Note: This web site is only kept up to date for OSG Software 1.2 (VDT 2.0.0). If you are looking for information for the most recent release, the RPM-based OSG Software 3.0, please see the OSG documentation web site

Terminology

Update
A new lettered version in a VDT series. 1.10.1, 1.10.1a, and 1.10.1b are all different updates within the 1.10.1 release. Generally speaking users always want the newest version of an update within a release. Old updates are not necessarily available. Also can be a verb, meaning "to switch an existing install to a newer (probably newest) update."
Release
A numbered version of the VDT, inclusive of all lettered updates. 1.9.1, 1.10.0, and 1.10.1 are all different releases. Old releases should generally remain available, and may receive further updates.
Upgrade
To go from an older VDT release to a newer VDT release.

Goals

We're considering distributing the VDT as a set of RPMs or debs. Package management would likely be via YUM (for RPMs) or APT (deb or RPM).

The big goal is to support upgrading from one VDT release (say, 1.10.0) to another (1.10.1).

Currently, this is done by making a fresh install of the new release in a new directory, then copying some of the configuration from the old install. This should be possible using RPM/deb, but is messy. It should be possible to handle this directly as an upgrade using the package system.

Smaller goals and assumptions:

General strategy

No matter which packaging system we use, there will be a package manager capable of automatically handling dependencies. This will allow a user to say "install Condor and Globus from the VDT," and actually have the VDT prerequisites automatically installed. The package manager can also handle commands like, "update my install" and "upgrade my install". The "commands" might be more complex than that in practice. For example, upgrading an install may require installing a new package by hand, then requesting an update. For RPM, we're talking about YUM (although APT is an option). For deb, this is built into the core system and is available via a number of tools, for for the sake of argument we'll use APT; a wildly available command line tool set, most notably apt-get.

One strategy that we are rejecting is having a single repository that contains all releases of the VDT. However, for completeness, this is what such a design would look like. We would be to incorporate the VDT release number into the name of the package itself. So the Condor package might be "VDT-1.10.1-Condor-7.0.2-1". The version of the package is "7.0.2-1" while the name of the package is "VDT-1.10.1-Condor". Installed packages would have the VDT version number in their directory name so that different versions could live side-by-side. So you might have /opt/vdt-1.10.1/condor and /opt/vdt-1.10.2/condor. This would support the old upgrade model where you do a side-by-side install, move configuration files, and uninstall the old release. This would be messy (you'd have the VDT version embedded in package names and directory names), fails to take advantage of upgrade capabilities available in RPM and deb, and is generally clumsy.

The plan we are going with is multiple repositories, one per releases. To upgrade, a user switches from the old repository to the new one. The details of how this is done vary from system to system. We could provide an RPM/deb with the information (Fedora Core style), an specialized tool to do it (Ubuntu style), or tell users to do it by hand (Debian style). There is nothing stopping us from using any of those three systems in all cases.

The upgrade solutions various distributions use are more complex than we need to worry about. The biggest issue we avoid is upgrading dependencies that the installer is built upon. For example, a user upgrading a Red Hat desktop from the desktop is relying on his GUI, libraries, and RPM to keep working throughout the upgrade. We're not providing or upgrading these core elements, so we don't need to worry about it.

Multiple simultaneous versions

Sometimes we need to support multiple versions of a package inside the same release. Known cases are the JDK (1.4, 1.5, and 1.6), MySQL (4 and 5), and Tomcat (5.0 and 5.5). The strategy used by Red Hat and Debian matches what the VDT is already doing: encode the major version number in the package name and provide one package per version. So we will continue to have a package called "JDK-1.4" next to a package called "JDK-1.5".

Debian and Red Hat both support the idea of /etc/alternates, a directory of symlinks for picking the "current" version of a package where multiple versions might be installed. Other binary locations (like /usr/bin) will link into /etc/alternates. So /usr/bin/java symlinks to /etc/alternates/java, which in turn symlinks to /usr/lib/jvm/jre-1.4.2-gcj/bin/java. The primary advantage is that changing which of several options is used is all collected in one place. We could support this, or something similar (say, $VDT_LOCATION/alternates), but it seems overly clever for our needs.

RPM under YUM

The current system, implemented by Red Hat, is to keep the repository information (and thus the release) in /etc/yum.repos.d/. The repository information itself is from an RPM package. To upgrade, a user hand-installs the repository information for the next release with something like "rpm -Uvh http://servername/packagename.rpm". Now "yum update" should find the new repository and handle the update.

Yum has simple but servicable support for multiple architectures. It's really in place so that a x686 system can intelligently get either i686 or i386 binaries, preferring the former. Yum has no direct support for different operating systems. We'll need to handle it ourselves, one repository per OS, using symlinks as necessary to share a single package between different OSes.

We'll need to carefully set the "Requires" and "Obsoletes" in the RPMs.

User modified files can be handled in one of two ways: we can replace them, saving the old version in $originalname.rpmsave, or leave them in place and save the new version $originalname.rpmnew.

Things to be wary of:

deb under APT

(We could also potentially do RPM under APT. Some distributions actually do this, but I don't believe any of the ones we support do.)

Out of the box, sources.list on Debian comes from the "apt" package; upgrading it by itself seems risky.

Step one is updating /etc/apt/sources.list, switching from the old repository to the new one. Then simply "apt-get dist-upgrade".

Debian recommends updating sources.list by hand, then using aptitude instead of apt-get as it "makes safer decisions".

(APT does support a configuration directory, which may be a better bet for us.)

Ubuntu uses dist-upgrader, which is a Python script that does the rewrite and manages the upgrade. There is additional cleverness built in, but at its core it rewrites sources.list and triggers an upgrade. dist-upgrader is provided as an external tarball that the GUI downloads, unpacks, and runs. This keeps the upgrade code entirely separate from the packaging system.

Things to be wary of:


Notes

Debian upgrade notes:
http://www.cyberciti.biz/tips/upgrading-debian-sarga-3-to-etch-4.html
http://www.debianadmin.com/upgrade-sarge-to-etch.html
http://www.debian.org/releases/stable/i386/release-notes/ch-upgrading

Debian packaging notes:
http://www.ibm.com/developerworks/linux/library/l-debpkg.html
 official, with a focus on being Debian compliant.
http://www.debian.org/doc/debian-policy/

Debian caches(?)
- have a "BrokenCount".  Should be 0.  Refused upgrade if non-zero.  Is this fixable?

Ubuntu design:
- Encode list of no-longer-present packages and add them to remove list.
- Has a blacklist of packages to never remove (but can upgrade).  More of a
  sanity check, I think.


- Check disk space before.  (Should be handled by Yum/dpkg automatically?)


General discussion of how Ubuntu did it (this is a proposal, but appears to
roughly match what happened with dist-upgrade.
https://wiki.ubuntu.com/AutomaticUpgrade


Yum automatically provides the architecture in $arch.  This is the 4's
return value from os.uname().  'i686' is typical.  It also provides
$basearch, based on arch.py.  This is a hard coded list of architectures
from most specific to least.  All chains end in "noarch" (but that's hidden
for $basearch).  'i386' would be the result from 'i686'.
- Yum can install both 32 and 64 bit binaries on 64-bit machines. How is
  this handled?

  

Fedora Core Upgrades

Debian Upgrades

APT using deb packages.

Ubuntu Upgrades

Ubuntu is built on Debian, but adds their own system to simplify upgrades.

AIX Build Problems

We can't use RPM 3.0.5 on AIX, so we'll need to build our own. While investigating, I learned the following:

beecrypt

You'll need beecrypt download.

Once you've configured beecrypt, edit config.h to set HAVE_SYNCH_H to 0. Then edit the Makefile to set LIBS to add "-lpthread"

RPM

export CPPFLAGS=-I/home/adesmet/rpm/install/include
export LDFLAGS=-L/home/adesmet/rpm/install/lib
./configure --prefix=/home/adesmet/rpm/install 

Current: missing rpcgen. Should be available on AIX.

checking for rpcgen... missing_rpcgen
configure: error: No rpcgen utility found.
mv: cannot rename Makefile to Makefile.orig: 
No such file or directory
cat: cannot open Makefile.orig
mv: cannot rename db.h to db.h.orig: 
No such file or directory
cat: cannot open db.h.orig
configure: error: ./configure failed for db3

Open