Note: This web site is only kept up to date for OSG Software 1.2 (VDT 2.0.0). If you are looking for information for the most recent release, the RPM-based OSG Software 3.0, please see the OSG documentation web site

Status of Native Packaging 2010/08/11

TODO List:

Packages needed to complete Glexec:

Testing

Assuming I have a GUMS server set up, I can currently test running /usr/bin/id via glexec. That's about it.

Ideally I should be able to make a package for VDT-Certification-Tests, and run those.

I need to test dependencies--right now my testing script ignores dependency checks. The dependency lists are generated as part of the build, just not used. I also need to replace packages like Curl and Expat with the distro-supplied versions and see if things work.

More Platforms

Currently packages only exist for x86_rhap5 and have only been tested on centos 5 x86. 64-bit packages would be nice, and last I checked we're eventually going to do debs.

Polish

Move things to /opt/vdt instead of /opt/vdt/glexec-single. Descriptions, summaries, license information. FHS / pseudo-FHS compliance.

Generalizability

In theory I can make RPMs out of any VDT package now. For NMI builds it's easy to add a step to create the RPM during the bundling stage. For non-NMI builds the wizard script generates enough to get started. Some manual work has to be done for packages that have environment variables or post-install steps. However, exceptions always pop up, and in some cases reorganization is useful--moving configure scripts into the main package, running them at build time if possible.

Build Process

  1. Check out file:///p/vdt/workspace/svn/vdt/branches/vdt-2.0.0-native from SVN.
  2. Run "make-nmi-rpm <testcache> <package>" for each package you want to rebuild. This is a shorthand for "make-vdt --nmi --no-transfer --transfer-rpm --test $testcache $package"
  3. Find the results in /p/vdt/public/html/test-nmi-rpm/$testcache/$nmi_platform/*.rpm

Install/Test Process

I've written a script named "glexec-single-rpm-install-test.pl" that will do the installation part of the testing. Copy it and the RPMs into a directory on a vdt test machine, make sure chrpath is installed, and run the script as root.

The script will clean up any RPMs from the old version. More specifically, it uninstalls RPMs in the 'Applications/Grid' group. N.B.: This is the same group the LIGO Globus packages are in, so they'll get uninstalled too.

Once the RPMs are uninstalled, the script will blow away any remnants in the /opt/vdt/glexec-single directory.

Then the script installs the new RPMs one by one, not doing any dependency checking. The new files will end up in /opt/vdt/glexec-single.

Finally it sets the gums_url in /etc/glexec/contrib/gums_interface/getmapping.cfg and provides some instructions for the next step.

Code Overview

make-vdt

I added two things to make-vdt. First is the vdt_software input method for transferring files to an NMI build. You put this in your foo.ftp file. An example is VDT-Base-Native/nmi/vdt-common.ftp:

method = ftp
vdt_software = vdt-common/!!VDT_VERSION!!/vdt-common-*.tar.gz
untar = true

This will load the input files from the /p/vdt/public/html/software tree. The reason I added this instead of using existing mechanisms like vdt_files, is that those don't expand globs. I have no idea ahead of time what the exact name of the vdt-common tarball that I'm going to use is, so I needed globs.

The second thing I added to make-vdt is vdt_rpm_result. You put that in your nmi/cmdfile. For VDT-Base-Native, it's

vdt_rpm_result = vdt-base-!!VDT_FULL_VERSION!!-*.*.rpm

and make-vdt will handle putting the RPM in its proper place.

VDTGlue.pm

I added some code to handle transferring the RPMs back to their proper place--make-vdt actually calls VDTGlue.pm (invoking it as a script!) to handle parts of the transfer. It did that for the regular vdt_result, so I used the same thing for vdt_rpm_result.

Most of the additions are in conf_to_spec, which creates an RPM spec file from a native.conf file. The native.conf file format is used for specifying metadata (e.g. license, or description), but also has mechanisms for running build-time scripts, post-install scripts, moving files, setting permissions, declaring config files, creating empty directories (e.g. /var/log/glexec), and creating a setup.sh analogue.

Build-time scripts are used for running configure_glexec during packaging, for example. The file movement functionality was added so files in $VDT_LOCATION/glexec-osg/etc would get moved to /etc/glexec.

The setupsh field in the native.conf file is used to construct a setup file that ends up in $VDT_LOCATION/setup.d/. This file is sourced whenever setup.sh is sourced. There is a 'dependency' mechanism that can be used to order how the files in setup.d are sourced.

pacman2native-wizard.pl

You run this from a checkout of the VDT and pass it a package directory, e.g. pacman2native-wizard.pl Fetch-CRL. The script will create a new package dir, e.g. Fetch-CRL-Native, which contains a defs file, and an nmi subdirectory with a cmdfile, input files, a native.conf, and an nmi-remote-task script. These are based off of the defs file and the pacman file in the original package. The input file uses vdt_download to get the software to repackage.

package commands in the pacman file get turned in to prereqs in the native.conf file. path and setenv commands are put in the setupsh section in the native.conf file. shell commands end up in the post-install script.

Note that I've only used this script for Fetch-CRL and Procd.