Note: This web site is only kept up to date for OSG Software 1.2 (VDT 2.0.0). If you are looking for information for the most recent release, the RPM-based OSG Software 3.0, please see the OSG documentation web site

Introduction

There's a lot of crud to keep track of when working on the VDT. Just editing file out in afs somewhere would be way more trouble then it's worth. To help us out we have a custom build system called make-vdt. It takes in a few different source files, does some magic and generates a (hopefully) working cache.

Basics

All our pacman files are stored in the VDT CVS repository. If you check out the VDT and take a look at the top level directory you'll see a few things:
make-vdt
This is the build script that puts it all together
defs
defs files are used by make-vdt to store configuration info. The syntax should be identical to Condor config files but there's probably some edge cases somewhere. The top level defs file includes information common to all packages, like the VDT version.
build-scripts
make-vdt has hooks to run scripts at a couple places in the build process. Common build scripts go here.
Lots of other directories
In the build system, each pacman package lives in it's own directory. The directory and the package need to have the same name.
Now take a look inside one of the package directories. How about Condor:
Condor.pacman
This is the source pacman file. It's pretty different from what you'll find out in the cache.
defs
This is the second level defs file. It contains information specific to the Condor package. Second level defs files are read after top level defs files and can override any information they contain.
questions.xml
The questions file. The questions system has its own documentation.
vdt_condor_install
This directory contains the files that become part of the Condor package. make-vdt tars them up for you.

Running It

Lets say you just modified an existing file. What do you do now? Hopefully this won't be too painful.

make-vdt is pretty simple for the operator. There are a couple of options, but most of them time you won't need them.

vail(nmueller): ../make-vdt --help
Usage: ../make-vdt [options] <dir>
        --force
        --test [test name]

Or: ../make-vdt --defs <defs file>

Or: ../make-vdt --nmi <dir>

If you changed something in the KX509 package you can install it pretty easilly.

vail(nmueller): ./make-vdt KX509/
Creating KX509-Questions package...
vdt/
vdt/setup/
vdt/setup/questions/
vdt/setup/questions/Licenses.xml
Updating pacman file at /p/vdt/public/html/vdt_131_cache/KX509.pacman...
Running post-build script ../build-scripts/copy-defs.pl...
Running post-build script ../build-scripts/copy-questions.pl...

And that's it. Sometimes make-vdt will fail to notice any differences between the installed package and the source. If that happens just give it -f:

vail(nmueller): ./make-vdt KX509/ 
No changes detected in KX509/.  Run me again with --force if you know something I don't.
vail(nmueller): ./make-vdt KX509/ -f
...

If you want to try your changes out before installing them in the main cache you can use the -t option. Let's say I make a change to PyGlobus:

vail(nmueller): vi PyGlobus.pacman 
vail(nmueller): ../make-vdt -t .
Updating pacman file at /p/vdt/public/html/test-cache/nmueller/PyGlobus.pacman...

I can now point pacman at http://vdt.cs.wisc.edu/test-cache/nmueller and do a test install. If I gave a test name I'd point it to http://vdt.cs.wisc.edu/test-cache/<test name>. Note: this still only copies in the current package. You'll probably want to seed your test cache by copying in files from the production cache.

Writing Packages

Packages for make-vdt are regular pacman files with a few additions. To keep things simple we'll call the source file format VPL (VDT Package Language) and the compiled format Eggshell. VPL is just a superset of Eggshell, so (if you really want) you can ignore the rest of this section and just check out the Eggshell documentation.
Macro substitution
You can include the value of any defs macro with !!<macro>!!. For example, the first line of VDT.pacman is:
description('Virtual Data Toolkit !!VDT_VERSION!!')
In the installed file !!VDT_VERSION!! will be replaced with the VDT version number from the defs file.
Proper if-then-else syntax
Eggshell comes with three things -- atoms, a short-circuited or and a short-circuited and. You can make more complex struc tures out of these, but it's a real mess. VPL comes with Perl style if-then-else syntax.
Proper boolean operators
The Eggshell and operator is ';' and the or operator is 'OR'. VPL adds the and/or operators from Perl ('and', 'or', '&&', '||').
User defined atoms
VPL allows the user to define and use new atoms, provided they can compile down to raw Eggshell. New atoms are defined wit h perl modules in vdt_src/build-scripts/atoms. The package name is the same as the atom name, and they most contain a fun ction gen. make-vdt passes &gen an array reference containing the arguments used and a hash containing the content s of the defs files. &gen must return a string of VPL representing the content of the atom. Current atoms include:
  • vdt_download() - takes a hash of platform names and files. Does the appropriate download.
  • shellDialog() - corrects the spelling of shellDialogue().
  • path() - redefines CORE::path(). If you set LD_LIBRARY_PATH it will set DYLD_LIBRARY_PATH on MacOSX. It will also add t he third 'no-check' argument to CORE::path().
Uninstallation.
You probably don't need an uninstall section. make-vdt will generate an uninstall section that cleans up files with vdt-uninstall and does a vdt-version --remove.

Tarball Generation

Most of the differences show up if you're using make-vdt to generate tarballs out of CVS. Some of our packages come from NMI (Globus, MyProxy...), some come from colaborators (VDS, DRM...) and some come from us (VDT-Version-Info, Configure-Globus-Server...). Anything that comes from us needs to be in CVS and managed by make-vdt. Lets take a look at VDT-Install-Service as an example. The defs file defines TOPDIR and TARBALL_FILE, which make-vdt will use to tar up the sources. Any files included in this package need to be in vdt_src/VDT-Install-Service/vdt.
TOPDIR=vdt
TARBALL_FILE=vdt-install-service-1.tar.gz
make-vdt will take the tarball filename you specified and tack a patch version on to the end. Because of that, you can't refer to the tarball directly in the pacman file:
description('VDT Service Installer')
url('!!WEB_ROOT!!')

shell('vdt-begin-install VDT-Install-Service')
shell('vdt-untar !!TARBALL_FILE!!')
shell('vdt-end-install')
Instead of untarring vdt-install-service-1.tar.gz you need to untar !!TARBALL_FILE!!. The actually tarball filename will change every time you run make-vdt. Additionally, you shouldn't put in a call to download() or vdt_download() -- make-vdt will do that for you.

NMI Integration

SSH Prereqs

make-vdt also includes hooks into the nmi build system. Since it's still a little early (and rough around the edges) you need to set up a bunch of ssh stuff first.
  1. Generate a public/private key pair from your CS account
    vail(nmueller): ssh-keygen -t dsa -b 4096
    Generating public/private dsa key pair.
    
    Enter file in which to save the key (/u/n/m/nmueller/.ssh/id_dsa):
    
    Created directory '/u/n/m/nmueller/.ssh'.
    
    Enter passphrase (empty for no passphrase):
    
    Enter same passphrase again:
    
    Your identification has been saved in /u/n/m/nmueller/.ssh/id_dsa.
    
    Your public key has been saved in /u/n/m/nmueller/.ssh/id_dsa.pub.
    
    The key fingerprint is:
    
    9a:93:4b:6c:7a:4d:df:96:9f:cd:20:7c:a7:8b:18:7e nmueller@vail.cs.wisc.edu
    
  2. scp the new key to grandcentral
    vail(nmueller): scp ~/.ssh/id_dsa.pub grandcentral:.ssh/authorized_keys
    
  3. Test it out. You should be able to ssh to grandcentral without entering a password.
  4. Generate a public/private key pair from your grandcentral account
  5. scp the new key back to your CS account
    grandcentral(nmueller): scp ~/.ssh/id_dsa.pub vail:
    
  6. Install the new key in your CS account
    vail(nmueller): mkdir ~/.ssh/public
    vail(nmueller): chmod 700 ~/.ssh/public
    vail(nmueller): fs sa ~/.ssh system:anyuser l
    vail(nmueller): fs sa ~/.ssh/public system:anyuser read
    vail(nmueller): mv ~/id_dsa.pub ~/.ssh/public/authorized_keys
    vail(nmueller): ln -s ~/.ssh/public/authorized_keys ~/.ssh
    
  7. Test it out. You should be able to ssh from grandcentral to your CS account without entering a password.

Basics

Now that that's over you can get around to building NMI packages. Take a look at a package that's built with NMI.

vail(nmueller): ls MySQL/nmi/
build-mysql.pl*  cmdfile  CVS/  glue.ftp  mysql.ftp
You'll see that the MySQL package contains an nmi subdirectory. That directory must contain a file called cmdfile. When make-vdt submits your build it will do so by copying cmdfile over to grandcentral and passing it on to the NMI system. These files are standard NMI files with a few optional VDT additions. See their documentation to learn more about vanilla NMI files.

To build an NMI package just run make-vdt with the --nmi option. make-vdt will read in your cmdfile, change a few things, copy everything you need to grandcentral and submit the build.

vail(nmueller): ../make-vdt -nmi .
Starting NMI build for MySQL...
cmdfile                                       100%  441     0.4KB/s   00:00    
glue.ftp                                      100%  110     0.1KB/s   00:00    
mysql.ftp                                     100%  122     0.1KB/s   00:00    
Submitting job(s).
Logging submit event(s).
1 job(s) submitted to cluster 27383.

Checking all your submit files for log file names.
This might take a while... 
Done.
-----------------------------------------------------------------------
File for submitting this DAG to Condor           : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.dag.condor.sub
Log of DAGMan debugging messages                 : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.dag.dagman.out
Log of Condor library debug messages             : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.dag.lib.out
Log of the life of condor_dagman itself          : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.dag.dagman.log

Condor Log file for all jobs of this DAG         : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.log
-----------------------------------------------------------------------
/nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109
gid = nmueller_grandcentral.cs.wisc.edu_1105615631_16109
runid = 1816

VDT Additions

To make your life easier make-vdt adds a few extra features to a normal NMI build.

defs Reference

There are two classes of options for defs files. The first are those understood by make-vdt itself.
ALT_CACHE = <cache name>
Some packages (CA-Certificates) are versioned independantly of the VDT and installed in a seperate cache. If ALT_CACHE is defined in a package's second level defs files the package will be installed in $(ROOT)/$(ALT_CACHE).
ALT_CACHE_BRANCH = <branch name>
ALT_CACHE packages are shared between multiple VDT versions, but they sill need one source location in CVS. This macro lets you specify the "real" branch for the package. If you are not editing this branch make-vdt will create a symlink from the current branch to the ALT_CACHE_BRANCH for the package in which this is defined.
ALLOW_X86_ON_X86_64 = 1
If this is equal to 1 in a package's defs file, then x86_64 platforms will use x86 binaries if no x86_64 binaries are available. It is possible for this to overlap with unsupported packages. In that case, the x86 binaries can be used if there are no x86_64 binaries, and the ALLOW_UNSUPPORTED flag is passed to pacman.
INCLUDE = <filename>
When the parser encounters an include macro it opens up the specified file and treats it as if it were #included.
LOCK_FILE = <filename>
Filename of universal lock file below ROOT. Since make-vdt needs to parse installed packages it can't be used by more then one person at a time. You need to have a shared lock file to prevent random tarballs from being deleted.
NATIVE_DIR = <directory>
Path to the native package directory below ROOT.
NMI_DIR = <directory>
Path to the nmi directory below ROOT.
PACMAN_VERSION = <version>
Required Pacman version for the release.
PLATFORM_ALIAS = <real platform>#<source platform>, <real platform>#<source platform>...
Pacman 2 doesn't handle platforms as well as we'd like. If you take a look at an installed package (say Globus-Base-RM-Server) you'll see that it has about a dozen download options and most are the same. To avoid redundancy in the source files make-vdt will default platforms with the data in the PLATFORM_ALIAS macro.
PLATFORM_MAP = <real platform>=<source platforms>/<real platform>#<source platforms>/...
The PLATFORM_MAP, as well as the specific declarations above it (PLATFORM_DEB_3_1, PLATFORM_FC_3, etc) are used to map real platforms to the preferred sources for tarballs. Each line for the specific platforms declares an ordered list of the preferred tarball build platforms. The PLATFORM_MAP line is a concatenates all this information, as well as associating the full platform name.
POST_BUILD_SCRIPT = <command>, <command>...
Any post build scripts are run after all work is done. They're only run if make-vdt detects any changes
PRE_BUILD_SCRIPT = <command>, <command>...
Pre build scripts are run before make-vdt looks for changes.
ROOT = <directory>
The root directory of the installation destination
RPM_ARCHES = <rpm arch>#<pacman arch>, <rpm arch>#<pacman arch>...
This isn't really used yet. Don't worry about it.
SEMI_SUPPORTED = <platform>, <platform>...
A list of semi-supported platforms. If a package does not specify a download for one of the platforms a nativelyInstalled section will be added to skip the install on that platform.
SOFTWARE_DIR = <directory>
Path to the software directory below ROOT.
TARBALL_FILE = <package>-<version>.tar.gz
The name of the tarball file make-vdt should create. This option is only respected in second level defs files.
TEST_DIR = <directory>
Path to the test cache directory below ROOT.
TOPDIR = <directory>
The top level directory for make-vdt to tar up. This option is only respected in second level defs files.
USER_ATOMS = <directory>
Path under vdt_src to the user-defined atom modules.
VDT_VERSION = <version>
make-vdt uses the VDT version to figure out where to put pacman files and other things.
WEB_ROOT = <url>
URL of ROOT
The second set is made up of options used by various pre and post install script and the web site.
CONTENTS_HTML = <package>, <package>...
Any packages in CONTENTS_HTML will be added to the release's contents.html file.
PACKAGE_DESC
PACKAGE_LONG_DESC
PACKAGE_URL
PACKAGE_VERSION
These are all used to generate contents.html and vdt-version.info, as well as individual pacman files.
VDT_VERSION_INFO = <package>, <package>...
Any packages VDT_VERSION_INFO will be added to the vdt-version.info file.