Note: This web site is only kept up to date for OSG Software
1.2 (VDT 2.0.0). If you are looking for information for the most recent
release, the RPM-based OSG Software 3.0, please see
the OSG documentation web site
There's a lot of crud to keep track of when working on the VDT. Just
editing file out in afs somewhere would be way more trouble then it's
worth. To help us out we have a custom build system called
. It takes in a few different source files, does some
magic and generates a (hopefully) working cache.
All our pacman files are stored in the VDT CVS
. If you check out the VDT and take a look at the top
level directory you'll see a few things:
- This is the build script that puts it all together
- defs files are used by make-vdt to store configuration
info. The syntax should be identical to Condor config files but
there's probably some edge cases somewhere. The top level defs file
includes information common to all packages, like the VDT version.
- make-vdt has hooks to run scripts at a couple places in
the build process. Common build scripts go here.
- Lots of other directories
- In the build system, each pacman package lives in it's own
directory. The directory and the package need to have the same
Now take a look inside one of the package directories. How about
- This is the source pacman file. It's pretty different from what
you'll find out in the cache.
- This is the second level defs file. It contains information
specific to the Condor package. Second level defs files are read
after top level defs files and can override any information they
- The questions file. The questions
system has its own documentation.
- This directory contains the files that become part of the Condor
package. make-vdt tars them up for you.
Lets say you just modified an existing file. What do you do now?
Hopefully this won't be too painful.
make-vdt is pretty simple for the operator. There are a
couple of options, but most of them time you won't need them.
vail(nmueller): ../make-vdt --help
Usage: ../make-vdt [options] <dir>
--test [test name]
Or: ../make-vdt --defs <defs file>
Or: ../make-vdt --nmi <dir>
If you changed something in the KX509 package you can install it
vail(nmueller): ./make-vdt KX509/
Creating KX509-Questions package...
Updating pacman file at /p/vdt/public/html/vdt_131_cache/KX509.pacman...
Running post-build script ../build-scripts/copy-defs.pl...
Running post-build script ../build-scripts/copy-questions.pl...
And that's it. Sometimes make-vdt will fail to notice any
differences between the installed package and the source. If that
happens just give it -f:
vail(nmueller): ./make-vdt KX509/
No changes detected in KX509/. Run me again with --force if you know something I don't.
vail(nmueller): ./make-vdt KX509/ -f
If you want to try your changes out before installing them in the
main cache you can use the -t option. Let's say I make a change to
vail(nmueller): vi PyGlobus.pacman
vail(nmueller): ../make-vdt -t .
Updating pacman file at /p/vdt/public/html/test-cache/nmueller/PyGlobus.pacman...
I can now point pacman at
http://vdt.cs.wisc.edu/test-cache/nmueller and do a test install. If
I gave a test name I'd point it to
http://vdt.cs.wisc.edu/test-cache/<test name>. Note: this still only copies in the current
package. You'll probably want to seed your test cache by copying in
files from the production cache.
Packages for make-vdt
are regular pacman files with a few
additions. To keep things simple we'll call the source file format
VPL (VDT Package Language) and the compiled format Eggshell. VPL is
just a superset of Eggshell, so (if you really want) you can ignore
the rest of this section and just check out the Eggshell
- Macro substitution
- You can include the value of any defs macro with !!<macro>!!. For example, the first line of VDT.pacman is:
description('Virtual Data Toolkit !!VDT_VERSION!!')
In the installed file !!VDT_VERSION!! will be replaced with the VDT version number from the defs file.
- Proper if-then-else syntax
- Eggshell comes with three things -- atoms, a short-circuited or and a short-circuited and. You can make more complex struc
tures out of these, but it's a real mess. VPL comes with Perl style if-then-else syntax.
- Proper boolean operators
- The Eggshell and operator is ';' and the or operator is 'OR'. VPL adds the and/or operators from Perl ('and', 'or', '&&',
- User defined atoms
- VPL allows the user to define and use new atoms, provided they can compile down to raw Eggshell. New atoms are defined wit
h perl modules in vdt_src/build-scripts/atoms. The package name is the same as the atom name, and they most contain a fun
ction gen. make-vdt passes &gen an array reference containing the arguments used and a hash containing the content
s of the defs files. &gen must return a string of VPL representing the content of the atom. Current atoms include:
- vdt_download() - takes a hash of platform names and files. Does the appropriate download.
- shellDialog() - corrects the spelling of shellDialogue().
- path() - redefines CORE::path(). If you set LD_LIBRARY_PATH it will set DYLD_LIBRARY_PATH on MacOSX. It will also add t
he third 'no-check' argument to CORE::path().
- You probably don't need an uninstall section. make-vdt
will generate an uninstall section that cleans up files with
vdt-uninstall and does a vdt-version --remove.
Most of the differences show up if you're using make-vdt
generate tarballs out of CVS. Some of our packages come from NMI
(Globus, MyProxy...), some come from colaborators (VDS, DRM...) and
some come from us (VDT-Version-Info, Configure-Globus-Server...).
Anything that comes from us needs to be in CVS and managed by
. Lets take a look at VDT-Install-Service as an
example. The defs file defines TOPDIR
, which make-vdt
to tar up the sources. Any files included in this package need to be
will take the tarball filename you specified and tack a patch version on to the end. Because of that, you can't refer to the tarball directly in the pacman file:
description('VDT Service Installer')
Instead of untarring vdt-install-service-1.tar.gz
you need to
. The actually tarball filename will
change every time you run make-vdt
. Additionally, you
shouldn't put in a call to download() or vdt_download() --
will do that for you.
also includes hooks into the nmi build system. Since it's still a little early (and rough around the edges) you need to set up a bunch of ssh stuff first.
- Generate a public/private key pair from your CS account
vail(nmueller): ssh-keygen -t dsa -b 4096
Generating public/private dsa key pair.
Enter file in which to save the key (/u/n/m/nmueller/.ssh/id_dsa):
Created directory '/u/n/m/nmueller/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /u/n/m/nmueller/.ssh/id_dsa.
Your public key has been saved in /u/n/m/nmueller/.ssh/id_dsa.pub.
The key fingerprint is:
- scp the new key to grandcentral
vail(nmueller): scp ~/.ssh/id_dsa.pub grandcentral:.ssh/authorized_keys
- Test it out. You should be able to ssh to grandcentral without entering a password.
- Generate a public/private key pair from your grandcentral account
- scp the new key back to your CS account
grandcentral(nmueller): scp ~/.ssh/id_dsa.pub vail:
- Install the new key in your CS account
vail(nmueller): mkdir ~/.ssh/public
vail(nmueller): chmod 700 ~/.ssh/public
vail(nmueller): fs sa ~/.ssh system:anyuser l
vail(nmueller): fs sa ~/.ssh/public system:anyuser read
vail(nmueller): mv ~/id_dsa.pub ~/.ssh/public/authorized_keys
vail(nmueller): ln -s ~/.ssh/public/authorized_keys ~/.ssh
- Test it out. You should be able to ssh from grandcentral to your CS account without entering a password.
Now that that's over you can get around to building NMI packages.
Take a look at a package that's built with NMI.
vail(nmueller): ls MySQL/nmi/
build-mysql.pl* cmdfile CVS/ glue.ftp mysql.ftp
You'll see that the MySQL package contains an nmi subdirectory. That
directory must contain a file called cmdfile
submits your build it will do so by copying
over to grandcentral and passing it on to the NMI
system. These files are standard NMI files with a few optional VDT
to learn more about vanilla NMI files.
To build an NMI package just run make-vdt with the --nmi option. make-vdt will read in your cmdfile, change a few things, copy everything you need to grandcentral and submit the build.
vail(nmueller): ../make-vdt -nmi .
Starting NMI build for MySQL...
cmdfile 100% 441 0.4KB/s 00:00
glue.ftp 100% 110 0.1KB/s 00:00
mysql.ftp 100% 122 0.1KB/s 00:00
Logging submit event(s).
1 job(s) submitted to cluster 27383.
Checking all your submit files for log file names.
This might take a while...
File for submitting this DAG to Condor : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.dag.condor.sub
Log of DAGMan debugging messages : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.dag.dagman.out
Log of Condor library debug messages : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.dag.lib.out
Log of the life of condor_dagman itself : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.dag.dagman.log
Condor Log file for all jobs of this DAG : /nmi/run/nmueller_grandcentral.cs.wisc.edu_1105615631_16109/run.log
gid = nmueller_grandcentral.cs.wisc.edu_1105615631_16109
runid = 1816
To make your life easier make-vdt adds a few extra features to a normal NMI build.
- All files in the nmi directory undergo variable substitution the
same way as pacman files (!!<variable>!!).
- A library of commonly used functions is provided for
you. VDTGlue.pm is transfered by default -- you just need to "use
VDTGlue" in your build script. It's current functions include:
- An overloaded system function to die in case of error.
- load_defs which reads the defs file shipped with the job.
- unpack_gpt_bundle(bundle) will recursivly untar a GPT bundle.
- fix_la_files(path) fixes all the .la files in the given path.
- A new parameter for ftp method input files -- vdt_files.
Instead of taring up your glue scripts every time you run
make-vdt, just specify a comma seperated list in
vdt_files and leave out ftp_root and
ftp_target. make-vdt will tar up the files for you
and fill in the necessary information.
- A new parameter for nmi method input files --
vdt_component. If your build relys on a previous NMI build
specify the component with vdt_component instead of hard
coding in the runid. make-vdt will search the NMI database
and prompt you for a runid when you submit the build.
- No more writing one off remote_declare scripts. Just specify
the task list in your cmdfile with the new vdt_declare_tasks
- make-vdt build can ship files back to /p/vdt when the
build completes. In your cmdfile just specify
vdt_result (wildcards can be used) and vdt_dest. The
file specified in vdt_result will be transfered back to
vdt_dest if the build completes successfully. You don't need
to wait around and do it by hand.
There are two classes of options for defs files. The first are those understood by make-vdt
- ALT_CACHE = <cache name>
- Some packages (CA-Certificates) are versioned independantly of the VDT and installed in a seperate cache. If ALT_CACHE is defined in a package's second level defs files the package will be installed in $(ROOT)/$(ALT_CACHE).
- ALT_CACHE_BRANCH = <branch name>
- ALT_CACHE packages are shared between multiple VDT versions, but they sill need one source location in CVS. This macro lets you specify the "real" branch for the package. If you are not editing this branch make-vdt will create a symlink from the current branch to the ALT_CACHE_BRANCH for the package in which this is defined.
- ALLOW_X86_ON_X86_64 = 1
- If this is equal to 1 in a package's defs file, then x86_64 platforms will use x86 binaries if no x86_64 binaries are available. It is possible for this to overlap with unsupported packages. In that case, the x86 binaries can be used if there are no x86_64 binaries, and the ALLOW_UNSUPPORTED flag is passed to pacman.
- INCLUDE = <filename>
- When the parser encounters an include macro it opens up the specified file and treats it as if it were #included.
- LOCK_FILE = <filename>
- Filename of universal lock file below ROOT. Since make-vdt needs to parse installed packages it can't be used by more then one person at a time. You need to have a shared lock file to prevent random tarballs from being deleted.
- NATIVE_DIR = <directory>
- Path to the native package directory below ROOT.
- NMI_DIR = <directory>
- Path to the nmi directory below ROOT.
- PACMAN_VERSION = <version>
- Required Pacman version for the release.
- PLATFORM_ALIAS = <real platform>#<source platform>, <real platform>#<source platform>...
- Pacman 2 doesn't handle platforms as well as we'd like. If you take a look at an installed package (say Globus-Base-RM-Server) you'll see that it has about a dozen download options and most are the same. To avoid redundancy in the source files make-vdt will default platforms with the data in the PLATFORM_ALIAS macro.
- PLATFORM_MAP = <real platform>=<source platforms>/<real platform>#<source platforms>/...
- The PLATFORM_MAP, as well as the specific declarations above it (PLATFORM_DEB_3_1, PLATFORM_FC_3, etc) are used to map real platforms to the preferred sources for tarballs. Each line for the specific platforms declares an ordered list of the preferred tarball build platforms. The PLATFORM_MAP line is a concatenates all this information, as well as associating the full platform name.
- POST_BUILD_SCRIPT = <command>, <command>...
- Any post build scripts are run after all work is done. They're only run if make-vdt detects any changes
- PRE_BUILD_SCRIPT = <command>, <command>...
- Pre build scripts are run before make-vdt looks for changes.
- ROOT = <directory>
- The root directory of the installation destination
- RPM_ARCHES = <rpm arch>#<pacman arch>, <rpm arch>#<pacman arch>...
- This isn't really used yet. Don't worry about it.
- SEMI_SUPPORTED = <platform>, <platform>...
- A list of semi-supported platforms. If a package does not
specify a download for one of the platforms a nativelyInstalled
section will be added to skip the install on that platform.
- SOFTWARE_DIR = <directory>
- Path to the software directory below ROOT.
- TARBALL_FILE = <package>-<version>.tar.gz
- The name of the tarball file make-vdt should create. This option is only respected in second level defs files.
- TEST_DIR = <directory>
- Path to the test cache directory below ROOT.
- TOPDIR = <directory>
- The top level directory for make-vdt to tar up. This option is only respected in second level defs files.
- USER_ATOMS = <directory>
- Path under vdt_src to the user-defined atom modules.
- VDT_VERSION = <version>
- make-vdt uses the VDT version to figure out where to put pacman files and other things.
- WEB_ROOT = <url>
- URL of ROOT
The second set is made up of options used by various pre and post
install script and the web site.
- CONTENTS_HTML = <package>, <package>...
- Any packages in CONTENTS_HTML will be added to the release's contents.html file.
- These are all used to generate contents.html and
vdt-version.info, as well as individual pacman files.
- VDT_VERSION_INFO = <package>, <package>...
- Any packages VDT_VERSION_INFO will be added to the vdt-version.info file.