Note: This web site is only kept up to date for OSG Software 1.2 (VDT 2.0.0). If you are looking for information for the most recent release, the RPM-based OSG Software 3.0, please see the OSG documentation web site

Building Globus 4 and Friends

Overview

The VDT build of Globus Toolkit 4 and related components (GSI-OpenSSH, KX.509, MyProxy, UberFTP) is a complicated process that requires frequent manual intervention. Just building Globus is complicated, but then we patch the source code a lot, tweak and use the NMI build glue to perform many of the steps, and divide the resulting build into packages in a funny way.

Overall, the process of a build goes something like this:

  1. Pick a version of Globus and a set of patches
  2. Build Globus
  3. Build each of the other components (MyProxy, etc.) against the Globus build
  4. Package all of the Globus components
  5. Install into the VDT

Strap yourself in, hang on, here we go.

Preflight Checks

The first step is to pick a version of the Globus Toolkit 4 and a set of patches to apply to it. There are a few steps you can take now to ensure that you won’t waste (as much) time during the actual build process.

If Globus has released a new, stable version since the last release of the VDT, chances are high that you’ll want to build the new version. Of course, building a new version — especially a big release — is a lot more work. For reference, here are the Globus 4 versions that have shipped in the VDT:

Globus VDT
4.0.1 1.3.7 – 1.3.10
4.0.2 1.3.11 – 1.5.0
4.0.3 1.5.1 –

Selecting a set of patches is complicated. Historically, Alain has been in charge of deciding which patches go into a new build. Some patches change Globus functionality and are incorporated at the request of community members, some patches are forward-looking bug fixes that come from the Globus bugzilla system and will be fixed in a future release, and, starting with VDT 1.3.11, a large set of patches exist for TeraGrid interoperability and came from the “community branch”.

When changing the Globus version or the set of patches, it’s a good idea to try applying the patches to the Globus source code to make sure they apply cleanly. Better to find out quickly from the command line than after waiting hours for an NMI to run and fail.

Getting Globus Source Code

To check out a copy of the Globus source code, you can consult a Globus webpage that describes the process in detail. Or, here are shell commands that worked for Globus 4.0.3:

cvs -d :pserver:anonymous@cvs.globus.org:/home/globdev/CVS/globus-packages co -r globus_4_0_3 packaging
cd packaging
./make-packages.pl --t2=globus_4_0_3 --t4=globus_4_0_3 --anonymous --skippackage --skipbundle

Preparing Patches

The patch files should go in a directory specific to the VDT release:

/p/vdt/public/html/patches/VERSION

The convention for patch filenames is to start with the integer sequence number of the patch — that is, add one to the last known patch number — then two underscore characters, then a brief name for the patch. Make sure that each patch file has 0644 permissions, while you’re at it.

It makes sense to start selecting patches by copying all patch files from the previous VDT release. Then, add new patches to the end of the list. That way, it should be easy to figure out what sequence number to use next.

When adding patches, make sure that the paths to the patched files start at the source-trees level of the packaging directory you checked out. That’s where the build script will apply patches. Look for the lines that start with --- or +++. So, if you get a patch that starts like this:

RCS file: /home/globdev/CVS/globus-packages/replica/rls/server/rli.c,v
retrieving revision 1.11
retrieving revision 1.11.4.1
diff -u -r1.11 -r1.11.4.1
--- rli.c       2005/04/18 20:56:48     1.11
+++ rli.c       2006/05/23 18:12:04     1.11.4.1
@@ -86,7 +86,7 @@

You’ll have to change the paths to look like this:

RCS file: /home/globdev/CVS/globus-packages/replica/rls/server/rli.c,v
retrieving revision 1.11
retrieving revision 1.11.4.1
diff -u -r1.11 -r1.11.4.1
--- source-trees/gt2-cvs/replica/rls/server/rli.c       2005/04/18 20:56:48     1.11
+++ source-trees/gt2-cvs/replica/rls/server/rli.c       2006/05/23 18:12:04     1.11.4.1
@@ -86,7 +86,7 @@

Installing Patches

Ultimately, you need all patch files in a single, flat directory all rolled up into a tarball for the build process. Now that we have the community branch patches in a multi-level directory, making the correct tarball is not a one-line command anymore. Here’s a Ruby script that might help (probably fancier than need be, but it gets the job done). Note that it assumes you checked out the Globus source right in the patches directory.

#!/unsup/ruby/bin/ruby -W0

system('/bin/rm -rf packaging/patches') if File.directory?('packaging/patches')
system('mkdir -p packaging/patches')

Dir.glob('[0-9][0-9]__*') do |filename|
  if File.directory?(filename)
    sequence_number = filename.match(/^(\d+)/)[0]
    Dir.glob("#{filename}/**/*") do |subfilename|
      next if subfilename =~ %r{/pkgdata-unused}
      new_filename = sequence_number + '__' + File.basename(subfilename) + '.diff'
      puts "Copying #{subfilename} to packaging/patches/#{new_filename}"
      system("cp -p #{subfilename} packaging/patches/#{new_filename}")
      system("chmod 0644 packaging/patches/#{new_filename}")
    end
  else
    puts "Copying #{filename} to packaging/patches/#{filename}"
    system("cp -p #{filename} packaging/patches/#{filename}")
    system("chmod 0644 packaging/patches/#{filename}")
  end
end

system('tar czf vdt-patches.tar.gz packaging/patches')

Verifying Patches

To screen the patches, you have to apply them in order and look at the output. If a patch hunk applies with an offset, that’s OK. But if it applies with fuzz or fails to apply, you probably need to fix the patch or remove it. With a new Globus version, there’s a good chance you’ll remove lots of patches, because many bug fixes will have been released into the main code.

To apply a patch, change to your packaging directory that you checked out earlier and then:

patch -p0 < <PATH_TO_PATCH_FILE>

Output from a successful patch might look like this:

patching file source-trees/gt2-cvs/gram/jobmanager/source/globus_gram_job_manager.h
patching file source-trees/gt2-cvs/gram/jobmanager/source/globus_gram_job_manager_request.c
patching file source-trees/gt2-cvs/gram/jobmanager/source/main.c

or, if patches apply successfully with offsets:

(Stripping trailing CRs from patch.)
patching file source-trees/gt2-cvs/gram/jobmanager/source/globus_gram_job_manager_query.c
Hunk #2 succeeded at 548 (offset 1 line).
Hunk #3 succeeded at 712 (offset 17 lines).
Hunk #4 succeeded at 983 (offset 1 line).

A patch that applies with fuzz might look like this (note the minimal difference between this and a successful patch):

(Stripping trailing CRs from patch.)
patching file source-trees/gt2-cvs/gram/jobmanager/setup/condor/condor.in
Hunk #3 succeeded at 73 with fuzz 2.

Outright failures are easy to spot:

patching file source-trees/gt2-cvs/gram/jobmanager/setup/fork/fork.in
Hunk #1 FAILED at 3.
Hunk #3 succeeded at 229 (offset 25 lines).
1 out of 4 hunks FAILED -- saving rejects to file source-trees/gt2-cvs/gram/jobmanager/setup/fork/fork.in.rej

If a patch has been applied to the source code already, it might look like this (and you’ll have to reply to the questions to proceed):

patching file source-trees/wsrf-cvs/ws-gram/service/java/setup/condor/pkgdata/pkg_data_src.gpt.in
Reversed (or previously applied) patch detected!  Assume -R? [n] n
Apply anyway? [n] n
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file source-trees/wsrf-cvs/ws-gram/service/java/setup/condor/pkgdata/pkg_data_src.gpt.in.rej

It gets a bit tedious to apply lots of patches manually, especially if you have to do it a few times to get everything right, so you might want to script the process a bit. Here’s a trivial shell script that might help:

#!/bin/sh
apply_patch() { echo; echo "Applying patch $1"; patch -p0 < patches/$1 }
cd packaging
apply_patch 11__gss_assist_gridmap.diff
apply_patch 13__job_manager_acct.diff
apply_patch 14__gatekeeper_acct.diff
apply_patch 17__lsf_acct.diff
# and so forth

The idea is to delete all of the lines after the cd packaging, then do something like this to append all of the calls to apply_patch:

for f in `ls -1 packaging/patches`; do echo "apply_patch $f" >> apply_patches.sh; done

If you change a patch, don’t forget to rebuild the patch tarball. Also, if the patch came from an outside developer, it’s nice to share your changes with them.

Building Globus

The actual build of Globus happens on the NMI Build & Test infrastructure, of course. Furthermore, we use the NMI glue scripts that were written for making an NMI release. NMI probably won’t have another release, but Charles Bacon of the Globus team still uses these scripts. We can change the scripts when we need to, but it’s best to do so in a way that won’t interfere with Charles’s builds.

Getting the NMI Glue

The first step is to get a copy of the NMI build glue. Do this step on an NMI submit machine, currently nmi-s001 or grandcentral.

  1. Set up SSH so that you can log into a CSL machine without a password. There are instructions for doing this somewhere.
  2. Set environment variables for CVS access and builds:
    export CVSROOT=:ext:<USERID>@<CSL_MACHINE>.cs.wisc.edu:/p/condor/repository/nmi
    export CVS_SERVER=/p/condor/public/bin/auth-cvs
    export CVS_RSH=ssh
    export _NMI_PROJECT_VERSION=<VDT_VERSION>
    export _NMI_PROJECT=VDT
  3. Check out the NMI build glue:
    cvs co nwo
  4. Look at the file nwo/glue/globus4/build/globus-glue.cvs and decide whether you need to change the hostname listed therein

Updating the Build

The nwo/glue/globus4/build directory contains the NMI glue files for building Globus itself. If you’re building the same version of Globus 4 as the last build, you probably don’t have to change much if anything. If you’re building a new version, who knows!

In any case, be sure to check over a few critical things:

cmdfile
component_version
platforms
globus-src.cvs
cvs_tag
pre_all.pl
execution of fait_accompli/installer.sh: values of --gt2-tag and --gt4-tag
vdt-patches.ftp
ftp_target for VDT version

Note: If you change any of the Perl scripts that are run remotely, you must check them in before submitting a build. At build time, the scripts are fetched from CVS, not from your working directory.

Building Globus

Once all the build glue files are ready, it’s time to build the software itself.

  1. Run the build
    nmi_submit cmdfile
  2. If the build fails anywhere, fix the root problem and run the build again

    Of course, this is one of the most difficult parts of the entire process. Work with the VDT team, the Condor team, and the Globus development team to work through problems.

  3. Once the build succeeds, pin it:
    nmi_pin --runid 25407 --days 365

    Pinning for a year is a good default time duration, unless you know better one way or the other. But keep in mind that letting a build lapse before it’s obsolete is not a good thing.

Builds take a long time. For builds of Globus 4.0.2, here are some typical run times — once a build machine was claimed, of course:

Task Time
pre_all (on submit machine) 30 minutes
build 3½–4 hours
build_tests 1 hour
bundle 1 hour

On some systems, it takes a lot longer. Plus, some systems — e.g., x86-64 RHEL 3 — are very busy and take a long time to claim.

Building the Other Components

Once there is a complete, successful build of Globus, there are four other components to build on top of it: GSI-OpenSSH, KX.509, MyProxy, UberFTP. Some of these components may be bundled with Globus, but in all cases, we get the source code from the original authors/maintainers, not Globus. The build steps are about the same as for Globus and the NMI build glue is part of the checkout done earlier, so this part of the overall process is fairly easy.

Even though there are four parallel build steps to manage, the steps are essentially the same for each component. Use the information in the following table as you go through the procedure for each component:

Component Dir Website
GSI-OpenSSH gsissh http://grid.ncsa.uiuc.edu/ssh/
KX.509 kx509 http://www.citi.umich.edu/projects/kerb_pki/
MyProxy myproxy http://grid.ncsa.uiuc.edu/myproxy/
UberFTP uberftp http://dims.ncsa.uiuc.edu/set/uberftp/

For each component, do the following steps:

  1. Change to the component’s directory (with DIR from the table above)
    cd nwo/glue/<DIR>/build
  2. Check websites (above) and component versions, and update download URLs as needed

    E.g., in gsissh-compat.ftp:

    method = ftp
    ftp_root = ftp://
    ftp_target = ftp.ncsa.uiuc.edu/aces/gssapi-openssh/bundle/3.6/gsi_openssh_compat-3.6-src.tar.gz
    ftp_args = -q
    becomes
    method = ftp
    ftp_root = ftp://
    ftp_target = ftp.ncsa.uiuc.edu/aces/gssapi-openssh/bundle/3.7/gsi_openssh_compat-3.7-src.tar.gz
    ftp_args = -q
  3. Change the prereq-globus.nmi to reflect the run ID of the new Globus build
    method = nmi
    input_runids = 24358
  4. Change the cvs_root line in the <DIR>-glue.cvs file for your user ID and machine:
    method = cvs
    cvs_root = :ext:<USER>@<MACHINE>.cs.wisc.edu:/p/condor/repository/nmi
    cvs_server = /afs/cs.wisc.edu/p/condor/public/bin/auth-cvs
    cvs_rsh = /nmi/scripts/ssh_no_x11
    cvs_module = nwo/glue/gsissh/build
    cvs_tag = trunk-VDT1_3_10-branch
  5. In the cmdfile, update component_version as needed and make sure platforms matches that of the Globus build:
    description = GSI-OpenSSH build for $(PROJECT)-$(PROJECT_VERSION)
    run_type = build
    
    project = $(PROJECT)
    project_release = $(PROJECT_VERSION)
    
    component = gsi-openssh
    component_version = 3.7
    
    inputs = gsissh-glue.cvs, prereq-globus.nmi, gsissh-compat.ftp,  gsissh-setup.ftp, gsissh-src.ftp, gpt.ftp, zlib-src.ftp
    
    remote_declare = nwo/glue/gsissh/build/remote_declare.pl
    remote_task = nwo/glue/gsissh/build/remote_task.pl
    remote_task_args = --nondebug-bins
    remote_post = nwo/glue/gsissh/build/remote_post.pl
    
    platforms = x86_rh_7.2, x86_rh_9, x86_fc_4, x86_64_fc_4, ia64_rhas_3, x86_64_rhas_3
    
    prereqs = java-1.4.2_05, apache-ant-1.6.2
    
    prereqs_ppc_aix_5.2 = tar-1.14, gzip-1.2.4, make-3.80, vac-6, vacpp-6, perl-5.8.5, tcp_wrappers-7.6
    prereqs_sun4u_sol_5.9 =tar-1.14, gzip-1.2.4, make-3.80, gcc-2.95.3, perl-5.8.5, tcp_wrappers-7.6
    
    notify = $(USER)@cs.wisc.edu
    
    priority = 1
  6. Make sure the GPT version is the same as the one used to build Globus itself

    Look at nwo/glue/globus4/build/gpt.ftp:

    method = ftp
    ftp_root = http://
    ftp_target = vdt.cs.wisc.edu/nmi/gpt-3.2autotools2004-NMI-9.0.tar.gz

    Then check for discrepancies against the local gpt.ftp:

    method = ftp
    ftp_root = ftp://
    ftp_target = ftp.isi.edu/pub/nmi/8.0/src/gpt/gpt-3.2autotools2004-NMI-8.0.tar.gz

    Make the local copy the same as the Globus one unless there’s some good reason not to.

  7. Check over all the files for correctness
  8. Check in changes to glue files used remotely
  9. Once all the build glue files are ready, submit the build the usual way:
    nmi_submit cmdfile
  10. If the build fails anywhere, fix the root problem and run the build again
  11. Once the build succeeds, pin it:
    nmi_pin --runid 25407 --days 365

It’s fine to have all four components building at the same time, although there may be a bit of resource contention as all four try to download the Globus build at the same time.

UNPROCESSED POST-BUILD NOTES

Tim took these notes while working through the packaging build and subsequent copying of binaries to the website. The notes languished on paper for a long time, so I just put them in here in raw form. I will sort through them later.

Packaging build dir: /p/vdt/workspace/post-build/make-globus-rpms

description files in per-version directories: gt4.0.2/descriptions

one set of description files for all platforms

file:
name — core of rpm name
script — post-install script added to RPM
package — Perl regular expression, make sure each package is listed exactly once

descriptions/nmi-remote-
fix release number   -rpm-release
descriptions_dir

make-nmi-tarball

grandcentral:
~alain/build-rpms
edit to point to files, builds

TODO (? already done?): copy gt2.4.3 stuff and cat-nmi-submit

BUILDS
make_rpms
stdout

in make-globus-rpms

Error: No package …

  1. failed to install — check NMI process
  2. typo in description files

    to debug: untar globus build(s), check Globus site
    final authority: GPT dependencies, XML files

Checking consistency:

Warning: No RPM will contain …
make sure it’s in one description file

Warning: … in both …
fatal RPM [something] — put in just one

sdk.i — all dev packages, .h, .a
exceptions: test packages include all parts

look at gpt dependencies — that is, the XML files in the original Globus source

cascommunity auth service (like VOMS)
dataGridFTP (server only)
drsdata replication service
infoMDS 2
jobmanagercommon all pre-WS jobmanagers
rftReliable File Transfer (RFT)
rls
rmPre-WS GRAM (i.e., server is gatekeeper)
ws-essentials
wsgram
wsjobmanager
wsmdsMDS 4

Warning: … more than once
overlapping Perl regular expressions — fix

Error: … doesn't specify any …
whole file is inappropriate — maybe need to suppress for the platform?

for platform-specific changes:

package _____ ! ia32 ia64 fc4
not 100% what the correct syntax is for the exclusions — look at existing ones

final tarballs go into a directory such as

/p/vdt/public/html/software/globus/4.0.2_VDT_1311
that is, the globus directory has subdirectories for each version

[this is how to manually copy out binaries — use copy script instead now]
go to each platform in the packaging NMI build: userdir/platform
copy/wget to local disk
untar results, get tarball, copy to software directory

for the defs file:
GLOBUS_SOFTWARE_DIR
GLOBUS_*_VERSION

same for GSI-OpenSSH and friends
in their own directories

make sure that the GPT in the VDT matches the one used in the Globus build itself!