Note: This web site is only kept up to date for OSG Software
1.2 (VDT 2.0.0). If you are looking for information for the most recent
release, the RPM-based OSG Software 3.0, please see
the OSG documentation web site
Goals for VDT Native Packaging
This page documents the VDT’s high-level goals for shipping native packages (RPMs, Debian packages) of the VDT
software stack. It is updated as necessary to reflect our current state, knowledge, priorities, and plans.
We aim to deliver high-quality native packages that meet users’ needs and conform to community standards.
Ship RPMs for VDT Software Components
The main activity is to ship RPMs (currently) for as many VDT software components as possible. Long-term, the plan is
to make all of the VDT software available via RPMs.
We are shipping one set of packages at a time, where a set is defined as one or a small number of critical packages
and all of its dependencies. For example, the first set of software that we shipped was centered on gLExec, and
included dependencies such as PRIMA, VOMS client, and Fetch CRL. We are prioritizing the software sets based on
current demand. Upcoming sets are listed below in current priority order (see Components).
Another part of this activity is to document our packaging policies and solicit
feedback on the policies and the resulting packages.
Make New Globus Toolkit Packages for LIGO
LIGO has formally requested that we update our Globus Tooklit source packages to version 5.0.x. At the same time,
they want us to add new packages for GRAM 5 and the jobmanagers. Also, we will need to update our packages that
depend upon GT — GSI-OpenSSH, MyProxy, and UberFTP — to work with the new GT5 build.
We have already shipped test packages of GT5 for Debian, and RPMs are soon to follow.
Create a New Configuration Layer for VDT Native Packages
There is a great deal of existing infrastructure in the VDT and in OSG (e.g., the configure-osg script) for
configuring software components once installed. However, the overall approach of this infrastructure does not match
well with native packaging standards. Thus, we must rework some elements of the configuration infrastructure to work
better with our native packages and to integrate better the existing system. This effort is in line with the
discussions of the OSG Blueprint Meeting held at BNL in May, and can ultimately be retrofitted into the VDT Pacman
Below are the component sets that we expect to work on next. They are listed in order of priority. We always welcome
input on priorities.
Xrootd, Bestman, HDFS, and GridFTP: We have strong requests for at least parts of this storage- and
transfer-related set of packages from both ATLAS and CMS. ATLAS contacts include Doug Benjamin, Paolo Calafiura,
Yushu Yao, and Michael Ernst. The primary CMS contact is Brian Bockelman. Also, Brian has done some work on native
packaging of xrootd, at least, and has agreed to contribute whatever he can. Tanya Levshina is the main contact for
Bestman/Xrootd work within OSG, but not for native packaging. Michael Thomas and Abhishek Rana have already
packaged HDFS and friends as a set of RPMs in an alternate VDT-hosted YUM
repository; it would be good to integrate their work with our own.
GlideinWMS VO Frontend: Burt Holzman (head of the GlideinWMS group) and Brian Bockelman are
interested in native packages for this software. Derek Weitzel has already made RPMs for the components, and wants
to add them to the existing BeStMan/HDFS/GridFTP repository mentioned above. So, the same ideas apply
here — we would want to verify the quality of the packages and ultimately move them into the main VDT YUM
repository. It is not clear yet who will support the packages in the long run.
Worker Node Client: Various OSG site administrators have requested native packages for worker nodes
in general, because they are using site-configuration tools that work best with RPMs. It meshes well with the
gLExec packaging. However, the requests for worker-node RPMs have been fairly casual and, to the best of our
knowledge, no one needs it desperately.
Other Miscellaneous Targets
These are all lower priority. No one actively needs them now. Of course, we want native packaging everywhere
eventually, and these are some logical sets to consider later.
VOMS and GUMS: Good stepping stones because they are smaller sets of services.
CE: Probably one of the largest, most comprehensive sets of components to tackle. Work on other
sets, above, will produce some of the dependent packages required for a full CE, so effectively, we are making
progress on this goal all of the time.
For now, we plan to focus on delivering RPM binary packages for the general-purpose VDT/OSG stack. The packages will
be distributed in a VDT YUM repository with test and production subsections. In the short-term, we will support:
- Red Hat Enterprise Linux 5, IA-32 and AMD-64
- Scientific Linux 5, IA-32 and AMD-64
For LIGO, we will continue to deliver native source packages in both the RPM and Debian formats. We already have the
corresponding test and production YUM and APT repositories. At LIGO’s official request, we currently support:
- CentOS 5, IA-32 and AMD-64
- Debian 5, IA-32 and AMD-64
Medium- to long-term, we hope to support other platforms (e.g., RHEL 4, RHEL 6) and other native packaging systems
(notably, Debian) for all of the VDT packages. However, at this time:
- There is little official demand for other platforms
- We have relatively little effort to devote to native packaging
- Adding another packaging system (like Debian) takes a lot of effort, because they do things differently
That being said, RHEL 4 RPMs are very similar to produce to RHEL 5 ones. Thus, we may make some packages
for RHEL 4 simply as a by-product of our normal build processes. But in order to stay focused on delivering as
many quality packages as possible, we plan to test, debug, refine, and ultimately deliver only RHEL 5 packages
during this period. Plans can change if need be.
Tim Cartwright leads the VDT native package effort, and is focusing on the Debian packages for LIGO
and on the configuration system. Given his other responsibilities, Tim can typically devote about 75% of his time
to VDT native packaging.
Mat Selmeci will spend the majority of his time working on new RPM packages of VDT components. On
average, about 90% of his time goes toward this work.
Alain Roy monitors, advises, and provides both high-level and technical guidance to the native
packaging effort. At times, Alain can contribute technical effort to specific tasks that arise, depending on the
other demands on his time.
Scot Kronenfeld primarily focuses on maintaining and extending the VDT Pacman packages, and thus is
not expected to contribute significantly to the native packaging effort. If there is greater need to deliver native
packages than to maintain Pacman packages, some of Scot’s could be diverted to native packaging. However, it
is worth noting that Scot has not done much native packaging work and thus would need extra spin-up time.
VDT Fermi Team — It is not clear what effort, if any, the Fermi team will have for native
packaging. Probably the best fit is if we get to some of the storage-related package sets, in which case the Fermi
team might have greater familiarity with and interest in the components. In any case, the Fermi team would need
extra spin-up time.
For the VDT native packaging effort, we are going to follow a flexible, incremental delivery approach, based very
loosely on the Agile Methodologies for software development. What this means in detail:
We will deliver working packages once per month. Specifically, all packages that install, run, update, and remove
correctly — even with incomplete functionality — will be placed into the test repository at
the end of the month. Once properly tested (to be defined), packages will be promoted to a production repository.
At the same time (end of month), Tim will write and distribute a report detailed what has been accomplished (i.e.,
delivered), what is in progress, and any discoveries since the last report. Discoveries include (and are not
limited to) technical issues (resolved and unresolved), changes to available effort (past and predicted), and new or
changed requests from stakeholders.
Based on what has been accomplished and on new information, Tim will work with the OSG Executive Team and other key
stakeholders (TBD) to determine what course corrections, if any, should be made, thereby setting the goals and
priorities for the next month’s deliveries.
Last Updated: 1 November 2010