Note: This web site is only kept up to date for OSG Software 1.2 (VDT 2.0.0). If you are looking for information for the most recent release, the RPM-based OSG Software 3.0, please see the OSG documentation web site

Apache in the VDT

Nate Mueller created and maintained our Apache build for a long time. Before he left, he talked with Tim about the process. This document started with from Tim’s notes from that time. Alain added to these notes on 12 April 2007 with wisdom gained from reading the GSI patch code and experimenting with the AJP connection.

We build Apache from source with lots of patches and a few extra modules.


Here are the patches we apply and what they’re all about.


This patch modifies the apachectl script to use the standard VDT location for the Apache config file instead of some system location.


This is the most complicated patch. It is a patch to mod_ssl that allows Apache to do SSL authentication with proxies, including the three flavors that we deal with in the grid world:

  • GT2-style, aka “old” proxies (-old)
  • GT3-style, aka “new” proxies (-new)
  • GT4-style, aka “RFC” proxies (-rfc)

Originally, we got a patch from Manchester University, and it dealt with old-style proxies. Nate augmented it to handle new- and RFC-style proxies. Note: VOMS always gives old-style proxies. The Manchester patch has been abandoned mostly. It became mod_gridsite, which doesn’t meet our needs: It and mod_jk don’t play nicely together and mod_gridsite directly modifies Apache memory.

So, we’ve been maintaining the patch for a while, originally for Apache 1.3, then migrated to 2.0, now 2.2. It is tested nightly, although the RFC format isn’being tested because of GUMS/Tomcat problems — new- and RFC-style proxies differ in their OID string representation, but are otherwise the same. In any case, migrations of this patch often involve “stupid patch munging” — Apache functions get moved around, renamed, etc.

See below for more about this patch.


mod_python still doesn’work with Apache 2.2, because mod_python uses a deprecated Apache API. Nate changed mod_python to work with new API, but didn’t really change functionality. Hopefully, the mod_python people will release an update to the module that will work with Apache 2.2.


This is another patch to mod_python which we got from Conrad Stevens, the Clarens developer. He doesn’t submit his patches to the mod_python developers, for various reasons. Presumably, Clarens won’t work without this patch.


I’m not sure if this is the complete list of Apache modules we build, but it’s a start.


XML parser. From


This is the Apache module that lets it pass web application requests to Tomcat. From

Proxy chain problem

As of 12 April 2007, we’ve had a problem for a while, probably since Nate added the GSI.patch: Tomcat will only get the user’s cert, not the entire chain. Alain looked into this in detail. He’s pretty sure that the GSI.patch is correct because he debugged it carefully.

Alain discovered bug reports and patches that implicate mod_jk (Apache’s code that talks to Tomcat) and the AJP connector (Tomcat’s code that talks to Apache): They simply don’t preserve the chain. Alain documented the state and the eventual solution in Ticket #1401

Understanding the GSI.patch

At first glance, the GSI.patch is a pretty large chunk of incomprehensible code. It’s really not too bad if you take a couple of hours to work through it. Alain has a few incomplete notes that he took while figuring it out. It may speed things up for someone later.

The patch is to Apache’s mod_ssl, which is the chunk of Apache that understands SSL. The majority of the patch is contained in ssl_engine_kernel.c. The only part that isn’t is in ssl_engine_init.c, and this just registers a callback function that verifies the peer’s certificate. It is registered with SSL_CTX_set_cert_verify_callback.

So everything boils down to one uber function, ssl_callback_GSI_verify. This is supposed to return 1 if the certificate can be verified, and 0 otherwise. It is given a X509_STORE_CTX * as a parameter. This is a structure that is used just for validating a single certificate. It’s defined in OpenSSL’s include/openssl/x509_vfy.h. The definition looks like this:

/* This is a used when verifying cert chains.  Since the
 * gathering of the cert chain can take some time (and have to be
 * 'retried', this needs to be kept and passed around. */
struct x509_store_ctx_st      /* X509_STORE_CTX */
    X509_STORE *ctx; 
    int current_method; /* used when looking up certs */

    /* The following are set by the caller */
    X509 *cert;     /* The cert to check */ AJR: Just the cert, no chain, struct x509_st
    STACK_OF(X509) *untrusted;  /* chain of X509s - untrusted - passed in */
    STACK_OF(X509_CRL) *crls;   /* set of CRLs passed in */

    X509_VERIFY_PARAM *param;
    void *other_ctx;    /* Other info for use with get_issuer() */

    /* Callbacks for various operations */
    int (*verify)(X509_STORE_CTX *ctx); /* called to verify a certificate */
    int (*verify_cb)(int ok,X509_STORE_CTX *ctx);       /* error callback */
    int (*get_issuer)(X509 **issuer, X509_STORE_CTX *ctx, X509 *x); /* get issuers cert from ctx */
    int (*check_issued)(X509_STORE_CTX *ctx, X509 *x, X509 *issuer); /* check issued */
    int (*check_revocation)(X509_STORE_CTX *ctx); /* Check revocation status of chain */
    int (*get_crl)(X509_STORE_CTX *ctx, X509_CRL **crl, X509 *x); /* retrieve CRL */
    int (*check_crl)(X509_STORE_CTX *ctx, X509_CRL *crl); /* Check CRL validity */
    int (*cert_crl)(X509_STORE_CTX *ctx, X509_CRL *crl, X509 *x); /* Check certificate against CRL */
    int (*check_policy)(X509_STORE_CTX *ctx);
    int (*cleanup)(X509_STORE_CTX *ctx);

    /* The following is built up */
    int valid;      /* if 0, rebuild chain */
    int last_untrusted; /* index of last untrusted cert */
    STACK_OF(X509) *chain;      /* chain of X509s - built up and trusted */
    X509_POLICY_TREE *tree; /* Valid policy tree */

    int explicit_policy;    /* Require explicit policy value */

    /* When something goes wrong, this is why */
    int error_depth;
    int error;
    X509 *current_cert;
    X509 *current_issuer;   /* cert currently being tested as valid issuer */
    X509_CRL *current_crl;  /* current CRL */

    CRYPTO_EX_DATA ex_data;
    } /* X509_STORE_CTX */;

If you have to debug the code, it’s useful to know about the X509 type, which is really a struct x509_st:

struct x509_st
	X509_CINF *cert_info;
	X509_ALGOR *sig_alg;
	ASN1_BIT_STRING *signature;
	int valid;
	int references;
	char *name;
	CRYPTO_EX_DATA ex_data;
	/* These contain copies of various extension values */
	long ex_pathlen;
	long ex_pcpathlen;
	unsigned long ex_flags;
	unsigned long ex_kusage;
	unsigned long ex_xkusage;
	unsigned long ex_nscert;
	struct AUTHORITY_KEYID_st *akid;
	X509_POLICY_CACHE *policy_cache;
#ifndef OPENSSL_NO_RFC3779
	STACK_OF(IPAddressFamily) *rfc3779_addr;
	struct ASIdentifiers_st *rfc3779_asid;
	unsigned char sha1_hash[SHA_DIGEST_LENGTH];
	X509_CERT_AUX *aux;
	} /* X509 */;
One important bit to notice here is the name: that’s the DN. If you are debugging something in this file, you can do something like this:
ap_log_error(APLOG_MARK, APLOG_ERR, 0, s, "AJR: Pushing first cert: %s\n", ctx->cert->name);