VDT Nightly Test Framwork

vail(nmueller): vdt-run-install-tests --help
Usage: vdt-run-install-tests
        --package 
        --version 
        --download 

Alternativly you can use the --download flag to download a spec file. Lines in the spec file have the format "version, version...: package, package..." and do the same thing as the old -v and -p tests. In addition, however, the spec file is passed through the gcc preprocesser with the hostname (all caps, - converted to _) and the day of the week (all caps) defined. This allows you do make more complex policy. The current spec file used is http://vdt.cs.wisc.edu/test-scripts/tests-to-run.

Installing a New Test Box

1. Follow the instructions to get host, LDAP, and HTTP certificates for the machine
2. Edit /p/vdt/workspace/grid-security/make-tarballs to include the new platform.
3. If needed, checkout a local copy of the nightly-tests.what CVS module:

   cvs -d /p/condor/workspaces/vdt/vdt_cvs co nightly-tests.what

4. Change to your nightly-tests.what directory
5. If you already had a nightly-tests.what checkout, make sure it’s up to date
6. Edit root/vdt-install-test/bin/vdt-run-install-tests and add the new host to the %archs hash definition; e.g.:

   'vdt-rhas3-amd64' => ["RHEL 3", "x86-64"],

7. Edit root/send-mail.sh and add the new host to the list of parameters:

   -a "Fedora Core 4,x86-64=1.3.11" -a "Fedora Core 4,32-on-x86-64=1.3.11"

8. Commit your changes
9. Follow the what instructions to set up what

How it all works

vdt-run-install-tests

This is the top level script that makes everything look easy. Use it (with appropriate -p and -v options) to run VDT install tests. The master copy lives in /p/vdt/workspace/nightly-tests. If you add a new test question you need to edit this script to pre-answer it in the hash near the top.

vdt-pacman-install-test

This is where most of the work is done. It sets a bunch of environment variables, runs pre-install scripts, installs the VDT, runs post-install scripts, tests the VDT and runs post-test scripts. It's big, nasty and takes more command line options then I care to admit. Don't attempt to run it directly. vdt-pacman-install-test also lives in /p/vdt/workspace/nightly-tests.

Server setup and teardown scripts

To make installation easier, vdt-pacman-install-test fetches it's pre and post scripts from the vdt website. Scripts are located at /p/vdt/public/html/test-scripts.

VDT-Test

The pacman package VDT-Test installs the VDT test harness (VDT-TestHarness) and the tests themselves (VDT-Certification-Tests). It lives in the regular VDT cvs tree. The test harness (vdt-run-tests) is executed by the nightly test scripts and stores it's output in the test database. It can still be used in the old standalone way.

The test database

This is where all the results are stored. Currently it's running on vail.cs.wisc.edu (Nate's desktop) with the data stored in /scratch/nmueller/vdt-test-db. This is less then ideal and I'm working as hard as I can to move it to the lab's database server (dumbo.cs.wisc.edu).

The web pages

http://www.cs.wisc.edu/vdt/test-results. These use Mason to display real-time test results. The source lives in the web tree at /p/vdt/public/html/test-results. Check out http://www.masonbook.com before you get too deep.

The email reports

The email reports are sent by /p/vdt/workspace/nightly-tests/vdt-test-email every night from my crontab.