################################################################
# VDT-dCache Package #
################################################################
The purpose of VDT-dCache package is to enable a user to install
and/or upgrade his/her dCache installation.
The package can be downloaded from the VDT website
(http:/vdt.cs.wisc.edu/software/dcache/server/)
The package currently provides:
o dCache installation/upgrade/init.d scripts
o dCache gratia probes installation script
################################################################
# SPECIAL NOTE FOR SL5 INSTALL #
################################################################
Before you do the dCache install on SL5 node, make sure the
following system-level libraries are installed:
o compat-readline
o openssl
################################################################
# WHICH SECTION OF THIS README SHOULD I READ FIRST? #
################################################################
You can read the entire file or jump to various sections as
necessary. In general:
Purpose Section
+++++++ +++++++
o Understand the directory structure DIRECTORY STRUCTURE
o Perform the installation INSTALLATION
o Perform your site specific customizations CUSTOMIZATION
o Learn about node configuration NODE CONFIGURATION
(config-node.pl)
o Learn about site configuration SITE CONFIGURATION
(site-info.def)
o Learn about install scripts INSTALL SCRIPTS
(java, postgres, pnfs, dCache, gratia probes)
o Clean up everything CLEANUP
(and begin from scratch)
o Upgrade the PNFS Server PNFS SERVER UPGRADE
o Get Help SUPPORT
################################################################
# DIRECTORY STRUCTURE #
################################################################
Upon untarring the vdt-dCache tarball, you should see following
directory structure:
RPMS - This directory contains dCache, Postgres, PNFS,
srmwatch, JDK and zsh rpms
install - This directory contains the following:
o site-info.def.template - template used by config-node.pl
o installation scripts
config-node.pl
find_java.sh
install.sh
install_dcache.sh
install_java.sh
install_pnfs.sh
install_postgres.sh
install_srmwatch.sh
install_initd_scripts.sh
install_gratia_probes.sh
rpm_unpack.sh
o README - This file
o config - This directory contains LinkGroupAuthorization.conf,
PoolManager.conf and storage-authzdb. The first two
serve as default configuration files. The last one
is an example authorization file
WARNING: If you are doing an upgrade, your previous
PoolManager.conf will be replaced. A backup of the
previous PoolManager.conf is saved for reference
when customizing PoolManager.conf
o util - This directory contains file "config_file". This is
Yaim Configuration and setup file for dCache 1.7.0-X
o init.d - This directory contains the init.d scripts for following:
pool
pnfs
dcache
################################################################
# INSTALLATION #
################################################################
Few important things to note, before you start the installation
process:
o The install script can be used to do a fresh install
of dCache 1.8.0-x or an upgrade of 1.7.0-x or 1.8.0-x
to 1.8.0-y
o Make sure all nodes on which dCache services will run,
including the pool nodes have valid host certificates
o Make sure NFS is not already running, else PNFS install
will fail
o The install script should be run as user "root"
o The installation process logs go to file "vdt-install.log"
o Error messages, if any, go to the file "vdt-install.err"
Alright, lets begin:
On all dCache nodes:
o Download the VDT-dCache package
o Untar the VDT-dCache package
o cd into the "install" directory
cd vdt-dcache-x.x/install
o Run the configuration script
./config-node.pl
Upon successful execution, a file named "site-info.def"
will be created
Note 1: In case of multi-node install, you can copy the
"site-info.def" file to remaining dCache nodes
and skip this step. You may have to edit this file
if Java location is different on these nodes
Note 2: If you are doing an upgrade, file "site-info.def"
already exists" and there are no changes in your
site configuration, you don't have to run the
configuration script again
o Edit the site-info.def file if site-dependent customizations
are desired (see CUSTOMIZATION section)
o Make sure rpcinfo is in your path
First on the PNFS node and then on remaining dCache nodes:
o If you would like to see what the installation script will
do and NOT do the actual install, run
./install.sh --dryrun
When you are ready to do the actual install, run
./install.sh
Post Install Configuration:
++++++++++++++++++++++++++
After the installation is over and before you start various dCache
services, you need to make some authorization configuration changes.
You should refer to main dCache website for this
(http:/www.dcache.org/manuals/Book/config/cf-gplazma.shtml), but here
is a minimal check list:
o customize the etc/dcachesrm-gplazma.policy file
o depending on how gPlazma is configured, it will use various
files. Make sure these files exist and have correct information
Start dCache:
++++++++++++
If using the init.d scripts
===========================
Follow the order as shown below
On PNFS node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o PNFS server should be running at this point
If not, start it
> /etc/init.d/pnfs start
On Admin node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o Start dCache core services
> /etc/init.d/dcache start
o Start gratia transfer probe services
> /etc/init.d/gratia-dcache-transfer start
On PNFS node:
o Start the PNFS domain
> /etc/init.d/dcache start
This starts up the pnfsManager, which by default runs on the PNFS
node
On SRM, Gsiftp,Gsidcap nodes:
o Start dCache core services
> /etc/init.d/dcache start
On Pool nodes:
o Start up the pools
> /etc/init.d/pool start
If NOT using the init.d scripts
===============================
Follow the order as shown below
On PNFS node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o PNFS server should be running at this point
If not, start it
> /opt/pnfs/bin/pnfs start
On Admin node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o Start dCache core services
> /opt/d-cache/bin/dcache start
On PNFS node:
o Start the PNFS domain
> /opt/d-cache/bin/dcache start
This starts up the pnfsManager, which by default runs on the PNFS
node
On SRM, Gsiftp,Gsidcap nodes:
o Start dCache core services
> /opt/d-cache/bin/dcache start
On Pool nodes:
o Start up the pools
> /opt/d-cache/bin/dcache start pool
################################################################
# NODE CONFIGURATION SCRIPT #
################################################################
The config-node.pl script will turn the site-info.def.template
file into a site-info.def file. If an existing site-info.def file
is found, it will use those values as defaults.
Internally,
o First, this script invokes the find_java.sh script which
finds java installed on your system in following order:
o Next, it will ask a number of questions for the user to answer.
o It first will ask how many non-pool and non-door nodes
you have. Based on this it makes a recommended
configuration using up to 5 nodes.
o It then asks which services should run on each of these
nodes (Note: node #1 is considered the DCACHE_ADMIN node)
The list of services should be entered on a single line
seperated by white space. The complete list of choices
are:
lmDomain
poolManager
pnfsManager
dirDomain
adminDoor
httpDomain
utilityDomain
gplazmaService
infoProvider
srm
replicaManager
statistics
o Next, the door nodes are configured. One instance each of dcap,
gsidcap, and gridftp will be made to run on each door node.
o Next, the pools with the directory locations on those nodes
will be set. By default, pool sizes will be made to be the size of
the entire partition the pool is are on. If another size is desired
for a pool, edit the file site-info.def, replacing the text "all"
with the desired size.
o Next, gratia probes are configured (gratia probes collect various
statistics about file transfer, space allocation by VOs and pool
utilization). The data collected by the probes will be sent to a
central collector. You will be ask to provide the following information:
URL of central collector
The username, password and port number of dCache admin interface
Name of your SE
Name of the Grid to which the SE belongs (OSG-ITB, OSG, LOCAL)
Also, gratia can notify you if gratia server went down
so you will be asked questions related to email settings
o Finally, it asks if you would like to use the init.d scripts
for starting/stopping various dCache services. If you choose 'y'
the scripts will be copied to the /etc/init.d/ directory,
chkconfig'ed, and permissions will be set to 744.
################################################################
# SITE CONFIGURATION #
################################################################
site-info.def file, defines which service shall run on which node.
If a service is required and no definition is given, it will run
on the admin node.The definition for services which may run on
more than one node may be a space-separated list of
fully-qualified hostnames. If a service needs further information,
such as pool size and location, the may appear in colon-separated
fields after the hostname.
################################################################
# INSTALLATION SCRIPTS #
################################################################
++++++++++++
+ Java +
++++++++++++
Installation script will
o first, check if java exists in $PATH. If it does,
it chooses that version (provided its a valid installation)
o then, check if $JAVA_HOME is defined. If it is,
it chooses that version
o finally, it checks in various standard locations and if
multiple java installations are found, the most recent
version is selected.
To give you an idea how this order works, consider
following situation:
You have a /usr/bin/java that points to a JRE installation
and have another JDK installation as well. In this case,
after you run the config_node.pl script, $JAVA_LOCATION in
your site-info.def will point to the JRE install (Note that
dCache requires JDK). In this case, you either can include
the java from JDK version in $PATH (before the JRE version),
or explicitly set $JAVA_LOCATION in site-info.def file to
point to JDK version.
++++++++++++
+ Postgres +
++++++++++++
Installation script will install Postgres only if
o this is a PNFS server node OR any of SRM, Billing, Space
Manager, Pin Manager or Replica Manager services are
running on this node AND
o it is not installed already
Note: dCache strongly recommends to use atleast version 8 or higher
++++++++++++
+ PNFS +
++++++++++++
Installation script will install PNFS server software only if
o this is the PNFS server node (defined by the value of
variable $DCACHE_PNFS in site-info.def) AND
o NFS server is not running on this node AND
o it (PNFS) is not installed already
Note 1: In a single dCache installation, there can only be one PNFS
server node
Note 2: Some PNFS rpms have a dependency on zsh. This is why zsh rpm
is installed as part of PNFS installation
Note 3: For existing PNFS installations:
o If PNFS is running when you run the installation script:
The installation script will check that the pnfs export
for the doors has value /pnfsdoors. If not, a warning
is issued and dCache installation might not work.
o If PNFS is not running when you run the install script:
You need to start PNFS first. The dCache installation
will not work till you do so.
++++++++++++
+ dCache +
++++++++++++
Installation script will unpack the dcache-server rpm if necessary
and modify the dCache configuration files
o Services will be turned on and pools will be created
according to the information in site-info.def
o Files etc/node_config.delta and etc/dCacheSetup.delta are
created, which show exactly what lines in etc/node_config
or config/dCacheSetup will be set. These files can be used
for custom settings as well (see CUSTOMIZATION section).
++++++++++++
+ gratia +
++++++++++++
Installation script will unpack the gratia probes on admin and
srm door nodes and will modify the configuration file
- gratia transfer probe will need to be started by
running "service gratia-dcache-transfer start" on admin node
- gratia storage probe is a cronjob that is run by default
every hour. The cron file gratia-probe-dcache-storage.cron
is located in /etc/cron.d
################################################################
# CUSTOMIZATION #
################################################################
Some dCache options not covered by the config-node.pl script may
be set by editing site-info.def after config-node.pl has been run.
MAX_FTP_STREAMS
The maximum number of client streams allowed by gridftp.
GRIDFTP_PRIVATE_NETWORK
The network portion of the ip on the private network of
the pools. For example, if the pools ip's are 192.168.1.x,
enter "192.168.1".
USE_MULTI_MOVER_QUEUES
Sets up default,WAN, and LAN queues on the pools, and
directs gridftp to use WAN and dcap and gsidcap to use LAN.
If desired, set to "yes".
BILLING_USE_DB
Causes billing info to be copied to a database as well as
the log file. If desired,set to "yes".
DCACHE_LOG_DIR
Set to a non-default log directory for dCache domain output
logs. Directory must exist before dCache is started.
Note: In order to correct a mismatch between gridftp and srm
paramters that exists in the default settings of dCache, the
following customization is always done:
REMOTE_GSI_FTP_MAX_TRANSFERS=2000
In addition, etc/node_config.delta and etc/dCacheSetup.delta can be
used to change any other lines in etc/node_config or
config/dCacheSetup. To change or add any other lines in these files,
before running install.sh add the line or lines to the *.delta file
(creating the file if necessary).
################################################################
# CLEANUP #
################################################################
You should follow instructions in this section, ONLY if you tried
to do the installation and all of below mentioned conditions are
true:
o installation didn't finish successfully
o you want to do a complete clean up
o you want to run dCache installation script again
++++++++++++
+ Postgres +
++++++++++++
This will erase all postgresql rpms and delete any existing postgres
databases and installations !! Make a backup of your databases and
anything else, you think you might need later.
Alright, lets begin
o Stop the postmaster
> /etc/init.d/postgresql stop
o Erase following postgresql rpms
> rpm -e postgresql postgresql-libs postgresql-server
o Delete the /var/lib/pgsql directory
> rm -rf /var/lib/pgsql
o As additional check, make sure following files do not
exist
/bin/postmaster
/sbin/postmaster
/usr/bin/postmaster
/usr/sbin/postmaster
/usr/local/bin/postmaster
++++++++++++
+ PNFS +
++++++++++++
This will erase pnfs-server rpm and delete all pnfs directories created
in /opt !! Make a backup of your databases and anything else, you think
you might need later.
Alright, lets begin
o Erase pnfs server rpm
> rpm -e pnfs-postgresql
o Erase zsh rpm
> rpm -e zsh
o Delete various pnfs directories (depending on where your install
failed, all of below might not have created)
> rm -rf /opt/pnfs.3.1.10
> rm -rf /opt/pnfs
> rm -rf /opt/pnfsdb
o Uninstalling pnfs does not unregister its RPC service
(a bug in PNFS). You should do it manually as follows:
> rpcinfo -p | grep nfs | grep -v grep
100003 2 udp 2049 nfs
> rpcinfo -d nfs 2
o Remove the init.d script
> rm -rf /etc/init.d/pnfs
> rm -rf /etx/init.d/dcache
++++++++++++
+ gratia +
++++++++++++
This will erase the gratia rpms.
On both dCache Admin and SRM nodes:
o Erase the following gratia rpms
rpm -e gratia-probe-extra-libs-0.32.4-1
rpm -e gratia-probe-extra-libs-arch-spec-0.32.4-1
rpm -e gratia-probe-common-0.32.4-1
On Admin node:
o Erase the transfer probe rpm
rpm -e gratia-probe-dCache-transfer-itb-0.32.4-1
On SRM node
o Erase the storage probe rpm
rpm -e gratia-probe-dCache-storage-itb-0.32.4-1
++++++++++++
+ dCache +
++++++++++++
This will erase the dcache-server rpm.!!Make a backup of /opt/d-cache and
anything else, you think you might need later.
Alright, lets begin
o Erase dcache-server rpm
> rpm -e --noscripts dcache-server-1.7.0-19.noarch
o Delete the d-cache directory
> rm -rf /opt/d-cache
o Delete the init.d script
> rm -rf /etc/init.d/dcache
> rm -rf /etc/init.d/pool (If it was a pool node)
################################################################
# PNFS SERVER UPGRADE #
################################################################
++++++++++++++
+ Disclaimer +
++++++++++++++
Though an attempt has been made to cover all that is necessary,
we do not take responsibility for loss of any data. Depending on
your PNFS setup, you may need to perform additional steps.
The following instructions have been tested while upgrading from
pnfs-postgresql-3.1.10-3.i386.rpm to pnfs-postgresql-3.1.10-7.i386.rpm
You need to be very careful while performing each step.
Also, please don't confuse this with the Postgres Database upgrade.
If you need to upgrade Postgres databases, then refer to
instructions on official postgres website www.postgresql.org.
++++++++++++++
+ Assumption +
++++++++++++++
The Postgres databases are located outside the PNFS installation
directory. So upgrading the PNFS server code will not affect the
Postgres databases in any way.
++++++++++++++
+ Upgrade +
++++++++++++++
Alright, lets begin
o Backup the PNFS Database structure information
(preferrably on different host)
This is very critical information and must be backed up,
before you upgrade the PNFS server code. If this is lost,
then the actual data stored in Postgres databases cannot be
accessed.
This location may be different for different sites, depending
on where you had installed PNFS.Common locations are
$PNFS_INSTALL_DIR/pnfsdb and $PNFS_INSTALL_DIR/pnfs/db
> scp -r /opt/pnfsdb fapl035:/usr/local/pnfs-backup/
D-0000 100% |******************************| 51 00:00
databases 100% |******************************| 78 00:00
shmservers 100% |******************************| 2 00:00
D-0001 100% |******************************| 51 00:00
admin 100% |******************************| 0 00:00
data1 100% |******************************| 0 00:00
o Backup the main PNFS Server configuration file
(preferrably on different host)
This file is usually present at location /usr/etc/pnfsSetup
> scp /usr/etc/pnfsSetup fapl035:/usr/local/pnfs-backup/
pnfsSetup 100% |******************************| 498 00:00
o Backup the existing PNFS Server rpm
(preferrably on different host)
> scp ../RPMS/pnfs-postgresql-3.1.10-3.i386.rpm fapl035:/usr/local/pnfs-backup/
o Remove the old and get the new PNFS Server rpm
o Remove the old pnfs server rpm from $VDT_DCACHE_HOME/dcache/RPMS/
> rm ../RPMS/pnfs-postgresql-3.1.10-3.i386.rpm
o Copy the new rpm in $VDT_DCACHE_HOME/dcache/RPMS/
o Install the new PNFS Server rpm
o Stop PNFS (if it's running)
> /opt/pnfs/bin/pnfs stop
o Stop Postmaster (if it's running)
> /etc/init.d/postgresql stop
o Make sure that old PNFS Server rpm is not installed.
If it is, erase it.
> rpm -qa | grep pnfs
pnfs-postgresql-3.1.10-3
> rpm -e pnfs-postgresql-3.1.10-3
o Uninstalling pnfs does not unregister its RPC service
(a bug in pnfs). You have to do it manually as follows:
> rpcinfo -p | grep nfs | grep -v grep
100003 2 udp 2049 nfs
> rpcinfo -d nfs 2
o Run the install script with dryrun option to make sure
that things are setup right for PNFS to be installed again.
> ./install.sh -s site-info.def --dryrun
o Start Postmaster if it is not already running
> /etc/init.d/postgresql start
o Run the install script (without dryrun option).
This will install new PNFS Server rpm.
> ./install.sh -s site-info.def
o Restore the backed up configuration and data to the new installation
Option 1
o Stop PNFS server
> /opt/pnfs/bin/pnfs stop
o Copy the backed up PNFS Server configuration file to
/usr/etc/pnfsSetup
> scp fapl035:/usr/etc/pnfsSetup /usr/etc/pnfsSetup
o Copy the backed up PNFS Database structure to
/opt/pnfsdb
> scp -r fapl035:/usr/local/pnfs-backup/pnfsdb /opt/
Option 2
o Stop PNFS server
> /opt/pnfs/bin/pnfs stop
o Copy the backed up PNFS Server configuration file to
/usr/etc/pnfsSetup
> scp fapl035:/usr/etc/pnfsSetup /usr/etc/pnfsSetup
o Make sure in file /opt/pnfs.3.1.10/pnfs/etc/pnfs_config,
the variable PNFS_OVERWRITE is set as follows:
PNFS_OVERWRITE = no
This basically means to not overwrite the pnfs database
specified by $PNFS_DB, if it exists already. Here make
sure value of $PNFS_DB point to right location.
o Start the PNFS Server
> /opt/pnfs/bin/pnfs start
################################################################
# SUPPORT #
################################################################
o For more detailed information about dCache or PNFS, refer the
dCache Book at http:/www.dcache.org/manuals/Book/
o For any issues/questions with/about the the VDT-dCache package,
please send email to osg-storage@opensciencegrid.org