################################################################
# VDT-dCache Package #
################################################################
The purpose of VDT-dCache package is to enable a user to install
and/or upgrade his/her dCache installation.
The package can be downloaded from the VDT website
(http:/vdt.cs.wisc.edu/software/dcache/server/)
The package currently provides:
o dCache installation/upgrade/init.d scripts
o gratia storage and transfer probes installation script
################################################################
# SPECIAL NOTE FOR SL5 INSTALL #
################################################################
Before you do the dCache install on SL5 node, make sure the
following system-level libraries are installed:
o compat-readline
o openssl
################################################################
# WHICH SECTION OF THIS README SHOULD I READ FIRST? #
################################################################
You can read the entire file or jump to various sections as
necessary. In general:
Purpose Section
+++++++ +++++++
o Understand the directory structure DIRECTORY STRUCTURE
o Perform the installation INSTALLATION
o Perform your site specific customizations CUSTOMIZATION
o Learn about node configuration NODE CONFIGURATION
(config-node.pl)
o Learn about site configuration SITE CONFIGURATION
(site-info.def)
o Learn about install scripts INSTALL SCRIPTS
(java, postgres, pnfs, dCache, gratia probes)
o Clean up everything CLEANUP
(and begin from scratch)
o Get Help SUPPORT
Note: For PNFS/Postgres Upgrade, read the UPGRADE file
################################################################
# DIRECTORY STRUCTURE #
################################################################
Upon untarring the vdt-dCache tarball, you should see following
directory structure:
RPMS - This directory contains following rpms:
dCache
Postgres
PNFS
srmwatch
gratia storage and transfer probe
JDK
install - This directory contains the following:
o site-info.def.template - template used by config-node.pl
o installation scripts
config-node.pl
find_java.sh
install.sh
install_dcache.sh
install_java.sh
install_pnfs.sh
install_postgres.sh
install_srmwatch.sh
install_initd_scripts.sh
install_gratia_probes.sh
rpm_unpack.sh
o README - This file
o UPGRADE - File which contains instructions on how to upgrade
Postgres and PNFS software
o config - This directory contains LinkGroupAuthorization.conf,
PoolManager.conf and storage-authzdb. The first two
serve as default configuration files. The last one
is an example authorization file
WARNING: If you are doing an upgrade, your previous
PoolManager.conf will be replaced. A backup of the
previous PoolManager.conf is saved for reference
when customizing PoolManager.conf
o util - This directory contains file "config_file". This is
Yaim Configuration and setup file for dCache 1.7.0-X
o init.d - This directory contains the init.d scripts for following:
pool
pnfs
dcache
################################################################
# INSTALLATION #
################################################################
Few important things to note, before you start the installation
process:
o The install script can be used to do a fresh install
of dCache 1.9.2-z or an upgrade of 1.8.0-x or 1.8.0-y
to 1.9.2-z
o Make sure all nodes on which dCache services will run,
including the pool nodes have valid host certificates
o Make sure NFS is not already running, else PNFS install
will fail
o The install script should be run as user "root"
o The installation process logs go to file "vdt-install.log"
o Error messages, if any, go to the file "vdt-install.err"
Alright, lets begin:
On all dCache nodes:
o Download the VDT-dCache package
o Untar the VDT-dCache package
o cd into the "install" directory
cd vdt-dcache-x.x/install
o Run the configuration script
./config-node.pl
Upon successful execution, a file named "site-info.def"
will be created
Note 1: In case of multi-node install, you can copy the
"site-info.def" file to remaining dCache nodes
and skip this step. You may have to edit this file
if Java location is different on these nodes
Note 2: If you are doing an upgrade, file "site-info.def"
already exists" and there are no changes in your
site configuration, you don't have to run the
configuration script again
o Edit the site-info.def file if site-dependent customizations
are desired (see CUSTOMIZATION section)
o Make sure rpcinfo is in your path
First on the PNFS node and then on remaining dCache nodes:
o If you would like to see what the installation script will
do and NOT do the actual install, run
./install.sh --dryrun
When you are ready to do the actual install, run
./install.sh
Post Install Configuration:
++++++++++++++++++++++++++
After the installation is over and before you start various dCache
services, you need to make some authorization configuration changes.
You should refer to main dCache website for this
(http:/www.dcache.org/manuals/Book/config/cf-gplazma.shtml), but here
is a minimal check list:
o customize the etc/dcachesrm-gplazma.policy file
o depending on how gPlazma is configured, it will use various
files. Make sure these files exist and have correct information
o Adjust your configuration according to the following:
****Important changes in gPlazma configuration****
gPlazma now persists authorizations in a database. Therefore, postgres
must be installed on the node which is running gPlazma. No special
configuration of postgres is needed by gPlazma, nor will it interfere
with other dCache uses of the database.
gPlazma no longer supports the convention of setting a literal
Role=NULL and/or Capability=NULL when no role or capability exist in a
user's proxy. This will affect sites that are using
/etc/grid-security/grid-vorolemap for authorization and are currently
using the convention. All instances of Role=NULL and Capability=NULL
should be removed from /etc/grid-security/grid-vorolemap. For example,
if a site is currently specifying fully-qualified attribute names
(groups and roles) in a form such as
"/DC=org/DC=doegrids/OU=People/CN=Ted Hesselroth 898521" "/cms/Role=cmsprod/Capability=NULL" cmsprod
"/DC=org/DC=doegrids/OU=People/CN=Ted Hesselroth 898521" "/cms/Role=NULL/Capability=NULL" uscms
such lines should be changed to
"/DC=org/DC=doegrids/OU=People/CN=Ted Hesselroth 898521" "/cms/Role=cmsprod" cmsprod
"/DC=org/DC=doegrids/OU=People/CN=Ted Hesselroth 898521" "/cms" uscms
gPlazma now includes a new authorization plugin, to support the XACML
authorization schema.Using XACML with SOAP messaging allows gPlazma to
acquire authorization mappings from any service which supports the obligation
profile for grid interoperability
(http://cd-docdb.fnal.gov/cgi-bin/ShowDocument?docid=2952). Servers presently
supporting XACML mapping are the latest releases of GUMS and SCAS. Using the new
plugin is optional, and previous configuration files are still compatible with
gPlazma. If the installation is an upgrade it will change
/opt/d-cache/config/gPlazma.batch. It is normally not necessary to change this
file, but if you have customized the previous copy, transfer your changes to
the new batch file.
If you wish to use the new plugin, add a line for xacml-vo-mapping in
/opt/d-cache/etc/dcachesrm-gplazma.policy, or rename the file
/opt/d-cache/etc/dcachesrm-gplazma.policy.rpmnew (created by the upgrade process) to
dcachesrm-gplazma.policy and edit it. The following configuration has xacml
mapping turned on and all other plugins turned off.
# Switches
xacml-vo-mapping="ON"
saml-vo-mapping="OFF"
kpwd="OFF"
grid-mapfile="OFF"
gplazmalite-vorole-mapping="OFF"
You will also need to set the endpoint for the xacml plugin by changing the line
for XACMLmappingServiceUrl.
# XACML-based grid VO role mapping
XACMLmappingServiceUrl="https://gums.oursite.edu:8443/gums/services/GUMSXACMLAuthorizationServicePort"
# Time in seconds to cache the mapping in memory
#xacml-vo-mapping-cache-lifetime="0"
If you are running the new GUMS as the result of an upgrade, the URL will be the same as the old
GUMS URL, except for the different service name. SCAS uses a direct secure socket connection and
does not require an endpoint field.
The new GUMS with XACML also supports the older SAML callout, which can be enabled by setting
saml-vo-mapping="ON" in the above set of lines. For the saml-vo-mapping, you will continue to use the
same endpoint as before the upgrade. The corresponding lines will be the same as they were in the
old policy file.
# SAML-based grid VO role mapping
mappingServiceUrl="https://gums.oursite.edu:8443/gums/services/GUMSAuthorizationServicePort"
# Time in seconds to cache the mapping in memory
#saml-vo-mapping-cache-lifetime="0
For testing purposes, one may turn off authorization caching by setting the lifetime to zero.
In production it is advisable to enable caching by setting the lifetime to 60 or 120 (seconds).
****SRM Space Manager Link Group Authorization****
In the link group authorization file, a leading slash before the group name is now required,
except when a user name is being used instead of a true group name. For example, if previously
authorizing the role cmsprod of the group cms with
cms/Role=cmsprod
the line should be changed to
/cms/Role=cmsprod
If a user has no group, the user name as mapped by gPlazma is used in place of the group name.
In this case no slash is used before the user name. For example, if a user with no group is
mapped to the user name cms999, then the line in the LinkGroup authorization file authorizing
the user to make a space reservation is
cms999
Note that no Role=* need be appended to the user name. Use of Role=* such as
/dteam/Role=*
is no longer needed - the wildcard is now assumed if no role is specified, e.g.,
/dteam
authorizes dteam group members with any or no role to make space reservations.
****Default Access Latency and Retention Policy****
The system wide default access latency and retention policy is now defined by the PnfsManager.
Default AccessLatency and RetentionPolicy defined by the variables SpaceManagerDefaultAccessLatency
and SpaceManagerDefaultRetentionPolicy in dCacheSetup will now need to be specified in dCacheSetup
on the PnfsManager node. The ones defined on the SRM node will have no effect.
Start dCache:
++++++++++++
If using the init.d scripts
===========================
Follow the order as shown below
On PNFS node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o PNFS server should be running at this point
If not, start it
> /etc/init.d/pnfs start
On Admin node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o Start dCache core services
> /etc/init.d/dcache start
o Start gratia transfer probe services
> /etc/init.d/gratia-dcache-transfer start
On PNFS node:
o Start the PNFS domain
> /etc/init.d/dcache start
This starts up the pnfsManager, which by default runs on the PNFS
node
On SRM, Gsiftp,Gsidcap nodes:
o Start dCache core services
> /etc/init.d/dcache start
On Pool nodes:
o Start up the pools
> /etc/init.d/pool start
If NOT using the init.d scripts
===============================
Follow the order as shown below
On PNFS node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o PNFS server should be running at this point
If not, start it
> /opt/pnfs/bin/pnfs start
On Admin node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o Start dCache core services
> /opt/d-cache/bin/dcache start
On PNFS node:
o Start the PNFS domain
> /opt/d-cache/bin/dcache start
This starts up the pnfsManager, which by default runs on the PNFS
node
On SRM, Gsiftp,Gsidcap nodes:
o Start dCache core services
> /opt/d-cache/bin/dcache start
On Pool nodes:
o Start up the pools
> /opt/d-cache/bin/dcache start pool
################################################################
# NODE CONFIGURATION SCRIPT #
################################################################
The config-node.pl script will turn the site-info.def.template
file into a site-info.def file. If an existing site-info.def file
is found, it will use those values as defaults.
Internally,
o First, this script invokes the find_java.sh script which
finds java installed on your system in following order:
o Next, it will ask a number of questions for the user to answer.
o It first will ask how many non-pool and non-door nodes
you have. Based on this it makes a recommended
configuration using up to 5 nodes.
o It then asks which services should run on each of these
nodes (Note: node #1 is considered the DCACHE_ADMIN node)
The list of services should be entered on a single line
seperated by white space. The complete list of choices
are:
lmDomain
poolManager
pnfsManager
dirDomain
adminDoor
httpDomain
utilityDomain
gplazmaService
infoProvider
srm
replicaManager
statistics
o Next, the door nodes are configured. One instance each of dcap,
gsidcap, and gridftp will be made to run on each door node.
o Next, the pools with the directory locations on those nodes
will be set. By default, pool sizes will be made to be the size of
the entire partition the pool is are on. If another size is desired
for a pool, edit the file site-info.def, replacing the text "all"
with the desired size.
o Next, gratia probes are configured (gratia probes collect various
statistics about file transfer, space allocation by VOs and pool
utilization). The data collected by the probes will be sent to a
central collector. You will be ask to provide the following information:
URL of central collector
The username, password and port number of dCache admin interface
Name of your SE
Name of the Grid to which the SE belongs (OSG-ITB, OSG, LOCAL)
Also, gratia can notify you if gratia server went down
so you will be asked questions related to email settings
o Finally, it asks if you would like to use the init.d scripts
for starting/stopping various dCache services. If you choose 'y'
the scripts will be copied to the /etc/init.d/ directory,
chkconfig'ed, and permissions will be set to 744.
################################################################
# SITE CONFIGURATION #
################################################################
site-info.def file, defines which service shall run on which node.
If a service is required and no definition is given, it will run
on the admin node.The definition for services which may run on
more than one node may be a space-separated list of
fully-qualified hostnames. If a service needs further information,
such as pool size and location, the may appear in colon-separated
fields after the hostname.
################################################################
# INSTALLATION SCRIPTS #
################################################################
++++++++++++
+ Java +
++++++++++++
Installation script will
o first, check if java exists in $PATH. If it does,
it chooses that version (provided its a valid installation)
o then, check if $JAVA_HOME is defined. If it is,
it chooses that version
o finally, it checks in various standard locations and if
multiple java installations are found, the most recent
version is selected.
To give you an idea how this order works, consider
following situation:
You have a /usr/bin/java that points to a JRE installation
and have another JDK installation as well. In this case,
after you run the config_node.pl script, $JAVA_LOCATION in
your site-info.def will point to the JRE install (Note that
dCache requires JDK). In this case, you either can include
the java from JDK version in $PATH (before the JRE version),
or explicitly set $JAVA_LOCATION in site-info.def file to
point to JDK version.
++++++++++++
+ Postgres +
++++++++++++
Installation script will install Postgres only if
o this is a PNFS server node OR any of SRM, Billing, Space
Manager, Pin Manager or Replica Manager services are
running on this node AND
o it is not installed already
Note: dCache strongly recommends to use atleast version 8 or higher
++++++++++++
+ PNFS +
++++++++++++
Installation script will install PNFS server software only if
o this is the PNFS server node (defined by the value of
variable $DCACHE_PNFS in site-info.def) AND
o NFS server is not running on this node AND
o it (PNFS) is not installed already
Note 1: In a single dCache installation, there can only be one PNFS
server node
Note 2: For existing PNFS installations:
o If PNFS is running when you run the installation script:
The installation script will check that the pnfs export
for the doors has value /pnfsdoors. If not, a warning
is issued and dCache installation might not work.
o If PNFS is not running when you run the install script:
You need to start PNFS first. The dCache installation
will not work till you do so.
++++++++++++
+ dCache +
++++++++++++
Installation script will unpack the dcache-server rpm if necessary
and modify the dCache configuration files
o Services will be turned on and pools will be created
according to the information in site-info.def
o Files etc/node_config.delta and etc/dCacheSetup.delta are
created, which show exactly what lines in etc/node_config
or config/dCacheSetup will be set. These files can be used
for custom settings as well (see CUSTOMIZATION section).
++++++++++++
+ gratia +
++++++++++++
Installation script will unpack the gratia probes on admin and
srm door nodes and will modify the configuration file
- gratia transfer probe will need to be started by
running "service gratia-dcache-transfer start" on admin node
- gratia storage probe is a cronjob that is run by default
every hour. The cron file gratia-probe-dcache-storage.cron
is located in /etc/cron.d
################################################################
# CUSTOMIZATION #
################################################################
Some dCache options not covered by the config-node.pl script may
be set by editing site-info.def after config-node.pl has been run.
MAX_FTP_STREAMS
The maximum number of client streams allowed by gridftp.
GRIDFTP_PRIVATE_NETWORK
The network portion of the ip on the private network of
the pools. For example, if the pools ip's are 192.168.1.x,
enter "192.168.1".
USE_MULTI_MOVER_QUEUES
Sets up default,WAN, and LAN queues on the pools, and
directs gridftp to use WAN and dcap and gsidcap to use LAN.
If desired, set to "yes".
BILLING_USE_DB
Causes billing info to be copied to a database as well as
the log file. If desired,set to "yes".
DCACHE_LOG_DIR
Set to a non-default log directory for dCache domain output
logs. Directory must exist before dCache is started.
Note: In order to correct a mismatch between gridftp and srm
paramters that exists in the default settings of dCache, the
following customization is always done:
REMOTE_GSI_FTP_MAX_TRANSFERS=2000
In addition, etc/node_config.delta and etc/dCacheSetup.delta can be
used to change any other lines in etc/node_config or
config/dCacheSetup. To change or add any other lines in these files,
before running install.sh add the line or lines to the *.delta file
(creating the file if necessary).
################################################################
# CLEANUP #
################################################################
You should follow instructions in this section, ONLY if you tried
to do the installation and all of below mentioned conditions are
true:
o installation didn't finish successfully
o you want to do a complete clean up
o you want to run dCache installation script again
++++++++++++
+ Postgres +
++++++++++++
This will erase all postgresql rpms and delete any existing postgres
databases and installations !! Make a backup of your databases and
anything else, you think you might need later.
Alright, lets begin
o Stop the postmaster
> /etc/init.d/postgresql stop
o Erase following postgresql rpms
> rpm -e postgresql postgresql-libs postgresql-server
o Delete the /var/lib/pgsql directory
> rm -rf /var/lib/pgsql
o As additional check, make sure following files do not
exist
/bin/postmaster
/sbin/postmaster
/usr/bin/postmaster
/usr/sbin/postmaster
/usr/local/bin/postmaster
++++++++++++
+ PNFS +
++++++++++++
This will erase pnfs-server rpm and delete all pnfs directories created
in /opt !! Make a backup of your databases and anything else, you think
you might need later.
Alright, lets begin
o Erase pnfs server rpm
> rpm -e pnfs-postgresql
o Delete various pnfs directories (depending on where your install
failed, all of below might not have created)
> rm -rf /opt/pnfs.3.1.10
> rm -rf /opt/pnfs
> rm -rf /opt/pnfsdb
o Uninstalling pnfs does not unregister its RPC service
(a bug in PNFS). You should do it manually as follows:
> rpcinfo -p | grep nfs | grep -v grep
100003 2 udp 2049 nfs
> rpcinfo -d nfs 2
o Remove the init.d script
> rm -rf /etc/init.d/pnfs
> rm -rf /etx/init.d/dcache
++++++++++++
+ gratia +
++++++++++++
This will erase the gratia rpms.
On both dCache Admin and SRM nodes:
o Erase the following gratia rpms
rpm -e gratia-probe-extra-libs-x
rpm -e gratia-probe-extra-libs-arch-spec-x
rpm -e gratia-probe-common-x
On Admin node:
o Erase the transfer probe rpm
rpm -e gratia-probe-dCache-transfer-x
On SRM node
o Erase the storage probe rpm
rpm -e gratia-probe-dCache-storage-x
#change the "x" to the version of gratia-probe rpms you have installed on your node
++++++++++++
+ dCache +
++++++++++++
This will erase the dcache-server rpm.!!Make a backup of /opt/d-cache and
anything else, you think you might need later.
Alright, lets begin
o Erase dcache-server rpm
> rpm -e --noscripts dcache-server-x.noarch #change the "x" to teh version of dcache-server rpm you have installed on your node
o Delete the d-cache directory
> rm -rf /opt/d-cache
o Delete the init.d script
> rm -rf /etc/init.d/dcache
> rm -rf /etc/init.d/pool (If it was a pool node)
################################################################
# SUPPORT #
################################################################
o For more detailed information about dCache or PNFS, refer the
dCache Book at http:/www.dcache.org/manuals/Book/
o For any issues/questions with/about the the VDT-dCache package,
please send email to osg-storage@opensciencegrid.org