################################################################
# VDT-dCache Package #
################################################################
The purpose of VDT-dCache package is to enable a user to install
and/or upgrade his/her dCache installation.
The package can be downloaded from the VDT website
(http:/vdt.cs.wisc.edu/software/dcache/server/)
################################################################
# SPECIAL NOTE - NEW CONFIGURATION FILE !!! #
################################################################
First thing that you need to do, once you download the package
and untar it, is the following:
If you are doing an upgrade
+++++++++++++++++++++++++++
You need to convert your old "site-info.def" file to the new format.
For this, run the following command before doing anything else -
o cd in to {VDT-DCACHE-INSTALL-HOME}/install directory
you will see file named "siteinfo.conf". This is the site
configuration file that various installation scripts will use.
This file has some "test" values, that will need to be replaced
with ones specific to your site.
o so, to port the values from old "site-info.def" file to new
configuration file "siteinfo.conf", run the following command
./convert.py
Note that this command expects your old configuration file to be
named "site-info.def"
After running this script, open the file "siteinfo.conf" and go
through it, just to make sure you have everything you need in
there.
If you are doing a fresh install
++++++++++++++++++++++++++++++++
Then you don't have any old configuration file. So, just open up the file
named "siteinfo.conf" (present in {VDT-DCACHE-INSTALL-HOME}/install directory)
and edit the values for various parameters according to your site.
################################################################
# SPECIAL NOTE FOR dCache gratia probe INSTALL #
################################################################
Before you install gratia probe, make sure the
following module is installed:
o pyOpenSSL
################################################################
# SPECIAL NOTE FOR SL5 INSTALL #
################################################################
Before you do the dCache install on SL5 node, make sure the
following system-level libraries are installed:
o compat-readline
o openssl
################################################################
# WHICH SECTION OF THIS README SHOULD I READ FIRST? #
################################################################
You can read the entire file or jump to various sections as
necessary. In general:
Purpose Section
+++++++ +++++++
o Understand the directory structure DIRECTORY STRUCTURE
o Perform the installation INSTALLATION
o Perform your site specific customizations CUSTOMIZATION
o Learn about site configuration SITE CONFIGURATION
(siteinfo.conf)
o Learn about install scripts INSTALL SCRIPTS
(java, postgres, pnfs, dCache, gratia probes
jython, dcache core operations and chronicle
rpms from OSG storage operations toolkit)
o Clean up everything CLEANUP
(and begin from scratch)
o Get Help SUPPORT
Note: For PNFS/Postgres Upgrade, read the UPGRADE file
################################################################
# DIRECTORY STRUCTURE #
################################################################
Upon untarring the vdt-dCache tarball, you should see following
directory structure:
RPMS - This directory contains following rpms:
dCache
Postgres
PNFS
srmwatch
Gratia storage and transfer probe
JDK
OSG Operations Toolkit - dCache Core Operations
dCache Chronicle
install - This directory contains the following:
o installation scripts
convert.py - to port values from old conf file to new conf file
install.py - invokes other installation scripts
validate.py - validates the site configuration file
install_java.sh - installs java
install_jython.sh - installs jython
install_dcache.sh - installs dcache
install_pnfs.sh - install pnfs
install_postgres.sh - installs postgres
install_srmwatch.sh - installs srmwatch
install_initd_scripts.sh - installs dcache/pnfs/pool init.d scripts
install_gratia_probes.sh - installs gratia storage/transfer probes
install_dcache_core_operations.sh - installs dcache core operations rpm
install_dcache_chronicle.sh - install dcache chronicle rpm
rpm_unpack.sh
o configuration files
siteinfo.conf - site configuration file that you need to edit
coretext.conf - file that you don't have to change at all, unless
you want to customize your site and not use the
default settings mentioned in this file.
o README - this file
o UPGRADE - file which contains instructions on how to upgrade
Postgres and PNFS software
o config - This directory contains LinkGroupAuthorization.conf,
PoolManager.conf and storage-authzdb. The first two
serve as default configuration files. The last one
is an example authorization file
WARNING: If you are doing an upgrade, your previous
PoolManager.conf will be replaced. A backup of the
previous PoolManager.conf is saved for reference
when customizing PoolManager.conf
o util - This directory contains file "config_file". This is
Yaim Configuration and setup file for dCache 1.7.0-X
o init.d - This directory contains the init.d scripts for following:
pool
pnfs
dcache
################################################################
# INSTALLATION #
################################################################
Few important things to note, before you start the installation
process:
o The install script can be used to do a fresh install
of dCache 1.9.2-z or an upgrade of 1.8.0-x or 1.8.0-y
to 1.9.2-z
o Make sure all nodes on which dCache services will run,
including the pool nodes have valid host certificates
o Make sure NFS is not already running, else PNFS install
will fail
o The install script should be run as user "root"
o The installation process logs go to file "[HOSTNAME]-vdt-install.log"
o Error messages, if any, go to the file "[HOSTNAME]-vdt-install.err"
o /etc/grid-security/certificates directory needs to exist
on all dcache nodes, else authentication will fail. Also,
the contents of this directory - namely, the certificates
and crls have to be updated regularly. There may be several
ways to do this, but one I know works is installing the
OSG WN package. Here is a sample install:
- wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-3.26.tar.gz
- tar -zxvf pacman-3.26.tar.gz
- cd pacman-3.26
- source setup.sh
- mkdir /usr/local/vdt-1.10.1
- cd /usr/local/vdt-1.10.1
- pacman -get OSG:wn-client
Upon successful installation, you will see entries in crontab
file such as:
/usr/local/vdt-1.10.1/fetch-crl/share/doc/fetch-crl-2.6.6/fetch-crl.cron
/usr/local/vdt-1.10.1/vdt/sbin/vdt-update-certs-wrapper --vdt-install /usr/local/vdt-1.10.1 --called-from-cron
Alright, lets begin:
On all dCache nodes:
o Download the VDT-dCache package
o Untar the VDT-dCache package
o cd into the "install" directory
cd vdt-dcache-x.x/install
o (Read the section on NEW CONFIGURATION FILE above and do the needful)
Edit the configuration file named "siteinfo.conf"
and populate all key-value pairs with correct values
for your site.
Note 1: In case of multi-node install, you can copy the
"siteinfo.conf" file to remaining dCache nodes
and skip this step. You may have to edit this file
if Java location is different on these nodes
Note 2: If you are doing an upgrade, file "siteinfo.conf"
already exists" and there are no changes in your
site configuration, you don't have to run the
configuration script again
o Edit the "coretext.conf" file if needed
o Make sure rpcinfo is in your path
First on the PNFS node and then on remaining dCache nodes:
o If you would like to see what the installation script will
do and NOT do the actual install, run
./install.py --dryrun
For usage of this command, run
./install.py --help
When you are ready to do the actual install, run
./install.py
Post Install Configuration:
++++++++++++++++++++++++++
After the installation is over and before you start various dCache
services, you need to make some authorization configuration changes.
You should refer to main dCache website for this
(http:/www.dcache.org/manuals/Book/config/cf-gplazma.shtml), but here
is a minimal check list:
o customize the etc/dcachesrm-gplazma.policy file
o depending on how gPlazma is configured, it will use various
files. Make sure these files exist and have correct information
o Adjust your configuration according to the following:
****Important changes in gPlazma configuration****
gPlazma now persists authorizations in a database. Therefore, postgres
must be installed on the node which is running gPlazma. No special
configuration of postgres is needed by gPlazma, nor will it interfere
with other dCache uses of the database.
gPlazma no longer supports the convention of setting a literal
Role=NULL and/or Capability=NULL when no role or capability exist in a
user's proxy. This will affect sites that are using
/etc/grid-security/grid-vorolemap for authorization and are currently
using the convention. All instances of Role=NULL and Capability=NULL
should be removed from /etc/grid-security/grid-vorolemap. For example,
if a site is currently specifying fully-qualified attribute names
(groups and roles) in a form such as
"/DC=org/DC=doegrids/OU=People/CN=Ted Hesselroth 898521" "/cms/Role=cmsprod/Capability=NULL" cmsprod
"/DC=org/DC=doegrids/OU=People/CN=Ted Hesselroth 898521" "/cms/Role=NULL/Capability=NULL" uscms
such lines should be changed to
"/DC=org/DC=doegrids/OU=People/CN=Ted Hesselroth 898521" "/cms/Role=cmsprod" cmsprod
"/DC=org/DC=doegrids/OU=People/CN=Ted Hesselroth 898521" "/cms" uscms
gPlazma now includes a new authorization plugin, to support the XACML
authorization schema.Using XACML with SOAP messaging allows gPlazma to
acquire authorization mappings from any service which supports the obligation
profile for grid interoperability
(http://cd-docdb.fnal.gov/cgi-bin/ShowDocument?docid=2952). Servers presently
supporting XACML mapping are the latest releases of GUMS and SCAS. Using the new
plugin is optional, and previous configuration files are still compatible with
gPlazma. If the installation is an upgrade it will change
/opt/d-cache/config/gPlazma.batch. It is normally not necessary to change this
file, but if you have customized the previous copy, transfer your changes to
the new batch file.
If you wish to use the new plugin, add a line for xacml-vo-mapping in
/opt/d-cache/etc/dcachesrm-gplazma.policy, or rename the file
/opt/d-cache/etc/dcachesrm-gplazma.policy.rpmnew (created by the upgrade process) to
dcachesrm-gplazma.policy and edit it. The following configuration has xacml
mapping turned on and all other plugins turned off.
# Switches
xacml-vo-mapping="ON"
saml-vo-mapping="OFF"
kpwd="OFF"
grid-mapfile="OFF"
gplazmalite-vorole-mapping="OFF"
You will also need to set the endpoint for the xacml plugin by changing the line
for XACMLmappingServiceUrl.
# XACML-based grid VO role mapping
XACMLmappingServiceUrl="https://gums.oursite.edu:8443/gums/services/GUMSXACMLAuthorizationServicePort"
# Time in seconds to cache the mapping in memory
#xacml-vo-mapping-cache-lifetime="0"
If you are running the new GUMS as the result of an upgrade, the URL will be the same as the old
GUMS URL, except for the different service name. SCAS uses a direct secure socket connection and
does not require an endpoint field.
The new GUMS with XACML also supports the older SAML callout, which can be enabled by setting
saml-vo-mapping="ON" in the above set of lines. For the saml-vo-mapping, you will continue to use the
same endpoint as before the upgrade. The corresponding lines will be the same as they were in the
old policy file.
# SAML-based grid VO role mapping
mappingServiceUrl="https://gums.oursite.edu:8443/gums/services/GUMSAuthorizationServicePort"
# Time in seconds to cache the mapping in memory
#saml-vo-mapping-cache-lifetime="0
For testing purposes, one may turn off authorization caching by setting the lifetime to zero.
In production it is advisable to enable caching by setting the lifetime to 60 or 120 (seconds).
****SRM Space Manager Link Group Authorization****
In the link group authorization file, a leading slash before the group name is now required,
except when a user name is being used instead of a true group name. For example, if previously
authorizing the role cmsprod of the group cms with
cms/Role=cmsprod
the line should be changed to
/cms/Role=cmsprod
If a user has no group, the user name as mapped by gPlazma is used in place of the group name.
In this case no slash is used before the user name. For example, if a user with no group is
mapped to the user name cms999, then the line in the LinkGroup authorization file authorizing
the user to make a space reservation is
cms999
Note that no Role=* need be appended to the user name. Use of Role=* such as
/dteam/Role=*
is no longer needed - the wildcard is now assumed if no role is specified, e.g.,
/dteam
authorizes dteam group members with any or no role to make space reservations.
****Default Access Latency and Retention Policy****
The system wide default access latency and retention policy is now defined by the PnfsManager.
Default AccessLatency and RetentionPolicy defined by the variables SpaceManagerDefaultAccessLatency
and SpaceManagerDefaultRetentionPolicy in dCacheSetup will now need to be specified in dCacheSetup
on the PnfsManager node. The ones defined on the SRM node will have no effect.
Start dCache:
++++++++++++
If using the init.d scripts
===========================
Follow the order as shown below
On PNFS node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o PNFS server should be running at this point
If not, start it
> /etc/init.d/pnfs start
On Admin node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o Start dCache core services
> /etc/init.d/dcache start
o Start gratia transfer probe services
> /etc/init.d/gratia-dcache-transfer start
On PNFS node:
o Start the PNFS domain
> /etc/init.d/dcache start
This starts up the pnfsManager, which by default runs on the PNFS
node
On SRM, Gsiftp,Gsidcap nodes:
o Start dCache core services
> /etc/init.d/dcache start
On Pool nodes:
o Start up the pools
> /etc/init.d/pool start
If NOT using the init.d scripts
===============================
Follow the order as shown below
On PNFS node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o PNFS server should be running at this point
If not, start it
> /opt/pnfs/bin/pnfs start
On Admin node:
o Postgres should be running at this point
If not, start it
> /etc/init.d/postgresql start
o Start dCache core services
> /opt/d-cache/bin/dcache start
On PNFS node:
o Start the PNFS domain
> /opt/d-cache/bin/dcache start
This starts up the pnfsManager, which by default runs on the PNFS
node
On SRM, Gsiftp,Gsidcap nodes:
o Start dCache core services
> /opt/d-cache/bin/dcache start
On Pool nodes:
o Start up the pools
> /opt/d-cache/bin/dcache start pool
################################################################
# SITE CONFIGURATION #
################################################################
"siteinfo.conf" file, defines which service shall run on which node.
If a service is required and no definition is given, it will run
on the admin node.The definition for services which may run on
more than one node may be a space-separated list of
fully-qualified hostnames. If a service needs further information,
such as pool size and location, the may appear in colon-separated
fields after the hostname.
################################################################
# INSTALLATION SCRIPTS #
################################################################
++++++++++++
+ Java +
++++++++++++
Installation script will install the JDK which is provided by the
package and exists under the RPMS directory.
++++++++++++
+ Jython +
++++++++++++
The jython installation script runs only if the node is an admin node
and if you have chosen to install the dcache core operations rpm.
Jython will be installed only if one of below conditions is not true:
o the dcache core operations rpm has already been installed OR
o the directory specified by value of JYTHON_LOCATION in siteinfo.conf
already exists.
In this case, the script assumes that jython has already been installed.
The jython installation process will be interactive, requiring input from you.
IMPORTANT: Standard install of Jython does not provide all packages.
When asked during the dialog, choose 'all/everything' option.
Also, when asked about the "target directory", enter the same value
as that of parameter JYTHON_LOCATION in your siteinfo.conf file.
++++++++++++
+ Postgres +
++++++++++++
Installation script will install Postgres only if
o this is a PNFS server node OR any of SRM, Billing, Space
Manager, Pin Manager or Replica Manager services are
running on this node AND
o it is not installed already
Note: dCache strongly recommends to use atleast version 8 or higher
++++++++++++
+ PNFS +
++++++++++++
Installation script will install PNFS server software only if
o this is the PNFS server node (defined by the value of
variable $DCACHE_PNFS in siteinfo.conf) AND
o NFS server is not running on this node AND
o it (PNFS) is not installed already
Note 1: In a single dCache installation, there can only be one PNFS
server node
Note 2: For existing PNFS installations:
o If PNFS is running when you run the installation script:
The installation script will check that the pnfs export
for the doors has value /pnfsdoors. If not, a warning
is issued and dCache installation might not work.
o If PNFS is not running when you run the install script:
You need to start PNFS first. The dCache installation
will not work till you do so.
++++++++++++
+ dCache +
++++++++++++
Installation script will unpack the dcache-server rpm if necessary
and modify the dCache configuration files
o Services will be turned on and pools will be created
according to the information in siteinfo.conf
o Files etc/node_config.delta and etc/dCacheSetup.delta are
created, which show exactly what lines in etc/node_config
or config/dCacheSetup will be set. These files can be used
for custom settings as well (see CUSTOMIZATION section).
++++++++++++
+ gratia +
++++++++++++
Installation script will unpack the gratia probes on admin and
srm door nodes and will modify the configuration file
- gratia transfer probe will need to be started by
running "service gratia-dcache-transfer start" on admin node
- gratia storage probe is a cronjob that is run by default
every hour. The cron file gratia-probe-dcache-storage.cron
is located in /etc/cron.d
################################################################
# CUSTOMIZATION #
################################################################
etc/node_config.delta and etc/dCacheSetup.delta can be
used to change any other lines in etc/node_config or
config/dCacheSetup. To change or add any other lines in these files,
before running install.sh add the line or lines to the *.delta file
(creating the file if necessary).
################################################################
# CLEANUP #
################################################################
You should follow instructions in this section, ONLY if you tried
to do the installation and all of below mentioned conditions are
true:
o installation didn't finish successfully
o you want to do a complete clean up
o you want to run dCache installation script again
++++++++++++
+ Postgres +
++++++++++++
This will erase all postgresql rpms and delete any existing postgres
databases and installations !! Make a backup of your databases and
anything else, you think you might need later.
Alright, lets begin
o Stop the postmaster
> /etc/init.d/postgresql stop
o Erase following postgresql rpms
> rpm -e postgresql postgresql-libs postgresql-server
o Delete the /var/lib/pgsql directory
> rm -rf /var/lib/pgsql
o As additional check, make sure following files do not
exist
/bin/postmaster
/sbin/postmaster
/usr/bin/postmaster
/usr/sbin/postmaster
/usr/local/bin/postmaster
++++++++++++
+ PNFS +
++++++++++++
This will erase pnfs-server rpm and delete all pnfs directories created
in /opt !! Make a backup of your databases and anything else, you think
you might need later.
Alright, lets begin
o Erase pnfs server rpm
> rpm -e pnfs-postgresql
o Delete various pnfs directories (depending on where your install
failed, all of below might not have created)
> rm -rf /opt/pnfs
> rm -rf /opt/pnfsdb
o Uninstalling pnfs does not unregister its RPC service
(a bug in PNFS). You should do it manually as follows:
> rpcinfo -p | grep nfs | grep -v grep
100003 2 udp 2049 nfs
> rpcinfo -d nfs 2
o Remove the init.d script
> rm -rf /etc/init.d/pnfs
> rm -rf /etx/init.d/dcache
++++++++++++
+ gratia +
++++++++++++
This will erase the gratia rpms.
On both dCache Admin and SRM nodes:
o Erase the following gratia rpms
rpm -e gratia-probe-extra-libs-[VERSION]
rpm -e gratia-probe-extra-libs-arch-spec-[VERSION]
rpm -e gratia-probe-common-[VERSION]
On Admin node:
o Erase the transfer probe rpm
rpm -e gratia-probe-dCache-transfer-itb-[VERSION]
On SRM node
o Erase the storage probe rpm
rpm -e gratia-probe-dCache-storage-itb-[VERSION]
++++++++++++
+ dCache +
++++++++++++
This will erase the dcache-server rpm.!!Make a backup of /opt/d-cache and
anything else, you think you might need later.
Alright, lets begin
o Erase dcache-server rpm
> rpm -e --noscripts dcache-server-[VERSION]
o Delete the d-cache directory
> rm -rf /opt/d-cache
o Delete the init.d script
> rm -rf /etc/init.d/dcache
> rm -rf /etc/init.d/pool (If it was a pool node)
++++++++++
+ Extra +
++++++++++
If dcache core operations and dcache chronicle rpms were installed:
o Erase the dcache core operations rpm
> rpm -e --nodeps osg-dcache-tools-core-operations-[VERSION]
o Erase the dcache chronicle rpm
> rpm -e --nodeps osg-dcache-tools-chronicle-[VERSION]
o Delete the directory in which you installed Jython (if you did)
################################################################
# SUPPORT #
################################################################
o For more detailed information about dCache or PNFS, refer the
dCache Book at http:/www.dcache.org/manuals/Book/
o For any issues/questions with/about the the VDT-dCache package,
please send email to osg-storage@opensciencegrid.org