##############################################################################################
#                       	DCACHE UPGRADE                                               #
##############################################################################################
The purpose of this document is to provide instructions for dCache upgrade procedure.

If existing namespace is PNFS
==============================
Assumptions:
        o You have a working dCache storage element
        o You are using pnfs namespace
        o You want to upgrade dCache (from 1.9.5-x to 1.9.5-y)

General procedure:
	o Keep Postgres servers running on all relevant nodes
	o Keep PNFS Server running on PNFS node
	o Stop all other dCache services on all relevant nodes (order - pools, gridftp/dcap doors, srm node, pnfs node, admin node)
	o Unmount /pnfs on all nodes except PNFS node itself
	o First on PNFS node and then on remaining nodes
		- cd in to $NEW-VDT-DCACHE-HOME/install
		- Modify siteinfo.conf according to your needs (or copy it over from pnfs node)
		- run install.py (with --dryrun option first)
	o Do Post install configuration (refer to INSTALL document)
	o Start dCache services in correct order (order - admin node,pnfs node, srm node, other door nodes, pool nodes )
	o Verify basic functionality is working
	o Follow instructions in MIGRATION document to now move from PNFS to Chimera

If existing namespace is CHIMERA
================================

Assumptions:
	o You have a working dCache storage element 
	o You are using chimera namespace already 
	o You want to upgrade dCache (from 1.9.5-x to 1.9.5-y)

Procedure::
 o Stop all currently running dCache services (follow the order)
	On Pool nodes:
		> /etc/init.d/pool stop
	On Gridftp and other door nodes:
		> /etc/init.d/dcache stop
	On SRM node:
		> /etc/init.d/dcache stop
	On Chimera node:
		> /etc/init.d/dcache stop
	On Admin node:
		> /etc/init.d/dcache stop
		- If enabled, stop gratia transfer probe service
                  > /etc/init.d/gratia-dcache-transfer stop

 o Unmount Chimera filesystem
	On SRM node:
		umount /pnfs
	On dCap node:(Only if you had it mounted)
		umount /mnt (or whatever the mount point is)
	On Chimera node:
		umount /pnfs
	
 o Keep postgres running on all nodes
 
 o Upgrade dCache on Chimera node first
	- make sure chimera fs is not mounted (run df)
	- stop chimera nfs server
		/etc/init.d/chimera-nfs-run.sh stop
	- cd $NEW-VDT-DCACHE-HOME/install
	- edit siteinfo.conf according to your needs
	- run install script 
		> ./install.py 
		sample output ---- (note the mount error below-- It is OK because chimera nfs 
					server is not running at this point)
		VDT install of dCache on: 
		2010-07-29 15:23:48.444526
		Unpacking dCache server software.
		Checking services running on this node ...
		PostgreSQL is already installed and running. Nothing will be done on this node.
		You have chosen NOT to setup Chimera.
		Installing dCache -
		Transferring previous settings from dCacheSetup
		DCACHE_REPLICA_MANAGER ... gwdca06.fnal.gov
		DCACHE_DOOR_SRM_IGNORE_ORDER ... true
		SRM_SPACEMGR_ENABLED ... yes
		SRM_IMPLICIT_SPACE_RES ... no
		SRM_LINK_GROUP_AUTH_FILE ... /opt/d-cache/etc/LinkGroupAuthorization.conf
		Modifying dCache configuration files
		Running dCache setup script.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		mount: mount to NFS server 'localhost' failed: System Error: Connection refused.
		Finished installing dCache.
		dCache rpm is already unpacked.
		Checking services running on this node ...
	- check the dcache has been upgraded
		> rpm -qa | grep dcache
	- start chimera nfs server
		> /etc/init.d/chimera-nfs-run.sh start
		Starting Chimera-NFSv3 interface
		Waiting for NFS server to register itself: ..
	- at this point chimera fs is not mounted. can run 'df' to check
		> df
	- mount it
		> mount localhost:/pnfs /pnfs
	- run 'df' to confirm it is mounted and also 'ls' on it to make sure all your old data 
	  files are there

 o Upgrade rest of the nodes (recommended order - admin, srm, doors, pools)
	- Login to the node
	- cd $NEW-VDT-DCACHE-HOME/install
        - edit siteinfo.conf according to your needs or copy from chimera node
        - run install script
               > ./install.py
	- check the dcache has been upgraded
               >  rpm -qa | grep dcache
 o Do Post Configuration settings for PoolManager.conf and authorization (refer to INSTALL)
 o Start services (follow the order)
	 On Admin node:
		- Postgres should be running at this point. If not, run
                	> /etc/init.d/postgresql start
    		- Start dCache core services
                	> /etc/init.d/dcache start
                	(this will also bring up pool, if its also a pool node)
    		- If enabled, start gratia transfer probe service
                	> /etc/init.d/gratia-dcache-transfer start
   	On Chimera node:
    		- Start the Chimera domain
                	> /etc/init.d/dcache start
                	(this starts up the pnfsManager, which by default runs on the Chimera node)
   	On SRM node:
   		- Postgres should be running at this point. If not, run
                	> /etc/init.d/postgresql start
   	On SRM, Gsiftp,Gsidcap nodes:
    		- Start dCache core services
                	> /etc/init.d/dcache start
   	On Pool nodes:
    		- Start up the pools
                	> /etc/init.d/pool start
	As a simple first level check, visit "http://[ADMIN_NODE]:2288"
	It should list all the services that are running and on which node
You should be all set!