gLite Data Transfer Agents Module Install Guide Paolo Badino $Id: INSTALL.xml,v 1.3 2005/10/14 10:32:54 badino Exp $ ---------------------------------------------------------------------- Table of Contents Installation Procedure Install External Dependencies Install GLite Data Transfer Agents modules Setup Directories glite Transfer Agent Daemon Configuration Files Setting up the Environment Start the daemon Note on Security Note on ServiceDiscovery Abstract This document describes the procedures that should be followed to install the gLite Data Transfer Agent. Installation Procedure Install External Dependencies In order to use the gLite File Transfer Agent, you need the following external dependencies installed on the system: o Globus Essentials from VDT 1.2.2 o MyProxy from VDT 1.2.2 o Oracle InstantClient o CGSI plugin for GSOAP 2.6 o xerces-c v. 2.5 o xalan-c v. 1.8.0 o log4cpp v. 0.3.4b o log4cxx v. 0.9.7 To install that libraries, you can use the following RPMs from the GLite repository: o gpt-VDT1.2.2rh9-1.i386.rpm o vdt_globus_essentials-VDT1.2.2rh9-1.i386.rpm o myproxy-1.14-EGEE.i386.rpm o perl-Expect.pm-1.01-9.i386.rpm (Required by myproxy-1.14-EGEE.i386.rpm) o CGSI_gSOAP_2.6.slc3-1.1.7-2.i386.rpm o glite-essentials-cpp-1.1.1-1_EGEE.i386.rpm except from the oracle instant client package (oracle-instantclient-basic-10.1.0.3-1.i386.rpm or greater) that should be downloaded from the Oracle web site. Note Please note that at the moment the myproxy VDT versions 1.2.0 and 1.2.2 are incompatible! The locations where those modules are installed are: o Globus: /opt/globus o MyProxy: /opt/globus o Oracle Instant Client: /usr/lib/oracle/10.1.0.3/client/lib/ o CGSI GSOAP: /usr/local/ o xerces-c: /opt/glite/externals/lib o xalan-c : /opt/glite/externals/lib o log4cpp : /opt/glite/externals/lib o log4cxx : /opt/glite/externals/lib Install GLite Data Transfer Agents modules In order to install the GLite GLite Data Transfer Agent, you need to download from the GLite repository the RPMs of the appropiate version of the following modules o org.glite.data.config-service o org.glite.data.catalog-api-c o org.glite.data.srm-api-c o org.glite.data.srm-cli o org.glite.data.transfer-url-copy o org.glite.data.transfer-agents o org.glite.servicediscovery.api-c o org.gridsite.core You can then install those RPMs issuing the command rpm -ivh here is the list of RPMS of the modules reported above. The default location where those modules are installed is /opt/glite. Setup Directories Create the following directories under the glite location directory (assuming /opt/glite): mkdir /opt/glite/etc mkdir /var/log/glite mkdir /var/glite/lock mkdir /var/glite/tmp glite Transfer Agent Daemon Configuration Files The VO agents are intended to be installed one for each VO, you need to create a pair of the configuration files for each VO. The same also apply for a Channel Agent: you need an agent for each Channel you want to mange. At this purpose, you can run the GLite Data ConfigGenerator tool, passing the proper configuration template provided by the org.glite.data.transfer-agents module and installed in /opt/glite/share/config/glite-data-transfer-agents. In order to configure a GLite Data Transfer VO Agent, for example, you need to run: export GLITE_LOCATION=/opt/glite export LD_LIBRARY_PATH=$GLITE_LOCATION/externals/lib:\ $GLITE_LOCATION/lib:$LD_LIBRARY_PATH /opt/glite/bin/glite_data_config_generator \ -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-vo-agent--.config.xml \ -o /opt/glite/etc/glite-transfer-vo-agent-.properties.xml \ -l /opt/glite/etc/glite-transfer-vo-agent-.log-properties where is the name of the DAO plugin to use (can be "mysql" or "oracle"), is the chosen deployment model ("fts" or "fps") and is the name of the Virtual Organization the agent will serve. For more details concerning the deployment scenarios (File Transfer Service or File Placement Service), please have a look to the README file. The output destination must be either $GLITE_LOCATION_USER/etc, $GLITE_LOCATION/etc or $GLITE_LOCATION_VAR/etc. All you need to doi is to enter the value of the configuration parameters that you want to put into the configuration. If you wnat to simplify the configuration, don't switch to the advanced mode and fill just the mandatory parameters, leaving the default value for all the others. In order to create the cofniguration for a Channel agent, you need to run: /opt/glite/bin/glite_data_config_generator \ -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-channel-agent-.config.xml \ -o /opt/glite/etc/glite-transfer-channel-agent-.properties.xml \ -l /opt/glite/etc/glite-transfer-channel-agent-.log-properties where is the name of the DAO plugin to use (can be "mysql" or "oracle"), and is the name of the Channel the agent is responsible for. In order to help the user in generating the configuration files, two additional templates have been provided with just the properties required by the supported transfer service /opt/glite/bin/glite_data_config_generator \ -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-channel-agent--.config.xml \ -o /opt/glite/etc/glite-transfer-channel-agent-.properties.xml \ -l /opt/glite/etc/glite-transfer-channel-agent-.log-properties where is the name of the Urlcopy Transfer Type configuration (can be "urlcopy" or "srmcopy"). You can also run the glite_data_config_generator tool without user interaction. To do that, you need to use the -p option and provide property file (key-value pairs) with the values you want to assign to the configuration parameters. /opt/glite/bin/glite_data_config_generator \ -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-vo-agent--.config.xml \ -o /opt/glite/etc/glite-transfer-vo-agent-.properties.xml \ -l /opt/glite/etc/glite-transfer-vo-agent-.log-properties -p -.properties> where -.properties> is the property file you have to pass. For a Channel Agent, you need to run: /opt/glite/bin/glite_data_config_generator \ -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-channel-agent-.config.xml \ -o /opt/glite/etc/glite-transfer-channel-agent-.properties.xml \ -l /opt/glite/etc/glite-transfer-channel-agent-.log-properties -p .properties> The structure of that property file requires that the key should be composed by the name of the component, as reported on the config template file, followed by the property name (separeted by a "."), and then "=" and the value to be assigned. . = . = ... . = log.FileName = log.Priority = ("log" is a reserverd key to generate the logging configuration file) If this option is used, rember that the property file should set all the mandatory properties, otherwise an error is returned. Note Due to a problem with the logging library not allowing to properly roll the files, it's strongly suggested to set the log Proprity at least to INFO or WARN, otherwise the scheduler will produce too many entries. From the org.glite.data.config-service version 1.1.10 it would be possible to provides multiple properties files as input parameters (specifying multiple -p options), as well as specifying directly the value of some parameters on the command line (using the -O option). These two methods can also be combined. You can then think to configure an agent in a similar way: /opt/glite/bin/glite_data_config_generator \ -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-channel-agent-.config.xml \ -o /opt/glite/etc/glite-transfer-channel-agent-.properties.xml \ -l /opt/glite/etc/glite-transfer-channel-agent-.log-properties -p db_connection.properties -p timeout.properties -O transfer-channel-agent.Name= -O transfer-channel-agent.Contact= The properties files can then be reused for installing different agent instances. Since the FTA configuration files (/opt/glite/etc/glite-transfer-vo-agent-.properties.xml and /opt/glite/etc/glite-transfer-channel-agent-.properties.xml) may contain some sensible information (like the user and the password to connect to the DB server), it's recommended that you change the privileges of these files so they can be read just by the account that is used to run the agent. Setting up the Environment Before running the daemon, you need to set up the environment variables GLITE_LOCATION_USER, GLITE_LOCATION, GLITE_LOCATION_VAR and LD_LIBRARY_PATH. You can do that either running the command illustrated above, creating a script or setting them in your shell profile. Remember to include GLITE_LOCATION/lib into the LD_LIBRARY_PATH. If Oracle is used as Data Source, the Oracle client library location should also be listed in LD_LIBRARY_PATH. In addition, GLITE_LOCATION_USER, GLITE_LOCATION and GLITE_LOCATION_VAR could be set in the ~/.glite.conf or /etc/glite.conf and described by the GLite Developer's Guide. Start the daemon Once all the previous steps are completed, you can run the daemon by using the following command (always with the service account credentials): /opt/glite/bin/glite_data_config_service glite-transfer-vo-agent- start or /opt/glite/bin/glite_data_config_service glite-transfer-channel-agent- start where is the name of the VO which the agent belongs to and is the name of the channel which the agent is responsible for. Note on Security Since the FTA will try to contect MyProxy in order to get the client delegated credentials for performing the transfer, contacting the ServiceDiscovery client and the Catalog, the use of a service/host certificate is compulsory. Note on ServiceDiscovery The File Transfer Agent make use of the ServiceDiscovery in order to find the appropriate endpoint for the Catalog, the Storage Element and the MyPRoxy Server (if not configured). Moreover, the information returned by ServiceDiscovery is used to detect the site involved on a transfer, in order to properly perform the channel allocation. This then requires that the ServiceDiscovery environement is properly set in order to include the information concerning all the Catalog and SRM Servers that could be involved in the transfer and replication, i.e. in all the sites reachables through one of the available channels. Since it's foreseen that different backends could be used as information system (file, RGMA, DBII), please refer to the ServiceDiscovery documentation to setup such environment. For what concern the properties that the ServiceDiscovery should provide, the FTA requires that the information system contains the entries related to at least one Fireman catalog (service type is org.glite.FiremanCatalog) (only in case of FPS) and to any possible SRMs (service type SRM) that could be involved in a transfer. The SRM service should specify an additional property (:SEMountPoint) that specify the parent path where the files would be stored. The prefix is used in order to allow to use a different mount point for each VOs. In case this parameter is the same for all the VO the property can be named simply SEMountPoint. All of these types are configurables as advanced parameters of the servicediscovery client module. An example of a configuration for the file based ServiceDiscovery is: https://lxb2028.cern.ch:8443/EGEE/glite-data-catalog-service-fr/services/FiremanCatalog org.glite.FiremanCatalog 2.0.0 cern.ch httpg://griftp.infn.it:8443/srm/managerv1 SRM 1.1.0 infn.it /castor/infn.it/grid/dteam/ /castor/infn.it/grid/alice/ /castor/infn.it/tmp/ httpg://griftp05.cern.ch:8443/srm/managerv1 SRM 1.1.0 cern.ch /castor/cern.ch/user/g/gproduct/EGEETEST/SE/