Commit 1f5627ab authored by Florido Paganelli's avatar Florido Paganelli
Browse files

Changed README file after decisions taken 2019-01-16

  * Removed obsolete or irrelevant sections like X509, A-REX, Setup and Config, Testing
  * Structure rearranged as decided in the meeting
  * Added references to existing webpages/documents instead of rewriting content
  * Updated links and statements about support
parent f15ad316
-------------------------------------------------------- =====================
# ARC MIDDLEWARE README ARC MIDDLEWARE README
-------------------------------------------------------- =====================
##### Last updated: 2019-01-11 Last updated: 2019-01-16
-------------------------------------------------------- `````````````````````````
Suggested structure
========================================
* Purpose of this document
* To point the reader to existing documentation
* To describe how to build from sources
We should assume who reads could be one or more of:
* reading github https://source.coderefinery.org/nordugrid/arc/blob/master/README
* has downloaded the tarball
* has cloned the git repo
* is a user
* is a developer
* Short description of the SW
* Alternatives to building (repos)
* Building the software
* Getting the software
* Build prerequisites
* Build dependencies
* Using distros
* Build process
* Without distros
* Mandatory
* Runtime
* ldap (maybe not needed after EGIIS phase out?)
* Optional
* Build
* Build & runtime
* Runtime
* Build process
* Installation, configuration and testing
* just point to relevant pages
* Contributing
* Developer's pages, mainly git wiki
* Contacts
* Support, documentation, mailing lists, contact
ARC Middleware
==============
The Advanced Resource Connector (ARC) middleware, developed by the Purpose of this Document
NorduGrid Collaboration (www.nordugrid.org), is an open source software ========================
solution enabling computing grid infrastructures with emphasis on
processing of large data volumes. ARC is being used to enable national
and international e-infrastructures since its first release in 2002.
This document is meant to introduce the reader to the
software and to point to existing documentation.
Dependencies Table of contents
============ =================
The core part of middleware is written in C/C++. Building the software * What is the ARC Middleware
from source or installing a pre-compiled binary requires different * License
external packages, furthermore the client and server packages have * Getting the software
different dependencies too. * Installation
* Configuration and Testing
Mandatory dependencies * Contributing
---------------------- * Support, Mailing lists
* Documentation
Build prerequisites: * Contact
o GNU make, autotools (autoconf>=2.56) (automake>=1.8)
o CVS
o m4
o GNU gettext
o C++ compiler and library
o libtool
o pkg-config
o doxygen
ARC supports rpm and apt based distributions, if you plan to build for
one of those such as CentOS or Ubuntu, follow the distribution-specific
instructions to resolve the rest of the dependencies.
If you want to build without any of those package managers, here's a
list of explicit requirements. Note that the list might not be
complete.
Build & runtime:
o e2fsprogs
o gthread-2.0 version 2.4.7 or later
o glibmm-2.4 version 2.4.7 or later
o libxml-2.0 version 2.4.0 or later
o openssl version 0.9.7a or later
If you are using LDAP based infosys:
o open-ldap server
o bdii version 5 or later
o glue-schema
Optional dependencies
----------------------
Build:
o CppUnit for unit testing
o Grid Packaging Tools (GPT) (compute client)
o swig version 1.3.28 or later (bindings)
Build & runtime:
o open-ldap client (LDAP DMC)
o python 2.4 or higher (bindings, APEL publisher by Jura, ACIX)
o java sdk 1.4 or later (bindings)
o globus-common 4 (compute client)
o globus-gssapi-gsi 4 (compute client)
o globus-ftp-client 4 (compute client)
o globus-ftp-control 4 (compute client)
o globus-io 4 (compute client)
o globus-openssl (compute client)
o xrootd (xrootd DMC)
o GFAL 2 (GFAL DMC)
o Berkeley DB C++ interface (Delegation)
o xmlsec1 1.2.4 or higher (Security)
o NSS 3 (credential)
Runtime dependencies:
o Perl, libxml-simple-perl, perl-Digest-SHA1, libjson-xs-perl (A-rex)
o Perl, perl-SOAP-Lite, perl-Crypt-OpenSSL-X509 (nordugridmap)
o perl-DBI (Boinc backend)
o GNU time (A-rex)
o VOMS
o Stomppy, m2Crypto (APEL publisher by Jura)
o pyOpenSSL, python-twisted-web, python-twisted-core, python-simplejson,
(Python 2.4 only) python-hashlib (ACIX)
Please note that depending on operating system distribution to build
ARC you may need to install development versions of mentioned packages.
# Getting the software
The middleware is free to deploy anywhere by anybody. Binary packages
are available from Ubuntu, Fedora and EPEL repositories, as well as
from NorduGrid's own repositories, as described at
http://download.nordugrid.org
The software code is released under the Apache 2.0 License (see the What is the ARC Middleware
LICENSE file). It is available from the NorduGrid's Subversion ==========================
repository, see http://svn.nordugrid.org for more details.
The Advanced Resource Connector (ARC) middleware, developed by the
NorduGrid Collaboration (<httpwww.nordugrid.org>), is an open source software
solution enabling e-Science computing infrastructures with emphasis on
processing of large data volumes.
ARC is being used to enable national and international e-infrastructures
since its first release in 2002.
License
=======
The software code is released under the Apache 2.0 License (see the
LICENSE file).
The necessary 3rd party libraries are usually available in standard The necessary 3rd party libraries are usually available in standard
Linux distributions; in rare cases NorduGrid repositories offer them. Linux distributions; in rare cases NorduGrid repositories offer them.
There are also nightly code snapshots and package builds available at Getting the software
http://download.nordugrid.org/nightlies/packages/nordugrid-arc/trunk/ . ====================
# Building & Installation
> Q1: Do we still release the tarball? @maiken
> Q2: Shall we instead just point at some git release? @all
The recommended way to install ARC is from repositories. If you want
to build it yourself, download the tarball, unpack it and cd into
the created directory (below, replace 6.0.0 with the desired tag):
tar -zxvf nordugrid-arc-6.0.0.tar.gz
cd nordugrid-arc-6.0.0
---REVISEBELOW
If you obtained the code from the Subversion repository, use the
'tags/1.0.0' directory.
svn co http://svn.nordugrid.org/repos/nordugrid/arc1/tags/1.0.0 nordugrid-arc
cd nordugrid-arc
Now configure the obtained code with
./autogen.sh
./configure --prefix=PLACE_TO_INSTALL_ARC
Choose the installation prefix wisely and according to the
requirements of your OS and personal preferences. ARC should function
properly from any location. By default installation goes into /usr/local if
you omit the '--prefix' option. If you install into another directory
than /usr/local you may need to set up an environment variable after
installation:
export ARC_LOCATION=PLACE_TO_INSTALL_ARC
On some systems 'autogen.sh' may produce few warnings. Ignore them as
long as 'configure' passes without errors. But in case of problems
during configure or compilation, collect them and present while
reporting problems.
If the previous commands finish without errors, do
touch src/doxygen/*.pdf
in order to get around an issue with timestamps and then compile and
install ARC:
make
make install
If you have already installed ARC libraries in the system default
location such as /usr/lib you may need to use the following
installation command instead in order to override installed pkgconfig
files and/or libtool archives which contain -L/usr/lib:
make LDFLAGS="-L<PLACE_TO_INSTALL_ARC>/lib" install All the information about how to get the software is available at
On some systems you may need to use gmake instead of make.
Depending on the chosen installation location you may need to run the last
command from a root account. That should install the following components:
sbin/arched - server executable
etc/init.d - daemon init scripts
bin/ - user tools and command line clients
lib/ - common libraries used by clients, server and plugins
lib/arc/ - plugins implementing Message Chain, Service and Security components
include/arc/ - C++ headers for application development
libexec/ - additional modules used by ARC services - currently only A-REX
share/arc - configuration examples, templates etc
share/doc/nordugrid-arc-* - documentation
share/locale - internationalization files - curently very limited support
share/man - manual pages for various utilities
X509 Certificates
=================
All ARC services use HTTPS or GridFTP as transport protocol so they require
proper setup of an X509 security infrastructure. Minimal requirements are:
* Host certificate aka public key in PEM format
* Corresponding private key
* Certificate of the Certification Authority (CA) which was used to sign the
host certificate
* Certificates of CAs of clients which are going to send requests to services,
unless of course clients use the same CA as the server.
More information about X509 certificates and their usage in Grid environment
can be found on http://www.nordugrid.org/documents/certificate_howto.html
and http://www.nordugrid.org/documents/arc-server-install.html#security
For testing purposes you can use pre-generated certificates and
keys available at:
http://svn.nordugrid.org/trac/nordugrid/browser/doc/trunk/tech_doc/sec/TestCA
Alternatively, you may choose to use KnowARC Instant CA service available at
https://arc-emi.grid.upjs.sk/instantCA/instantCA . It is especially useful when
testing installations consisting of multiple hosts.
Please remember that it is not safe to use such instant keys in publicly
accessible installations of ARC. Make sure that even the generated CA
certificate is removed before making your services available to the
outside world.
You can put host certificates and private keys anywhere. Common locations
for servers running from root account are /etc/grid-security/hostcert.pem
and /etc/grid-security/hostkey.pem, respectively. The content of the
private key must not be encrypted nor protected by a password since
a service has no way to ask a password. Therefore it must be properly protected
by means of file system permissions (some services enforce that the private key
is only readable by the user running the service).
It is possible to configure the ARC server to accept either a single CA
certificate or multiple CA certificates located in the specified directory. The
latter option is recommended. The common location is
/etc/grid-security/certificates/ . In that case the names of the certificate
files have to follow the hash values of the certificates. These are obtainable
by running the command
openssl x509 -hash -noout -in path_to_certificate
The corresponding file name for the certificate should be <hash_value>.0 .
The value for the pre-generated CA certificate is 4457e417.
For the ARC client tools you may use the pre-generated user certificate and key
located at the same place above. Generally the key and certificate are not used
directly but a passphraseless proxy certificate is generated and used instead.
ARC comes with a proxy generation utility arcproxy - see 'man arcproxy' for
usage and options. Locations of the credentials are provided to the client
tools via the client configuration file.
The set of pre-generated keys and certificates also includes a user
certificate in PKCS12 format which you can import into your browser
for accessing ARC services capable of producing HTML output.
IMPORTANT: If during the configuration stage you see a message "OpenSSL
contains no support for proxy credentials" that means you won't be
able to use proxy credentials generated by utilities like grid-proxy-init,
voms-proxy-init or arcproxy. Because of that all user private keys must
be kept unencrypted.
<http://download.nordugrid.org/>
Installation
============
ARC Server Setup & Configuration If you want to install for a specific distribution, see the list of
================================ supported distributions here:
The configuration of the ARC server is specified in a file which by default <http://download.nordugrid.org/repos.html>
is at /etc/arc.conf. A different location can be specified by the ARC_CONFIG
environment variable. For configuration details and examples please refer to And follow the instructions here:
the reference in share/doc/nordugrid-arc/arc.conf.reference or the service
manual of the particular services you wish to run.
<http://www.nordugrid.org/documents/arc6/user/index.html>
If you want to build and install the software yourself, please read
The A-REX Service <TBD>
=================
ARC comes with an OGSA BES compliant Grid job management service called A-REX. Configuration and Testing
To deploy A-REX refer to "ARC Computing Element: System Administrator Guide" =========================
(NORDUGRID-MANUAL-20) which contains extensive information on set up and
configuration of A-REX.
Testing and Using A-REX (clients)
=================================
Instructions below refer to the Web Service interface; similar tests
can be done for the original GridFTP-based interface (gsiftp:// protocol).
Now you may use the command line utility 'arcinfo' to obtain a service
description. You can do something like
arcinfo -c https://localhost:60000/arex -l
This should produce a description list of the resources A-REX represents. Below
you can see a truncated example of proper output.
---
Cluster: localhost
Health State: ok
Location information:
Domain information:
Service information:
Service Name: MINIMAL Computing Element
Service Type: org.nordugrid.execution.arex
Endpoint information:
URL: https://localhost:60000/arex
Capabilities:
executionmanagement.jobexecution
Technology: webservice
Interface Name: OGSA-BES
Supported Profiles:
WS-I 1.0
HPC-BP
Implementor: NorduGrid
Implementation Name: A-REX
Implementation Version: 0.9
QualityLevel: development
Health State: ok
Serving State: production
Issuer CA: /O=Grid/O=NorduGrid/CN=NorduGrid Certification Authority
Trusted CAs:
/C=BE/O=BELNET/OU=BEGrid/CN=BEGrid CA/emailAddress=gridca@belnet.be
/C=FR/O=CNRS/CN=CNRS2-Projets
/DC=org/DC=ugrid/CN=UGRID CA
[...]
Staging: staginginout
Job Descriptions:
ogf:jsdl:1.0
Queue information:
Mapping Queue: default
Max Total Jobs: 100
Max Running Jobs: 10
Max Waiting Jobs: 99
Max Pre LRMS Waiting Jobs: 0
Max User Running Jobs: 5
Max Slots Per Job: 1
Doesn't Support Preemption
Total Jobs: 0
Running Jobs: 0
Waiting Jobs: 0
Suspended Jobs: 0
Staging Jobs: 0
Pre-LRMS Waiting Jobs: 0
Free Slots: 10
Free Slots With Duration:
P68Y1M5DT3H14M7S: 10
Used Slots: 0
Requested Slots: 0
Manager information:
Resource Manager: torque
Doesn't Support Advance Reservations
Doesn't Support Bulk Submission
Total Physical CPUs: 10
Total Logical CPUs: 10
Total Slots: 10
Non-homogeneous Resource
Working area is nor shared among jobs
Working Area Total Size: 15
Working Area Free Size: 4
Working Area Life Time: P7D
Cache Area Total Size: 15
Cache Area Free Size: 4
Execution Environment information:
Execution environment is a physical machine
Execution environment does not support inbound connections
Execution environment does not support outbound connections
---
A-REX accepts jobs described in XRSL, which is described in "Extended Resource
Specification Language: Reference Manual for ARC versions 0.8 and above"
(NORDUGRID-MANUAL-4). To submit a job to the A-REX service one may use the
'arcsub' command:
arcsub -c https://localhost:60000/arex -f simple.xrsl
If everything goes fine, somewhere in its output there should be a message
"Job submitted", and a job identifier is obtained which will be stored locally
in a client job store. One can then query job state with the 'arcstat' utility:
arcstat <job id>
State: Running
arcstat <job id>
State: Finished
For more information on these and other arc* job and data management commands
please see the man pages of those utilities or "ARC Clients: User Manual for
ARC 11.05 (client versions 1.0.0) and above" (NORDUGRID-MANUAL-13).
See the official online documentation:
<http://www.nordugrid.org/documents/arc6/user/arc6_install_guide.html>
Contributing Contributing
============ ============
...@@ -438,35 +77,47 @@ The open source development of the ARC middleware is coordinated by ...@@ -438,35 +77,47 @@ The open source development of the ARC middleware is coordinated by
the NorduGrid Collaboration which is always open to new members. the NorduGrid Collaboration which is always open to new members.
Contributions from the community to the software and the documentation Contributions from the community to the software and the documentation
is welcomed. Sources can be downloaded from the software repository is welcomed. Sources can be downloaded from the software repository
at download.nordugrid.org or the Subversion code repository at at <http://download.nordugrid.org> or the Coderefinery gitlab repository
svn.nordugrid.org. at <https://https://source.coderefinery.org/nordugrid/arc>
The technical coordination group defines outstanding issues that have The Technical Coordination Group (TCG) defines outstanding issues that
to be addressed in the framework of ARC development. Feature have to be addressed in the framework of ARC development. Feature
requests and enhancement proposals are recorded in the Bugzilla bug requests and enhancement proposals are recorded in the Bugzilla bug
tracking system at bugzilla.nordugrid.org. For a more detailed tracking system at <https://bugzilla.nordugrid.org>. For a more detailed
description, write access to the code repository and further description, write access to the code repository and further
questions, write to the nordugrid-discuss mailing list (see questions, subscribe and write to the *nordugrid-discuss* mailing list:
www.nordugrid.org for details). Ongoing and completed Grid research <https://mail.nordugrid.org/mailman/listinfo>
projects and student assignments related to the middleware are listed
on the NorduGrid web site as well.
Ongoing and completed e-Science research projects and student
assignments related to the middleware are listed
on the NorduGrid web site: <http://www.nordugrid.org>
Support, documentation, mailing lists, contact Support, Mailing lists
============================================== ==============================================
User support and site installation assistance is provided via the User support and site installation assistance is provided via
request tracking system available at nordugrid-support@nordugrid.org. several mailing lists, among which the nordugrid-discuss mailing list
In addition, the NorduGrid runs several mailing lists, among which the is a general forum for all kind of issues related to the ARC middleware.
nordugrid-discuss mailing list is a general forum for all kind of <https://mail.nordugrid.org/mailman/listinfo>
issues related to the ARC middleware. The Bugzilla problem tracking system
(bugzilla.nordugrid.org) accepts requests for features or enhancements, The Bugzilla problem tracking system accepts requests for features or
and is the prime medium to track and report problems. enhancements, and is the prime medium to track and report problems.
<https://bugzilla.nordugrid.org>
Documentation
=============
Research papers, overview talks, reference manuals, user guides, Research papers, overview talks, reference manuals, user guides,
installation instructions, conference presentations, FAQ and even installation instructions, conference presentations, FAQ and even
tutorial materials can be fetched from the documentation section of tutorial materials can be fetched from the documentation section of
www.nordugrid.org <http://www.nordugrid.org>
ARC6 specific documentation is at
<http://www.nordugrid.org/documents/arc6/>
Contact
=======
Contact information is kept updated on the www.nordugrid.org web site. Contact information is kept updated on the <http://www.nordugrid.org>
web site.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment