Commit 79788ecf authored by Andrii Salnikov's avatar Andrii Salnikov

Merge branch 'restructuring' into 'master'

Docs tree restructuring

See merge request !1
parents 9fab970e 9ce005bd
Pipeline #4294 passed with stages
in 17 minutes and 50 seconds
......@@ -12,7 +12,6 @@ build_docs:
- dnf -y install texlive-collection-latexextra latexmk
- dnf -y install python-isodate python-ldap
- dnf -y install doxygen
- cd playground
- bash
- tar czf $CI_PROJECT_DIR/sphinx-docs.tar.gz -C build/html .
- mv build/html $CI_PROJECT_DIR/
Contributing to Documentation
NorduGrid ARC6 documentation is mainly written in `reStructuredText <>`_ and build with `Sphinx <>`_ to HTML pages, LaTeX (for printable PDF versions) and ePub.
Contribution is possible by direct push to the repo, no need for merge requests!
Recent source tree CI ``master`` build is available instantly via `GitLab Pages <>`_ and deployed to the ` <>`_ on nightly basis.
Documentation structure
As reflected in the :doc:`main index <index>` page, the documentation logically divided into the following groups:
* :ref:`users_docs`
* :ref:`admins_docs`
* :ref:`developers_docs`
* :ref:`tech_docs`
Admins documentation is the most developed part for ARC6. We aimed to have dedicated documents for each particular topic and then bound them in the following way:
* The introductory :doc:`/admins/try_arc6` that contains a zero-configuration case hands-on instruction to help new users and admins getting started with ARC CE.
* Main :doc:`/admins/arc6_install_guide` that contains general installation and configuration flow for production *Computing Element* deployment. The guide itself should be brief enough and holds only main examples. All detailed instruction for each particular subsystem configuration should be written in the dedicated document that linked to this guide.
* :doc:`/admins/reference` automatically rendered from text version in the code source tree. Targeted for web-search. The ultimate configuration options description we have.
* All other dedicated documents goes to :doc:`/admins/details/index` section in case someone wants to access them directly instead of following installation guides links.
Source tree directory structure
Sources of documents in rST format are placed into the ``source`` directory that contains the following structure:
* ``_static`` - logo and CSS files [#f1]_
* ``_templates`` - Sphinx HTML theme layout tuning [#f1]_
* ``admins`` - directory used to store :ref:`admins_docs`. The main documents that represent the entry points to *ARC Computing Element* installation and configuration resides directly in this directory.
* ``details`` - this subdirectory designed to hold detailed configuration instructions for ARC CE subsystems, like :doc:`/admins/details/rtes` that describe all aspects of RTEs in-depth. This documents intentionally moved deeper in the main TOC tree to prevent first levels flooding. If you can find a chapter in main :doc:`install guide </admins/arc6_install_guide>` that cat point to the in-depth document in question - this is a right place to store it.
* ``commands`` - place for automatically generated command line option reference for ARC tools. At the time of writing the Python tools represented there.
* ``archery`` - for the documents that not related to *Computing Element* another subdirectory should be created. At the time of writing only ARCHERY documentation of that kind is available.
* ``developers`` - directory used to store :ref:`developers_docs`. As long as we don't have many of this kind of documents everything that contains implementation details (how stuff coded, which variable used, etc) should go there.
* ``repos`` - this directory holds repository configuration instructions that needed for users as well as admins and developers. This documents are hidden in the main TOC tree, instead they are linked from other various documents when needed. Main index page have repos pointers as well as :doc:`Try ARC6 </admins/try_arc6>` and :doc:`Install Guide </admins/arc6_install_guide>`
* ``sdk`` - place for Doxygen SDK documentation integration to Sphinx build [#f1]_
* ``tech`` - directory used to store :ref:`tech_docs`. Now it holds only index document that contains references to the available PDFs that is verified to be applicable to ARC6 release. To add more, use template inside the index file.
* ``users`` - directory used to store :ref:`users_docs`. If document is user-oriented - place it here.
* ``wip`` - "work in progress" area hidden in the TOC tree. You can use it for incomplete documents that should not be publicly advertised yet, but still will be built and available via ``/wip/`` URL in produced HTML files.
.. [#f1] Do not touch unless you are modifying the Sphinx build itself. Nothing there affects documentation writting process.
Index files
Each directory in the source tree contains ``index.rst`` file that is used to link other documents in the same TOC tree. Upper-level ``index.rst`` file contains references to ``index.rst`` files in the subdirectories, that in turns contains pointers to the other documents.
.. note::
When you add a new document, add a reference to the ``index.rst`` in the same directory
Just follow the ``index.rst`` chain (look for ``toctree`` keyword) starting from the ``source`` directory to get familiar with TOC linking structure.
Storing images
When you need to add images to your document you should upload the image file itself and refer to it from the ``.rst`` document using ``image::`` or ``figure::`` keywords.
To be consistent image files are stored inside the ``images`` subdirectory in the document location. Then referencing is done by relative path, e.g:
.. code-block:: rst
.. figure:: images/shared_sessiondir_yes.svg
Building the docs
The top directory in the source tree contains ```` script that:
* checkout the ARC source code tree and configures it with packaging-like paths
* builds Doxygen SDK documentation
* converts ``arc.conf.reference`` to rST
* copies documentation parts from ARC source code tree (for :ref:`developers <developers_docs>` section)
* prepare automatically generated documentation for CLI commands
* builds HTML
* builds PDF (with LaTeX)
* builds ePub
On commit the GitLab CI configured to automatically invoke the ```` to produce rendered documentation. Documentation archive is avaiable as CI job artifacts and for ``master`` branch is deployed to `GitLab Pages <>`_.
To build the docs on your local machine you should have at least Sphinx installed. Additionally you should be able to configure ARC source tree and ARC Python command dependencies for auto-generated parts.
Complete and up-to-date list of dependencies defined for CI build and can be found in ``.gitlab-ci.yml``.
PDF builds with LaTeX is the most heavy part from the both time and needed additional packages perspective. For HTML rendering local debugging it is recommended to use the ``html`` script argument the skips PDF and ePUB:
.. code-block:: console
[user@localhost doc]$ ./ html
Point your browser to ``file:///path/to/doc/build/html/index.html`` to view the HTML rendering locally.
Writing Documentation in reStructuredText
General Syntax
Code snippets
Adding notes
Adding images
This repository is intended to be the initial workplace for ARC6 documentation.
Some (major) reorganization may take place as ARC6 release date is approached.
Contribution is possibly by direct push, no need for merge requests!
Recent version of built source tree is avaiable here:
......@@ -19,17 +19,20 @@ fi
make -C src/doxygen SDK PYTHON=/usr/bin/python
# copy README
cp README.rst source/
export PYTHONPATH="$CODE_DIR/src/utils/python"
# generate reference.rst
$CODE_DIR/src/utils/python/arcconfig-reference --convert-to-rst -r $CODE_DIR/src/doc/arc.conf.reference > "${DOCROOT}/source/user/reference.rst"
$CODE_DIR/src/utils/python/arcconfig-reference --convert-to-rst -r $CODE_DIR/src/doc/arc.conf.reference > "${DOCROOT}/source/admins/reference.rst"
# copy in-code developers documentation
cp $CODE_DIR/src/services/a-rex/lrms/README.lrms.rst ${DOCROOT}/source/developer/lrms.rst
cp $CODE_DIR/src/utils/python/README.configparser.rst ${DOCROOT}/source/developer/configparser.rst
cp $CODE_DIR/src/services/a-rex/lrms/README.lrms.rst ${DOCROOT}/source/developers/lrms.rst
cp $CODE_DIR/src/utils/python/README.configparser.rst ${DOCROOT}/source/developers/configparser.rst
# substitute code dir for commands
for cmdf in ${DOCROOT}/source/commands/*.rst; do
for cmdf in ${DOCROOT}/source/admins/commands/*.rst; do
sed "s#__CODE_DIR__#${CODE_DIR}#g" -i "$cmdf"
......@@ -50,5 +53,5 @@ rm -rf build/html/sdk
mv $CODE_DIR/src/doxygen/SDK/html build/html/sdk
# undo in-place changes of files
git checkout -- ${DOCROOT}/source/commands
git checkout -- ${DOCROOT}/source/admins/commands
.. NorduGrid ARC documentation master file, created by
sphinx-quickstart on Fri Jul 20 15:14:52 2018.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to NorduGrid ARC's documentation!
.. toctree::
:maxdepth: 2
:caption: Contents:
.. Indices and tables
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Documentation for ARC administrators
.. toctree::
:maxdepth: 4
.. # Dynamic content
# ./arcconfig-reference --convert-to-rst -r ../../doc/arc.conf.reference > reference.rst
......@@ -26,3 +26,4 @@ div.dnldpdf {
div.dnldpdf a {
color: #888;
.. _arc6_install_guide:
ARC6 Installation Guide
ARC Computing Element Installation and Configuration Guide
......@@ -112,7 +113,7 @@ Each authgroup represents a set of users, whose identities are matched to config
Authorization as well as mapping are then configured based on authgroup membership.
Once defined, authgroups can be applied to filter access to the CE per interface (:ref:`[arex/ws/emies] <reference_arex_ws_emies>`, :ref:`[gridftpd/jobs] <reference_gridftpd_jobs>`) and per queue. The ``allowaccess`` option in the corresponding interface block defines the allowed authgroups.
Once defined, authgroups can be applied to filter access to the CE per interface (:ref:`[arex/ws/jobs] <reference_arex_ws_jobs>`, :ref:`[gridftpd/jobs] <reference_gridftpd_jobs>`) and per queue. The ``allowaccess`` option in the corresponding interface block defines the allowed authgroups.
In the shipped configuration the empty ``[authgroup: zero]`` is defined and applied to A-REX WS interface, the effect of which is to deny any access by default.
......@@ -340,7 +341,7 @@ For example, to enable the system-defined ``ENV/PROXY`` RTE, run:
[root ~]# arcctl rte enable ENV/PROXY
More details on operating RunTime Environments can be found in :doc:`rtes`.
More details on operating RunTime Environments can be found in :doc:`/admins/details/rtes`.
Information system
ARCHERY Documentation
.. toctree::
:maxdepth: 2
In-deps ARC CE Configuration and Tuning Guides
.. toctree::
:maxdepth: 2
......@@ -129,7 +129,7 @@ In ARC6 release operating RunTime Environments is changed significantly and rely
userrte [label="User-defined RTEs"];
userrte2 [label="User-defined RTEs",style="dashed",color="grey",fontcolor="grey"];
arcctl [label="arcctl", color="red", shape="oval", fontsize=16, fontcolor=red, href="../commands/arcctl.html" target="_blank"];
arcctl [label="arcctl", color="red", shape="oval", fontsize=16, fontcolor=red, href="../admins/commands/arcctl.html" target="_blank"];
systemrte -> arcctl;
userrte -> arcctl;
......@@ -138,9 +138,9 @@ In ARC6 release operating RunTime Environments is changed significantly and rely
subgraph clusterControlDir {
label="Control Directory"; fontsize=16;
node [shape=folder];
enabled [label="Enabled RTEs", href="../user/rtes.html#enabling-rtes", target="_top"];
default [label="Default RTEs", href="../user/rtes.html#default-rtes", target="_top"];
params [label="RTE parameters", href="../user/rtes.html#rte-parameters", target="_top", group=galign1];
enabled [label="Enabled RTEs", href="../admins/detals/rtes.html#enabling-rtes", target="_top"];
default [label="Default RTEs", href="../admins/details/rtes.html#default-rtes", target="_top"];
params [label="RTE parameters", href="../admins/details/rtes.html#rte-parameters", target="_top", group=galign1];
arcctl -> enabled;
......@@ -284,7 +284,7 @@ The first tag describe RTE origin (``system``, ``user`` or :ref:`dummy<dummy_rte
The special ``masked`` keyword indicates that RTE name used more that once and *by-name* operations will apply to other RTE script. In example ``ENV/PROXY`` will be enabled from user-defined location and system-defined will be masked. However it is possible to enable ``masked`` RTE :ref:`by path<rte_enable_by_path>`.
Listing the particular kind of RTEs (e.g. :ref:`enabled<enabling_rtes>`) is possible with appropriate argument (see :doc:`/commands/arcctl` for all available options):
Listing the particular kind of RTEs (e.g. :ref:`enabled<enabling_rtes>`) is possible with appropriate argument (see :doc:`/admins/commands/arcctl` for all available options):
.. code-block:: console
ARC Computing Element Deployment
.. toctree::
:maxdepth: 2
.. Dynamic content: arcconfig-reference --convert-to-rst -r ../../doc/arc.conf.reference > reference.rst
Deploying Other ARC Components
.. toctree::
:maxdepth: 2
......@@ -12,7 +12,7 @@ ARC6 comes with so-called *zero configuration* included and works out of the box
Step 1. Enable NorduGrid ARC6 repos
Latest alpha release of ARC6 is available from :doc:`NorduGrid Repositories <repository>`.
Latest alpha release of ARC6 is available from :doc:`NorduGrid Repositories </repos/repository>`.
.. note::
Aplha packages are in *testing* repository, so please make sure it is enabled, e.g. on RHEL-based systems you can use ``yum --enablerepo=nordugrid-testing`` to enable it for one transaction or ``yum-config-manager --enable nordugrid-testing`` to enable permanently.
......@@ -97,7 +97,9 @@ html_theme = 'alabaster'
# documentation.
html_theme_options = {
'logo': 'ARClogo.png'
'logo': 'ARClogo.png',
'sidebar_includehidden': False,
'sidebar_collapse': True
# Add any paths that contain custom static files (such as style sheets) here,
......@@ -120,6 +122,7 @@ html_sidebars = {
# Customizations
#def setup(app):
# app.add_stylesheet('css/svgwidth.css')
Documentation for developers
Implemantation Details for Developers
.. toctree::
:maxdepth: 4
:maxdepth: 2
Welcome to NorduGrid ARC's documentation!
The *Advanced Resource Connector (ARC)* middleware, developed by the
`NorduGrid Collaboration <>`_, is an open source software solution
enabling e-Science computing infrastructures with emphasis on processing of large data volumes.
ARC is being used to enable national and international e-infrastructures since its first release in 2002.
ARC is available for variety of GNU/Linux flavors via stable :doc:`Repositores <repos/repository>` or :doc:`Nigtly Builds <repos/nightlies-repo>` if you want to test the latest development release.
If you are new to ARC get started with the :doc:`Try ARC6 <admins/try_arc6>` quickstart guide to get an overview of main operations in the simple test case.
For production *Computing Element* deployment follow the :doc:`Installation and Configuration Guide <admins/arc6_install_guide>` that contains the structure and pointers to precise configuration of every ARC subsystem.
The ultimate description of each configuration option can be found in the :doc:`admins/reference`.
.. _users_docs:
Documentation for Infrastructure Users
This part of the documentation targeted to distributed computing infrastructure users that use either clients or SDK to run jobs and handle data transfers.
.. toctree::
:maxdepth: 2
.. _admins_docs:
Documentation for Infrastructure Admins and Operators
This section contains a documentation about all ARC middleware services deployment, configuration and operations.
If you are looking for ARC Computing Element setup instruction or performance tuning parameters you are in the right place.
.. toctree::
:maxdepth: 2
.. _tech_docs:
Technical Documents Describing ARC Components
Following documents gives a deep technical description of the various ARC components. If you are looking
for architecture internals (how parts of ARC was designed) you can follows this section.
.. toctree::
:maxdepth: 2
.. _developers_docs:
Documentation for Developers
If you are looking for development internal details of ARC (like how some stuff was coded) this part of
documentation is for you. Mainly for those who want to contribute to the project development,
advanced troubleshooters or just interested.
.. toctree::
:maxdepth: 2
.. Not linked directly to the TOC but still built and available via linking or if you know what you are looking for
.. toctree::
......@@ -123,7 +123,7 @@ Once the NorduGrid repositories are configured, install the packages with:
Or with ``dnf`` for Fedora.
Please refer to the :doc:`arc6_install_guide` for package selection.
Please refer to the :doc:`/admins/arc6_install_guide` for package selection.
Note that the NorduGrid repositories for RedHat Enterprise Linux/CentOS depends on the `EPEL <>`_ repositories which must also be part of the YUM configuration.
......@@ -233,5 +233,5 @@ Install the packages with:
[root~]# apt-get install <list of package names>
Please refer to the :doc:`arc6_install_guide` for package selection.
Please refer to the :doc:`/admins/arc6_install_guide` for package selection.
Index of ARC Technical Documents
.. |clearfloat| raw:: html
<div class="clearer"></div>
.. note::
Technical documents `exists <>`_ for ARC5 only. Those that are verified to be relevant for ARC6 will be listed below
Hosting Environment of the Advanced Resource Connector middleware
.. image:: images/doc-manuals.png
:align: left
Document gives a deep technical description of the HED service container.
A Client Library for ARC
.. image:: images/doc-manuals.png
:align: left
Document describes from a technical viewpoint the plugin-based client library of ARC.
ARC Client Installation Instructions
.. note::
There are no client installation instructions available in the new documentation format yet. Please use the
`old installation instructions <>`_ for main points
and hints.
CLI and SDK Documentation
.. toctree::
:maxdepth: 2
DTR (Data Transfer Reloaded)
This page describes the data staging framework for ARC, code-named
DTR (Data Transfer Reloaded).
Work-in-progress Docs
Hidden area that holds unfinished documents to be build and available in the doc tree, but not yet linked to the right place.
.. toctree::
:maxdepth: 2
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment