Next generation ARC accounting with A-REX Accounting Records SQLite Database (Fixes BUGZ-3883)

New ARC accounting subsystem is aimed to improve salability, eliminate bottlenecks of the current legacy architecture and provide much more information about the ARC CE jobs on site.

The new accounting subsystem:

• store everything in SQLite database optimized for common local queries
• records are put into accounting database by A-REX from the beginning
• arcctl will provide admin-friendly interface to access all accounting data locally
• new jura tool will get information directly from A-REX accounting database to publish records to SGAS/APEL
• republishing will recreate XMLs from the accounting database, eliminating the need of accounting archive

There are several major development steps:

• Database schema design
• SQLite queries performance testing and optimization on huge legacy accounting archive databases obtained after ARC 6.0.0 rollout
• Develop A-REX accounting database subsystem to write data
• Develop Python modules to interact with accounting database
• Develop arcctl interfaces for local stats queries
• Develop Python modules for SGAS UR XML generation
• Develop Python modules for APEL CAR/Summary and Sync messages generation
• Develop Python modules for APEL SSM integration (send to APEL)
• Develop Python modules for SGAS records POSTing
• Develop arcctl interface for republishing using SGAS/APEL modules
• Develop new jura-ng tool for regular publishing of data SGAS and APEL services
• Wrap it up, keeping both legacy and new accounting subsystem to allow transition
• Add missing accounting metrics measurements
• Rollout and test publishing to APEL
• Rollout and test publishing to SGAS