ERS - Error reporting system

DAQling uses the ERS - error reporting system for logging and error reporting/handling.

For more information about the ERS API, refer to the atlas-tdaq-software/ers.

For information on using the more complex features of the API, like creating custom streams, refer to the docs.

Logging

ERS provides 6 default ERS streams: debug, log, info, warning, error and fatal.

The first four can be written to by macros of the form:

ERS_DEBUG(0,"Text to send to the debug stream.");
ERS_INFO("Text to send to" <<" the info stream.");
...

The debug stream requires an extra integer value indicating the debug level.

The debug level can be set in the environment, and only messages with a debug level lower than or equal to the set debug level will be sent to the debug stream. The default debug level is 0.

Note: debug level messages are compiled out when building with the CMake "Release" flag, through the ERS_NO_DEBUG complile definition.

For all 6 streams "ERS Issues" can be sent to the stream. Issues will be explained in the next section.

Example:

ers::fatal(IssueInstance);

where the IssueInstance is an object derived from the ers::Issue base class.

Logging formats

Below an example of a log entry from DAQling is shown:

[2021-06-16 11:05:24,327] [readoutinterface01] [core] [WARNING] [Statistics.cpp:64] [daqling::core::Statistics::Statistics(nlohmann::json&)] m_name = readoutinterface01

On a more generic form a log entry looks like this:

[timestamp] [module-name] [package] [level] [file:line] [function name] "message"

Explanation of fields:

  • timestamp - timestamp of when the logging operation occured.
  • module-name - name of the module producing the logging message. Can also be 'unknown' or 'core' if the source is unknown or the message is a result of a general DAQling core operation.
  • package - tag linked to the package/library producing the logging message. Can be 'module', 'core' or 'connection'. Different log levels can be set for these packages.
  • level - severity level of the message. One of the 6 ERS streams.
  • file & line - the file and line number where the message was produced / issue was thrown.
  • function name - name of the function, including namespaces, where the message originates from. This field can be disabled by adding add_compile_definitions(DONT_LOG_FUNCTION_NAME) to the CMakeLists.txt file in the top directory.
  • message - logging message.

Thread tagging

ERS doesn't provide any mechanism for tagging the individual modules in a DAQling component with their names. However, it does provide the thread-id as part of the context of an issue.

This is utilized by creating a thread tagger, mapping thread-ids to module names. This mapping is automatically done for the runner thread of the modules. However if the runner thread spawns child threads, this must be done manually by the module developer.

To do this, simply call the addTag() method of the DAQProcess class, within the new thread context.

A simple way to do this would be:

thread_runner_function(){
  addTag();
  while(m_run){
    ...
  }
}

Log levels

In DAQling it is possible to set the minimum logging levels in the configurations.

All logging messages with a lower severity level than the set level, will be discarded.

The logging levels can be set for the three different package tags used in DAQling: core, module and connection.

Below an example of setting the logging level for a component in a configuration file can be seen:

"loglevel": {"core": "INFO", "module": "INFO", "connection": "WARNING"}

Issues

ERS provides an object type called Issue.

Issues are a combined exception and message object, supporting chained issues. This allows for chaining issues and sending them to one of the streams, providing very detailed error information. This functionality is very powerful, since it allows higher level objects to gather detailed information from lower level objects, and react on this by recovering when possible, or terminating the program otherwise.

Information on how to create custom Issues can be found here.

Below is an example of usage of Issues in DAQling:

From ConnectionManager.hpp:

namespace daqling {
ERS_DECLARE_ISSUE(core, ConnectionIssue, "", ERS_EMPTY)
ERS_DECLARE_ISSUE_BASE(core, CannotAddChannel, core::ConnectionIssue, "Failed to add channel!",
                    ERS_EMPTY, ERS_EMPTY)
ERS_DECLARE_ISSUE_BASE(core, CannotAddStatsChannel, core::ConnectionIssue,
                    "Failed to add stats channel! ZMQ returned: " << eWhat, ERS_EMPTY,
                    ((const char *)eWhat))
ERS_DECLARE_ISSUE_BASE(core, CannotGetChidAndType, core::ConnectionIssue,
                    "Failed to get chid and connection type " << eWhat, ERS_EMPTY,
                    ((const char *)eWhat))
namespace core {

Notice how the Issues are declared in the daqling namespace, and inside the declaration assign core as the namespace.

This makes it easy to access from the daqling::core namespace as seen below:

From ConnectionManager.cpp:

  try {
    m_stats_context = std::make_unique<zmq::context_t>(ioT);
    m_stats_socket = std::make_unique<zmq::socket_t>(*(m_stats_context.get()), ZMQ_PUB);
    m_stats_socket->connect(connStr);
    ERS_INFO(" Statistics are published on: " << connStr);
  } catch (std::exception &e) {
    throw CannotAddStatsChannel(ERS_HERE, e.what());
    return false;
  }

Here the ConnectionManager catches an exception and wraps it in an issue, from which the owner of the ConnectionManager can deduce where and why an error occured, or simply chain and pass on the Issue as seen below:

From DAQProcess.hpp:

try {
    if (!m_connections.setupStatsConnection(1, statsURI)) {
        return false;
    }
} catch (ers::Issue &i) {
    throw core::CannotSetupStatPublishing(ERS_HERE, i);
}

Note that ERS_HERE is required for all Issues, and captures the program context and passes this to the created issue. All other parameters are specific to the type of Issue being created.