Monitoring

Metrics registration

Each Module has its own instance of "Statistics" class (m_statistics) which provides the interface for metrics monitoring. While defining your custom Module you can register the metric using the template function:

void registerMetric(T* pointer, std::string name, metrics::metric_type mtype)

in the configure() function of your Module, where:

  • pointer - pointer to the variable storing the metric value
  • name - name of the metric published to database
  • mtype - type of the metric

For example:

m_statistics->registerMetric<std::atomic<int> >(&m_metric6, "RandomMetric1-int", daqling::core::metrics::LAST_VALUE);

The acceptable types of variables are:

  • std::atomic<int>
  • std::atomic<float>
  • std::atomic<double>
  • std::atomic<bool>
  • std::atomic<size_t>

There are 4 possible metrics types:

  • LAST_VALUE - measure the current value of metrics
  • ACCUMULATE - accumulate the current variable value to metric value and reset the variable
  • AVERAGE - calculate the average value of metric over given time interval
  • RATE - calculate the rate over given time interval

All of them are defined in the daqling::core::metrics namespace.

Metrics configuration

The metrics values can be send directly to influxDB or published via ZMQ socket. The metrics destination can be set in the JSON configuration of the Module by specifying the following settings:

  • stats_uri - URI of ZMQ publisher socket
  • influxDb_uri - URI used for sending the values directly to influxDB
  • influxDb_name - name of the influxDB database

Either stats_uri or influxDb_uri must be specified in order to allow for the metrics registration in the module. Example settings:

"settings": {
  "stats_uri": "tcp://*:6007",
  "influxDb_uri": "http://localhost:8086/write?db=",
  "influxDb_name": "DAQlingMetrics"
}

In this configuration the values will be sent to the influxDB database as well as published via ZMQ publisher.

It is also possible to configure the interval between stats posting. This is done by adding the field: * stats_interval - interval between posting in ms. Default is 500.

Metrics visualization

The default metrics visualization tool is Grafana (https://grafana.com/) connected to influxDB (https://www.influxdata.com). Both Grafana and influxDB can be installed and configured automatically (see the installation instructions in README.md file). In this case the influxDB database DAQlingMetrics is configured as well as default Grafana dashboard called DAQling demo. The Grafana dashboard can be displayed under the following link: <host>:3000

  • grafana default login: admin
  • default password: admin

ZMQ subscriber

When "stats_uri" option in the JSON Module settings is specified, the metrics values are published via ZMQ socket. Then it is possible to create a custom ZMQ subscriber. The example subscriber can be found here:

scripts/Monitoring/metric-manager.py

with a JSON configuration file: metrics-config.json.

This subscriber gives the possibility to send the metrics values to Redis or/and InfluxDB. It is also possible to set the list of subscribed metrics as well as the individual destination of defined metric.

Metrics configuration:

"metrics": [
  "<metric_name>,<destination>",
  ...
]

Possible destinations:

  • i - InfluxDB
  • r - Redis
  • b - both (InfluxDB and Redis)

If the <destination> parameter is not provided or not valid (e.g. w), the i option is set as default.

Example settings:

"metrics": [
  "metricssimulator-RandomMetric1-int,i",
  "metricssimulator-RandomMetric2-float,w", 
  "metricssimulator-RandomMetric3-double,b",
  "metricssimulator-RandomMetric8-double_average,r"
]