Distributed Systems

DAQling supports natively the distribution of components over multiple host computers.

The DAQling demo is "single-host", in particular "build", "control" and "runner" are hosted in the same compute node. In a distributed system, the "runner" can be hosted on several compute nodes, other than the "build and control" one.

Configuration

The distribution of components over "runner" nodes requires some small modifications to the configuration.

{
  "name": "example01"
  ...
  "modules":[
    {
      ...
      "name":"example01",
      "host": "host01", <---- executes on host01
      "port": 5555,
      "connections": {
        "senders": [
          {
            "type": "ZMQPair",
            "chid": 0,
            "transport": "tcp",
            "host": "*",  <---- binds to host01 port 8101
            "port": 8101
          }
        ]
      }
    }
  ]
},
{
  "name": "example02"
  "host": "host02", <---- executes on host02
  "port": 5556,
  "modules":[
    {
      ...
      "connections": {
        "receivers": [
          {
            "type": "ZMQPair",
            "chid": 0,
            "transport": "tcp",
            "host": "host01", <---- connects to host01 port 8101
            "port": 8101
          }
        ]
      }
    }
  ]
}

Setup of "runner" nodes

The following describes how to set up "runner" nodes working on the same AlmaLinux or RHEL 8,9 operating system as the "build and control" node.

Note that is possible to control "runner" nodes working with a different OS with respect to the "build and control" node, but some of the next steps should be adapted and carried out manually.

Run-time dependencies

Run-time dependencies can be installed on the "runner" hosts with:

dnf install -y boost-devel xmlrpc-c-devel cppzmq-devel

Note that these are a subset of the "Build dependencies", as reported in the DAQling's README.md.

Supervisor

"Supervisor" and its extension "Twiddler" are needed to spawn new DAQling executables on the "runner" node.

In order to perform the setup with Ansible, first add the ~/.ssh/id_rsa.pub of the "build and control" node to the ~/.ssh/authorized_keys of the "runner" nodes, in order to have password-less login.

Modify the ansible/hosts file adding the "runner" nodes' hostnames under the [daq] header.

Now modify the ansible/set-up-host.yml, replacing the line hosts: localhost with hosts: daq. Now the set-up-host.yml playbook will execute the operation on all the hosts listed in the [daq] section of ansible/hosts.

Execute the playbook with:

ansible-playbook -i hosts set-up-host.yml -t remote --ask-become

DAQling binaries

The "build and control" DAQling build/ directory ($DAQ_BUILD_DIR) must be available on the "runner" nodes at the same absolute path as in the "build and control" node.

In order to obtain this, it's possible to simply copy the build/ folder to all the "runner" nodes after every rebuild of the source code.

Alternatively, the build/ folder can be mounted as a "Network File System" (NFS) in order to share the content over all "runner" nodes.

Firewall rules

In order to enable communication between the distributed nodes, it is necessary to open the ports used by DAQling.

In particular, taking the DAQling demo, one notices that ports 5555,5556, 8101, 8102 are used for control and data movement.

In order to open the relevant port ranges execute:

firewall-cmd --add-port=5555-5556/tcp --permanent
firewall-cmd --add-port=8101-8102/tcp --permanent
firewall-cmd --reload

Note that the ranges have to be changed according to the specific DAQling configuration used.