Preparing docker swarm
Installation requirements
Hardware configuration
Configuration must be defined according to the intended use of the of the software package. If the processing pipelines processing pipelines, you will need to increase the number of CPUs accordingly. If data volumes are high, it will be necessary to memory and disk space will need to be increased accordingly.
| Memory | CPU | Disk | *Minimum |
|---|---|---|---|
Minimal |
8GB |
4 Cores |
50GB |
Standard 16GB 8 to 16 cores 200GB |
Intensive |
32Go |
16 to 32 cores |
For multi-machine implementations, the machines must be must be connected via a GigaBit Ethernet network, to guarantee performance.
Machine configuration
-
Machines can be physical, virtual or in the cloud.
-
Machines must communicate with each other over the network, using fixed IP addressing.
-
The IP address must be assigned to a network interface available to the host operating system.
-
Open the following ports between machines (for Docker):
-
TCP port 2377 for cluster management communications management)
-
TCP and UDP port 7946 for inter-node communications
-
UDP port 4789 for docker overlay network traffic
-
-
Allow IP protocol 50 (ESP) for overlay network encryption
System and software requirements
The operating system used in this document is Ubuntu 22 LTS. The application is validated on Debian/ubuntu environments Debian/ubuntu environments, and is functional on any recent Linux system, with the prerequisites installed.
As a matter of good Linux practice, we recommend distinguishing between system data volumes, docker (the /var/lib/docker directory can become very become very large), and datachain directories.
| Requirements | Version | Comments and Installation |
|---|---|---|
*Docker |
Docker |
20 minimum (current version 23.0.0) |
Cluster Swarm
This section briefly describes how to set up a multi-node Swarm cluster.
Note that this procedure is also valid for a single-node of a single-node cluster.
Cluster creation
Cluster initialization
This command initiates the cluster and creates the first manager node. manager.
To be executed once only on the host manager.
$ docker swarm init [--advertise-addr <MANAGER-IP>]Example:
$ docker swarm init --advertise-addr 192.168.99.100
Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a
manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
192.168.99.100:2377
To add a manager to this swarm, run 'docker swarm join-token manager'
and follow the instructions.You can test the correct instantiation of the cluster:
$ docker info
Containers: 2
Running: 0
Paused: 0
Stopped: 2
...snip...
Swarm: active
NodeID: dxn1zf6l61qsb1josjja83ngz
Is Manager: *true*
Managers: 1
Nodes: 1
...snip...This shows that Swarm mode has been activated on the host manager and the cluster is ready for use.
For single node mode, the procedure ends here. To proceed to add additional nodes, see next section.
Add nodes to the Swarm cluster
Run on each machine you wish to add as a node.
$ docker swarm join --token <MANAGER-TOKEN> <MANAGER-IP>:2377Example:
$ docker swarm join
--token
SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
192.168.99.100:2377
This node joined a swarm as a worker.To retrieve the token again from the manager:
docker swarm join-token workerYou can test the node addition by running the following command on the manager - several nodes are now referenced:
$ docker node lsDatachain installation
This chapter introduces you to the DataChain © solution images and their settings.