Datachain solution architecture

The purpose of this page is to present:

The technical architecture of the solution,
The different types of deployment that can be implemented.

The solution is made up of 5 modules, 2 of which are mandatory: Datachain-Core and Dc-Console.

The entire Datachain solution is delivered as docker containers, allowing deployment on platforms such as docker swarm or K8s.

All these modules share a common SSO. (In V9, Keycloak remains mandatory if the Maestro and Dc-Code modules are deployed. Otherwise, the solution can integrate with SSO solutions available on the market that support OAUTH authentication.)

Requests to the different modules can be routed by HTTP reverse proxy solutions (such as Traefik for docker or Kubernetes ingress).

Datachain Core

Datachain Core is made available as a docker container, enabling simple single-server deployments, right up to highly available multi-server architectures.

Solution application architecture

The architecture below shows the minimal version of a deployment, in standalone mode, i.e. without using any external resource.

The table below summarizes the internal flows between the different modules.

Source	Target	Protocol	Port	Function
traefik	web_ui	http	80	proxies requests to the web_ui

web_ui	backend	http	8089	proxies requests to the backend

backend	spark	http	8090	Data exchange with Spark
backend	redis	redis	6379	Asynchronous information exchange with Spark
backend	pg_all	sql	5432	Retrieval of application and exposure data
backend	keycloak	http	8080	Retrieval of authentication data

spark	redis	redis	6379	Asynchronous information exchange with the backend
spark	pg_all	sql	5432	Generation of exposure data
spark	keycloak	http	8080	Retrieval of authentication data (to be removed in a future version)

keycloak	pg_all	sql	5432	Retrieval of Keycloak data

pg_migration	pg_all	sql	5432	Schema and data migration

Source

Target

Protocol

Port

Function

traefik

web_ui

http

proxies requests to the web_ui

web_ui

backend

http

8089

proxies requests to the backend

backend

spark

http

8090

Data exchange with Spark

backend

redis

6379

Asynchronous information exchange with Spark

backend

pg_all

sql

5432

Retrieval of application and exposure data

backend

keycloak

http

8080

Retrieval of authentication data

spark

redis

6379

Asynchronous information exchange with the backend

spark

pg_all

sql

5432

Generation of exposure data

spark

keycloak

http

8080

Retrieval of authentication data (to be removed in a future version)

keycloak

pg_all

sql

5432

Retrieval of Keycloak data

pg_migration

pg_all

sql

5432

Schema and data migration

Focus on the Spark deployment

In the Datachain solution, the Spark service is the entry point for the computations performed by Datachain to run pipelines. This node therefore requires suitable CPU/memory power.

Different deployment scenarios can address the performance/availability requirements of the Spark node(s).

The Spark service can process Datachain pipelines in parallel. The mode used in this case is Spark’s FAIR mode. It can however remain in FIFO mode, which processes only one task at a time.

The following diagrams only show the deployment diagrams of the Spark contexts, in order to focus on them. The other elements of the solution must be deployed to obtain an operational platform.

Case 1: Single server, single service, with vertical scalability

The service is configured with enough CPU and memory to perform the processing. If additional resources are needed, the physical resources of the machine must be increased, and these new resources allocated to the existing service. The number of jobs processed in parallel can potentially be increased.

All projects use the same Spark service. Under heavy load, contention may occur. This solution reaches its limits when the memory requirements become too high for all the parallel processing. In that case, case 2 can be used.

Configuration in this case: from 1 cpu/1 GB RAM to 60 cpu/60 GB RAM.
If the physical machine is dedicated to the Spark service, make sure to leave 1 cpu/1 GB RAM to the host system.

Case 2: Single server, multiple services

Several Spark services are deployed on the same machine: they can share the same data directory (HDFS) in order to share data. This is useful when a project can use any of these contexts interchangeably.

Case 3: Multi-server, single/multiple services

One or more Spark services are deployed on several machines that do not share a disk. This deployment mode makes it possible to deploy contexts per BU (Business Unit) of an organization, each having one or more dedicated Spark contexts. Each BU thus has its own computing power: its jobs will be processed even if another BU uses its resources very intensively.

A second possibility is to share development and production on the same Datachain instance. Jobs of projects under development can be run on production contexts, while production jobs are scheduled through Maestro on the production context. A temporary overload of the development contexts will not affect the performance of the production contexts.

Case 4: Multi-server, single/multiple services deployed on a Spark cluster

In this scenario, a Spark/Hadoop cluster is deployed on a first set of machines. Then, one or more Spark services are deployed, configured to use the Spark/Hadoop cluster. Jobs are thus distributed across the different machines of the cluster.

Technical architecture

Hardware requirements

The configuration must be defined according to the intended use of the software package. Indeed, if the processing pipelines require a lot of computation, the number of CPUs must be adapted accordingly. If data volumes are large, memory and disk space will need to be adapted.

Configuration type	Memory	CPU	Disk
Minimal	8 GB	4 cores	50 GB
Standard	16 GB	8 to 16 cores	200 GB
Intensive	32 GB	16 to 32 cores	500 GB
Very intensive	64 GB	32 to 64 cores	1 TB
Extended	64 GB and more	64 cores and more	1 TB and more

Configuration type

Memory

CPU

Disk

Minimal

8 GB

4 cores

50 GB

Standard

16 GB

8 to 16 cores

200 GB

Intensive

32 GB

16 to 32 cores

500 GB

Very intensive

64 GB

32 to 64 cores

1 TB

Extended

64 GB and more

64 cores and more

1 TB and more

When deploying on several machines, they must be connected through a Gigabit Ethernet network, in order to guarantee optimal performance.

CPU core usage by the Spark component

When an execution request is sent to the Datachain application, it triggers one or more jobs on the Spark component, which then determines an execution plan to optimize processing, making the best use of the resources allocated to it. Submitted jobs are split into tasks, which can be executed in parallel. The processing unit of a task in Spark is the CPU core.

Thus, if a Spark context is allocated a maximum of 3 cores, it can run at most 3 tasks in parallel.

The number of cores allocated to the Spark context is therefore a way to improve application performance.

Dernière mise à jour le 2 novembre 2026 à 15:16