Operations - General

This page presents the operations common to all stacks of the Datachain solution: deployment scripts, starting and stopping, start order, updates, certificates, and logging and monitoring conventions. The operations specific to each stack are described in the dedicated pages (see Operations).

Directory tree and deployment scripts

When Datachain Core is deployed using the deployment methods provided by Adobis, we provide a set of shell scripts to manage the deployed services, regardless of the granularity of the deployed stacks.

PLEASE NOTE

The scripts provided are meant to facilitate the deployment of the solution. You are free to adapt them, and they are not subject to any license.

Conversely, these scripts are not covered by any warranty.

In the rest of this chapter, we use the standard paths provided by Adobis for the product.

The following directory tree is deployed on all cluster nodes:

.
└── opt
    ├── adobis
    ├── products  // contains the scripts and configuration files of the different stacks
    │   ├── stack1
    │   └── stack2  // same overall directory tree
    ├── data  // contains the data of the different modules
    │   └── stack1  // application data directory
    │       └── xxx  // data directory of a service of the application
    └── logs  // contains the log files

The following scripts are deployed on the cluster master node:

.
└── opt
    ├── adobis
    └── products  // contains the scripts and configuration files of the different stacks
        └── stack1
            ├── shared  // scripts and elements used by the start, stop, ... scripts
            ├── previous_stacks_version  // history of certain elements.
            ├── start.sh  // stack start script
            ├── stop.sh  // stack stop script
            ├── restart.sh  // script that stops then restarts the stack
            ├── runStack.sh  // script used by the start, stop and restart scripts
            ├── *.env  // configuration of a specific service (example: backend.env for the configuration of the Datachain backend)
            ├── .env.harbor  // keys of the Adobis Harbor registry
            ├── .env.platform  // global settings (root URL), used in the other env files or in the docker-compose file.
            ├── .env.global  // specifies the stack name, as well as the networks this stack 'owns'. For the latter, the network is created by the start or restart script when the stack is launched, if it does not exist yet.
            ├── .env.local  // potentially contains the Traefik URL, used in the docker-compose file.
            └── docker-compose-${stack-name} // Deployment descriptor of the ${stack-name} stack

The .env.* files shown above are not mandatory. They can be merged into a single one, keeping the name of one of the files (i.e. all parameters can be grouped in a single '.env.local' file)

Below is a table of the variables usually used:

Parameter	Description
HARBOR_REGISTRY	Name of the registry from which to pull the docker images
HARBOR_USER	Registry user name, if authentication is required
HARBOR_KEY	Registry user key, if authentication is required
DOCKER_NETWORKS	Network owned by the stack(s). The network is created automatically if it does not exist. (From a docker swarm point of view, we create an encrypted, non-attachable overlay network.)
STACKS_NAMES	Name of the stack(s), separated by spaces. If STACKS_NAMES=dc, the deployment directory must contain a docker-compose-dc.yml file. There must be as many docker-compose-xxx.yml files as there are stacks.
DOCKER_NETWORKS	Network owned by the stack(s). The network is created automatically if it does not exist. (From a docker swarm point of view, we create an encrypted, non-attachable overlay network.)
ADDITIONALS_COMPOSE_FILE_NAMES	When there is a single stack but several docker compose files, ending of the file name to use to add the additional docker-compose files. For example, if the description of the Redis service has been externalized in a docker-compose-redis.yml file, you can declare: ADDITIONALS_COMPOSE_FILE_NAMES=redis The docker-compose-STACK_NAME and docker-compose-redis.yml files will be used when the stack is started.
*_URL	Externalization of the URLs for the Traefik reverse proxy, used by the docker-compose-xxx.yml file(s)

Parameter

Description

HARBOR_REGISTRY

Name of the registry from which to pull the docker images

HARBOR_USER

Registry user name, if authentication is required

HARBOR_KEY

Registry user key, if authentication is required

DOCKER_NETWORKS

Network owned by the stack(s). The network is created automatically if it does not exist. (From a docker swarm point of view, we create an encrypted, non-attachable overlay network.)

STACKS_NAMES

Name of the stack(s), separated by spaces. If STACKS_NAMES=dc, the deployment directory must contain a docker-compose-dc.yml file. There must be as many docker-compose-xxx.yml files as there are stacks.

DOCKER_NETWORKS

Network owned by the stack(s). The network is created automatically if it does not exist. (From a docker swarm point of view, we create an encrypted, non-attachable overlay network.)

ADDITIONALS_COMPOSE_FILE_NAMES

When there is a single stack but several docker compose files, ending of the file name to use to add the additional docker-compose files. For example, if the description of the Redis service has been externalized in a docker-compose-redis.yml file, you can declare:

ADDITIONALS_COMPOSE_FILE_NAMES=redis

The docker-compose-STACK_NAME and docker-compose-redis.yml files will be used when the stack is started.

*_URL

Externalization of the URLs for the Traefik reverse proxy, used by the docker-compose-xxx.yml file(s)

Starting and stopping a stack

To start the stack, run the following shell command:

bash start.sh

To stop the stack, run the following shell command:

bash stop.sh

To restart the stack, run the following command:

bash restart.sh

Changing version

When changing the stack version, or on first launch, you can specify the version on the command line using the '-v' option followed by the version number to deploy.

bash start.sh -v 8.6.0

bash restart.sh -v 8.6.0

Start order of the solution components

Some services require others to be already running in order to be operational. Below is a suggested start order for the different stacks. Depending on your own architecture and deployment, some stacks may be grouped together. The ones presented below follow a standard split.

Traefik: as a reverse proxy, no application is accessible without this stack being deployed. The absence of the Traefik stack does not cause any application malfunction. The other applications are simply not accessible from a browser.
Keycloak: when the Keycloak stack is deployed but not started, some modules of the Datachain stack cannot start, because they use security information coming from this stack.
Spark cluster: when a Spark cluster is deployed, it must be operational before the Datachain stack is launched, and the Datachain stack must be restarted if the cluster is restarted. Otherwise, the Datachain Spark contexts will not be operational.
Datachain: last module to start.

Components are stopped in the reverse order of the start order.

The operations and files specific to each stack are described in the corresponding Operations pages.

Version update

Updating a stack consists in redeploying its services with the new version of the images, using the -v option of the scripts (see Changing version). For all stacks:

Read the technical release notes of each intermediate version: they describe the configuration, image and port changes, and the exceptional migration procedures.
Back up the stack data before stopping it (see the operations page of the stack concerned, and Data management for the databases).
Stop the stack, then restart it with the new version. The migration services (pg_migration for Datachain, db-migration and airflow-migration for Dc-Maestro) run once and stop: check in their logs that the migrations went well before using the application.
Follow the release notes' instructions regarding third-party components (Keycloak, PostgreSQL): version to keep, or recommended migration.

Application certificates

The Datachain Java images (Spark, Console, Marketplace) make it possible to add trusted authorities or server certificates, for example to access connectors or repositories over HTTPS. The certificates to add are placed in the /java_cert_import volume of the container: they are imported into the Java keystore when the container starts.

The .cert extension is mandatory, one certificate per file, PEM or DER encoding.
After adding a new certificate, restart the service concerned.
The detailed format and behavior are described in Application certificates.

A manual alternative, which consists in providing a complete keystore to the JVM, is presented in the FAQ.

Logs

The Datachain Java images (backend, Spark, Console, Marketplace) write their logs to files under /logs/datachain/<module>, with daily and size-based rotation, compression and a 30-day retention. Mounting a volume on /logs is recommended to keep them. See the configuration page of each module.
The other images (Web, migration, Keycloak, PostgreSQL, Valkey) write to the container’s standard output, which can be read with docker service logs (Swarm) or docker logs.
Dc-Maestro writes to the standard output and to a rotating file (LOG_FILENAME). Airflow writes the task execution logs under /opt/airflow/logs.

Monitoring

The Datachain modules expose their metrics in Prometheus format: natively (Spring Actuator) for the backend, Spark, the Console and the Marketplace, and through external components for the web server (nginx-prometheus-exporter) and PostgreSQL (postgres-exporter). The setup is described in the operations pages of each stack.

Dernière mise à jour le 2 novembre 2026 à 15:16