Operations - Datachain Core

This page describes the operations of the Datachain Core stack: startup and shutdown, logs, monitoring and backup. The module settings are described in Configuration - Datachain Core.

Startup and shutdown

The Datachain stack is started last, after Keycloak and, where applicable, the Spark cluster. The startup, shutdown and restart scripts are described in Operations - General.

At startup, the migration service (pg_migration) runs once and stops: the database patches must be applied before using the application. Check its logs after every update.

In the case of a Spark cluster, it must be operational before the Datachain stack is launched, and the Datachain stack must be restarted if the cluster is restarted.

Stack configuration files (in the stack directory):

File

Service concerned

web_ui.env

Web module: configuration

backend.env

Backend module: configuration

spark.env

Spark module: configuration

Access and service health checks

Module Port URL Usage

Web

8080

/

Datachain application

Web

9080

/stub_status

Nginx metrics

Backend

8089

/

Application API

Backend

9089

/actuator-backend/*

Spring Actuator

Spark

8090

/

Computation module API

Spark

9090

/management/*

Spring Actuator (default prefix, can be changed with management.endpoints.web.base-path)

Spark

4040

/

Spark driver interface: analysis of job execution

Logs

Module Location

Web

Container standard output (Nginx access logs)

Backend

/logs/datachain/backend: dc-backend.log (application), access.log, management.log

Spark

/logs/datachain/spark: dc-spark.log (application), access.log, management.log

Migration

Container standard output

Application logs are rotated daily (and every 200MB), compressed, and kept for 30 days, within a limit of 10GB. See Configuration - Datachain Core for details.

Monitoring

All metrics are made available in Prometheus format and can be integrated into most monitoring tools.

Module

Monitoring type

Web

External component nginx-prometheus-exporter

Backend

Internal, via Spring Actuator

Spark

Internal, via Spring Actuator

As with most applications, the elements to monitor are system resources and connection pools (to avoid reaching saturation).

Web module

Monitoring is done through the nginx-prometheus-exporter image, added to the deployment file. It reads the Nginx metrics exposed on port 9080 (/stub_status).

docker-compose.yml file
version: "3.7"
services:

  # Déclaration des autres services
  # Other service declaration

  nginx_exporter:
    image: nginx/nginx-prometheus-exporter:${NGINX_PROMETHEUS_EXPORTER_VERSION}
    networks:
      # Disponible sur le réseau traefik, car disponible via le proxy traefik
      traefik_network:
      # Disponible sur le réseau dc_network, car interroge le module web_ui, dont un alias sur ce réseau est 'dc-web-ui'
      dc_network:
    deploy:
      mode: replicated
      replicas: 1
      labels:
        # Exposition via traefik
        - "traefik.enable=true"
        - "traefik.docker.network=traefik_network"
        # disponible sur l'URL ADMIN_URL à l'uri /nginx-exporter (voir configuration ci-dessous)
        - "traefik.http.routers.nginx_exporter.rule=Host(`${ADMIN_URL}`) && PathPrefix(`/nginx-exporter`)"
        # URL traefik sécurisée
        - "traefik.http.routers.nginx_exporter.entrypoints=websecure"
        - "traefik.http.routers.nginx_exporter.tls=true"
        - "traefik.http.routers.nginx_exporter.tls.certresolver=resolver1"
        - "traefik.http.routers.nginx_exporter.service=nginx_exporter"
        - "traefik.http.routers.nginx_exporter.middlewares=nginx_exporter-auth@docker"
        - "traefik.http.services.nginx_exporter.loadbalancer.server.port=9113"
        # Protection de l'url par un mot de passe traefik généré avec la commande : echo $(htpasswd -nbB  adobis motdepasse) | sed -e s/\\$/\\$\\$/g
        - "traefik.http.middlewares.nginx_exporter-auth.basicauth.users=adobis:$$2y$$05$$zMWiM6s1zgLJ8zBnTf3Dm.gxnEKhyt6.099nsOWkdyVon4tElSLIm"
        - "traefik.http.middlewares.nginx_exporter-auth.basicauth.removeheader=true"
    environment:
      # récupération de l'information dans la page stub_status
      # La la status est exposé sur le port 8080 à partir de la version 8.6
      - SCRAPE_URI=http://dc-web-ui:8080/stub_status
      - NGINX_RETRIES=20
      # disponible sous /nginx-exporter
      - TELEMETRY_PATH=/nginx-exporter

Backend module

Monitoring is native for the component, via Spring Actuator, by enabling and exposing the metrics.

Module configuration (usually backend.env)
# Prometheus configuration
management.endpoints.web.base-path=/actuator-backend
management.server.address=0.0.0.0
management.metrics.export.prometheus.enabled=true
docker-compose.yml file
version: "3.7"
services:

  # Déclaration des autres services
  # Other service declaration

  dc_backend:
    image: ${HARBOR_REGISTRY}/dc/backend:${PRODUCT_VERSION}
    networks:
      # Disponible sur le réseau traefik, car disponible via le proxy traefik
      traefik_network:
      # Disponible sur le réseau dc_network
      dc_network:
        aliases:
          - dc-backend
    env_file: backend.env
    deploy:
      mode: replicated
      replicas: 1
      labels:
        # Exposition via traefik
        - "traefik.enable=true"
        - "traefik.docker.network=traefik_network"
        # disponible sur l'URL ADMIN_URL à l'uri /actuator-backend (voir configuration ci-dessous)
        - "traefik.http.routers.dc_backend_actuator.rule=Host(`${ADMIN_URL}`) && PathPrefix(`/actuator-backend`)"
        # URL traefik sécurisée
        - "traefik.http.routers.dc_backend_actuator.entrypoints=websecure"
        - "traefik.http.routers.dc_backend_actuator.tls=true"
        - "traefik.http.routers.dc_backend_actuator.tls.certresolver=resolver1"
        - "traefik.http.routers.dc_backend_actuator.service=dc_backend_actuator"
        - "traefik.http.services.dc_backend_actuator.loadbalancer.server.port=9089"
        - "traefik.http.routers.dc_backend_actuator.middlewares=dc_backend_actuator-auth@docker"
        # Protection de l'url par un mot de passe traefik généré avec la commande : echo $(htpasswd -nbB  adobis motdepasse) | sed -e s/\\$/\\$\\$/g
        - "traefik.http.middlewares.dc_backend_actuator-auth.basicauth.users=adobis:$$2y$$05$$zMWiM6s1zgLJ8zBnTf3Dm.gxnEKhyt6.099nsOWkdyVon4tElSLIm"
        - "traefik.http.middlewares.dc_backend_actuator-auth.basicauth.removeheader=true"

Spark module

Monitoring is native for the component, via Spring Actuator, by enabling and exposing the metrics.

Module configuration (usually spark.env)
# Prometheus configuration
management.endpoints.web.base-path=/actuator-spark
management.server.address=0.0.0.0
management.prometheus.metrics.export.enabled=true
docker-compose.yml file
version: "3.7"
services:

  # Déclaration des autres services
  # Other service declaration

  dc_spark:
    image: ${HARBOR_REGISTRY}/dc/spark:${PRODUCT_VERSION}
    networks:
      # Disponible sur le réseau traefik, car disponible via le proxy traefik
      traefik_network:
      # Disponible sur le réseau dc_network
      dc_network:
        aliases:
          - dc-spark
    env_file: spark.env
    deploy:
      mode: replicated
      replicas: 1
      labels:
        # Exposition via traefik
        - "traefik.enable=true"
        - "traefik.docker.network=traefik_network"
        # disponible sur l'URL ADMIN_URL à l'uri /actuator-spark (voir configuration ci-dessous)
        - "traefik.http.routers.dc_spark_actuator.rule=Host(`${ADMIN_URL}`) && PathPrefix(`/actuator-spark`)"
        # URL traefik sécurisée
        - "traefik.http.routers.dc_spark_actuator.entrypoints=websecure"
        - "traefik.http.routers.dc_spark_actuator.tls=true"
        - "traefik.http.routers.dc_spark_actuator.tls.certresolver=resolver1"
        - "traefik.http.routers.dc_spark_actuator.service=dc_spark_actuator"
        - "traefik.http.services.dc_spark_actuator.loadbalancer.server.port=9090"
        - "traefik.http.routers.dc_spark_actuator.middlewares=dc_spark_actuator-auth@docker"
        # Protection de l'url par un mot de passe traefik généré avec la commande : echo $(htpasswd -nbB  adobis motdepasse) | sed -e s/\\$/\\$\\$/g
        - "traefik.http.middlewares.dc_spark_actuator-auth.basicauth.users=adobis:$$2y$$05$$zMWiM6s1zgLJ8zBnTf3Dm.gxnEKhyt6.099nsOWkdyVon4tElSLIm"
        - "traefik.http.middlewares.dc_spark_actuator-auth.basicauth.removeheader=true"

Backup and restore

We recommend performing cold backups, after first stopping the services that use the elements to back up. For Datachain Core, the elements to back up are:

  • the adobis database (configuration, pipelines, tasks), and the expose database (exposed data): see Operations - Data management;

  • the Spark file system (directory mounted on /data).

Backup and restore of the file system

In the current version of Datachain, the HDFS file system is split into two directories, input and output. The output directory mainly contains the elements produced by the processing pipelines, publications, as well as the cache (notably persistence data). As a result, the output directory can quickly become very large.

Apart from software backups, other technologies (outside the scope of this documentation) can back up file systems (VM snapshots, disk replication, …​). It may be more appropriate to use this type of backup rather than the backups proposed below.

Without an HDFS cluster

For Spark contexts without a Spark/Hadoop cluster, the files are directly accessible in the docker volume. You therefore only need to back up the directory.

  dc_spark1:
    image: ${HARBOR_REGISTRY}/dc/spark:${PRODUCT_VERSION}
    # ...
    volumes:
        - /opt/adobis/data/datachain/hdfs:/data # Standard volume
    # ...

In the case above, you must therefore back up the /opt/adobis/data/datachain/hdfs directory, on each machine hosting a Spark context.

With an HDFS cluster

For Spark contexts with a Spark/Hadoop cluster, Hadoop natively replicates data across all the cluster nodes. Using standard backup techniques seems hardly appropriate.

If a software backup procedure is needed, please follow the Hadoop recommendations on the subject.

Dernière mise à jour le 2 novembre 2026 à 15:16