Operations - Data management

This page describes the operations of the data management components: the PostgreSQL databases and the Valkey service. Their settings are described in Configuration - Data management.

Elements to back up

We recommend performing cold backups, after first stopping the services that use the elements to back up. Depending on the deployed architecture, these databases are hosted by one or several PostgreSQL instances (see the image DATABASE_MODE variable).

Database	Stack	Content
adobis	Datachain Core	Pipeline configuration, settings, completed and running tasks
expose	Datachain Core	Exposed data intended for consumers
dc_marketplace	Marketplace	Marketplace data
dc_console	Console	Console data (users, groups, roles)
keycloak	SSO	Keycloak users, groups and sessions
dc_maestro	Dc-Maestro	Maestro data (pipelines, schedules, permissions)
airflow	Dc-Maestro	Airflow metadata (DAGs, runs)
dc_code	DC Code	DC Code data

Database

Stack

Content

adobis

Datachain Core

Pipeline configuration, settings, completed and running tasks

expose

Datachain Core

Exposed data intended for consumers

dc_marketplace

Marketplace

Marketplace data

dc_console

Console

Console data (users, groups, roles)

keycloak

SSO

Keycloak users, groups and sessions

dc_maestro

Dc-Maestro

Maestro data (pipelines, schedules, permissions)

airflow

Dc-Maestro

Airflow metadata (DAGs, runs)

dc_code

DC Code

DC Code data

To back up a whole PostgreSQL instance, back up each of the databases it hosts.

Database backup and restore

The Datachain Core database servers are PostgreSQL servers. For more information on backup/restore procedures, please refer to the online documentation: https://www.postgresql.org/docs/current/app-pgdump.html & https://www.postgresql.org/docs/current/app-pgrestore.html.

For a containerized database, we propose two methods below to perform backups/restores.

For backups/restores, the commands must be run on the machines (docker workers) where the containers actually run.

With a backup volume

A backup volume is mounted on the underlying PostgreSQL service (pg, pg_all, etc.). For example

  dc_pg:
    image: ${HARBOR_REGISTRY}/dc/pg:${PRODUCT_VERSION}
    # ...
    volumes:
        - /opt/adobis/data/datachain/pg:/var/lib/postgresql/data # Standard volume
        - /opt/adobis/data/backup/datachain/pg:/pg_dump # Backup volume
    # ...

For the commands below, the Datachain stack name is dc, and the database to back up is adobis.

Backup

docker exec -t $(docker ps -qf "name=dc_dc_pg\\.") pg_dump -U postgres --format=c --blobs --clean --create adobis -f /pg_dump/$(date +"%Y-%m-%d")_pg_dc_dump.dump

This command generates, in the /pg_dump directory (and therefore in /opt/adobis/data/backup/datachain/pg on the server), a file named <DumpDate in YYYY-MM-DD format>_pg_dc_dump.dump.

Restore

The backup file is 20240924_pg_dc_dump.dump. The restore command is as follows:

echo "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'adobis';" | docker exec -i $(docker ps -qf "name=dc_pg\\.") /usr/bin/psql -U postgres && docker exec -t $(docker ps -qf "name=dc_dc_pg\\.") pg_restore -U postgres -c -C -d postgres /pg_dump/20240924_pg_dc_dump.dump

Without a backup volume

Backup

In this case, the script first performs the postgres dump, then copies it to the final storage directory.

docker exec -t $(docker ps -qf "name=dc_dc_pg\\.") pg_dump -U postgres --format=c --blobs --clean --create adobis -f /tmp/pg_dc_dump.dump && docker cp $(docker ps -qf "name=dc_dc_pg\\."):/tmp/pg_dc_dump.dump /tmp/$(date +"%Y-%m-%d")_pg_dc_dump.dump && docker exec -t $(docker ps -qf "name=dc_dc_pg\\.") rm /tmp/pg_dc_dump.dump

This command generates, in the server /tmp directory, a file named <DumpDate in YYYY-MM-DD format>_pg_dc_dump.dump.

Restore

The backup file is 20240924_pg_dc_dump.dump. The restore command is as follows:

docker cp 20240924_pg_dc_dump.dump $(docker ps -qf "name=dc_dc_pg\\."):/tmp/adobis.dump && echo "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'adobis';" | docker exec -i $(docker ps -qf "name=dc_pg\\.") /usr/bin/psql -U postgres && docker exec -t $(docker ps -qf "name=dc_dc_pg\\.") pg_restore -U postgres -c -C -d postgres /tmp/adobis.dump && docker exec -t $(docker ps -qf "name=dc_dc_pg\\.") rm /tmp/adobis.dump

PostgreSQL monitoring

Monitoring is done through the postgres-exporter image, by deploying an additional service in the deployment file.

docker-compose.yml file

version: "3.7"
services:

  # Déclaration des autres services
  # Other service declaration

  pg_exporter:
    image: quay.io/prometheuscommunity/postgres-exporter:${PG_PROMETHEUS_EXPORTER_VERSION}
    networks:
      # Disponible sur le réseau traefik, car disponible via le proxy traefik
      traefik_network:
      # Disponible sur le réseau dc_network, car interroge le module pg, dont un alias sur ce réseau est 'dc-pg'
      dc_network:
    deploy:
      mode: replicated
      replicas: 1
      labels:
        # Exposition via traefik
        - "traefik.enable=true"
        - "traefik.docker.network=traefik_network"
        # disponible sur l'URL ADMIN_URL à l'uri /pg-exporter (voir configuration ci-dessous)
        - "traefik.http.routers.pg_exporter.rule=Host(`${ADMIN_URL}`) && PathPrefix(`/pg-exporter`)"
        # URL traefik sécurisée
        - "traefik.http.routers.pg_exporter.tls=true"
        - "traefik.http.routers.pg_exporter.tls.certresolver=resolver1"
        - "traefik.http.routers.pg_exporter.entrypoints=websecure"
        - "traefik.http.routers.pg_exporter.service=pg_exporter"
        - "traefik.http.services.pg_exporter.loadbalancer.server.port=9187"
        - "traefik.http.routers.pg_exporter.middlewares=pg_exporter-auth@docker"
        # Protection de l'url par un mot de passe traefik généré avec la commande : echo $(htpasswd -nbB  adobis motdepasse) | sed -e s/\\$/\\$\\$/g
        - "traefik.http.middlewares.pg_exporter-auth.basicauth.users=adobis:$$2y$$05$$zMWiM6s1zgLJ8zBnTf3Dm.gxnEKhyt6.099nsOWkdyVon4tElSLIm"
        - "traefik.http.middlewares.pg_exporter-auth.basicauth.removeheader=true"
    environment:
      # récupération des informations auprès de la base dc-pg
      - DATA_SOURCE_URI=dc-pg:5432/postgres?sslmode=disable
      - DATA_SOURCE_USER=postgres
      # - DATA_SOURCE_PASS_FILE peut être utilisé avec docker secret
      - DATA_SOURCE_PASS=motdepasse
      # disponible sous /pg-exporter
      - PG_EXPORTER_WEB_TELEMETRY_PATH=/pg-exporter

PostgreSQL version upgrade

Major PostgreSQL version changes are described in the technical release notes (for example the migration to version 18 in version 9.1.0). The procedure consists in stopping the services that use the database, backing up the data, starting the new version, then restoring the data. Mounting the data volume on /var/lib/postgresql also allows using pg_upgrade --link.

Valkey

Valkey handles asynchronous message exchanges between the modules (cache and queues). The image has no Adobis-specific configuration: refer to the Valkey documentation for optional data persistence.

Logs are written to the container standard output.
Valkey is used by Datachain Core and by Dc-Maestro (database number 1 for Airflow, database 0 reserved for Datachain notifications).

Dernière mise à jour le 2 novembre 2026 à 15:16