Operations - Dc-Maestro

This page describes the operating procedures of the Dc-Maestro stack: startup and shutdown, logs, backup, execution history purge and maintenance DAGs. The image settings are described in Configuration - Dc-Maestro.

Services and startup order

Dc-Maestro is made of permanent services and run-once services (migrations). The services rely on Keycloak, PostgreSQL (dc_maestro and airflow databases) and Valkey, which must be started beforehand (see Operations - General).

Service Type Remarks

db-migration

Run once

Migrates the Maestro database to the image version, after an optional backup. To be run before the backend starts. Must not be restarted automatically.

airflow-migration

Run once

Migrates the Airflow database and creates the Maestro user in Airflow. To be run before the Airflow services start.

socket-proxy

Permanent

Filtered access to the Docker API: required for the Airflow worker to launch the task containers.

airflow-webserver, airflow-scheduler, airflow-worker

Permanent

Pipeline execution engine. They share the DAG directory (/opt/airflow/dags) with the Upflower.

maestro-upflower

Permanent

Drops the DAGs generated by Maestro into the DAG directory, and delivers the maintenance DAGs at startup.

backend, synchronizer, websockets

Permanent

Maestro API, synchronization with Airflow, broadcast of execution events.

front-ui

Permanent

Maestro web interface.

Recommended startup order: migrations (db-migration, airflow-migration), then socket-proxy, the Airflow services and the Upflower, then the backend (and its two associated services) and the interface. Shutdown is done in the reverse order.

Service health check

Service Port Check

backend

8080

curl http://localhost:8080/ (also used as the image health check)

front-ui

8080

Web interface

airflow-webserver

8080

Airflow web interface and API (/api/v1/, used by Maestro)

maestro-upflower

8000

/ (image health check)

socket-proxy

2375

Filtered Docker API, used by the Airflow worker

The synchronizer and websockets services do not listen on any port: check their logs.

Logs

Service Location

backend, synchronizer, websockets

Standard output of the container, and the rolling file LOG_FILENAME (for example /log/maestro_logs.log). Rotation is set with LOG_MAXBYTES and LOG_BACKUPCOUNT. Mounting a volume on /log is recommended.

db-migration

Standard output, and /log if LOG_FILENAME points to it

Airflow services

Standard output. Task execution logs under /opt/airflow/logs (mount a volume to keep them)

maestro-upflower

Standard output (level set by LOGLEVEL)

After each update, check in the logs of db-migration and airflow-migration that the migrations went well.

Backup and restore

We recommend cold backups, after stopping the services concerned. The items to back up are:

  • the dc_maestro (pipelines, schedules, rights) and airflow (metadata and execution history) databases: see Operations - Data management;

  • the Airflow DAG directory (/opt/airflow/dags), which contains the DAGs generated by Maestro (maestro subdirectory). These DAGs can be regenerated by Maestro, but backing them up allows a faster recovery. The maintenance DAGs (ops subdirectory) are dropped again at each Upflower startup;

  • optionally, the task execution logs (/opt/airflow/logs).

The db-migration service can also back up the Maestro database before the migration (variable DUMP_DB set to true): the dump_maestro_<date>.tar file is written in the volume mounted on /data/pg_dump.

Execution history purge

Maestro keeps the execution history of the pipelines. To limit the size of the database, a purge can be enabled on the backend: it is disabled by default, nothing is deleted as long as HISTORY_PURGE_ENABLED is not true. The parameters are described in Configuration - Dc-Maestro.

Recommended procedure: first enable HISTORY_PURGE_DRY_RUN to check in the logs what would be deleted, then disable the simulation mode. Keep the same retention values (HISTORY_KEEP_RUNS, HISTORY_RETENTION_DAYS, HISTORY_RULE) as those of the airflow_db_cleanup maintenance DAG.

Airflow maintenance DAGs

The Upflower delivers two maintenance DAGs, airflow_db_cleanup (Airflow database) and airflow_logs_cleanup (Airflow log files). Their settings are described in Configuration - Airflow maintenance DAGs (ops). From an operating point of view:

  • The DAGs are created paused and in simulation mode: they only act after activation.

  • Commissioning: activate the DAG (unpause, only once: in the Airflow interface or with airflow dags unpause), let it run in simulation, check in the logs of its tasks what would be deleted, back up the Airflow database, then enable deletions.

  • Deletions are enabled with OPS_DB_CLEANUP_APPLY / OPS_LOGS_CLEANUP_APPLY (Upflower environment, restart required), with the apply parameter when the DAG is launched manually, or by setting the Airflow variable db_cleanup_apply / logs_cleanup_apply to true.

  • A parameter change in the Upflower environment is taken into account when the Upflower restarts, which rewrites the DAG file.

  • On a large database, reduce the retention gradually (for example OPS_DB_CLEANUP_KEEP_RUNS to 5000, then 1000, then 500).

Update

  1. Back up the dc_maestro and airflow databases, and the DAG directory.

  2. Stop the stack, then restart it with the new version of the images (see Updating a version).

  3. The migrations run (db-migration, airflow-migration). When the synchronizer starts, the existing DAGs are migrated to the format of the new version (MIGRATE_ON_UPDATE).

  4. The Upflower drops the maintenance DAGs delivered with the new version again.

  5. See the technical release notes for configuration changes (ports, image user, etc.).

Dernière mise à jour le 2 novembre 2026 à 15:16