Configuration - Dc-Maestro

This page describes how to configure the images of the Dc-Maestro module, the pipeline orchestrator of the Datachain solution.

The images are configured through Docker *.env files or environment variables.

Unlike the Console and Marketplace images, the Dc-Maestro images currently do not support variables suffixed with _FILE: the procedure described in About environment variables does not apply here. Sensitive values (passwords, Keycloak secrets) must be provided in clear text in the environment variables.

Overview

Dc-Maestro is delivered as several images. A single image can carry several services, which are distinguished by their startup command.

The images run without root privileges and are compatible with OpenShift: see Running without privileges.

Service Image Ports Role

backend

maestro/maestro-backend

8080

Maestro REST API

synchronizer

maestro/maestro-backend

None

Synchronization of pipelines with Airflow

websockets

maestro/maestro-backend

None

Real-time broadcast of execution events

db-migration

maestro/maestro-migrate

None

Maestro database migration (single run)

front-ui

maestro/maestro-ui

8080

Web interface

airflow-webserver, airflow-scheduler, airflow-worker, airflow-migration

maestro/maestro-airflow

8080 (airflow-webserver only)

Airflow execution engine (Celery executor)

maestro-upflower

maestro/maestro-upflower

8000

Maestro / Airflow link: deposits the DAGs generated by Maestro

socket-proxy

maestro/docker-proxy

2375

Filtered access to the Docker API, for launching tasks

Two additional images are not services: they are the images in which pipeline tasks (maestro/maestro-python3.12-airflow-dc) and DCCode notebooks (maestro/maestro-notebook) run.

Dc-Maestro also uses:

  • one PostgreSQL database for Maestro (dc_maestro) and one for Airflow (airflow);

  • a Valkey (Redis-compatible) server: cache and event queues for Maestro, Celery broker for Airflow.

In an installation integrated into the Datachain suite, these databases are hosted by the dc/pg image (dc_maestro and airflow values of DATABASE_MODE) and the Datachain Valkey: see Data management. Airflow then uses Valkey database number 1, database 0 being reserved for Datachain notifications.

Backend module - Image maestro/maestro-backend

This image carries the Maestro API, as well as the two background processes (Airflow synchronization and WebSocket events). In deployment, three separate services use the same image with a different command.

Image description

Description

Carries the Dc-Maestro REST API: management of Datachain instances, pipelines, schedules and permissions. Generates the Airflow DAGs, synchronizes their state and broadcasts execution events.

Size

~580MB

Technical components

Python 3.12 / FastAPI / Uvicorn / SQLAlchemy (Alembic migrations)

User

dc (uid 38330, primary group root (gid 0), home directory /home/maestro). The image also accepts any arbitrary uid imposed at startup (OpenShift): this uid belongs to the root group (gid 0). The directory mounted on /log must be writable by uid 38330 or by group 0.

Commands
Command Purpose

python run.py webserver

API only, without WebSocket or Airflow synchronization. Used for the backend service.

python run.py synchronizer

Airflow synchronization loop. At startup, runs the migration of existing DAGs if MIGRATE_ON_UPDATE is 1. Used for the synchronizer service.

python run.py websockets

Execution event broadcast loop. Used for the websockets service.

python run.py standalone

The three components in a single container. Default command of the image, suited to a simple deployment.

Volumes
Path Purpose Required

/log

Directory of the log file, when LOG_FILENAME points to it (value used in the provided deployments: /log/maestro_logs.log)

No, but recommended

Logs

Application logs are written to the container’s standard output and to a rotating file (LOG_FILENAME).

Log Purpose

maestro_logs.log

Application logs of the API, synchronization and events. Rotation by size (LOG_MAXBYTES), number of files kept defined by LOG_BACKUPCOUNT.

URLs
Port URL Purpose

8080

/

API listening port (PORT variable). Also used as the image health check (curl http://localhost:8080/). In the provided deployments, external access goes through the /api prefix of the Maestro domain name, which is removed by the reverse proxy before reaching the container.

Environment variables

Variables are read from the container’s environment variables; failing that, from the file designated by ENV_FILE (by default .env in the application directory). An environment value always takes precedence over the one in the file. Booleans accept 0/1 or true/false.

Variables without a default value are required.

Maestro database

Variable Default Description

PG_HOST

(none)

PostgreSQL server host

PG_PORT

5432

PostgreSQL server port

PG_DB

(none)

Database name (dc_maestro for an installation with dc/pg)

PG_USER

(none)

PostgreSQL user

PG_PWD

(none)

PostgreSQL user password

DB_URL

(built)

Full SQLAlchemy URL (postgresql+asyncpg://…​). If absent, it is built from the PG_* variables above.

EXTRA_ENGINE_PARAMS

(none)

SQLAlchemy connection pool parameters, in JSON format. Example: {"pool_size":20,"max_overflow":5}

CONNECT_ARGS

(none)

Driver connection arguments, in JSON format

ECHO_SQL

0

1: logs SQL queries

Valkey

Variable Default Description

VALKEY_HOST

(none)

Valkey server host. The legacy names REDIS_HOST, REDIS_PORT and REDIS_PWD are accepted as a fallback.

VALKEY_PORT

6379

Valkey server port

VALKEY_PWD

(none)

Valkey password, if protected

VALKEY_LOCK_TIMEOUT_S

60

Lifetime (in seconds) of the locks set by the different Maestro instances

Keycloak

Variable Default Description

DC_KEYCLOAK_HOST

(none)

Keycloak server URL

DC_KEYCLOAK_REALM

(none)

Keycloak realm (dc-realm in standard installations)

DC_KEYCLOAK_CLIENT_ID

(none)

Keycloak client of the Maestro interface (maestro)

DC_KEYCLOAK_CLIENT_SECRET

(none)

Secret of this client. Required for startup; leave empty if the client is public.

DC_KEYCLOAK_CLIENT_BACKEND

(none)

Keycloak client representing the API (maestro_backend)

DC_KEYCLOAK_CLIENT_AIRFLOW_ID

(none)

Keycloak client used by Airflow (maestro_airflow)

DC_KEYCLOAK_CLIENT_AIRFLOW_SECRET

(none)

Secret of the Airflow client

DC_KEYCLOAK_ADMIN

(none)

Keycloak administrator account used by Maestro

DC_KEYCLOAK_ADMIN_PWD

(none)

Password of this account

DC_KEYCLOAK_ADMIN_REALM

master

Realm in which the administrator account is defined

DC_MAESTRO_ADMIN_PWD

(none)

Password of the maestro-sys system user, defined in Keycloak (Credentials tab) with the maestro_admin role of the maestro_backend client. Maestro uses it for its internal operations, notably the DAG migration.

DC_KEYCLOAK_GROUPS_LABEL

user-groups

Name of the Keycloak attribute that carries the user’s groups

Instance identification

Variable Default Description

PLATEFORM_ID

default

Platform identifier

MODULE_INSTANCE_ID

dcm-1

Module instance identifier, useful when several backends coexist

Airflow integration

Variable Default Description

AIRFLOW_API_URL

(none)

Airflow REST API URL. Example: http://airflow-api:8080/api/v1/

AIRFLOW_UPFLOWER_URL

(none)

Upflower URL. Example: http://maestro-upflower:8000/

AIRFLOW_USER

(none)

Airflow user used by Maestro (maestro)

AIRFLOW_PASSWORD

(none)

Password of this user

AIRFLOW_DB_URI

(none)

Direct access to the Airflow database, in the format user:password@host/airflow. Optional: speeds up synchronization.

AIRFLOW_ONLY_API

1

1: synchronization uses only the Airflow API. 0: also uses the direct connection AIRFLOW_DB_URI.

AIRFLOW_SYNC_DELAY_MIN

5

Frequency of the synchronization with Airflow, in minutes

STATUS_STREAM_DELAY_SEC

10

Frequency of the execution event broadcast, in seconds

MIGRATE_ON_UPDATE

1

1: at each synchronizer startup, migrates existing DAGs to the format of the installed version. 0: disabled (to be reserved for services other than the synchronizer).

Task execution

These variables define how pipeline tasks are launched by Airflow.

Variable Default Description

DC_AFJOB_IMAGE

(none)

Image in which Datachain tasks run (maestro/maestro-python3.12-airflow-dc:<version>). Required in practice: Maestro does not start without it.

DEPLOY_MODE

SWARM

Deployment mode: SWARM or K8S

DC_AF_OPERATOR

DockerOperator

Airflow operator used by default

DOCKER_URI

tcp://socket-proxy:2375

Address of the Docker API used by tasks (in practice the socket-proxy service)

DOCKER_NETWORK

(none)

Docker network to which each task container is attached

DOCKER_NETWORKS

(none)

For DCCode tasks: list of networks separated by ;. Task containers must join the backend network to reach MAESTRO_PRIVATE_API_URL.

DOCKER_SWARM_PLACEMENT

(none)

Swarm placement constraints for task containers

DOCKER_FORCE_PULL

false

true: pulls the image before each task

ADOBIS_REGISTRY_CONN_ID

(none)

Identifier of the Airflow connection used to authenticate against the image registry when pulling the image of a Docker task. The connection is declared on the Airflow side (AIRFLOW_CONN_<IDENTIFIER> variable). If not set: no authentication is attempted.

K8S_NAMESPACE

datachain

Kubernetes: namespace in which to launch tasks

K8S_REGISTRY_SECRETS

datachain-ovh-registry-credential

Kubernetes: secret used to pull images

K8S_AFFINITY

(none)

Kubernetes: affinity rules of the task pods

K8S_SECURITY_CONTEXT

{"fsGroup": 1000, "runAsUser": 1000}

Kubernetes: security context of the DCCode pods

K8S_DCCODE_IMAGE

(none)

Kubernetes: image of the DCCode tasks

DCCode

Variable Default Description

DCCODE_LINKER_URL

(none)

(Private) URL of the dccode_linker service

DCCODE_PUBLIC_URL

localhost

Public URL of DCCode

DCCODE_RUN_IMAGE

(reference image)

Image in which DCCode notebooks run (maestro/maestro-notebook:<version>). It is recommended to set it explicitly: the default value references a development version.

Email sending (SMTP)

Variable Default Description

SMTP_SERVER

(none)

SMTP server

SMTP_PORT

(none)

SMTP server port (25, 465 or 587)

SMTP_SENDER

maestro-noreply@adobisgroup.fr

Sender address

DC_MAESTRO_ADMIN_EMAIL

(none)

Email address of the instance administrator: it is added to the recipients of pipeline notification emails. Must be a valid address, otherwise startup fails.

SMTP_USER

(none)

SMTP account. Without SMTP_USER and SMTP_PASSWORD, no authentication is performed (internal relay, port 25 for example).

SMTP_PASSWORD

(none)

SMTP account password

SMTP_STARTTLS

true

STARTTLS after connection (ports 25 and 587)

SMTP_SSL

false

Implicit SSL from the connection (port 465). Takes precedence over SMTP_STARTTLS.

SMTP_TIMEOUT

30

Timeout, in seconds, of the connection and of each SMTP exchange

TEMPLATE_PATH

app/core/templates/

Directory of email templates

Maestro addresses

Variable Default Description

MAESTRO_PUBLIC_URL

(none)

Address where users open Maestro (the web interface): used for the links in emails. If absent, the legacy MAESTRO_FRONT_URL, then MAESTRO_HOST_URL, are read. With none of the three, emails contain no link.

MAESTRO_PRIVATE_API_URL

(none)

Address of the Maestro API, as seen by task containers (DCM_HOST_URL). In Swarm: the name of the service and not of a container, for example http://maestro-backend. Without it, pipeline trigger tasks fail (Environment variable DCM_HOST_URL is missing).

DC_MAESTRO_API_PREFIX

(empty)

URL prefix (for example /api) under which a reverse proxy publishes the API, when it removes it before forwarding the request to the container. It does not change the addresses served by the container: it is only used so that the interactive API documentation (/docs) builds correct addresses behind this proxy. Optional: not needed for standard deployments. Must start with /.

USE_INSTANCE_PRIVATE_URL

(not set)

If the variable is set (whatever its value, including 0), Maestro reaches Datachain instances through their private (internal network) address rather than the public one. To disable, leave the variable unset.

History purge

The purge of the execution history is disabled by default: nothing is deleted unless HISTORY_PURGE_ENABLED is true.

Variable Default Description

HISTORY_PURGE_ENABLED

false

Enables the purge

HISTORY_PURGE_DRY_RUN

false

true: the purge only logs what it would delete

HISTORY_KEEP_RUNS

500

Maximum number of runs kept per pipeline (the most recent ones)

HISTORY_RETENTION_DAYS

548

Maximum age of a run, in days (548 days = 18 months)

HISTORY_RULE

cap

cap: a run is purged as soon as it exceeds the maximum number OR the maximum age. protect: only if it exceeds both.

HISTORY_PURGE_DELETED_PIPELINES

true

Also purges the history of pipelines that no longer exist

HISTORY_SYNC_KEEP_DAYS

30

Retention period, in days, of synchronization rows (the last successful synchronization is always kept)

HISTORY_PURGE_HOUR

3

The purge runs once a day, starting at this hour (0-23, server time)

HISTORY_PURGE_BATCH

500

Number of runs deleted per transaction

HISTORY_PURGE_MAX_SECONDS

600

Maximum duration of a purge, in seconds (the rest is processed the next day)

The HISTORY_KEEP_RUNS, HISTORY_RETENTION_DAYS and HISTORY_RULE values have the same defaults as the airflow_db_cleanup Airflow maintenance DAG: it is recommended to keep them identical on both sides.

Logs and execution

Variable Default Description

LOG_FILENAME

/opt/adobis/logs/maestro.log

Log file path. The provided deployments use /log/maestro_logs.log.

LOG_MAXBYTES

10000000

Maximum size of a log file before rotation, in bytes (10 MB by default)

LOG_BACKUPCOUNT

3

Number of archived log files kept

DEBUG

0

Enables debug logs

HOST

0.0.0.0 (image)

API listening address

PORT

8080 (image)

API listening port

UVICORN_WORKERS

8 (image)

Number of Uvicorn processes of the API. Only the backend service uses it.

TTL_CACHE_MIN

5

Lifetime, in minutes, of the internal cache

ENV_FILE

.env

Variables file read in addition to the environment

DEV

0

Development mode (automatic code reload, single process). Do not use in production.

Configuration file

The image embeds a default .env file, which the container’s environment variables override. Example of a backend.env file for a Docker Swarm deployment (values between angle brackets must be replaced):

# Maestro database
PG_HOST=<postgresql host>
PG_PORT=5432
PG_DB=dc_maestro
PG_USER=<user>
PG_PWD=<password>

# Valkey
VALKEY_HOST=<valkey host>

# Keycloak
DC_KEYCLOAK_HOST=<public keycloak url>
DC_KEYCLOAK_REALM=dc-realm
DC_KEYCLOAK_CLIENT_ID=maestro
DC_KEYCLOAK_CLIENT_SECRET=<maestro client secret>
DC_KEYCLOAK_ADMIN=<keycloak administrator>
DC_KEYCLOAK_ADMIN_PWD=<password>
DC_KEYCLOAK_ADMIN_REALM=master
DC_KEYCLOAK_CLIENT_AIRFLOW_ID=maestro_airflow
DC_KEYCLOAK_CLIENT_AIRFLOW_SECRET=<maestro_airflow client secret>
DC_KEYCLOAK_CLIENT_BACKEND=maestro_backend
DC_MAESTRO_ADMIN_PWD=<maestro-sys password>

# Airflow
AIRFLOW_UPFLOWER_URL=http://maestro-upflower:8000/
AIRFLOW_API_URL=http://airflow-api:8080/api/v1/
AIRFLOW_USER=maestro
AIRFLOW_PASSWORD=<password of the maestro airflow user>
AIRFLOW_ONLY_API=1
DOCKER_URI=tcp://socket-proxy:2375
DC_AFJOB_IMAGE=<registry>/maestro/maestro-python3.12-airflow-dc:<version>
DEPLOY_MODE=SWARM

# Maestro addresses
MAESTRO_PUBLIC_URL=https://<maestro domain name>
MAESTRO_PRIVATE_API_URL=http://maestro-backend

# SMTP
SMTP_SERVER=<smtp server>
SMTP_PORT=587
SMTP_SENDER=<sender>
SMTP_USER=<smtp account>
SMTP_PASSWORD=<password>

# Logs
LOG_FILENAME=/log/maestro_logs.log
LOG_MAXBYTES=104800
LOG_BACKUPCOUNT=3

EXTRA_ENGINE_PARAMS={"pool_size":20,"max_overflow":5}
STATUS_STREAM_DELAY_SEC=10
AIRFLOW_SYNC_DELAY_MIN=5

Each of the following variables is read by only one service. To adjust them, define them at the level of that service in the deployment file (docker-compose) rather than in backend.env, which is shared by all services:

Variable Service Default Role

UVICORN_WORKERS

backend

8 (image)

Number of Uvicorn processes of the API

MIGRATE_ON_UPDATE

synchronizer

1

Migration of existing DAGs at each startup

AIRFLOW_SYNC_DELAY_MIN

synchronizer

5

Frequency of the synchronization with Airflow, in minutes

STATUS_STREAM_DELAY_SEC

websockets

10

Frequency of the execution event broadcast, in seconds

As an example, the provided Docker Swarm deployments reduce the synchronization frequency to 2 minutes and the event frequency to 5 seconds, and may limit the API to 3 processes.

Migration module - Image maestro/maestro-migrate

This image applies the Maestro database migrations (Alembic). It is run once at each installation or update, before the backend starts, and then stops.

Image description

Description

Migrates the Maestro database schema to the image version (alembic upgrade head), after an optional database backup.

Size

~540MB

Technical components

Python 3.12 / Alembic / PostgreSQL client (pg_dump)

User

dc (uid 38330, primary group root (gid 0), home directory /home/maestro). The image also accepts any arbitrary uid imposed at startup (OpenShift): this uid belongs to the root group (gid 0). The directories mounted on /data/pg_dump and /log must be writable by uid 38330 or by group 0; an empty named volume inherits the correct permissions from the image on first startup.

Volumes
Path Purpose Required

/data/pg_dump

Directory in which the database backup is written before migration (dump_maestro_<date>.tar, in pg_dump tar format)

Only if DUMP_DB is true

/log

Log directory, if LOG_FILENAME points to it

No

URLs

None: the container runs and then stops. The service must not be restarted automatically (restart policy none).

Environment variables

Variable Default Description

PG_HOST

(none)

PostgreSQL server host

PG_PORT

5432

PostgreSQL server port

PG_DB

(none)

Database name (dc_maestro)

PG_USER

(none)

PostgreSQL user. Must be able to modify the schema.

PG_PWD

(none)

PostgreSQL user password

DUMP_DB

(disabled)

The value true (lowercase, any other value is ignored) triggers a database backup before the migration. The container waits for PostgreSQL to respond before starting it.

RESET_DB

(disabled)

The value true deletes all Maestro data (alembic downgrade base) after interactive confirmation. Unusable as a Swarm service (no input possible): reserved for development environments.

The service must access the same database as the backend. The PG_* variables are the same as those of the backend.

Configuration file

PG_HOST=<postgresql host>
PG_PORT=5432
PG_DB=dc_maestro
PG_USER=<user>
PG_PWD=<password>
DUMP_DB=true

Interface module - Image maestro/maestro-ui

This image serves the Maestro web interface (Angular application). Its configuration is injected into the application files when the container starts: it does not require rebuilding the image.

Image description

Description

Dc-Maestro web interface: design, scheduling and monitoring of pipelines.

Size

~85MB

Technical components

Angular application served by Nginx (alpine)

User

nginx (uid 101, primary group root (gid 0)). The image also accepts any arbitrary uid imposed at startup (OpenShift): this uid belongs to the root group (gid 0).

URLs
Port URL Purpose

8080

/

Interface listening port. The interface calls the Maestro API and WebSockets directly (MAESTRO_API_URL, MAESTRO_WS_URL): these addresses must be reachable from the browser.

Environment variables

Variable Default Description

MAESTRO_API_URL

(none)

Public URL of the Maestro API, as seen from the browser. Example: https://<maestro domain name>/api

MAESTRO_WS_URL

(none)

Public URL of the Maestro WebSockets. Example: wss://<maestro domain name>/api

MAESTRO_KEYCLOAK_AUTH_URL

(none)

Keycloak authentication URL, as seen from the browser (ends with /auth)

MAESTRO_KEYCLOAK_REALM

(none)

Keycloak realm (dc-realm)

MAESTRO_KEYCLOAK_CLIENT_ID

(none)

Keycloak client of the interface (maestro)

DOCUMENTATION_BASE_URL

https://docs.adobisdc.fr/fr/datachain-maestro/current/ (image)

Documentation URL, used by the interface help links

SUPPORT_BASE_URL

https://adobisgroup.atlassian.net/servicedesk/customer/portal/10 (image)

Support URL, used by the interface help links

IS_EXPOSV2_ACTIVE

(none)

true: enables v2 exposure of pipelines

IS_DCCODE_ACTIVE

(none)

true: enables the DCCode integration (prerequisite: DCCode deployed)

AIRFLOW_TIME_ZONE

(none)

Airflow time zone, used to display schedules

Configuration file

MAESTRO_API_URL=https://<maestro domain name>/api
MAESTRO_WS_URL=wss://<maestro domain name>/api
MAESTRO_KEYCLOAK_AUTH_URL=https://<public keycloak url>/auth
MAESTRO_KEYCLOAK_REALM=dc-realm
MAESTRO_KEYCLOAK_CLIENT_ID=maestro
DOCUMENTATION_BASE_URL=https://docs.adobisdc.fr/fr/datachain-maestro/current/
SUPPORT_BASE_URL=<support url>
IS_EXPOSV2_ACTIVE=true
IS_DCCODE_ACTIVE=false
AIRFLOW_TIME_ZONE=Europe/Paris

Airflow module - Image maestro/maestro-airflow

Apache Airflow 2.11 image (Python 3.12) enriched with Maestro extensions (plugins, email templates, Datachain client). The same image carries four services, which are distinguished by their command.

Image description

Description

Runs the pipelines generated by Maestro as Airflow DAGs, with the Celery executor.

Size

~2.8GB

Technical components

Apache Airflow 2.11 / Python 3.12 / Celery

User

airflow (uid 50000, gid 0) by default in the image. The provided deployments impose 38330:0 (uid 38330, group root): the directories mounted on /opt/airflow/dags and /opt/airflow/logs must be writable by this uid, or by group 0.

Commands
Command Purpose

webserver

Airflow web interface and REST API (airflow-webserver service). The API is used by Maestro; the interface is exposed externally only if necessary.

scheduler

DAG scheduler (airflow-scheduler service)

celery worker

Runs tasks (airflow-worker service). It launches task containers through the socket-proxy service.

airflow db upgrade then reset of the maestro user password

Airflow database migration and creation of the Maestro user (airflow-migration service, single run)

Volumes
Path Purpose Required

/opt/airflow/dags

Airflow DAGs. This directory is shared with the Upflower, which deposits the DAGs generated by Maestro there (maestro subdirectory).

Yes

/opt/airflow/logs

Task execution logs

Yes, to keep the logs

URLs
Port URL Purpose

8080

/

Airflow web interface and API (airflow-webserver service only). The API is used by Maestro on /api/v1/.

Environment variables

Airflow is configured through the standard AIRFLOW__<SECTION>__<KEY> variables, described in the Apache Airflow documentation. The variables below are those set in the provided deployments, and shared by the four services through an airflow_common.env file.

Variable Deployment value Description

AIRFLOW__CORE__EXECUTOR

CeleryExecutor

Task executor

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

postgresql+psycopg2://<user>:<password>@<host>/airflow

Airflow metadata database

AIRFLOW__CELERY__RESULT_BACKEND

db+postgresql://<user>:<password>@<host>/airflow

Celery task results (same database)

AIRFLOW__CELERY__BROKER_URL

redis://<valkey host>:6379/1

Celery broker (Valkey). Use a database different from those already used elsewhere.

AIRFLOW__CORE__FERNET_KEY

(to be defined)

Encryption key for Airflow connections and variables

AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION

true

DAGs are created paused

AIRFLOW__CORE__LOAD_EXAMPLES

false

Do not load example DAGs

AIRFLOW__CORE__MIN_SERIALIZED_DAG_UPDATE_INTERVAL

5

Minimum update interval of serialized DAGs, in seconds

AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG

1

Maximum number of simultaneous runs of a single DAG

AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG

5

Maximum number of simultaneous tasks of a single DAG

AIRFLOW__CORE__PARALLELISM

15

Maximum number of simultaneous tasks across the whole of Airflow

AIRFLOW__API__AUTH_BACKENDS

airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session

API authentication methods. Basic authentication is required by Maestro.

AIRFLOW__WEBSERVER__BASE_URL

http://airflow-api

Web server base address

AIRFLOW__WEBSERVER__PAGE_SIZE

20

Number of items per page

AIRFLOW__WEBSERVER__WEB_SERVER_WORKER_TIMEOUT

300

Web server worker timeout, in seconds

AIRFLOW__LOGGING__LOGGING_LEVEL

INFO

Log level

AIRFLOW__LOGGING__EXTRA_LOGGER_NAMES

maestro

Additional loggers taken into account

AIRFLOW__SMTP__SMTP_HOST

(smtp server)

SMTP server for Airflow emails

AIRFLOW__SMTP__SMTP_PORT

587

SMTP port

AIRFLOW__SMTP__SMTP_STARTTLS

True

STARTTLS

AIRFLOW__SMTP__SMTP_USER

(smtp account)

SMTP account

AIRFLOW__SMTP__SMTP_PASSWORD

(password)

SMTP password

AIRFLOW__SMTP__SMTP_MAIL_FROM

(sender)

Sender address

ADOBIS_REGISTRY_CONN_ID, AIRFLOW_CONN_ADOBIS_REGISTRY

(empty if DCCode inactive)

Airflow connection to the image registry, used to pull the image of Docker tasks and, with DCCode, that of notebooks (ADOBIS_REGISTRY_CONN_ID: connection identifier, to be set in the backend as well; AIRFLOW_CONN_ADOBIS_REGISTRY: its definition). To be set when the registry requires authentication, or when IS_DCCODE_ACTIVE is true on the interface side.

The following variables are set in the image and can be overridden: AIRFLOW__EMAIL__EMAIL_BACKEND (custom_utils.email_handler.send_email_with_id), AIRFLOW__EMAIL__HTML_CONTENT_TEMPLATE (/opt/airflow/mail_template.j2.html) and AIRFLOW__EMAIL__SUBJECT_TEMPLATE (/opt/airflow/mail_subject.j2.html).

Variables specific to the migration service (airflow-migration):

Variable Value Description

_AIRFLOW_DB_MIGRATE

true

Migrates the Airflow database at startup

_AIRFLOW_WWW_USER_CREATE

true

Creates the Maestro Airflow user

_AIRFLOW_WWW_USER_USERNAME

maestro

User name. Must match the backend’s AIRFLOW_USER.

_AIRFLOW_WWW_USER_PASSWORD

(password)

User password. Must match the backend’s AIRFLOW_PASSWORD. The password is reset each time the migration runs.

Configuration file

AIRFLOW__CORE__EXECUTOR=CeleryExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://<user>:<password>@<postgresql host>/airflow
AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://<user>:<password>@<postgresql host>/airflow
AIRFLOW__CELERY__BROKER_URL=redis://<valkey host>:6379/1
AIRFLOW__CORE__FERNET_KEY=<fernet key>
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=true
AIRFLOW__CORE__LOAD_EXAMPLES=false
AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG=1
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG=5
AIRFLOW__CORE__PARALLELISM=15
AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session
AIRFLOW__WEBSERVER__BASE_URL=http://airflow-api
AIRFLOW__LOGGING__EXTRA_LOGGER_NAMES=maestro
AIRFLOW__LOGGING__LOGGING_LEVEL=INFO
AIRFLOW__SMTP__SMTP_HOST=<smtp server>
AIRFLOW__SMTP__SMTP_PORT=587
AIRFLOW__SMTP__SMTP_STARTTLS=True
AIRFLOW__SMTP__SMTP_USER=<smtp account>
AIRFLOW__SMTP__SMTP_PASSWORD=<password>
AIRFLOW__SMTP__SMTP_MAIL_FROM=<sender>

Upflower module - Image maestro/maestro-upflower

The Upflower is the link service between Maestro and Airflow: it receives the generated DAGs from Maestro and deposits them in the Airflow DAG directory. It verifies the Keycloak token of the calls it receives.

Image description

Description

API for depositing the Airflow DAGs generated by Maestro. Also delivers, at startup, the maintenance DAGs provided with the image (ops subdirectory of the DAG directory).

Size

~200MB

Technical components

Python 3.12 / FastAPI / Uvicorn (3 processes)

User

dc (uid 38330, primary group root (gid 0), home directory /home/maestro). The image also accepts any arbitrary uid imposed at startup (OpenShift): this uid belongs to the root group (gid 0). The directory mounted on /opt/airflow/dags must be writable by uid 38330 or by group 0. The deposited DAG files are assigned to the Upflower user: see AIRFLOW_UID and AIRFLOW_GID.

Volumes
Path Purpose Required

/opt/airflow/dags

Airflow DAG directory: must be the same as the one mounted in the Airflow services

Yes

URLs
Port URL Purpose

8000

/

Upflower API, called by the backend (AIRFLOW_UPFLOWER_URL). Also used as the image health check.

Environment variables

Variable Default Description

DC_KEYCLOAK_HOST

localhost

Keycloak server URL

DC_KEYCLOAK_REALM

dc-realm

Keycloak realm

DC_KEYCLOAK_CLIENT_ID

maestro

Keycloak client

DC_KEYCLOAK_CLIENT_SECRET

(none)

Keycloak client secret

DC_KEYCLOAK_AUDIENCE

maestro_backend

Audience expected in received tokens

AIRFLOW_DAG_PATH

/opt/airflow/dags

DAG directory in the container

AIRFLOW_MAESTRO_FOLDER

maestro

Subdirectory of the DAGs generated by Maestro

AIRFLOW_OPS_FOLDER

ops

Subdirectory of the maintenance DAGs delivered with the image

UPFLOWER_SEED_OPS_DAGS

1

0: does not deliver the maintenance DAGs at startup

UPFLOWER_OPS_DAGS_DIR

/app/ops_dags

Directory, in the image, of the delivered maintenance DAGs

AIRFLOW_UID

1000

Owner of the deposited DAG files. Since the Upflower does not run as root, it can only assign files to its own uid: set 38330 (uid of the Upflower and of the provided Airflow services). Another value does not prevent DAGs from being deposited, but the files remain owned by uid 38330.

AIRFLOW_GID

0

Group of the deposited DAG files: must be a group of the Upflower (root, gid 0, by default) and allow reading by the Airflow services.

LOGLEVEL

INFO

Log level

The parameters of the maintenance DAGs (history retention, for example) are also set through Upflower environment variables: the Upflower writes them into the delivered DAG file, which avoids configuring the Airflow services. See Airflow maintenance DAGs (ops).

Configuration file

DC_KEYCLOAK_HOST=<public keycloak url>
DC_KEYCLOAK_REALM=dc-realm
DC_KEYCLOAK_CLIENT_ID=maestro
DC_KEYCLOAK_CLIENT_SECRET=<maestro client secret>
DC_KEYCLOAK_AUDIENCE=maestro_backend
AIRFLOW_DAG_PATH=/opt/airflow/dags
AIRFLOW_MAESTRO_FOLDER=maestro

Airflow maintenance DAGs (ops)

The maestro/maestro-upflower image delivers two maintenance DAGs, which clean up Airflow’s technical data (Airflow 2.x). They are not Maestro DAGs: they are deposited in the ops subdirectory of the DAG directory (variable AIRFLOW_OPS_FOLDER), not in maestro.

DAG Role

airflow_db_cleanup

Cleans up the Airflow database: run history, technical tables (jobs, sessions, Celery results, import errors) and audit log. It can also purge the metadata of DAGs that were deleted without being cleaned up.

airflow_logs_cleanup

Deletes old Airflow log files: execution logs of each DAG, scheduler logs, empty folders, and log folders of deleted DAGs.

How it works

  • Delivery: at startup, the Upflower copies the files delivered with the image (directory /app/ops_dags, variable UPFLOWER_OPS_DAGS_DIR) into the ops directory. A file is rewritten only if its content changes: an image or configuration update is therefore applied when the Upflower restarts. Manual changes made to a delivered file are overwritten.

  • Configuration: all the parameters of each DAG are set through environment variables of the Upflower (OPS_DB_CLEANUP_<PARAMETER> and OPS_LOGS_CLEANUP_<PARAMETER>, in upper case). The Upflower writes the values into the DAG file at delivery time: the Airflow services have nothing to configure. The type of the value is that of the parameter (boolean: true/false, 1/0, yes/no, on/off; integer; text). An invalid value is ignored (warning in the Upflower logs) and the default value is kept.

  • Deactivation: UPFLOWER_SEED_OPS_DAGS=0 prevents the delivery of the DAGs. The files must then be provided by other means, and their parameters changed directly in the SETTINGS block of the file.

  • Cautious by default: each DAG is created paused and only performs a simulation (dry-run): it reports in its logs what it would delete, without deleting anything.

A DAG is started (unpaused) only when it is created. If the DAG already exists in a paused state, reactivate it once in the Airflow interface (or with airflow dags unpause), even after setting ENABLED to true.

Setting up a maintenance DAG

  1. Set OPS_DB_CLEANUP_ENABLED=true (or OPS_LOGS_CLEANUP_ENABLED=true) in the Upflower environment, adjust the other parameters if needed, then restart the Upflower.

  2. Let the DAG run in simulation mode and check in its logs what would be deleted.

  3. Back up the Airflow database (pg_dump) before the first actual deletion.

  4. Enable deletions: OPS_DB_CLEANUP_APPLY=true (or OPS_LOGS_CLEANUP_APPLY=true) in the Upflower environment, or, without restarting, run the DAG manually with the apply parameter set to true, or create the Airflow variable db_cleanup_apply (respectively logs_cleanup_apply) set to true.

On a large database, reduce the retention gradually (for example OPS_DB_CLEANUP_KEEP_RUNS to 5000, then 1000, then 500).

Scope

The SCOPE parameter of both DAGs defines the DAGs concerned:

  • maestro (default): only the DAGs whose file path contains SCOPE_MARKER (by default /maestro/maestro_, i.e. the DAGs generated by Maestro). Other DAGs (handwritten ones, the maintenance DAGs themselves) are never touched;

  • all: all the DAGs of the Airflow instance, whether they come from Maestro or not.

airflow_db_cleanup DAG

The DAG never deletes DAGs: the dag and serialized_dag tables are not modified (except with PURGE_ORPHANS, see below). Only the history is cleaned up:

  • technical tables (job, celery_taskmeta, celery_tasksetmeta, session, import_error): rows older than TECH_DAYS days, whatever the scope (these tables are global);

  • runs (dag_run, with their task instances and xcom): according to RUNS_MODE;

  • audit log (log): rows older than LOG_DAYS days. A running or queued run is never deleted.

Variable (prefix OPS_DB_CLEANUP_) Default Description

ENABLED

false

true: the DAG is created unpaused. No effect if the DAG already exists

APPLY

false

true: actually deletes. false: simulation only

SCHEDULE

0 3 * * 0

Schedule (cron expression: without quotes in a Docker environment file), by default every Sunday at 03:00. none: DAG to be run manually

SCOPE

maestro

Scope: maestro or all

SCOPE_MARKER

/maestro/maestro_

maestro scope: text that the DAG file path must contain

TECH_DAYS

14

Technical tables: maximum age, in days

RUNS_MODE

keep_runs

Run cleanup: keep_runs (limits below) or skip (runs are not touched)

KEEP_RUNS

500

keep_runs mode: maximum number of runs kept per DAG (the most recent ones)

RUNS_DAYS

548

keep_runs mode: maximum age of the runs, in days (548 days, i.e. 18 months)

RUNS_RULE

cap

cap: a run is deleted as soon as it exceeds either of the two limits. protect: only if it exceeds both (a DAG that is rarely run then keeps its entire history)

LOG_DAYS

90

Audit log (log table): maximum age, in days

PURGE_ORPHANS

false

true: also purges the metadata of DAGs that were incompletely deleted (file gone, but rows left in the database)

ORPHAN_MIN_AGE_DAYS

7

PURGE_ORPHANS: the DAG must have been inactive for at least N days

ORPHAN_MAX

200

PURGE_ORPHANS: maximum number of DAGs purged per run (the purge is done in stages)
The KEEP_RUNS, RUNS_DAYS and RUNS_RULE values are the same as those of the Maestro history purge (HISTORY_KEEP_RUNS, HISTORY_RETENTION_DAYS and HISTORY_RULE of the backend): keep the same values on both sides.

A DAG is purged (PURGE_ORPHANS) only if it has been inactive for more than ORPHAN_MIN_AGE_DAYS days, is within the chosen scope, and has no running execution started less than 3 days ago. Runs are deleted first, in batches, then the DAG itself, as the Airflow REST API does (DELETE /dags/{dag_id}). The audit log rows are kept.

The tasks use the database connection of the Airflow worker: this DAG works with Airflow 2.x only. No archive table is kept.

airflow_logs_cleanup DAG

The DAG deletes, in the Airflow log directory (base_log_folder):

  • the execution log folders (dag_id=<id>/run_id=<run>) of each DAG in scope, according to RUNS_LOGS_KEEP, RUNS_LOGS_DAYS and RUNS_LOGS_RULE. Age and rank are those of the files (last write); the database is not used. A folder modified during the last RUNS_LOGS_GRACE_DAYS days is never deleted;

  • optionally, the log folders of DAGs that no longer exist (DELETED_DAGS_LOGS);

  • the files of the scheduler/ folder older than SCHEDULER_LOGS_DAYS days;

  • empty folders, older than one day;

  • as a safety net, whatever the scope, any log file older than MAX_AGE_DAYS days.

Variable (prefix OPS_LOGS_CLEANUP_) Default Description

ENABLED

false

true: the DAG is created unpaused. No effect if the DAG already exists

APPLY

false

true: actually deletes. false: simulation only

SCHEDULE

0 2 * * *

Schedule (cron expression), by default every day at 02:00. none: DAG to be run manually

SCOPE

maestro

Scope: maestro or all

SCOPE_MARKER

/maestro/maestro_

maestro scope: text that the DAG file path must contain

RUNS_LOGS_KEEP

100

Maximum number of runs whose logs are kept per DAG (the most recent ones)

RUNS_LOGS_DAYS

548

Maximum age of the execution logs, in days (18 months)

RUNS_LOGS_RULE

cap

cap: deleted as soon as one of the two limits is exceeded. protect: only if both are

RUNS_LOGS_GRACE_DAYS

1

A run folder modified during the last N days is never deleted

RUNS_LOGS_MAX

5000

Maximum number of run folders deleted per DAG run

DELETED_DAGS_LOGS

false

true: also deletes the log folders of DAGs that no longer exist

DELETED_DAGS_MIN_AGE_DAYS

7

DELETED_DAGS_LOGS: the DAG must have been inactive, and nothing written in its folder, for at least N days

DELETED_DAGS_MAX

500

DELETED_DAGS_LOGS: maximum number of DAG folders deleted per run

MAX_AGE_DAYS

730

Safety net: any log file older than N days is deleted, for all DAGs, known to the database or not (730 days, i.e. 2 years; 0: disabled)

SCHEDULER_LOGS_DAYS

7

Files of the scheduler/ folder: maximum age, in days

Log retention is independent of the run history retention (database). With the maestro scope, the deletion of the logs of a DAG deleted from the database can no longer be linked to Maestro: these logs are deleted by MAX_AGE_DAYS, or by DELETED_DAGS_LOGS with the all scope. To avoid any deletion by mistake, the DAG refuses to run if the database knows none of the DAGs present in the log directory (wrong or empty database). The log directory seen by the worker is that of its host: it is shared by the Airflow services of the same host.

If a handwritten log cleanup DAG already exists in the DAG directory, remove it so that the two do not run at the same time.

Configuration example

Variables to add to the Upflower environment file, to enable both DAGs with actual deletions:

# Airflow database cleanup
OPS_DB_CLEANUP_ENABLED=true
OPS_DB_CLEANUP_APPLY=true
OPS_DB_CLEANUP_KEEP_RUNS=500
OPS_DB_CLEANUP_RUNS_DAYS=548

# Log file cleanup
OPS_LOGS_CLEANUP_ENABLED=true
OPS_LOGS_CLEANUP_APPLY=true
OPS_LOGS_CLEANUP_SCHEDULE=0 2 * * *

Docker Proxy module - Image maestro/docker-proxy

This image (HAProxy 2.6) exposes the node’s Docker API with limited permissions. It allows the Airflow worker to launch task containers without direct access to the Docker socket.

Image description

Description

Filters calls to the Docker API: only the call families allowed by the variables below are forwarded to the Docker socket.

Size

~25MB

Technical components

HAProxy 2.6 (alpine)

User

haproxy (uid 99, gid 99) by default in the image. The provided deployments run the service as root (0:0) so it can read the Docker socket.

Volumes
Path Purpose Required

/var/run/docker.sock

Host Docker socket, mounted read-only

Yes

URLs
Port URL Purpose

2375

/

Filtered Docker API. Do not expose outside the internal network: this is the backend’s DOCKER_URI address (tcp://socket-proxy:2375).

This service makes it possible to create containers on the cluster. It must be attached only to the internal Datachain network, never published, and placed on a manager node.

Environment variables

Each variable allows (1) or forbids (0) a family of Docker API calls. By default, the image only allows read access: EVENTS, PING and VERSION. The provided deployment enables the families required to launch containers.

Variable Image default Provided deployment Description

POST

0

1

Allows calls that modify state (without it, the API is read-only)

CONTAINERS

0

1

Container management

IMAGES

0

1

Image management

NETWORKS

0

1

Network management

SERVICES, TASKS, SWARM, NODES, CONFIGS

0

1

Management of Swarm services and cluster state

VOLUMES

0

1

Volume management

EXEC, BUILD, COMMIT, DISTRIBUTION, PLUGINS, SESSION, SYSTEM, INFO, AUTH

0

1

Other families of Docker API calls

SECRETS, GRPC

0

0

Unchanged: calls forbidden

ALLOW_RESTARTS

0

0

Allows stopping, restarting and destroying containers, even when POST is 0

PING, VERSION

1

1

Required for the Docker client version negotiation

EVENTS

1

1

Docker event stream

SOCKET_PATH

/var/run/docker.sock

unix@/var/run/docker.sock

Socket path. The unix@ prefix is required for HAProxy to treat it as a Unix socket.

LOG_LEVEL

info

info

HAProxy log level

Since the group owning the Docker socket varies from machine to machine, the provided deployments run this service as root (user: "0:0") rather than assuming a fixed group.

Pipeline execution images

Image maestro/maestro-python3.12-airflow-dc

Image in which the Datachain tasks of a pipeline run (triggering an exposure, an export, etc.). It is designated by the backend’s DC_AFJOB_IMAGE variable.

Description

Runs the Maestro task scripts with the Datachain Python client.

Size

~215MB

Technical components

Python 3.12 / datachain Python client

User

dc (uid 38330, primary group root (gid 0), home directory /home/maestro). The image also accepts any arbitrary uid imposed at startup (OpenShift): this uid belongs to the root group (gid 0).

Configuration

No configuration to provide: Maestro passes the required variables to each task container (DC_HOST, DC_PORT, DC_PROTOCOL, DC_TOKEN, DCM_HOST_URL, etc.). The container must be able to reach the Datachain instance and the Maestro API (MAESTRO_PRIVATE_API_URL).

Image maestro/maestro-notebook

Image in which the notebooks of the DCCode integration run. It is designated by the backend’s DCCODE_RUN_IMAGE variable (and K8S_DCCODE_IMAGE for Kubernetes).

Description

Prepares the notebook’s Python environment (uv sync), then runs it with Papermill, passing it its parameters.

Size

~160MB

Technical components

Python 3.12 / uv / Papermill

User

dc (uid 38330, primary group root (gid 0), home directory /home/maestro). The image also accepts any arbitrary uid imposed at startup (OpenShift): this uid belongs to the root group (gid 0).

Configuration

No configuration to provide: the working directory (/home/maestro/dccodenb), the notebook and its parameters are prepared and mounted by Maestro at each run.

Dernière mise à jour le 2 novembre 2026 à 15:16