Configuration - Data management

This page describes the data management components of the Datachain solution and how to configure them: the PostgreSQL databases (main image and authentication database) and the Valkey cache and queue service.

Images can be configured via docker *.env files or environment variables.

Datachain PostgreSQL module - Image dc/dc_pg:18-trixie

Custom PostgreSQL image based on postgres:18, with business database initialization, health check, and lifecycle management scripts.

Image description

Description

Custom PostgreSQL image (based on postgres:18), which creates the Datachain suite databases on first startup. The databases created are selected with the DATABASE_MODE environment variable: all by default (all databases), or one or more comma-separated values, for example core,expose.

DATABASE_MODE value Database created Content

core

adobis

Datachain Core: the configuration of the processing pipelines (developed via the Datachain interface), as well as the settings, the executed tasks and the tasks currently running

expose

expose

The data exposed to consumers

marketplace

dc_marketplace

Datachain Marketplace

dc_console

dc_console

Datachain administration console

keycloak

keycloak

The Keycloak SSO data

dc_maestro

dc_maestro

Dc-Maestro

airflow

airflow

Airflow, used by Dc-Maestro

dc_code

dc_code

DC Code

The PostgreSQL unaccent extension is enabled on the adobis, expose, dc_marketplace and dc_console databases.

Several instances of the image can be deployed, each dedicated to certain databases through DATABASE_MODE (for example one instance for the expose database and another for the keycloak database).

Size

~485MB

Technical components

Native PostgreSQL database

User

The container starts as root to prepare the data directory (it gives ownership of it to postgres), then PostgreSQL runs as postgres (uid 999, gid 999). This rule also applies to the dc/pg_keycloak image.

Volumes
Path Purpose Required

/var/lib/postgresql

Data storage (PGDATA data directory: /var/lib/postgresql/18/docker). Mounting the volume on /var/lib/postgresql allows using pg_upgrade --link between major PostgreSQL versions

yes

Logs

Module logs on the docker standard output

URLs
Port URL Purpose

5432

/

Data access from the other services.

Environment variables

Variable Default Description

DATABASE_MODE

all

Databases to create: all, or one or more values among core, marketplace, dc_console, keycloak, expose, dc_maestro, airflow, dc_code, separated by commas

POSTGRES_PASSWORD

PostgreSQL superuser password (required if POSTGRES_PASSWORD_FILE is not defined)

POSTGRES_PASSWORD_FILE

Path to a file containing the password, alternative to POSTGRES_PASSWORD (Docker secrets)

POSTGRES_USER

postgres

Superuser

POSTGRES_DB

postgres

Default database

PGDATA

/var/lib/postgresql/18/docker

Data directory, versioned since PG18 (/var/lib/postgresql/<version>/docker). Mount the volume on /var/lib/postgresql to allow pg_upgrade --link between major versions

PG_RESET_WAL_ON_START

false

Reset WAL at startup if an abrupt shutdown is detected. Set to true to enable, or manage via a K8s initContainer

PG_GRACEFUL_STOP

true

Clean shutdown via pg_ctl on SIGTERM. Set to false if managed by a K8s preStop hook

Connections

Variable Default PG parameter Description

PG_MAX_CONNECTIONS

200

max_connections

Maximum number of simultaneous connections

Memory

Variable Default PG parameter Description

PG_SHARED_BUFFERS

128MB

shared_buffers

Memory allocated to the PostgreSQL shared cache. Recommended: ~25% of available RAM

PG_WORK_MEM

4MB

work_mem

Memory per sort or hash operation. Warning: consumed per connection × operation

PG_MAINTENANCE_WORK_MEM

64MB

maintenance_work_mem

Memory for maintenance operations (VACUUM, CREATE INDEX, etc.)

PG_EFFECTIVE_CACHE_SIZE

4GB

effective_cache_size

Estimate of the memory available for the OS + PG cache. Hint for the planner, recommended: ~75% of RAM

WAL

Variable Default PG parameter Description

PG_MAX_WAL_SIZE

1GB

max_wal_size

Maximum size of WAL files before a checkpoint is triggered

PG_WAL_LEVEL

replica

wal_level

WAL detail level: minimal, replica, logical

Logging

Variable Default PG parameter Description

PG_LOG_MIN_DURATION

-1

log_min_duration_statement

Minimum duration (ms) above which a query is logged. -1 disables, 0 logs everything

Parallelism

Variable Default PG parameter Description

PG_MAX_PARALLEL_WORKERS

8

max_parallel_workers

Maximum number of parallel workers. Reduce in CPU-constrained environments

Maintenance (scheduled VACUUM FULL)

Variable Default PG parameter Description

PG_VACUUM_FULL_SCHEDULE

empty

pg_cron

Cron schedule (e.g. 0 3 * * 0) of a VACUUM FULL on each existing database. Empty: disabled, and the vacuum_full_* jobs already scheduled are removed at startup. VACUUM FULL locks each table while it is being rewritten and requires free disk space

PG_VACUUM_FULL_STAGGER_MIN

10

—

Minutes between the VACUUM FULL runs of two databases (alphabetical order: the 2nd starts at +N, the 3rd at +2N…). 0: all databases at the same time. Requires a schedule with numeric minute and hour, otherwise the offset is ignored

Authentication PostgreSQL module - Image dc/pg_keycloak

Can be replaced by the dc/pg_all image.

No Adobis-specific configuration. See https://hub.docker.com/_/postgres

The description of this image is identical to that of the other PostgreSQL images: see Image description.

Cache and queues - Valkey (formerly Redis)

Image description

Description

Standard Valkey image (formerly Redis): ensures asynchronous message exchange between the modules

Size

~44MB

Technical components

Native Valkey application

User

The container starts as root to prepare the data directory (it gives ownership of it to valkey), then Valkey runs as valkey (uid 999, gid 1000).

Volumes

Only the potential standard Valkey volumes

Logs

Module logs on the docker standard output

URLs
Port URL Purpose

6379

/

Queue access for the other services.

Configuration

No Adobis-specific configuration. See the Valkey configuration documentation.

Since version 9.1.0, the Redis service is replaced by Valkey (image valkey/valkey:9.0-alpine). See the technical notes of version 9.1.0 for the migration procedure.

Dernière mise à jour le 2 novembre 2026 à 15:16