Frequently asked questions

One or more PostgreSQL images no longer boot

When the Datachain stack is shut down (often unexpectedly), the database may no longer start, with the following error log:

2023-03-24 11:12:35.314 CET [24] LOG: database system stopped at 2023-03-17 06:04:58 CET
2023-03-24 11:12:35.314 CET [24] LOG: invalid resource manager identifier in primary checkpoint record
2023-03-24 11:12:35.314 CET [24] PANIC: could not locate a valid checkpoint record

In this case, proceed as follows:

  • on the machine where the PostgreSQL is located, identify the PostgreSQL data directory:

    • either it’s a volume name mounted on the machine and is in the docker-compose deployment file

    • or it’s a docker volume, and is located in /var/lib/docker/volumes/NOM_DE_STACK_NOM_DU_VOLUME/_data (stack name from startup (up.sh?), and volume name the volume in the docker-compose file (potentially pg_data). The 'docker volume ls' command will validate that this is the correct volume name. Example on a file:

version: "3.7
services:
  dc_pg:
    image: dc/pg:dev
    networks:
      dc_network:
        aliases:
          - dc-pg
    deploy:
      mode: replicated
      replicas: 1
    volumes:
      - pg_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=adobis
...

If the stack name is datachain, then the volume will be datachain_pg_data, and the path /var/lib/docker/volumes/datachain_pg_data/_data.

  • Once the path has been validated, stop the stack.

  • On the target postgres machine, run the following command.

docker run -it -u postgres -v /var/lib/docker/volumes/datachain_pg_data/_data/:/var/lib/postgresql/data ${REGISTRY}/dc/pg:7.6.0 pg_resetwal /var/lib/postgresql/data/

Note: you could use the standard postgres 12 image, which would also work. * Restart the stack.

I want to remove the passwords from the docker-compose.yml deployment files. How can I do this?

In swarm, all datachain passwords can be stored directly at swarm cluster level in "docker secret".

Datachain environment variables containing passwords usually have a second variable suffixed with _FILE. The procedure for storing a password is as follows:

  • create the secret in docker from the Swarm manager

printf "my super secret password" | docker secret create my_secret -

  • use this secret in the deployment file (or .en file, depending on your preference) and replace the environment variable with the suffixed variable.

version: "3.7"
services:
...
  dc_pg_migration:
    image: dc/pg_migration:dev
    networks:
      dc_network:
        aliases:
          - dc-pg-migration
    environment:
      - PG_HOST=dc-pg
      - PG_PASSWORD=12345
    deploy:
      restart_policy:
        condition: none
    volumes:
      - pg_dump:/data/pg_dump

becomes

version: "3.7
services:
...
  dc_pg_migration:
    image: dc/pg_migration:dev
    networks:
      dc_network:
        aliases:
          - dc-pg-migration
    environment:
      - PG_HOST=dc-pg
      - PG_PASSWORD_FILE=/run/secrets/pg_password
      - PG_EXPOSE_PASSWORD_FILE=/run/secrets/pg_password
      - PG_KC_PASSWORD_FILE=/run/secrets/pg_password
    deploy:
      restart_policy:
        condition: none
    volumes:
      - pg_dump:/data/pg_dump
    secrets:
      - pg_password
...

secrets:
  pg_password:
    external: true