DC-Maestro Documentation
Who is DC-Maestro for?
DC-Maestro is intended in particular for all DataChain users who are members of a Project and who wish to orchestrate and automate the execution of DataChain tasks.
What is its role in data processing?
DC-Maestro allows you to move data processing into production and to be part of a DataOps approach: the data used and the processing performed are updated automatically at a chosen frequency.
How does DC-Maestro work?
DC-Maestro allows you to reference one or more DataChain instances (sharing the same SSO) and to create Schedules.
A Schedule is mostly made of a Pipeline which is a collection of tasks.
Different tasks are used to schedule processes to be performed on DataChain datasets, such as a DataBlock Persistence.
Pipeline tasks can be scheduled from a Workflow and execute automatically at a defined frequency.
Each task execution generates a status and execution log that allows the tasks to be tracked and properly executed.
Main features of DC-Maestro
Orchestration
Manage and orchestrate complex pipelines intuitively from the Workflow view.
Select a DataChain Project and then add and connect tasks to run. Order the tasks to run in the order you define from the Workflow view.
Automation
Define the frequency of your Schedules execution according to your needs: every day at a specific time, once a month, every hour…
Execute Schedules when needed, beyond the set frequency, and halt the automation at your discretion.