Guide: Building a Pipeline with Workflow
|
OnlyDC-Maestro userscan create and manage Schedules. Only the Owner of the Schedule and contributors can modify and execute it. |
Introduction
The Workflow makes it easy to visualize and organize the Pipeline of a Schedule.
The execution order of the tasks depends on their scheduling in the Workflow.
To learn more, consult the page dedicated to the execution of Schedules.
Each Schedule is associated with a unique DataChain Project, defined when the Schedule is created.
The set of elements available for DataChain tasks all come from this Project.
Select a Schedule
From the list of Schedules, click on the line of the Schedule you wish to modify.
To quickly filter the list and display only your Schedules, select 
at the top right of the list.
You can also search by keywords (tag, description, project) and filter by creation or modification date.
Add and set up tasks
The Workflow displays the last known status of tasks.
If the Schedule has just been created and has never been executed, it contains only one task, and its status is 
.
To learn more, see the paragraph on tasks execution status.
Add Task
To add a task, click on the icon 
of the Workflow menu on the right and click on the task you want to add.
The task appears grayed out in the Workflow: it must be configured before being saved and executed.
| As long as you do not leave the page, non-parameterized tasks remain visible in the Workflow. |
Set up a task
Click on a task in the Workflow to select it and display the parameters tab using the icon 
.
We advise you to take the time to modify the label of each task because this information is visible in the Workflow tasks.
The label must allow you to quickly identify what type of task it is, we recommend using a short code.
Add a description to provide details if needed.
Click 
to display the list of available DataChain elements.
This list contains only items
-
available for this task (For example DataBlocks for an Exposing a DataBlock task)
-
accessible in the selected Project (defined during the Planning initialization step)
-
on which you have sufficient DataChain rights and permissions (global and Project permissions, item rights)
| Some tasks, such as updating a Publication, expect 2 parameters: a dashboard, then the Publication to update. |
In the example above, the letters correspond to the type of element (EM for Business Entity and DB for DataBlock) followed by the identifier of the element.
- WARNING
-
Until you click
, the Schedule Pipeline will not be updated.
This allows working on the Workflow while an automatic Schedule continues to run automatically in the latest updated stable version.
Build a Pipeline
Repeat the operation to add and configure as many tasks as desired.
Link tasks together to guide the order of execution.
The Workflow is automatically organized each time you reload the page: this allows all Schedules to share the same organizational logic.
You can click 
to organize the Pipeline as you go.
To delete a task or a link, click on the component from the Workflow then on the cross.
|
When the Pipeline is edited from the Workflow, the working mode behaves like a draft. This means that an automatic Schedule continues to launch executions for known tasks in the registered version. The modifications will be taken into account in the next execution. |
| Tasks are automatically organized in the Workflow each time you enter the page. |
To avoid executing a pipeline while it is being created, you can disable scheduling from the settings tab: in this case, it is no longer possible to activate scheduling from the list, and automatic execution is paused.
Set execution order
Tasks that are not connected to any other tasks run in random order and regardless of the success or failure other tasks execution.
Connected tasks run from left to right.
If a task fails the next one will not be executed, its status will be "blocked".
If a task is in success, the next one will be executed and so on.
To delete a link, click on it then on the dedicated cross.