Guide: Building a Pipeline with Workflow
|
Only DC-Maestro User can create and manage Schedules. Only the Owner and Contributors of the Schedule can modify and execute it. |
Introduction
The Workflow makes it easy to visualize and organize the Pipeline of a Schedule.
The execution order of the tasks depends on their scheduling in the Workflow.
To learn more, consult the page dedicated to the execution of Schedules.
Each Schedule is associated with a unique DataChain Project, defined when the Schedule is created.
The set of elements available for DataChain tasks all come from this Project.
Select a Schedule
From the list of Schedules, click on the line of the Schedule you wish to modify.
To quickly filter the list and display only your Schedules, select 
at the top right of the list.
You can also search by keywords (tag, description, project) and filter by creation or modification date.
Add and set up tasks
The Workflow displays the last known status of tasks.
If the Schedule has just been created and has never been executed, it contains no task.
To learn more, see the paragraph on tasks execution status.
You can create a custom task pipeline or use the automatic calculation function to automatically generate an optimized workflow.
| Export tasks must be created and set up manually +. Remember to save them before launching an automatic calculation so that they are taken into account. |
Add Task
To add a task, click on the icon Add 
of the Workflow menu on the right and click on the task you want to add.
The task appears grayed out in the Workflow: it must be configured before being saved and executed.
| As long as you do not leave the page, non-parameterized tasks remain visible in the Workflow. |
Set up a task
Click on a task in the Workflow to select it and display parameters 
.
We recommend that you take the time to modify the label of each task, as this information is visible in the workflow tasks.
The label should make it easy to identify the type of task involved. We recommend using a short code.
You can also copy the label of a linked item and add a description to add details if required.
Click on Select to display the list of available DataChain elements.
This list contains only items
-
available for this task (For example DataBlocks for an Exposing a DataBlock task)
-
accessible in the selected Project (defined during the Planning initialization step)
-
on which you have sufficient DataChain rights and permissions (global and Project permissions, item rights)
In the example above, the letters correspond to the type of element (EM for Business Entity and DB for DataBlock) followed by the identifier of the element.
| Some tasks, such as updating a Publication, expect 2 parameters: a dashboard, then the Publication to update. |
Build a Pipeline
Repeat the operation to add and set up as many tasks as required.
Link tasks together to direct the order of execution.
You can click on 
to organise the Task Pipeline as you go if required.
To delete a task or link, click on the task and then on the icon 
.
Adding and deleting tasks and links video::task-linking.mp4[width=840]
|
When the Pipeline is modified from the Workflow, the work mode behaves like a draft +.
This means that Automatic Scheduling continues to launch executions for known tasks in the last saved version sent to the orchestrator. When you save, the changes are taken into account in the next run and do not affect a current run. |
| To avoid executing a pipeline while it is being created, you can disable scheduling from the parameters tab: in this case, it is no longer possible to activate scheduling from the list and automatic executions are paused. |
Set execution order
Tasks that are not connected to any other tasks run in random order and regardless of the success or failure other tasks execution.
Connected tasks run from left to right.
If a task fails the next one will not be executed, its status will be "blocked".
If a task is in success, the next one will be executed and so on.
To delete a link, click on it and then on the icon .
| If the Workflow has been generated automatically, it is important not to modify the optimal task execution order that has been calculated. |
