I wonder if Prefect could be used to queue up the same automatic and manual pipelines with different parameters and distribute this execution queue to different machines for processing

Archive
1

I am new in the field and I am trying to design an architecture to improve our current system. At first I have two cases:

  1. Several ETL pipelines that will be triggered periodically at specified times.

  2. ETL pipelines that will be triggered manually by users with custom parameters.

The user-triggered pipelines would often be the same as the automatic ones. I wonder if Prefect could be used to queue up the same automatic and manual pipelines with different parameters and distribute this execution queue to different machines for processing. Is there a better tool to solve this problem? Below is a diagram of the problem.

TL;DR: I want to use Prefect to create Tasks inside each ETL, creating a flow, then queue these Flows executions into something that will be responsible for distributing the work to multiple nodes.

[Album] imgur.com