View in #prefect-community on Slack
@Alex_To: Hello prefect community. I am in the process of evaluating various products to replace our in-house orchestration tool. Prefect stands out from the crowd with intuitive UI and flexible workflow branching (conditional, loop, dynamic) which our tool does not do and the other contender does not do very well. (:cough: bareflow)
Our use case is slightly different: we will be using the tool for container execution orchestration in which each task simply invokes a container (within our k8s cluster or ECS) or a job (databrick job). Each container is an atomic unit of work written in any language. This architecture de-couples orchestration from the actual functional task (container) and avoid recoding of hundreds of existing tasks/containers. This has been working well for us using our internal tool.
Our flow would be simply as task1: call container-A; task2 call Databrick-job-B; task3: call container-C after task1 and task2 are completed.
My questions are:
- Based on the documentations, my best option is to run local agent on some EC2 instances with localDashExecutor. Using any other agent type would require additional resources and add more latency. e.g: With Kubernetes agent, one pod for the Flow run and another for the actual container; Latency = double spin up time. The downside is scaling problem with local agent. Do I understand it correctly? Any other approach?
- Any plan to add ECSRunTask to AWS Tasks? This is to run any arbitrary task defined outside of prefect context in ECS. Similar to Airflow ECS Operator? I am surprised it’s not already on the list.
Thanks
@Kevin_Kho: For number 1, I don’t think you need massive scale because you are just kicking off jobs right? Yes you are understanding it right though. I think the LocalAgent doesn’t need to be big in this case.
Certainly not opposed to the AWS ECSRun. Will ask our head of integrations, but can’t guarantee a timeline on that.
@Anna_Geller: @Alex_To This is not an official task (yet), but it works to run ECS task from Prefect.
• task: ecsruntask.py · GitHub
• usage example: ecsruntask_usage.py · GitHub
@Alex_To: @Kevin_Kho Thanks for confirming. You are right. No actual computing is required on the Flow Runs. However, at peak, it might have to kick off a couple of hundreds tasks. Would it present a problem then with LocalAgent?
@Anna_Geller You are amazing. Thanks for the examples.