Task-specific environment/memory requirements

[sbsd]

Is it possible to define a flow where each task has it’s own environment and memory specifications? For example, task1 uses containerA with 16 workers and 16Gb RAM, task2 uses containerB with 1 worker and 1Gb RAM).
Thanks!

[anna_geller]

@sbsd

each task has it’s own environment and memory specifications

Welcome to Discourse @sbsd!

Yes, it’s definitely possible. Technically, such a task would involve calling a subflow (or child flow), and Prefect would orchestrate those subflows as if they would belong to a single (parent) flow.

Check out this topic for details on how you can use that with a Docker or Kubernetes agent:

Can Prefect run each task in a different Docker container?

Packaging code dependencies can be challenging, especially if some of your tasks require different Python library versions than other tasks. There are two patterns supported by Prefect to separate code dependencies on a per task level: 1. Flow-of-flows orchestrator pattern Prefect allows you to manage dependencies not only between tasks in a single data pipeline but also to manage dependencies between individual data pipelines. Prefect can provide this high degree of flexibility with respect…

LMK if you have any specific questions about it.

[sbsd]

Thanks @anna_geller, those pages are quite helpful (I should have seen those before posting). I will give those approaches a shot and drop a line here if I run into any walls - thanks again!