Task-specific environment/memory requirements

[anna_geller]

@sbsd

each task has it’s own environment and memory specifications

Welcome to Discourse @sbsd!

Yes, it’s definitely possible. Technically, such a task would involve calling a subflow (or child flow), and Prefect would orchestrate those subflows as if they would belong to a single (parent) flow.

Check out this topic for details on how you can use that with a Docker or Kubernetes agent:

Can Prefect run each task in a different Docker container?

Packaging code dependencies can be challenging, especially if some of your tasks require different Python library versions than other tasks. There are two patterns supported by Prefect to separate code dependencies on a per task level: 1. Flow-of-flows orchestrator pattern Prefect allows you to manage dependencies not only between tasks in a single data pipeline but also to manage dependencies between individual data pipelines. Prefect can provide this high degree of flexibility with respect…

LMK if you have any specific questions about it.