The same pattern for Prefect 2.0
Note about pros and cons of this approach
Pros
- higher isolation for dependency management
Cons
Running each task in a separate container may be less beneficial than it may initially seem to be:
- it introduces a lot of overhead in the build process
- overhead at runtime,
- more storage requirements and thus costs,
- makes it harder to pass data between tasks/processes
Conclusion
Running everything in Docker is not a silver bullet, so make sure to carefully evaluate your use case and consider the trade-offs mentioned above