How do you recommend approaching CI pipelines with Docker storage in a monorepo?

[anna_geller]

View in #prefect-community on Slack

@Alvaro_Durán_Tovar: How do you recommend approaching CI pipelines with docker storage? So I think we might end up having N flows in python scripts, using docker as storage deploying to kubernetes. I guess we will run all flows on every release, that means creating a new docker image for each of the flows and registering them again.

Can be ok, I mean, I not going to do it manually… but seems a bit overkill to rebuild and upgrade all flows if only 1 has changed. Any recommendation on how to manage flows through CI pipelines if we have all flows on the same repo?

@Kevin_Kho: Hi @Alvaro_Durán_Tovar, have you seen this page?

Multi Flow Storage | Prefect Docs

This discussion thread might also have some info about this

GitHub: Flow deployment patterns · Discussion #4042 · PrefectHQ/prefect

@Alvaro_Durán_Tovar: nice discussion, thanks!

@Billy_McMonagle: Hi Alvaro, my post is one of those in the discussion Kevin posted, but I can elaborate slightly bc we’re running in production now… I am using custom docker images for each of my flow groups. For performance, I rely heavily on some of the caching features available in buildkit, and build my flows with a makefile that runs multiple builds in parallel. Right now I have 1 base image and then 10 different flow images that extend it. Typical build time is around 4 minutes. Like you, I’m also interested in ways to selectively rebuild individual images, but I am hoping to put off fully optimizing this for a while… by making effective use of build caching, it really doesn’t matter if everything is getting rebuilt.

@Anna_Geller: @Alvaro_Durán_Tovar I think the problem is not specific to Prefect, but in general building CI/CD pipelines in monorepos is a bit more challenging. You could have a look at this blog post from AWS showing how you could e.g. use GitHub event payload to determine what a CI pipeline should do e.g. triggering only build for Project A and building a Docker image only for that flow: https://aws.amazon.com/blogs/devops/integrate-github-monorepo-with-aws-codepipeline-to-run-project-specific-ci-cd-pipelines/.

Based on this, it seems that many CI tools have option to filter based on path in the repository.

Amazon Web Services: Integrate GitHub monorepo with AWS CodePipeline to run project-specific CI/CD pipelines | Amazon Web Services

DevOps Stack Exchange: Git-based CI/CD Pipeline for Monorepo