Prefect deployments often have critical, implicit dependencies on files and build artifacts, such as containers, that are created and stored outside of Prefect.
Each of these dependencies is a potential stumbling block when deploying a flow - you must ensure they’re satisfied for your flow to run successfully. In this release, we’re introducing two new beta features, workers and projects, to help you better manage your flow deployment process.
Additionally, we’re releasing variables for centralized management of management and expanding events and automations to include blocks. There are a lot of highlighted features this week — but we’ve also made some significant performance improvements alongside a slew of bug fixes and enhancements!
Workers are next-generation agents, designed from the ground up to interact with work pools. Each worker manages flow run infrastructure of a specific type and must pull from a work pool with a matching type.
Existing work pools are all “agent” typed for backwards compatibility with our agents — but new work pools can be assigned a specific infrastructure type. Specifying a type for a work pool simplifies choosing what kind of infrastructure will be used when creating a flow run.
Work pools expose rich configuration of their infrastructure. Every work pool type has a base configuration with sensible defaults such that you can begin executing work with just a single command. The infrastructure configuration is fully customizable from the Prefect UI.
For example, you can now customize the entire payload used to run flows on Kubernetes — you are not limited to the fields Prefect exposes in its SDK. We provide templating to inject runtime information and common settings into infrastructure creation payloads. Advanced users can add custom template variables which are then exposed the same as Prefect’s default options in an easy to use UI.
If the work pool’s configuration is updated, all workers automatically begin using the new settings — you no longer need to redeploy your agents to change infrastructure settings. For advanced use cases, you can override settings on a per-deployment basis.
This release includes Process, Kubernetes, and Docker worker types. Additional worker types will be included in subsequent releases.
Creating a Kubernetes work pool:
Adding a new variable to the advanced work pool configuration will expose it in the basic config:
See the updated work pool, workers, & agents concepts documentation for more information.
A project is a directory of files that define one or more flows, deployments, Python packages, or any other dependencies that your flow code needs to run. If you’ve been using Prefect, or working on any non-trivial Python project, you probably have an organized structure like this already.
Prefect projects are minimally opinionated, so they can work with the structure you already have in place and with the containerization, version control, and build automation tools that you know and love. With projects as directories, you can make relative references between files while retaining portability. We expect most projects to map directly to a git repository. In fact, projects offer a first-class way to clone a git repository so they can be easily shared and synced.
Projects also include a lightweight build system that you can use to define the process for deploying flows in that project. That procedure is specified in a new
prefect.yaml file, in which you can specify steps to build the necessary artifacts for a project’s deployments, push those artifacts, and retrieve them at runtime.
Projects are a contract between you and a worker, specifying what you do when you create a deployment, and what the worker will do before it kicks off that deployment. Together, projects and workers bridge your development environment, where your flow code is written, and your execution environment, where your flow code runs. Create your first Prefect project by following this tutorial.
See the new project concept doc for more information.
Variables enable you to store and reuse non-sensitive bits of data, such as configuration information. Variables are named, mutable string values, much like environment variables. They are scoped to a Prefect Server instance or a single workspace in Prefect Cloud. Variables can be created or modified at any time.
While variable values are most commonly loaded during flow runtime, they can be loaded in other contexts, at any time, such that they can be used to pass configuration information to Prefect configuration files, such as project steps. You can access any variable via the Python SDK via the
from prefect import variables # from a synchronous context answer = variables.get('the_answer') print(answer) # 42 # from an asynchronous context answer = await variables.get('the_answer') print(answer) # 42 # without a default value answer = variables.get('not_the_answer') print(answer) # None # with a default value answer = variables.get('not_the_answer', default='42') print(answer) # 42
See the new variables concept doc for more information or the pull request for implementation details.
Continuing the rollout of events as the primary unit of observability in Prefect Cloud, Prefect will now emit events for all block method calls by default.
These events can be viewed in the Event feed, allowing you to analyze the interactions your flows and tasks have with external systems such as storage locations, notification services, and infrastructure. Additionally, you can trigger automations based on these events. For example, you can create an automation that is triggered when a file is uploaded to a storage location.
We’re releasing many new features every week and we know not everyone is on the latest version of Prefect. We’ve added versioning to our documentation website to make it easier to find the docs for the version of Prefect that you’re using.
Now, when you visit the Prefect documentation site, you’ll see a version selector at the top of the page.
There’s one breaking change to be aware of in this release:
- Unused options for sorting logs have been removed from the API — #7873
Thanks so much to the external contributors to this release!
- @mianos made their first contribution in #9077!
- @dominictarro made their first contribution in #8965!
Enhancements and Fixes
The list of enhancements and fixes this week is long, so check out the 2.10.0 release notes to see them all! We added a quick 2.10.1 release with a fix for projects, but 2.10.0 is the place to read about all the changes.