When do I need to reregister my flow? Which changes in flow metadata result in Prefect bumping up the flow version?

Archive
, , , , , , , , , ,
1

Not all changes that you make to your flow lead to Prefect bumping up your flow version.

When do I need to re-register my flow?

Redeploying your flow, i.e. re-registering your flow, is necessary only when your metadata has changed. What counts as metadata is:

  1. Your flow structure (tasks and edges)
  2. Your storage and run configuration
  3. Your flow name or project name
  4. The Prefect Core version
  5. Your reference tasks
  6. Your serialized flow (only relevant when using the default pickle-based storage rather than script storage - for more details, see the section " Pickle vs. Script-based storage" below)

Registering via flow.register() vs. via CLI

When using the default flow.register("project_name") Prefect always bumps up the flow version, unless you use the idempotency key as shown here:

 flow.register(
    project_name="Hello, World!",
    idempotency_key=flow.serialized_hash()
)

However, if you use the Prefect register CLI, Prefect does this automatically and only reregisters if the serialized hash of the flow has changed:

prefect register --project yourporject -p yourflow.py

Can I force re-registration from the CLI even if my metadata hasn’t changed?

Yes, you can do that by adding the -f or --force flag.

prefect register --project yourporject -p yourflow.py --force

For more about flow registration via CLI, type in your terminal:

prefect register --help

Pickle vs. Script-based storage

Especially because of #6 the answer to whether you need to reregister depends on whether you use a pickle or script-based storage.

For instance, if you use script-based storage and some code within your task has changed, you don’t have to re-register your flow, because nothing in your flow structure has changed i.e. there are no new tasks or edges and the serialized flow (#6) is only relevant when using pickle-based storage.

However, if you add a new task or you change some upstream/downstream task’s dependency, then you need to reregister your flow because then your flow structure i.e. the DAG has changed and this gets stored in the backend (#1).

:bulb: Important note about executors :bangbang:

Note that the type and configuration of your flow’s executor are not stored in the Prefect backend as part of the flow metadata.

Why? Because the executor may contain sensitive information such as your Dask scheduler address which you may want to keep private.

So how does Prefect know which executor to use? Prefect retrieves that information from your flow’s storage at runtime.

More details & references