Prefect 1.0 -> Prefect 2.0 User Configurations to Configuration Profiles

Archive
, , ,
1

Summary

Prefect installations house a number of configuration settings that can be set within a users environment that affect how, when, and where prefect runs and deploys flows. in Prefect 1.0 if a user wanted to adjust these settings this could be specified in a user configuration file or set as environment variables.

In Prefect 2.0 these settings are persisted in configuration profiles of which a user can create multiple to store configurations for multiple environments and can easily switch between them.

Video link

Audience

Profiles are useful for users that need/want to have different configuration settings applied for different environments and want to be able to easily switch between these configurations.

What is Staying the Same

Using multiple profiles is not a requirement for interacting with prefect. Users are able to set configuration settings for a default profile in the profiles.toml file, similar to the config.toml created for Prefect 1.0. Users can also they can continue to set environment variables as necessary for their individual flows.

What is Changing

Profiles include 2 Major changes:

  1. As mentioned previously Users are able to persist multiple profiles that house individualized configuration settings
  2. Users are able to interact with configuration settings and profiles via the CLI allowing them to set and remove individual configurations for the current profile as well as create new ones

How to Convert from 1.0 to 2.0

For the most part users won’t need to adjust many of the default settings available, for a full list you can run this command in the CLI

# Displays a full list of configuration settings for the current profile
prefect config view --show-defaults

That said it can be extremely useful for users interacting with multiple environments through Prefect Cloud or Orion.

The following example creates 2 profiles:

  • One for interacting with prefect cloud
  • One for interacting with a local instance of Orion.
#  Creates a new profile democonfig
prefect profile create democonfig

# Sets democonfig to the active profile
prefect profile use democonfig

# sets the prefect api url to the workspace within an account
prefect config set PREFECT_API_URL="https://api.prefect.cloud/api/accounts/xxxxxxxxxxxxxxxxxxxxxxxxxxxx/workspaces/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

# creates new profile demo2orion
prefect profile create demo2orion 

# sets the PREFECT_API_URL to the localhost for the demo2orion profile
# The -p flag specifies the profile to update even though democonfig is the active profile
 prefect -p demo2orion config set PREFECT_API_URL="http://127.0.0.1:4200/api"

Troubleshooting

  • For prefect cloud If you don’t specify the PREFECT_API_URL and the PREFECT_API_KEY manually within your active profile then when logging into cloud via the CLI you’ll be prompted to create a new profile to store the cloud api URL. If setting the API url manually it needs to be in this format “https://api.prefect.cloud/api/accounts/xxxxxxxxxxxxxxxxxxxxxxxxxxxx/workspaces/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx” including the account id and workspace id located in the Prefect Cloud URL when you’re viewing the page through the UI.

  • Setting the Workspace via the CLI after authenticating to prefect cloud updates the URL to the api-beta url, this should still function correctly as it should reroute to the correct url but is something to be conscious of if you experience issues.

2 Likes