How to set up prefect server and local agent to make flow of flows

Archive
, ,
1

Hello,

I am running workflows on a self hosted infrastructure, with the default prefect server (the one launched running prefect server start) and a separate local agent in version 1.2.2.
It all works well, until I have to create a flow of flows, then each task creating a flow fails with the following error:

Task 'create_flow_run': Exception encountered during task execution!
Traceback (most recent call last):
  File "/home/debian/.local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/home/debian/.local/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/home/cloud/.local/lib/python3.9/site-packages/prefect/tasks/prefect/flow_run.py", line 167, in create_flow_run
  File "/home/debian/.local/lib/python3.9/site-packages/prefect/client/client.py", line 1000, in get_cloud_url
    tenant_slug = self.get_default_tenant_slug()
  File "/home/debian/.local/lib/python3.9/site-packages/prefect/client/client.py", line 1027, in get_default_tenant_slug
    if tenant.id == self.tenant_id:
  File "/home/debian/.local/lib/python3.9/site-packages/prefect/client/client.py", line 262, in tenant_id
    self._tenant_id = self._get_auth_tenant()
  File "/home/debian/.local/lib/python3.9/site-packages/prefect/client/client.py", line 153, in _get_auth_tenant
    raise ValueError("You have not set an API key for authentication.")
ValueError: You have not set an API key for authentication.

The weird thing is that the children flows are created and ran without issues, even with that error.
But I would rather have the parent flow reporting a success.
What do I need to do to make it work?

Best regards,
Nicolas

2

The API key is requested by the client when your agent assumes you are running flows with the Prefect Cloud backend. To switch to Server, either run:

prefect backend server

before starting the agent, or set the env variable on your agent:

export PREFECT__BACKEND="server"

More on that here:

3

I did have this line in the backend.toml file of the machine hosting the prefect agent.
Do I need to put it on the machine hosting the prefect server, and on the ones hosting the dask clusters?
(I am using a remote dask cluster, so all the tasks including the creation of the the subflow are run on those dask workers)

4

This is definitely missing on the machine from which you are running your agent/execution layer

5

I confirm this was the issue. Adding the prefect config.toml and backend.toml files on the dask workers solved it.
Thanks a lot!

1 Like