Troubleshooting Prefect Server issues - list of resources you may check

[anna_geller]

Getting started

How to deploy self-hosted Server and register first flows to the Server backend?

This documentation shows how to deploy Server to a single machine. It’s important to: Switch your backend to Server (default is Cloud): prefect backend server Include the following configuration to the ~/.prefect/config.toml on the machine from which you register your flows (e.g. your laptop) so that Prefect knows which API endpoint it should talk to. [server] endpoint = "http://YOUR_MACHINES_PUBLIC_IP:4200/graphql" [server.ui] apollo_url = "http://YOUR_MACHINES_PUBLIC_IP:4200/gra…

How to ensure I'm using Server backend?

There are several ways how you can configure that. CLI prefect backend server This will update the backend.toml file to “server” as well: [image] Note that “cloud” is the default. Using an environment variable export PREFECT__BACKEND="server" In a Kubernetes YAML manifest When deploying to Kubernetes, you can do: You need to add the following environment variables: env: - name: PREFECT__BACKEND value: server - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN …

Can’t connect to a Prefect Server from other machine despite port 4200 being open

First, check that the API is working on the Server deployment by going to the Interactive API and testing that the API is healthy. The hello endpoint can be used as follows: query { hello } If it is working, it means that you need to start the Server with the --expose flag as follows: prefect server start --expose This will allow outside connections. You can find more information in this Github issue and this documentation page.

How to set Server endpoint when deploying a Kubernetes agent?

You need to add the following environment variables: env: - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN value: '' - name: PREFECT__CLOUD__API value: "http://some_ip:4200/graphql" # paste your GraphQL Server endpoint here - name: PREFECT__BACKEND value: server

How to disable sending telemetry data from Server?

View in #prefect-community on Slack [Omar_Sultan] @Omar_Sultan: Hi Everyone, we have a prefect server running on Kubernetes, setup was done using the HELM Chart. Everything is running smoothly but occasionally we would get this error File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 341, in _raise_timeout self, url, "Read timed out. (read timeout=%s)" % timeout_value urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='prefect-apollo.prefect', port=4…

Migrations

How to run Server database migrations manually?

View in #prefect-server on Slack [Scarlett_King] @Scarlett_King: Hi guys, I’ve just redeployed prefect server to AKS via helm chart and having trouble connecting to graphql from ui. How can I troubleshoot this? [Anna_Geller] @Anna_Geller: You need to ensure that the right ports are open. Networking is a bit challenging on Azure and I’m no networking expert, but you need to ensure that: • API endpoint is reachable - the default port is 4200 and it would need to be open to allow the API to be r…

Troubleshooting Server migration to Hasura 2.0 and Prefect >= 1.1

View in #prefect-community on Slack [Henry] @Henry: Hello. We upgrade from 0.15.14 to 1.1.0 and it seems like our flows are still running but nothign shows up on the dashboard unless we pick a project [Kevin_Kho] @Kevin_Kho: You mean the main dashboard just to confirm? [Henry] @Henry: yes the main dashboard with the overview/flows/calendar tab [Kevin_Kho] @Kevin_Kho: you’re on Cloud or Server? [Henry] @Henry: server migrations seemed to run fine [Kevin_Kho] @Kevin_Kho: when you go to the…

Maintenance

How can I free up Postgres database space?

If you want to perform a clean up to free up space on your Postgres database instance, there are a couple of ways you can approach it: you can manually delete old records, especially logs from the flow run table using DELETE FROM in SQL, you can do the same in an automated fashion, e.g. some users have an actual flow that runs on schedule and purges old data from the database, alternatively, you can use the open-source pg_cron job scheduler for Postgres to schedule such DB administration tasks…

How can I delete flow runs older than 30 days using GraphQL API to clean up database space in Prefect Server?

Here is an example script you can use: import prefect from datetime import datetime, timezone client = prefect.Client() flows = client.graphql( { 'query': { 'flow': { 'id' } } } ) for flow in flows['data']['flow']: flow_runs = client.graphql( { 'query': { 'flow (where: {id: {_eq: "'+flow['id']+'"}})':{ 'flow_runs':{ 'id', 'name', …

Timeout

How to check a GraphQL query timeout settings on Server? How to increase that timeout value?

How to increase the default timeout? You can set that on your config.toml: How to check the current value? Check the value in the config.toml file or via: prefect.context.config.cloud.request_timeout Slack discussion about the same topic

Secrets

How to set Secrets (e.g. GITHUB_ACCESS_TOKEN) on Server?

The Prefect Server uses local Secrets. Therefore, the local Secret e.g. the Secret for your storage access token must be set: either within the ~/.prefect/config.toml or as an environment variable. We generally recommend setting it via an environment variable, but here are several options how you can configure it. 1) config.toml You can add your Secret within the configuration file as follows: [context.secrets] GITHUB_ACCESS_TOKEN = "your_token" 2) Environment variable set on agent sta…

What is the correct (and secure) way to pass GITLAB_ACCESS_TOKEN to a KubernetesAgent using Server deployed with Helm chart?

Related topics: How to set Storage secrets (e.g. GITHUB_ACCESS_TOKEN) on Server?

General features

Is there a way to set concurrency limits on the agent when using Prefect Server?

Performance and Scale of Prefect Server

How scalable is Prefect Server for scheduling concurrent runs of tens of thousands of flows?

View in #prefect-server on Slack [Sam_Brownlow] @Sam_Brownlow: tl;dr How scalable is Prefect Server for scheduling concurrent runs of 10s of thousands of flows? I’m evaluating Prefect Server as a solution for managing a data pipeline. How I envisioned this working was using the Prefect framework as a computationally minimal “glue.” Prefect would be the central pipe through which the state of the data flows (but not the data itself). It would hand off all computation to microservices via (a)…

Which Prefect Server components are stateless and can be scaled horizontally?

View in #prefect-community on Slack [M._Siddiqui] @M._Siddiqui: Hello everyone, I am currently looking to try out a self hosted Prefect Server and UI instance on AWS. Based on the services I saw in the docker-compose file over here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/docker-compose.yml Has anyone attempted to deploy this on an ECS Fargate cluster ? Apart from the Postgres instance, I would like to deploy each service into its own Fargate instance. I am just wond…