How to disable sending telemetry data from Server?

anna_geller · April 8, 2022, 9:56am

@Omar_Sultan: Hi Everyone, we have a prefect server running on Kubernetes, setup was done using the HELM Chart. Everything is running smoothly but occasionally we would get this error

   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 341, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='prefect-apollo.prefect', port=4200): Read timed out. (read timeout=15)

This happens especially when we use the Task StartFlowRun it does not happen very often , but I was wondering if there was a way to force retry or if anyone knows why this would be happening? Thanks

@Anna_Geller: This error means that some action on the backend took too long to respond to the API request. E.g. it could be that the task run state updates are taking too long.

Here is how you can increase the read timeout on Server:
• Is it possible to increase the GraphQL API request timeout?
• How to check a GraphQL query timeout settings on Server? How to increase that timeout value?

To set retry on the StartFlowRun task, you can do:

from datetime import timedelta
from prefect import Flow
from prefect.tasks.prefect import StartFlowRun

start_flow_run = StartFlowRun(project_name="PROJECT_NAME", wait=True, max_retries=3, retry_delay=timedelta(minutes=5))

with Flow("FLOW_NAME") as flow:
    staging = start_flow_run(flow_name="child_flow_name")

@Omar_Sultan: Thank you so much for that
Hey Anna, quick followup , so I applied the change in the HELM Values and apply the configuration. And I can now see the env variable being assigned to the apollo pods , however when I print

prefect.context.config.cloud.request_timeout

from any task that runs on the server it still shows 15, is there anything else I need to apply? do I need to restart the agent pod for example?

@Anna_Geller: You may restart the Apollo pod so that it can pick up the changes. But not sure what the max allowed value is here, what did you set?

@Omar_Sultan: I set it at 60 , I believe the document said that the max allowed value was 60

@Anna_Geller: I see. So if this doesn’t help I’m afraid we have to find out the root cause for those timeouts can you share your flow with StartFlowRun that times out? Could it be that the child flow run gets stuck and the parent flow run polling for its status at some point times out?

@Omar_Sultan: Hey Anna, so i’ve been doing some investigations and it seems that this error only happens right around the same time the Hasura pod tries to submit telemetry. Because I am working in a closed environment without internet access this operation is timing out and it seems its causing the flows to timeout as well. Not sure if that makes anysense.

I can see from Hasura documentaiton that telemetry can be disabled, however, not sure how i can do that in the helm values file. Any ideas?

@Anna_Geller: Nice work finding that out! There is an easy way to disable it, check out this page: Telemetry | Prefect Docs

On a Hełm chart you should be able to set an environment variable

name: PREFECT__SERVER__TELEMETRY__ENABLED 
value: false

@Omar_Sultan: So I followed the link above and it disabled telemetry for prefect and reflected correctly. But I believe the issue is coming still from the Hasura image itself, it needs to have an env variable in the pod called HASURA_GRAPHQL_ENABLE_TELEMETRY set as false … I’m looking at the vlaues.yaml of the helm chart but can’t find anywhere to pass env variables to the service
sorry to keep bugging you with this

@Anna_Geller: I have a meeting with a Prefect employee who knows more about this telemetry thing in just 10 min will ask and update you afterwards

@Omar_Sultan: WoW nice thank yo u so much

@Anna_Geller: @Omar_Sultan sorry for getting back to you a little later than planned: I check and this is part of the code where the telemetry info is set

github.com

PrefectHQ/server/blob/master/services/apollo/src/index.js#L19


      
          const express = require('express')
          const APOLLO_API_PORT = process.env.APOLLO_API_PORT || '4200'
          const APOLLO_API_BIND_ADDRESS = process.env.APOLLO_API_BIND_ADDRESS || '0.0.0.0'
          const APOLLO_API_BODY_LIMIT = process.env.APOLLO_API_BODY_LIMIT || '5mb'
          const APOLLO_API_ENABLE_PLAYGROUND = process.env.APOLLO_API_ENABLE_PLAYGROUND == 'false' ? false : true
          
          
const PREFECT_API_HEALTH_URL =
            process.env.PREFECT_API_HEALTH_URL || 'http://localhost:4201/health'
          const PREFECT_SERVER_VERSION = process.env.PREFECT_SERVER_VERSION || 'UNKNOWN'
          
          
const TELEMETRY_ENABLED_RAW =
            process.env.PREFECT_SERVER__TELEMETRY__ENABLED || 'false'
          // Convert from a TOML boolean to a JavaScript boolean
          const TELEMETRY_ENABLED = TELEMETRY_ENABLED_RAW == 'true' ? true : false
          const TELEMETRY_ID = uuidv4()
          // --------------------------------------------------------------------
          // Server
          const app = express()
          const depthLimit = require('graphql-depth-limit')
          class PrefectApolloServer extends ApolloServer {
            async createGraphQLServerOptions(req, res) {

You can see here that the environment variable PREFECT_SERVER__TELEMETRY__ENABLED is the right one. I seemed to have mistyped the underscores (only one underscore between PREFECT and SERVER), which may be the reason why this didn’t work for you.

Also, once you set that, you may need to restart the Apollo pod to make sure the changes get applied.

To set it on the Helm chart, see server/values.yaml at master · PrefectHQ/server · GitHub, e.g.:

helm upgrade \
    $NAME \
    prefecthq/prefect-server \
    --set apollo.options.telemetryEnabled=true

Topic		Replies	Views
How can I set task-level timeouts in Prefect 2? Archive prefect-2-0 , timeout , prefect-future , wait	9	2399	June 21, 2023
Prefect flows failing due to PrefectHTTPStatusError when number of flows is increased Help prefect-2-0 , logging , failure , troubleshooting	0	27	April 22, 2024
On task mark as success, still inner loops are in progress Archive prefect-1-0 , graphql	12	1427	May 6, 2022
How do I stop or cancel a task run? Archive prefect-2-0 , cancellation	3	3087	March 29, 2022
Flow run crashing with asyncio.exceptions.CancelledError Help	1	507	June 29, 2023

How to disable sending telemetry data from Server?

Related Topics