How to disable the SSL verification when setting up a Prefect agent? I'm getting SSLCertVerificationError: "CERTIFICATE_VERIFY_FAILED certificate verify failed: unable to get local issuer certificate"

View in #prefect-community on Slack

@Italo_Barros: Hello everyone, does someone knows how to disable the SSL Verification when using prefect? Is there’s something like the “–set ssl_verify True”?

I’m facing some problems with auth and running the agent due to a corporate network. The following error occurs when I use the “prefect auth login --key”:

requests.exceptions.SSLError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

davzucky @davzucky: Just to confirm. Are you using perfect cloud or server?

@Italo_Barros: Prefect Cloud, I had faced this error before but updating the “certifi” lib solved the issue. But for some reason, this turnaround is not working anymore.

@Anna_Geller: A couple of questions:

  1. What type of agent do you use?
  2. How did you start your agent?
  3. Can you confirm that you can reach the Internet from the machine on which you run the agent?
  4. Do you have port 443 open to outbound traffic?

davzucky @davzucky: Is your company using a proxy CA certificate to wrap external call?

@Italo_Barros: @Anna_Geller I’m using the local agent with is started by the command “prefect agent local start --agent-address http://localhost:8056”. I’m able to reach the internet, and port 443 is open to outbound traffic. I believe that the problem with the agent is also related to the SSL error.
@davzucky I don’t know if this answers your question but my company is using Zscaler. To use the conda, pip, and also the urllib I need to set the ssl_verify to False to properly enable some GET or POST calls.
Since prefect uses urllib in the background, I’m unable to disable de SSL verification during the CLI command. Maybe will work if I change the source code to disable, but I’m looking to a easier way to do that haha

davzucky @davzucky: Ok. I don’t know them. But they may inject their own certificate in the request. Do you have a company CA certificate that you are using?
Can you try this hack to disable CA validation? https://stackoverflow.com/questions/48391750/disable-python-requests-ssl-validation-for-an-imported-module

Stack Overflow: Disable Python requests SSL validation for an imported module

If that work will meet to check the certificate chain

@Anna_Geller: @Italo_Barros can you confirm whether setting this env variable as @davzucky described works?

export CURL_CA_BUNDLE=""

also, do you run this on Unix or on Windows VM? StackOverflow users confirmed this works on Unix but not on Windows

Were you able to start the Prefect agent? Can you see the agent being healthy and querying for flow runs?

This thread discusses agent health checks - it’s not about SSL issues, but it may be helpful to check as well

@Italo_Barros: Hi @Anna_Geller and @davzucky I already tried to set some environment variables to FALSE like the CURL_CA_BUNDLE and PYTHONHTTPSVERIFY, doesn’t work.

Regarding the OS, I’m running on Windows 10 with Python 3.9. I will try to run in another version of Python just to check.

I also can’t start the prefect agent due to the same SSL error.

@Anna_Geller: I see, this confirms what SO users said that it doesn’t work on Windows… I’ll ask the team since debugging SSL issues on windows is a bit beyond my level :laughing:
Can you ask your IT DevOps folks for support here? I talked to someone from my team who used to work in a similar environment and he said it may be an issue totally not related to Prefect, e.g. can you download pip packages from this VM?

davzucky @davzucky: sorry, I won’t be able to help a lot as well on Windows, usually I only run on Linux. will be interested to see what is the root cause
One think I can think about, Are you able to run podman on Windows ?
@Anna_Geller on the Cloud are you exposing the GraphQL playground endpoint ?

@Anna_Geller: yes, we do, it’s enabled by default and I don’t even know if it can be disabled - why do you ask? is this related to the SSL issue in some way?

@Italo_Barros: @Anna_Geller, I’m able to download pip and conda packages disabling putting the SSL Verification to FALSE as default. There’s no way to do that on prefect?

davzucky @davzucky: @Anna_Geller yes. Could you paste it? I want @Italo_Barros to go there with the browser and check the certificate? I want to know who issue it

@Anna_Geller: @davzucky Do you want me to paste something or Italo? Regarding the interactive API, the URL is:

https://cloud.prefect.io/teamname/api

davzucky @davzucky: Yes exactly. @Italo_Barros can you check on chrome or edge what issues the certificate?

A similar question on Slack:

Bennett_Lambert @Bennett_Lambert: Hi all, my company is testing prefect and I am having trouble installing a kubernetes agent, which I suspect is because of the company proxy. When I try to install the agent the logs say that it is being registered, but then eventually I get an SSL error. requests.exceptions.SSLError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))
Is there a way to provide proxy credentials when starting the agent? I’ve tried using the --env flag to supply proxy information. Or is there something similar to a --verify=false that I can set?

Anna_Geller @Anna_Geller: Hi @Bennett_Lambert, you could try setting this environment variable:

export CURL_CA_BUNDLE=''
export PYTHONHTTPSVERIFY='false'

Solution

Bennett_Lambert @Bennett_Lambert: ok thanks for the help!
I’m on Windows and setting CURL_CA_BUNDLE=’’, PYTHONHTTPSVERIFY=‘false’ and the HTTP_PROXY and HTTPS_PROXY environment variables did the trick

Anna_Geller @Anna_Geller: Thanks so much for verifying that! :raised_hands: