Cloudpickle version mismatch causing errors

View in #prefect-community on Slack

James_Keegan @James_Keegan: cloudpickle v2.1.0 was just released an hour ago and it has broken the healthcheck when we try to register a flow. We’re using the prefecthq/prefect:0.15.6-python3.7 as a base image.

We now get this error when building the docker image.

Step 21/21 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/prefect-dbt-run-modelling.prefect"]' '(3, 7)'
 ---> Running in 1d14c333ced9
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 152, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
AttributeError: Can't get attribute '_make_function' on <module 'cloudpickle.cloudpickle' from '/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle.py'>

I re-ran a successful pipeline from ~4 hours ago to confirm that it wasn’t to do with any code changes we made and yeah, it also failed

Kevin_Kho @Kevin_Kho: Will try to replicate and will create a ticket for it

Michael_Adkins @Michael_Adkins: This sounds like mismatched cloudpickle versions
We can’t update the version that’s inside that container. You’ll want to register flows with a matching version.

James_Keegan @James_Keegan: Interesting - so update the version of cloudpickle in the pipeline env to also be 2.1.0?

Kevin_Kho @Kevin_Kho: I think more like make your registration cloudpickle version the same as the one in the prefecthq/prefect:0.15.6-python3.7 container
I can replicate if my local registration is cloudpickle 2.1 using that same image. cloudpickle version 1.6 worked fine. I think you need to downgrade the registration version

James_Keegan @James_Keegan: Thanks for the advice all - downgrading didn’t work, but updating the version to 2.1.0 in the Dockerfile (in an extra_dockerfile_command) did! Yeah must have been a version mismatch as stated

Ilya_Galperin @Ilya_Galperin: Thank you everyone in this thread, helped us out :slightly_smiling_face: Specifically adding the following command to our docker storage extra_dockerfile_command per James’ suggestion: RUN pip install cloudpickle==2.1.0

Kyle_McChesney @Kyle_McChesney: Same issue. Just to confirm, is it the cloud pickle version in server or agent that needs to be the same

Michael_Adkins @Michael_Adkins: @Kevin_Kho let’s open an issue / discourse topic for this so it’s discoverable, I ran into a cloudpickle compat issue today too.

Kevin_Kho @Kevin_Kho: Base image for Docker storage and flow registration (and probably agent). I will turn this into Discourse

1 Like

adding: RUN pip install cloudpickle==2.1.0 to the dockerfile for our agent fixed it.

1 Like