View in #prefect-community on Slack
@James_Keegan: cloudpickle
v2.1.0 was just released an hour ago and it has broken the healthcheck when we try to register a flow. We’re using the prefecthq/prefect:0.15.6-python3.7
as a base image.
We now get this error when building the docker image.
Step 21/21 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/prefect-dbt-run-modelling.prefect"]' '(3, 7)'
---> Running in 1d14c333ced9
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 152, in <module>
flows = cloudpickle_deserialization_check(flow_file_paths)
File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
flows.append(cloudpickle.loads(flow_bytes))
AttributeError: Can't get attribute '_make_function' on <module 'cloudpickle.cloudpickle' from '/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle.py'>
I re-ran a successful pipeline from ~4 hours ago to confirm that it wasn’t to do with any code changes we made and yeah, it also failed
@Kevin_Kho: Will try to replicate and will create a ticket for it
@Michael_Adkins: This sounds like mismatched cloudpickle versions
We can’t update the version that’s inside that container. You’ll want to register flows with a matching version.
@James_Keegan: Interesting - so update the version of cloudpickle in the pipeline env to also be 2.1.0?
@Kevin_Kho: I think more like make your registration cloudpickle
version the same as the one in the prefecthq/prefect:0.15.6-python3.7
container
I can replicate if my local registration is cloudpickle 2.1 using that same image. cloudpickle version 1.6 worked fine. I think you need to downgrade the registration version
@James_Keegan: Thanks for the advice all - downgrading didn’t work, but updating the version to 2.1.0 in the Dockerfile (in an extra_dockerfile_command
) did! Yeah must have been a version mismatch as stated
@Ilya_Galperin: Thank you everyone in this thread, helped us out Specifically adding the following command to our docker storage extra_dockerfile_command per James’ suggestion: RUN pip install cloudpickle==2.1.0
@Kyle_McChesney: Same issue. Just to confirm, is it the cloud pickle version in server or agent that needs to be the same
@Michael_Adkins: @Kevin_Kho let’s open an issue / discourse topic for this so it’s discoverable, I ran into a cloudpickle compat issue today too.
@Kevin_Kho: Base image for Docker storage and flow registration (and probably agent). I will turn this into Discourse