Why do I get the error TypeError: cannot pickle object?

There are two scenarios where this error happens.

1. Dask

The first is when using a DaskExecutor and using a task input or output that is not serializable by cloudpickle. Dask uses cloudpickle as the mechanism to send data from the client to the workers. This specific error is often raised with mapped tasks that use client-type objects, such as connections to databases or HTTP clients, as inputs.

Solution: Such database or HTTP connections need to be instantiated (and closed) inside your Prefect tasks.

2. Results

The second way this can happen is through Results. By default, task outputs are saved as LocalResults, and the default Serializer is the PickleSerializer, which uses cloudpickle. If you have a task that returns a client or connection, you can avoid serialization of this task by turning off checkpointing for that task with @task(checkpoint=False).

Similar discussion on Slack