How to use Dask without mapping in Prefect 1.0: using Dask worker_client to call client.submit() inside a Prefect task

Archive
, , , , , , ,
7

For that you would need to persist the results yourself - there is no such feature in Prefect to automatically infer what you want to persist :smile: unless you return it and checkpoint the result.

Do you happen to use AWS S3? If so, you could persist this data in a single command using awswrangler - I love this library!

https://aws-data-wrangler.readthedocs.io/en/stable/

Alternatively, you could manually call .write() method on results within your task:

from prefect import Flow, task
from prefect.engine.results import S3Result

@task
def my_task():
    s3_result = S3Result(bucket='bucket_of_models')
    s3_result.write(your_dask_dataframe)

More on that:

Also, you can check: