How to use Prefect `sync_compatible` functions in Jupyter

Archive
,
1

Really like the sync_compatible decorator in prefect/asyncutils.py at main · PrefectHQ/prefect (github.com).

One question I have is how best to use decorated functions in Jupyter notebook environments? For example the following code works in a plain Python script

from prefect.blocks.system import Secret
secret_block = Secret.load("my-secret")
secret_block  # This is a Secret block

However, if I run it in Jupyter, secret_block is a coroutine, presumably because the sync_compatible decorator detects the async event loop running in the Jupyter notebook. So I have to remember to do await secret_block = Secret.load("my-secret") when I am in a notebook.

Is there a way to get the code to work the same regardless of whether it is running in Jupyter or as a script?

1 Like
2

You can use Prefect in a Jupyter notebook by making your flows and block method calls async. Here is an example async flow:

import asyncio

from prefect import task, flow


@task(log_prints=True)
async def extract() -> list:
    return list(range(1, 11))


@task(log_prints=True)
async def transform(x: int) -> int:
    return x * 2


@task(log_prints=True)
async def load(x: int) -> None:
    print(f"final result {x}")


@flow(log_prints=True)
async def async_etl():
    nrs = await extract()
    for n in nrs:
        final = await transform(n)
        await load(final)


if __name__ == "__main__":
    asyncio.run(async_etl())
3

Since Jupyter already has a tornado loop running, you can simply call await where needed:

from prefect.blocks.system import Secret
secret_block = await Secret.load("my-secret")
secret_block  # This is a Secret block

I encounter this a lot too so maybe I can find a way to patch sync_compatible to detect whether it’s a Jupyter notebook and run accordingly.

1 Like
4

I added an issue here Calling `sync_compatible` functions in Jupyter notebooks require `await` · Issue #7722 · PrefectHQ/prefect · GitHub

2 Likes