How to use Prefect `sync_compatible` functions in Jupyter

Really like the sync_compatible decorator in prefect/ at main · PrefectHQ/prefect (

One question I have is how best to use decorated functions in Jupyter notebook environments? For example the following code works in a plain Python script

from prefect.blocks.system import Secret
secret_block = Secret.load("my-secret")
secret_block  # This is a Secret block

However, if I run it in Jupyter, secret_block is a coroutine, presumably because the sync_compatible decorator detects the async event loop running in the Jupyter notebook. So I have to remember to do await secret_block = Secret.load("my-secret") when I am in a notebook.

Is there a way to get the code to work the same regardless of whether it is running in Jupyter or as a script?

1 Like

You can use Prefect in a Jupyter notebook by making your flows and block method calls async. Here is an example async flow:

import asyncio

from prefect import task, flow

async def extract() -> list:
    return list(range(1, 11))

async def transform(x: int) -> int:
    return x * 2

async def load(x: int) -> None:
    print(f"final result {x}")

async def async_etl():
    nrs = await extract()
    for n in nrs:
        final = await transform(n)
        await load(final)

if __name__ == "__main__":

Since Jupyter already has a tornado loop running, you can simply call await where needed:

from prefect.blocks.system import Secret
secret_block = await Secret.load("my-secret")
secret_block  # This is a Secret block

I encounter this a lot too so maybe I can find a way to patch sync_compatible to detect whether it’s a Jupyter notebook and run accordingly.

1 Like

I added an issue here Calling `sync_compatible` functions in Jupyter notebooks require `await` · Issue #7722 · PrefectHQ/prefect · GitHub