Unreliable Activation of on_cancellation Hook in Process Work Pool - Prefect 2.19.7

Help
1

Bug summary
The on_cancellation hook can’t be triggered Consistently when the flow is deployed in a Process work pool. Even though the workflow status is shown as “canceled” in both the Prefect UI and the client, the absence of the expected log message from the hook suggests that it fails to execute reliably.

Reproduction
We created a Process work pool and started a worker to operate within it.

prefect work-pool create default-process-pool --type process

prefect worker start --pool default-process-pool

Here is the code to reproduce the issue

from prefect import flow, get_run_logger
import os
import time


def hook_cancellantion(flow, flow_run, state):
    logger = get_run_logger()
    logger.info("here is hook cancellation")


@flow(
    name="cancellation_hook_test",
    on_cancellation=[hook_cancellantion],
)
def cancellation_hook_test():
    logger = get_run_logger()
    for i in range(10):
        logger.info(f"flow is running {i}")
        time.sleep(10)


if __name__ == "__main__":
    cancellation_hook_test.from_source(
        source=os.path.dirname(__file__),
        entrypoint=f"{os.path.basename(__file__)}:cancellation_hook_test") \
        .deploy(
        work_pool_name="default-process-pool",
        name="cancel_test",
    )

Error
Then I clicked the cancel button while the flow was halfway through its execution.

<frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
17:47:31.957 | INFO    | Flow run 'judicious-jerboa' - flow is running 0
17:47:41.960 | INFO    | Flow run 'judicious-jerboa' - flow is running 1
17:47:45.491 | INFO    | prefect.worker.process.processworker b3c0d2c6-189f-4bab-a974-b1e7f92cd3db - Found 1 flow runs awaiting cancellation.
^C17:47:45.521 | INFO    | prefect.flow_runs.worker - Cancelled flow run 'd9874f2b-172d-4cb7-854e-de620dac150b'!
17:47:45.522 | ERROR   | prefect.flow_runs.worker - Process 61088 exited with status code: 3221225786; Process was terminated due to a Ctrl+C or Ctrl+Break signal. Typically, this is caused by manual cancellation.

image
image968×826 105 KB

The flow has been canceled. The expected log message Running hook 'hook_cancellantion' in response to entering state 'Cancelling should appear, but it does not.

Versions (prefect version output)

Version:             2.19.7
API version:         0.8.4
Python version:      3.11.8
Git commit:          60f05122
Built:               Fri, Jun 28, 2024 11:27 AM
OS/Arch:             win32/AMD64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.45.2

Additional context
Although the on_cancellation hook reliably triggers when a flow is deployed using the .serve() method, we are looking to utilize a local work-pool due to our need to manage hundreds of flows. Opening hundreds of CLI or tmux sessions on our server is impractical for our operations.