We are running flows using Kubernetes-Job infrastructure and we are using the on_completion annotations on our flows to capture state. We can use the opencensus python library to write logs to Azure app insights from within the flows but our understanding is the on_completion function runs outside of the flow and we’re trying to figure out how to send those messages to app insights. Does it run on the kubernetes job pod as part of the flow run or does it run from the agent that kicked off the kubernetes job after the flow run completes? Or is it run somewhere else?
We are running:
Prefect Cloud (Prefect 2.0)
Agent version 2.8.3
K8s Job version 2.8.3
I can provide additional context if necessary. Thanks in advance!