Welcome to Discourse!
Before we dive deeper here, where are you in your Prefect adoption? Are you new to Prefect or using it for some time already? If you have the option to choose Prefect 2.0, your use case is much easier to solve:
https://discourse.prefect.io/t/should-i-start-with-prefect-2-0-orion-skipping-prefect-1-0/544
LMK and this way I can give a better answer