Does Prefect support data-aware scheduling?

View in #prefect-community on Slack

Todd_de_Quincey @Todd_de_Quincey: Hi all,

I’m curious if the Prefect team have any plans to introduce concepts similar to Dagster’s Software-defined-assets or Airflow’s Data-aware scheduling?

I haven’t used either of the above, but conceptually, they are very attractive solutions (especially Dagster’s implementation).

Bianca_Hoch @Bianca_Hoch: Hi Todd! We are working on a much more explicit (yet still flexible) way to track such metadata related to your workflows using Resources and Events, which will additionally allow you to track SLAs related to your data. For instance, you’ll be able to define that a given Snowflake table should always be updated by 9 AM, and if such SLA is not satisfied, you can take automated actions and track related failures in a more robust way, allowing for faster detection and resolution.

Todd_de_Quincey @Todd_de_Quincey: Any rough ETAs?

Bianca_Hoch @Bianca_Hoch: No firm ETAs yet, but these are high priority features that are actively being worked on for release in the near future. Definitely monitor our posts in the announcements channel, where we share release notes for each new version of Prefect.

1 Like