How to Use Prefect and Monte Carlo to Achieve More Reliable Data Pipelines

This post demonstrates the Monte Carlo integration that adds even more observability into your data workflows.

Key points discussed in the article

  • What is Monte Carlo (explained by someone new to the platform)
  • How is Monte Carlo different from Prefect?
  • The problem that Prefect’s integration for Monte Carlo can solve including:
    • ability to track data lineage
    • schema changes
    • anomaly detection incl. data volume and freshness
  • New Prefect tasks for Monte Carlo allow to:
    • add missing upstream and downstream nodes in Monte Carlo’s data lineage graph
    • enrich tables with metadata from Prefect flows including the last time the data was updated, whether data passed data quality tests, and much more

The code for the demo from the article:

Live-workshop recording

There is also a recorded video of a workshop available here:

The code from the workshop: