You No Longer Need Two Separate Systems for Batch Processing and Streaming - by Anna Geller

This blog post shows how you can use Orion to handle both batch-processing and real-time streaming.

Key points

  • maintaining two separate systems for batch processing and real-time streaming introduces additional burdens and costs
  • a more flexible workflow orchestration system such as Orion can handle both paradigms
  • demo with a sample e-commerce application streaming real-time orders showing how Orion can extract and process real-time data
  • demo of how Orion can help make a real-time data pipeline more robust and easier to maintain by:
    • adding retries
    • parametrization
    • allowing conditional logic
    • providing observability into failed components
    • allowing to send notifications on failure
    • orchestrating downstream processes

Link is now broken , any updated ?
oh nvm , it redirected to acutal link.

If I process 1000 events every second, I can actually send ~100 million Prefect API calls a day without any extra cost?

In this scenario, Prefect is simply observing the state of your streaming runs - check this post, it will clarify a lot:

For pricing questions, Iā€™d encourage you to reach out to since Iā€™m not up to speed on that.

1 Like