You No Longer Need Two Separate Systems for Batch Processing and Streaming - by Anna Geller

This blog post shows how you can use Orion to handle both batch-processing and real-time streaming.

Key points

  • maintaining two separate systems for batch processing and real-time streaming introduces additional burdens and costs
  • a more flexible workflow orchestration system such as Orion can handle both paradigms
  • demo with a sample e-commerce application streaming real-time orders showing how Orion can extract and process real-time data
  • demo of how Orion can help make a real-time data pipeline more robust and easier to maintain by:
    • adding retries
    • parametrization
    • allowing conditional logic
    • providing observability into failed components
    • allowing to send notifications on failure
    • orchestrating downstream processes