This blog post shows how you can use Orion to handle both batch-processing and real-time streaming.
Key points
- maintaining two separate systems for batch processing and real-time streaming introduces additional burdens and costs
- a more flexible workflow orchestration system such as Orion can handle both paradigms
- demo with a sample e-commerce application streaming real-time orders showing how Orion can extract and process real-time data
- demo of how Orion can help make a real-time data pipeline more robust and easier to maintain by:
- adding retries
- parametrization
- allowing conditional logic
- providing observability into failed components
- allowing to send notifications on failure
- orchestrating downstream processes