Session: Equipping easy-to-use and scalable stream processing technologies on Kubernetes
Stream processing and data analysis are crucial for various roles, including data engineers, platform engineers, DevOps, and many more. Simplifying real-time stream data processing while ensuring it’s cost-efficient and resilient to K8s pod restarts or node upgrades remains a significant challenge. While existing stream processing solutions exist, they are often hard to manage, operationally demanding, and expensive. This presentation will share our journey in developing a generic open-source Kubernetes-native stream processing framework, it allows developers to effortlessly and quickly execute large-scale stream processing tasks without the burden of heavy and costly data processing platforms. By leveraging this platform, Intuit’s application developers handle approximately 5 billion messages for analytics daily, while ML engineers train 135K models and make 60M predictions. It’s also the engine behind Intuit’s extensive anomaly detection platform, running in more than 200+ Kubernetes clusters.
