Mastering real-time anomaly detection with open source tools
Talk (60 min)
We'll use Apache Kafka for streaming data, Apache Flink for processing it in real time, and ARIMA models to detect unusual patterns. Whether it's spotting fraud, monitoring systems, or tracking IoT devices, this solution is flexible and reliable.
We'll start by exploring how Kafka helps collect and manage fast-moving data streams. Then, we'll demonstrate how Flink processes this data in real time and integrates ARIMA-based anomaly detection to uncover events as they occur. We'll dive into the details of how ARIMA works—covering how it models trends, seasonality, and residual noise to identify outliers in time-series data.
We'll also show how Apache Iceberg can be used to store historical data efficiently, enabling retrospective analysis, ongoing model evaluation, and performance improvements over time.
By combining real-time detection with long-term storage, you can build a robust system that evolves as your data grows. This talk includes clear examples and practical steps to help you build your own pipeline. It's ideal for anyone looking to use open-source tools to monitor and react to issues in real-time data streams.
Olena Kutsenko
Olena is a Staff Developer Advocate at Confluent and a recognized expert in data streaming and analytics. With two decades of experience in software engineering, she has built mission-critical applications, led high-performing teams, and driven large-scale technology adoption at industry leaders like Nokia, HERE Technologies, AWS, and Aiven.
A passionate advocate for real-time data processing and AI-driven applications, Olena empowers developers and organizations to use the power of streaming data. She is an AWS Community Builder, a dedicated mentor, and a volunteer instructor at a nonprofit tech school, helping to shape the next generation of engineers.
As an international speaker and thought leader, Olena regularly presents at top global conferences, sharing deep technical insights and hands-on expertise. Whether through her talks, workshops, or content, she is committed to making complex technologies accessible and inspiring innovation in the developer community.