What’s Inside the Black Box? Using ML to Tune and Manage Kafka
We use machine learning to delve deep into the internals of how systems like Kafka work. In this talk I’ll dive into what variables affect performance and reliability, including previously unknown leading indicators of major performance problems, failure conditions and how to tune for specific use cases. I’ll cover some of the specific methodology we use, including Bayesian optimization, and reinforcement learning. I’ll also talk about our own internal infrastructure that makes heavy use of Kafka and Kubernetes to deliver real-time predictions to our customers.
Speakers
Matthew Stump
CEO, Vorstella
