Tackling Kafka, with a Small Team

This is a story about what happens when a distributed system becomes a big part of a small team’s infrastructure. This distributed system was Kafka and the team size was one engineer. I will discuss my failures along with my journey of deploying Kafka at scale with very little prior distributed systems experience. In this presentation, we will discuss how unique insights in the following organization culture, engineering and metrics created tailwinds and headwinds. This presentation will be a tactical approach to conquering a complex system with an understaffed team while your business is growing fast. I will discuss how the use case and resilience requirements for our Kafka cluster change as the user base grew from 100K users to over 6 million.

Speakers
speakerimage
Jaren Glover
Infrastructure Engineer, Robinhood