{"componentChunkName":"component---src-templates-session-details-js","path":"/sessions/lessons-learned-scaling-stateful-kafka-streams-topologies","result":{"data":{"contentfulSessionDetails":{"title":"Lessons Learned Scaling Stateful Kafka Streams Topologies","slug":"lessons-learned-scaling-stateful-kafka-streams-topologies","timing":"2:00 PM - 2:45 PM","details":{"details":"Kafka Streams is a powerful engine to do stream processing, and its stateful operations have allowed us to implement event-driven architectures in a simple, efficient and productive way.\n\nOur use case is about real estate listings websites, and at relatively low volumes of data (few millions) everything worked out of the box. However, when we started scaling things got a bit more difficult: High latency on every rolling-update, topologies eternally in rebalance, write stalls, excessive AWS bills and even losing data.\n\nI will explain a bunch of actions we have done that helped us scale our topologies to process hundreds of millions of listings: Use kubernetes StatefulSets, tune RocksDB configurations, use Horizontal Pod Scaling wisely, activate consumer Rack Awareness, and more."},"speakers":[{"name":"Ferran Galí i Reniu","designation":"Data Engineer","organisation":"LIFULL Connect","image":{"file":{"url":"//images.ctfassets.net/w1ipbbcgm10q/4DrsdtNDiuoS8X77FJRac1/ed48c1ac5b46f0a68546515bd0ad313f/ferran-gali-i-reniu.jpeg"}}}],"schedule_details":[{"schedule":[{"date":"May 17, 2023","agenda":[{"events":[{"displayName":"Kafka Summit London","slug":"kafka-summit-london-2023","startDate":"May 16","eventEndDate":"17, 2023","endDate":"May 17, 2023","eventStartMonth":"May","eventEndMonth":"May","registrationUrl":null,"registrationCtaText":null,"transparentHeader":true,"bannerImageLeft":null,"bannerImageRight":null,"bannerImageLeftMobile":null,"bannerImageRightMobile":null,"seoMetadata":{"title":"Kafka Summit London 2023 | May 16 - 17 | London, UK","description":{"description":"Kafka Summit London offers 2 days of technical sessions on Apache Kafka, data streaming, and real-time data technologies from leading technologists and industry experts."}}}]}]}]}]}},"pageContext":{"slug":"lessons-learned-scaling-stateful-kafka-streams-topologies"}},"staticQueryHashes":["3974625060"]}