How Did We Move the Mountain? - Migrating 1 Trillion+ Messages Per Day Across Data Centers at PayPal
Date : September 15, 2021
Time : 02:00 PM - 03:00 PM

Have you ever migrated Kafka clusters from one data center to another being completely transparent to client applications? At PayPal, as part of a massive datacenter migration initiative, Kafka team successfully moved all PayPal Kafka traffic across data centers. This initiative involved migrating 20+ Kafka clusters (1000+ broker and zookeeper nodes), as well as 60+ mirrormaker groups which seamlessly handle Kafka traffic volumes as high as 1 trillion messages per day. Throughout the course of this migration, applications required no modification, encountered 0% service outage, 0% message loss and duplicated messages. The whole migration process was fully transparent to Kafka applications. In this session, you will learn the strategies, techniques and tools the PayPal Kafka team has utilized for managing the migration process. You will also learn the lessons and pitfalls they experienced during this exercise, as well as the secret sauce of making the migration successful.

Na Yang
Sr. Engineer Manager, PayPal
Lei Huang
Member of Technical Staff 2, PayPal