Getting data between systems, particularly at scale, is a common challenge faced by data engineers. Pipelines need to be reliable, flexible, and scalable, and without requiring us to write the same boilerplate code each time. Kafka Connect is a framework that provides scalable & fault-tolerant integration between Apache Kafka and other systems. It can be deployed on containers making it easy to scale for increased capacity, throughput, and resilience. We will give a short intro to Kafka Connect and container technologies before proceeding to a deep dive into practical applications. Attendees will learn about: * Real-world Kafka Connect pipelines. * How to build custom connector container images * Configuration, and orchestration of Kafka Connect pipelines with Kubernetes using GitOps.
