A fun introduction to the world of Kafka Connect and Kafka Streams by using it to process data from Xbox Live. Gaming is a social activity, so Xbox includes a social aspect. Details about what games you play and when you’re playing are shared with your friends on the Xbox Live service. Achievements that you earn and complete in games are shared with your friends. This makes for a fun source of data, that we can play with using Kafka Connect and Kafka Streams. 1) Getting the data into Kafka I’ll show how I’m using Kafka Connect to surface data from Xbox REST APIs as streams of events on Kafka topics. 2) Processing the events I’ll show how I’m using Kafka Streams to process the Xbox events as they’re emitted - such as cleaning up some of the noisy events with filters, enriching events with additional context using transformations, and creating gamerscore leaderboards using aggregations. As a project, it is just a bit of fun. But it is a useful exercise for illustrating how to get started with Kafka Connect and Kafka Streams, and a way to demonstrate the principles using a real source of real events, rather than using fake or generated events. Plus it is a good excuse for me to spend more time gaming, and justifying it as work!
