Kafka Donuts
Introduction
It's time to build something fun in Kafka!
The last couple of months, we have tested Kafka quite a bit, played with it, destroyed it, destroyed it again. It has proven to be a really usable and durable platform thus far.
It feels like it is time to do something FUNKY with it! I was pondering about what to do and then I figured, let's follow our dream an sell donuts! (Ok, so maybe that is not really the dream here, but it does play well for a fun blog series...)
Let's set off on an adventure and see what we can do with Kafka as our mainstream broker and throw in a couple of other technologies to see what awesome things we can create.
This will be an evolving series - we will update the list of available posts - as they grow - on this main page, at the bottom.
ENJOY!
The Scene
We are a fictional company that sells donuts online and deliver them to you, in 60 seconds.
Kafka Donuts - Done in 60 seconds
We will build most of our business using Kafka as our message broker and use other tools to support and leverage it.
Simulators will be used to bake donuts, buy donuts, act as producers for ingredients and act as drone shipping companies for those long haul orders.
We will write these in different languages to show off Kafka's integration with them.
As we progress we will introduce some real world scenarios and see how we can handle them.
The Kit
As we go along we will do most of this on our local machine. We will use Docker intensively.
Setting Up Shop
Let us start by defining our business.
- We sell a variety of Donuts worldwide
- We take orders through an online portal
- We bake them at our shop
- Ingredients are sourced from suppliers who deliver
- We order ingredients online
- We deliver donuts through Courier companies
- We can introduce special pricing and promotions at any given time
- We have a small stock room where we can store 2000 donuts
- Donuts reaching their Ship Before Date are disposed
- We would like to automate ordering ingredients as they fall below their minimums
Summary
With the seen set, follow the posts below to see how we evolve the various parts of the system. All of the code will be available on Github.
Episodes
Completed:
- Donut Broker - Message Broker - Store and stream all donut data - Kafka
- Donut Baker - Kafka Publisher - Create donuts - .Net Core
Future:
- Donut Buyer - Fake Data - Buy donuts - .Net Core
- Donut Integrator - Integration - Send donut orders to Kafka - Apache Nifi
- Donut Insights - Real Time Processing - Analyze donut orders - KSQL
- Donut Dashboard - Data Visualization - See where our orders come from - Elastic Search, Kibana
- Donut Promotions - Machine Learning - Predict suitable donuts - Tensorflow
- Donut Artisans - Real Time Processing - Do complex processing on donut orders - Apache Flink