Experiences with Kafka and exactly-once processing in IoT apps

This is a 5 min readSome context on message brokers and delivery guarantees (If you have fair amount of experiences with message processing and delivery guarantees please skip to the next part of this post.) Message delivery guarantee is one of the canonical requirements for message brokers and they are very relevant for all types of brokers: the ones based on queue semantics and the ones… Continue reading

TAPO for Airports – A Streaming usecase

This is a 1 min readAirports, especially the busy ones face an interesting challenge when it comes to serving the commuters, they need a smoother way to handle passengers in queues without long frustrating waits and thereby elevate the overall experience. No one likes to wait/stand in long queues. But airports, unfortunately, have lots of queues one for check-in, baggage… Continue reading

Apache Flink CEP and ATM Fraud usecase- Part 2

This is a 2 min readOn the 1st part of this multi-part series on Apache Flink CEP library, I briefly covered the case for a dedicated CEP framework among the toolsets of open-source stream processing frameworks. Quick recap on the use case For a customer, an ATM Withdrawal Txn >= 10,000 made more than ‘3 times’ in a location >  50 mile radius… Continue reading

“Exactly-once” with a Kafka-Storm Integration

This is a 4 min readUpdate 4, Nov 2016: When I first wrote this post it was outright mockery and contempt. But the Google Data flow paper (The Unified google framework for Batch (FlumeJava) and Stream processing (MillWheel)) and the Google MillWheel paper clearly explains that this is exactly the same approach google team has taken to solve the duplicate events problem…. Continue reading

What wikipedia can’t tell you about Apache storm and Apache spark streaming

This is a 1 min readI am seeing a lot of questions around Spark streaming and Storm in Quora. When to choose what and what are their performances, reliability and support like. There are a lot of comparisons as usual available in the web , if you google around you could find. But instead comparing them side by side I thought of talking… Continue reading

What you didn’t know about Real-time notification systems

This is a 2 min readI have been intrigued by Event Notification systems for a long time now, In fact this started from my programming days in legacy environments like iSeries. So I started working on a toy project which evolved into a solid project. I thought I will muse about that recent project the RealTimeNotification. But before going into the details of the… Continue reading

RTOS Roundup for IoT

This is a 2 min readI have been musing about my views on IoT gateways, protocols and how to handle message inflow from different forms of  “things” in IoT. For a change I wanted to focus on the ingredients of a “thing”. “Things” may or may not be constrained or constrained for different things depending on the context. either case the key ingredient is… Continue reading