What are Streaming APIs?

Streaming APIs are not to be confused with multimedia streaming API services like Netflix or Youtube. Industry is starting to use a newer breed of REST APIs called the Streaming APIs to offer a “high-throughput” pipeline to receive curated data. With these APIs you can capture information in real time. It’s perfect if you’re working with a continuous stream that has no defined end like live location updates that you are trying to show in a map.

Stream Definitions

Each API call will accept a set of filters which defines a stream. For instance, a stream definition could be “stream me locations of my facebook friends”.

How it works ?

One more time on the goal – we have something happening right now live and we are trying to provide a means for streaming that live information to say a web or mobile client. Lets assume the clients call a REST API to initiate the streaming call and Streaming APIs need a kick-ass infrastrcuture in the behind. Internally these APIs uses a Transient pub/sub architecture. A Live Stream Processor will publish and API Server will Subscribe. Every stream definition that comes through a new API request will be registered/stored in a persistent datastore. Then the API server opens a pipe to the live stream processor (Imagine any Pub/Sub infra like MQTT pipe or REDIS pub/sub) hoping live stream processor will give the data what the API client needs. On the other hand Live Stream processor which takes real-time streams of data from the real-world runs queries against each of the available stream definitions. The satisfied list of stream definitions are supposed to be pushed to respective pipes opened with API server and in turn to the clients.

Sorry if I have lost you with this, try and see if this figure helps a bit.