Event Streams

Event streams are a log of one or more "things that happen", which are usually referred to as events. Event streams are conceptually focused around events than objects or tables, which are the typical storage unit of relational databases.

Apache Kafka and Gazette are a popular open source implementations of event streams. Amazon Web Services' Kinesis and Azure Event-Hubs are proprietary hosted implementations.

Why do event streams matter to developers?

The way that data is stored affects how you can work with it. Constraints and guarantees like consistency make it easier to code certain applications but harder to build other types of applications that need performance in different ways. Event streams make it easier to build applications that analyze large amounts of constantly-updated data because the events are not stored relationally.

How are event streams typically stored?

Some applications, such as aggregating millions of sensors, or thousands of streaming cameras, constantly output large amounts of data with no breaks. It is difficult to process such large volumes of data in traditional data stores, so event streams are built off of a simpler data structure: logs.

Logs are ordered sequences of events and they typically have less constraints than a database. In event streams, logs are also immutable. They do not change once they are written. Instead, newer events are written in the sequence as state changes. The reduced constraints compared to a database and the immutability mean that logs can handle the high throughput writes needed to keep up with the constant flood of data from the source of an event stream.

Event stream resources

What topic do you want to learn next?

Tell me about standard relational databases.

How should Python libraries be installed on a server?

What editor should I use to code my Python app?


Matt Makai 2012-2022