Compendium

Message Queue and Streaming Catalog — Kafka, Pulsar, NATS, RabbitMQ, Cloud Streams, Flink

24 min read

Message Queue and Streaming Catalog

A working catalog of the production messaging and stream-processing landscape circa 2026. Three overlapping product categories blur together here and are easier to read in one document: the message brokers (RabbitMQ, ActiveMQ, IBM MQ), the distributed event logs (Kafka, Pulsar, Redpanda), and the stream processors that consume from both (Flink, Spark Structured Streaming, Beam, Materialize, RisingWave, ksqlDB). They have been converging for a decade — RabbitMQ added quorum queues and streams, Kafka added KRaft and tiered storage, Pulsar built tiered storage and Functions, and the cloud vendors wrap all three with serverless veneers.

The selection axes that matter:

  1. Durability and replication — fsync-per-write vs batched vs replicated vs object-storage-backed. The choice trades off cost and latency.
  2. Throughput vs latency — millions of events/sec at 10ms p99 (Kafka, Pulsar) vs hundreds of thousands at sub-millisecond (NATS, ZeroMQ) vs everything-in-between.
  3. Delivery semantics — at-most-once, at-least-once, exactly-once. Exactly-once is achievable in Kafka (transactions + idempotent producer) and Pulsar (transactions 2.7+), but with significant client-side complexity.
  4. Ordering guarantees — partition-ordered (Kafka, Kinesis), queue-ordered (FIFO SQS, RocketMQ), no-order (standard SQS, NATS pub/sub), key-ordered subsets.
  5. Fan-out patterns — pub/sub broadcast (NATS, Pulsar, EventBridge) vs consumer-group partitioning (Kafka, Kinesis) vs work-queue distribution (RabbitMQ, SQS) vs request/reply (NATS, ZeroMQ).
  6. Replay capability — replayable from offset (Kafka, Pulsar, Kinesis, EventHubs) vs consume-and-acknowledge with no replay (SQS, RabbitMQ classic queues, NATS without JetStream).
  7. Operational model — broker-based (Kafka, RabbitMQ, ActiveMQ), brokerless library (ZeroMQ, nanomsg), single-binary (NATS, Redpanda), and serverless cloud (Confluent Cloud, WarpStream, MSK Serverless, EventHubs).
  8. Wire-protocol gravity — the Kafka wire protocol is the dominant gravity well; Redpanda, WarpStream, BufStream, Azure Event Hubs Kafka surface, Confluent Cloud, and AWS MSK all speak it. AMQP 0.9.1 (RabbitMQ), AMQP 1.0 (Azure Service Bus, ActiveMQ Artemis, Solace), MQTT (IoT brokers), and STOMP form the other major standards.

The distributed event log — Kafka and its successors

The category that started with LinkedIn’s internal infrastructure and now sits at the center of nearly every modern data platform. Distributed, partitioned, replicated, append-only commit log. Producers append, consumers read by offset; data is retained according to time, size, or compaction policy rather than deleted on consumption. This is the “log as primary abstraction” thesis (Jay Kreps, “The Log: What every software engineer should know about real-time data’s unifying abstraction” 2013).

  • Apache Kafka — LinkedIn 2010 (Jay Kreps + Neha Narkhede + Jun Rao); Apache Software Foundation 2011 (top-level project 2012); Apache 2.0. The dominant event-streaming platform of the 2015-2026 era. Architecture: brokers form a cluster, topics partition the data, partitions replicate via leader/follower per-partition (ISR — in-sync replica set), producers publish with idempotent + transactional options, consumers read in consumer groups (partition assignment, offset commits). Storage: per-partition segment files on disk with sparse index; retention by time (default 7 days) or by size; log compaction (key-based deduplication, used for change-data-capture and configuration topics). Major releases of note:
    • 0.10 (2016) — Kafka Streams library, exactly-once-ish semantics in motion.
    • 0.11 (2017) — idempotent producer, transactions, exactly-once semantics (EOS) for the producer→Kafka→consumer pipeline.
    • 2.4 (2019) — incremental cooperative rebalancing.
    • 3.0 (2021) — KRaft (Kafka Raft) consensus available as preview, deprecating ZooKeeper dependency.
    • 3.3 (2022) — KRaft production-ready.
    • 3.5 (2023) — Tiered Storage GA (S3/object-storage offload of old segments).
    • 3.6 (2023) — JBOD support in KRaft, KIP-405 tiered storage stable.
    • 3.7 (2024) — KRaft fully replaces ZooKeeper for new clusters; ZooKeeper marked deprecated; first release where Zk-mode is “legacy compatibility only.”
    • 3.8 / 3.9 (2024-2025) — queue-style semantics improvements (KIP-932 “Queues for Kafka”), share groups, performance work on tiered storage cold reads. Cluster scale at the high end: LinkedIn runs 100+ clusters with thousands of brokers each; Uber’s MultiKafka is 38 clusters and several trillion messages/day; Netflix runs 36 clusters of ~200 brokers each. The Confluent Cloud control plane handles 10K+-broker tenants.
  • Confluent Platform — commercial Kafka distribution from Confluent Inc; founded 2014 by the original LinkedIn Kafka authors (Jay Kreps CEO, Neha Narkhede, Jun Rao); IPO Jun 2021 raising 9.1B initial market cap, dipped to ~$8B 2024 after platform-software re-rating. The platform layers Schema Registry (Avro/Protobuf/JSON Schema with compatibility rules), Connect (source/sink ETL plugin framework — 200+ connectors), ksqlDB (SQL-on-Kafka-Streams), Control Center (web UI), MirrorMaker 2 (cross-cluster replication), REST Proxy, Cluster Linking (Confluent-only), Stream Governance (lineage + data quality), Audit Logs, Tiered Storage (in CP since 6.0).
  • Confluent Cloud — fully managed serverless Kafka launched 2017; tiers Basic (single-AZ, dev), Standard (multi-AZ shared infra), Dedicated (single-tenant). Pricing: per-eCKU (elastic Confluent Kafka Unit, ~10 MB/s ingress + 30 MB/s egress + 5K partitions) plus retention storage plus data-in/data-out. Confluent Freight (2024 announced GA 2025) is the BYOC + S3-native cheaper tier that competes head-on with WarpStream. Confluent Flink (managed Apache Flink SQL service) and Stream Governance layer above; eBPF-based broker-level metering. Tableflow (2024) writes Kafka topics directly to Apache Iceberg tables — Kafka topic → Iceberg table for analytics-without-ETL.
  • Redpanda — Vectorized Data Inc 2019 (Alexander Gallego ex-Akamai); Redpanda Data brand from 2022. C++ rewrite of the Kafka protocol using the Seastar framework (per-core sharded, thread-per-core, no JVM, no ZooKeeper from day one). Wire-compatible with Kafka client libraries — drop-in replacement for the consumer/producer API. Source-available Redpanda Community Edition (BSL with conversion to Apache 2.0 after 4 years) + Enterprise + Redpanda Cloud managed offering. Pricing per provisioned-CPU AKU (Active Kafka Unit). Highlights: lower p99 latency claims (sub-10ms vs Kafka’s typical 100ms+), single binary deployment, integrated Schema Registry + HTTP Proxy, Tiered Storage (S3-backed), Iceberg integration 2024 (Iceberg sink direct), Redpanda Connect (formerly Benthos, acquired Mar 2024 — stream-processing connector framework). Series C Apr 2022 100M Series A+B+C total) at ~$725M valuation. 2024 pivot to AI/agentic workload positioning — “the data plane for AI agents” tagline.
  • Apache Pulsar — Yahoo 2013 (open-sourced 2016); Apache top-level 2018; Apache 2.0. Separated compute and storage architecture from day one — brokers are stateless dispatchers; persistence delegated to Apache BookKeeper (which itself is a separately-Apache distributed log built by Yahoo for HDFS namenode HA). This separation allows independent scaling of broker count and storage capacity, and clean tiered storage to S3/GCS/Azure Blob via offloaders. Multi-tenant from day one: tenants → namespaces → topics; per-namespace policies for quotas, retention, replication, schema. Geo-replication is built-in (configure per namespace; bidirectional or unidirectional; no MirrorMaker analog needed). Pulsar Functions (lightweight stream processing inline in the broker — alternative to Flink for simple transforms). Pulsar IO (Kafka Connect analog). Releases: 2.10 (2022) transactions GA, 2.11 (2023), 3.0 LTS (2023) marks first LTS, 3.2 (2024) Picnic-protocol stream sharing improvements, 3.3 (2024), 3.4 / 4.0 (2025) continued. Tiered offload of cold segments via JCloud (S3/Azure/GCS).
  • StreamNative — Pulsar commercial behind-the-creators company founded 2019 by Sijie Guo (Pulsar PMC chair) + Jia Zhai (ex-Yahoo Pulsar team). Offers StreamNative Cloud (fully managed Pulsar on AWS/GCP/Azure), StreamNative Platform (self-managed enterprise), StreamNative Lakehouse (Iceberg integration). Pricing per active partition + storage + traffic. Series A 2021 $23M.
  • DataStax Luna Streaming — DataStax-built and -supported Pulsar distribution; DataStax (the Cassandra company) entered the streaming space 2019 with Astra Streaming (managed Pulsar) and then folded it into a single Astra data platform after IBM’s acquisition of DataStax announced Feb 2025.

Brokered message queues — the AMQP / JMS lineage

Traditional queues with brokered routing, acknowledgments, dead-letter queues, and consumer-driven delivery. The older but still extremely common pattern.

  • RabbitMQ — Rabbit Technologies 2007; SpringSource 2010 → VMware 2010 → Pivotal 2013 → VMware 2018 → Broadcom Nov 2023 (as part of the $61B VMware acquisition closed Nov 2023, restructured 2024). Apache 2.0; Mozilla Public License 2.0 for some plugins. Implements AMQP 0.9.1 (advanced message queuing protocol — the lingua franca of broker-based messaging). Also speaks STOMP, MQTT, AMQP 1.0 (plugin). Routing model: producer → exchange (direct/topic/fanout/headers) → binding → queue → consumer. Storage: per-queue Mnesia + on-disk for persistent messages.
    • Classic queues — the original, single-node mastered with mirroring (deprecated).
    • Quorum queues — Raft-based replicated queues since 3.8 (2019), recommended default in 4.0.
    • Streams — append-only log type since 3.9 (2021); Kafka-style replay; consumed by librabbitmq-stream-client or AMQP 1.0.
    • 4.0 release (2024) — quorum queues default, classic queue v1 removed, streams stable, AMQP 1.0 native (no plugin), Khepri (Ra-Raft replicated metadata store replacing Mnesia) opt-in. Managed: CloudAMQP (84codes; Sweden; managed RabbitMQ + LavinMQ + Karafka), AWS Amazon MQ (managed RabbitMQ + ActiveMQ), Azure Cluster on VMs.
  • Apache ActiveMQ — Apache 2.0; Sun → Apache 2004. The original Java-based AMQP/JMS broker. ActiveMQ Classic (5.x branch; still widely deployed, maintenance-only) plus ActiveMQ Artemis (next-gen broker rewritten from HornetQ donated by Red Hat 2014; the 6.x line; multi-protocol AMQP 1.0 + MQTT + STOMP + OpenWire + Core; high-performance journal). Adoption declining outside legacy enterprise; managed by AWS Amazon MQ.
  • IBM MQ (formerly MQSeries, then WebSphere MQ, then IBM MQ) — IBM 1993. Closed-source commercial; the entrenched enterprise messaging platform for banks, airlines, governments, and insurance. Speaks JMS, AMQP 1.0, MQTT, native MQI. Used as the transport beneath core banking systems globally.
  • TIBCO EMS — TIBCO Software (now Cloud Software Group after Vista Equity 2022 merger with Citrix); JMS-compliant enterprise messaging. Used heavily in financial trading floors and telco. Maintenance plateau.
  • Solace PubSub+ — Solace Corp; Toronto-based; event mesh positioning — a logical distributed event broker fabric spanning data centers and clouds. Speaks AMQP 1.0, MQTT 3.1.1/5, REST, JMS, SMF (Solace Message Format), Kafka. Used at JPMorgan Chase, RBC, SAP, Boeing for cross-system event distribution. Solace Cloud managed.
  • Apache RocketMQ — Alibaba 2012 (original developer Feng Jia); Apache 2017; Apache 2.0. Financial-grade ordered messaging — strictly ordered messages within a topic when needed; deep Alibaba production heritage running double-eleven (Singles Day) at peak >55M TPS in 2024. Topics → message queues (partitions) → producers + consumer groups. Transactional messages (two-phase commit pattern). RocketMQ 5.x (2022+) unifies the stream + queue paradigms — same data accessible as stream (offset-based replay) or queue (consumer-group ack). Cloud offering at Alibaba Cloud (now Apsara MQ for RocketMQ); recently pulled deeper into Alibaba Cloud roadmap rather than community-led.

Brokerless and library-style — no central server

Smaller, lower-level patterns where the application embeds the messaging layer directly.

  • ZeroMQ (0MQ, ØMQ) — Pieter Hintjens et al. 2007; MPL-2.0 (formerly LGPL). Not a broker, not a server — a socket library with eight standard patterns (PUB/SUB, REQ/REP, PUSH/PULL, DEALER/ROUTER, PAIR/PAIR, plus thread-safe RADIO/DISH and CLIENT/SERVER added in 4.x). Provides reliable transport over TCP, IPC, inproc, multicast. Used as the messaging substrate inside many other systems (IPython kernel protocol, Mongrel2, parts of OpenStack, scientific computing pipelines). After Hintjens’ death 2016 the project continued under iMatix.
  • nanomsg + nng — Garrett D’Amore (ex-Joyent; Solaris kernel engineer) 2014; positioned as ZeroMQ successor with cleaner code, MPL-2.0, no STREAM transport but with NNG (nanomsg-next-generation) reaching 1.x stable. Patterns SURVEYOR/RESPONDENT, BUS, PUSH/PULL, etc. Smaller community than ZeroMQ but cleaner C/Go bindings.
  • NATS — Synadia Communications (Derek Collison; ex-Apcera, ex-VMware Cloud Foundry messaging architect); originally written 2011 in Ruby, rewritten in Go 2014; Apache 2.0. Designed for cloud-native, sub-millisecond latency, brokerless-feeling clustering. Single small binary (15 MB), starts in milliseconds. Subject-based routing (orders.us.east.> hierarchical wildcard). Core delivery: at-most-once pub/sub. JetStream (added 2020): durable persistence, consumer groups, replay-by-offset, deduplication windows, KV store, Object Store. NATS 2.x (2024) is the durable streaming-capable line; recent releases: 2.10 (2024), 2.11 (2025) bringing observability and per-account leafnode improvements. Connect via NATS protocol, MQTT bridge, WebSocket bridge, JetStream API. Multi-tenant via accounts + decentralized JWT-based auth. Operated by Synadia Cloud managed ($22M Series A 2024). Used in Cloud Foundry, Pinterest, Walmart, NetApp.

Hyperscaler-managed streaming and queueing

The cloud-vendor offerings — these are where most net-new deployments land after 2020 because the operational burden of self-managed Kafka is significant.

AWS

  • Amazon SQS (Simple Queue Service) — launched July 2006; the oldest AWS service (predates S3 by 4 months). Fully managed, transparent durability, three replicas across AZs. Two flavors: Standard (at-least-once, best-effort ordering, unlimited throughput) and FIFO (exactly-once processing within a message group, strict ordering, 3000 msg/s with batching or 300 without). Pricing: 0.50 per million (FIFO).
  • Amazon SNS (Simple Notification Service) — pub/sub topic broker; subscribers can be SQS queues, Lambda, HTTPS endpoints, email, SMS, mobile push. Tightly coupled with SQS for the canonical SNS-fanout-to-SQS pattern.
  • Amazon EventBridge — formerly CloudWatch Events; rebranded 2019. Event-bus with schema registry, rules that match events and route to targets (Lambda, SQS, SNS, Step Functions, API destinations, partner events). Schemas discovered automatically by inspecting traffic. Pipes (2022) bridges sources to targets with filtering/enrichment.
  • Amazon Kinesis — AWS’s Kafka-equivalent suite:
    • Kinesis Data Streams — partitioned log; shards 1 MB/s ingest each; up to 1 year retention.
    • Kinesis Data Firehose — managed delivery to S3/Redshift/OpenSearch/Splunk; serverless, no shard management.
    • Kinesis Data Analytics — managed Apache Flink (rebranded Amazon Managed Service for Apache Flink in 2024).
    • Kinesis Video Streams — H.264/H.265 video ingestion for ML pipelines and security cameras.
  • Amazon MSK (Managed Streaming for Kafka) — managed Apache Kafka. MSK Provisioned (broker EC2 instances) since 2019; MSK Serverless since 2022 (pay per ingress/egress, no broker management). MSK Connect for managed Kafka Connect connectors.

Google Cloud

  • Google Cloud Pub/Sub — fully managed, planet-scale, exactly-once delivery (since 2020 GA), ordering keys (since 2020). Two delivery models: push (HTTPS endpoint) and pull (client library). Filtering at subscription level. Pricing per message TB.
  • Pub/Sub Lite — zonal, lower-cost, Kafka-equivalent priced (per provisioned partition + storage); positioned for high-throughput log-style workloads where the cross-region pricing of Pub/Sub is overkill.
  • Cloud Dataflow — managed Apache Beam runner for unified batch + streaming.

Azure

  • Azure Service Bus — enterprise broker; AMQP 1.0 native; queues + topics + subscriptions. JMS 2.0 support via Premium tier. Sessions for in-order processing.
  • Azure Event Hubs — Azure’s Kafka equivalent. Stream-style partitioned log. Kafka-compatible endpoint (speaks Kafka 1.0+ wire protocol since 2018), making Event Hubs a drop-in replacement for self-hosted Kafka for many use cases. Event Hubs Schema Registry (2021). Capture to Azure Blob Storage or Data Lake. Dedicated clusters for high throughput.
  • Azure Event Grid — EventBridge-equivalent event router; subscribes to Azure resource events (blob created, VM started); supports CloudEvents schema natively; MQTT broker mode added 2023 (positions as IoT message broker).

The S3-native Kafka wave — 2023-2025 reset

The most important shift in the streaming space in years: rebuild Kafka-compatible brokers with object storage (S3) as the primary durability layer rather than as a tiered-storage offload. This collapses inter-AZ replication traffic (the largest cost on AWS for cross-AZ Kafka) and changes the cost curve dramatically.

  • WarpStream — Richard Artoul + Ryan Worl (both ex-Datadog Husky engineers) launched 2023; Apache 2.0 agent + commercial control plane. Zero local disk, no inter-AZ replication, no SSDs, no broker storage. Producers write directly to S3 (batched into “files” of typically 4 MB); the metadata service tracks offsets and assigns reads. Latency target ~500 ms (vs Kafka’s 10 ms) — explicit trade-off for ~80% lower cost than self-managed MSK. Wire-compatible Kafka API. BYOC (bring-your-own-cloud) deployment — agents run in the customer’s VPC, only metadata goes through WarpStream’s control plane. Confluent acquired WarpStream Sep 9, 2024 for $220M (announced); rebranded as Confluent Freight for the S3-native serverless tier.
  • BufStream — Buf Technologies 2024; from the team behind buf.build (Protocol Buffers tooling — Buf Schema Registry, buf CLI, ConnectRPC). S3-native + Schema Registry-native + Protobuf-first. Differentiates from WarpStream by integrating schema enforcement directly into the broker rather than as a separate Schema Registry service. Apache 2.0 broker + paid hosted control plane.
  • Confluent Freight (2024, GA 2025) — Confluent’s own S3-native tier, post-WarpStream-acquisition. Same model as WarpStream but rolled into Confluent Cloud billing.
  • AWS MSK Serverless Express Brokers — AWS’s response (announced Nov 2024); reduces replication overhead and offers faster broker scaling but still uses EBS-backed storage rather than pure S3.

This category effectively makes “Kafka as cheap as S3 + compute” the new baseline, and is reshaping pricing across the industry.

Stream processors — Flink, Spark, Beam, ksqlDB, and the SQL-streaming wave

The compute layer that reads from the brokers above and produces transformed/aggregated/joined outputs. Stateful, windowed, exactly-once-aware processing.

  • Apache Flink — original at TU Berlin / Stratosphere project 2009; Apache top-level 2014; Apache 2.0. The reference distributed stream-processor. Native streaming model (not micro-batch) with event-time semantics, watermarks, exactly-once via distributed snapshots (Chandy-Lamport), keyed state, windowed aggregations, joins, broadcast state, async I/O. Two main APIs: DataStream (Java/Scala/Python — low-level) and Flink SQL (declarative, increasingly the default). Stateful Functions (StateFun) extension. Flink CDC (Change Data Capture from Postgres/MySQL/MongoDB/Oracle — donated by Alibaba, joined Apache Flink in 2024).
    • 1.17 (2023) PyFlink improvements, RocksDB upgrades.
    • 1.18 (2023) generic upsert/merge improvements.
    • 1.19 (2024) dynamic configuration changes, async sink improvements, batch performance.
    • 1.20 (2024) Materialized table support; Flink SQL improvements.
    • 2.0 (2025) state v2 with disaggregated state (decouple compute from state storage), DataStream API 2.0, ForSt RocksDB-derived state backend for tiered local + remote state. Managed offerings: Confluent Flink (Confluent Cloud’s managed Flink SQL service, ex-Immerok acquisition Jan 2023), Amazon Managed Service for Apache Flink (renamed from Kinesis Data Analytics 2024), Aiven for Apache Flink, Ververica Platform (Ververica = original Flink creators’ company, sold to Alibaba 2019), Decodable (multi-tenant managed Flink + Debezium CDC; founded by Eric Sammer ex-Splunk).
  • Apache Spark Streaming + Structured Streaming — UC Berkeley AMPLab 2009; Apache 2014; Apache 2.0. The original Spark Streaming (DStream) was micro-batch; Structured Streaming (added Spark 2.0 2016) is the modern API, treats stream as unbounded table, supports event-time + watermarks + windowed aggregates. Still micro-batch under the hood (vs Flink’s true streaming). Continuous processing mode (since 2.3, experimental) reaches lower latency. Dominant deployment is Databricks managed Spark — Databricks IPO speculation 2025; ~$60B private valuation Series J. Spark 4.0 (2025) further unifies batch + streaming + ML.
  • Apache Beam — Google donated Dataflow SDK to Apache 2016; Apache 2.0. Portable unified batch + streaming model — write once in Java/Python/Go, run on Google Cloud Dataflow, Flink, Spark, Samza, Apex, or Direct (testing). Concepts: PCollection, PTransform, Pipeline, ParDo, GroupByKey, Window, Trigger. Most heavily used on its native runner (Google Cloud Dataflow) where it transparently scales.
  • ksqlDB — Confluent 2017 (formerly KSQL); Confluent Community License; built on Kafka Streams library. Continuous SQL queries against Kafka topics, materialized state via RocksDB, push and pull queries. Confluent Cloud’s managed ksqlDB has been deprioritized 2024 in favor of Confluent Flink, but the standalone ksqlDB remains supported.
  • Kafka Streams — Confluent-led client library (Apache 2.0; in Kafka core repo since 0.10). Embedded stream processor running in your application process — no separate cluster. Topology DSL + Processor API; state stores backed by RocksDB; exactly-once semantics via Kafka transactions. The right choice when you don’t want a separate stream-processing cluster.
  • Materialize — Materialize Inc. 2019; founders Frank McSherry (Cambridge/Microsoft Research; author of differential dataflow), Arjun Narayan, Nikhil Benesch. BSL-licensed. Incrementally-maintained materialized views in Postgres-wire-protocol SQL. Built atop Timely Dataflow + Differential Dataflow (McSherry’s open-source Rust libraries) which together provide formally-correct incremental view maintenance. v0.27 (2024) brought emit-changes-at-source improvements; v1.0 (2024) marked production-readiness. Materialize Cloud managed service. Pricing per compute-unit-hour. Differentiated by correctness guarantees — the answer at any moment matches the equivalent batch SQL exactly.
  • RisingWave — Singularity Data 2022 (Yingjun Wu ex-AWS Redshift + ex-IBM Streams); Apache 2.0. Postgres-wire-protocol streaming database in Rust. Positioned as Flink alternative for SQL-heavy streaming workloads. State stored in S3 + local cache. Materialized views, sinks to Kafka/Postgres/Iceberg. 2.0 (2024) brought UDFs in WASM, Iceberg sink improvements. RisingWave Cloud managed.
  • Decodable — multi-tenant managed Flink + Debezium CDC, founded 2021 by Eric Sammer.
  • Imply Polaris — managed Apache Druid + Flink for real-time analytics (Imply Data; Druid creators).
  • Apache Storm — Twitter 2011 → Apache 2014; declining; replaced by Flink in nearly all greenfield deployments.
  • Apache Samza — LinkedIn 2014; partner to Kafka but never gained Flink-level traction outside LinkedIn.
  • Hazelcast Jet (now Hazelcast Platform) — Hazelcast Inc.; stream processing co-located with in-memory data grid.

Specialized brokers and niche systems

  • Memphis.dev → Superstream — Memphis was a Kafka simplification play 2022-2023; pivoted to Superstream mid-2024 as a Kafka cost-optimization layer rather than a broker. Original Memphis broker repo is in maintenance-only.
  • Apache Kafka REST Proxy — Confluent-built HTTP gateway in front of Kafka; useful for languages without a good Kafka client library.
  • Apache Camel — Apache 2.0; routing and mediation engine implementing the 60+ Enterprise Integration Patterns (Gregor Hohpe). Adapters for ~350 systems. Used to bridge brokers, files, FTP, SaaS APIs.
  • MQTT brokers — for IoT specifically: Mosquitto (Eclipse Foundation), EMQX (EMQ Tech, China; clustered MQTT 5.0 broker; EMQX 5.0 2022 with Mria distributed engine, EMQX 5.7+ with HTTP/gRPC bridge), HiveMQ (HiveMQ GmbH, Germany; enterprise MQTT; HiveMQ Cloud), VerneMQ (Erlang; declining).
  • LavinMQ (84codes) — RabbitMQ-compatible AMQP broker rewritten in Crystal for higher throughput, smaller footprint; 84codes’ own answer to Broadcom-era RabbitMQ concerns.
  • Apache ActiveMQ Artemis — see above; Red Hat AMQ Broker is the supported distribution.
  • Solace PubSub+ Event Mesh — see above.

CDC — change data capture as a streaming source

A cross-cutting category that increasingly defines real-time data architecture: stream the changes from an OLTP database as events into Kafka/Pulsar/Kinesis.

  • Debezium — Red Hat / open source; Apache 2.0. Postgres logical replication, MySQL binlog, MongoDB oplog, Oracle LogMiner / XStream, SQL Server CT, Cassandra commitlog, Vitess. Runs as Kafka Connect source connectors. Embedded mode for non-Kafka use.
  • Flink CDC — donated by Alibaba; runs Debezium-style sources directly inside Flink jobs (no Kafka Connect required).
  • AWS DMS (Database Migration Service) — managed CDC + initial load.
  • GCP Datastream — managed CDC for Oracle, MySQL, Postgres → BigQuery / GCS.
  • Striim, Fivetran HVR (acquired 2021), Qlik Replicate (formerly Attunity), Airbyte CDC, Estuary Flow (Materialize-team adjacent; real-time CDC + transforms) — the broader commercial CDC market.

Schema registries and serialization

  • Confluent Schema Registry — Confluent Community License (pre-2025), now Confluent Source-Available; Avro + JSON Schema + Protobuf. The de facto standard.
  • Apicurio Registry — Red Hat / Apache 2.0 alternative; same Avro/JSON/Protobuf coverage plus AsyncAPI.
  • AWS Glue Schema Registry — AWS managed; integrates with MSK + Kinesis.
  • Buf Schema Registry (BSR) — Buf Technologies; Protobuf-first; integrates with BufStream.
  • Karapace — Aiven; Apache 2.0; Schema Registry API-compatible.

Serialization formats commonly registered: Avro (compact, schema-required, native to Kafka ecosystem), Protobuf (Google’s; widely used in microservices), JSON Schema (most permissive, largest payloads), MessagePack (rare in streaming), Thrift (Facebook’s; declining).

  1. Kafka-API as the de facto streaming interface — Redpanda, Azure Event Hubs, WarpStream, BufStream, Confluent Freight all speak it. Native non-Kafka APIs (Pulsar binary, Kinesis HTTP, Event Hubs AMQP) are now secondary at most vendors.
  2. S3-native architectures collapsing the inter-AZ-replication cost line — WarpStream’s $220M acquisition by Confluent (Sep 2024) marks the category coming of age.
  3. Tiered storage everywhere — Kafka 3.5+, Pulsar (since launch), Redpanda, Event Hubs (Capture), Confluent Cloud all offload cold data to S3/object storage; reduces broker disk pressure and enables longer retention cheaply.
  4. Kafka topics → Iceberg tables natively — Confluent Tableflow (2024), Redpanda Iceberg sink, AutoMQ + Iceberg, BufStream Iceberg — eliminating the historical pipeline (Kafka → ETL → warehouse) for analytical consumers.
  5. Flink SQL eating Kafka Streams + ksqlDB in managed cloud deployments — Confluent Flink, MSK Serverless + Managed Flink, GCP Dataflow + Flink.
  6. Schema enforcement at broker — Confluent Stream Governance, BufStream Protobuf-first, Redpanda Cloud schema validation, Azure Event Hubs Schema Registry.
  7. Queue semantics inside Kafka — KIP-932 “Queues for Kafka” + share groups bringing classic work-queue patterns to the log.
  8. MQ → event-stream consolidation — many enterprises now standardize on a single Kafka-API platform that covers both work-queue and event-log patterns, displacing legacy MQ.
  9. AI/agentic workload positioning — Redpanda, NATS, Kafka all marketing toward AI agent communication; stream-of-tool-calls and stream-of-events for agents replacing point-to-point HTTP loops.
  10. Exactly-once becoming routine — what was a 2017 differentiator (Kafka EOS) is now table stakes across Pulsar, Flink, Materialize, RisingWave, Cloud Pub/Sub.

Connectors and integration — the Kafka Connect ecosystem and friends

The “moving data into and out of brokers” layer is its own substantial category.

  • Kafka Connect — bundled with Apache Kafka since 0.9 (2015); plugin framework with source + sink connector model. 200+ connectors across the Confluent Hub, including JDBC, S3, Snowflake, BigQuery, Salesforce, MongoDB, Elasticsearch, HDFS, Debezium (CDC for Postgres / MySQL / Oracle / SQL Server / MongoDB). Self-managed via Connect worker JVMs; managed by Confluent Cloud Connectors, AWS MSK Connect, Aiven Kafka Connect.
  • Apache NiFi — NSA → Apache 2014; flow-based programming for data routing. Drag-and-drop UI for data movement. Used heavily in government, telco, healthcare for ETL pipelines that combine batch + streaming. NiFi 2.0 (2024) modernized the stack with Java 21 + improved provenance + StateLess NiFi.
  • Benthos (now Redpanda Connect) — Ash Burlaczenko 2017 → Jeffail → Redpanda Data acquired Mar 2024. Single-binary stream-processing connector with 200+ inputs/outputs/processors written declaratively in YAML. Apache 2.0 (core) + Enterprise. Direct competitor / alternative to Kafka Connect; runs as a sidecar rather than requiring a worker cluster.
  • Fluentd + Fluent Bit — Treasure Data → CNCF; log routing daemons that often bridge log sources to Kafka/Kinesis/Pulsar. Fluent Bit is the embedded-C version for edge.
  • Logstash — Elastic; the original ELK-stack ingest tool; declining vs Fluent Bit + Vector.
  • Vector — Datadog (acquired Timber.io 2021 $40M); Rust-written observability data router; Apache 2.0; common as a Kafka/Pulsar producer for log + metric streams.
  • Telegraf — InfluxData; metrics collection agent with Kafka output.

Use-case selection

Choosing among the above is rarely a clean cut — most production architectures use 2-4 of these in combination. The rough decision tree:

NeedReach for
Replayable event log, high throughput, ecosystemKafka (Confluent or MSK)
Same as Kafka but lower cost on cloudWarpStream / Confluent Freight / BufStream
Same as Kafka but lower latency, simpler opsRedpanda
Multi-tenant + geo-replication built inPulsar
Simple work queue, point-to-pointSQS / RabbitMQ
Fanout pub/sub, low latencyNATS / SNS / EventBridge
Ordered processing within keyKafka / Kinesis / FIFO SQS / RocketMQ
In-app embedded stream processingKafka Streams
Stateful complex streaming SQLFlink (Confluent Flink, managed)
Always-correct materialized viewsMaterialize
Streaming Postgres-wire SQLRisingWave / Materialize
IoT device trafficEMQX / HiveMQ / Mosquitto (MQTT)
Brokerless library-style IPCZeroMQ / nanomsg
Cross-system event mesh enterpriseSolace PubSub+
Legacy bank / airline / mainframeIBM MQ

Operational considerations

Beyond the platform choice, several cross-cutting concerns drive production design:

  1. Partition count sizing — Kafka and Kinesis partition count is the unit of parallelism and the unit of ordering. Too few partitions: consumer throughput caps out. Too many: rebalances are slow, ZooKeeper / KRaft pressure grows, metadata bloats. Rule of thumb: 10× the expected peak consumer count, capped at 4096 per topic for KRaft clusters.
  2. Consumer-group rebalancing — historical Kafka stop-the-world rebalancing was a major operational pain. Incremental cooperative rebalancing (KIP-429, Kafka 2.4) and static membership (KIP-345) reduced this. Pulsar’s separation of subscriptions from consumers gives it a different (often easier) model.
  3. Schema evolution — backward-compatible (consumers can read new) vs forward-compatible (producers can write old) vs full vs none. Schema Registry enforces this at the wire layer; without it, every producer + consumer change risks an outage.
  4. Tiered storage tuning — Kafka 3.5+‘s tiered storage defaults to a local.retention.ms and local.retention.bytes that controls how much sits on local disk vs S3. Hot reads from local disk are fast; cold reads from S3 are 10-100× slower; tune based on consumer lag patterns.
  5. Exactly-once cost — Kafka transactional producer + read-process-write consumer adds ~10% throughput overhead vs at-least-once. Worth it for financial / billing pipelines; usually overkill for telemetry.
  6. Dead-letter queue (DLQ) handling — SQS, RabbitMQ, Pulsar, Service Bus all have first-class DLQs. Kafka has none — convention is a separate topic and a DLT handler in the consumer.
  7. Backpressure — push-based (RabbitMQ, NATS, MQTT) needs flow control; pull-based (Kafka, Kinesis, Pulsar consumer) is naturally backpressured but can lag indefinitely. Monitor lag aggressively.
  8. Compaction vs retention — Kafka log compaction keeps the latest value per key forever (good for CDC + config topics); time-based retention deletes everything older than N days. Don’t accidentally compact a topic with no key — you’ll silently drop messages.

Adjacent

Graph View