Apache Kafka

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, and mission-critical applications. In this Apache Kafka review, we examine how Kafka handles millions of events per second with fault tolerance and low latency, and how it compares to managed alternatives like Confluent Cloud, Amazon MSK, and Redpanda.

Overview

Apache Kafka was originally developed at LinkedIn in 2011 and donated to the Apache Software Foundation, where it became a top-level project. More than 80% of Fortune 100 companies use Kafka, with particularly strong adoption in manufacturing (10/10 top companies), insurance (10/10), energy and utilities (10/10), telecom (8/10), and banking (7/10).

Kafka's core abstraction is the distributed commit log: producers write events to topics, topics are split into partitions distributed across a cluster of brokers, and consumers read events in order. This architecture enables throughput of millions of messages per second with sub-10ms latency on commodity hardware. Events are persisted to disk and replicated across brokers, providing durability guarantees even when individual nodes fail.

As of Kafka 3.x, the project has moved toward removing its ZooKeeper dependency in favor of KRaft (Kafka Raft), a built-in consensus protocol that simplifies deployment and improves scalability. KRaft mode is production-ready as of Kafka 3.5 and eliminates the need to manage a separate ZooKeeper ensemble.

Key Features and Architecture

Publish-Subscribe Messaging

Producers write events to named topics, and consumers subscribe to topics via consumer groups. Each consumer group receives every event exactly once, enabling parallel processing across multiple consumers. Kafka guarantees ordering within a partition and supports exactly-once semantics (EOS) for end-to-end processing guarantees.

Partitioning and Replication

Topics are divided into partitions, each replicated across a configurable number of brokers (typically replication factor 3). Partitions enable horizontal scaling — adding partitions increases throughput linearly. If a broker fails, a replica is automatically promoted to leader, ensuring zero data loss with acks=all producer configuration.

Kafka Connect

A framework for streaming data between Kafka and external systems without writing custom code. Kafka Connect provides source connectors (ingest from databases, files, cloud services) and sink connectors (write to data warehouses, search indexes, object storage). The Confluent Hub hosts 200+ pre-built connectors for systems like PostgreSQL, MySQL, MongoDB, Elasticsearch, S3, BigQuery, Snowflake, and Salesforce.

Kafka Streams and ksqlDB

Kafka Streams is a Java client library for building stream processing applications directly on Kafka — no separate cluster required. It supports stateful operations like windowed aggregations, joins, and exactly-once processing. ksqlDB extends this with a SQL interface for stream processing, enabling analysts to write continuous queries like SELECT * FROM orders WHERE amount > 1000 EMIT CHANGES.

Schema Registry

The Confluent Schema Registry manages Avro, Protobuf, and JSON Schema definitions for Kafka topics, enforcing schema compatibility rules (backward, forward, full) to prevent breaking changes. This is critical for organizations with multiple producer and consumer teams who need to evolve data formats safely.

Tiered Storage

Introduced in Kafka 3.6 (KIP-405), tiered storage offloads older log segments from broker local disks to remote object storage (S3, GCS, Azure Blob). This decouples storage capacity from broker compute, enabling longer retention periods (weeks or months) without scaling broker disk. Confluent Cloud and Amazon MSK both support tiered storage.

Ideal Use Cases

Real-Time Event Streaming

Companies like Netflix, Uber, and LinkedIn use Kafka as the central nervous system for event-driven architectures. Every user action, system event, and metric flows through Kafka topics, enabling real-time dashboards, fraud detection, and personalization engines processing millions of events per second.

Change Data Capture (CDC)

Kafka Connect with Debezium captures row-level changes from databases (PostgreSQL, MySQL, SQL Server, MongoDB) and streams them to Kafka topics in real time. Downstream consumers can replicate data to warehouses, update search indexes, or trigger business workflows — all without modifying the source database.

Log Aggregation

Organizations replace traditional log collection systems (syslog, Flume) with Kafka to aggregate logs from thousands of application instances into centralized topics. Consumers route logs to Elasticsearch for search, S3 for archival, or stream processing applications for real-time alerting.

Microservices Communication

Kafka serves as an asynchronous message bus between microservices, decoupling producers from consumers. Services publish domain events to Kafka topics, and other services consume events they care about. This pattern enables independent scaling, fault isolation, and eventual consistency across service boundaries.

Pricing and Licensing

Apache Kafka itself is free and open-source under the Apache License 2.0. The cost of running Kafka comes from infrastructure and operations. Self-hosted and managed options:

Self-hosted Kafka requires significant operational expertise: broker configuration, partition rebalancing, monitoring, upgrades, and security. Many organizations start self-hosted and migrate to managed services as operational costs exceed the managed service premium.

Pros and Cons

Pros

• Extreme throughput — handles millions of messages per second with sub-10ms latency on commodity hardware
• Durability and fault tolerance — partition replication with configurable acks ensures zero data loss; automatic leader election on broker failure
• Massive ecosystem — 200+ Kafka Connect connectors, Kafka Streams, ksqlDB, Schema Registry, and integration with every major data tool
• Proven at scale — battle-tested at LinkedIn (7 trillion messages/day), Netflix, Uber, and 80%+ of Fortune 100
• Open source (Apache 2.0) — no licensing costs, full source code access, large community (27,000+ GitHub stars)
• KRaft mode — eliminates ZooKeeper dependency, simplifying deployment and reducing operational complexity

Cons

• Operational complexity — self-hosted Kafka requires expertise in broker tuning, partition management, monitoring, and upgrades; a dedicated team of 1–3 engineers is typical
• No built-in exactly-once by default — EOS requires careful configuration of enable.idempotence, transactional producers, and isolation.level=read_committed consumers
• JVM resource requirements — Kafka brokers run on the JVM and typically need 6–16GB heap per broker plus OS page cache for optimal performance
• Partition count limits — while Kafka scales well, clusters with 100,000+ partitions can experience increased leader election time and metadata overhead
• Rebalancing disruption — consumer group rebalances during scaling or failures can cause temporary processing pauses; cooperative rebalancing (KIP-429) mitigates but doesn't eliminate this

Alternatives and How It Compares

Confluent Platform / Confluent Cloud

Confluent, founded by Kafka's original creators, offers a commercial distribution with Schema Registry, ksqlDB, Confluent Control Center, and 200+ managed connectors. Confluent Cloud provides fully managed Kafka with serverless pricing starting at $0.0075/GB. For teams that want Kafka without operational overhead, Confluent is the most feature-complete managed option — but costs 3–5x more than self-hosted at scale.

Amazon MSK

Amazon MSK is AWS's managed Kafka service, handling broker provisioning, patching, and monitoring. MSK Serverless offers pay-per-use pricing for intermittent workloads. MSK is the natural choice for AWS-centric organizations but lacks Confluent's ksqlDB, Schema Registry management, and connector marketplace. MSK pricing starts lower than Confluent but operational features are more limited.

Redpanda

Redpanda is a Kafka-compatible streaming platform written in C++ (no JVM) that claims 10x lower tail latency and simpler operations. It's API-compatible with Kafka, so existing producers and consumers work without code changes. Redpanda eliminates ZooKeeper and JVM tuning complexity. The trade-off is a smaller ecosystem and community compared to Kafka's 12+ years of production hardening.

Apache Pulsar

Pulsar is an alternative open-source messaging platform with native multi-tenancy, geo-replication, and tiered storage. Pulsar separates compute (brokers) from storage (BookKeeper), enabling independent scaling. Pulsar has a smaller community than Kafka but offers features like built-in schema registry and functions (lightweight stream processing). StreamNative provides managed Pulsar.

Amazon Kinesis Data Streams

Kinesis is AWS's proprietary streaming service with simpler operations than Kafka but less flexibility. Kinesis pricing is per-shard ($0.015/shard-hour + $0.014/GB), and each shard handles 1MB/s ingress and 2MB/s egress. Kinesis is easier to operate than self-hosted Kafka but has lower throughput limits per shard and no ecosystem of connectors comparable to Kafka Connect.