Apache Pulsar

Apache Pulsar is a cloud-native distributed messaging and streaming platform with built-in multi-tenancy, geo-replication, and tiered storage. In this Apache Pulsar review, we examine how Pulsar's architecture compares to Apache Kafka and whether its unique features justify the operational complexity for modern data infrastructure teams.

Overview

Apache Pulsar provides a distributed pub-sub messaging system where messages are stored in Apache BookKeeper (a distributed log storage system) rather than on the broker nodes themselves. This separation of serving and storage is Pulsar's key architectural differentiator: brokers are stateless and can be added, removed, or restarted without data movement, while BookKeeper handles durable storage with configurable replication. Pulsar supports both traditional messaging patterns (queues, pub/sub) and streaming (log-based consumption with replay). The platform includes built-in multi-tenancy with namespace isolation, built-in geo-replication across data centers, tiered storage for offloading old data to S3/GCS/HDFS, and Pulsar Functions for lightweight serverless stream processing. Pulsar is used in production at Yahoo (2M+ topics), Tencent, Verizon Media, Splunk, and Iterable.

Key Features and Architecture

• Separated compute and storage — stateless brokers handle serving while Apache BookKeeper handles durable storage, enabling independent scaling of throughput and storage capacity
• Built-in multi-tenancy — native tenant, namespace, and topic isolation with per-tenant quotas, authentication, and authorization — no need for separate clusters per team
• Geo-replication — built-in cross-datacenter replication with configurable replication policies (sync, async, active-active) without external tools like Kafka MirrorMaker
• Tiered storage — automatically offload older messages to S3, GCS, HDFS, or Azure Blob Storage while keeping recent data on BookKeeper for fast access, reducing storage costs by 60–80%
• Pulsar Functions — lightweight serverless compute framework for stream processing directly within Pulsar, without needing Spark or Flink for simple transformations
• Schema Registry — built-in schema enforcement and evolution (Avro, Protobuf, JSON) with compatibility checks on produce
• Multi-protocol support — native Pulsar protocol plus Kafka protocol compatibility (KoP), AMQP, and MQTT protocol handlers
• Message deduplication — built-in exactly-once semantics with broker-side deduplication for producers

Pricing and Licensing

Apache Pulsar is free and open-source under the Apache 2.0 license. Costs come from infrastructure and managed services:

• Apache Pulsar: $0 (Apache 2.0) — all features including geo-replication and multi-tenancy are free
• StreamNative Cloud (managed Pulsar): From $0.10/hour per Pulsar Unit ($75/month minimum), scaling based on throughput and storage
• Self-hosted on AWS/GCP/Azure: $500–$5,000/month for a production cluster (3 brokers + 3 BookKeeper nodes + ZooKeeper)
• StreamNative Enterprise: Custom pricing with support, security features, and SLA guarantees

For comparison, Kafka managed services: Confluent Cloud from $0.004/partition-hour (first $400 free), Amazon MSK from $0.21/broker-hour (~$150/month), Redpanda Cloud from $0.08/partition-hour.

Ideal Use Cases

• Multi-tenant messaging platforms — organizations running a shared messaging infrastructure for multiple teams or business units where Kafka would require separate clusters per tenant, increasing operational overhead and infrastructure costs significantly
• Global applications with geo-replication — systems that need active-active or active-passive replication across data centers or cloud regions with built-in conflict resolution, without the complexity of configuring and maintaining Kafka MirrorMaker 2
• Cost-optimized long-term message retention — use cases requiring weeks or months of message retention where tiered storage (offloading to S3 at $0.023/GB vs BookKeeper at $0.10/GB) provides 60–80% storage cost savings compared to keeping all data on broker disks
• IoT and edge messaging — high-topic-count scenarios (100K+ topics) where Pulsar's architecture handles topic creation and management more efficiently than Kafka's partition-based model

Pros and Cons

Pros:

• Separated compute and storage enables independent scaling — add throughput without adding storage and vice versa
• Built-in multi-tenancy eliminates the need for separate clusters per team, reducing operational overhead
• Native geo-replication is simpler and more reliable than Kafka MirrorMaker or Confluent Replicator
• Tiered storage reduces long-term retention costs by 60–80% by offloading to object storage
• Kafka protocol compatibility (KoP) allows gradual migration from Kafka without rewriting producers and consumers
• Pulsar Functions provide lightweight stream processing without deploying Spark or Flink for simple transformations

Cons:

• Higher operational complexity — requires managing brokers, BookKeeper nodes, and ZooKeeper (3 separate distributed systems)
• Smaller ecosystem — fewer connectors, client libraries, and third-party integrations compared to Kafka's massive ecosystem
• Smaller community — fewer tutorials, Stack Overflow answers, blog posts, and conference talks than Kafka
• BookKeeper expertise required — debugging storage issues requires understanding BookKeeper internals, which is a niche skill
• StreamNative Cloud is less mature than Confluent Cloud — fewer regions, fewer integrations, and less enterprise tooling
• Throughput can be lower than Kafka for simple pub/sub — the storage separation adds latency for the highest-throughput use cases

Who Should Use Apache Pulsar

Apache Pulsar is best suited for platform engineering teams at mid-to-large organizations that need a shared messaging infrastructure serving multiple teams with isolation guarantees. Organizations with global deployments requiring built-in geo-replication will benefit from Pulsar's native cross-datacenter support. Companies with long message retention requirements (weeks to months) will see significant cost savings from tiered storage. Teams already running Kafka who are hitting multi-tenancy or geo-replication pain points should evaluate Pulsar as a migration target using the Kafka protocol compatibility layer. Small teams or organizations with simple messaging needs should stick with Kafka or Redpanda for the larger ecosystem and simpler operations.

Alternatives and How It Compares

• Apache Kafka — the industry standard for event streaming with the largest ecosystem (1,000+ connectors), community, and managed service options. Better for most use cases due to ecosystem size. Lacks native multi-tenancy and geo-replication. Free, Confluent Cloud from $0.004/partition-hour.
• Redpanda — Kafka-compatible streaming platform written in C++ with no JVM or ZooKeeper dependency. Simpler to operate than both Kafka and Pulsar. Better for teams wanting Kafka compatibility with lower operational overhead. Free, Cloud from $0.08/partition-hour.
• Amazon MSK — managed Kafka on AWS with minimal operational overhead. Better for AWS-native teams who want Kafka without cluster management. From $0.21/broker-hour.
• RabbitMQ — traditional message broker for task queues and request-reply patterns. Better for application messaging; not designed for high-throughput event streaming. Free (open-source).
• NATS — lightweight, high-performance messaging system with JetStream for persistence. Better for microservice communication; less feature-rich for event streaming. Free (open-source).

Conclusion

Apache Pulsar is a technically impressive messaging platform with genuine architectural advantages over Kafka: separated compute and storage, built-in multi-tenancy, native geo-replication, and tiered storage. These features matter for large organizations running shared messaging infrastructure across multiple teams and data centers. However, the higher operational complexity (three distributed systems to manage), smaller ecosystem, and smaller community mean Pulsar is only the right choice when its unique features are actually needed. For most messaging and streaming use cases, Kafka's ecosystem and community make it the safer default. Choose Pulsar when multi-tenancy, geo-replication, or tiered storage are genuine requirements, not just nice-to-haves.