You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "zfchai (via GitHub)" <gi...@apache.org> on 2025/07/30 05:53:16 UTC
[I] Suggest removing ZooKeeper and streamlining the architecture [pulsar]
zfchai opened a new issue, #24584:
URL: https://github.com/apache/pulsar/issues/24584
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
### Problem Statement
Apache Pulsar currently relies on external ZooKeeper for metadata storage and management, which introduces several significant drawbacks:
1. **Performance Bottlenecks**
- ZooKeeper becomes a performance bottleneck under high concurrency scenarios
- Network latency between Pulsar and ZooKeeper affects overall system performance
- Additional network hops for metadata operations
2. **Operational Complexity**
- Requires maintaining two separate distributed systems (ZooKeeper + Pulsar)
- Complex configuration management coordinating both systems
- Difficult troubleshooting as issues can occur in either system
3. **Scalability Challenges**
- ZooKeeper's scalability limitations impact Pulsar's growth
- Scaling requires coordination between two systems
### Proposed Solution
Implement a built-in metadata management system within Pulsar to replace external ZooKeeper dependency:
#### Key Benefits
1. **Simplified Architecture**
- Unified metadata management reduces external dependencies
- Single system to maintain and monitor
- Easier deployment and configuration
2. **Performance Improvements**
- Eliminate network hops for metadata operations
- Optimize storage engine specifically for messaging use cases
- Better integration between metadata and messaging layers
3. **Enhanced Controllability**
- Deep integration between metadata storage and messaging system
- More flexible scaling and optimization strategies
- Customizable metadata management policies
### Implementation Approach
1. **Native Metadata Storage Engine**
- Develop a dedicated metadata storage engine within Pulsar
- Support clustering and high availability configurations
- Implement efficient metadata caching mechanisms
2. **Gradual Migration Path**
- Provide compatibility mode for smooth migration
- Maintain optional ZooKeeper support during transition period
- Offer migration tools and documentation
3. **Performance Optimization**
- Optimize storage structures for messaging scenarios
- Implement advanced caching and indexing strategies
- Ensure consistency and reliability of metadata operations
### Expected Outcomes
- **Reduced Operational Overhead**: Single system management instead of coordinating multiple systems
- **Improved Performance**: Faster metadata operations and reduced latency
- **Better Scalability**: Independent scaling of metadata and messaging components
- **Enhanced Reliability**: Simplified failure modes and easier recovery procedures
This enhancement aligns with modern distributed system design principles and would significantly improve Pulsar's usability, performance, and operational efficiency.
### Solution
_No response_
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Suggest removing ZooKeeper and streamlining the architecture [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #24584: Suggest removing ZooKeeper and streamlining the architecture
URL: https://github.com/apache/pulsar/issues/24584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Suggest removing ZooKeeper and streamlining the architecture [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on issue #24584:
URL: https://github.com/apache/pulsar/issues/24584#issuecomment-3135204149
Regarding the performance bottlenecks, does it come from a test result of real workload? The metadata operations are not in the critical paths like produce or consume. In addition, there is a [metadata cache](https://github.com/apache/pulsar/blob/master/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/cache/impl/MetadataCacheImpl.java) widely used internally.
I admit that ZooKeeper has its limitation, that's why https://github.com/oxia-db/oxia is born. But implement a built-in metadata management system might not be a good idea. I can hardly see Pulsar users suffering ZooKeeper related issues in real production environments.
This suggestion looks very similar to Kafka's KRaft proposal. However, it's different in Pulsar. Kafka has more heavy metadata. For example, a dynamically changed ISR list maintained for each topic. Before introducing KRaft, there is only one single centralized controller node in Kafka, which is not scalable. Well, now Kafka does not have a ZooKeeper cluster, but it has a KRaft controller cluster and its separated [KRaft Quorum Monitoring Metrics](https://kafka.apache.org/documentation/#kraft_quorum_monitoring).
Each Pulsar broker can serve any metadata related request, and Pulsar brokers don't communicate with each other like Kafka brokers. That's the main difference.
Actually, Pulsar can do the similar thing by renaming ZooKeeper as `pulsar-metadata` and `BookKeeper` as `pulsar-storage` so that it can claim that now Pulsar has only 1 component. But it's meaningless.
IMHO, developing a better metadata service or even using a RDS might be good, but it's not a important part from a practical perspective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Suggest removing ZooKeeper and streamlining the architecture [pulsar]
Posted by "zfchai (via GitHub)" <gi...@apache.org>.
zfchai commented on issue #24584:
URL: https://github.com/apache/pulsar/issues/24584#issuecomment-3138415625
@lhotari Can you provide a documentation of the latest version of Pulsar and Oxia cluster? This would be a great solution for many people who don't want to use bulky ZK. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Suggest removing ZooKeeper and streamlining the architecture [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #24584:
URL: https://github.com/apache/pulsar/issues/24584#issuecomment-3135719201
@zfchai You can already use [Oxia](https://github.com/oxia-db/oxia) with Pulsar to replace the usage of ZooKeeper as the metadata store of Pulsar and BookKeeper. The [Apache Pulsar Helm chart](https://github.com/apache/pulsar-helm-chart) has experimental support for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Suggest removing ZooKeeper and streamlining the architecture [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #24584:
URL: https://github.com/apache/pulsar/issues/24584#issuecomment-6388693138
Thanks @zfchai. Pulsar no longer requires ZooKeeper: Oxia can be the metadata store for both Pulsar and BookKeeper (PIP-335, https://github.com/apache/pulsar/blob/master/pip/pip-335.md, since 3.3.0). The Helm chart supports it with `components.zookeeper=false` and `components.oxia=true`, and PIP-454 adds a framework for migrating existing clusters off ZooKeeper. As discussed above, a metadata engine built into the broker is not the planned direction. Closing as completed. Gaps in the Oxia deployment docs can be reported in https://github.com/apache/pulsar-site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org