You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/08/09 13:06:44 UTC
[GitHub] [pulsar] zengguan opened a new issue, #17021: [proxy] proxy support multi pulsar cluster
zengguan opened a new issue, #17021:
URL: https://github.com/apache/pulsar/issues/17021
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
In [PIP-121](https://github.com/apache/pulsar/issues/13315)，we support cluster level controlled cluster failover on client side。But I have some issue with this
- when we upgrade pulsar in future，it is more difficult to upgrade client than upgrade proxy。
- client side only support Java，We need to implement other programming language such as go，python。
- cluster level auto failover is great，but i want to implement controlled in topic level。it is more flexible and smaller influences。In this way we can migrate request step by step with topic level，and Observe whether there is an impact on the task associated with the topic。
### Solution
- [Idea] We propose an idea，which proxy support multi pulsar cluster，with specified configuration multi cluster。
- [Implement] In this mode，proxy can discover multi pulsar cluster with multi zookeeper connections。when we creating topic，we can specified Mapping between topics and clusters，such as TopicA -> Cluster A, Topic B -> Cluster B, store this metadata in zookeeper。If clients send lookup request to proxy with topicA，and the proxy will respond the specified cluster url (such as Cluster A)。
- [migration] for example，in cluster A we have 10 topics reading and writing., And TopicA -> Cluster A in proxy. I send the swith request(TopicA -> Cluster B) to proxy, the proxy change the mapping metadata in zookeeper, and after then request topic A become mapping Cluster B.
- [dependent] we should turn on cluster level geo-replication to ensure the topic data sync between all primary and secondary clusters. Otherwise, it may distribute the topic data into different clusters. And the consumers won’t get the whole data of the topic.
### Alternatives
_No response_
### Anything else?
- current architecture situation :
- after support multi pulsar cluster architecture :
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zengguan closed issue #17021: [proxy] proxy support multi pulsar cluster
Posted by GitBox <gi...@apache.org>.
zengguan closed issue #17021: [proxy] proxy support multi pulsar cluster
URL: https://github.com/apache/pulsar/issues/17021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #17021: [proxy] proxy support multi pulsar cluster
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #17021:
URL: https://github.com/apache/pulsar/issues/17021#issuecomment-1241460128
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org