You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/05/25 04:40:18 UTC
[GitHub] [pulsar] lukestephenson opened a new issue #7028: Support shared subscriptions on compacted topics
lukestephenson opened a new issue #7028:
URL: https://github.com/apache/pulsar/issues/7028
**Is your feature request related to a problem? Please describe.**
I'm evaluating Pulsar. A primary use case of mine is large (more than 1 billion unique keyed messages) compacted topics. To handle these efficiently, I had planned to make use of multiple partitions on the brokers, as well as have multiple consumers with a "key shared" subscription.
Currently the pulsar client rejects setting up a subscription on a compacted topic unless the subscription type is exclusive or failover.
https://github.com/apache/pulsar/blob/v2.5.2/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java#L309-L315
**Describe the solution you'd like**
Ability to have multiple consumers sharing a subscription to a compacted topic (partitioned or not partitioned).
**Describe alternatives you've considered**
Allocate more resources to a single consumer. Might not be easy in kubernetes, we might get to a point where the pod needs more resources than the nodes in our cluster. I don't believe this is the correct solution. We should be building services that can scale out to many small nodes from the beginning.
I'd be happy as well with each consumer being allocated a different partition from the topic (i.e. similar to what Kafka does).
**Additional context**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lukestephenson commented on issue #7028: Support shared subscriptions on compacted topics
Posted by GitBox <gi...@apache.org>.
lukestephenson commented on issue #7028:
URL: https://github.com/apache/pulsar/issues/7028#issuecomment-633970533
I'm able to get the desired behaviour I needed using a Failover subscription. Feel free to close this issue, or keep it around if it's still a worthwhile change. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gmethvin commented on issue #7028: Support shared subscriptions on compacted topics
Posted by GitBox <gi...@apache.org>.
gmethvin commented on issue #7028:
URL: https://github.com/apache/pulsar/issues/7028#issuecomment-633864492
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7028: Support shared subscriptions on compacted topics
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7028:
URL: https://github.com/apache/pulsar/issues/7028#issuecomment-633763777
@lukestephenson
If you use a partitioned topic, the data is already partitioned. each partition will be compacted if you enable compaction. So each partition can have its own compacted state. Then you can read the compacted topic use Failover subscription. This would achieve the exact same thing as you did in Kafka.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hozumi commented on issue #7028: Support shared subscriptions on compacted topics
Posted by GitBox <gi...@apache.org>.
hozumi commented on issue #7028:
URL: https://github.com/apache/pulsar/issues/7028#issuecomment-663175287
Hi, I'm thinking the same use case discussed here.
It would be nice if users could manage the load by simply adding or removing consumers to a single logical subscription, like turning a faucet, instead of manually assigning subscriptions to underlying partitions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org