You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/09/28 10:12:33 UTC
[GitHub] [pulsar] sollecitom opened a new issue #12224: Partitions rebalance listener for Consumer
sollecitom opened a new issue #12224:
URL: https://github.com/apache/pulsar/issues/12224
When using processors with in-memory state, it's important to know the partitions assigned and de-assigned. This allows to load and unload the state, and to recover based from a durable eventually consistent snapshot by loading it based on these partitions.
At the moment, you can try to derive this information from the Message IDs, but there are 2 problems with this approach.
1. You don't know when a partition is de-assigned for a Consumer. This forces you to use a time-based cache eviction policy.
2. When you recover from a crash, you don't know the whole set of partitions assigned, meaning you're forced to load from durable storage the state for each partition when you encounter it in a MessageID. This means a less efficient startup.
Ideally, you should expose something like the [ConsumerRebalanceListener](https://kafka.apache.org/30/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html) in Kafka.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #12224: Partitions rebalance listener for Consumer
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #12224:
URL: https://github.com/apache/pulsar/issues/12224#issuecomment-1053811368
@sollecitom Close this for now, feel free to reopen this if you have further questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #12224: Partitions rebalance listener for Consumer
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12224:
URL: https://github.com/apache/pulsar/issues/12224#issuecomment-1053791837
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 closed issue #12224: Partitions rebalance listener for Consumer
Posted by GitBox <gi...@apache.org>.
Jason918 closed issue #12224:
URL: https://github.com/apache/pulsar/issues/12224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #12224: Partitions rebalance listener for Consumer
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #12224:
URL: https://github.com/apache/pulsar/issues/12224#issuecomment-934292736
Maybe you can check the docs in `org.apache.pulsar.client.api.ConsumerEventListener`.
Once the broker choose a consumer for receiving messages in failover mode, it will send an event to client. You can add a listener to this event through ConsumerEventListener.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #12224: Partitions rebalance listener for Consumer
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #12224:
URL: https://github.com/apache/pulsar/issues/12224#issuecomment-929164058
I think there's no rebalance concept in Pulsar. Pulsar consumer doesn't maintain a partition assignment internally like what Kafka consumer does. Pulsar's consumption model is push, while Kafka's is pull. In Pulsar, the dispatcher at broker side determines which partition should the message be sent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #12224: Partitions rebalance listener for Consumer
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #12224:
URL: https://github.com/apache/pulsar/issues/12224#issuecomment-929164058
I think there's no rebalance concept in Pulsar. Pulsar consumer doesn't maintain a partition assignment internally like what Kafka consumer does. Pulsar's consumption model is push, while Kafka's is pull. In Pulsar, the dispatcher at broker side determines which partition should the message be sent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org