You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Peter Romianowski (JIRA)" <ji...@apache.org> on 2012/05/18 16:35:07 UTC
[jira] [Created] (KAFKA-345) Add a listener to
ZookeeperConsumerConnector to get notified on rebalance events
Peter Romianowski created KAFKA-345:
---------------------------------------
Summary: Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
Key: KAFKA-345
URL: https://issues.apache.org/jira/browse/KAFKA-345
Project: Kafka
Issue Type: Improvement
Components: core
Affects Versions: 0.7, 0.8
Reporter: Peter Romianowski
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-345) Add a listener to
ZookeeperConsumerConnector to get notified on rebalance events
Posted by "Peter Romianowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288179#comment-13288179 ]
Peter Romianowski commented on KAFKA-345:
-----------------------------------------
Jun,
regarding your comments:
1. Adding it to the scala-version makes sense, of course. Bear with me, I'm just a Java-guy. I'll add that.
2. I'll move the listener. I did name it "ConsumerListener" and "setListener" intentionally, because I thought it would be a good extension point for future stuff, that is not necessarily related to rebalancing. That's also why I made it an abstract class and not an interface. If you still think it should be renamed, I'll do it.
3. I'll remove the spaces and unused imports.
> Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
> --------------------------------------------------------------------------------
>
> Key: KAFKA-345
> URL: https://issues.apache.org/jira/browse/KAFKA-345
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Peter Romianowski
> Attachments: KAFKA-345.patch
>
>
> A sample use-case
> In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-345) Add a listener to
ZookeeperConsumerConnector to get notified on rebalance events
Posted by "Peter Romianowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Romianowski updated KAFKA-345:
------------------------------------
Description:
A sample use-case
In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
was:
h1. A sample use-case
In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
> Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
> --------------------------------------------------------------------------------
>
> Key: KAFKA-345
> URL: https://issues.apache.org/jira/browse/KAFKA-345
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Peter Romianowski
>
> A sample use-case
> In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-345) Add a listener to
ZookeeperConsumerConnector to get notified on rebalance events
Posted by "Peter Romianowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Romianowski updated KAFKA-345:
------------------------------------
Description:
h1. A sample use-case
In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
> Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
> --------------------------------------------------------------------------------
>
> Key: KAFKA-345
> URL: https://issues.apache.org/jira/browse/KAFKA-345
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Peter Romianowski
>
> h1. A sample use-case
> In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-345) Add a listener to
ZookeeperConsumerConnector to get notified on rebalance events
Posted by "Peter Romianowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Romianowski updated KAFKA-345:
------------------------------------
Status: Patch Available (was: Open)
Added a patch against trunk.
> Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
> --------------------------------------------------------------------------------
>
> Key: KAFKA-345
> URL: https://issues.apache.org/jira/browse/KAFKA-345
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Peter Romianowski
> Attachments: KAFKA-345.patch
>
>
> A sample use-case
> In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-345) Add a listener to
ZookeeperConsumerConnector to get notified on rebalance events
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281327#comment-13281327 ]
Jun Rao commented on KAFKA-345:
-------------------------------
Peter, thanks for the patch. Some comments:
1. It seems that we should add setListener in the scala version of ConsumerConnector too. Instead of calling it setListener, should we call it setRebalanceListener?
2. Since ConsumerListener needs to be used in both the java and scala version of ConsumerConnector, should we put it in the consumer package, instead of javaapi.consumer? Also, we probably should rename it to ConsumerRebalanceListener.
3. ZKLoadBalanceTest:
3.1 testLoadBalance(): There seems to be a mix of space and tab. Maybe that's in the original code already, but could you fix that?
3.2 remove unused imports
I am not sure if there is an easy way to fail rebalance in unit test. The unit test in the patch seem sufficient to me.
> Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
> --------------------------------------------------------------------------------
>
> Key: KAFKA-345
> URL: https://issues.apache.org/jira/browse/KAFKA-345
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Peter Romianowski
> Attachments: KAFKA-345.patch
>
>
> A sample use-case
> In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-345) Add a listener to
ZookeeperConsumerConnector to get notified on rebalance events
Posted by "Peter Romianowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Romianowski updated KAFKA-345:
------------------------------------
Attachment: KAFKA-345.patch
> Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
> --------------------------------------------------------------------------------
>
> Key: KAFKA-345
> URL: https://issues.apache.org/jira/browse/KAFKA-345
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Peter Romianowski
> Attachments: KAFKA-345.patch
>
>
> A sample use-case
> In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-345) Add a listener to
ZookeeperConsumerConnector to get notified on rebalance events
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288276#comment-13288276 ]
Jun Rao commented on KAFKA-345:
-------------------------------
Peter,
Regarding #2, naming it as ConsumerListener is fine.
> Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
> --------------------------------------------------------------------------------
>
> Key: KAFKA-345
> URL: https://issues.apache.org/jira/browse/KAFKA-345
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Peter Romianowski
> Attachments: KAFKA-345.patch
>
>
> A sample use-case
> In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-345) Add a listener to
ZookeeperConsumerConnector to get notified on rebalance events
Posted by "Peter Romianowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279022#comment-13279022 ]
Peter Romianowski commented on KAFKA-345:
-----------------------------------------
I tried to add a test to verify that the listener gets called if an error occurs (ConsumerListener#afterRebalance(false)) but I failed to introduce some error into ZookeeperConsumerConnector.ZKRebalancerListener#syncedRebalance.
Any ideas how to provoke an error during rebalancing?
> Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
> --------------------------------------------------------------------------------
>
> Key: KAFKA-345
> URL: https://issues.apache.org/jira/browse/KAFKA-345
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Peter Romianowski
> Attachments: KAFKA-345.patch
>
>
> A sample use-case
> In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira