You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@helix.apache.org by Kanak Biscuitwala <ka...@apache.org> on 2014/01/09 02:35:10 UTC
Review Request 16746: [HELIX-281] Prevent message callbacks from
indefinitely starving other callbacks
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16746/
-----------------------------------------------------------
Review request for helix, Zhen Zhang and Kishore Gopalakrishna.
Bugs: HELIX-281
Repository: helix-git
Description
-------
The key part of this is a new queue for callbacks. When a ClusterEvent is added, if an event with the same name already exists in the queue, it replaces that event. Otherwise, it is added to the tail of the queue. This prevents a large number of message events from starving other events.
commit a0e0eff78ce56d28f436e943303220579897dc32
Author: Kanak Biscuitwala <ka...@apache.org>
Date: Wed Jan 8 17:18:35 2014 -0800
[HELIX-281] Prevent message callbacks from indefinitely starving other callbacks
:100644 100644 9fef2da... b467bff... M helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
:000000 100644 0000000... 986437b... A helix-core/src/main/java/org/apache/helix/controller/stages/ClusterEventBlockingQueue.java
:000000 100644 0000000... 2dba7b6... A helix-core/src/test/java/org/apache/helix/controller/stages/TestClusterEventBlockingQueue.java
:100644 100644 30f5807... 67298b3... M helix-core/src/test/java/org/apache/helix/integration/TestSchedulerMessage.java
Diffs
-----
helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java 9fef2da
helix-core/src/main/java/org/apache/helix/controller/stages/ClusterEventBlockingQueue.java PRE-CREATION
helix-core/src/test/java/org/apache/helix/controller/stages/TestClusterEventBlockingQueue.java PRE-CREATION
helix-core/src/test/java/org/apache/helix/integration/TestSchedulerMessage.java 30f5807
Diff: https://reviews.apache.org/r/16746/diff/
Testing
-------
Ran this on a cluster with 100 resources, 100 partitions per resource, 100 nodes, FULL_AUTO. Took them all up and all down a few times, leading to a massive amount of entropy in the cluster. The controller was able to process all the callbacks in the expected order and outstanding callbacks were automatically coalesced.
Existing tests pass locally, added unit tests for the queue.
Thanks,
Kanak Biscuitwala
Re: Review Request 16746: [HELIX-281] Prevent message callbacks from
indefinitely starving other callbacks
Posted by Zhen Zhang <ne...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16746/#review32559
-----------------------------------------------------------
Ship it!
Ship It!
- Zhen Zhang
On Jan. 9, 2014, 1:35 a.m., Kanak Biscuitwala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16746/
> -----------------------------------------------------------
>
> (Updated Jan. 9, 2014, 1:35 a.m.)
>
>
> Review request for helix, Zhen Zhang and Kishore Gopalakrishna.
>
>
> Bugs: HELIX-281
>
>
> Repository: helix-git
>
>
> Description
> -------
>
> The key part of this is a new queue for callbacks. When a ClusterEvent is added, if an event with the same name already exists in the queue, it replaces that event. Otherwise, it is added to the tail of the queue. This prevents a large number of message events from starving other events.
>
> commit a0e0eff78ce56d28f436e943303220579897dc32
> Author: Kanak Biscuitwala <ka...@apache.org>
> Date: Wed Jan 8 17:18:35 2014 -0800
>
> [HELIX-281] Prevent message callbacks from indefinitely starving other callbacks
>
> :100644 100644 9fef2da... b467bff... M helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
> :000000 100644 0000000... 986437b... A helix-core/src/main/java/org/apache/helix/controller/stages/ClusterEventBlockingQueue.java
> :000000 100644 0000000... 2dba7b6... A helix-core/src/test/java/org/apache/helix/controller/stages/TestClusterEventBlockingQueue.java
> :100644 100644 30f5807... 67298b3... M helix-core/src/test/java/org/apache/helix/integration/TestSchedulerMessage.java
>
>
> Diffs
> -----
>
> helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java 9fef2da
> helix-core/src/main/java/org/apache/helix/controller/stages/ClusterEventBlockingQueue.java PRE-CREATION
> helix-core/src/test/java/org/apache/helix/controller/stages/TestClusterEventBlockingQueue.java PRE-CREATION
> helix-core/src/test/java/org/apache/helix/integration/TestSchedulerMessage.java 30f5807
>
> Diff: https://reviews.apache.org/r/16746/diff/
>
>
> Testing
> -------
>
> Ran this on a cluster with 100 resources, 100 partitions per resource, 100 nodes, FULL_AUTO. Took them all up and all down a few times, leading to a massive amount of entropy in the cluster. The controller was able to process all the callbacks in the expected order and outstanding callbacks were automatically coalesced.
>
> Existing tests pass locally, added unit tests for the queue.
>
>
> Thanks,
>
> Kanak Biscuitwala
>
>
Re: Review Request 16746: [HELIX-281] Prevent message callbacks from
indefinitely starving other callbacks
Posted by Zhen Zhang <ne...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16746/#review32555
-----------------------------------------------------------
helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
<https://reviews.apache.org/r/16746/#comment61440>
remove comment out lines
helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
<https://reviews.apache.org/r/16746/#comment61445>
log on thread start/terminate
probably need to catch ZkInterrupteException and Throwable also:
while(!isInterrupted()) {
try {
// do sth.
} catch (InterruptedException e) {
interrupt();
} catch (ZkInterruptedException e) {
interrupt();
} catch (Throwable e) {
LOG.error(...);
}
}
- Zhen Zhang
On Jan. 9, 2014, 1:35 a.m., Kanak Biscuitwala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16746/
> -----------------------------------------------------------
>
> (Updated Jan. 9, 2014, 1:35 a.m.)
>
>
> Review request for helix, Zhen Zhang and Kishore Gopalakrishna.
>
>
> Bugs: HELIX-281
>
>
> Repository: helix-git
>
>
> Description
> -------
>
> The key part of this is a new queue for callbacks. When a ClusterEvent is added, if an event with the same name already exists in the queue, it replaces that event. Otherwise, it is added to the tail of the queue. This prevents a large number of message events from starving other events.
>
> commit a0e0eff78ce56d28f436e943303220579897dc32
> Author: Kanak Biscuitwala <ka...@apache.org>
> Date: Wed Jan 8 17:18:35 2014 -0800
>
> [HELIX-281] Prevent message callbacks from indefinitely starving other callbacks
>
> :100644 100644 9fef2da... b467bff... M helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
> :000000 100644 0000000... 986437b... A helix-core/src/main/java/org/apache/helix/controller/stages/ClusterEventBlockingQueue.java
> :000000 100644 0000000... 2dba7b6... A helix-core/src/test/java/org/apache/helix/controller/stages/TestClusterEventBlockingQueue.java
> :100644 100644 30f5807... 67298b3... M helix-core/src/test/java/org/apache/helix/integration/TestSchedulerMessage.java
>
>
> Diffs
> -----
>
> helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java 9fef2da
> helix-core/src/main/java/org/apache/helix/controller/stages/ClusterEventBlockingQueue.java PRE-CREATION
> helix-core/src/test/java/org/apache/helix/controller/stages/TestClusterEventBlockingQueue.java PRE-CREATION
> helix-core/src/test/java/org/apache/helix/integration/TestSchedulerMessage.java 30f5807
>
> Diff: https://reviews.apache.org/r/16746/diff/
>
>
> Testing
> -------
>
> Ran this on a cluster with 100 resources, 100 partitions per resource, 100 nodes, FULL_AUTO. Took them all up and all down a few times, leading to a massive amount of entropy in the cluster. The controller was able to process all the callbacks in the expected order and outstanding callbacks were automatically coalesced.
>
> Existing tests pass locally, added unit tests for the queue.
>
>
> Thanks,
>
> Kanak Biscuitwala
>
>