You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Kevin Peek (JIRA)" <ji...@apache.org> on 2016/11/21 17:53:58 UTC
[jira] [Updated] (STORM-2210) ShuffleGrouping does not produce even
distribution
[ https://issues.apache.org/jira/browse/STORM-2210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Peek updated STORM-2210:
------------------------------
Description:
When testing the ShuffleGrouping in a multithreaded environment, it produces an extremely uneven distribution.
This appears to be a result of the Collection.shuffle call here. https://github.com/apache/storm/blob/1.0.x-branch/storm-core/src/jvm/org/apache/storm/grouping/ShuffleGrouping.java#L58
Because current was set to zero before the shuffle, other threads are able to access the arrayList while it is being shuffled.
Stephen's gist here includes a test that results in a very uneven distribution of taskIds from the ShuffleGrouping: https://gist.github.com/Crim/61537958df65a5e13b3844b2d5e28cde
I would have expected the taskIds from the ShuffleGrouping to be almost uniformly distributed.
was:
When testing the ShuffleGrouping in a multithreaded environment, it produces an extremely uneven distribution.
This appears to be a result of the Collection.shuffle call here. https://github.com/apache/storm/blob/1.0.x-branch/storm-core/src/jvm/org/apache/storm/grouping/ShuffleGrouping.java#L58
Because current was set to zero before the shuffle, other threads are able to access the arrayList while it is being shuffled.
> ShuffleGrouping does not produce even distribution
> --------------------------------------------------
>
> Key: STORM-2210
> URL: https://issues.apache.org/jira/browse/STORM-2210
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
> Affects Versions: 1.0.2
> Reporter: Kevin Peek
> Priority: Minor
>
> When testing the ShuffleGrouping in a multithreaded environment, it produces an extremely uneven distribution.
> This appears to be a result of the Collection.shuffle call here. https://github.com/apache/storm/blob/1.0.x-branch/storm-core/src/jvm/org/apache/storm/grouping/ShuffleGrouping.java#L58
> Because current was set to zero before the shuffle, other threads are able to access the arrayList while it is being shuffled.
> Stephen's gist here includes a test that results in a very uneven distribution of taskIds from the ShuffleGrouping: https://gist.github.com/Crim/61537958df65a5e13b3844b2d5e28cde
> I would have expected the taskIds from the ShuffleGrouping to be almost uniformly distributed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)