[jira] [Resolved] (STORM-2210) ShuffleGrouping does not produce even distribution

["Robert Joseph Evans (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-2210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans resolved STORM-2210.
----------------------------------------
       Resolution: Fixed
         Assignee: Kevin Peek
    Fix Version/s: 1.0.3
                   1.1.0
                   2.0.0

Thanks [~kevpeek],

I merged this into master, 1.x-branch and 1.0.x-branch.  Keep up the good work.

> ShuffleGrouping does not produce even distribution
> --------------------------------------------------
>
>                 Key: STORM-2210
>                 URL: https://issues.apache.org/jira/browse/STORM-2210
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-core
>    Affects Versions: 1.0.2
>            Reporter: Kevin Peek
>            Assignee: Kevin Peek
>            Priority: Critical
>             Fix For: 2.0.0, 1.1.0, 1.0.3
>
>          Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> When testing the ShuffleGrouping in a multithreaded environment, it produces an extremely uneven distribution.
> This appears to be a result of the Collection.shuffle call here. https://github.com/apache/storm/blob/1.0.x-branch/storm-core/src/jvm/org/apache/storm/grouping/ShuffleGrouping.java#L58
> Because current was set to zero before the shuffle, other threads are able to access the arrayList while it is being shuffled.
> Stephen's gist here includes a test that results in a very uneven distribution of taskIds from the ShuffleGrouping: https://gist.github.com/Crim/61537958df65a5e13b3844b2d5e28cde
> I would have expected the taskIds from the ShuffleGrouping to be almost uniformly distributed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)