[jira] [Updated] (PIG-1592) ORDER BY distribution is uneven when record size is correlated with order key

["Thejas M Nair (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thejas M Nair updated PIG-1592:
-------------------------------

    Fix Version/s:     (was: 0.9.0)
                   0.10

> ORDER BY distribution is uneven when record size is correlated with order key
> -----------------------------------------------------------------------------
>
>                 Key: PIG-1592
>                 URL: https://issues.apache.org/jira/browse/PIG-1592
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Dmitriy V. Ryaboy
>            Assignee: Thejas M Nair
>             Fix For: 0.10
>
>
> The partitioner contributed in PIG-545 distributes the order key space between partitions so that each partition gets approximately the same number of keys, even when the keys have a non-uniform distribution over the key space.
> Unfortunately this still allows for severe partition imbalance when record size is correlated with the order key. By way of motivating example, consider this script which attempts to produce a list of genuses based on how many species each genus contains:
> {code}
> set default_parallel 60;
> critters = load 'biodata'' as (genus, species);
> genus_counts = foreach (group critters by genus) generate group as genus, COUNT(critters) as num_species, critters;
> ordered_genuses = order genus_counts by num_species desc;
> store ordered_genuses....
> {code}
> The higher the value of genus_counts, the more species tuples will be contained in the critters bag, the wider the row. This can cause a severe processing imbalance, as the partitioner processing the records with the highest values of genus_counts will have the same number of *records* as the partitioner processing the lowest number, but it will have far more actual *bytes* to work on.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira