You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2017/05/24 22:11:04 UTC
[jira] [Assigned] (PIG-4449) Optimize the case of Order by + Limit
in nested foreach
[ https://issues.apache.org/jira/browse/PIG-4449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy reassigned PIG-4449:
---------------------------------------
Assignee: Rohini Palaniswamy
Fix Version/s: 0.18.0
> Optimize the case of Order by + Limit in nested foreach
> -------------------------------------------------------
>
> Key: PIG-4449
> URL: https://issues.apache.org/jira/browse/PIG-4449
> Project: Pig
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Labels: Performance
> Fix For: 0.18.0
>
>
> This is one of the very frequently used patterns
> {code}
> grouped_data_set = group data_set by id;
> capped_data_set = foreach grouped_data_set
> {
> ordered = order joined_data_set by timestamp desc;
> capped = limit ordered $num;
> generate flatten(capped);
> };
> {code}
> But this performs very poorly when there are millions of rows for a key in the groupby with lot of spills. This can be easily optimized by pushing the limit into the InternalSortedBag and maintain only $num records any time and avoid memory pressure.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)