You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Alan Gates (JIRA)" <ji...@apache.org> on 2008/09/19 21:02:44 UTC
[jira] Created: (PIG-441) Order by desc does not work with more
than one reducer
Order by desc does not work with more than one reducer
------------------------------------------------------
Key: PIG-441
URL: https://issues.apache.org/jira/browse/PIG-441
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Alan Gates
Assignee: Alan Gates
Fix For: types_branch
Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-441) Order by desc does not work with more
than one reducer
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-441:
---------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch checked in.
> Order by desc does not work with more than one reducer
> ------------------------------------------------------
>
> Key: PIG-441
> URL: https://issues.apache.org/jira/browse/PIG-441
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Alan Gates
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: PIG-441.patch
>
>
> Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-441) Order by desc does not work with more
than one reducer
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633542#action_12633542 ]
Olga Natkovich commented on PIG-441:
------------------------------------
+1
> Order by desc does not work with more than one reducer
> ------------------------------------------------------
>
> Key: PIG-441
> URL: https://issues.apache.org/jira/browse/PIG-441
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Alan Gates
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: PIG-441.patch
>
>
> Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-441) Order by desc does not work with more
than one reducer
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-441:
---------------------------
Attachment: PIG-441.patch
This patch does a couple of things:
1) add object comparators to the PigXRawComparator classes so that in places where hadoop uses the object instead of null comparators the desc behavior can still be handled.
2) Change FindQuantiles UDF to take in its constructor an array that indicates ascending vs descending order so that the quantiles can be correclty computed.
> Order by desc does not work with more than one reducer
> ------------------------------------------------------
>
> Key: PIG-441
> URL: https://issues.apache.org/jira/browse/PIG-441
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Alan Gates
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: PIG-441.patch
>
>
> Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-441) Order by desc does not work with more
than one reducer
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-441:
---------------------------
Status: Patch Available (was: Open)
> Order by desc does not work with more than one reducer
> ------------------------------------------------------
>
> Key: PIG-441
> URL: https://issues.apache.org/jira/browse/PIG-441
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Alan Gates
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: PIG-441.patch
>
>
> Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.