You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2013/11/21 21:47:35 UTC
[jira] [Updated] (PIG-3534) Make order by work
[ https://issues.apache.org/jira/browse/PIG-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-3534:
----------------------------
Attachment: PIG-3534-1.patch
Initial naive implementation to simulate MR. Currently need 5 vertaxes (3 Dags) to do the order by:
1. materialize the previous output into hdfs (Dag end)
2. map phase for sampler
3. reduce phase for sampler, material the sample file into hdfs (Dag end)
4. map phase for the sort
5. reduce phase for the sort (open end)
#2, #3 can be optimized out, we need to use SampleInputHandler? to replace the SampleLoader. #1 does not need to cut a Dag boundary.
The comparator is not fully right. Currently only numerical/byte type order correctly. Those are interweaved with the comparator changes in PIG-3527. I can fix them after PIG-3527.
There are also many duplicate code with MR. Some refectory will be desired.
> Make order by work
> ------------------
>
> Key: PIG-3534
> URL: https://issues.apache.org/jira/browse/PIG-3534
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: tez-branch
>
> Attachments: PIG-3534-1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)