You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Jeff Hammerbacher (JIRA)" <ji...@apache.org> on 2010/06/12 08:01:13 UTC
[jira] Created: (HIVE-1402) Add parallel ORDER BY to Hive
Add parallel ORDER BY to Hive
-----------------------------
Key: HIVE-1402
URL: https://issues.apache.org/jira/browse/HIVE-1402
Project: Hadoop Hive
Issue Type: New Feature
Reporter: Jeff Hammerbacher
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1402) Add parallel ORDER BY to Hive
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880120#action_12880120 ]
Jeff Zhang commented on HIVE-1402:
----------------------------------
Hi, I make a draft implementation for one special case. And it works, but since it is only for one special case, so I have some hard coding. I hope someone can give some help or instruction for the next step.
One big problem of parallel ORDER BY is that the output key type of ExecMapper is HiveKey, and it has been serialized by LazyBinarySerDe, so the original column type is lost here. But when do sampling and partition, I should use the original column type.
The following is my initial design.
1. During parse stage, extract one SampleOperator which has two children: TableScanOperator, SelectOperator ( I am not familiar with Hive Parse Stage, and the code is not clear for me, could anyone give some help or recommend some documentation about the Hive parser ? )
2. Modify the TotalOrderPartitioner. Add a Deserializer to convert the HiveKey to its original column type. and deserialie the HiveKey in method getPartition().
Welcome any comments and help.
> Add parallel ORDER BY to Hive
> -----------------------------
>
> Key: HIVE-1402
> URL: https://issues.apache.org/jira/browse/HIVE-1402
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.5.0
> Reporter: Jeff Hammerbacher
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1402) Add parallel ORDER BY to Hive
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1402:
---------------------------------
Affects Version/s: (was: 0.5.0)
> Add parallel ORDER BY to Hive
> -----------------------------
>
> Key: HIVE-1402
> URL: https://issues.apache.org/jira/browse/HIVE-1402
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Jeff Hammerbacher
> Assignee: Venkatesh S
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1402) Add parallel ORDER BY to Hive
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1402:
---------------------------------
Affects Version/s: 0.5.0
Component/s: Query Processor
> Add parallel ORDER BY to Hive
> -----------------------------
>
> Key: HIVE-1402
> URL: https://issues.apache.org/jira/browse/HIVE-1402
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.5.0
> Reporter: Jeff Hammerbacher
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1402) Add parallel ORDER BY to Hive
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878213#action_12878213 ]
Jeff Hammerbacher commented on HIVE-1402:
-----------------------------------------
>From Ning Zhang:
bq. order by is supported in trunk with certain limititions in strict mode (has to have a limit)
>From John Sichi:
> If someone is interested in adding parallel ORDER BY to Hive (using TotalOrderPartitioner), here's a good starting point: http://wiki.apache.org/hadoop/Hive/HBaseBulkLoad
>
> The goal would be to take that manual two-step sample-then-sort process and turn it into an automatic plan within Hive. I have a better example for the sampling query which I > haven't published yet.
>
> We would also need to name the final output files in such a way that the total order could be iterated via the filenames.
> Add parallel ORDER BY to Hive
> -----------------------------
>
> Key: HIVE-1402
> URL: https://issues.apache.org/jira/browse/HIVE-1402
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1402) Add parallel ORDER BY to Hive
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi reassigned HIVE-1402:
--------------------------------
Assignee: Venkatesh S
> Add parallel ORDER BY to Hive
> -----------------------------
>
> Key: HIVE-1402
> URL: https://issues.apache.org/jira/browse/HIVE-1402
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.5.0
> Reporter: Jeff Hammerbacher
> Assignee: Venkatesh S
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.