You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Rui Li <ru...@intel.com> on 2014/09/18 12:39:14 UTC
Review Request 25774: Support merging small files
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25774/
-----------------------------------------------------------
Review request for hive and Xuefu Zhang.
Bugs: HIVE-8043
https://issues.apache.org/jira/browse/HIVE-8043
Repository: hive-git
Description
-------
Support merging files for spark.
For non-rc files, the merging task is simply a MapWork.
For RC/Orc files, the merging task is a MergeFileWork. And SparkMergeFileRecordHandler is added to handle it.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 5078a3a
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunctionResultList.java c54bffe
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 2537789
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 9b11fe4
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 3eea26a
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java 94ebcdd
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java b0a9407
ql/src/test/queries/clientpositive/disable_merge_for_bucketing.q 471d296
ql/src/test/queries/clientpositive/merge1.q c7249af
ql/src/test/queries/clientpositive/merge2.q bb86dc2
ql/src/test/results/clientpositive/spark/merge1.q.out 772984d
ql/src/test/results/clientpositive/spark/merge2.q.out 8d8dcb8
Diff: https://reviews.apache.org/r/25774/diff/
Testing
-------
Thanks,
Rui Li
Re: Review Request 25774: Support merging small files
Posted by Rui Li <ru...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25774/
-----------------------------------------------------------
(Updated Sept. 20, 2014, 3:37 a.m.)
Review request for hive and Xuefu Zhang.
Changes
-------
Fix a bug in getting file system for the output dir
Bugs: HIVE-8043
https://issues.apache.org/jira/browse/HIVE-8043
Repository: hive-git
Description
-------
Support merging files for spark.
For non-rc files, the merging task is simply a MapWork.
For RC/Orc files, the merging task is a MergeFileWork. And SparkMergeFileRecordHandler is added to handle it.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 5078a3a
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunctionResultList.java c54bffe
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 2537789
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 9b11fe4
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 3eea26a
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java 94ebcdd
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java b0a9407
ql/src/test/queries/clientpositive/disable_merge_for_bucketing.q 471d296
ql/src/test/queries/clientpositive/merge1.q c7249af
ql/src/test/queries/clientpositive/merge2.q bb86dc2
ql/src/test/results/clientpositive/spark/merge1.q.out 772984d
ql/src/test/results/clientpositive/spark/merge2.q.out 8d8dcb8
ql/src/test/results/clientpositive/spark/union_remove_10.q.out f561fdf
ql/src/test/results/clientpositive/spark/union_remove_11.q.out 10b0c9c
ql/src/test/results/clientpositive/spark/union_remove_16.q.out a59a352
ql/src/test/results/clientpositive/spark/union_remove_4.q.out 518dc24
ql/src/test/results/clientpositive/spark/union_remove_5.q.out f7f9627
ql/src/test/results/clientpositive/spark/union_remove_9.q.out 0ec55de
Diff: https://reviews.apache.org/r/25774/diff/
Testing
-------
Thanks,
Rui Li
Re: Review Request 25774: Support merging small files
Posted by Rui Li <ru...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25774/
-----------------------------------------------------------
(Updated Sept. 18, 2014, 12:33 p.m.)
Review request for hive and Xuefu Zhang.
Changes
-------
Update golden files for failed tests
Bugs: HIVE-8043
https://issues.apache.org/jira/browse/HIVE-8043
Repository: hive-git
Description
-------
Support merging files for spark.
For non-rc files, the merging task is simply a MapWork.
For RC/Orc files, the merging task is a MergeFileWork. And SparkMergeFileRecordHandler is added to handle it.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 5078a3a
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunctionResultList.java c54bffe
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 2537789
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 9b11fe4
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 3eea26a
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java 94ebcdd
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java b0a9407
ql/src/test/queries/clientpositive/disable_merge_for_bucketing.q 471d296
ql/src/test/queries/clientpositive/merge1.q c7249af
ql/src/test/queries/clientpositive/merge2.q bb86dc2
ql/src/test/results/clientpositive/spark/merge1.q.out 772984d
ql/src/test/results/clientpositive/spark/merge2.q.out 8d8dcb8
ql/src/test/results/clientpositive/spark/union_remove_10.q.out f561fdf
ql/src/test/results/clientpositive/spark/union_remove_11.q.out 10b0c9c
ql/src/test/results/clientpositive/spark/union_remove_16.q.out a59a352
ql/src/test/results/clientpositive/spark/union_remove_4.q.out 518dc24
ql/src/test/results/clientpositive/spark/union_remove_5.q.out f7f9627
ql/src/test/results/clientpositive/spark/union_remove_9.q.out 0ec55de
Diff: https://reviews.apache.org/r/25774/diff/
Testing
-------
Thanks,
Rui Li