You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by j....@gmail.com on 2014/09/05 09:51:51 UTC
Re: Review Request 24627: HIVE-7704: Create tez task for fast file
merging
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/
-----------------------------------------------------------
(Updated Sept. 5, 2014, 7:51 a.m.)
Review request for hive and Gunther Hagleitner.
Changes
-------
Addressed Gunther's review comments in this patch.
Bugs: HIVE-7704
https://issues.apache.org/jira/browse/HIVE-7704
Repository: hive-git
Description
-------
Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a0a5f54
itests/src/test/resources/testconfiguration.properties 4ad72a3
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java aa094ee
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199
ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 2bcb481
ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 44b6a43
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 82aa6ba
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java c2ba782
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb
ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8
ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java 8513f99
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java d58c59d
ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION
ql/src/test/queries/clientpositive/orc_merge1.q ee65b98
ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION
ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION
ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION
ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out ea37c36
ql/src/test/results/clientpositive/list_bucket_dml_10.q.out e9367ac
ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 99496d5
ql/src/test/results/clientpositive/list_bucket_dml_6.q.out d5deadb
ql/src/test/results/clientpositive/list_bucket_dml_7.q.out 4aea4db
ql/src/test/results/clientpositive/list_bucket_dml_9.q.out f94a3cc
ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648
ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469
ql/src/test/results/clientpositive/orc_createas1.q.out b0c58dd
ql/src/test/results/clientpositive/orc_merge1.q.out fc3e206
ql/src/test/results/clientpositive/orc_merge3.q.out 258f538
ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION
ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION
ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION
ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036
ql/src/test/results/clientpositive/rcfile_merge1.q.out ac6a2bd
ql/src/test/results/clientpositive/rcfile_merge2.q.out d8a61f3
ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5
ql/src/test/results/clientpositive/tez/orc_merge1.q.out f79f878
ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION
ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68
ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498
ql/src/test/results/clientpositive/union_remove_12.q.out 033475f
ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa
ql/src/test/results/clientpositive/union_remove_14.q.out 315c958
ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0
ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2
Diff: https://reviews.apache.org/r/24627/diff/
Testing
-------
Thanks,
Prasanth_J
Re: Review Request 24627: HIVE-7704: Create tez task for fast file
merging
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/#review52679
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91637>
All these were exisiting code taken from MergeMapper.java. Anyways, I rewrote the comment in the new patch. Also merged fixTmpPath and fixTmpPathConcatenate method to single method in the new patch.
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91638>
Updated in new patch.
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91639>
Updated comment in new patch.
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91640>
This is all gone in new patch.
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
<https://reviews.apache.org/r/24627/#comment91641>
Thats the eclipse. I used intellij. Perhaps both seems to do the opposite. :)
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
<https://reviews.apache.org/r/24627/#comment91642>
Fixed it.
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
<https://reviews.apache.org/r/24627/#comment91643>
If I use Map interface here, then I need to cast it to LinkedHashMap when I set aliasToWork(). To avoid casting I am using ListkedHashMap on LHS.
ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91644>
Fixed it.
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java
<https://reviews.apache.org/r/24627/#comment91645>
This is a dummy output format which is set in DagUtils/MergeFileTask. I am just using to make sure the operator pipeline for fast file merge is initialized properly. If the operator pipeline is wrongly initialized with say TS -> FS, then FS will get record writer from this output format throwing RuntimeException. If OFM or RFM operators are initialized then this will never be called.
The RCFile and ORC file writers handles output file opening and closing themselves. It does not use the standard record writer interfaces for writing the output. Both RCFile and ORC use custom interfaces for block level and stripe level writing respectively.
- Prasanth_J
On Sept. 6, 2014, 2:03 a.m., Prasanth_J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24627/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2014, 2:03 a.m.)
>
>
> Review request for hive and Gunther Hagleitner.
>
>
> Bugs: HIVE-7704
> https://issues.apache.org/jira/browse/HIVE-7704
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18
> itests/src/test/resources/testconfiguration.properties 99049ca
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 6f23575
> ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683
> ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199
> ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221
> ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2d9b9c3
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8
> ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419
> ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION
> ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d
> ql/src/test/queries/clientpositive/orc_merge1.q ee65b98
> ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION
> ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION
> ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION
> ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out ea37c36
> ql/src/test/results/clientpositive/list_bucket_dml_10.q.out e9367ac
> ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 99496d5
> ql/src/test/results/clientpositive/list_bucket_dml_6.q.out d5deadb
> ql/src/test/results/clientpositive/list_bucket_dml_7.q.out 4aea4db
> ql/src/test/results/clientpositive/list_bucket_dml_9.q.out f94a3cc
> ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648
> ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469
> ql/src/test/results/clientpositive/orc_createas1.q.out b0c58dd
> ql/src/test/results/clientpositive/orc_merge1.q.out fc3e206
> ql/src/test/results/clientpositive/orc_merge3.q.out 258f538
> ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036
> ql/src/test/results/clientpositive/rcfile_merge1.q.out ac6a2bd
> ql/src/test/results/clientpositive/rcfile_merge2.q.out d8a61f3
> ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5
> ql/src/test/results/clientpositive/tez/orc_merge1.q.out f79f878
> ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68
> ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498
> ql/src/test/results/clientpositive/union_remove_12.q.out 033475f
> ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa
> ql/src/test/results/clientpositive/union_remove_14.q.out 315c958
> ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0
> ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2
>
> Diff: https://reviews.apache.org/r/24627/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Prasanth_J
>
>
Re: Review Request 24627: HIVE-7704: Create tez task for fast file
merging
Posted by Vikram Dixit Kumaraswamy <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/#review53087
-----------------------------------------------------------
Ship it!
Ship It!
- Vikram Dixit Kumaraswamy
On Sept. 9, 2014, 7:32 a.m., Prasanth_J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24627/
> -----------------------------------------------------------
>
> (Updated Sept. 9, 2014, 7:32 a.m.)
>
>
> Review request for hive and Gunther Hagleitner.
>
>
> Bugs: HIVE-7704
> https://issues.apache.org/jira/browse/HIVE-7704
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31aeba9
> itests/src/test/resources/testconfiguration.properties 99049ca
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 6f23575
> ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683
> ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199
> ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221
> ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bbf3f6
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8
> ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419
> ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION
> ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d
> ql/src/test/queries/clientpositive/orc_merge1.q ee65b98
> ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION
> ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION
> ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION
> ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 11c7578
> ql/src/test/results/clientpositive/list_bucket_dml_10.q.out 8de452f
> ql/src/test/results/clientpositive/list_bucket_dml_4.q.out b1c060e
> ql/src/test/results/clientpositive/list_bucket_dml_6.q.out 3450d63
> ql/src/test/results/clientpositive/list_bucket_dml_7.q.out f6a4cb5
> ql/src/test/results/clientpositive/list_bucket_dml_9.q.out 796c7af
> ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648
> ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469
> ql/src/test/results/clientpositive/orc_createas1.q.out 993c853
> ql/src/test/results/clientpositive/orc_merge1.q.out 7f88125
> ql/src/test/results/clientpositive/orc_merge3.q.out 258f538
> ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036
> ql/src/test/results/clientpositive/rcfile_merge1.q.out cadb2c6
> ql/src/test/results/clientpositive/rcfile_merge2.q.out 807b8fb
> ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5
> ql/src/test/results/clientpositive/tez/orc_merge1.q.out e88bb6d
> ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68
> ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498
> ql/src/test/results/clientpositive/union_remove_12.q.out 033475f
> ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa
> ql/src/test/results/clientpositive/union_remove_14.q.out 315c958
> ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0
> ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2
>
> Diff: https://reviews.apache.org/r/24627/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Prasanth_J
>
>
Re: Review Request 24627: HIVE-7704: Create tez task for fast file
merging
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/
-----------------------------------------------------------
(Updated Sept. 9, 2014, 7:32 a.m.)
Review request for hive and Gunther Hagleitner.
Changes
-------
Addressed Vikram's review comment.s
Bugs: HIVE-7704
https://issues.apache.org/jira/browse/HIVE-7704
Repository: hive-git
Description
-------
Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31aeba9
itests/src/test/resources/testconfiguration.properties 99049ca
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 6f23575
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199
ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221
ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bbf3f6
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb
ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419
ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION
ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d
ql/src/test/queries/clientpositive/orc_merge1.q ee65b98
ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION
ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION
ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION
ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 11c7578
ql/src/test/results/clientpositive/list_bucket_dml_10.q.out 8de452f
ql/src/test/results/clientpositive/list_bucket_dml_4.q.out b1c060e
ql/src/test/results/clientpositive/list_bucket_dml_6.q.out 3450d63
ql/src/test/results/clientpositive/list_bucket_dml_7.q.out f6a4cb5
ql/src/test/results/clientpositive/list_bucket_dml_9.q.out 796c7af
ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648
ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469
ql/src/test/results/clientpositive/orc_createas1.q.out 993c853
ql/src/test/results/clientpositive/orc_merge1.q.out 7f88125
ql/src/test/results/clientpositive/orc_merge3.q.out 258f538
ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION
ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION
ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION
ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036
ql/src/test/results/clientpositive/rcfile_merge1.q.out cadb2c6
ql/src/test/results/clientpositive/rcfile_merge2.q.out 807b8fb
ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5
ql/src/test/results/clientpositive/tez/orc_merge1.q.out e88bb6d
ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION
ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68
ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498
ql/src/test/results/clientpositive/union_remove_12.q.out 033475f
ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa
ql/src/test/results/clientpositive/union_remove_14.q.out 315c958
ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0
ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2
Diff: https://reviews.apache.org/r/24627/diff/
Testing
-------
Thanks,
Prasanth_J
Re: Review Request 24627: HIVE-7704: Create tez task for fast file
merging
Posted by Vikram Dixit Kumaraswamy <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/#review52632
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91490>
inputDepth or tmpPath?
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91489>
inputDepth perhaps?
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91491>
nit updated.
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91502>
nit: has slight difference w.r.t
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
<https://reviews.apache.org/r/24627/#comment91505>
Usually done in the reverse way. The java imports come before the hadoop/hive imports.
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
<https://reviews.apache.org/r/24627/#comment91507>
Don't really need it here. Can be moved down to line 582.
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
<https://reviews.apache.org/r/24627/#comment91506>
HashMap only on the LHS. Don't use LinkedHashMap.
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
<https://reviews.apache.org/r/24627/#comment91508>
Nice work! Although this section usually comes above the hadoop imports though.
ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java
<https://reviews.apache.org/r/24627/#comment91509>
nit: comma after merge makes it easier to read.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
<https://reviews.apache.org/r/24627/#comment91515>
Nice work!
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java
<https://reviews.apache.org/r/24627/#comment91517>
What purpose does this class serve? Doesn't look like it does anything. Comments please.
- Vikram Dixit Kumaraswamy
On Sept. 6, 2014, 2:03 a.m., Prasanth_J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24627/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2014, 2:03 a.m.)
>
>
> Review request for hive and Gunther Hagleitner.
>
>
> Bugs: HIVE-7704
> https://issues.apache.org/jira/browse/HIVE-7704
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18
> itests/src/test/resources/testconfiguration.properties 99049ca
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 6f23575
> ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683
> ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199
> ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221
> ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2d9b9c3
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b
> ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8
> ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419
> ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION
> ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d
> ql/src/test/queries/clientpositive/orc_merge1.q ee65b98
> ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION
> ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION
> ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION
> ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out ea37c36
> ql/src/test/results/clientpositive/list_bucket_dml_10.q.out e9367ac
> ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 99496d5
> ql/src/test/results/clientpositive/list_bucket_dml_6.q.out d5deadb
> ql/src/test/results/clientpositive/list_bucket_dml_7.q.out 4aea4db
> ql/src/test/results/clientpositive/list_bucket_dml_9.q.out f94a3cc
> ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648
> ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469
> ql/src/test/results/clientpositive/orc_createas1.q.out b0c58dd
> ql/src/test/results/clientpositive/orc_merge1.q.out fc3e206
> ql/src/test/results/clientpositive/orc_merge3.q.out 258f538
> ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036
> ql/src/test/results/clientpositive/rcfile_merge1.q.out ac6a2bd
> ql/src/test/results/clientpositive/rcfile_merge2.q.out d8a61f3
> ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5
> ql/src/test/results/clientpositive/tez/orc_merge1.q.out f79f878
> ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68
> ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498
> ql/src/test/results/clientpositive/union_remove_12.q.out 033475f
> ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa
> ql/src/test/results/clientpositive/union_remove_14.q.out 315c958
> ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0
> ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2
>
> Diff: https://reviews.apache.org/r/24627/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Prasanth_J
>
>
Re: Review Request 24627: HIVE-7704: Create tez task for fast file
merging
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/
-----------------------------------------------------------
(Updated Sept. 6, 2014, 2:03 a.m.)
Review request for hive and Gunther Hagleitner.
Changes
-------
Fixed test failures
Bugs: HIVE-7704
https://issues.apache.org/jira/browse/HIVE-7704
Repository: hive-git
Description
-------
Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18
itests/src/test/resources/testconfiguration.properties 99049ca
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 6f23575
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199
ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221
ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2d9b9c3
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b
ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb
ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419
ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION
ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d
ql/src/test/queries/clientpositive/orc_merge1.q ee65b98
ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION
ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION
ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION
ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out ea37c36
ql/src/test/results/clientpositive/list_bucket_dml_10.q.out e9367ac
ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 99496d5
ql/src/test/results/clientpositive/list_bucket_dml_6.q.out d5deadb
ql/src/test/results/clientpositive/list_bucket_dml_7.q.out 4aea4db
ql/src/test/results/clientpositive/list_bucket_dml_9.q.out f94a3cc
ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648
ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469
ql/src/test/results/clientpositive/orc_createas1.q.out b0c58dd
ql/src/test/results/clientpositive/orc_merge1.q.out fc3e206
ql/src/test/results/clientpositive/orc_merge3.q.out 258f538
ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION
ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION
ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION
ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036
ql/src/test/results/clientpositive/rcfile_merge1.q.out ac6a2bd
ql/src/test/results/clientpositive/rcfile_merge2.q.out d8a61f3
ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5
ql/src/test/results/clientpositive/tez/orc_merge1.q.out f79f878
ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION
ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68
ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498
ql/src/test/results/clientpositive/union_remove_12.q.out 033475f
ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa
ql/src/test/results/clientpositive/union_remove_14.q.out 315c958
ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0
ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2
Diff: https://reviews.apache.org/r/24627/diff/
Testing
-------
Thanks,
Prasanth_J