You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Gunther Hagleitner <gh...@hortonworks.com> on 2013/12/23 16:03:38 UTC
Review Request 16440: HIVE-6098: Merge Tez branch into trunk
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/
-----------------------------------------------------------
Review request for hive.
Repository: hive
Description
-------
I think the Tez branch is at a point where we can consider merging it back into trunk after review.
Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.)
Limitations:
Union all is not yet supported on Tez
SMB is not yet supported on Tez
Bucketed map-join is executed as broadcast join (bucketing is ignored)
Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR.
I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well.
I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try.
Diffs
-----
Diff: https://reviews.apache.org/r/16440/diff/
Testing
-------
Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
Thanks,
Gunther Hagleitner
Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
> On Jan. 8, 2014, 6:51 p.m., Vikram Dixit Kumaraswamy wrote:
> > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java, line 44
> > <https://reviews.apache.org/r/16440/diff/3/?file=417745#file417745line44>
> >
> > Could it ever be a non-tez task?
Yes, this can happen since some analyze commands are run as MR on Tez.
> On Jan. 8, 2014, 6:51 p.m., Vikram Dixit Kumaraswamy wrote:
> > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java, line 153
> > <https://reviews.apache.org/r/16440/diff/3/?file=417720#file417720line153>
> >
> > Need to update java doc.
Will do.
> On Jan. 8, 2014, 6:51 p.m., Vikram Dixit Kumaraswamy wrote:
> > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java, line 215
> > <https://reviews.apache.org/r/16440/diff/3/?file=417720#file417720line215>
> >
> > No param conf.
Will do.
> On Jan. 8, 2014, 6:51 p.m., Vikram Dixit Kumaraswamy wrote:
> > trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java, line 42
> > <https://reviews.apache.org/r/16440/diff/3/?file=417725#file417725line42>
> >
> > spelling error "commone"
Will change.
> On Jan. 8, 2014, 6:51 p.m., Vikram Dixit Kumaraswamy wrote:
> > trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java, line 38
> > <https://reviews.apache.org/r/16440/diff/3/?file=417728#file417728line38>
> >
> > nit: reword the sentence.
Yup, will do.
> On Jan. 8, 2014, 6:51 p.m., Vikram Dixit Kumaraswamy wrote:
> > trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java, line 150
> > <https://reviews.apache.org/r/16440/diff/3/?file=417740#file417740line150>
> >
> > Could rename the mrTask to be more generic.
I'm not sure. I thought mvTask is move task, mrTask is merge task in this function. I'd like to keep that, if it's all the same to you.
> On Jan. 8, 2014, 6:51 p.m., Vikram Dixit Kumaraswamy wrote:
> > trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java, line 393
> > <https://reviews.apache.org/r/16440/diff/3/?file=417741#file417741line393>
> >
> > Why doesn't the same apply for the value table desc below?
Because the keys are special. If we're using reduce sink we need to specify sort order etc. The value side of things is unchanged.
> On Jan. 8, 2014, 6:51 p.m., Vikram Dixit Kumaraswamy wrote:
> > trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java, line 422
> > <https://reviews.apache.org/r/16440/diff/3/?file=417741#file417741line422>
> >
> > Comment needs to be fixed.
Will do.
> On Jan. 8, 2014, 6:51 p.m., Vikram Dixit Kumaraswamy wrote:
> > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java, line 29
> > <https://reviews.apache.org/r/16440/diff/3/?file=417745#file417745line29>
> >
> > Could add some documentation explaining the use of this class.
Ok.
- Gunther
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/#review31347
-----------------------------------------------------------
On Jan. 7, 2014, 12:50 a.m., Gunther Hagleitner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16440/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2014, 12:50 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive
>
>
> Description
> -------
>
> I think the Tez branch is at a point where we can consider merging it back into trunk after review.
> Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.)
> Limitations:
> Union all is not yet supported on Tez
> SMB is not yet supported on Tez
> Bucketed map-join is executed as broadcast join (bucketing is ignored)
> Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR.
> I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well.
> I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try.
>
>
> Diffs
> -----
>
> trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1556067
> trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1556067
> trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1556067
> trunk/conf/hive-default.xml.template 1556067
> trunk/data/conf/tez/hive-site.xml UNKNOWN
> trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1556067
> trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1556067
> trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1556067
> trunk/itests/qtest/pom.xml 1556067
> trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1556067
> trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1556067
> trunk/pom.xml 1556067
> trunk/ql/pom.xml 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1556067
> trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1556067
> trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out UNKNOWN
> trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION
> trunk/ql/src/test/templates/TestCliDriver.vm 1556067
> trunk/ql/src/test/templates/TestCompareCliDriver.vm 1556067
> trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1556067
> trunk/ql/src/test/templates/TestParse.vm 1556067
> trunk/ql/src/test/templates/TestParseNegative.vm 1556067
> trunk/shims/0.23/pom.xml 1556067
> trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1556067
>
> Diff: https://reviews.apache.org/r/16440/diff/
>
>
> Testing
> -------
>
> Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
>
>
> Thanks,
>
> Gunther Hagleitner
>
>
Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
Posted by Vikram Dixit Kumaraswamy <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/#review31347
-----------------------------------------------------------
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
<https://reviews.apache.org/r/16440/#comment59818>
Need to update java doc.
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
<https://reviews.apache.org/r/16440/#comment59817>
No param conf.
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
<https://reviews.apache.org/r/16440/#comment59831>
spelling error "commone"
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java
<https://reviews.apache.org/r/16440/#comment59832>
nit: reword the sentence.
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java
<https://reviews.apache.org/r/16440/#comment59834>
Could rename the mrTask to be more generic.
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
<https://reviews.apache.org/r/16440/#comment59836>
Why doesn't the same apply for the value table desc below?
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
<https://reviews.apache.org/r/16440/#comment59835>
Comment needs to be fixed.
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java
<https://reviews.apache.org/r/16440/#comment59838>
Could add some documentation explaining the use of this class.
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java
<https://reviews.apache.org/r/16440/#comment59839>
Could it ever be a non-tez task?
- Vikram Dixit Kumaraswamy
On Jan. 7, 2014, 12:50 a.m., Gunther Hagleitner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16440/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2014, 12:50 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive
>
>
> Description
> -------
>
> I think the Tez branch is at a point where we can consider merging it back into trunk after review.
> Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.)
> Limitations:
> Union all is not yet supported on Tez
> SMB is not yet supported on Tez
> Bucketed map-join is executed as broadcast join (bucketing is ignored)
> Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR.
> I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well.
> I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try.
>
>
> Diffs
> -----
>
> trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1556067
> trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1556067
> trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1556067
> trunk/conf/hive-default.xml.template 1556067
> trunk/data/conf/tez/hive-site.xml UNKNOWN
> trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1556067
> trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1556067
> trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1556067
> trunk/itests/qtest/pom.xml 1556067
> trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1556067
> trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1556067
> trunk/pom.xml 1556067
> trunk/ql/pom.xml 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1556067
> trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1556067
> trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out UNKNOWN
> trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION
> trunk/ql/src/test/templates/TestCliDriver.vm 1556067
> trunk/ql/src/test/templates/TestCompareCliDriver.vm 1556067
> trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1556067
> trunk/ql/src/test/templates/TestParse.vm 1556067
> trunk/ql/src/test/templates/TestParseNegative.vm 1556067
> trunk/shims/0.23/pom.xml 1556067
> trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1556067
>
> Diff: https://reviews.apache.org/r/16440/diff/
>
>
> Testing
> -------
>
> Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
>
>
> Thanks,
>
> Gunther Hagleitner
>
>
Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/#review31424
-----------------------------------------------------------
Ship it!
Ship It!
- Thejas Nair
On Jan. 9, 2014, 1:52 a.m., Gunther Hagleitner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16440/
> -----------------------------------------------------------
>
> (Updated Jan. 9, 2014, 1:52 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive
>
>
> Description
> -------
>
> I think the Tez branch is at a point where we can consider merging it back into trunk after review.
> Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.)
> Limitations:
> Union all is not yet supported on Tez
> SMB is not yet supported on Tez
> Bucketed map-join is executed as broadcast join (bucketing is ignored)
> Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR.
> I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well.
> I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try.
>
>
> Diffs
> -----
>
> trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1556697
> trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1556697
> trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1556697
> trunk/conf/hive-default.xml.template 1556697
> trunk/data/conf/tez/hive-site.xml UNKNOWN
> trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1556697
> trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1556697
> trunk/hcatalog/webhcat/svr/src/test/data/status/hive/stderr 1556697
> trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1556697
> trunk/itests/qtest/pom.xml 1556697
> trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1556697
> trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1556697
> trunk/pom.xml 1556697
> trunk/ql/pom.xml 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1556697
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1556697
> trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1556697
> trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out UNKNOWN
> trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION
> trunk/ql/src/test/templates/TestCliDriver.vm 1556697
> trunk/ql/src/test/templates/TestCompareCliDriver.vm 1556697
> trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1556697
> trunk/ql/src/test/templates/TestParse.vm 1556697
> trunk/ql/src/test/templates/TestParseNegative.vm 1556697
> trunk/shims/0.23/pom.xml 1556697
> trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1556697
>
> Diff: https://reviews.apache.org/r/16440/diff/
>
>
> Testing
> -------
>
> Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
>
>
> Thanks,
>
> Gunther Hagleitner
>
>
Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/
-----------------------------------------------------------
(Updated Jan. 9, 2014, 1:52 a.m.)
Review request for hive.
Changes
-------
Addresses review comments.
Repository: hive
Description
-------
I think the Tez branch is at a point where we can consider merging it back into trunk after review.
Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.)
Limitations:
Union all is not yet supported on Tez
SMB is not yet supported on Tez
Bucketed map-join is executed as broadcast join (bucketing is ignored)
Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR.
I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well.
I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try.
Diffs (updated)
-----
trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1556697
trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1556697
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1556697
trunk/conf/hive-default.xml.template 1556697
trunk/data/conf/tez/hive-site.xml UNKNOWN
trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1556697
trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1556697
trunk/hcatalog/webhcat/svr/src/test/data/status/hive/stderr 1556697
trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1556697
trunk/itests/qtest/pom.xml 1556697
trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1556697
trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1556697
trunk/pom.xml 1556697
trunk/ql/pom.xml 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1556697
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1556697
trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1556697
trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out UNKNOWN
trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION
trunk/ql/src/test/templates/TestCliDriver.vm 1556697
trunk/ql/src/test/templates/TestCompareCliDriver.vm 1556697
trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1556697
trunk/ql/src/test/templates/TestParse.vm 1556697
trunk/ql/src/test/templates/TestParseNegative.vm 1556697
trunk/shims/0.23/pom.xml 1556697
trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1556697
Diff: https://reviews.apache.org/r/16440/diff/
Testing
-------
Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
Thanks,
Gunther Hagleitner
Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/#review31386
-----------------------------------------------------------
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java
<https://reviews.apache.org/r/16440/#comment59842>
Yes, there are cases where we execute stats via MR on Tez. In this case we need to fall back to the MR aggregator.
- Gunther Hagleitner
On Jan. 7, 2014, 12:50 a.m., Gunther Hagleitner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16440/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2014, 12:50 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive
>
>
> Description
> -------
>
> I think the Tez branch is at a point where we can consider merging it back into trunk after review.
> Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.)
> Limitations:
> Union all is not yet supported on Tez
> SMB is not yet supported on Tez
> Bucketed map-join is executed as broadcast join (bucketing is ignored)
> Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR.
> I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well.
> I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try.
>
>
> Diffs
> -----
>
> trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1556067
> trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1556067
> trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1556067
> trunk/conf/hive-default.xml.template 1556067
> trunk/data/conf/tez/hive-site.xml UNKNOWN
> trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1556067
> trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1556067
> trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1556067
> trunk/itests/qtest/pom.xml 1556067
> trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1556067
> trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1556067
> trunk/pom.xml 1556067
> trunk/ql/pom.xml 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1556067
> trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1556067
> trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION
> trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1556067
> trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION
> trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out UNKNOWN
> trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION
> trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION
> trunk/ql/src/test/templates/TestCliDriver.vm 1556067
> trunk/ql/src/test/templates/TestCompareCliDriver.vm 1556067
> trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1556067
> trunk/ql/src/test/templates/TestParse.vm 1556067
> trunk/ql/src/test/templates/TestParseNegative.vm 1556067
> trunk/shims/0.23/pom.xml 1556067
> trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1556067
>
> Diff: https://reviews.apache.org/r/16440/diff/
>
>
> Testing
> -------
>
> Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
>
>
> Thanks,
>
> Gunther Hagleitner
>
>
Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/
-----------------------------------------------------------
(Updated Jan. 7, 2014, 12:50 a.m.)
Review request for hive.
Changes
-------
The latest patch removes the tez refactoring. (which is in trunk now)
Repository: hive
Description
-------
I think the Tez branch is at a point where we can consider merging it back into trunk after review.
Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.)
Limitations:
Union all is not yet supported on Tez
SMB is not yet supported on Tez
Bucketed map-join is executed as broadcast join (bucketing is ignored)
Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR.
I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well.
I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try.
Diffs (updated)
-----
trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1556067
trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1556067
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1556067
trunk/conf/hive-default.xml.template 1556067
trunk/data/conf/tez/hive-site.xml UNKNOWN
trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1556067
trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1556067
trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1556067
trunk/itests/qtest/pom.xml 1556067
trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1556067
trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1556067
trunk/pom.xml 1556067
trunk/ql/pom.xml 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1556067
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1556067
trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1556067
trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out UNKNOWN
trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION
trunk/ql/src/test/templates/TestCliDriver.vm 1556067
trunk/ql/src/test/templates/TestCompareCliDriver.vm 1556067
trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1556067
trunk/ql/src/test/templates/TestParse.vm 1556067
trunk/ql/src/test/templates/TestParseNegative.vm 1556067
trunk/shims/0.23/pom.xml 1556067
trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1556067
Diff: https://reviews.apache.org/r/16440/diff/
Testing
-------
Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
Thanks,
Gunther Hagleitner
Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/
-----------------------------------------------------------
(Updated Dec. 27, 2013, 6:38 a.m.)
Review request for hive.
Changes
-------
Updated patch contains the latest changes (rebase + class path stuff)
Repository: hive
Description
-------
I think the Tez branch is at a point where we can consider merging it back into trunk after review.
Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.)
Limitations:
Union all is not yet supported on Tez
SMB is not yet supported on Tez
Bucketed map-join is executed as broadcast join (bucketing is ignored)
Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR.
I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well.
I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try.
Diffs (updated)
-----
trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1553631
trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1553631
trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1553631
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1553631
trunk/conf/hive-default.xml.template 1553631
trunk/data/conf/tez/hive-site.xml UNKNOWN
trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1553631
trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1553631
trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1553631
trunk/itests/qtest/pom.xml 1553631
trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1553631
trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1553631
trunk/pom.xml 1553631
trunk/ql/pom.xml 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableLoader.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MuxOperator.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecReducer.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HashTableLoader.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/DefaultGraphWalker.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TableAccessAnalyzer.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1553631
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1553631
trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestOperators.java 1553631
trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 1553631
trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1553631
trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION
trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 1553631
trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 1553631
trunk/ql/src/test/results/clientpositive/join32.q.out 1553631
trunk/ql/src/test/results/clientpositive/join32_lessSize.q.out 1553631
trunk/ql/src/test/results/clientpositive/join33.q.out 1553631
trunk/ql/src/test/results/clientpositive/join34.q.out 1553631
trunk/ql/src/test/results/clientpositive/multi_insert_move_tasks_share_dependencies.q.out 1553631
trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out UNKNOWN
trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/vector_left_outer_join.q.out 1553631
trunk/ql/src/test/results/clientpositive/vectorized_mapjoin.q.out 1553631
trunk/ql/src/test/results/compiler/plan/case_sensitivity.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/cast1.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/groupby1.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/groupby2.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/groupby3.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/groupby4.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/groupby5.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/groupby6.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input1.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input2.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input20.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input3.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input4.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input5.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input6.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input7.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input8.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input9.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input_part1.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input_testsequencefile.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input_testxpath.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/input_testxpath2.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/join1.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/join2.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/join3.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/join4.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/join5.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/join6.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/join7.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/join8.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/sample7.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/subq.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/udf1.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/udf4.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/udf6.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/udf_case.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/udf_when.q.xml 1553631
trunk/ql/src/test/results/compiler/plan/union.q.xml 1553631
trunk/ql/src/test/templates/TestCliDriver.vm 1553631
trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1553631
trunk/ql/src/test/templates/TestParse.vm 1553631
trunk/ql/src/test/templates/TestParseNegative.vm 1553631
trunk/shims/0.23/pom.xml 1553631
trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1553631
Diff: https://reviews.apache.org/r/16440/diff/
Testing
-------
Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
Thanks,
Gunther Hagleitner
Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/
-----------------------------------------------------------
(Updated Dec. 23, 2013, 3:24 p.m.)
Review request for hive.
Repository: hive
Description
-------
I think the Tez branch is at a point where we can consider merging it back into trunk after review.
Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.)
Limitations:
Union all is not yet supported on Tez
SMB is not yet supported on Tez
Bucketed map-join is executed as broadcast join (bucketing is ignored)
Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR.
I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well.
I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try.
Diffs (updated)
-----
trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1553125
trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1553125
trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1553125
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1553125
trunk/conf/hive-default.xml.template 1553125
trunk/data/conf/tez/hive-site.xml UNKNOWN
trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1553125
trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1553125
trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1553125
trunk/itests/qtest/pom.xml 1553125
trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1553125
trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1553125
trunk/pom.xml 1553125
trunk/ql/pom.xml 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableLoader.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MuxOperator.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecReducer.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HashTableLoader.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/DefaultGraphWalker.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TableAccessAnalyzer.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1553125
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1553125
trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestOperators.java 1553125
trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 1553125
trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION
trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1553125
trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out UNKNOWN
trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION
trunk/ql/src/test/results/compiler/plan/case_sensitivity.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/cast1.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/groupby1.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/groupby2.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/groupby3.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/groupby4.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/groupby5.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/groupby6.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input1.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input2.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input20.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input3.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input4.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input5.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input6.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input7.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input8.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input9.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input_part1.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input_testsequencefile.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input_testxpath.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/input_testxpath2.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/join1.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/join2.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/join3.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/join4.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/join5.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/join6.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/join7.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/join8.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/sample7.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/subq.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/udf1.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/udf4.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/udf6.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/udf_case.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/udf_when.q.xml 1553125
trunk/ql/src/test/results/compiler/plan/union.q.xml 1553125
trunk/ql/src/test/templates/TestCliDriver.vm 1553125
trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1553125
trunk/ql/src/test/templates/TestParse.vm 1553125
trunk/ql/src/test/templates/TestParseNegative.vm 1553125
trunk/shims/0.23/pom.xml 1553125
trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1553125
Diff: https://reviews.apache.org/r/16440/diff/
Testing
-------
Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver.
Thanks,
Gunther Hagleitner