You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Zoltan Haindrich <ki...@rxd.hu> on 2018/01/30 18:14:02 UTC
Review Request 65422: HIVE-17626
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
-----------------------------------------------------------
Review request for hive and Ashutosh Chauhan.
Repository: hive-git
Description
-------
preview
Diffs
-----
cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a
hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java ad31287879
hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 533f0bcd6f
itests/src/test/resources/testconfiguration.properties d86ff58840
ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33
ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08
ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 190771ea6b
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java cbadfa4f07
ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java dcf8d31eaf
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java d56002d192
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/65422/diff/1/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 3691 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950474#file1950474line3696>
> >
> > Instead of config this should be explain modifier. WE already have explain rewrite select .. We similarly can add explain reoptimize select ...
yes...I agree; it turned out that its very inconvinient to use it this way...
I've employed a semanticAnlayzer hook to handle the reoptimize keyword
> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 5066 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950474#file1950474line5071>
> >
> > Instead of iterating over _this_ which can be very large, more efficient is to iterate on other list.
I wasn't aware that the iterator() creates a new map on the fly....I'm now using getProps() to get access to the actual values
> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
> > Lines 127 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950478#file1950478line127>
> >
> > Currently its only reexcuted once. Alternatively, we can keep re-running it if it fails again. e.g. in case of OOM, its possible that there are many joins which are mis-planed, but we get stats only for first join.
> > To avoid, very large number of retrials we can limit to some max attempts.
I aggree
> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java
> > Lines 21 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950481#file1950481line21>
> >
> > Incorrect import ?
I've just taken a look at null analysis; but it detects too many issues to just turn on...so I'll remove it for now :)
> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java
> > Lines 50 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950503#file1950503line50>
> >
> > Instead of relying on ids, better is to use (and extend) logic in SharedWorkOptimizer::compareOperator() ?
t
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review196950
-----------------------------------------------------------
On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2018, 6:13 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> preview
>
>
> Diffs
> -----
>
> cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java ad31287879
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 533f0bcd6f
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9
> ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33
> ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08
> ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
> ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 190771ea6b
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java cbadfa4f07
> ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java dcf8d31eaf
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java d56002d192
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/65422/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Zoltan Haindrich
>
>
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 3689 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950474#file1950474line3694>
> >
> > Default should be reoptimize.
ok; but disabled it in all the other conf-s for now
> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
> > Lines 36 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950478#file1950478line36>
> >
> > Comments on what this driver does?
I've added an apidoc; but I feel more and more that this should be probably re-thinked - and instead of using an abstract type; the re-execution parts could be "pluggable" ; which could enable more than 1 at a time ... this would be trivial after finishing this; currently I feel that the patch keeps some questions open elsewhere.
> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
> > Lines 4748-4749 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950493#file1950493line4748>
> >
> > To avoid this we should switch to find equivalence of 2 operators based on their signature. e.g., Operator::logicalEquals().
removed from here
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review196950
-----------------------------------------------------------
On Feb. 26, 2018, 6:19 p.m., Zoltan Haindrich wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2018, 6:19 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> preview
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6
> data/conf/llap/hive-site.xml c4c299c5de
> itests/src/test/resources/testconfiguration.properties 2776fe95f1
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae
> ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93
> ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
> ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
> ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
> ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
> ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1
> ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f9a6386ecf
> ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
> ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
> ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out cab45c9410
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out 43ca0d9ad7
> ql/src/test/results/clientpositive/llap/join32_lessSize.q.out 5b5be13702
> ql/src/test/results/clientpositive/llap/join46.q.out 4625b9fed8
> ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 4dd7432ac5
> ql/src/test/results/clientpositive/llap/llap_partitioned.q.out ac754674ea
> ql/src/test/results/clientpositive/llap/mapjoin46.q.out f530c1e8e1
> ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68
> ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708
> ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e
> ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/semijoin_hint.q.out cdfa73349b
> ql/src/test/results/clientpositive/llap/subquery_in_having.q.out cc0b6a597a
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 67a22c812f
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out 50c18c8a68
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out cec3daa085
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 89eab37d07
> ql/src/test/results/clientpositive/llap/tez_join_tests.q.out 75879fb157
> ql/src/test/results/clientpositive/llap/tez_joins_explain.q.out 3790b37760
> ql/src/test/results/clientpositive/llap/vector_reduce_groupby_duplicate_cols.q.out 07c1e4188c
> ql/src/test/results/clientpositive/perf/tez/query72.q.out 55ddd5a0f8
> ql/src/test/results/clientpositive/perf/tez/query78.q.out 9c2d7b3491
> ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af
>
>
> Diff: https://reviews.apache.org/r/65422/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Zoltan Haindrich
>
>
Re: Review Request 65422: HIVE-17626
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review196950
-----------------------------------------------------------
First pass review.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3689 (patched)
<https://reviews.apache.org/r/65422/#comment276939>
Default should be reoptimize.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3691 (patched)
<https://reviews.apache.org/r/65422/#comment276940>
Instead of config this should be explain modifier. WE already have explain rewrite select .. We similarly can add explain reoptimize select ...
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 5066 (patched)
<https://reviews.apache.org/r/65422/#comment276964>
Instead of iterating over _this_ which can be very large, more efficient is to iterate on other list.
ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
Lines 36 (patched)
<https://reviews.apache.org/r/65422/#comment276962>
Comments on what this driver does?
ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
Lines 127 (patched)
<https://reviews.apache.org/r/65422/#comment276958>
Currently its only reexcuted once. Alternatively, we can keep re-running it if it fails again. e.g. in case of OOM, its possible that there are many joins which are mis-planed, but we get stats only for first join.
To avoid, very large number of retrials we can limit to some max attempts.
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java
Lines 21 (patched)
<https://reviews.apache.org/r/65422/#comment276959>
Incorrect import ?
ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java
Lines 25 (patched)
<https://reviews.apache.org/r/65422/#comment276961>
Add comments on what this Driver does.
ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
Lines 29 (patched)
<https://reviews.apache.org/r/65422/#comment276963>
java class docs.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
Lines 4748-4749 (patched)
<https://reviews.apache.org/r/65422/#comment276968>
To avoid this we should switch to find equivalence of 2 operators based on their signature. e.g., Operator::logicalEquals().
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 2364 (patched)
<https://reviews.apache.org/r/65422/#comment276965>
This should extend to all operators.
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java
Lines 33 (patched)
<https://reviews.apache.org/r/65422/#comment276966>
javadoc
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java
Lines 32 (patched)
<https://reviews.apache.org/r/65422/#comment276969>
Operator::logicalEquals() ?
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java
Lines 50 (patched)
<https://reviews.apache.org/r/65422/#comment276967>
Instead of relying on ids, better is to use (and extend) logic in SharedWorkOptimizer::compareOperator() ?
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java
Lines 56 (patched)
<https://reviews.apache.org/r/65422/#comment276970>
LOG.debug
- Ashutosh Chauhan
On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2018, 6:13 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> preview
>
>
> Diffs
> -----
>
> cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java ad31287879
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 533f0bcd6f
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9
> ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33
> ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08
> ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
> ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 190771ea6b
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java cbadfa4f07
> ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java dcf8d31eaf
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java d56002d192
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/65422/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Zoltan Haindrich
>
>
Re: Review Request 65422: HIVE-17626
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review198652
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 653 (patched)
<https://reviews.apache.org/r/65422/#comment278887>
I meant this should be run only if first query execution failed. Also when it runs it should simultaneously collect stats from executed tree.
Actually I am not sure how its computing even signatures, since its not passing any reference to plan itself.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java
Lines 35 (patched)
<https://reviews.apache.org/r/65422/#comment278888>
Add comment about what is key and what is value.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java
Lines 68 (patched)
<https://reviews.apache.org/r/65422/#comment278889>
I think we agreed on that signature of operator is stored in a flat structure which can be compared without traversing full tree. Can you leave a TODO for this?
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 2506 (patched)
<https://reviews.apache.org/r/65422/#comment278890>
Whats the need for this condition? Runtime stats are always accurate even when state is determined as complete during compile time.
- Ashutosh Chauhan
On March 5, 2018, 8:08 a.m., Zoltan Haindrich wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> -----------------------------------------------------------
>
> (Updated March 5, 2018, 8:08 a.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-17626
> https://issues.apache.org/jira/browse/HIVE-17626
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> preview
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eca35731292df6e1adef7ec04164e39a7baf13e1
> data/conf/llap/hive-site.xml cdda875ddbcc7b6fe977770523779487d30a043a
> druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java PRE-CREATION
> itests/src/test/resources/testconfiguration.properties 32b75518cfea11367e0adee26b08c47887a59b54
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2d0aca0a11bda6b9dce31ccf6acfcfd81e412d5a
> itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java c01d87bf51942f01a38e21ffbd66a9313a91e9a6
> ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b6dadee4a5cd640c8cea39669869477
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2d7e4597de623d892702cae6e732ec5eb09d87da
> ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f335c6806460fb6b83ee4da8bf842bd5a
> ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/IDriver.java d4494cc72ed706886f829195a58514082793be4f
> ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1ecddc720d6b6e3680194354a6e157c
> ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b1812903a1cff9d3f52078eff92fc2aa3cec5
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f91151773edd49f42145209b8877cc8
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 085d6a7d94ca40faed3ec87f95962cb7fd774100
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 0d31e6e42203b3a9146b678b1694c457dd56ab2a
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c467df479be0b5984fe2a9bf9f5ab7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b1412ebb9783da3bf28a9c1b9f10dbc
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d981ae95f02d1d5acded1d8661272128
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd746869540df35c235ab28a33d2625e9a8b5
> ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f9cbe1ed969bd7ac53196dd49d081a3
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e1cbae2386cd2f475e47b44f750d088
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 190771ea6b1cbf4b669a8919271b25a689af941b
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/Signature.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/SignatureUtils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java d1c53cf34515d73f24425a7b3a6afdf2d076b9d4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java cbadfa4f07e04f6575faa9147df687cb860cb5ca
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c436e44258947c459e3e0c0e24eb9d622
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 907362312da329010dcde53d8fb6f5790f022fbd
> ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 35f9edfecc74b85fe64a2cbe6290ca44a341c63e
> ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 714cf3914b0190c360a1246934902db1c8ab4012
> ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba819fd73d1bded1a7a246049a0a2f420d
> ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 7332693513fe4a9374422a7671e82351590988f2
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 5d3fdb8b632c9bfd3ee40b0f6e30b71e5aa3cf46
> ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 92b8031da6d1157d8f979455f8a8a1556b1c5ccd
> ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08e802c4869a9aede5a957efb1358863
> ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java e90a3987ec3ed3a60728120a0d3b624badbec922
> ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3abea1614d421acb44273717357eb8f0d
> ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af283cda23becb2ceaea6b466ace3e8b54
> ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1c19bd9c2bf30f760074d71ba5c6265
> ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 3837a4993494ec7c4bc96b966b740aed0a3d8482
> ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00930d3a39c3ebc14a2e4dfa592d404d
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 23dd19bbaeb8b00a975e6b9b70eda1d91a697828
> ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a268c0b54006435b09e4e9a40ba41c59c
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09f81a79ec3462f7641d4f8fb7159cd7
> ql/src/java/org/apache/hadoop/hive/ql/plan/ScriptDesc.java 858de98229cf1884bd31bfaee67396bd32208418
> ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e38e7e468fd6eaddfb5f5749b27dd44f8682f0f0
> ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6b607f8d6fef5e36fee9ab36a741718
> ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b9fbb5cf248d76205e47dbaa113026f
> ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java cf8e6e547ed8bd39dd353a3f6fb075ad71faf55d
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda152b3a0f3d47d4ec4bb9960e4447349
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOpSigFactory.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
> ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55a0342a0d51dc938028b071306ddadd
> ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f876c3bd249e7230d2fbc0a33e354d26
> ql/src/test/org/apache/hive/testutils/TestX1.java PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 981f2607fb537b0754c4cfa7c7201b1985dd29af
> ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1d3430743d7816f65b51174ef61e2a8
> ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f6852e2fe72e67a4cfe9688bae4291625
> ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a8f769a48e95b54cd6ca7e698cf9963
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out c5302c347068eb7a64becfaeae769babda4e7615
> ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e2f682a6442df574a291bd101356efb
> ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f404966b8971fba739ef4d9091e4140cea3
> ql/src/test/results/clientpositive/show_functions.q.out 43e4a5de393d4b23c4c0257f08c32dd650eaaadc
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java PRE-CREATION
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java 3d64c7341a17b0f7e7213cc1606adebc8874820a
>
>
> Diff: https://reviews.apache.org/r/65422/diff/7/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Zoltan Haindrich
>
>
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
-----------------------------------------------------------
(Updated March 6, 2018, 11:06 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
11 ; address review comments; disable opstats collection for successfull first query ; added a conf variable to enable it for tests.
Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626
Repository: hive-git
Description
-------
preview
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edea129579
data/conf/llap/hive-site.xml c4c299c5de
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java cbeac2c00a
itests/src/test/resources/testconfiguration.properties e8aa827523
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 07e6eaad88
itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java c01d87bf51
ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 6999777297
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 08b791ad42
ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 131127e50d
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 77e9263e0f
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 1e28ca843f
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 2b345d6ec7
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 085d6a7d94
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 0d31e6e422
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java 8c1bcb3f62
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/Signature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/SignatureUtils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java d1c53cf345
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 0c6aece1df
ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 35f9edfecc
ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 714cf3914b
ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81
ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 7332693513
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 5d3fdb8b63
ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e
ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08
ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b
ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab
ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28
ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1
ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 3837a49934
ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00
ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2
ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a26
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09
ql/src/java/org/apache/hadoop/hive/ql/plan/ScriptDesc.java 858de98229
ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e38e7e468f
ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 59968fa8ef
ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java cf8e6e547e
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java 6c469bf944
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOpSigFactory.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55
ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f8
ql/src/test/queries/clientnegative/bad_exec_hooks.q 709d8d9c85
ql/src/test/queries/clientpositive/hook_order.q b6753b308b
ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out cd3023f9d5
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out e7a1a5b675
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 75a2908fd1
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out d9572e73fe
ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f4049
ql/src/test/results/clientpositive/show_functions.q.out 82fb5f4e0f
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java d3be5dd0a2
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java 3d64c7341a
Diff: https://reviews.apache.org/r/65422/diff/9/
Changes: https://reviews.apache.org/r/65422/diff/8-9/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
-----------------------------------------------------------
(Updated March 5, 2018, 10:12 p.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
10
Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626
Repository: hive-git
Description
-------
preview
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edea129579
data/conf/llap/hive-site.xml c4c299c5de
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java cbeac2c00a
itests/src/test/resources/testconfiguration.properties e8aa827523
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 07e6eaad88
itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java c01d87bf51
ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 6999777297
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 08b791ad42
ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 131127e50d
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 77e9263e0f
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 1e28ca843f
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 2b345d6ec7
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 085d6a7d94
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 0d31e6e422
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java 8c1bcb3f62
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/Signature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/SignatureUtils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java d1c53cf345
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 0c6aece1df
ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 35f9edfecc
ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 714cf3914b
ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81
ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 7332693513
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 5d3fdb8b63
ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e
ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08
ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b
ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab
ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28
ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1
ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 3837a49934
ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00
ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2
ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a26
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09
ql/src/java/org/apache/hadoop/hive/ql/plan/ScriptDesc.java 858de98229
ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e38e7e468f
ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 59968fa8ef
ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java cf8e6e547e
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java 6c469bf944
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOpSigFactory.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55
ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f8
ql/src/test/org/apache/hive/testutils/TestX1.java PRE-CREATION
ql/src/test/queries/clientnegative/bad_exec_hooks.q 709d8d9c85
ql/src/test/queries/clientpositive/hook_order.q b6753b308b
ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out cd3023f9d5
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out e7a1a5b675
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 75a2908fd1
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out d9572e73fe
ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f4049
ql/src/test/results/clientpositive/show_functions.q.out 82fb5f4e0f
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java d3be5dd0a2
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java 3d64c7341a
Diff: https://reviews.apache.org/r/65422/diff/8/
Changes: https://reviews.apache.org/r/65422/diff/7-8/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
-----------------------------------------------------------
(Updated March 5, 2018, 8:08 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
08
Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626
Repository: hive-git
Description
-------
preview
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eca35731292df6e1adef7ec04164e39a7baf13e1
data/conf/llap/hive-site.xml cdda875ddbcc7b6fe977770523779487d30a043a
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java PRE-CREATION
itests/src/test/resources/testconfiguration.properties 32b75518cfea11367e0adee26b08c47887a59b54
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2d0aca0a11bda6b9dce31ccf6acfcfd81e412d5a
itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java c01d87bf51942f01a38e21ffbd66a9313a91e9a6
ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b6dadee4a5cd640c8cea39669869477
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2d7e4597de623d892702cae6e732ec5eb09d87da
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f335c6806460fb6b83ee4da8bf842bd5a
ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/IDriver.java d4494cc72ed706886f829195a58514082793be4f
ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1ecddc720d6b6e3680194354a6e157c
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b1812903a1cff9d3f52078eff92fc2aa3cec5
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f91151773edd49f42145209b8877cc8
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 085d6a7d94ca40faed3ec87f95962cb7fd774100
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 0d31e6e42203b3a9146b678b1694c457dd56ab2a
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c467df479be0b5984fe2a9bf9f5ab7a
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b1412ebb9783da3bf28a9c1b9f10dbc
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d981ae95f02d1d5acded1d8661272128
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd746869540df35c235ab28a33d2625e9a8b5
ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f9cbe1ed969bd7ac53196dd49d081a3
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e1cbae2386cd2f475e47b44f750d088
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 190771ea6b1cbf4b669a8919271b25a689af941b
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/Signature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/SignatureUtils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java d1c53cf34515d73f24425a7b3a6afdf2d076b9d4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java cbadfa4f07e04f6575faa9147df687cb860cb5ca
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c436e44258947c459e3e0c0e24eb9d622
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 907362312da329010dcde53d8fb6f5790f022fbd
ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 35f9edfecc74b85fe64a2cbe6290ca44a341c63e
ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 714cf3914b0190c360a1246934902db1c8ab4012
ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba819fd73d1bded1a7a246049a0a2f420d
ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 7332693513fe4a9374422a7671e82351590988f2
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 5d3fdb8b632c9bfd3ee40b0f6e30b71e5aa3cf46
ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 92b8031da6d1157d8f979455f8a8a1556b1c5ccd
ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08e802c4869a9aede5a957efb1358863
ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java e90a3987ec3ed3a60728120a0d3b624badbec922
ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3abea1614d421acb44273717357eb8f0d
ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af283cda23becb2ceaea6b466ace3e8b54
ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1c19bd9c2bf30f760074d71ba5c6265
ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 3837a4993494ec7c4bc96b966b740aed0a3d8482
ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00930d3a39c3ebc14a2e4dfa592d404d
ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 23dd19bbaeb8b00a975e6b9b70eda1d91a697828
ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a268c0b54006435b09e4e9a40ba41c59c
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09f81a79ec3462f7641d4f8fb7159cd7
ql/src/java/org/apache/hadoop/hive/ql/plan/ScriptDesc.java 858de98229cf1884bd31bfaee67396bd32208418
ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e38e7e468fd6eaddfb5f5749b27dd44f8682f0f0
ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6b607f8d6fef5e36fee9ab36a741718
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b9fbb5cf248d76205e47dbaa113026f
ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java cf8e6e547ed8bd39dd353a3f6fb075ad71faf55d
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda152b3a0f3d47d4ec4bb9960e4447349
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOpSigFactory.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55a0342a0d51dc938028b071306ddadd
ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f876c3bd249e7230d2fbc0a33e354d26
ql/src/test/org/apache/hive/testutils/TestX1.java PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 981f2607fb537b0754c4cfa7c7201b1985dd29af
ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1d3430743d7816f65b51174ef61e2a8
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f6852e2fe72e67a4cfe9688bae4291625
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a8f769a48e95b54cd6ca7e698cf9963
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out c5302c347068eb7a64becfaeae769babda4e7615
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e2f682a6442df574a291bd101356efb
ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f404966b8971fba739ef4d9091e4140cea3
ql/src/test/results/clientpositive/show_functions.q.out 43e4a5de393d4b23c4c0257f08c32dd650eaaadc
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java PRE-CREATION
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java 3d64c7341a17b0f7e7213cc1606adebc8874820a
Diff: https://reviews.apache.org/r/65422/diff/7/
Changes: https://reviews.apache.org/r/65422/diff/6-7/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
> On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote:
> > data/conf/hive-site.xml
> > Lines 338 (patched)
> > <https://reviews.apache.org/r/65422/diff/6/?file=1968339#file1968339line338>
> >
> > Is there a reason for this? AFAICT existing tests should not get affected by this, so this shouldn't impact tests, so we might as well turn it to true.
at some point I was not sure if this patch is broking some tests or not;
but I'll submit it with and without this for testing right now...I think some negative tests will be broken - because the reexecution will cause some new log lines...I'll remove this if possible in the next patch
> On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java
> > Lines 652 (patched)
> > <https://reviews.apache.org/r/65422/diff/6/?file=1968353#file1968353line661>
> >
> > We want to compute operator signatures and store stats corresponding to it only if we decide to reexecute it. Currently, it seems it unconditionally does this. Also, at this point plan hasn't been executed, so there are no stats for it. So, is this only computing signatures?
added a conditional; yes, it's only computing the signatures - the idea is that this call will calculate the "plan independent" part of the planmapper.
> On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
> > Lines 1581 (patched)
> > <https://reviews.apache.org/r/65422/diff/6/?file=1968362#file1968362line1588>
> >
> > Also add a comment that its currently used only when reoptimization is attempted to match operators on 2 trees.
I think in HIVE-18703 I will probably move this method to somewhere else...
> On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java
> > Lines 111 (patched)
> > <https://reviews.apache.org/r/65422/diff/6/?file=1968412#file1968412line111>
> >
> > Can there be a semantics about whether subsequent plugin run depends on results on plugins before it? I am thinking of a scenario where ReOptimize plugin executes first but it fails to generate an alternate plan (either because it can't get stats for an operator it was looking for or can't match op trees or whatever). So, we will know before executing second time via Reoptimize plugin that it won't be successful. At that time we make Overlay plugin go with auto.convert.join=off. I think this will be useful, although can be a follow-up.
that would be an interesting extension...I don't know how would it be the best right now...but I think this should be investigated in a followup
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review198543
-----------------------------------------------------------
On March 1, 2018, 3:35 p.m., Zoltan Haindrich wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> -----------------------------------------------------------
>
> (Updated March 1, 2018, 3:35 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-17626
> https://issues.apache.org/jira/browse/HIVE-17626
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> preview
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6
> data/conf/hive-site.xml b56cbd2469
> data/conf/llap/hive-site.xml c4c299c5de
> data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7
> data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f
> data/conf/rlist/hive-site.xml 630e481edc
> data/conf/spark/local/hive-site.xml 8ff6256638
> data/conf/spark/standalone/hive-site.xml 84851c769c
> data/conf/spark/yarn-client/hive-site.xml 6c633623ab
> data/conf/tez/hive-site.xml 236adc7087
> druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java cbeac2c00a
> itests/src/test/resources/testconfiguration.properties 2776fe95f1
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae
> itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java c01d87bf51
> ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93
> ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
> ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
> ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
> ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 08b791ad42
> ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 131127e50d
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03
> ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 1e28ca843f
> ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 2b345d6ec7
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 085d6a7d94
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 0d31e6e422
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
> ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java 8c1bcb3f62
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java d1c53cf345
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1
> ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 714cf3914b
> ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81
> ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 7332693513
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 5d3fdb8b63
> ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e
> ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08
> ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b
> ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab
> ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28
> ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1
> ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 3837a49934
> ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2
> ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a26
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09
> ql/src/java/org/apache/hadoop/hive/ql/plan/ScriptDesc.java 858de98229
> ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e38e7e468f
> ql/src/java/org/apache/hadoop/hive/ql/plan/Signature.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/SignatureUtils1.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
> ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 59968fa8ef
> ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java cf8e6e547e
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java 6c469bf944
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOpSigFactory.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
> ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55
> ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f8
> ql/src/test/org/apache/hive/testutils/TestX1.java PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
> ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
> ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68
> ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708
> ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e
> ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f4049
> ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java d3be5dd0a2
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java 3d64c7341a
>
>
> Diff: https://reviews.apache.org/r/65422/diff/6/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Zoltan Haindrich
>
>
Re: Review Request 65422: HIVE-17626
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review198543
-----------------------------------------------------------
data/conf/hive-site.xml
Lines 338 (patched)
<https://reviews.apache.org/r/65422/#comment278729>
Is there a reason for this? AFAICT existing tests should not get affected by this, so this shouldn't impact tests, so we might as well turn it to true.
data/conf/llap/hive-site.xml
Lines 363 (patched)
<https://reviews.apache.org/r/65422/#comment278730>
Set to true ?
ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 652 (patched)
<https://reviews.apache.org/r/65422/#comment278763>
We want to compute operator signatures and store stats corresponding to it only if we decide to reexecute it. Currently, it seems it unconditionally does this. Also, at this point plan hasn't been executed, so there are no stats for it. So, is this only computing signatures?
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
Lines 1581 (patched)
<https://reviews.apache.org/r/65422/#comment278737>
Also add a comment that its currently used only when reoptimization is attempted to match operators on 2 trees.
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
Lines 306 (patched)
<https://reviews.apache.org/r/65422/#comment278739>
Add it in non reserved list in IdentifiersParser.g
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java
Lines 36 (patched)
<https://reviews.apache.org/r/65422/#comment278762>
These are not just filters, better name it as operators?
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java
Lines 111 (patched)
<https://reviews.apache.org/r/65422/#comment278765>
Can there be a semantics about whether subsequent plugin run depends on results on plugins before it? I am thinking of a scenario where ReOptimize plugin executes first but it fails to generate an alternate plan (either because it can't get stats for an operator it was looking for or can't match op trees or whatever). So, we will know before executing second time via Reoptimize plugin that it won't be successful. At that time we make Overlay plugin go with auto.convert.join=off. I think this will be useful, although can be a follow-up.
- Ashutosh Chauhan
On March 1, 2018, 3:35 p.m., Zoltan Haindrich wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> -----------------------------------------------------------
>
> (Updated March 1, 2018, 3:35 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-17626
> https://issues.apache.org/jira/browse/HIVE-17626
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> preview
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6
> data/conf/hive-site.xml b56cbd2469
> data/conf/llap/hive-site.xml c4c299c5de
> data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7
> data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f
> data/conf/rlist/hive-site.xml 630e481edc
> data/conf/spark/local/hive-site.xml 8ff6256638
> data/conf/spark/standalone/hive-site.xml 84851c769c
> data/conf/spark/yarn-client/hive-site.xml 6c633623ab
> data/conf/tez/hive-site.xml 236adc7087
> druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java cbeac2c00a
> itests/src/test/resources/testconfiguration.properties 2776fe95f1
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae
> itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java c01d87bf51
> ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93
> ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
> ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
> ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
> ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 08b791ad42
> ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 131127e50d
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03
> ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 1e28ca843f
> ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 2b345d6ec7
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 085d6a7d94
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 0d31e6e422
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
> ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java 8c1bcb3f62
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java d1c53cf345
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1
> ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 714cf3914b
> ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81
> ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 7332693513
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 5d3fdb8b63
> ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e
> ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08
> ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b
> ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab
> ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28
> ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1
> ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 3837a49934
> ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2
> ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a26
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09
> ql/src/java/org/apache/hadoop/hive/ql/plan/ScriptDesc.java 858de98229
> ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e38e7e468f
> ql/src/java/org/apache/hadoop/hive/ql/plan/Signature.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/SignatureUtils1.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
> ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 59968fa8ef
> ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java cf8e6e547e
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java 6c469bf944
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOpSigFactory.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
> ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55
> ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f8
> ql/src/test/org/apache/hive/testutils/TestX1.java PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
> ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
> ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68
> ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708
> ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e
> ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f4049
> ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java d3be5dd0a2
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java 3d64c7341a
>
>
> Diff: https://reviews.apache.org/r/65422/diff/6/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Zoltan Haindrich
>
>
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
-----------------------------------------------------------
(Updated March 1, 2018, 3:35 p.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
06
Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626
Repository: hive-git
Description
-------
preview
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6
data/conf/hive-site.xml b56cbd2469
data/conf/llap/hive-site.xml c4c299c5de
data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7
data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f
data/conf/rlist/hive-site.xml 630e481edc
data/conf/spark/local/hive-site.xml 8ff6256638
data/conf/spark/standalone/hive-site.xml 84851c769c
data/conf/spark/yarn-client/hive-site.xml 6c633623ab
data/conf/tez/hive-site.xml 236adc7087
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java cbeac2c00a
itests/src/test/resources/testconfiguration.properties 2776fe95f1
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae
itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java c01d87bf51
ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 08b791ad42
ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 131127e50d
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 1e28ca843f
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 2b345d6ec7
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 085d6a7d94
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 0d31e6e422
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java 8c1bcb3f62
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java d1c53cf345
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1
ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 714cf3914b
ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81
ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 7332693513
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 5d3fdb8b63
ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e
ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08
ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b
ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab
ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28
ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1
ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 3837a49934
ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00
ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2
ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a26
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09
ql/src/java/org/apache/hadoop/hive/ql/plan/ScriptDesc.java 858de98229
ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e38e7e468f
ql/src/java/org/apache/hadoop/hive/ql/plan/Signature.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/SignatureUtils1.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 59968fa8ef
ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java cf8e6e547e
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java 6c469bf944
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOpSigFactory.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55
ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f8
ql/src/test/org/apache/hive/testutils/TestX1.java PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e
ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f4049
ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java d3be5dd0a2
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java 3d64c7341a
Diff: https://reviews.apache.org/r/65422/diff/6/
Changes: https://reviews.apache.org/r/65422/diff/5-6/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
-----------------------------------------------------------
(Updated Feb. 27, 2018, 6:36 p.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
5
Repository: hive-git
Description
-------
preview
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6
data/conf/hive-site.xml b56cbd2469
data/conf/llap/hive-site.xml c4c299c5de
data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7
data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f
data/conf/rlist/hive-site.xml 630e481edc
data/conf/spark/local/hive-site.xml 8ff6256638
data/conf/spark/standalone/hive-site.xml 84851c769c
data/conf/spark/yarn-client/hive-site.xml 6c633623ab
data/conf/tez/hive-site.xml 236adc7087
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java cbeac2c00a
itests/src/test/resources/testconfiguration.properties 2776fe95f1
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae
itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java c01d87bf51
ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 08b791ad42
ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 131127e50d
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 1e28ca843f
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 2b345d6ec7
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 085d6a7d94
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 0d31e6e422
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java 8c1bcb3f62
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1
ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java 6c469bf944
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55
ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f8
ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e
ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f4049
ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java d3be5dd0a2
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java 3d64c7341a
Diff: https://reviews.apache.org/r/65422/diff/5/
Changes: https://reviews.apache.org/r/65422/diff/4-5/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
-----------------------------------------------------------
(Updated Feb. 27, 2018, 11:17 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
03
Repository: hive-git
Description
-------
preview
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6
data/conf/hive-site.xml b56cbd2469
data/conf/llap/hive-site.xml c4c299c5de
data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7
data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f
data/conf/rlist/hive-site.xml 630e481edc
data/conf/spark/local/hive-site.xml 8ff6256638
data/conf/spark/standalone/hive-site.xml 84851c769c
data/conf/spark/yarn-client/hive-site.xml 6c633623ab
data/conf/tez/hive-site.xml 236adc7087
itests/src/test/resources/testconfiguration.properties 2776fe95f1
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae
ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1
ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e
ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af
Diff: https://reviews.apache.org/r/65422/diff/4/
Changes: https://reviews.apache.org/r/65422/diff/3-4/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
-----------------------------------------------------------
(Updated Feb. 26, 2018, 6:19 p.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
update to 02
Repository: hive-git
Description
-------
preview
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6
data/conf/llap/hive-site.xml c4c299c5de
itests/src/test/resources/testconfiguration.properties 2776fe95f1
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae
ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f9a6386ecf
ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out cab45c9410
ql/src/test/results/clientpositive/llap/explainuser_1.q.out 43ca0d9ad7
ql/src/test/results/clientpositive/llap/join32_lessSize.q.out 5b5be13702
ql/src/test/results/clientpositive/llap/join46.q.out 4625b9fed8
ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 4dd7432ac5
ql/src/test/results/clientpositive/llap/llap_partitioned.q.out ac754674ea
ql/src/test/results/clientpositive/llap/mapjoin46.q.out f530c1e8e1
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e
ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/semijoin_hint.q.out cdfa73349b
ql/src/test/results/clientpositive/llap/subquery_in_having.q.out cc0b6a597a
ql/src/test/results/clientpositive/llap/subquery_multi.q.out 67a22c812f
ql/src/test/results/clientpositive/llap/subquery_notin.q.out 50c18c8a68
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out cec3daa085
ql/src/test/results/clientpositive/llap/subquery_select.q.out 89eab37d07
ql/src/test/results/clientpositive/llap/tez_join_tests.q.out 75879fb157
ql/src/test/results/clientpositive/llap/tez_joins_explain.q.out 3790b37760
ql/src/test/results/clientpositive/llap/vector_reduce_groupby_duplicate_cols.q.out 07c1e4188c
ql/src/test/results/clientpositive/perf/tez/query72.q.out 55ddd5a0f8
ql/src/test/results/clientpositive/perf/tez/query78.q.out 9c2d7b3491
ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af
Diff: https://reviews.apache.org/r/65422/diff/3/
Changes: https://reviews.apache.org/r/65422/diff/2-3/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
-----------------------------------------------------------
(Updated Feb. 20, 2018, 6:41 p.m.)
Review request for hive and Ashutosh Chauhan.
Repository: hive-git
Description
-------
preview
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3d777f992b
itests/src/test/resources/testconfiguration.properties d4f2e539fb
ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
ql/src/java/org/apache/hadoop/hive/ql/Driver.java d00e639643
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 27b53b8b33
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g e431271d3a
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f9a6386ecf
ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/65422/diff/2/
Changes: https://reviews.apache.org/r/65422/diff/1-2/
Testing
-------
Thanks,
Zoltan Haindrich
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
> > Lines 139 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950494#file1950494line140>
> >
> > Should also compare with parent to check rowcount and other stats are not greater than parent.
ok
> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
> > Lines 310 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950494#file1950494line311>
> >
> > Should also compare with parent to check rowcount and other stats are not greater than parent and normalize to parent stats in that case.
ok
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review197649
-----------------------------------------------------------
On Feb. 26, 2018, 6:19 p.m., Zoltan Haindrich wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2018, 6:19 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> preview
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6
> data/conf/llap/hive-site.xml c4c299c5de
> itests/src/test/resources/testconfiguration.properties 2776fe95f1
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae
> ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93
> ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4
> ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588
> ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88
> ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
> ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java b0cf3bd94e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 783a672c47
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 9a3f81c98f
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1
> ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f9a6386ecf
> ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java fa3f89fda1
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b
> ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1
> ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out cab45c9410
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out 43ca0d9ad7
> ql/src/test/results/clientpositive/llap/join32_lessSize.q.out 5b5be13702
> ql/src/test/results/clientpositive/llap/join46.q.out 4625b9fed8
> ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 4dd7432ac5
> ql/src/test/results/clientpositive/llap/llap_partitioned.q.out ac754674ea
> ql/src/test/results/clientpositive/llap/mapjoin46.q.out f530c1e8e1
> ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68
> ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708
> ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out c7c8993d0e
> ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/semijoin_hint.q.out cdfa73349b
> ql/src/test/results/clientpositive/llap/subquery_in_having.q.out cc0b6a597a
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 67a22c812f
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out 50c18c8a68
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out cec3daa085
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 89eab37d07
> ql/src/test/results/clientpositive/llap/tez_join_tests.q.out 75879fb157
> ql/src/test/results/clientpositive/llap/tez_joins_explain.q.out 3790b37760
> ql/src/test/results/clientpositive/llap/vector_reduce_groupby_duplicate_cols.q.out 07c1e4188c
> ql/src/test/results/clientpositive/perf/tez/query72.q.out 55ddd5a0f8
> ql/src/test/results/clientpositive/perf/tez/query78.q.out 9c2d7b3491
> ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af
>
>
> Diff: https://reviews.apache.org/r/65422/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Zoltan Haindrich
>
>
Re: Review Request 65422: HIVE-17626
Posted by Zoltan Haindrich <ki...@rxd.hu>.
> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
> > Lines 131 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950478#file1950478line131>
> >
> > This is hackish.. as pointed above it needs to happen via explain modifier.
I agree
> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java
> > Lines 21 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950481#file1950481line21>
> >
> > Use java's nonnull annotation.
I've not found any "standard" annotation...I may just as well remove these markers...
https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use/42695253#42695253
> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
> > Lines 54 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950484#file1950484line54>
> >
> > Why is this needed?
this is not needed...but enables the user to set a different set of configuration during re-executions
> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java
> > Lines 36 (patched)
> > <https://reviews.apache.org/r/65422/diff/1/?file=1950498#file1950498line36>
> >
> > A flat map of operators looses hierarichal info in which operators are organized which is tree. So, this match needs to happen via sub-graph matching pattern. See SharedWorkOptimizer::areMergeable() .
I will try to retain this concept for now at least; the idea is that imagine that we have N operator stats gathered; and the current plan consist of M operators; if we have only a cmp(A,B) oracle; that means we will have to do N*M comparisions; which could become really bad if N starts to become large...
I'm thinking of serving the existing operator infos in a map alike fashion - at least it should be visible as one for the outside.
If an operator could self-describe its whole context; then it could be match... for example a matching the textual representation of a RelNode contains all the upstream operations as well; and enables matching.
It looked promising to do it; I wanted to do it with HIVE-18703 - but unfortunately there were some complications...
- Zoltan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review197649
-----------------------------------------------------------
On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2018, 6:13 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> preview
>
>
> Diffs
> -----
>
> cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java ad31287879
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 533f0bcd6f
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9
> ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33
> ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08
> ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
> ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 190771ea6b
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java cbadfa4f07
> ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java dcf8d31eaf
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java d56002d192
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/65422/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Zoltan Haindrich
>
>
Re: Review Request 65422: HIVE-17626
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review197649
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
Lines 131 (patched)
<https://reviews.apache.org/r/65422/#comment277871>
This is hackish.. as pointed above it needs to happen via explain modifier.
ql/src/java/org/apache/hadoop/hive/ql/Context.java
Lines 72 (patched)
<https://reviews.apache.org/r/65422/#comment277873>
Use java.util.Optional instead
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java
Lines 21 (patched)
<https://reviews.apache.org/r/65422/#comment277882>
Use java's nonnull annotation.
ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java
Lines 40 (patched)
<https://reviews.apache.org/r/65422/#comment277884>
Need to check for org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionError and java.lang.OutOfMemoryError only.
ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
Lines 43 (patched)
<https://reviews.apache.org/r/65422/#comment277885>
Need to check for org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionError and java.lang.OutOfMemoryError only.
ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
Lines 54 (patched)
<https://reviews.apache.org/r/65422/#comment277886>
Why is this needed?
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 139 (patched)
<https://reviews.apache.org/r/65422/#comment277893>
Should also compare with parent to check rowcount and other stats are not greater than parent.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 310 (patched)
<https://reviews.apache.org/r/65422/#comment277894>
Should also compare with parent to check rowcount and other stats are not greater than parent and normalize to parent stats in that case.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 2314 (patched)
<https://reviews.apache.org/r/65422/#comment277895>
Should also compare with parent to check rowcount and other stats are not greater than parent and normalize to parent stats in that case.
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java
Lines 36 (patched)
<https://reviews.apache.org/r/65422/#comment277890>
A flat map of operators looses hierarichal info in which operators are organized which is tree. So, this match needs to happen via sub-graph matching pattern. See SharedWorkOptimizer::areMergeable() .
ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java
Lines 75 (patched)
<https://reviews.apache.org/r/65422/#comment277891>
LOG.debug
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java
Lines 64 (patched)
<https://reviews.apache.org/r/65422/#comment277892>
throw MapJoinMemoryExhaustionError
- Ashutosh Chauhan
On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2018, 6:13 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> preview
>
>
> Diffs
> -----
>
> cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java ad31287879
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 533f0bcd6f
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9
> ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33
> ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08
> ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 8dd7cfe58c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java 134fc0ff0b
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java 1eb72ce4d9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java 384bd74686
> ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 190771ea6b
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java cbadfa4f07
> ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java dcf8d31eaf
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java d56002d192
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION
> ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/65422/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Zoltan Haindrich
>
>