You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Jason Dere <jd...@hortonworks.com> on 2015/06/17 20:38:41 UTC
Review Request 35576: HIVE-11028 Tez: table self join and join with
another table fails with IndexOutOfBoundsException
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35576/
-----------------------------------------------------------
Review request for hive, Gunther Hagleitner and John Pullokkaran.
Bugs: HIVE-11028
https://issues.apache.org/jira/browse/HIVE-11028
Repository: hive-git
Description
-------
Change TezCompiler to only run short-cutting of expressions rather than full constant folding.
Diffs
-----
itests/src/test/resources/testconfiguration.properties b9f39fb
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java 0027960
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcCtx.java 6bb2a09
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java 4a4814d
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 56707af
ql/src/test/queries/clientpositive/tez_self_join.q PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_self_join.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/35576/diff/
Testing
-------
qfile test added
Thanks,
Jason Dere
Re: Review Request 35576: HIVE-11028 Tez: table self join and join
with another table fails with IndexOutOfBoundsException
Posted by Jason Dere <jd...@hortonworks.com>.
> On June 17, 2015, 7:39 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java, line 46
> > <https://reviews.apache.org/r/35576/diff/1/?file=986551#file986551line46>
> >
> > As we discussed, Couldn't we move foldExpr from ConstantPropagate to ExprNodeDescUtils.
> >
> > Then DPP could use ExprNodeDescUtils directly without depending on ConstantProp.
> >
> > IMO, its a better seperation.
It looks like ConstantPropagate does additional things that we actually want to be done, such as being able to delete filter operators if the filter expression consists of a single constant True expression. We end up with those kinds of situationss from the dynamic partition pruning. I believe it would be easier and a more generic reusable solution to use ConstantPropagate here, than to have to re-implement similar logic in DynamicPartitionPrunerProc.
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35576/#review88269
-----------------------------------------------------------
On June 17, 2015, 6:38 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35576/
> -----------------------------------------------------------
>
> (Updated June 17, 2015, 6:38 p.m.)
>
>
> Review request for hive, Gunther Hagleitner and John Pullokkaran.
>
>
> Bugs: HIVE-11028
> https://issues.apache.org/jira/browse/HIVE-11028
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Change TezCompiler to only run short-cutting of expressions rather than full constant folding.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties b9f39fb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java 0027960
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcCtx.java 6bb2a09
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java 4a4814d
> ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 56707af
> ql/src/test/queries/clientpositive/tez_self_join.q PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_self_join.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/35576/diff/
>
>
> Testing
> -------
>
> qfile test added
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 35576: HIVE-11028 Tez: table self join and join
with another table fails with IndexOutOfBoundsException
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35576/#review88269
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java (line 46)
<https://reviews.apache.org/r/35576/#comment140696>
As we discussed, Couldn't we move foldExpr from ConstantPropagate to ExprNodeDescUtils.
Then DPP could use ExprNodeDescUtils directly without depending on ConstantProp.
IMO, its a better seperation.
- John Pullokkaran
On June 17, 2015, 6:38 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35576/
> -----------------------------------------------------------
>
> (Updated June 17, 2015, 6:38 p.m.)
>
>
> Review request for hive, Gunther Hagleitner and John Pullokkaran.
>
>
> Bugs: HIVE-11028
> https://issues.apache.org/jira/browse/HIVE-11028
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Change TezCompiler to only run short-cutting of expressions rather than full constant folding.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties b9f39fb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java 0027960
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcCtx.java 6bb2a09
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java 4a4814d
> ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 56707af
> ql/src/test/queries/clientpositive/tez_self_join.q PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_self_join.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/35576/diff/
>
>
> Testing
> -------
>
> qfile test added
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 35576: HIVE-11028 Tez: table self join and join
with another table fails with IndexOutOfBoundsException
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35576/#review88433
-----------------------------------------------------------
Ship it!
Ship It!
- John Pullokkaran
On June 17, 2015, 6:38 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35576/
> -----------------------------------------------------------
>
> (Updated June 17, 2015, 6:38 p.m.)
>
>
> Review request for hive, Gunther Hagleitner and John Pullokkaran.
>
>
> Bugs: HIVE-11028
> https://issues.apache.org/jira/browse/HIVE-11028
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Change TezCompiler to only run short-cutting of expressions rather than full constant folding.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties b9f39fb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java 0027960
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcCtx.java 6bb2a09
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java 4a4814d
> ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 56707af
> ql/src/test/queries/clientpositive/tez_self_join.q PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_self_join.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/35576/diff/
>
>
> Testing
> -------
>
> qfile test added
>
>
> Thanks,
>
> Jason Dere
>
>