You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by pengcheng xiong <px...@hortonworks.com> on 2014/12/01 19:57:17 UTC
Re: Review Request 27713: CBO: enable groupBy index
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27713/
-----------------------------------------------------------
(Updated Dec. 1, 2014, 6:57 p.m.)
Review request for hive and John Pullokkaran.
Repository: hive-git
Description
-------
Right now, even when groupby index is build, CBO is not able to use it. In this patch, we are trying to make it use groupby index that we build. The basic problem is that
for SEL1-SEL2-GRY-...-SEL3,
the previous version only modify SEL2, which immediately precedes GRY.
Now, with CBO, we have lots of SELs, e.g., SEL1.
So, the solution is to modify all of them.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java 9ffa708
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java 02216de
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 0f06ec9
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java 74614f3
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java d699308
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_1.q PRE-CREATION
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_2.q PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out fdc1dc6
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/27713/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 27713: CBO: enable groupBy index
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27713/#review63710
-----------------------------------------------------------
Ship it!
Ship It!
- John Pullokkaran
On Dec. 2, 2014, 11:18 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27713/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2014, 11:18 p.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Right now, even when groupby index is build, CBO is not able to use it. In this patch, we are trying to make it use groupby index that we build. The basic problem is that
> for SEL1-SEL2-GRY-...-SEL3,
> the previous version only modify SEL2, which immediately precedes GRY.
> Now, with CBO, we have lots of SELs, e.g., SEL1.
> So, the solution is to modify all of them.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties fc1f345
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java 9ffa708
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java 02216de
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 0f06ec9
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java 74614f3
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java d699308
> ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_1.q PRE-CREATION
> ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_2.q PRE-CREATION
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out fdc1dc6
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27713/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 27713: CBO: enable groupBy index
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27713/
-----------------------------------------------------------
(Updated Dec. 5, 2014, 10:52 p.m.)
Review request for hive and John Pullokkaran.
Changes
-------
remove space, update golden files
Repository: hive-git
Description
-------
Right now, even when groupby index is build, CBO is not able to use it. In this patch, we are trying to make it use groupby index that we build. The basic problem is that
for SEL1-SEL2-GRY-...-SEL3,
the previous version only modify SEL2, which immediately precedes GRY.
Now, with CBO, we have lots of SELs, e.g., SEL1.
So, the solution is to modify all of them.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties fc1f345
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java 9ffa708
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java 02216de
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 0f06ec9
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java 74614f3
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java d699308
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_1.q PRE-CREATION
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_2.q PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out fdc1dc6
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/27713/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 27713: CBO: enable groupBy index
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27713/
-----------------------------------------------------------
(Updated Dec. 3, 2014, 7:40 p.m.)
Review request for hive and John Pullokkaran.
Repository: hive-git
Description
-------
Right now, even when groupby index is build, CBO is not able to use it. In this patch, we are trying to make it use groupby index that we build. The basic problem is that
for SEL1-SEL2-GRY-...-SEL3,
the previous version only modify SEL2, which immediately precedes GRY.
Now, with CBO, we have lots of SELs, e.g., SEL1.
So, the solution is to modify all of them.
Diffs
-----
itests/src/test/resources/testconfiguration.properties fc1f345
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java 9ffa708
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java 02216de
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 0f06ec9
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java 74614f3
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java d699308
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_1.q PRE-CREATION
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_2.q PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out fdc1dc6
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/27713/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 27713: CBO: enable groupBy index
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27713/#review63615
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java
<https://reviews.apache.org/r/27713/#comment105900>
Remove empty spaces
- John Pullokkaran
On Dec. 2, 2014, 11:18 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27713/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2014, 11:18 p.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Right now, even when groupby index is build, CBO is not able to use it. In this patch, we are trying to make it use groupby index that we build. The basic problem is that
> for SEL1-SEL2-GRY-...-SEL3,
> the previous version only modify SEL2, which immediately precedes GRY.
> Now, with CBO, we have lots of SELs, e.g., SEL1.
> So, the solution is to modify all of them.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties fc1f345
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java 9ffa708
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java 02216de
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 0f06ec9
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java 74614f3
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java d699308
> ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_1.q PRE-CREATION
> ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_2.q PRE-CREATION
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out fdc1dc6
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27713/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 27713: CBO: enable groupBy index
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27713/
-----------------------------------------------------------
(Updated Dec. 3, 2014, 7:40 p.m.)
Review request for hive and John Pullokkaran.
Changes
-------
remove white spaces
Repository: hive-git
Description
-------
Right now, even when groupby index is build, CBO is not able to use it. In this patch, we are trying to make it use groupby index that we build. The basic problem is that
for SEL1-SEL2-GRY-...-SEL3,
the previous version only modify SEL2, which immediately precedes GRY.
Now, with CBO, we have lots of SELs, e.g., SEL1.
So, the solution is to modify all of them.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties fc1f345
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java 9ffa708
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java 02216de
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 0f06ec9
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java 74614f3
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java d699308
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_1.q PRE-CREATION
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_2.q PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out fdc1dc6
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/27713/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 27713: CBO: enable groupBy index
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27713/#review63711
-----------------------------------------------------------
Ship it!
- John Pullokkaran
On Dec. 2, 2014, 11:18 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27713/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2014, 11:18 p.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Right now, even when groupby index is build, CBO is not able to use it. In this patch, we are trying to make it use groupby index that we build. The basic problem is that
> for SEL1-SEL2-GRY-...-SEL3,
> the previous version only modify SEL2, which immediately precedes GRY.
> Now, with CBO, we have lots of SELs, e.g., SEL1.
> So, the solution is to modify all of them.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties fc1f345
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java 9ffa708
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java 02216de
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 0f06ec9
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java 74614f3
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java d699308
> ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_1.q PRE-CREATION
> ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_2.q PRE-CREATION
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out fdc1dc6
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27713/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 27713: CBO: enable groupBy index
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27713/
-----------------------------------------------------------
(Updated Dec. 2, 2014, 11:18 p.m.)
Review request for hive and John Pullokkaran.
Changes
-------
put q test in minimrclidriver
Repository: hive-git
Description
-------
Right now, even when groupby index is build, CBO is not able to use it. In this patch, we are trying to make it use groupby index that we build. The basic problem is that
for SEL1-SEL2-GRY-...-SEL3,
the previous version only modify SEL2, which immediately precedes GRY.
Now, with CBO, we have lots of SELs, e.g., SEL1.
So, the solution is to modify all of them.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties fc1f345
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java 9ffa708
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java 02216de
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 0f06ec9
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java 74614f3
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java d699308
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_1.q PRE-CREATION
ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_2.q PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out fdc1dc6
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/27713/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 27713: CBO: enable groupBy index
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27713/#review63459
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java
<https://reviews.apache.org/r/27713/#comment105714>
I don't think you can allow function wraping index key. Since we don't know if UDF is going to mutate the values (Non Null -> Null, Null -> Non Null).
Example:
select a, count(b) from (select a, (case a is null then 1 else a) as b from r1)r2 group by a;
- John Pullokkaran
On Dec. 1, 2014, 6:57 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27713/
> -----------------------------------------------------------
>
> (Updated Dec. 1, 2014, 6:57 p.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Right now, even when groupby index is build, CBO is not able to use it. In this patch, we are trying to make it use groupby index that we build. The basic problem is that
> for SEL1-SEL2-GRY-...-SEL3,
> the previous version only modify SEL2, which immediately precedes GRY.
> Now, with CBO, we have lots of SELs, e.g., SEL1.
> So, the solution is to modify all of them.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java 9ffa708
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java 02216de
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 0f06ec9
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java 74614f3
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java d699308
> ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_1.q PRE-CREATION
> ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx_cbo_2.q PRE-CREATION
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out fdc1dc6
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27713/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>