You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by Harish Butani <rh...@gmail.com> on 2014/09/02 22:23:12 UTC

Re: Review Request 25179: HIVE-7905: CBO: more cost model changes

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25179/
-----------------------------------------------------------

(Updated Sept. 2, 2014, 8:23 p.m.)


Review request for hive, Gunther Hagleitner, John Pullokkaran, and Mostafa Mokhtar.


Repository: hive-git


Description
-------

CBO: more cost model changes


Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/HiveDefaultRelMetadataProvider.java 2c08772 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdSelectivity.java df70de2 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdUniqueKeys.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6c293c8 

Diff: https://reviews.apache.org/r/25179/diff/


Testing
-------

existing tests.


Thanks,

Harish Butani

Re: Review Request 25179: HIVE-7905: CBO: more cost model changes

Posted by John Pullokkaran <jp...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25179/#review52523
-----------------------------------------------------------

Ship it!


Ship It!

- John Pullokkaran


On Sept. 3, 2014, 11:19 p.m., Harish Butani wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25179/
> -----------------------------------------------------------
> 
> (Updated Sept. 3, 2014, 11:19 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, John Pullokkaran, and Mostafa Mokhtar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> CBO: more cost model changes
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/HiveDefaultRelMetadataProvider.java 2c08772 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdSelectivity.java df70de2 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdUniqueKeys.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8fe8b3c 
> 
> Diff: https://reviews.apache.org/r/25179/diff/
> 
> 
> Testing
> -------
> 
> existing tests.
> 
> 
> Thanks,
> 
> Harish Butani
> 
>

Re: Review Request 25179: HIVE-7905: CBO: more cost model changes

Posted by Harish Butani <rh...@gmail.com>.


> On Sept. 4, 2014, 8:17 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java, line 80
> > <https://reviews.apache.org/r/25179/diff/2/?file=676157#file676157line80>
> >
> >     Can't getRowCount(SemiJoinRel) combine with getRowCount(JoinRelBase)

The reason for this is because RelMdRowCount has a getRowCount method on SemiJoinRel


- Harish


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25179/#review52340
-----------------------------------------------------------


On Sept. 3, 2014, 11:19 p.m., Harish Butani wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25179/
> -----------------------------------------------------------
> 
> (Updated Sept. 3, 2014, 11:19 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, John Pullokkaran, and Mostafa Mokhtar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> CBO: more cost model changes
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/HiveDefaultRelMetadataProvider.java 2c08772 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdSelectivity.java df70de2 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdUniqueKeys.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8fe8b3c 
> 
> Diff: https://reviews.apache.org/r/25179/diff/
> 
> 
> Testing
> -------
> 
> existing tests.
> 
> 
> Thanks,
> 
> Harish Butani
> 
>

Re: Review Request 25179: HIVE-7905: CBO: more cost model changes

Posted by John Pullokkaran <jp...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25179/#review52340
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java
<https://reviews.apache.org/r/25179/#comment91104>

    Can't getRowCount(SemiJoinRel) combine with getRowCount(JoinRelBase)


- John Pullokkaran


On Sept. 3, 2014, 11:19 p.m., Harish Butani wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25179/
> -----------------------------------------------------------
> 
> (Updated Sept. 3, 2014, 11:19 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, John Pullokkaran, and Mostafa Mokhtar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> CBO: more cost model changes
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/HiveDefaultRelMetadataProvider.java 2c08772 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdSelectivity.java df70de2 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdUniqueKeys.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8fe8b3c 
> 
> Diff: https://reviews.apache.org/r/25179/diff/
> 
> 
> Testing
> -------
> 
> existing tests.
> 
> 
> Thanks,
> 
> Harish Butani
> 
>

Re: Review Request 25179: HIVE-7905: CBO: more cost model changes

Posted by Harish Butani <rh...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25179/
-----------------------------------------------------------

(Updated Sept. 3, 2014, 11:19 p.m.)


Review request for hive, Gunther Hagleitner, John Pullokkaran, and Mostafa Mokhtar.


Changes
-------

have added the changes
- to check range against rowCounts. This depends on HIVE-7915, which has not being merged into CBO branch
- apply ndv scaling, where applicable
- log the detailsof PK - FK relationship


Repository: hive-git


Description
-------

CBO: more cost model changes


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/HiveDefaultRelMetadataProvider.java 2c08772 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdSelectivity.java df70de2 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdUniqueKeys.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8fe8b3c 

Diff: https://reviews.apache.org/r/25179/diff/


Testing
-------

existing tests.


Thanks,

Harish Butani

Re: Review Request 25179: HIVE-7905: CBO: more cost model changes

Posted by Harish Butani <rh...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25179/#review52086
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java
<https://reviews.apache.org/r/25179/#comment90821>

    You referring to the join with Date Dimensions in the case of TPCDS, right? 
    AS we discussed will address the special case of the child RelNode trees containing only TScan/Filter/Project.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdUniqueKeys.java
<https://reviews.apache.org/r/25179/#comment90822>

    yes will add logging.
    BTW the check based on range depends on HIVE-7915 going in, that exposes the high and low values in the ColumnStatistics object.


- Harish Butani


On Sept. 2, 2014, 8:23 p.m., Harish Butani wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25179/
> -----------------------------------------------------------
> 
> (Updated Sept. 2, 2014, 8:23 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, John Pullokkaran, and Mostafa Mokhtar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> CBO: more cost model changes
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/HiveDefaultRelMetadataProvider.java 2c08772 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdSelectivity.java df70de2 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdUniqueKeys.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6c293c8 
> 
> Diff: https://reviews.apache.org/r/25179/diff/
> 
> 
> Testing
> -------
> 
> existing tests.
> 
> 
> Thanks,
> 
> Harish Butani
> 
>

Re: Review Request 25179: HIVE-7905: CBO: more cost model changes

Posted by Mostafa Mokhtar <mm...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25179/#review52079
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java
<https://reviews.apache.org/r/25179/#comment90814>

    My concern is that the computation of selectivity doesn't take into account differences in NDV which can result in under estimation.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdUniqueKeys.java
<https://reviews.apache.org/r/25179/#comment90816>

    Currently NDV in the stats is not accurate.
    As you mentioned max(col) - min(col) is more accurate, hence we are less likely to see this code in effect due to inaccuracies in NDV.
    Is it possible to add some logging statements for : Table name, column name, row count, NDV , min and max ?


- Mostafa Mokhtar


On Sept. 2, 2014, 8:23 p.m., Harish Butani wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25179/
> -----------------------------------------------------------
> 
> (Updated Sept. 2, 2014, 8:23 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, John Pullokkaran, and Mostafa Mokhtar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> CBO: more cost model changes
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/HiveDefaultRelMetadataProvider.java 2c08772 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdRowCount.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdSelectivity.java df70de2 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/stats/HiveRelMdUniqueKeys.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6c293c8 
> 
> Diff: https://reviews.apache.org/r/25179/diff/
> 
> 
> Testing
> -------
> 
> existing tests.
> 
> 
> Thanks,
> 
> Harish Butani
> 
>