You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Ashutosh Chauhan <ha...@apache.org> on 2014/06/19 04:35:18 UTC

Review Request 22770: Allow partial partition spec in analyze command

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/
-----------------------------------------------------------

Review request for hive.


Bugs: HIVE-7255
    https://issues.apache.org/jira/browse/HIVE-7255


Repository: hive-git


Description
-------

So that stats collection can happen for multiple partitions through one statement.


Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877 
  ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4 
  ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927 
  ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION 
  ql/src/test/queries/clientpositive/columnstats_partlvl_incorrect_num_keys.q PRE-CREATION 
  ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb 
  ql/src/test/results/clientpositive/columnstats_partlvl.q.out a4c4677 
  ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/columnstats_partlvl_incorrect_num_keys.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/22770/diff/


Testing
-------

Moved -ve tests to +ve.


Thanks,

Ashutosh Chauhan


Re: Review Request 22770: Allow partial partition spec in analyze command

Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/
-----------------------------------------------------------

(Updated June 30, 2014, 4:36 a.m.)


Review request for hive.


Changes
-------

-- Incorporated Szehon's feedback.
-- Added new tests as per Gunther's suggestions.
-- Fixed couple of bugs.


Bugs: HIVE-7255
    https://issues.apache.org/jira/browse/HIVE-7255


Repository: hive-git


Description
-------

So that stats collection can happen for multiple partitions through one statement.


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877 
  ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4 
  ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927 
  ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION 
  ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb 
  ql/src/test/results/clientpositive/columnstats_partlvl.q.out 6128770 
  ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/22770/diff/


Testing
-------

Moved -ve tests to +ve.


Thanks,

Ashutosh Chauhan


Re: Review Request 22770: Allow partial partition spec in analyze command

Posted by Ashutosh Chauhan <ha...@apache.org>.

> On June 19, 2014, 3:51 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java, line 73
> > <https://reviews.apache.org/r/22770/diff/1/?file=612845#file612845line73>
> >
> >     Looks like no longer used.

will get rid of it.


> On June 19, 2014, 3:51 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java, line 422
> > <https://reviews.apache.org/r/22770/diff/1/?file=612845#file612845line422>
> >
> >     We can remove 'totalRows' variable as well.

yup. will remove


> On June 19, 2014, 3:51 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java, line 425
> > <https://reviews.apache.org/r/22770/diff/1/?file=612845#file612845line425>
> >
> >     Do we need to cleanup the fetch operator at some point?

Yeah, will do.


> On June 19, 2014, 3:51 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java, line 65
> > <https://reviews.apache.org/r/22770/diff/1/?file=612849#file612849line65>
> >
> >     Same question, cant we still display partial partition spec?

not sure, if its any useful in explain, but will see if I can add it back.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/#review46179
-----------------------------------------------------------


On June 19, 2014, 2:35 a.m., Ashutosh Chauhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22770/
> -----------------------------------------------------------
> 
> (Updated June 19, 2014, 2:35 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-7255
>     https://issues.apache.org/jira/browse/HIVE-7255
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> So that stats collection can happen for multiple partitions through one statement.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877 
>   ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4 
>   ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927 
>   ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/columnstats_partlvl_incorrect_num_keys.q PRE-CREATION 
>   ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb 
>   ql/src/test/results/clientpositive/columnstats_partlvl.q.out a4c4677 
>   ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/columnstats_partlvl_incorrect_num_keys.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/22770/diff/
> 
> 
> Testing
> -------
> 
> Moved -ve tests to +ve.
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>


Re: Review Request 22770: Allow partial partition spec in analyze command

Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/#review46179
-----------------------------------------------------------


Hi Ashutosh, it looks good , had some minor questions on this patch.


ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
<https://reviews.apache.org/r/22770/#comment81446>

    Looks like no longer used.



ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
<https://reviews.apache.org/r/22770/#comment81453>

    We can remove 'totalRows' variable as well.



ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
<https://reviews.apache.org/r/22770/#comment81445>

    Do we need to cleanup the fetch operator at some point?



ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
<https://reviews.apache.org/r/22770/#comment81451>

    Sorry if I misunderstand this, but cant we still set a partial partition spec, for explain?



ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java
<https://reviews.apache.org/r/22770/#comment81452>

    Same question, cant we still display partial partition spec?


- Szehon Ho


On June 19, 2014, 2:35 a.m., Ashutosh Chauhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22770/
> -----------------------------------------------------------
> 
> (Updated June 19, 2014, 2:35 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-7255
>     https://issues.apache.org/jira/browse/HIVE-7255
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> So that stats collection can happen for multiple partitions through one statement.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877 
>   ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4 
>   ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927 
>   ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/columnstats_partlvl_incorrect_num_keys.q PRE-CREATION 
>   ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb 
>   ql/src/test/results/clientpositive/columnstats_partlvl.q.out a4c4677 
>   ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/columnstats_partlvl_incorrect_num_keys.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/22770/diff/
> 
> 
> Testing
> -------
> 
> Moved -ve tests to +ve.
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>


Re: Review Request 22770: Allow partial partition spec in analyze command

Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/#review46180
-----------------------------------------------------------



ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q
<https://reviews.apache.org/r/22770/#comment81448>

    I don't think these tests have enough coverage. For instance, as far as I can tell nothing validates that the stats computed are actually correct and that all col stats are populated.
    
    Some ideas are:
    
    - test with different data in different partitions
    - validate correctness of stats
    - run analyze without specifying any partition key (i suspect that would be the common case)
    - test with schema evolution (eg: what happens when we add a column/drop/rearrange columns and run a full table analyze
    - test with partially populated stats for a table (i.e.: incremental update stats scenario)
    
    Are these covered elsewhere?


- Gunther Hagleitner


On June 19, 2014, 2:35 a.m., Ashutosh Chauhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22770/
> -----------------------------------------------------------
> 
> (Updated June 19, 2014, 2:35 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-7255
>     https://issues.apache.org/jira/browse/HIVE-7255
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> So that stats collection can happen for multiple partitions through one statement.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877 
>   ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4 
>   ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927 
>   ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/columnstats_partlvl_incorrect_num_keys.q PRE-CREATION 
>   ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb 
>   ql/src/test/results/clientpositive/columnstats_partlvl.q.out a4c4677 
>   ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/columnstats_partlvl_incorrect_num_keys.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/22770/diff/
> 
> 
> Testing
> -------
> 
> Moved -ve tests to +ve.
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>