You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Ashutosh Chauhan <ha...@apache.org> on 2014/06/19 04:35:18 UTC
Review Request 22770: Allow partial partition spec in analyze command
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-7255
https://issues.apache.org/jira/browse/HIVE-7255
Repository: hive-git
Description
-------
So that stats collection can happen for multiple partitions through one statement.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73
ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520
ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd
ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c
ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877
ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4
ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927
ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION
ql/src/test/queries/clientpositive/columnstats_partlvl_incorrect_num_keys.q PRE-CREATION
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb
ql/src/test/results/clientpositive/columnstats_partlvl.q.out a4c4677
ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION
ql/src/test/results/clientpositive/columnstats_partlvl_incorrect_num_keys.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/22770/diff/
Testing
-------
Moved -ve tests to +ve.
Thanks,
Ashutosh Chauhan
Re: Review Request 22770: Allow partial partition spec in analyze command
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/
-----------------------------------------------------------
(Updated June 30, 2014, 4:36 a.m.)
Review request for hive.
Changes
-------
-- Incorporated Szehon's feedback.
-- Added new tests as per Gunther's suggestions.
-- Fixed couple of bugs.
Bugs: HIVE-7255
https://issues.apache.org/jira/browse/HIVE-7255
Repository: hive-git
Description
-------
So that stats collection can happen for multiple partitions through one statement.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73
ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520
ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd
ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c
ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877
ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4
ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927
ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb
ql/src/test/results/clientpositive/columnstats_partlvl.q.out 6128770
ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/22770/diff/
Testing
-------
Moved -ve tests to +ve.
Thanks,
Ashutosh Chauhan
Re: Review Request 22770: Allow partial partition spec in analyze command
Posted by Ashutosh Chauhan <ha...@apache.org>.
> On June 19, 2014, 3:51 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java, line 73
> > <https://reviews.apache.org/r/22770/diff/1/?file=612845#file612845line73>
> >
> > Looks like no longer used.
will get rid of it.
> On June 19, 2014, 3:51 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java, line 422
> > <https://reviews.apache.org/r/22770/diff/1/?file=612845#file612845line422>
> >
> > We can remove 'totalRows' variable as well.
yup. will remove
> On June 19, 2014, 3:51 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java, line 425
> > <https://reviews.apache.org/r/22770/diff/1/?file=612845#file612845line425>
> >
> > Do we need to cleanup the fetch operator at some point?
Yeah, will do.
> On June 19, 2014, 3:51 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java, line 65
> > <https://reviews.apache.org/r/22770/diff/1/?file=612849#file612849line65>
> >
> > Same question, cant we still display partial partition spec?
not sure, if its any useful in explain, but will see if I can add it back.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/#review46179
-----------------------------------------------------------
On June 19, 2014, 2:35 a.m., Ashutosh Chauhan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22770/
> -----------------------------------------------------------
>
> (Updated June 19, 2014, 2:35 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-7255
> https://issues.apache.org/jira/browse/HIVE-7255
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> So that stats collection can happen for multiple partitions through one statement.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73
> ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520
> ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd
> ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c
> ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877
> ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4
> ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927
> ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION
> ql/src/test/queries/clientpositive/columnstats_partlvl_incorrect_num_keys.q PRE-CREATION
> ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb
> ql/src/test/results/clientpositive/columnstats_partlvl.q.out a4c4677
> ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION
> ql/src/test/results/clientpositive/columnstats_partlvl_incorrect_num_keys.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22770/diff/
>
>
> Testing
> -------
>
> Moved -ve tests to +ve.
>
>
> Thanks,
>
> Ashutosh Chauhan
>
>
Re: Review Request 22770: Allow partial partition spec in analyze command
Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/#review46179
-----------------------------------------------------------
Hi Ashutosh, it looks good , had some minor questions on this patch.
ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
<https://reviews.apache.org/r/22770/#comment81446>
Looks like no longer used.
ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
<https://reviews.apache.org/r/22770/#comment81453>
We can remove 'totalRows' variable as well.
ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
<https://reviews.apache.org/r/22770/#comment81445>
Do we need to cleanup the fetch operator at some point?
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
<https://reviews.apache.org/r/22770/#comment81451>
Sorry if I misunderstand this, but cant we still set a partial partition spec, for explain?
ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java
<https://reviews.apache.org/r/22770/#comment81452>
Same question, cant we still display partial partition spec?
- Szehon Ho
On June 19, 2014, 2:35 a.m., Ashutosh Chauhan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22770/
> -----------------------------------------------------------
>
> (Updated June 19, 2014, 2:35 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-7255
> https://issues.apache.org/jira/browse/HIVE-7255
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> So that stats collection can happen for multiple partitions through one statement.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73
> ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520
> ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd
> ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c
> ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877
> ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4
> ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927
> ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION
> ql/src/test/queries/clientpositive/columnstats_partlvl_incorrect_num_keys.q PRE-CREATION
> ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb
> ql/src/test/results/clientpositive/columnstats_partlvl.q.out a4c4677
> ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION
> ql/src/test/results/clientpositive/columnstats_partlvl_incorrect_num_keys.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22770/diff/
>
>
> Testing
> -------
>
> Moved -ve tests to +ve.
>
>
> Thanks,
>
> Ashutosh Chauhan
>
>
Re: Review Request 22770: Allow partial partition spec in analyze command
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22770/#review46180
-----------------------------------------------------------
ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q
<https://reviews.apache.org/r/22770/#comment81448>
I don't think these tests have enough coverage. For instance, as far as I can tell nothing validates that the stats computed are actually correct and that all col stats are populated.
Some ideas are:
- test with different data in different partitions
- validate correctness of stats
- run analyze without specifying any partition key (i suspect that would be the common case)
- test with schema evolution (eg: what happens when we add a column/drop/rearrange columns and run a full table analyze
- test with partially populated stats for a table (i.e.: incremental update stats scenario)
Are these covered elsewhere?
- Gunther Hagleitner
On June 19, 2014, 2:35 a.m., Ashutosh Chauhan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22770/
> -----------------------------------------------------------
>
> (Updated June 19, 2014, 2:35 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-7255
> https://issues.apache.org/jira/browse/HIVE-7255
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> So that stats collection can happen for multiple partitions through one statement.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8ae1c73
> ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 47a6871
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 1270520
> ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java a4ba4bd
> ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java b75f78c
> ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 86e6877
> ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q b4887c4
> ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q 2f8e927
> ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q PRE-CREATION
> ql/src/test/queries/clientpositive/columnstats_partlvl_incorrect_num_keys.q PRE-CREATION
> ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out d48d8cb
> ql/src/test/results/clientpositive/columnstats_partlvl.q.out a4c4677
> ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out PRE-CREATION
> ql/src/test/results/clientpositive/columnstats_partlvl_incorrect_num_keys.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22770/diff/
>
>
> Testing
> -------
>
> Moved -ve tests to +ve.
>
>
> Thanks,
>
> Ashutosh Chauhan
>
>