You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Selina Zhang <se...@yahoo-inc.com> on 2014/03/18 23:09:37 UTC
Review Request 19373: Limit table partitions involved in a table scan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19373/
-----------------------------------------------------------
Review request for hive and Gunther Hagleitner.
Bugs: HIVE-6492
https://issues.apache.org/jira/browse/HIVE-6492
Repository: hive-git
Description
-------
Introduce a new configure parameter to limit the table partitions involved in a table scan. It applies to "select *" query and any queries need issue MR jobs.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ecd4c5d
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ecce21e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java 7f2bb60
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 73603ab
ql/src/test/queries/clientnegative/limit_partition.q PRE-CREATION
ql/src/test/queries/clientnegative/limit_partition_stats.q PRE-CREATION
ql/src/test/queries/clientpositive/limit_partition_metadataonly.q PRE-CREATION
ql/src/test/results/clientnegative/limit_partition.q.out PRE-CREATION
ql/src/test/results/clientnegative/limit_partition_stats.q.out PRE-CREATION
ql/src/test/results/clientpositive/limit_partition_metadataonly.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/19373/diff/
Testing
-------
3 tests are added
Thanks,
Selina Zhang
Re: Review Request 19373: Limit table partitions involved in a table scan
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19373/#review38493
-----------------------------------------------------------
Few comments. Looks good otherwise.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java
<https://reviews.apache.org/r/19373/#comment70697>
We can do: tso.getTableDesc().setIsMetadataOnly();
- Ashutosh Chauhan
On March 18, 2014, 10:09 p.m., Selina Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19373/
> -----------------------------------------------------------
>
> (Updated March 18, 2014, 10:09 p.m.)
>
>
> Review request for hive and Gunther Hagleitner.
>
>
> Bugs: HIVE-6492
> https://issues.apache.org/jira/browse/HIVE-6492
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Introduce a new configure parameter to limit the table partitions involved in a table scan. It applies to "select *" query and any queries need issue MR jobs.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ecd4c5d
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ecce21e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java 7f2bb60
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 73603ab
> ql/src/test/queries/clientnegative/limit_partition.q PRE-CREATION
> ql/src/test/queries/clientnegative/limit_partition_stats.q PRE-CREATION
> ql/src/test/queries/clientpositive/limit_partition_metadataonly.q PRE-CREATION
> ql/src/test/results/clientnegative/limit_partition.q.out PRE-CREATION
> ql/src/test/results/clientnegative/limit_partition_stats.q.out PRE-CREATION
> ql/src/test/results/clientpositive/limit_partition_metadataonly.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/19373/diff/
>
>
> Testing
> -------
>
> 3 tests are added
>
>
> Thanks,
>
> Selina Zhang
>
>
Re: Review Request 19373: Limit table partitions involved in a table scan
Posted by Selina Zhang <se...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19373/
-----------------------------------------------------------
(Updated March 25, 2014, 9:03 p.m.)
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-6492
https://issues.apache.org/jira/browse/HIVE-6492
Repository: hive-git
Description
-------
Introduce a new configure parameter to limit the table partitions involved in a table scan. It applies to "select *" query and any queries need issue MR jobs.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java affcbb4
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ecd4c5d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java 7f2bb60
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e3291be
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 10bae4d
ql/src/test/queries/clientnegative/limit_partition.q PRE-CREATION
ql/src/test/queries/clientnegative/limit_partition_stats.q PRE-CREATION
ql/src/test/queries/clientpositive/limit_partition_metadataonly.q PRE-CREATION
ql/src/test/results/clientnegative/limit_partition.q.out PRE-CREATION
ql/src/test/results/clientnegative/limit_partition_stats.q.out PRE-CREATION
ql/src/test/results/clientpositive/limit_partition_metadataonly.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/19373/diff/
Testing
-------
3 tests are added
Thanks,
Selina Zhang
Re: Review Request 19373: Limit table partitions involved in a table scan
Posted by Selina Zhang <se...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19373/
-----------------------------------------------------------
(Updated March 25, 2014, 9:02 p.m.)
Review request for hive and Gunther Hagleitner.
Changes
-------
Added changes based on the review by Ashutosh.
Bugs: HIVE-6492
https://issues.apache.org/jira/browse/HIVE-6492
Repository: hive-git
Description
-------
Introduce a new configure parameter to limit the table partitions involved in a table scan. It applies to "select *" query and any queries need issue MR jobs.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java affcbb4
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ecd4c5d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java 7f2bb60
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e3291be
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 10bae4d
ql/src/test/queries/clientnegative/limit_partition.q PRE-CREATION
ql/src/test/queries/clientnegative/limit_partition_stats.q PRE-CREATION
ql/src/test/queries/clientpositive/limit_partition_metadataonly.q PRE-CREATION
ql/src/test/results/clientnegative/limit_partition.q.out PRE-CREATION
ql/src/test/results/clientnegative/limit_partition_stats.q.out PRE-CREATION
ql/src/test/results/clientpositive/limit_partition_metadataonly.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/19373/diff/
Testing
-------
3 tests are added
Thanks,
Selina Zhang
Re: Review Request 19373: Limit table partitions involved in a table scan
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19373/#review38485
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
<https://reviews.apache.org/r/19373/#comment70685>
Instead of variable in TSOperator, we should have this in TSDesc. Thats because all compile time info & description belongs in *Desc classes. Operators are meant for pure execution logic.
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
<https://reviews.apache.org/r/19373/#comment70687>
Better to have this as scanLimit > -1
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
<https://reviews.apache.org/r/19373/#comment70692>
I thought you want this limit to be applied on cumulative partitions count or limit is meant for per TSOperator?
- Ashutosh Chauhan
On March 18, 2014, 10:09 p.m., Selina Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19373/
> -----------------------------------------------------------
>
> (Updated March 18, 2014, 10:09 p.m.)
>
>
> Review request for hive and Gunther Hagleitner.
>
>
> Bugs: HIVE-6492
> https://issues.apache.org/jira/browse/HIVE-6492
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Introduce a new configure parameter to limit the table partitions involved in a table scan. It applies to "select *" query and any queries need issue MR jobs.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ecd4c5d
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ecce21e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java 7f2bb60
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 73603ab
> ql/src/test/queries/clientnegative/limit_partition.q PRE-CREATION
> ql/src/test/queries/clientnegative/limit_partition_stats.q PRE-CREATION
> ql/src/test/queries/clientpositive/limit_partition_metadataonly.q PRE-CREATION
> ql/src/test/results/clientnegative/limit_partition.q.out PRE-CREATION
> ql/src/test/results/clientnegative/limit_partition_stats.q.out PRE-CREATION
> ql/src/test/results/clientpositive/limit_partition_metadataonly.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/19373/diff/
>
>
> Testing
> -------
>
> 3 tests are added
>
>
> Thanks,
>
> Selina Zhang
>
>