You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Russell Melick <rm...@cs.hmc.edu> on 2011/04/07 07:27:22 UTC
Review Request: HIVE-1644 Use filter pushdown for automatically accessing
indexes
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/
-----------------------------------------------------------
Review request for hive.
Summary
-------
Review request for HIVE-1644.12.patch
This addresses bug HIVE-1644.
https://issues.apache.org/jira/browse/HIVE-1644
Diffs
-----
ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
ql/src/test/queries/clientpositive/index_opt_where.q PRE-CREATION
ql/src/test/queries/clientpositive/index_opt_where_partitioned.q PRE-CREATION
ql/src/test/queries/clientpositive/index_opt_where_simple.q PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589
conf/hive-default.xml c42197f
Diff: https://reviews.apache.org/r/558/diff
Testing
-------
Thanks,
Russell
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by John Sichi <js...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review399
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/558/#comment748>
For consistency with my review in HIVE-1694, I suggest hive.optimize.index.filter as the name for this configuration parameter.
(In HIVE-1694 I suggested hive.optimize.index.groupby, and we want it to be possible to enable/disable them independently)
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/558/#comment749>
In line with the previous comment, suggest hive.optimize.index.filter.compact.minSize/maxSize.
Namit's suggestion for minSize was 5G.
I think the default for maxSize should be infinity (I can't think of a case where we want it in effect by default).
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
<https://reviews.apache.org/r/558/#comment750>
HIVE-1803 is changing this to hive.index.blockfilter.file. Assuming that gets committed first, we should use that, since it's generic rather than tied to the index type.
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java
<https://reviews.apache.org/r/558/#comment751>
What are the units here? Also, don't use colon after parameter name.
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment752>
The non-functional changes in this file are gonna conflict with HIVE-1803, so get rid of them.
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment755>
Use HiveUtils.unparseIdentifier for quoting table names in generated SQL.
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment756>
Isn't it incorrect to set properties on the original table scan here since this is only tentative?
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment757>
Likewise, modifying inputs is incorrect before we have a definite plan.
Some more work on the new HiveIndexHandler interface method is required for resolving this plus the residuals.
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment753>
If searchConditions.size() == 0, it means we didn't find anything which could be handled by the index. In that case, we should bail out immediately and not try to do anything more with this index.
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment759>
We collect the residual here, but we don't do anything with it. Don't we need to pass it back so that Hive can decide what to leave in the Filter operator?
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
<https://reviews.apache.org/r/558/#comment760>
The list actually contains index objects, not index table names. Also typo: "is exists"
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java
<https://reviews.apache.org/r/558/#comment761>
Only cast once.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment764>
Indentation is wrong here.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment763>
In my review for HIVE-1694, I noted that we should not be swallowing exceptions. I think some of this code was copied from there. If we can't access the metastore during optimization, it should be treated as a fatal error.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment765>
The plan still looks wrong (there are two Stage-0's, one for the index scan, one for the final fetch), so the relabeling is still not quite working correctly.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment766>
no space after !
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment767>
Suggested rename for method: arePartitionsCoveredByIndex
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment768>
This checks that the metadata matches. But it does not actually check that the index partitions exist.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java
<https://reviews.apache.org/r/558/#comment769>
Is that true? Why couldn't index be used to optimize a map-only task?
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java
<https://reviews.apache.org/r/558/#comment770>
As noted above, we DO want to fail here.
ql/src/test/queries/clientpositive/index_opt_where.q
<https://reviews.apache.org/r/558/#comment772>
Could you add more tests for cases where automatic indexing should decide not to kick in:
* index can't be used because of min/max size config
* index can't be used because predicate isn't supported
* index can't be used columns aren't covered
Also:
* case where multiple indexes apply and we pick one (currently arbitrarily, but make sure it's at least deterministic so that test doesn't become flaky)
ql/src/test/queries/clientpositive/index_opt_where_partitioned.q
<https://reviews.apache.org/r/558/#comment771>
Could you add a test which shows the index NOT being used in the case where required partitions haven't been built yet?
ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out
<https://reviews.apache.org/r/558/#comment773>
Here's the duplicate Stage-0 I referred to in the code.
- John
On 2011-04-07 05:27:22, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-07 05:27:22)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_opt_where.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_opt_where_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_opt_where_simple.q PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589
> conf/hive-default.xml c42197f
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by John Sichi <js...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review495
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1027>
Oh, and reading your original comment more carefully: yeah, they are two separate entities (one for the table partition, and one for the index partition), so even if the equals method were tied to metastore object identity, it still wouldn't work.
The getSpec() method on the Partition class is what gives you the actual key/value pairs for the partition, suitable for comparison.
- John
On 2011-04-16 06:04:26, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-16 06:04:26)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589
> conf/hive-default.xml c42197f
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14015d0
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
> ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review492
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
<https://reviews.apache.org/r/558/#comment998>
Still need to change hive.index.compact.file to hive.index.blockfilter.file , but hopefully bitmap gets committed soon.
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment997>
I'm not sure the way I'm doing it currently will work with partitions. I don't take them into account when generating the index query.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1000>
see later comment about why this abort needs to be skipped for anything to run.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment999>
This doesn't seem to work (it always returns false here). This checks whether the partitions equal each other, which I don't think can happen since they're on different tables. What information in a partition do I need to be checking?
ql/src/test/queries/clientpositive/index_auto_multiple.q
<https://reviews.apache.org/r/558/#comment995>
Is there a multiple column table? Or, what's the best way to create a multi-column table and populate it with data? I can't figure out a good way to query the value column, so the src table seems less than ideal.
ql/src/test/queries/clientpositive/index_auto_unused.q
<https://reviews.apache.org/r/558/#comment996>
How do unbuilt partitions work? I didn't see any way to delay the building, so I don't know how to have an index with unbuilt partitions.
- Russell
On 2011-04-16 06:04:26, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-16 06:04:26)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589
> conf/hive-default.xml c42197f
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14015d0
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
> ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review533
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1110>
I thought this might what caused the original table to be used, instead of the stale index. By adding the index table, we keep the original table around. However, clearing the inputs before adding the index table didn't change anything.
ql/src/test/results/clientpositive/index_auto_test_if_used.q.out
<https://reviews.apache.org/r/558/#comment1109>
We shouldn't be seeing this output. We're still generating the right plan, but something is wrong when we run it.
- Russell
On 2011-04-23 06:41:49, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-23 06:41:49)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2cdaeb6
> conf/hive-default.xml 79ea477
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 69ee03b
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c41bb32
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
> ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review610
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1253>
This does not work because the GenMRTableScan1.process() does not get called on these objects after we set these, as far as I can tell.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1254>
At this point, the inputFormat for this work is actually null. I do not know which work I need to change the inputFormat for, and I can't think of an easy way to get to other work.
- Russell
On 2011-04-29 00:01:06, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-29 00:01:06)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f77f46c
> conf/hive-default.xml 6bd615e
> eclipse-templates/.classpath 8d2dc52
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 24e16e4
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java f90d64f
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2207ac4
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
> ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/
-----------------------------------------------------------
(Updated 2011-05-01 19:20:02.130293)
Review request for hive.
Summary
-------
Review request for HIVE-1644.12.patch
This addresses bug HIVE-1644.
https://issues.apache.org/jira/browse/HIVE-1644
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d28dad0
conf/hive-default.xml 89b5236
eclipse-templates/.classpath 8d2dc52
ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 24e16e4
ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 953cc4c
ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java f90d64f
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 092484a
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 404d1fa
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 0462749
ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/558/diff
Testing
-------
Thanks,
Russell
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/
-----------------------------------------------------------
(Updated 2011-04-29 00:01:06.921150)
Review request for hive.
Changes
-------
HIVE-1644.17.patch
Summary
-------
Review request for HIVE-1644.12.patch
This addresses bug HIVE-1644.
https://issues.apache.org/jira/browse/HIVE-1644
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f77f46c
conf/hive-default.xml 6bd615e
eclipse-templates/.classpath 8d2dc52
ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 24e16e4
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java f90d64f
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2207ac4
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/558/diff
Testing
-------
Thanks,
Russell
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/
-----------------------------------------------------------
(Updated 2011-04-23 06:41:49.007802)
Review request for hive.
Changes
-------
HIVE-1644.16.patch
Summary
-------
Review request for HIVE-1644.12.patch
This addresses bug HIVE-1644.
https://issues.apache.org/jira/browse/HIVE-1644
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2cdaeb6
conf/hive-default.xml 79ea477
ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 69ee03b
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c41bb32
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_test_if_used.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_test_if_used.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/558/diff
Testing
-------
Thanks,
Russell
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review528
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment1094>
When we run a query on a non-partitioned table, we get a single partition in queryPartitions of the whole table (with an empty partSpec). Then, when we add the partition columns to the list of indexed column, we end up adding all the columns in the src table, instead of just the partitioned ones. If we make sure the partSpec isn't empty, this doesn't happen.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1095>
We need to cast the work in this task to MapredWork in order to get the input size out (line 176). I'm not sure if this is the best place to do that checking.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1096>
See above comment about MapredWork
- Russell
On 2011-04-22 03:50:54, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-22 03:50:54)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2cdaeb6
> conf/hive-default.xml 79ea477
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 69ee03b
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c41bb32
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
> ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by John Sichi <js...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review530
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
<https://reviews.apache.org/r/558/#comment1106>
Create a followup task for dealing with jobs which access multiple tables. For that, we need to associate the index formats/files with specific tables, and that requires modifying the way the index input format works.
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
<https://reviews.apache.org/r/558/#comment1105>
Create a followup task for displaying these in the plan (to indicate that a table scan's input is being filtered by the intermediate file). We only want to do that when they are non-null (to avoid upsetting all the existing test reference files).
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment1099>
spacing
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment1100>
spacing
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1102>
spacing
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1101>
When logging errors being propagated, use the two-arg version of the method and pass e as the second arg. Same thing in a few other places.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1103>
curly bracket placement
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1104>
create a followup for this one
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1098>
This is not an error, just a condition that prevents usage of the index, so it should be logged as info rather than error.
- John
On 2011-04-22 03:50:54, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-22 03:50:54)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2cdaeb6
> conf/hive-default.xml 79ea477
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 69ee03b
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c41bb32
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
> ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/
-----------------------------------------------------------
(Updated 2011-04-22 03:50:54.602032)
Review request for hive.
Changes
-------
HIVE-1644.15.patch
Summary
-------
Review request for HIVE-1644.12.patch
This addresses bug HIVE-1644.
https://issues.apache.org/jira/browse/HIVE-1644
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2cdaeb6
conf/hive-default.xml 79ea477
ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 69ee03b
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c41bb32
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/558/diff
Testing
-------
Thanks,
Russell
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by John Sichi <js...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review494
-----------------------------------------------------------
conf/hive-default.xml
<https://reviews.apache.org/r/558/#comment1019>
BTW, these property names should be all-lowercase.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java
<https://reviews.apache.org/r/558/#comment1022>
When you add an overload, add Javadoc as well (including the new param's meaning).
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java
<https://reviews.apache.org/r/558/#comment1020>
Could you explain the usage interaction better (along the lines of what I explained in my review comment)?
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment1021>
You're right. Either we need to treat them as index columns (so that the predicates on them will automatically be collected by the predicate analyzer), or we need to explicitly generate corresponding equality predicates based on the partition values which have already been identified by partition pruning.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1024>
From an efficiency perspective, you certainly don't want to be doing this over and over inside the outer for loop; just do it once first outside.
Also, for a table with a huge number of partitions, fetching all of them is a bad idea; it's better to selectively query the partitions of interest (but batching them if possible).
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1023>
This doesn't work because the Partition class does not override the default Java equals method (which is based on object identity rather than value), and different metastore queries return different object instances for the same underlying entity.
ql/src/test/queries/clientpositive/index_auto_multiple.q
<https://reviews.apache.org/r/558/#comment1025>
I don't understand what you mean? src has two columns, key and value.
ql/src/test/queries/clientpositive/index_auto_unused.q
<https://reviews.apache.org/r/558/#comment1026>
From the index design doc, there's an optional PARTITION clause when rebuilding an index which allows you to build just one specific partition, leaving the others unbuilt. I think there are some examples in the unit tests.
ALTER INDEX index_name ON table_name [ PARTITION (...) ] REBUILD
- John
On 2011-04-16 06:04:26, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-16 06:04:26)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589
> conf/hive-default.xml c42197f
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14015d0
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
> ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/
-----------------------------------------------------------
(Updated 2011-04-16 06:04:26.681814)
Review request for hive.
Changes
-------
HIVE-1644.14.patch
Summary
-------
Review request for HIVE-1644.12.patch
This addresses bug HIVE-1644.
https://issues.apache.org/jira/browse/HIVE-1644
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589
conf/hive-default.xml c42197f
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14015d0
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/558/diff
Testing
-------
Thanks,
Russell
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by John Sichi <js...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review482
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment982>
A few comments here.
1) Rather than passing in the entire table scan object and letting the handler set properties on it, I think we should just have the handler pass back the necessary information (input format and intermediate file).
2) The generateIndexQuery method's parameter list is growing. For plugin interfaces, a good pattern we've been using in other places is to introduce a new context class (say HiveIndexQueryContext) with getters and setters for the information to be communicated in both directions. Then the caller instantiates one of these and passes in an instance. The plugin reads and writes to the context. On return, the caller gets the modified information out. The main benefit is that in the future, if we need to pass more information, we just add new members to the context class, and none of the existing plugin implementations break.
In this case, you could also put the context objects in a map (instead of having to keep multiple maps indexQueryTasks/additionalInputs etc).
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment983>
Just put it as a TODO for now; create the followup JIRA issue and reference it in the TODO.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment990>
Look in Hive.java; there are methods like
public List<Partition> getPartitionsByNames(Table tbl, List<String> partNames)
which look up the actual partitions for a table from the metastore. You can pass in indexTable.
ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out
<https://reviews.apache.org/r/558/#comment991>
Hmm...what if we could avoid relabeling altogether? If you look in Driver.java, there's a method compile which calls TaskFactory.resetId(). This is what causes us to start back over from 0.
If you add an optional parameter resetTaskIds=true, and then pass false for the Driver instance used for compiling the reentrant query, that might do it.
- John
On 2011-04-15 08:08:14, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-15 08:08:14)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589
> conf/hive-default.xml c42197f
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_opt_where.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_opt_where_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_opt_where_simple.q PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review472
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment953>
I would have liked to just make a copy of pctx before I called rewriteForIndex(...) for every index, and then just use whichever of those corresponded to the index rewrite we chose. However, the pctx did not seem to have an easy way to copy it.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment957>
Do we need to propagate the residual predicate any further?
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment955>
I'm kind of confused about how to check the actual table and not the metadata. When we call indexTable.getPartitionKeys() and part.getTable.getPartitionKeys(), that method calls getPartitionKeys() on the underlying Thrift Tables. Is there a way besides getPartitionKeys() that we should be using?
ql/src/test/queries/clientpositive/index_opt_where.q
<https://reviews.apache.org/r/558/#comment956>
I have not yet added the additional unit tests
ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out
<https://reviews.apache.org/r/558/#comment954>
I fixed the labeling for this case, but would it make sense to label our stages differently for indexing? We only relabel correctly as long as we're overwriting the highest numbered stage, since we only relabel a single task. Or, should it relabel all tasks in the whole plan? We only have easy access to the context.currentTask when we iterate through in IndexWhereProcessor (line 153)
- Russell
On 2011-04-15 08:08:14, Russell Melick wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/558/
> -----------------------------------------------------------
>
> (Updated 2011-04-15 08:08:14)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Review request for HIVE-1644.12.patch
>
>
> This addresses bug HIVE-1644.
> https://issues.apache.org/jira/browse/HIVE-1644
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589
> conf/hive-default.xml c42197f
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
> ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
> ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
> ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
> ql/src/test/queries/clientpositive/index_opt_where.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_opt_where_partitioned.q PRE-CREATION
> ql/src/test/queries/clientpositive/index_opt_where_simple.q PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/558/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Russell
>
>
Re: Review Request: HIVE-1644 Use filter pushdown for automatically
accessing indexes
Posted by Russell Melick <rm...@cs.hmc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/
-----------------------------------------------------------
(Updated 2011-04-15 08:08:14.640798)
Review request for hive.
Changes
-------
HIVE-1644.13.patch
Summary
-------
Review request for HIVE-1644.12.patch
This addresses bug HIVE-1644.
https://issues.apache.org/jira/browse/HIVE-1644
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589
conf/hive-default.xml c42197f
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b
ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9
ql/src/test/queries/clientpositive/index_opt_where.q PRE-CREATION
ql/src/test/queries/clientpositive/index_opt_where_partitioned.q PRE-CREATION
ql/src/test/queries/clientpositive/index_opt_where_simple.q PRE-CREATION
ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/558/diff
Testing
-------
Thanks,
Russell