You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Russell Melick (JIRA)" <ji...@apache.org> on 2011/04/23 05:43:05 UTC
[jira] [Created] (HIVE-2128) Automatic Indexing with multiple
tables
Automatic Indexing with multiple tables
---------------------------------------
Key: HIVE-2128
URL: https://issues.apache.org/jira/browse/HIVE-2128
Project: Hive
Issue Type: Improvement
Components: Indexing
Affects Versions: 0.8.0
Reporter: Russell Melick
Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Status: Patch Available (was: Open)
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070943#comment-13070943 ]
Hudson commented on HIVE-2128:
------------------------------
Integrated in Hive-trunk-h0.21 #848 (See [https://builds.apache.org/job/Hive-trunk-h0.21/848/])
HIVE-2128. Automatic Indexing with multiple tables.
(Syed Albiz via jvs)
jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150962
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java
* /hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out
* /hive/trunk/ql/src/test/queries/clientpositive/index_auto_mult_tables_compact.q
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
* /hive/trunk/ql/src/test/queries/clientpositive/index_auto_self_join.q
* /hive/trunk/ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java
* /hive/trunk/ql/src/test/results/clientpositive/index_auto_self_join.q.out
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
* /hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables.q.out
* /hive/trunk/ql/src/test/queries/clientpositive/index_auto_mult_tables.q
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Fix For: 0.8.0
>
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch, HIVE-2128.7.patch, HIVE-2128.8.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Attachment: HIVE-2128.4.patch
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060217#comment-13060217 ]
John Sichi commented on HIVE-2128:
----------------------------------
I don't think this approach works...what if both tables in the query have an applicable filter?
In that case, don't we need a separate blockfilter file per table alias?
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069299#comment-13069299 ]
jiraposter@reviews.apache.org commented on HIVE-2128:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1010/
-----------------------------------------------------------
(Updated 2011-07-21 23:52:23.929900)
Review request for hive and John Sichi.
Changes
-------
Added order by to testcases. This revealed an existing bug where we would walk the entire operator tree for each task in the task tree in IndexWhereTaskDispatcher. I amended this to only walk the subset of the operator tree in the current task.
Summary
-------
Grab the indexed tables during optimized query generation, grab the associated path URIs, and keep those around in the Configuration object. When the job is passed to ExecDriver, this data is extracted and used in HiveIndexedInputFormat to decide whether to use the index file or delegate to the parent (HiveInputFormat) class. Not sure if this is robust.
This addresses bug HIVE-2128.
https://issues.apache.org/jira/browse/HIVE-2128
Diffs (updated)
-----
ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out 4c9efd1
ql/src/test/results/clientpositive/index_auto_self_join.q.out PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java b9b586e
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java f1ee95d
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dbc489f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java da084f6
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java a03a9a6
ql/src/test/queries/clientpositive/index_auto_mult_tables.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_mult_tables_compact.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_self_join.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_mult_tables.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1010/diff
Testing
-------
added new testcase index_auto_mult_tables.q
Thanks,
Syed
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060199#comment-13060199 ]
John Sichi commented on HIVE-2128:
----------------------------------
I don't see the new testcase in either the patch or the reviewboard entry?
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Attachment: HIVE-2128.5.patch
removed unnecessary imports
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Attachment: HIVE-2128.8.patch
Ah, sorry, forgot to git rebase before regenerating the patch, some of the recently landed patches introduced changes to the testcase output.
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch, HIVE-2128.7.patch, HIVE-2128.8.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Status: Patch Available (was: Open)
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch, HIVE-2128.7.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-2128:
-----------------------------
Status: Open (was: Patch Available)
I got failures in the new tests:
testCliDriver_index_auto_mult_tables
testCliDriver_index_auto_mult_tables_compact
testCliDriver_index_auto_self_join
Plus this existing test:
testCliDriver_index_bitmap_auto_partitioned
Maybe you forgot to update the logs?
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060888#comment-13060888 ]
John Sichi commented on HIVE-2128:
----------------------------------
I was thinking of the case of compact indexes (one on each table).
Your test case is similar, but for bitmap indexes. We certainly should not be trying to combine the indexes in this case since they are on different tables! The plan looks strange already because it is applying the srcpart predicate twice, and the src index not at all. (It's hard to tell what's going on since the same predicate is applied on both tables; use a different predicate to see if it's two copies of the same vs one of each.)
Regardless of index type, I think we *should* be able to use indexes on different tables at once in the same query.
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-2128:
-----------------------------
Status: Open (was: Patch Available)
Reran with latest patch and still got failures in these three:
testCliDriver_index_auto_mult_tables
testCliDriver_index_auto_mult_tables_compact
testCliDriver_index_auto_self_join
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch, HIVE-2128.7.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Attachment: HIVE-2128.7.patch
Sorry, I forgot to regenerate the testcase outputs. Fixed in this patch
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch, HIVE-2128.7.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067878#comment-13067878 ]
jiraposter@reviews.apache.org commented on HIVE-2128:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1010/#review1112
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java
<https://reviews.apache.org/r/1010/#comment2271>
Why was this comment truncated?
ql/src/test/queries/clientpositive/index_auto_mult_tables.q
<https://reviews.apache.org/r/1010/#comment2273>
All of these SELECT statements need ORDER BY for determinism.
- John
On 2011-07-19 03:15:17, Syed Albiz wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1010/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-19 03:15:17)
bq.
bq.
bq. Review request for hive and John Sichi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Grab the indexed tables during optimized query generation, grab the associated path URIs, and keep those around in the Configuration object. When the job is passed to ExecDriver, this data is extracted and used in HiveIndexedInputFormat to decide whether to use the index file or delegate to the parent (HiveInputFormat) class. Not sure if this is robust.
bq.
bq.
bq. This addresses bug HIVE-2128.
bq. https://issues.apache.org/jira/browse/HIVE-2128
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. ql/src/test/results/clientpositive/index_auto_self_join.q.out PRE-CREATION
bq. ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out PRE-CREATION
bq. ql/src/test/queries/clientpositive/index_auto_self_join.q PRE-CREATION
bq. ql/src/test/results/clientpositive/index_auto_mult_tables.q.out PRE-CREATION
bq. ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java a03a9a6
bq. ql/src/test/queries/clientpositive/index_auto_mult_tables.q PRE-CREATION
bq. ql/src/test/queries/clientpositive/index_auto_mult_tables_compact.q PRE-CREATION
bq. ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java b9b586e
bq. ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java f1ee95d
bq. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5
bq. ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946
bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dbc489f
bq. ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java 617723e
bq.
bq. Diff: https://reviews.apache.org/r/1010/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. added new testcase index_auto_mult_tables.q
bq.
bq.
bq. Thanks,
bq.
bq. Syed
bq.
bq.
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069851#comment-13069851 ]
John Sichi commented on HIVE-2128:
----------------------------------
+1. Will commit when tests pass.
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067398#comment-13067398 ]
John Sichi commented on HIVE-2128:
----------------------------------
Could you make sure the latest patch is uploaded here and matching Review Board, and then click Submit Patch? Also make sure all spurious changes (like extra imports) are gone; I'm seeing some of those in Review Board.
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Status: Patch Available (was: Open)
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz reassigned HIVE-2128:
-----------------------------------
Assignee: Syed S. Albiz
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-2128:
-----------------------------
Resolution: Fixed
Fix Version/s: 0.8.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Syed!
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Fix For: 0.8.0
>
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch, HIVE-2128.7.patch, HIVE-2128.8.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Attachment: HIVE-2128.6.patch
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Status: Patch Available (was: Open)
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch, HIVE-2128.7.patch, HIVE-2128.8.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067470#comment-13067470 ]
jiraposter@reviews.apache.org commented on HIVE-2128:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1010/
-----------------------------------------------------------
(Updated 2011-07-19 03:15:17.006396)
Review request for hive and John Sichi.
Changes
-------
removed unnecessary imports from patch
Summary
-------
Grab the indexed tables during optimized query generation, grab the associated path URIs, and keep those around in the Configuration object. When the job is passed to ExecDriver, this data is extracted and used in HiveIndexedInputFormat to decide whether to use the index file or delegate to the parent (HiveInputFormat) class. Not sure if this is robust.
This addresses bug HIVE-2128.
https://issues.apache.org/jira/browse/HIVE-2128
Diffs (updated)
-----
ql/src/test/results/clientpositive/index_auto_self_join.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_self_join.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_mult_tables.q.out PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java a03a9a6
ql/src/test/queries/clientpositive/index_auto_mult_tables.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_mult_tables_compact.q PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java b9b586e
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java f1ee95d
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dbc489f
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java 617723e
Diff: https://reviews.apache.org/r/1010/diff
Testing
-------
added new testcase index_auto_mult_tables.q
Thanks,
Syed
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Attachment: HIVE-2128.2.patch
Hmm, you're right, we would need multiple blockfilter files in that case. I'm not sure if that is possible though in the framework we have, given that we choose one single index query to generate each time at each stage. In the case where we have multiple indexes involved at each stage, the index handler is responsible for combining index inputs into a single output file. For example, I have attached a case that seems like it might cause problems (both src and srcpart have indexes built and filters applied), however since the index queries are generated separately for separate table scans it seems like there is no collision between blockfilter files. Does this address the issue? It seems like there might still be a possibility of collision between table and blockfilter file, so I wonder if you had a different case in mind?
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060207#comment-13060207 ]
jiraposter@reviews.apache.org commented on HIVE-2128:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1010/
-----------------------------------------------------------
(Updated 2011-07-06 00:03:20.513755)
Review request for hive and John Sichi.
Changes
-------
updated patch to include testcase
Summary
-------
Grab the indexed tables during optimized query generation, grab the associated path URIs, and keep those around in the Configuration object. When the job is passed to ExecDriver, this data is extracted and used in HiveIndexedInputFormat to decide whether to use the index file or delegate to the parent (HiveInputFormat) class. Not sure if this is robust.
This addresses bug HIVE-2128.
https://issues.apache.org/jira/browse/HIVE-2128
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 090ecfc
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java 617723e
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java f1ee95d
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dbc489f
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java a03a9a6
ql/src/test/queries/clientpositive/index_auto_mult_tables.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_mult_tables.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1010/diff
Testing
-------
added new testcase index_auto_mult_tables.q
Thanks,
Syed
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Attachment: HIVE-2128.1.patch
updated to include testcase
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2128:
--------------------------------
Attachment: HIVE-2128.1.patch
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Attachments: HIVE-2128.1.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049555#comment-13049555 ]
John Sichi commented on HIVE-2128:
----------------------------------
HiveInputFormat already keeps track of the mapping from path to input format. So the idea here is that instead of setting HiveIndexedInputFormat globally for the entire job, we need to be associating it only with the paths that are supposed to have index filtering applied.
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-2128:
-----------------------------
Status: Open (was: Patch Available)
Comments on Review Board. After that it looks good to go!
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060075#comment-13060075 ]
jiraposter@reviews.apache.org commented on HIVE-2128:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1010/
-----------------------------------------------------------
Review request for hive and John Sichi.
Summary
-------
Grab the indexed tables during optimized query generation, grab the associated path URIs, and keep those around in the Configuration object. When the job is passed to ExecDriver, this data is extracted and used in HiveIndexedInputFormat to decide whether to use the index file or delegate to the parent (HiveInputFormat) class. Not sure if this is robust.
This addresses bug HIVE-2128.
https://issues.apache.org/jira/browse/HIVE-2128
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 090ecfc
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java 617723e
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java f1ee95d
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dbc489f
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java a03a9a6
Diff: https://reviews.apache.org/r/1010/diff
Testing
-------
added new testcase index_auto_mult_tables.q
Thanks,
Syed
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Attachments: HIVE-2128.1.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple
tables
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064253#comment-13064253 ]
jiraposter@reviews.apache.org commented on HIVE-2128:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1010/
-----------------------------------------------------------
(Updated 2011-07-13 00:29:56.738368)
Review request for hive and John Sichi.
Changes
-------
Revamped approach. We already uniquely assign filenames to each index query result, so instead of throwing those away, keep them in the indexIntermediateFile variable, and take the union of those input paths to generate the next set of input splits.
Summary
-------
Grab the indexed tables during optimized query generation, grab the associated path URIs, and keep those around in the Configuration object. When the job is passed to ExecDriver, this data is extracted and used in HiveIndexedInputFormat to decide whether to use the index file or delegate to the parent (HiveInputFormat) class. Not sure if this is robust.
This addresses bug HIVE-2128.
https://issues.apache.org/jira/browse/HIVE-2128
Diffs (updated)
-----
ql/src/test/results/clientpositive/index_auto_self_join.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/Driver.java b278ffe
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java 617723e
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java b9b586e
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java f1ee95d
ql/src/java/org/apache/hadoop/hive/ql/index/TableBasedIndexHandler.java 02ab78c
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dbc489f
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java a03a9a6
ql/src/test/queries/clientpositive/index_auto_mult_tables.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_mult_tables_compact.q PRE-CREATION
ql/src/test/queries/clientpositive/index_auto_self_join.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_mult_tables.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1010/diff
Testing
-------
added new testcase index_auto_mult_tables.q
Thanks,
Syed
> Automatic Indexing with multiple tables
> ---------------------------------------
>
> Key: HIVE-2128
> URL: https://issues.apache.org/jira/browse/HIVE-2128
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.8.0
> Reporter: Russell Melick
> Assignee: Syed S. Albiz
> Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch
>
>
> Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira