You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Syed S. Albiz (JIRA)" <ji...@apache.org> on 2011/08/07 00:23:27 UTC
[jira] [Created] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Support automatic rebuilding of indexes when they go stale
----------------------------------------------------------
Key: HIVE-2354
URL: https://issues.apache.org/jira/browse/HIVE-2354
Project: Hive
Issue Type: Improvement
Reporter: Syed S. Albiz
Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086851#comment-13086851 ]
He Yongqiang commented on HIVE-2354:
------------------------------------
please add more explain statements to cover all cases where a indexupdater task may added.
otherwise it looks good.
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087908#comment-13087908 ]
jiraposter@reviews.apache.org commented on HIVE-2354:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1456/
-----------------------------------------------------------
(Updated 2011-08-19 19:20:00.521433)
Review request for hive, Yongqiang He, John Sichi, and Ning Zhang.
Summary
-------
Chain a second set of tasks to the MoveTask for INSERT statements.
This addresses bug HIVE-2354.
https://issues.apache.org/jira/browse/HIVE-2354
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 815f15a
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 6f311c6
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cb7a542
ql/src/test/queries/clientpositive/index_auto_update.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_update.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_stale.q.out 24900a1
Diff: https://reviews.apache.org/r/1456/diff
Testing
-------
Added testcase "index_auto_update.q"
Thanks,
Syed
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086934#comment-13086934 ]
jiraposter@reviews.apache.org commented on HIVE-2354:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1456/
-----------------------------------------------------------
(Updated 2011-08-18 10:28:02.416294)
Review request for hive, John Sichi and Ning Zhang.
Summary
-------
Chain a second set of tasks to the MoveTask for INSERT statements.
This addresses bug HIVE-2354.
https://issues.apache.org/jira/browse/HIVE-2354
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b46976f
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 6f311c6
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cec0d46
ql/src/test/queries/clientpositive/index_auto_update.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_update.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1456/diff
Testing
-------
Added testcase "index_auto_update.q"
Thanks,
Syed
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087939#comment-13087939 ]
jiraposter@reviews.apache.org commented on HIVE-2354:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1456/
-----------------------------------------------------------
(Updated 2011-08-19 20:20:22.217903)
Review request for hive, Yongqiang He, John Sichi, and Ning Zhang.
Changes
-------
Moves a bunch of the logic to update index timestamps into the AlterIndex DDLTask rather than the DDLSemanticAnalyzer, so that the timestamp is guaranteed to be updated *after* the base table is touched. This change is reflected in index_auto_update.q, as now we see that the check for index staleness in the SELECT passes and we use the index in index_auto_update.q as desired.
Summary
-------
Chain a second set of tasks to the MoveTask for INSERT statements.
This addresses bug HIVE-2354.
https://issues.apache.org/jira/browse/HIVE-2354
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 815f15a
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c031f40
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java c17891c
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 6f311c6
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cb7a542
ql/src/java/org/apache/hadoop/hive/ql/plan/AlterIndexDesc.java f0dfd00
ql/src/test/queries/clientpositive/index_auto_update.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_update.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_stale.q.out 24900a1
Diff: https://reviews.apache.org/r/1456/diff
Testing
-------
Added testcase "index_auto_update.q"
Thanks,
Syed
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch, HIVE-2354.6.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-2354:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed, thanks Syed!
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch, HIVE-2354.6.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Status: Patch Available (was: Open)
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch, HIVE-2354.6.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089792#comment-13089792 ]
He Yongqiang commented on HIVE-2354:
------------------------------------
+1, doing another run of tests
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch, HIVE-2354.6.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Status: Patch Available (was: Open)
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2354:
---------------------------------
Component/s: Indexing
Fix Version/s: 0.8.0
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Fix For: 0.8.0
>
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch, HIVE-2354.6.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087905#comment-13087905 ]
jiraposter@reviews.apache.org commented on HIVE-2354:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1456/
-----------------------------------------------------------
(Updated 2011-08-19 19:10:10.969158)
Review request for hive, Yongqiang He, John Sichi, and Ning Zhang.
Changes
-------
Made the configuration variable false by default, and set it to true in the relevant tests
Summary
-------
Chain a second set of tasks to the MoveTask for INSERT statements.
This addresses bug HIVE-2354.
https://issues.apache.org/jira/browse/HIVE-2354
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 815f15a
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 6f311c6
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cb7a542
ql/src/test/queries/clientpositive/index_auto_update.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_update.q.out PRE-CREATION
ql/src/test/results/clientpositive/index_stale.q.out 24900a1
Diff: https://reviews.apache.org/r/1456/diff
Testing
-------
Added testcase "index_auto_update.q"
Thanks,
Syed
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Attachment: HIVE-2354.2.patch
added clause for updating on LOAD DATA commands
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Status: Open (was: Patch Available)
just realized this does not cover the case where indexes go stale due to LOAD DATA/ALTER TABLE/ALTER PARTITION invocations
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087520#comment-13087520 ]
He Yongqiang commented on HIVE-2354:
------------------------------------
running tests
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Attachment: HIVE-2354.5.patch
I was setting the default setting on the AUTOUPDATE configuration variable to true, which was why the testcase was screwing up. Changed it to false by default, and explicitly set it in index_auto_update.q
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087823#comment-13087823 ]
He Yongqiang commented on HIVE-2354:
------------------------------------
index_auto_update.q and index_stale.q failed. Can u take a look?
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz reassigned HIVE-2354:
-----------------------------------
Assignee: Syed S. Albiz
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Status: Patch Available (was: Open)
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086935#comment-13086935 ]
jiraposter@reviews.apache.org commented on HIVE-2354:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1456/
-----------------------------------------------------------
(Updated 2011-08-18 10:28:36.489366)
Review request for hive, Yongqiang He, John Sichi, and Ning Zhang.
Changes
-------
updated testcases to include an EXPLAIN statement on the INSERT command to show the auto-update jobs
Summary
-------
Chain a second set of tasks to the MoveTask for INSERT statements.
This addresses bug HIVE-2354.
https://issues.apache.org/jira/browse/HIVE-2354
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b46976f
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 6f311c6
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cec0d46
ql/src/test/queries/clientpositive/index_auto_update.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_update.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1456/diff
Testing
-------
Added testcase "index_auto_update.q"
Thanks,
Syed
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082227#comment-13082227 ]
jiraposter@reviews.apache.org commented on HIVE-2354:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1456/
-----------------------------------------------------------
Review request for hive, John Sichi and Ning Zhang.
Summary
-------
Chain a second set of tasks to the MoveTask for INSERT statements.
This addresses bug HIVE-2354.
https://issues.apache.org/jira/browse/HIVE-2354
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b46976f
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cec0d46
ql/src/test/queries/clientpositive/index_auto_update.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_update.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1456/diff
Testing
-------
Added testcase "index_auto_update.q"
Thanks,
Syed
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085365#comment-13085365 ]
jiraposter@reviews.apache.org commented on HIVE-2354:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1456/
-----------------------------------------------------------
(Updated 2011-08-15 22:01:14.270380)
Review request for hive, John Sichi and Ning Zhang.
Changes
-------
added a clause for LOAD DATA
Summary
-------
Chain a second set of tasks to the MoveTask for INSERT statements.
This addresses bug HIVE-2354.
https://issues.apache.org/jira/browse/HIVE-2354
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b46976f
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 6f311c6
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cec0d46
ql/src/test/queries/clientpositive/index_auto_update.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_update.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1456/diff
Testing
-------
Added testcase "index_auto_update.q"
Thanks,
Syed
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087938#comment-13087938 ]
Syed S. Albiz commented on HIVE-2354:
-------------------------------------
Moved a bunch of the logic to update index timestamps into the AlterIndex DDLTask rather than the DDLSemanticAnalyzer, so that the timestamp is guaranteed to be updated *after* the base table is touched. This change is reflected in index_auto_update.q, as now we see that the check for index staleness in the SELECT passes and we use the index in index_auto_update.q as desired.
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch, HIVE-2354.6.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086850#comment-13086850 ]
He Yongqiang commented on HIVE-2354:
------------------------------------
can u do an explain on "INSERT OVERWRITE TABLE temp SELECT * FROM src;" ?
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087910#comment-13087910 ]
Syed S. Albiz commented on HIVE-2354:
-------------------------------------
Actually, looking more closely at the testcase output, there is an issue with this patch. The timestamps set when the index is rebuilt are still coming up stale, so we don't see index usage in the EXPLAIN plan of the SELECT query. I will investigate further
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Attachment: HIVE-2354.1.patch
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Status: Open (was: Patch Available)
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Attachment: HIVE-2354.6.patch
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch, HIVE-2354.5.patch, HIVE-2354.6.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Attachment: HIVE-2354.4.patch
updated testcase with EXPLAIN statement on INSERT command
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch, HIVE-2354.4.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2354) Support automatic rebuilding of
indexes when they go stale
Posted by "Syed S. Albiz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed S. Albiz updated HIVE-2354:
--------------------------------
Attachment: HIVE-2354.3.patch
oops, wrong patch
> Support automatic rebuilding of indexes when they go stale
> ----------------------------------------------------------
>
> Key: HIVE-2354
> URL: https://issues.apache.org/jira/browse/HIVE-2354
> Project: Hive
> Issue Type: Improvement
> Reporter: Syed S. Albiz
> Assignee: Syed S. Albiz
> Attachments: HIVE-2354.1.patch, HIVE-2354.2.patch, HIVE-2354.3.patch
>
>
> Support a mode where indexes will be automatically rebuilt when the table/partition they are based on is modified. So if index foo is built on table bar, and bar has it's contents overwritten, we should support a mode where index foo will automatically rebuild itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira