You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "John Sichi (JIRA)" <ji...@apache.org> on 2011/02/02 02:19:30 UTC
[jira] Created: (HIVE-1941) support explicit view partitioning
support explicit view partitioning
----------------------------------
Key: HIVE-1941
URL: https://issues.apache.org/jira/browse/HIVE-1941
Project: Hive
Issue Type: New Feature
Components: Query Processor
Affects Versions: 0.6.0
Reporter: John Sichi
Assignee: John Sichi
Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
For more information, see
http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1941:
-----------------------------
Status: Patch Available (was: Open)
HIVE-1941.4.patch passes tests. Review board request updated here:
https://reviews.apache.org/r/390/
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1941:
-----------------------------
Status: Patch Available (was: Open)
https://reviews.apache.org/r/390/
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang updated HIVE-1941:
----------------------------
Status: Open (was: Patch Available)
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998832#comment-12998832 ]
Paul Yang commented on HIVE-1941:
---------------------------------
Patch looks good once we have the aforementioned changes.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995546#comment-12995546 ]
John Sichi commented on HIVE-1941:
----------------------------------
Yongqiang, does that address your comments?
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1941:
-----------------------------
Attachment: HIVE-1941.2.patch
HIVE-1941.2.patch should pass tests.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001281#comment-13001281 ]
Paul Yang commented on HIVE-1941:
---------------------------------
+1 tests passed
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.8.0
>
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch, HIVE-1941.5.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993769#comment-12993769 ]
He Yongqiang commented on HIVE-1941:
------------------------------------
will take a look.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995666#comment-12995666 ]
Paul Yang commented on HIVE-1941:
---------------------------------
It looks like it's possible with the current thrift add_partition() method to create a partition for a view with a non-null SD/location. Can we put in a check to guard against this case?
Other than that, it looks good from the metastore/replication side.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996593#comment-12996593 ]
Paul Yang commented on HIVE-1941:
---------------------------------
@John - Yes, that's what I meant. I'll take a look at the whole patch as well.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999194#comment-12999194 ]
John Sichi commented on HIVE-1941:
----------------------------------
HIVE-1941.5 adds metastore-level checks and fixes conflicts with latest trunk.
I'm rerunning tests now.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch, HIVE-1941.5.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995626#comment-12995626 ]
He Yongqiang commented on HIVE-1941:
------------------------------------
sorry, i was in the middle of other things. will go through the code again.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996064#comment-12996064 ]
He Yongqiang commented on HIVE-1941:
------------------------------------
Paul, can you help review the whole patch? I am still in middle of sth else...
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang updated HIVE-1941:
----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed. Thanks John!
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.8.0
>
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch, HIVE-1941.5.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992211#comment-12992211 ]
John Sichi commented on HIVE-1941:
----------------------------------
Chatted with Paul: hook should get view+view partitions as output, underlying table+table partitions as input.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996006#comment-12996006 ]
John Sichi commented on HIVE-1941:
----------------------------------
Paul, just to confirm: alter_view_failure4.q already verifies that we can't set a location from the QL level, but you're saying we should add a metastore-level check too, right?
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1941:
-----------------------------
Fix Version/s: 0.8.0
Status: Patch Available (was: Open)
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.8.0
>
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch, HIVE-1941.5.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1941:
-----------------------------
Attachment: HIVE-1941.4.patch
HIVE-1941.4.patch fixes a few test failures in the previous patch; rerunning tests now.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1941:
-----------------------------
Status: Open (was: Patch Available)
Improvement needed from code review session at Facebook:
* ALTER VIEW ADD/DROP PARTITION needs to derive dependencies on underlying table partitions in order for replication to work (report them via hook input/outputs)
Followups needed:
* Store partition-level dependencies in metastore; use these later for stuff like retrieving stats from underlying table partitions
* CREATE OR REPLACE VIEW to avoid having to drop all partitions during schema evolution
* For a join view with join on partition column, optimizer should be smart enough to push predicate down both ways in order to avoid errors in strict mode (investigate whether this is already happening)
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1941:
-----------------------------
Attachment: HIVE-1941.3.patch
Running HIVE-1941.3.patch through tests now.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993783#comment-12993783 ]
He Yongqiang commented on HIVE-1941:
------------------------------------
How about the question raised in our internal review:
the relation between the view's partition key and the underlying tables' partition keys?
For example:
Is there a use case to support a view def like:
CREATE VIEW vp1
PARTITIONED ON (value)
AS
SELECT key, value
FROM src
WHERE key=86;
Because the view is partitioned on value, which is not the partition key of the underlying table. So the Partition on the view seems not very useful.
But if in an example:
CREATE VIEW vp2
PARTITIONED ON (ds)
AS
SELECT src.key, srcpart.value, srcpart.ds
FROM src
join srcpart
on src.key = srcpart.key
;
The vp2's ds originate from srcpart. So if there is a partition ds='2011' existing in srcpart, it makes sense to add a partition ds='2011' in vp2.
Another example:
create table srcpart_1 (key int, value string) partitioned by (ds string);
create table srcpart_2 (key int, value string) partitioned by (ds string, hr int);
CREATE VIEW vp2
PARTITIONED ON (ds, hr)
AS
SELECT srcpart_1.key, srcpart_2.value, srcpart_1.ds, srcpart_2.hr
FROM srcpart_1
join srcpart_2
on srcpart_1.key = srcpart_2.key
and srcpart_1.ds=srcpart_2.ds
;
>From this case the ds originate from srcpart_1, but the join condition put the same ds on srcpart_2. So a query like "select * from vp2 where ds='2000' and hr=11" is very smooth to run.
Some negative examples:
create table srcpart_1 (key int, value string) partitioned by (ds string);
create table srcpart_2 (key int, value string) partitioned by (ds string, hr int);
CREATE VIEW vp2
PARTITIONED ON (ds, hr)
AS
SELECT srcpart_1.key, srcpart_2.value, srcpart_1.ds, srcpart_2.hr
FROM srcpart_1
join srcpart_2
on srcpart_1.key = srcpart_2.key
;
We should not support this because it is hard to a partition pruning.
Another one:
CREATE VIEW vp2
PARTITIONED ON (ds, hr)
AS
SELECT srcpart_1.key, srcpart_2.value, srcpart_1.ds, srcpart_2.hr
FROM srcpart_1
join srcpart_2
on srcpart_1.key = srcpart_2.key
and srcpart_1.ds=srcpart_2.hr
;
we should probably also throw an error this definition.
I am not sure. But i think it makes sense to include the view's partition keys are from the underlying tables. Even not, we should make sure a prefix of the partition keys are from underlying tables.
And for partition keys from base tables, we should also make sure they cover a prefix of the base tables' partition keys (not randomly chosen).
A big problem of this implicit partition relationship between a partitioned view and the base tables is how to manage the relationships to keep consistent. For example, what if the base table's partition got dropped and the view's partition is still there? I think we should be fine here.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995668#comment-12995668 ]
Paul Yang commented on HIVE-1941:
---------------------------------
Similarly, we should handle the case when calling append_partition() on a view.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1941:
-----------------------------
Attachment: HIVE-1941.1.patch
HIVE-1941.1.patch is preliminary; I haven't run through tests yet.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993791#comment-12993791 ]
John Sichi commented on HIVE-1941:
----------------------------------
A use case for a partitioned view on top of an unpartitioned table would be when the table is an external table (being loaded by some other mechanism) or a non-native table (e.g. HBase). I think both of these are likely to come up eventually, so we might as well allow them from day one. The view partition would be used to indicate that some particular portion of the dataset has been fully loaded.
I verified that no join case currently works in strict mode because the predicate does not get pushed down both ways. I will open a followup for that (it's really something we should be optimizing regardless of views instead of forcing the user to duplicate the predicate).
If you look at my negative test cases, notice that in strict mode we prevent the case where the underlying table partitioning is not satisfied by the view. However, the exception is raised during ADD PARTITION (rather than during CREATE VIEW) because that's when we compile the dummy query.
Regarding prefixes: as Raghu mentioned for indexing, a prefix would only be relevant if we move to a hierarchical model for partitioning, which we don't currently have.
Regarding dependencies, we should work on that after HIVE-1073 is in place to track them (we need them in the metastore).
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1941:
-----------------------------
Attachment: HIVE-1941.5.patch
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch, HIVE-1941.5.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1941) support explicit view partitioning
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999243#comment-12999243 ]
John Sichi commented on HIVE-1941:
----------------------------------
Ready for commit.
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.8.0
>
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch, HIVE-1941.5.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1941) support explicit view partitioning
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1941:
---------------------------------
Component/s: Views
> support explicit view partitioning
> ----------------------------------
>
> Key: HIVE-1941
> URL: https://issues.apache.org/jira/browse/HIVE-1941
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor, Views
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.8.0
>
> Attachments: HIVE-1941.1.patch, HIVE-1941.2.patch, HIVE-1941.3.patch, HIVE-1941.4.patch, HIVE-1941.5.patch
>
>
> Allow creation of a view with an explicit partitioning definition, and support ALTER VIEW ADD/DROP PARTITION for instantiating partitions.
> For more information, see
> http://wiki.apache.org/hadoop/Hive/PartitionedViews
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira