You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "He Yongqiang (JIRA)" <ji...@apache.org> on 2009/11/10 00:08:32 UTC
[jira] Created: (HIVE-918) Allow partition-wise file format
Allow partition-wise file format
---------------------------------
Key: HIVE-918
URL: https://issues.apache.org/jira/browse/HIVE-918
Project: Hadoop Hive
Issue Type: Improvement
Reporter: He Yongqiang
Assignee: He Yongqiang
Right now all partitions in a hive table share the same file format. We should allow partition wise file format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-918) Allow partition-wise file format
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao resolved HIVE-918.
-----------------------------
Resolution: Duplicate
Fix Version/s: 0.5.0
> Allow partition-wise file format
> ---------------------------------
>
> Key: HIVE-918
> URL: https://issues.apache.org/jira/browse/HIVE-918
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Fix For: 0.5.0
>
>
> Right now all partitions in a hive table share the same file format. We should allow partition wise file format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-918) Allow partition-wise file format
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775238#action_12775238 ]
He Yongqiang commented on HIVE-918:
-----------------------------------
bq. 1) Every partition has its own Storage Descriptor object which contains information about file descriptors, serdes and other storage related information that a table can have. There is no need to add more metadata to partition.
Thanks! So we will not need to add new metadata into patition.
bq. 2)
If we already filled all sd metadata of partitions. We will not need to add mnew, we can just change the table's existing sd and use that for new partitions. I will check that.
Thanks for the information, Prasad!
> Allow partition-wise file format
> ---------------------------------
>
> Key: HIVE-918
> URL: https://issues.apache.org/jira/browse/HIVE-918
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
>
> Right now all partitions in a hive table share the same file format. We should allow partition wise file format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-918) Allow partition-wise file format
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775212#action_12775212 ]
He Yongqiang commented on HIVE-918:
-----------------------------------
>>and a new one(let's say mold) for newly added
I mean " and a new one(let's say mnew) for newly added "
> Allow partition-wise file format
> ---------------------------------
>
> Key: HIVE-918
> URL: https://issues.apache.org/jira/browse/HIVE-918
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
>
> Right now all partitions in a hive table share the same file format. We should allow partition wise file format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-918) Allow partition-wise file format
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775233#action_12775233 ]
Prasad Chakka commented on HIVE-918:
------------------------------------
Regarding metadata,
1) Every partition has its own Storage Descriptor object which contains information about file descriptors, serdes and other storage related information that a table can have. There is no need to add more metadata to partition.
2) If we have 1) what is the need for this? Can we not get all the different file formats that a table has by querying the storage descriptors of all partitions of a table?
The plan for allowing metadata evolution is that the 'alter table ...' cmd changes metadata for a table but not for existing partitions. All new partitions inherit the new metadata of the table. Hive QL should figure out the list of partitions that a query touches on using just the latest table metadata and then fetch the partition metadata for any further stuff. Are we changing this to something different, if so what?
> Allow partition-wise file format
> ---------------------------------
>
> Key: HIVE-918
> URL: https://issues.apache.org/jira/browse/HIVE-918
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
>
> Right now all partitions in a hive table share the same file format. We should allow partition wise file format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-918) Allow partition-wise file format
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775211#action_12775211 ]
He Yongqiang commented on HIVE-918:
-----------------------------------
Just had an offline discussion with Namit.
1) Will add metadata about file format and serde into partition metadata.
2) And also will add in table new metadata about new partitions' fileformat and serde. So there will be two set of metadata, one (let's say mold) about old fileformat and serde, and a new one(let's say mold) for newly added partitions. The old one is specifies when creating the table and will not changed. The new one can be changed anytime.
For 1, will need to add a command " alter table T PARTITION p SET tableRowFormat tableSerde [FORCE] "
this will change the fileformat and serde metadata about the specified patition. And if force is present, will conver the partition's data by running an insert overwrite job.
For 2, will need to add a new command "ALTER TABLE SET tableRowFormat tableSerde". This will change mnew, and will only influence the new paritions added. We will copy the mnew into new partitions added. If mnew is null, we will just use mold and will not copy mold into partition metadata.
Any comments?
> Allow partition-wise file format
> ---------------------------------
>
> Key: HIVE-918
> URL: https://issues.apache.org/jira/browse/HIVE-918
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
>
> Right now all partitions in a hive table share the same file format. We should allow partition wise file format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.