You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Sushanth Sowmyan (JIRA)" <ji...@apache.org> on 2013/08/06 22:07:47 UTC
[jira] [Created] (HIVE-5011) Dynamic partitioning in HCatalog
broken on external tables
Sushanth Sowmyan created HIVE-5011:
--------------------------------------
Summary: Dynamic partitioning in HCatalog broken on external tables
Key: HIVE-5011
URL: https://issues.apache.org/jira/browse/HIVE-5011
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Dynamic partitioning with HCatalog has been broken as a result of HCATALOG-500 trying to support user-set paths for external tables.
The goal there was to be able to support other custom destinations apart from the normal "hive-style" partitions. However, it is not currently possible for users to set paths for dynamic ptn writes, since we don't support any way for users to specify "patterns"(like, say "${rootdir}/$v1.$v2/") into which writes happen, only "locations", and the values for dyn. partitions are not known ahead of time. Also, specifying a custom path messes with the way dynamic ptn. code tries to determine what was written to where from the output committer, which means that even if we supported patterned-writes instead of location-writes, we still have to do some more deep diving into the output committer code to support it.
Thus, my current proposal is that we honour writes to user-specified paths for external tables *ONLY* for static partition writes - i.e., if we can determine that the write is a dyn. ptn. write, we will ignore the user specification. (Note that this does not mean we ignore the table's external location - we honour that - we just don't honour any HCatStorer/etc provided additional location - we stick to what metadata tells us the root location is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira