[jira] [Commented] (HIVE-5011) Dynamic partitioning in HCatalog broken on external tables

["Sushanth Sowmyan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731226#comment-13731226 ] 

Sushanth Sowmyan commented on HIVE-5011:
----------------------------------------

Basic bug synopsis:

Say a table t1 partitioned by key dayofweek:string is present in location "hdfs://blah/foo/t1/".

Ordinarily, if we try to write to it specifying that we're writing a partition dayofweek="sunday", then the location it'll write to is "hdfs://blah/foo/t1/dayofweek=sunday/".

Now, this is known before the MR jobs start, and will be set as the location, and all is good. If the table is specified as an external table, and the user wants to specify a custom location format for the location, such that they want "hdfs://blah/foo/t1/sunday/", then HCat Storer currently allows them to specify that, and that will be honoured too.

That was the intent of HCATALOG-500, and the way it works for static partitioning.

With dynamic partitioning on external tables, with HCATALOG-500, however, this is what winds up happening.

All the partitions being written to wind up having their location set as "hdfs://blah/foo/t1/dayofweek=__DEFAULT_HIVE_PARTITION__" if no override is provided , or to "hdfs://blah/foo/t1/whatever" if that location was provided as an override.

This results in the first partition writes from the drones writing to this location, and all other drones not being able to open to write, stalling, getting retried, and having the job fail. It would be possible, in theory, if there were only one reducer in the job, and all data present in only one partition worth of writing, that the job might not fail, but that's a highly constrained mode of writing which makes the dynamic partitioning feature itself meaningless.
                
> Dynamic partitioning in HCatalog broken on external tables
> ----------------------------------------------------------
>
>                 Key: HIVE-5011
>                 URL: https://issues.apache.org/jira/browse/HIVE-5011
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>
> Dynamic partitioning with HCatalog has been broken as a result of HCATALOG-500 trying to support user-set paths for external tables.
> The goal there was to be able to support other custom destinations apart from the normal "hive-style" partitions. However, it is not currently possible for users to set paths for dynamic ptn writes, since we don't support any way for users to specify "patterns"(like, say "$\{rootdir\}/$v1.$v2/") into which writes happen, only "locations", and the values for dyn. partitions are not known ahead of time. Also, specifying a custom path messes with the way dynamic ptn. code tries to determine what was written to where from the output committer, which means that even if we supported patterned-writes instead of location-writes, we still have to do some more deep diving into the output committer code to support it.
> Thus, my current proposal is that we honour writes to user-specified paths for external tables *ONLY* for static partition writes - i.e., if we can determine that the write is a dyn. ptn. write, we will ignore the user specification. (Note that this does not mean we ignore the table's external location - we honour that - we just don't honour any HCatStorer/etc provided additional location - we stick to what metadata tells us the root location is.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira