[jira] [Commented] (HIVE-11344) HIVE-9845 makes HCatSplit.write modify the split so that PartitionInfo objects are unusable after it

["Sushanth Sowmyan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-11344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637678#comment-14637678 ] 

Sushanth Sowmyan commented on HIVE-11344:
-----------------------------------------

There are three routes I see available here:

a) There is decompress logic in PartInfo.setTableInfo, and compress logic in PartInfo.writeObject. we could make it so that PartInfo.writeObject does the "compression", writes itself, and then does the decompression back.
b) We could decompress on demand - wherein if a user calls getInputFormatClassName(), we then fetch that info if it's not available, and always return values consistently.
c) We could add a new conf parameter that controls whether or not we do compression - users with 100k splits would prefer compression, and be okay with the fact that PartInfo objects are not usable, and users that want to use the PartInfo objects will be okay with the fact that they are going to hog a little bit more serialized space.

(c) is a bad solution all-round. [~ashutoshc] would be mad at me for adding another conf parameter, and it is entirely possible that those that are trying to implement other streaming interfaces/etc and are mimicing M/R will run into a large number of partitions as well.
(b) is nifty, and I probably like the idea of, but I'm not entirely certain if it will run afoul of other serialization methods in the future that call getters to get fields (some json serializers) which might result in a bloated serialized PartInfo object anyway. Also, it spreads the decompression logic across multiple getters, and pushes the assert statement in multiple places as well.
(a) is probably the cleanest solution, although it makes a code reader wonder why we're going through the gymnastics we are. Some code comments might help with that.


> HIVE-9845 makes HCatSplit.write modify the split so that PartitionInfo objects are unusable after it
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-11344
>                 URL: https://issues.apache.org/jira/browse/HIVE-11344
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 1.2.0
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>
> HIVE-9845 introduced a notion of compression for HCatSplits so that when serializing, it finds commonalities between PartInfo and TableInfo objects, and if the two are identical, it nulls out that field in PartInfo, thus making sure that when PartInfo is then serialized, info is not repeated.
> This, however, has the side effect of making the PartInfo object unusable if HCatSplit.write has been called.
> While this does not affect M/R directly, since they do not know about the PartInfo objects and once serialized, the HCatSplit object is recreated by deserializing on the backend, which does restore the split and its PartInfo objects, this does, however, affect framework users of HCat that try to mimic M/R and then use the PartInfo objects to instantiate distinct readers.
> Thus, we need to make it so that PartInfo is still usable after HCatSplit.write is called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)