[jira] [Commented] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects

["Peter Vary (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283516#comment-16283516 ] 

Peter Vary commented on HIVE-17990:
-----------------------------------

Hi [~alangates],

Thanks for working on this. I have taken a quick look at this patch, and have some questions:
AFAIK the current MetaStore API is quite verbose - and because of the limitations of Thrift - every object is serialized in the response as many times as it is referenced in the result. This causes serious memory issues in HS2 and/or HMS when running queries on tens of thousands of partitions.
If I understand the proposal correctly, this specific change adds even more data to the response, and in most of the cases this data will be the same for every partition in the request. This is true for the initial changes and even more so for the eventual results.

I am totally positive for the Schema Registry integration, and if we could find a way to not to increase the memory pressure, or we have plans for help this in the future, I would be happy.

Also it would be nice to have unit tests for the new HMS API interface methods in this jira, or one follow-up jira.

Please feel free to correct me if I am wrong in anything stated above.
Thanks,
Peter

> Add Thrift and DB storage for Schema Registry objects
> -----------------------------------------------------
>
>                 Key: HIVE-17990
>                 URL: https://issues.apache.org/jira/browse/HIVE-17990
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Standalone Metastore
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: Adding-Schema-Registry-to-Metastore.pdf
>
>
> This JIRA tracks changes to Thrift, RawStore, and DB scripts to support objects in the Schema Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)