You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sergey Shelukhin (JIRA)" <ji...@apache.org> on 2018/03/27 18:19:00 UTC
[jira] [Comment Edited] (HIVE-19040) get_partitions_by_expr()
implementation in HiveMetaStore causes backward incompatibility easily
[ https://issues.apache.org/jira/browse/HIVE-19040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416039#comment-16416039 ]
Sergey Shelukhin edited comment on HIVE-19040 at 3/27/18 6:18 PM:
------------------------------------------------------------------
Hmm.. this only applies to CLI, right? I think we can assume metastore and HS2 will be the same version.
With standalone metastore, to utilize byexpr, Hive jar would need to be placed on metastore classpath (at least at the current state of this); so, again, metastore only serves as a conduit of the expression between Hive (HS2) and Hive (that jar); since both are server side we can expect them to be the same version.
CLI otoh would be affected.
The historical reason the proxy exists btw is because there's no metastore client module - so, metastore server cannot depend on QL. I guess with standalone metastore it will be set in stone that metastore cannot depend on Hive.
was (Author: sershe):
Hmm.. this only applies to CLI, right? I think we can assume metastore and HS2 will be the same version.
With standalone metastore, to utilize byexpr, Hive jar would need to be placed on metastore classpath (at least at the current state of this); so, again, metastore only serves as a conduit of the expression between Hive (HS2) and Hive (that jar).
The historical reason the proxy exists btw is because there's no metastore client module - so, metastore server cannot depend on QL. I guess with standalone metastore it will be set in stone that metastore cannot depend on Hive.
> get_partitions_by_expr() implementation in HiveMetaStore causes backward incompatibility easily
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-19040
> URL: https://issues.apache.org/jira/browse/HIVE-19040
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
> Affects Versions: 2.0.0
> Reporter: Aihua Xu
> Priority: Major
>
> In the HiveMetaStore implementation of {{public PartitionsByExprResult get_partitions_by_expr(PartitionsByExprRequest req) throws TException}} , an expression is serialized into byte array from the client side and passed through PartitionsByExprRequest. Then HMS will deserialize back into the expression and filter the partitions by it.
> Such partition filtering expression can contain various UDFs. If there are some changes to one of the UDFs between different Hive versions, HS2 on the older version will serialize the expression in old format which won't be able to be deserialized by HMS on the newer version. One example of that is, GenericUDFIn class adds {{transient}} to the field constantInSet which will cause such incompatibility.
> One approach I'm thinking of is, instead of converting the expression object to byte array, we can pass the expression string directly.
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)