You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Ratandeep Ratti (JIRA)" <ji...@apache.org> on 2018/04/16 05:56:00 UTC

[jira] [Commented] (HIVE-19040) get_partitions_by_expr() implementation in HiveMetaStore causes backward incompatibility easily

    [ https://issues.apache.org/jira/browse/HIVE-19040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439005#comment-16439005 ] 

Ratandeep Ratti commented on HIVE-19040:
----------------------------------------

{quote} It's also possible to create support for native expressions in metastore that would handle most Hive cases, i.e. basically replace Filter.g with Hive expression parsing and include common UDFs like IN, etc {quote}

I like this idea. Maybe we could just send {{org.apache.hadoop.hive.ql.plan.ExprNodeDesc#getExprString}} and convert that into a Hive expression on the Metastore side?  This would at least make us immune to UDF impl changes, kryo dependency coupling between server and client.

> get_partitions_by_expr() implementation  in HiveMetaStore causes backward incompatibility easily
> ------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-19040
>                 URL: https://issues.apache.org/jira/browse/HIVE-19040
>             Project: Hive
>          Issue Type: Improvement
>          Components: Standalone Metastore
>    Affects Versions: 2.0.0
>            Reporter: Aihua Xu
>            Priority: Major
>
> In the HiveMetaStore implementation of {{public PartitionsByExprResult get_partitions_by_expr(PartitionsByExprRequest req) throws TException}} , an expression is serialized into byte array from the client side and passed through  PartitionsByExprRequest. Then HMS will deserialize back into the expression and filter the partitions by it.
> Such partition filtering expression can contain various UDFs. If there are some changes to one of the UDFs between different Hive versions, HS2 on the older version will serialize the expression in old format which won't be able to be deserialized by HMS on the newer version.  One example of that is, GenericUDFIn class adds {{transient}}  to the field constantInSet which will cause such incompatibility.
> One approach I'm thinking of is, instead of converting the expression object to byte array, we can pass the expression string directly. 
>  
>  
>   
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)