You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2016/11/02 05:19:58 UTC

[jira] [Resolved] (SPARK-17992) HiveClient.getPartitionsByFilter throws an exception for some unsupported filters when hive.metastore.try.direct.sql=false

     [ https://issues.apache.org/jira/browse/SPARK-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Reynold Xin resolved SPARK-17992.
---------------------------------
       Resolution: Fixed
         Assignee: Michael Allman
    Fix Version/s: 2.1.0

> HiveClient.getPartitionsByFilter throws an exception for some unsupported filters when hive.metastore.try.direct.sql=false
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-17992
>                 URL: https://issues.apache.org/jira/browse/SPARK-17992
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>            Reporter: Michael Allman
>            Assignee: Michael Allman
>             Fix For: 2.1.0
>
>
> We recently added (and enabled by default) table partition pruning for partitioned Hive tables converted to using {{TableFileCatalog}}. When the Hive configuration option {{hive.metastore.try.direct.sql}} is set to {{false}}, Hive will throw an exception for unsupported filter expressions. For example, attempting to filter on an integer partition column will throw a {{org.apache.hadoop.hive.metastore.api.MetaException}}.
> I discovered this behavior because VideoAmp uses the CDH version of Hive with a Postgresql metastore DB. In this configuration, CDH sets {{hive.metastore.try.direct.sql}} to {{false}} by default, and queries that filter on a non-string partition column will fail. That would be a rather rude surprise for these Spark 2.1 users...
> I'm not sure exactly what behavior we should expect, but I suggest that {{HiveClientImpl.getPartitionsByFilter}} catch this metastore exception and return all partitions instead. This is what Spark does for Hive 0.12 users, which does not support this feature at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org