[jira] [Resolved] (SPARK-36137) HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS

["L. C. Hsieh (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-36137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

L. C. Hsieh resolved SPARK-36137.
---------------------------------
    Fix Version/s: 3.3.0
       Resolution: Fixed

Issue resolved by pull request 33382
[https://github.com/apache/spark/pull/33382]

> HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-36137
>                 URL: https://issues.apache.org/jira/browse/SPARK-36137
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.2.0
>            Reporter: Chao Sun
>            Assignee: Chao Sun
>            Priority: Major
>             Fix For: 3.3.0
>
>
> At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use {{getAllPartitionsOf}} when {{hive.metastore.try.direct.sql}} is enabled in the remote HMS. However, in certain cases the remote HMS will fallback to use ORM (which only support string type for partition columns) to query the underlying RDBMS even if this config is set to true, and Spark will not be able to recover from the error and will just fail the query. 
> For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2, and Spark was not able to pushdown filter for {{date}} column.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org