[jira] [Created] (SPARK-36137) HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS

["Chao Sun (Jira)" <ji...@apache.org>]

Chao Sun created SPARK-36137:
--------------------------------

             Summary: HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS
                 Key: SPARK-36137
                 URL: https://issues.apache.org/jira/browse/SPARK-36137
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.2.0
            Reporter: Chao Sun


At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use {{getAllPartitionsOf}} when {{hive.metastore.try.direct.sql}} is enabled in the remote HMS. However, in certain cases the remote HMS will fallback to use ORM to query the underlying RDBMS even if this config is set to true, and Spark will not be able to recover from the error and will just fail the query. 

For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2, and Spark was not able to pushdown filter for {{date}} column.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org