You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2026/07/09 01:32:00 UTC

[jira] [Resolved] (SPARK-45040) Caught Hive MetaException attempting to get partition metadata by filter from Hive

     [ https://issues.apache.org/jira/browse/SPARK-45040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-45040.
----------------------------------
    Resolution: Not A Problem

Resolving as Not A Problem: this 'Caught Hive MetaException attempting to get partition metadata by filter from Hive' error is a metastore-integration issue, and the exception message itself documents the workaround — set spark.sql.hive.manageFilesourcePartitions=false (or ensure the external metastore, e.g. AWS Glue, supports partition filter pushdown). No Spark code change is actionable here. Please reopen with a reproducer against a supported version if you believe this is a Spark bug.

> Caught Hive MetaException attempting to get partition metadata by filter from Hive
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-45040
>                 URL: https://issues.apache.org/jira/browse/SPARK-45040
>             Project: Spark
>          Issue Type: Bug
>          Components: Java API
>    Affects Versions: 2.4.0
>         Environment: AWS Glue
>            Reporter: Yangbo Yu
>            Priority: Critical
>         Attachments: Screenshot Capture - 2023-08-31 - 19-02-31.png
>
>
> We are integrating Spark 2.4 with our AWS Glue ETL job.
> And we recently realized that a lot of our jobs are failed because of the below error:
> {{Exception in User Class: java.lang.RuntimeException : Caught Hive MetaException attempting to get partition metadata by filter from Hive. You can set the Spark configuration setting spark.sql.hive.manageFilesourcePartitions to false to work around this problem, however this will result in degraded performance. Please report a bug: }}{{https://issues.apache.org/jira/browse/SPARK
> }}
> This error first happens from Aug 30th, and it occurs from time to time. It is gone for several hours and occurs again. During the time that the error occurs, most of the jobs fail, only few succeed. 
> And we tried to set {{spark.sql.hive.manageFilesourcePartitions}} to false but it did not work out. Some other issue is coming out.
> Can you look into the error and let me know if there is any work around to mitigate the issue? 
> Let me know if you need anything from my end.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org