You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/04/23 15:12:15 UTC
[PR] fix: use FAILED_READ_FILE.FILE_NOT_EXIST in Spark 4.0 FileNotFound shim [datafusion-comet]
andygrove opened a new pull request, #4048:
URL: https://github.com/apache/datafusion-comet/pull/4048
## Which issue does this PR close?
Closes #2946.
## Rationale for this change
The Spark 4.0 `ShimSparkErrorConverter` translates native FileNotFound errors into a `SparkFileNotFoundException` with error class `_LEGACY_ERROR_TEMP_2055`. That error class exists in Spark 3.4 and 3.5 but was removed in Spark 4.0, so Spark's `ErrorClassesJsonReader` throws an `INTERNAL_ERROR` when it tries to look up the message template ("Cannot find main error class '_LEGACY_ERROR_TEMP_2055'").
This manifests as test failures on `org.apache.spark.sql.hive.HiveMetadataCacheSuite` (and any other suite that uses `checkError`/`checkErrorMatchPVals` against `FAILED_READ_FILE.FILE_NOT_EXIST`) when run against Spark 4.0 with Comet enabled:
```
null did not equal "FAILED_READ_FILE.FILE_NOT_EXIST"
```
with underlying cause:
```
SparkException: [INTERNAL_ERROR] Cannot find main error class '_LEGACY_ERROR_TEMP_2055'
```
The previously ignored `sql_hive-1` CI job for Spark 4.0 (issue #2946) surfaces these failures once the job is re-enabled.
## What changes are included in this PR?
In the Spark 4.0 shim, delegate to `QueryExecutionErrors.fileNotExistError(path, cause)`, which is the 4.0 replacement for `readCurrentFileNotFoundError`. It produces a `SparkException` with the expected error class `FAILED_READ_FILE.FILE_NOT_EXIST` and `path` parameter that tests assert on. The 3.4 and 3.5 shims are unchanged.
## How are these changes tested?
Covered by the existing Spark test `org.apache.spark.sql.hive.HiveMetadataCacheSuite` (run in the `sql_hive-1` CI job for Spark 4.0), which previously failed on:
- `SPARK-16337 temporary view refresh`
- `view refresh`
- `partitioned table is cached when partition pruning is true`
- `partitioned table is cached when partition pruning is false`
Re-enabling that job will be handled separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: use FAILED_READ_FILE.FILE_NOT_EXIST in Spark 4.0 FileNotFound shim [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #4048: fix: use FAILED_READ_FILE.FILE_NOT_EXIST in Spark 4.0 FileNotFound shim
URL: https://github.com/apache/datafusion-comet/pull/4048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: use FAILED_READ_FILE.FILE_NOT_EXIST in Spark 4.0 FileNotFound shim [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #4048:
URL: https://github.com/apache/datafusion-comet/pull/4048#issuecomment-4305642989
this is included in #4047 - no need for separate PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org