You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (Jira)" <ji...@apache.org> on 2024/09/17 22:49:00 UTC
[jira] [Commented] (SPARK-49617) Stop ignoring auth errors when ignoreCorruptFiles=1 in FileScanRDD
[ https://issues.apache.org/jira/browse/SPARK-49617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882529#comment-17882529 ]
Xiao Li commented on SPARK-49617:
---------------------------------
[~yhosny] when did we introduce this behavior change?
> Stop ignoring auth errors when ignoreCorruptFiles=1 in FileScanRDD
> ------------------------------------------------------------------
>
> Key: SPARK-49617
> URL: https://issues.apache.org/jira/browse/SPARK-49617
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Yousof Hosny
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> We currently lose data when the ignoreCorruptFiles flag is set to true because file Authorization Errors are handled as if they were corrupt files. We ignore these files despite them having no corrupt records. We’d like to fix this bug by failing the stream when authorization errors are found.
> To prevent regressions we will add a SQLConf flag that can be temporarily set to revert to previous behavior. The idea is to deprecate the flag at a later major version release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org