You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2023/03/06 11:24:00 UTC

[jira] [Resolved] (SPARK-42668) Catch exception while trying to close compressed stream in HDFSStateStoreProvider abort

     [ https://issues.apache.org/jira/browse/SPARK-42668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim resolved SPARK-42668.
----------------------------------
    Fix Version/s: 3.5.0
       Resolution: Fixed

Issue resolved by pull request 40273
[https://github.com/apache/spark/pull/40273]

> Catch exception while trying to close compressed stream in HDFSStateStoreProvider abort
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-42668
>                 URL: https://issues.apache.org/jira/browse/SPARK-42668
>             Project: Spark
>          Issue Type: Task
>          Components: Structured Streaming
>    Affects Versions: 3.4.0
>            Reporter: Anish Shrigondekar
>            Assignee: Anish Shrigondekar
>            Priority: Major
>             Fix For: 3.5.0
>
>
> Catch exception while trying to close compressed stream in HDFSStateStoreProvider abort
> We have seen some cases where the task exits as cancelled/failed which triggers the abort in the task completion listener for HDFSStateStoreProvider. As part of this, we cancel the backing stream and close the compressed stream. However, different stores such as Azure blob store could throw exceptions which are not caught in the current path, leading to job failures. This change proposes to fix this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org