[jira] [Updated] (SPARK-60445) Stop the transformWithState state server before a failed task aborts its state store

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-60445:
-----------------------------------
    Labels: pull-request-available  (was: )

> Stop the transformWithState state server before a failed task aborts its state store
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-60445
>                 URL: https://issues.apache.org/jira/browse/SPARK-60445
>             Project: Spark
>          Issue Type: Task
>          Components: Structured Streaming
>    Affects Versions: 4.3.0
>            Reporter: Dylan Wong
>            Priority: Major
>              Labels: pull-request-available
>
> When a Python {{transformWithState}} task fails or is killed (for example, a stage abort after an OOM, or a job cancel), the RocksDB store's failure listener aborts the store, which sets its stamp to -1. The state server is only stopped by a completion listener, and completion listeners run after failure listeners. So a Python worker that is still running can make one more state call, which hits {{{}STATE_STORE_INVALID_STAMP{}}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org