[jira] [Resolved] (SPARK-39083) Fix FsHistoryProvider race condition between update and clean app data

["Sean R. Owen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-39083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean R. Owen resolved SPARK-39083.
----------------------------------
    Fix Version/s: 3.3.0
                   3.2.2
       Resolution: Fixed

Issue resolved by pull request 36424
[https://github.com/apache/spark/pull/36424]

> Fix FsHistoryProvider race condition between update and clean app data
> ----------------------------------------------------------------------
>
>                 Key: SPARK-39083
>                 URL: https://issues.apache.org/jira/browse/SPARK-39083
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 3.2.1
>         Environment: CentOS Linux release 7.7.1908
> openjdk version "1.8.0_262"
> OpenJDK Runtime Environment (build 1.8.0_262-b10)
> OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)
>            Reporter: Vu Tan
>            Priority: Major
>             Fix For: 3.3.0, 3.2.2
>
>
> After SPARK-29043, FsHistoryProvider will list the log info without waiting all `mergeApplicationListing` task finished.
> However, the method `cleanAppData` for stale data is not thread safe and it could lead to a scenario when `cleanAppData` delete the entry of 
> ApplicationInfoWrapper for an application right after it has been updated by `mergeApplicationListing`.
> So there will be cases when the HS Web UI displays `Application not found` for applications whose logs does exist.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org