[jira] [Commented] (SPARK-21078) JobHistory applications synchronized is invalid

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047860#comment-16047860 ] 

Sean Owen commented on SPARK-21078:
-----------------------------------

I agree there's a problem here, which might or might not actually cause a problem in practice, but should be touched up.

First, {{applications}} doesn't need to be a mutable List, because it's not modified, and this is the justification for not synchronizing any other access to it. It's only ever swapped out for a new value. But that happens in two places and only one of them is protected in any way.

Is it acceptable to just return immediately from {{mergeApplicationListing}} or {{cleanLogs}} if it is already being run by another thread? that would solve it. It may be that the assumption is that these are only ever run by one thread anyway, in which case no synchronization is necessary

CC [~parth.brahmbhatt] [~tgraves] for https://issues.apache.org/jira/browse/SPARK-13988

> JobHistory applications synchronized is invalid
> -----------------------------------------------
>
>                 Key: SPARK-21078
>                 URL: https://issues.apache.org/jira/browse/SPARK-21078
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.1.1
>            Reporter: fangfengbin
>
> FsHistoryProvider has some threads to mergeApplicationListing.
> mergeApplicationListing function has code like this:
> applications.synchronized { 
> ...
> applications = mergedApps 
> ...
> } 
> applications object is changed,  so the synchronized do not work.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org