You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Herman van Hövell (Jira)" <ji...@apache.org> on 2022/06/02 14:51:00 UTC

[jira] [Created] (SPARK-39366) BlockInfoManager should not release write locks on task end

Herman van Hövell created SPARK-39366:
-----------------------------------------

             Summary: BlockInfoManager should not release write locks on task end
                 Key: SPARK-39366
                 URL: https://issues.apache.org/jira/browse/SPARK-39366
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.2.1
            Reporter: Herman van Hövell
            Assignee: Herman van Hövell


The BlockInfoManager releases all locks held by a task when the task is done. It also release write locks, the problem with that is that a thread (other than the main task thread) might still be modifying the block. By releasing it the block now seems readable, and a reader might observe a block in a partial or non-existent state.

This is fortunately not the massive problem as it appears to be because the BlockManager (only place where we write blocks) is well behaved and always puts the block in a consistent state. This means the errors caused by this are only transient.

Given the fact that the write code is well behaved we don't need to release the write locks on task end. We should remove that.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org