You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by da...@apache.org on 2016/04/27 08:08:43 UTC
spark git commit: [SPARK-14911] [CORE] Fix a potential data race in
TaskMemoryManager
Repository: spark
Updated Branches:
refs/heads/master 8fda5a73d -> b2a456064
[SPARK-14911] [CORE] Fix a potential data race in TaskMemoryManager
## What changes were proposed in this pull request?
[[SPARK-13210][SQL] catch OOM when allocate memory and expand array](https://github.com/apache/spark/commit/37bc203c8dd5022cb11d53b697c28a737ee85bcc) introduced an `acquiredButNotUsed` field, but it might not be correctly synchronized:
- the write `acquiredButNotUsed += acquired` is guarded by `this` lock (see [here](https://github.com/apache/spark/blame/master/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java#L271));
- the read `memoryManager.releaseExecutionMemory(acquiredButNotUsed, taskAttemptId, tungstenMemoryMode)` (see [here](https://github.com/apache/spark/blame/master/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java#L400)) might not be correctly synchronized, and thus might not see `acquiredButNotUsed`'s most recent value.
This patch makes `acquiredButNotUsed` volatile to fix this.
## How was this patch tested?
This should be covered by existing suits.
Author: Liwei Lin <lw...@gmail.com>
Closes #12681 from lw-lin/fix-acquiredButNotUsed.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b2a45606
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b2a45606
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b2a45606
Branch: refs/heads/master
Commit: b2a45606481a6da6e1f68d14d1095a8dcf2a0e57
Parents: 8fda5a7
Author: Liwei Lin <lw...@gmail.com>
Authored: Tue Apr 26 23:08:40 2016 -0700
Committer: Davies Liu <da...@gmail.com>
Committed: Tue Apr 26 23:08:40 2016 -0700
----------------------------------------------------------------------
core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/b2a45606/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
index 6b7d9aa..2796114 100644
--- a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
+++ b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
@@ -114,7 +114,7 @@ public class TaskMemoryManager {
/**
* The amount of memory that is acquired but not used.
*/
- private long acquiredButNotUsed = 0L;
+ private volatile long acquiredButNotUsed = 0L;
/**
* Construct a new TaskMemoryManager.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org