You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2014/06/18 19:57:50 UTC
git commit: [SPARK-2162] Double check in doGetLocal to avoid read on
removed block.
Repository: spark
Updated Branches:
refs/heads/master 587d32012 -> 5ad5e3486
[SPARK-2162] Double check in doGetLocal to avoid read on removed block.
other wise, it will either read in vain in memory level case, or throw exception in disk level case when it believe the block is there while actually it had been removed.
Author: Raymond Liu <ra...@intel.com>
Closes #1103 from colorant/bm and squashes the following commits:
daac114 [Raymond Liu] Address comments
d1ea287 [Raymond Liu] Double check in doGetLocal to avoid read on removed block.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5ad5e348
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5ad5e348
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5ad5e348
Branch: refs/heads/master
Commit: 5ad5e3486aa4d13b0991de13f5f17d9897dd2753
Parents: 587d320
Author: Raymond Liu <ra...@intel.com>
Authored: Wed Jun 18 10:57:45 2014 -0700
Committer: Reynold Xin <rx...@apache.org>
Committed: Wed Jun 18 10:57:45 2014 -0700
----------------------------------------------------------------------
.../main/scala/org/apache/spark/storage/BlockManager.scala | 7 +++++++
1 file changed, 7 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/5ad5e348/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
index f52bc70..373987c 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
@@ -363,6 +363,13 @@ private[spark] class BlockManager(
val info = blockInfo.get(blockId).orNull
if (info != null) {
info.synchronized {
+ // Double check to make sure the block is still there, since removeBlock
+ // method also synchronizes on BlockInfo object, so the block might have
+ // been removed when we actually come here.
+ if (blockInfo.get(blockId).isEmpty) {
+ logDebug(s"Block $blockId had been removed")
+ return None
+ }
// If another thread is writing the block, wait for it to become ready.
if (!info.waitForReady()) {