You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/17 13:19:14 UTC
[GitHub] [spark] hvanhovell opened a new pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
hvanhovell opened a new pull request #34632:
URL: https://github.com/apache/spark/pull/34632
### What changes were proposed in this pull request?
This PR moves the BlockInfoManager from a single mutex per instance to much more fine grained locking at the block level. Concretely this PR makes the following changes:
- The use of `this.synchronized` for guarding against concurrent creation of a block has been replaced with a striped lock. We have effectively replaced a single coarse-grained lock with a lock per block.
- The use of `this.synchronized` for `wait()` and `notifyAll()` has been replaced with per-block Conditions.
- Extract common logic from `lockForWriting` and `lockForReading` into an acquireLock helper method. This deduplication is important given the size of the changes that this PR needed to make to the shared code.
- Optimization: call `currentTaskAttemptId` only once per method by storing its result into a local variable.
The PR is the first in a series of PRs. The next one will add group based locking and removal so we can remove broadcasts and cached RDDs in a constant time operation.
### Why are the changes needed?
The main motivation for this change is to increase and stabilize throughput of clusters that run a significant number of concurrent queries. In the current situation these queries are fighting for the BlockInfoManager lock when they create broadcasts. The worse problem arrises after we do a full GC. This triggers clean-up of unused broadcasts. The number of broadcasts in the system can be significant (>> 10K) when we run under load. We need to acquire a lock per broadcast block. We end up with all the BM worker threads, the DAG Scheduler thread, and the query threads fighting for a single lock. This causes the throughput to drop to close to 0 during these clean-up periods.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Functionally this is covered by existing tests. The performance has been checked by running 32 concurrent streams and hammering a cluster with queries. That shows the throughput drops are mostly gone now. I am not sure how well we can capture this in a non invasive benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971693578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #34632:
URL: https://github.com/apache/spark/pull/34632#discussion_r752099503
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -166,6 +189,48 @@ private[storage] class BlockInfoManager extends Logging {
Option(TaskContext.get()).map(_.taskAttemptId()).getOrElse(BlockInfo.NON_TASK_WRITER)
}
+ /**
+ * Helper for lock acquisistion.
+ */
+ private def acquireLock(
+ blockId: BlockId,
+ blocking: Boolean)(
+ f: BlockInfo => Boolean): Option[BlockInfo] = {
+ var done = false
+ var result: Option[BlockInfo] = None
+ while(!done) {
+ val wrapper = blockInfoWrappers.get(blockId)
+ if (wrapper == null) {
+ done = true
+ } else {
+ wrapper.withLock { (info, condition) =>
+ if (f(info)) {
+ result = Some(info)
+ done = true
+ } else if (!blocking) {
+ done = true
+ } else {
+ condition.await()
Review comment:
`Object.wait()` can throw an interrupted exception right? AFAICT this should behave the same. I also checked `Lock.lock()` and this seems to behave the same as synchronized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #34632:
URL: https://github.com/apache/spark/pull/34632#discussion_r751886165
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -122,24 +140,30 @@ private[storage] class BlockInfoManager extends Logging {
* set-if-not-exists operation ([[lockNewBlockForWriting()]]) and are removed
* by [[removeBlock()]].
*/
- @GuardedBy("this")
- private[this] val infos = new mutable.HashMap[BlockId, BlockInfo]
+ private[this] val blockInfoWrappers = new ConcurrentHashMap[BlockId, BlockInfoWrapper]
+
+ /**
+ * Stripe used to control multi-threaded access to block information.
+ *
+ * We are using this instead of the synchronizing on the [[BlockInfo]] objects to avoid race
+ * conditions in the `lockNewBlockForWriting` method. When this method returns successfully is is
Review comment:
nit: `is is` -> `it is`
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -166,6 +189,48 @@ private[storage] class BlockInfoManager extends Logging {
Option(TaskContext.get()).map(_.taskAttemptId()).getOrElse(BlockInfo.NON_TASK_WRITER)
}
+ /**
+ * Helper for lock acquisistion.
+ */
+ private def acquireLock(
+ blockId: BlockId,
+ blocking: Boolean)(
+ f: BlockInfo => Boolean): Option[BlockInfo] = {
+ var done = false
+ var result: Option[BlockInfo] = None
+ while(!done) {
+ val wrapper = blockInfoWrappers.get(blockId)
+ if (wrapper == null) {
+ done = true
+ } else {
+ wrapper.withLock { (info, condition) =>
+ if (f(info)) {
+ result = Some(info)
+ done = true
+ } else if (!blocking) {
+ done = true
+ } else {
+ condition.await()
Review comment:
Note: There is a difference now - `synchronized` does not result in `InterruptedException`, while `await` can throw the exception.
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -319,18 +376,35 @@ private[storage] class BlockInfoManager extends Logging {
*/
def lockNewBlockForWriting(
blockId: BlockId,
- newBlockInfo: BlockInfo): Boolean = synchronized {
+ newBlockInfo: BlockInfo): Boolean = {
logTrace(s"Task $currentTaskAttemptId trying to put $blockId")
- lockForReading(blockId) match {
- case Some(info) =>
- // Block already exists. This could happen if another thread races with us to compute
- // the same block. In this case, just keep the read lock and return.
- false
- case None =>
- // Block does not yet exist or is removed, so we are free to acquire the write lock
- infos(blockId) = newBlockInfo
- lockForWriting(blockId)
- true
+ // Get the lock that will be associated with the to-be written block and lock it for the entire
+ // duration of this operation. This way we prevent race conditions when two threads try to write
+ // the same block at the same time.
+ val lock = locks.get(blockId)
+ lock.lock()
+ try {
+ val wrapper = new BlockInfoWrapper(newBlockInfo, lock)
+ while (true) {
+ val previous = blockInfoWrappers.putIfAbsent(blockId, wrapper)
+ if (previous == null) {
+ // New block lock it for writing.
+ val result = lockForWriting(blockId, blocking = false)
+ assert(result.isDefined)
Review comment:
QQ: Can there be a race (and cause assertion failure) between `putIfAbsent` and `lockForWriting` ?
Do we want to return `true` if `result.isDefined` - else retry loop ?
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -341,106 +415,103 @@ private[storage] class BlockInfoManager extends Logging {
*
* @return the ids of blocks whose pins were released
*/
- def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = synchronized {
+ def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = {
val blocksWithReleasedLocks = mutable.ArrayBuffer[BlockId]()
- val readLocks = readLocksByTask.remove(taskAttemptId).getOrElse(ImmutableMultiset.of[BlockId]())
- val writeLocks = writeLocksByTask.remove(taskAttemptId).getOrElse(Seq.empty)
-
- for (blockId <- writeLocks) {
- infos.get(blockId).foreach { info =>
+ val writeLocks = Option(writeLocksByTask.remove(taskAttemptId)).getOrElse(Collections.emptySet)
+ writeLocks.forEach { blockId =>
+ blockInfo(blockId) { (info, condition) =>
assert(info.writerTask == taskAttemptId)
info.writerTask = BlockInfo.NO_WRITER
+ condition.signalAll()
}
blocksWithReleasedLocks += blockId
}
- readLocks.entrySet().iterator().asScala.foreach { entry =>
+ val readLocks = Option(readLocksByTask.remove(taskAttemptId))
+ .getOrElse(ImmutableMultiset.of[BlockId])
+ readLocks.entrySet().forEach { entry =>
Review comment:
I am still checking, but do we want to do both the `remove` before iterating over them ?
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -341,106 +415,103 @@ private[storage] class BlockInfoManager extends Logging {
*
* @return the ids of blocks whose pins were released
*/
- def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = synchronized {
+ def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = {
val blocksWithReleasedLocks = mutable.ArrayBuffer[BlockId]()
- val readLocks = readLocksByTask.remove(taskAttemptId).getOrElse(ImmutableMultiset.of[BlockId]())
- val writeLocks = writeLocksByTask.remove(taskAttemptId).getOrElse(Seq.empty)
-
- for (blockId <- writeLocks) {
- infos.get(blockId).foreach { info =>
+ val writeLocks = Option(writeLocksByTask.remove(taskAttemptId)).getOrElse(Collections.emptySet)
+ writeLocks.forEach { blockId =>
+ blockInfo(blockId) { (info, condition) =>
assert(info.writerTask == taskAttemptId)
info.writerTask = BlockInfo.NO_WRITER
+ condition.signalAll()
}
blocksWithReleasedLocks += blockId
}
- readLocks.entrySet().iterator().asScala.foreach { entry =>
+ val readLocks = Option(readLocksByTask.remove(taskAttemptId))
+ .getOrElse(ImmutableMultiset.of[BlockId])
+ readLocks.entrySet().forEach { entry =>
val blockId = entry.getElement
val lockCount = entry.getCount
blocksWithReleasedLocks += blockId
- get(blockId).foreach { info =>
+ blockInfo(blockId) { (info, condition) =>
info.readerCount -= lockCount
assert(info.readerCount >= 0)
+ condition.signalAll()
}
}
- notifyAll()
-
blocksWithReleasedLocks.toSeq
}
/** Returns the number of locks held by the given task. Used only for testing. */
private[storage] def getTaskLockCount(taskAttemptId: TaskAttemptId): Int = {
- readLocksByTask.get(taskAttemptId).map(_.size()).getOrElse(0) +
- writeLocksByTask.get(taskAttemptId).map(_.size).getOrElse(0)
+ Option(readLocksByTask.get(taskAttemptId)).map(_.size()).getOrElse(0) +
+ Option(writeLocksByTask.get(taskAttemptId)).map(_.size).getOrElse(0)
}
/**
* Returns the number of blocks tracked.
*/
- def size: Int = synchronized {
- infos.size
- }
+ def size: Int = blockInfoWrappers.size
/**
* Return the number of map entries in this pin counter's internal data structures.
* This is used in unit tests in order to detect memory leaks.
*/
- private[storage] def getNumberOfMapEntries: Long = synchronized {
+ private[storage] def getNumberOfMapEntries: Long = {
size +
readLocksByTask.size +
- readLocksByTask.map(_._2.size()).sum +
+ readLocksByTask.asScala.map(_._2.size()).sum +
writeLocksByTask.size +
- writeLocksByTask.map(_._2.size).sum
+ writeLocksByTask.asScala.map(_._2.size).sum
}
/**
* Returns an iterator over a snapshot of all blocks' metadata. Note that the individual entries
* in this iterator are mutable and thus may reflect blocks that are deleted while the iterator
* is being traversed.
*/
- def entries: Iterator[(BlockId, BlockInfo)] = synchronized {
- infos.toArray.toIterator
+ def entries: Iterator[(BlockId, BlockInfo)] = {
+ blockInfoWrappers.entrySet().iterator().asScala.map(kv => kv.getKey -> kv.getValue.info)
}
/**
* Removes the given block and releases the write lock on it.
*
* This can only be called while holding a write lock on the given block.
*/
- def removeBlock(blockId: BlockId): Unit = synchronized {
- logTrace(s"Task $currentTaskAttemptId trying to remove block $blockId")
- infos.get(blockId) match {
- case Some(blockInfo) =>
- if (blockInfo.writerTask != currentTaskAttemptId) {
- throw new IllegalStateException(
- s"Task $currentTaskAttemptId called remove() on block $blockId without a write lock")
- } else {
- infos.remove(blockId)
- blockInfo.readerCount = 0
- blockInfo.writerTask = BlockInfo.NO_WRITER
- writeLocksByTask.removeBinding(currentTaskAttemptId, blockId)
- }
- case None =>
- throw new IllegalArgumentException(
- s"Task $currentTaskAttemptId called remove() on non-existent block $blockId")
+ def removeBlock(blockId: BlockId): Unit = {
+ val taskAttemptId = currentTaskAttemptId
+ logTrace(s"Task $taskAttemptId trying to remove block $blockId")
+ blockInfo(blockId) { (info, condition) =>
+ if (info.writerTask != taskAttemptId) {
+ throw new IllegalStateException(
+ s"Task $taskAttemptId called remove() on block $blockId without a write lock")
+ } else {
+ blockInfoWrappers.remove(blockId)
+ info.readerCount = 0
+ info.writerTask = BlockInfo.NO_WRITER
+ writeLocksByTask.get(taskAttemptId).remove(blockId)
+ }
+ condition.signalAll()
}
- notifyAll()
}
/**
* Delete all state. Called during shutdown.
*/
- def clear(): Unit = synchronized {
- infos.valuesIterator.foreach { blockInfo =>
- blockInfo.readerCount = 0
- blockInfo.writerTask = BlockInfo.NO_WRITER
+ def clear(): Unit = {
+ blockInfoWrappers.values().forEach { wrapper =>
+ wrapper.withLock { (info, condition) =>
+ info.readerCount = 0
+ info.writerTask = BlockInfo.NO_WRITER
+ condition.signalAll()
+ }
Review comment:
nit: Given we are enhancing this codepath, do we want to do a `tryLock` for `clear` instead ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971634840
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49805/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971977875
**[Test build #145347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145347/testReport)** for PR 34632 at commit [`259c52d`](https://github.com/apache/spark/commit/259c52debf9c012691161722f568c021c9386658).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972871375
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49859/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972166075
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49818/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971590836
**[Test build #145336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145336/testReport)** for PR 34632 at commit [`a5e7e5a`](https://github.com/apache/spark/commit/a5e7e5a66459462d8bcb00ea169d4a88715f88b6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #34632:
URL: https://github.com/apache/spark/pull/34632#discussion_r752141810
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -341,106 +415,103 @@ private[storage] class BlockInfoManager extends Logging {
*
* @return the ids of blocks whose pins were released
*/
- def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = synchronized {
+ def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = {
val blocksWithReleasedLocks = mutable.ArrayBuffer[BlockId]()
- val readLocks = readLocksByTask.remove(taskAttemptId).getOrElse(ImmutableMultiset.of[BlockId]())
- val writeLocks = writeLocksByTask.remove(taskAttemptId).getOrElse(Seq.empty)
-
- for (blockId <- writeLocks) {
- infos.get(blockId).foreach { info =>
+ val writeLocks = Option(writeLocksByTask.remove(taskAttemptId)).getOrElse(Collections.emptySet)
+ writeLocks.forEach { blockId =>
+ blockInfo(blockId) { (info, condition) =>
assert(info.writerTask == taskAttemptId)
info.writerTask = BlockInfo.NO_WRITER
+ condition.signalAll()
}
blocksWithReleasedLocks += blockId
}
- readLocks.entrySet().iterator().asScala.foreach { entry =>
+ val readLocks = Option(readLocksByTask.remove(taskAttemptId))
+ .getOrElse(ImmutableMultiset.of[BlockId])
+ readLocks.entrySet().forEach { entry =>
Review comment:
I don't think it does. What are you worried about? Just FYI you will get NPEs if you try to take out locks for tasks that have been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971748935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971587532
**[Test build #145334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145334/testReport)** for PR 34632 at commit [`0138918`](https://github.com/apache/spark/commit/0138918012968614746c8188c629aba1a10a3e7b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972786425
**[Test build #145386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145386/testReport)** for PR 34632 at commit [`08a139c`](https://github.com/apache/spark/commit/08a139c680d2ad649c58333314f89c63dce26bcd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972278962
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145347/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971677488
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49807/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972128099
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49818/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #34632:
URL: https://github.com/apache/spark/pull/34632#discussion_r752140190
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -319,18 +376,35 @@ private[storage] class BlockInfoManager extends Logging {
*/
def lockNewBlockForWriting(
blockId: BlockId,
- newBlockInfo: BlockInfo): Boolean = synchronized {
+ newBlockInfo: BlockInfo): Boolean = {
logTrace(s"Task $currentTaskAttemptId trying to put $blockId")
- lockForReading(blockId) match {
- case Some(info) =>
- // Block already exists. This could happen if another thread races with us to compute
- // the same block. In this case, just keep the read lock and return.
- false
- case None =>
- // Block does not yet exist or is removed, so we are free to acquire the write lock
- infos(blockId) = newBlockInfo
- lockForWriting(blockId)
- true
+ // Get the lock that will be associated with the to-be written block and lock it for the entire
+ // duration of this operation. This way we prevent race conditions when two threads try to write
+ // the same block at the same time.
+ val lock = locks.get(blockId)
+ lock.lock()
+ try {
+ val wrapper = new BlockInfoWrapper(newBlockInfo, lock)
+ while (true) {
+ val previous = blockInfoWrappers.putIfAbsent(blockId, wrapper)
+ if (previous == null) {
+ // New block lock it for writing.
+ val result = lockForWriting(blockId, blocking = false)
+ assert(result.isDefined)
Review comment:
So my reasoning is this. You need to hold the lock for a block when we try to add it to the map. Nothing else can modify the block in the meantime. If the value returned is null it means it is a new block, and this means we should be able to acquire the lock without any issue (this also the reason why we use `blocking=false`).
I don't mind being defensive here. I can add a retry if you think we should.
TBH this is one of the more subtle changes in this PR. One of the problems is that the caller relies on the fact that the `BlockInfo` object passed in is also the one persisted in the BlockInfoManager if you acquire the write lock. If you would lock the block after putIfAbsent you have to deal with the situation where another thread might 'steal' the lock, and take out a read lock on an uninitialized block (or a few other weird scenarios). While there are probably ways to deal with that, I opted to use a simpler approach where this is not a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #34632:
URL: https://github.com/apache/spark/pull/34632#discussion_r752476094
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -166,6 +189,48 @@ private[storage] class BlockInfoManager extends Logging {
Option(TaskContext.get()).map(_.taskAttemptId()).getOrElse(BlockInfo.NON_TASK_WRITER)
}
+ /**
+ * Helper for lock acquisistion.
+ */
+ private def acquireLock(
+ blockId: BlockId,
+ blocking: Boolean)(
+ f: BlockInfo => Boolean): Option[BlockInfo] = {
+ var done = false
+ var result: Option[BlockInfo] = None
+ while(!done) {
+ val wrapper = blockInfoWrappers.get(blockId)
+ if (wrapper == null) {
+ done = true
+ } else {
+ wrapper.withLock { (info, condition) =>
+ if (f(info)) {
+ result = Some(info)
+ done = true
+ } else if (!blocking) {
+ done = true
+ } else {
+ condition.await()
Review comment:
You are right, `acquireLock` gets used only where we previous had a `wait`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972871375
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49859/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972858129
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49859/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972245605
**[Test build #145347 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145347/testReport)** for PR 34632 at commit [`259c52d`](https://github.com/apache/spark/commit/259c52debf9c012691161722f568c021c9386658).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971632765
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49807/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971713047
**[Test build #145334 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145334/testReport)** for PR 34632 at commit [`0138918`](https://github.com/apache/spark/commit/0138918012968614746c8188c629aba1a10a3e7b).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #34632:
URL: https://github.com/apache/spark/pull/34632#discussion_r752507849
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -341,106 +415,103 @@ private[storage] class BlockInfoManager extends Logging {
*
* @return the ids of blocks whose pins were released
*/
- def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = synchronized {
+ def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = {
val blocksWithReleasedLocks = mutable.ArrayBuffer[BlockId]()
- val readLocks = readLocksByTask.remove(taskAttemptId).getOrElse(ImmutableMultiset.of[BlockId]())
- val writeLocks = writeLocksByTask.remove(taskAttemptId).getOrElse(Seq.empty)
-
- for (blockId <- writeLocks) {
- infos.get(blockId).foreach { info =>
+ val writeLocks = Option(writeLocksByTask.remove(taskAttemptId)).getOrElse(Collections.emptySet)
+ writeLocks.forEach { blockId =>
+ blockInfo(blockId) { (info, condition) =>
assert(info.writerTask == taskAttemptId)
info.writerTask = BlockInfo.NO_WRITER
+ condition.signalAll()
}
blocksWithReleasedLocks += blockId
}
- readLocks.entrySet().iterator().asScala.foreach { entry =>
+ val readLocks = Option(readLocksByTask.remove(taskAttemptId))
+ .getOrElse(ImmutableMultiset.of[BlockId])
+ readLocks.entrySet().forEach { entry =>
Review comment:
I was trying to investigate if the delay between writeLocks remove and readLocks remove might cause any potential issues (given the condition signal for write locks iteration).
I did not find cause for concern though, so resolving conversation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971587532
**[Test build #145334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145334/testReport)** for PR 34632 at commit [`0138918`](https://github.com/apache/spark/commit/0138918012968614746c8188c629aba1a10a3e7b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971674843
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49805/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971693575
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972922616
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145386/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972815707
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49859/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972278962
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145347/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #34632:
URL: https://github.com/apache/spark/pull/34632#discussion_r752144293
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -341,106 +415,103 @@ private[storage] class BlockInfoManager extends Logging {
*
* @return the ids of blocks whose pins were released
*/
- def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = synchronized {
+ def releaseAllLocksForTask(taskAttemptId: TaskAttemptId): Seq[BlockId] = {
val blocksWithReleasedLocks = mutable.ArrayBuffer[BlockId]()
- val readLocks = readLocksByTask.remove(taskAttemptId).getOrElse(ImmutableMultiset.of[BlockId]())
- val writeLocks = writeLocksByTask.remove(taskAttemptId).getOrElse(Seq.empty)
-
- for (blockId <- writeLocks) {
- infos.get(blockId).foreach { info =>
+ val writeLocks = Option(writeLocksByTask.remove(taskAttemptId)).getOrElse(Collections.emptySet)
+ writeLocks.forEach { blockId =>
+ blockInfo(blockId) { (info, condition) =>
assert(info.writerTask == taskAttemptId)
info.writerTask = BlockInfo.NO_WRITER
+ condition.signalAll()
}
blocksWithReleasedLocks += blockId
}
- readLocks.entrySet().iterator().asScala.foreach { entry =>
+ val readLocks = Option(readLocksByTask.remove(taskAttemptId))
+ .getOrElse(ImmutableMultiset.of[BlockId])
+ readLocks.entrySet().forEach { entry =>
val blockId = entry.getElement
val lockCount = entry.getCount
blocksWithReleasedLocks += blockId
- get(blockId).foreach { info =>
+ blockInfo(blockId) { (info, condition) =>
info.readerCount -= lockCount
assert(info.readerCount >= 0)
+ condition.signalAll()
}
}
- notifyAll()
-
blocksWithReleasedLocks.toSeq
}
/** Returns the number of locks held by the given task. Used only for testing. */
private[storage] def getTaskLockCount(taskAttemptId: TaskAttemptId): Int = {
- readLocksByTask.get(taskAttemptId).map(_.size()).getOrElse(0) +
- writeLocksByTask.get(taskAttemptId).map(_.size).getOrElse(0)
+ Option(readLocksByTask.get(taskAttemptId)).map(_.size()).getOrElse(0) +
+ Option(writeLocksByTask.get(taskAttemptId)).map(_.size).getOrElse(0)
}
/**
* Returns the number of blocks tracked.
*/
- def size: Int = synchronized {
- infos.size
- }
+ def size: Int = blockInfoWrappers.size
/**
* Return the number of map entries in this pin counter's internal data structures.
* This is used in unit tests in order to detect memory leaks.
*/
- private[storage] def getNumberOfMapEntries: Long = synchronized {
+ private[storage] def getNumberOfMapEntries: Long = {
size +
readLocksByTask.size +
- readLocksByTask.map(_._2.size()).sum +
+ readLocksByTask.asScala.map(_._2.size()).sum +
writeLocksByTask.size +
- writeLocksByTask.map(_._2.size).sum
+ writeLocksByTask.asScala.map(_._2.size).sum
}
/**
* Returns an iterator over a snapshot of all blocks' metadata. Note that the individual entries
* in this iterator are mutable and thus may reflect blocks that are deleted while the iterator
* is being traversed.
*/
- def entries: Iterator[(BlockId, BlockInfo)] = synchronized {
- infos.toArray.toIterator
+ def entries: Iterator[(BlockId, BlockInfo)] = {
+ blockInfoWrappers.entrySet().iterator().asScala.map(kv => kv.getKey -> kv.getValue.info)
}
/**
* Removes the given block and releases the write lock on it.
*
* This can only be called while holding a write lock on the given block.
*/
- def removeBlock(blockId: BlockId): Unit = synchronized {
- logTrace(s"Task $currentTaskAttemptId trying to remove block $blockId")
- infos.get(blockId) match {
- case Some(blockInfo) =>
- if (blockInfo.writerTask != currentTaskAttemptId) {
- throw new IllegalStateException(
- s"Task $currentTaskAttemptId called remove() on block $blockId without a write lock")
- } else {
- infos.remove(blockId)
- blockInfo.readerCount = 0
- blockInfo.writerTask = BlockInfo.NO_WRITER
- writeLocksByTask.removeBinding(currentTaskAttemptId, blockId)
- }
- case None =>
- throw new IllegalArgumentException(
- s"Task $currentTaskAttemptId called remove() on non-existent block $blockId")
+ def removeBlock(blockId: BlockId): Unit = {
+ val taskAttemptId = currentTaskAttemptId
+ logTrace(s"Task $taskAttemptId trying to remove block $blockId")
+ blockInfo(blockId) { (info, condition) =>
+ if (info.writerTask != taskAttemptId) {
+ throw new IllegalStateException(
+ s"Task $taskAttemptId called remove() on block $blockId without a write lock")
+ } else {
+ blockInfoWrappers.remove(blockId)
+ info.readerCount = 0
+ info.writerTask = BlockInfo.NO_WRITER
+ writeLocksByTask.get(taskAttemptId).remove(blockId)
+ }
+ condition.signalAll()
}
- notifyAll()
}
/**
* Delete all state. Called during shutdown.
*/
- def clear(): Unit = synchronized {
- infos.valuesIterator.foreach { blockInfo =>
- blockInfo.readerCount = 0
- blockInfo.writerTask = BlockInfo.NO_WRITER
+ def clear(): Unit = {
+ blockInfoWrappers.values().forEach { wrapper =>
+ wrapper.withLock { (info, condition) =>
+ info.readerCount = 0
+ info.writerTask = BlockInfo.NO_WRITER
+ condition.signalAll()
+ }
Review comment:
Sure will update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971737392
**[Test build #145336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145336/testReport)** for PR 34632 at commit [`a5e7e5a`](https://github.com/apache/spark/commit/a5e7e5a66459462d8bcb00ea169d4a88715f88b6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971977875
**[Test build #145347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145347/testReport)** for PR 34632 at commit [`259c52d`](https://github.com/apache/spark/commit/259c52debf9c012691161722f568c021c9386658).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972786425
**[Test build #145386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145386/testReport)** for PR 34632 at commit [`08a139c`](https://github.com/apache/spark/commit/08a139c680d2ad649c58333314f89c63dce26bcd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972166075
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49818/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971590836
**[Test build #145336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145336/testReport)** for PR 34632 at commit [`a5e7e5a`](https://github.com/apache/spark/commit/a5e7e5a66459462d8bcb00ea169d4a88715f88b6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972909271
**[Test build #145386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145386/testReport)** for PR 34632 at commit [`08a139c`](https://github.com/apache/spark/commit/08a139c680d2ad649c58333314f89c63dce26bcd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972922616
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145386/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-973282031
Thanks for the reviews! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell closed pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
hvanhovell closed pull request #34632:
URL: https://github.com/apache/spark/pull/34632
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971593952
@JoshRosen can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-971748935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34632:
URL: https://github.com/apache/spark/pull/34632#issuecomment-972039929
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49818/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #34632: [SPARK-37356][CORE] Add fine grained locking to the BlockInfoManager
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #34632:
URL: https://github.com/apache/spark/pull/34632#discussion_r752505963
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala
##########
@@ -319,18 +376,35 @@ private[storage] class BlockInfoManager extends Logging {
*/
def lockNewBlockForWriting(
blockId: BlockId,
- newBlockInfo: BlockInfo): Boolean = synchronized {
+ newBlockInfo: BlockInfo): Boolean = {
logTrace(s"Task $currentTaskAttemptId trying to put $blockId")
- lockForReading(blockId) match {
- case Some(info) =>
- // Block already exists. This could happen if another thread races with us to compute
- // the same block. In this case, just keep the read lock and return.
- false
- case None =>
- // Block does not yet exist or is removed, so we are free to acquire the write lock
- infos(blockId) = newBlockInfo
- lockForWriting(blockId)
- true
+ // Get the lock that will be associated with the to-be written block and lock it for the entire
+ // duration of this operation. This way we prevent race conditions when two threads try to write
+ // the same block at the same time.
+ val lock = locks.get(blockId)
+ lock.lock()
+ try {
+ val wrapper = new BlockInfoWrapper(newBlockInfo, lock)
+ while (true) {
+ val previous = blockInfoWrappers.putIfAbsent(blockId, wrapper)
+ if (previous == null) {
+ // New block lock it for writing.
+ val result = lockForWriting(blockId, blocking = false)
+ assert(result.isDefined)
Review comment:
Sounds good, I wanted to make sure I understood the behavior now.
I am fine with leaving it as is - the assertion failure, if it happens, will indicate if this assumption is getting violated anyway !.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org