You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2025/09/10 03:09:50 UTC
Re: [PR] [SPARK-53446][CORE] Optimize BlockManager remove operations with cached block mappings [spark]
cloud-fan commented on code in PR #52210:
URL: https://github.com/apache/spark/pull/52210#discussion_r2335396295
##########
core/src/main/scala/org/apache/spark/storage/BlockManager.scala:
##########
@@ -288,6 +288,65 @@ private[spark] class BlockManager(
securityManager.getIOEncryptionKey())
private val maxRemoteBlockToMem = conf.get(config.MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM)
+ // SPARK-53446 Performance optimization: Cache mappings to avoid O(n) scans in remove operations
+ private[this] val rddToBlockIds =
+ new ConcurrentHashMap[Int, ConcurrentHashMap.KeySetView[BlockId, java.lang.Boolean]]()
+ private[this] val broadcastToBlockIds =
+ new ConcurrentHashMap[Long, ConcurrentHashMap.KeySetView[BlockId, java.lang.Boolean]]()
+ private[this] val sessionToBlockIds =
+ new ConcurrentHashMap[String, ConcurrentHashMap.KeySetView[BlockId, java.lang.Boolean]]()
+
+ /**
+ * Add a block ID to the appropriate cache mapping based on its type.
+ */
+ private def addToCache(blockId: BlockId): Unit = {
+ blockId match {
+ case rddBlockId: RDDBlockId =>
+ rddToBlockIds
+ .computeIfAbsent(rddBlockId.rddId, _ => ConcurrentHashMap.newKeySet())
+ .add(blockId)
+ case broadcastBlockId: BroadcastBlockId =>
+ broadcastToBlockIds
+ .computeIfAbsent(broadcastBlockId.broadcastId, _ => ConcurrentHashMap.newKeySet())
+ .add(blockId)
+ case cacheId: CacheId =>
+ sessionToBlockIds
+ .computeIfAbsent(cacheId.sessionUUID, _ => ConcurrentHashMap.newKeySet())
+ .add(blockId)
+ case _ => // Do nothing for other block types
+ }
+ }
+
+ /**
+ * Remove a block ID from the appropriate cache mapping based on its type.
+ */
+ private def removeFromCache(blockId: BlockId): Unit = {
+ blockId match {
+ case rddBlockId: RDDBlockId =>
+ Option(rddToBlockIds.get(rddBlockId.rddId)).foreach { blockSet =>
+ blockSet.remove(blockId)
+ if (blockSet.isEmpty) {
+ rddToBlockIds.remove(rddBlockId.rddId)
Review Comment:
I think there is a race condition here:
1. thread 1 calls `addToCache`, gets the map for its RDD id
2. thread 2 calls `removeFromCache`, gets the map for the same RDD id, remove the last block id, and then removes this RDD id from the cache
3. thread 1 adds the block id, but it's noop as this map entire is dangling now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org