Re: [PR] [SPARK-53002] Show accurate memory usage statistics when bounded memory is enabled for RocksDB [spark]

["anishshri-db (via GitHub)" <gi...@apache.org>]

anishshri-db commented on code in PR #51709:
URL: https://github.com/apache/spark/pull/51709#discussion_r2249002688


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBMemoryManager.scala:
##########
@@ -97,6 +97,31 @@ object RocksDBMemoryManager extends Logging with UnmanagedMemoryConsumer {
     logDebug(s"Unregistered instance $uniqueId")
   }
 
+

Review Comment:
   nit: extra newlines ?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBMemoryManager.scala:
##########
@@ -97,6 +97,31 @@ object RocksDBMemoryManager extends Logging with UnmanagedMemoryConsumer {
     logDebug(s"Unregistered instance $uniqueId")
   }
 
+
+  def getNumRocksDBInstances(boundedMemory: Boolean): Long = {
+    instanceMemoryMap.values().asScala.count(_.isBoundedMemory == boundedMemory)
+  }
+
+  /**
+   * Get the memory usage for a specific instance, accounting for bounded memory sharing.
+   * @param uniqueId The instance's unique identifier
+   * @param totalMemoryUsage The total memory usage of this instance
+   * @return The adjusted memory usage accounting for sharing in bounded memory mode
+   */
+  def getInstanceMemoryUsage(uniqueId: String, totalMemoryUsage: Long): Long = {
+    val instanceInfo = instanceMemoryMap.get(uniqueId)
+    if (instanceInfo.isBoundedMemory) {
+      // In bounded memory mode, divide by the number of bounded instances
+      // since they share the same memory pool
+      val numBoundedInstances = getNumRocksDBInstances(true)
+      totalMemoryUsage / numBoundedInstances
+    } else {
+      // In unbounded memory mode, each instance has its own memory
+      totalMemoryUsage
+    }
+  }
+
+

Review Comment:
   nit: same here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org