You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zhangxiong0301 (via GitHub)" <gi...@apache.org> on 2024/03/13 09:33:14 UTC

[PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

zhangxiong0301 opened a new pull request, #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574

   ### What changes were proposed in this pull request?
   
   In the implementation of the methods `flushBuffer`, `handleEventAndUpdateMetrics`, and `removeBufferByShuffleId`, read-write locks have been added to manage concurrency. This ensures that a `ShuffleBuffer` successfully converted into a `flushEvent` won't be cleaned up again by `removeBufferByShuffleId`, and a `ShuffleBuffer` already cleaned up by `removeBufferByShuffleId` won't be transformed back into a `flushEvent`. This effectively resolves the concurrency issue.
   
   ### Why are the changes needed?
   
   Fix https://github.com/apache/incubator-uniffle/issues/1572 & https://github.com/apache/incubator-uniffle/issues/1560 & https://github.com/apache/incubator-uniffle/issues/1542
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Existing UTs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2000166575

   @jerqi @zuston Please rebuild this PR, thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-1996253083

   I want to know that if the flush event has been pushed into handler queue before the app is expired, what will happen when this event is handling and app has been expired? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#discussion_r1526651778


##########
server/src/main/java/org/apache/uniffle/server/ShuffleTaskManager.java:
##########
@@ -692,35 +702,42 @@ private boolean isAppExpired(String appId) {
    * @param shuffleIds
    */
   public void removeResourcesByShuffleIds(String appId, List<Integer> shuffleIds) {
-    if (CollectionUtils.isEmpty(shuffleIds)) {
-      return;
-    }
+    Lock writeLock = getAppWriteLock(appId);
+    writeLock.lock();
+    try {
+      if (CollectionUtils.isEmpty(shuffleIds)) {
+        return;
+      }
 
-    LOG.info("Start remove resource for appId[{}], shuffleIds[{}]", appId, shuffleIds);
-    final long start = System.currentTimeMillis();
-    final ShuffleTaskInfo taskInfo = shuffleTaskInfos.get(appId);
-    if (taskInfo != null) {
-      for (Integer shuffleId : shuffleIds) {
-        taskInfo.getCachedBlockIds().remove(shuffleId);
-        taskInfo.getCommitCounts().remove(shuffleId);
-        taskInfo.getCommitLocks().remove(shuffleId);
+      LOG.info("Start remove resource for appId[{}], shuffleIds[{}]", appId, shuffleIds);
+      final long start = System.currentTimeMillis();
+      final ShuffleTaskInfo taskInfo = shuffleTaskInfos.get(appId);

Review Comment:
   > I think we need to limit the lock scope here.
   > 
   > `writeLock.lock()` should be started here rather than from the beginning of the method.
   
   Cleaning up resources is not a frequent operation. It only happens when an `appId` expires and is removed, or when unregistering a `shuffleId`. Therefore, I don't think there's a need to narrow the scope of the lock, as it won't cause any performance bottlenecks. And the code will be more readable. 
   Also, we have conducted performance stress tests on this PR and it does not impact performance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2000182087

   This PR should be merged ASAP, because this issue is a deeply hidden and serious problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-1993962299

   @jerqi @zuston PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2000160460

   > I want to know that if the flush event has been pushed into handler queue before the app is expired, what will happen when this event is handling but app has been expired?
   
   Before this PR:
   1. A `ShuffleBuffer` is turned into a `FlushEvent`, and its blocks and size are cleared
   ->
   2. The `FlushEvent` is added to the flushing queue
   ->
   3. The method `removeBufferByShuffleId` is executed, which causes the following things to happen:
   
   3.1. Running the following code snippet, but please note that in the code below, `buffer.getBlocks()` is empty and size is 0, because of the step 1 above:
   ```
   for (ShuffleBuffer buffer : buffers) {
     buffer.getBlocks().forEach(spb -> spb.getData().release());
     ShuffleServerMetrics.gaugeTotalPartitionNum.dec();
     size += buffer.getSize();
   }
   ```
   
   3.2. `appId` is removed from the `bufferPool`
       
   ->
   4. The `FlushEvent` is taken out from the queue and encounters an `EventInvalidException` because the `appId` was removed before
   ->
   5. When handling the `EventInvalidException`, nothing is done and the `event.doCleanup()` method is not called, causing a memory leak.
   Of course, this is just one scenario of concurrency exceptions. In the previous code, without locking, in the `processFlushEvent` method, it is possible that the event may become invalid at any time when continuing executing in `processFlushEvent` method, which is why there is https://github.com/apache/incubator-uniffle/issues/1542. Also, there is https://github.com/apache/incubator-uniffle/issues/1560.
   
   After this PR:
   We will set a read lock for steps 1 and 2 above, a write lock for step 3, a read lock for step 4, and when encountering an `EventInvalidException` in step 5, we will call the `event.doCleanup()` method to release the memory.
   
   In this way, we can ensure the following things when resources are being cleaned up:
   1. `ShuffleBuffers` that have not yet been converted to `FlushEvents` will not be converted in the future, but will be directly cleaned up.
   2. `FlushEvents` that have been converted from `ShuffleBuffers` will definitely encounter an `EventInvalidException`, and we will eventually handle this exception correctly, releasing memory.
   3. If there is already a `FlushEvent` being processed and it is about to be flushed to disk, the resource cleanup task will wait for all `FlushEvents` related to the `appId` to be completed before starting the cleanup task, ensuring that the cleanup and flushing tasks are completely independent and do not interfere with each other.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2001851361

   > > I want to know that if the flush event has been pushed into handler queue before the app is expired, what will happen when this event is handling but app has been expired?
   > 
   > To make it clear for everyone, I will try to explain this issue in detail.
   > 
   > Before this PR:
   > 
   > 1. A `ShuffleBuffer` is turned into a `FlushEvent`, and **_its blocks and size are cleared_**
   >    →
   > 2. The `FlushEvent` is added to the flushing queue
   >    →
   > 3. The method `removeBufferByShuffleId` is executed, which causes the following things to happen:
   > 
   > 3.1. Running the following code snippet, but please note that in the code below, `buffer.getBlocks()` **_is empty and size is 0_**, because of the step 1 above:
   > 
   > ```
   > for (ShuffleBuffer buffer : buffers) {
   >   buffer.getBlocks().forEach(spb -> spb.getData().release());
   >   ShuffleServerMetrics.gaugeTotalPartitionNum.dec();
   >   size += buffer.getSize();
   > }
   > ```
   > 
   > 3.2. `appId` is removed from the `bufferPool` → 4. The `FlushEvent` is taken out from the queue and encounters an `EventInvalidException` because the `appId` was removed before → 5. When handling the `EventInvalidException`, nothing is done and the `event.doCleanup()` method **_is not called, causing a memory leak_**. Of course, this is just one scenario of concurrency exceptions. In the previous code, without locking, in the `processFlushEvent` method, it is possible that the event may become invalid at any time when continuing executing in `processFlushEvent` method, which is why there is #1542. Also, there is #1560.
   > 
   > After this PR: We will set a read lock for steps 1 and 2 above, a write lock for step 3, a read lock for step 4, and when encountering an `EventInvalidException` in step 5, we will call the `event.doCleanup()` method to release the memory.
   > 
   > In this way, we can ensure the following things when resources are being cleaned up:
   > 
   > 1. `ShuffleBuffers` that have not yet been converted to `FlushEvents` will not be converted in the future, but will be directly cleaned up.
   > 2. `FlushEvents` that have been converted from `ShuffleBuffers` will definitely encounter an `EventInvalidException`, and we will eventually handle this exception correctly, releasing memory.
   > 3. If there is already a `FlushEvent` being processed and it is about to be flushed to disk, the resource cleanup task will wait for all `FlushEvents` related to the `appId` to be completed before starting the cleanup task, ensuring that the cleanup and flushing tasks are completely independent and do not interfere with each other.
   
   @leslizhang Could you update this to the description of pull request? Could you add a ut for this case?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#discussion_r1527908608


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataFlushEvent.java:
##########
@@ -40,6 +40,7 @@ public class ShuffleDataFlushEvent {
   private final long size;
   private final List<ShufflePartitionedBlock> shuffleBlocks;
   private final Supplier<Boolean> valid;
+

Review Comment:
   remove this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#discussion_r1527766916


##########
server/src/main/java/org/apache/uniffle/server/ShuffleTaskManager.java:
##########
@@ -692,35 +702,42 @@ private boolean isAppExpired(String appId) {
    * @param shuffleIds
    */
   public void removeResourcesByShuffleIds(String appId, List<Integer> shuffleIds) {
-    if (CollectionUtils.isEmpty(shuffleIds)) {
-      return;
-    }
+    Lock writeLock = getAppWriteLock(appId);
+    writeLock.lock();
+    try {
+      if (CollectionUtils.isEmpty(shuffleIds)) {
+        return;
+      }
 
-    LOG.info("Start remove resource for appId[{}], shuffleIds[{}]", appId, shuffleIds);
-    final long start = System.currentTimeMillis();
-    final ShuffleTaskInfo taskInfo = shuffleTaskInfos.get(appId);
-    if (taskInfo != null) {
-      for (Integer shuffleId : shuffleIds) {
-        taskInfo.getCachedBlockIds().remove(shuffleId);
-        taskInfo.getCommitCounts().remove(shuffleId);
-        taskInfo.getCommitLocks().remove(shuffleId);
+      LOG.info("Start remove resource for appId[{}], shuffleIds[{}]", appId, shuffleIds);
+      final long start = System.currentTimeMillis();
+      final ShuffleTaskInfo taskInfo = shuffleTaskInfos.get(appId);

Review Comment:
   > Therefore, I don't think there's a need to narrow the scope of the lock, as it won't cause any performance bottlenecks.
   
   I don't think the rational for not narrowing down the lock scope is that it won't cause any performance issues.  Narrowing down the lock scope usually helps reduces potential deadlock issues, for example, when the lock scope is long enough, it's more possible that another operation(with locking) is added by later developers without knowing it would cause dead locks. Therefore, it's considered best practice to always narrow down lock scope.
   
   For this particular case though, other methods already have long lock scope, which should be addressed later. I'm ok to leave it as it is for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2003048966

   @zuston @advancedxy I think we can merge this for now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#discussion_r1526660607


##########
server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java:
##########
@@ -291,23 +293,37 @@ protected void flushBuffer(
       int startPartition,
       int endPartition,
       boolean isHugePartition) {
-    ShuffleDataFlushEvent event =
-        buffer.toFlushEvent(
-            appId,
-            shuffleId,
-            startPartition,
-            endPartition,
-            () -> bufferPool.containsKey(appId),
-            shuffleFlushManager.getDataDistributionType(appId));
-    if (event != null) {
-      event.addCleanupCallback(() -> releaseMemory(event.getSize(), true, false));
-      updateShuffleSize(appId, shuffleId, -event.getSize());
-      inFlushSize.addAndGet(event.getSize());
-      if (isHugePartition) {
-        event.markOwnedByHugePartition();
+
+    ReentrantReadWriteLock.ReadLock readLock = shuffleTaskManager.getAppReadLock(appId);
+    readLock.lock();
+    if (!bufferPool.getOrDefault(appId, new HashMap<>()).containsKey(shuffleId)) {
+      LOG.info(
+          "Shuffle[{}] for app[{}] has already been removed, no need to flush the buffer",
+          shuffleId,
+          appId);
+      return;
+    }
+    try {
+      ShuffleDataFlushEvent event =
+          buffer.toFlushEvent(
+              appId,
+              shuffleId,
+              startPartition,
+              endPartition,
+              shuffleFlushManager.getDataDistributionType(appId));
+      if (event != null) {
+        event.setReadLock(readLock);

Review Comment:
   The codehas been refactored, no need to do this anymore.



##########
server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java:
##########
@@ -291,23 +293,37 @@ protected void flushBuffer(
       int startPartition,
       int endPartition,
       boolean isHugePartition) {
-    ShuffleDataFlushEvent event =
-        buffer.toFlushEvent(
-            appId,
-            shuffleId,
-            startPartition,
-            endPartition,
-            () -> bufferPool.containsKey(appId),
-            shuffleFlushManager.getDataDistributionType(appId));
-    if (event != null) {
-      event.addCleanupCallback(() -> releaseMemory(event.getSize(), true, false));
-      updateShuffleSize(appId, shuffleId, -event.getSize());
-      inFlushSize.addAndGet(event.getSize());
-      if (isHugePartition) {
-        event.markOwnedByHugePartition();
+
+    ReentrantReadWriteLock.ReadLock readLock = shuffleTaskManager.getAppReadLock(appId);
+    readLock.lock();
+    if (!bufferPool.getOrDefault(appId, new HashMap<>()).containsKey(shuffleId)) {
+      LOG.info(
+          "Shuffle[{}] for app[{}] has already been removed, no need to flush the buffer",
+          shuffleId,
+          appId);
+      return;
+    }
+    try {
+      ShuffleDataFlushEvent event =
+          buffer.toFlushEvent(
+              appId,
+              shuffleId,
+              startPartition,
+              endPartition,
+              shuffleFlushManager.getDataDistributionType(appId));
+      if (event != null) {
+        event.setReadLock(readLock);

Review Comment:
   The code has been refactored, no need to do this anymore.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#discussion_r1524156846


##########
server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java:
##########
@@ -291,23 +293,37 @@ protected void flushBuffer(
       int startPartition,
       int endPartition,
       boolean isHugePartition) {
-    ShuffleDataFlushEvent event =
-        buffer.toFlushEvent(
-            appId,
-            shuffleId,
-            startPartition,
-            endPartition,
-            () -> bufferPool.containsKey(appId),
-            shuffleFlushManager.getDataDistributionType(appId));
-    if (event != null) {
-      event.addCleanupCallback(() -> releaseMemory(event.getSize(), true, false));
-      updateShuffleSize(appId, shuffleId, -event.getSize());
-      inFlushSize.addAndGet(event.getSize());
-      if (isHugePartition) {
-        event.markOwnedByHugePartition();
+
+    ReentrantReadWriteLock.ReadLock readLock = shuffleTaskManager.getAppReadLock(appId);
+    readLock.lock();

Review Comment:
   ditto..



##########
server/src/main/java/org/apache/uniffle/server/ShuffleTaskManager.java:
##########
@@ -692,35 +702,42 @@ private boolean isAppExpired(String appId) {
    * @param shuffleIds
    */
   public void removeResourcesByShuffleIds(String appId, List<Integer> shuffleIds) {
-    if (CollectionUtils.isEmpty(shuffleIds)) {
-      return;
-    }
+    Lock writeLock = getAppWriteLock(appId);
+    writeLock.lock();
+    try {
+      if (CollectionUtils.isEmpty(shuffleIds)) {
+        return;
+      }
 
-    LOG.info("Start remove resource for appId[{}], shuffleIds[{}]", appId, shuffleIds);
-    final long start = System.currentTimeMillis();
-    final ShuffleTaskInfo taskInfo = shuffleTaskInfos.get(appId);
-    if (taskInfo != null) {
-      for (Integer shuffleId : shuffleIds) {
-        taskInfo.getCachedBlockIds().remove(shuffleId);
-        taskInfo.getCommitCounts().remove(shuffleId);
-        taskInfo.getCommitLocks().remove(shuffleId);
+      LOG.info("Start remove resource for appId[{}], shuffleIds[{}]", appId, shuffleIds);
+      final long start = System.currentTimeMillis();
+      final ShuffleTaskInfo taskInfo = shuffleTaskInfos.get(appId);

Review Comment:
   I think we need to limit the lock scope here.
   
   `writeLock.lock()` should be started here rather than from the beginning of the method.



##########
server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java:
##########
@@ -291,23 +293,37 @@ protected void flushBuffer(
       int startPartition,
       int endPartition,
       boolean isHugePartition) {
-    ShuffleDataFlushEvent event =
-        buffer.toFlushEvent(
-            appId,
-            shuffleId,
-            startPartition,
-            endPartition,
-            () -> bufferPool.containsKey(appId),
-            shuffleFlushManager.getDataDistributionType(appId));
-    if (event != null) {
-      event.addCleanupCallback(() -> releaseMemory(event.getSize(), true, false));
-      updateShuffleSize(appId, shuffleId, -event.getSize());
-      inFlushSize.addAndGet(event.getSize());
-      if (isHugePartition) {
-        event.markOwnedByHugePartition();
+
+    ReentrantReadWriteLock.ReadLock readLock = shuffleTaskManager.getAppReadLock(appId);
+    readLock.lock();
+    if (!bufferPool.getOrDefault(appId, new HashMap<>()).containsKey(shuffleId)) {
+      LOG.info(
+          "Shuffle[{}] for app[{}] has already been removed, no need to flush the buffer",
+          shuffleId,
+          appId);
+      return;
+    }
+    try {
+      ShuffleDataFlushEvent event =
+          buffer.toFlushEvent(
+              appId,
+              shuffleId,
+              startPartition,
+              endPartition,
+              shuffleFlushManager.getDataDistributionType(appId));
+      if (event != null) {
+        event.setReadLock(readLock);

Review Comment:
   This doesn't seem idiomatic.
   A more appropriate way would be that passing the read lock to the constructor of `ShuffleDataFlushEvent` and the readLock shall never be changed after construction.
   
   The lock logic could be hide behind the `ShuffleDataFlushEvent` internal processing logic or exposed by:
   ```java
   Class ShuffleDataFlushEvent {
       <T> T withReadLock(Supplier<T> action) {
       readLock.lock();
       try {
         return action.get();
       } finally {
         readLock.unlock();
       }
     }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2003088043

   > This is not a consistently reproducible issue, so it is difficult to conduct positive and negative tests. We have already run more than 10 20TB TPCH SQL tests in our stress test environment without encountering any problems. Before fixing this issue, the frequency of its occurrence was relatively high. @advancedxy
   
   Got your point, but unit test could mock this problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "leslizhang (via GitHub)" <gi...@apache.org>.
leslizhang commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2003031457

   > > I don't think we need to add new UTs. Because in this PR, we have already added locks into existing UTs. If anything goes wrong, we will know it.
   > 
   > > For UTs, the logic of read-write lock synchronization has been added to all previous unit tests(ShuffleBufferManagerTest,ShuffleFlushManagerTest, etc.) for caching block or flushing block. That is to say, the unit testing of the PR is implemented by adapting and modifying the previous unit test code.
   > 
   > Hmmm, I think the new unit test cases means that is it possible to adding new UTs to demonstrate the issues you described in your PR? Since concurrency issues are happened in production env, they indicates that the existing UTs doesn't cover the concurrency issues.
   > 
   > If it's too hard or too complicated to add cases like that, you can simply state that and I think we are fine to merge it for now. Otherwise, the existing test cases cannot prove you have indeed fixed the concurrency issues.
   
   This is not a consistently reproducible issue, so it is difficult to conduct positive and negative tests. We have already run more than 10 20TB TPCH SQL tests in our stress test environment without encountering any problems. Before fixing this issue, the frequency of its occurrence was relatively high. @advancedxy 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "leslizhang (via GitHub)" <gi...@apache.org>.
leslizhang commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2001923704

   > > > I want to know that if the flush event has been pushed into handler queue before the app is expired, what will happen when this event is handling but app has been expired?
   > > 
   > > 
   > > To make it clear for everyone, I will try to explain this issue in detail.
   > > Before this PR:
   > > 
   > > 1. A `ShuffleBuffer` is turned into a `FlushEvent`, and **_its blocks and size are cleared_**
   > >    →
   > > 2. The `FlushEvent` is added to the flushing queue
   > >    →
   > > 3. The method `removeBufferByShuffleId` is executed, which causes the following things to happen:
   > > 
   > > 3.1. Running the following code snippet, but please note that in the code below, `buffer.getBlocks()` **_is empty and size is 0_**, because of the step 1 above:
   > > ```
   > > for (ShuffleBuffer buffer : buffers) {
   > >   buffer.getBlocks().forEach(spb -> spb.getData().release());
   > >   ShuffleServerMetrics.gaugeTotalPartitionNum.dec();
   > >   size += buffer.getSize();
   > > }
   > > ```
   > > 
   > > 
   > >     
   > >       
   > >     
   > > 
   > >       
   > >     
   > > 
   > >     
   > >   
   > > 3.2. `appId` is removed from the `bufferPool` → 4. The `FlushEvent` is taken out from the queue and encounters an `EventInvalidException` because the `appId` was removed before → 5. When handling the `EventInvalidException`, nothing is done and the `event.doCleanup()` method **_is not called, causing a memory leak_**. Of course, this is just one scenario of concurrency exceptions. In the previous code, without locking, in the `processFlushEvent` method, it is possible that the event may become invalid at any time when continuing executing in `processFlushEvent` method, which is why there is #1542. Also, there is #1560.
   > > After this PR: We will set a read lock for steps 1 and 2 above, a write lock for step 3, a read lock for step 4, and when encountering an `EventInvalidException` in step 5, we will call the `event.doCleanup()` method to release the memory.
   > > In this way, we can ensure the following things when resources are being cleaned up:
   > > 
   > > 1. `ShuffleBuffers` that have not yet been converted to `FlushEvents` will not be converted in the future, but will be directly cleaned up.
   > > 2. `FlushEvents` that have been converted from `ShuffleBuffers` will definitely encounter an `EventInvalidException`, and we will eventually handle this exception correctly, releasing memory.
   > > 3. If there is already a `FlushEvent` being processed and it is about to be flushed to disk, the resource cleanup task will wait for all `FlushEvents` related to the `appId` to be completed before starting the cleanup task, ensuring that the cleanup and flushing tasks are completely independent and do not interfere with each other.
   > 
   > @leslizhang Could you update this to the description of pull request? Could you add a ut for this case?
   
   Thank you for the suggestion.
   > Could you update this to the description of pull request?
    done
   
   > Could you add a ut for this case?
   A bit more suggestion:
   The essence of this PR is to add lock control to the read and write operations of blocks in the shuffle server memory: tasks to delete blocks add write locks (considered as write operations on blocks), and tasks to flush blocks (including the conversion of blocks to events, and the flushing operation of generated events) add read locks. This PR does not change the processing flow of block lifecycle. For UTs, the logic of read-write lock synchronization has been added to all previous unit tests(ShuffleBufferManagerTest,ShuffleFlushManagerTest, etc.) for caching block or flushing block. That is to say, the unit testing of the PR is implemented by adapting and modifying the previous unit test code.
   thx again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-1994094552

   ## Test Results
    2 340 files  ±0   2 340 suites  ±0   6h 19m 14s :stopwatch: + 1h 48m 15s
      904 tests ±0     884 :white_check_mark:  -  18   2 :zzz: ±0   10 :x: + 10    8 :fire: +  8 
   10 505 runs  ±0  10 230 :white_check_mark:  - 252  23 :zzz: ±0  140 :x: +140  112 :fire: +112 
   
   For more details on these failures and errors, see [this check](https://github.com/apache/incubator-uniffle/runs/22604715307).
   
   Results for commit e3f730ac. ± Comparison against base commit 9737d579.
   
   [test-results]:data:application/gzip;base64,H4sIAJKE8WUC/03NSQ7DIAwF0KtErLuwgRToZSpEsISaoWJYVb17oQooy/9sf38YhdUn9pi4kHCbWCohX/NSos3h2JtwNctKdZ7bhoGRnqk4V0nrC73Cu10NIBvWCghDfIxHbGdVYtnTfzrDfMbeisAFDDtrRYdeK8dKr0Vsz92xbSHXyLwgJcA6rixpQLhrZxb0xEkZYQittgso1Oz7A+yyE5sXAQAA
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2002883564

   > I don't think we need to add new UTs. Because in this PR, we have already added locks into existing UTs. If anything goes wrong, we will know it.
   
   > For UTs, the logic of read-write lock synchronization has been added to all previous unit tests(ShuffleBufferManagerTest,ShuffleFlushManagerTest, etc.) for caching block or flushing block. That is to say, the unit testing of the PR is implemented by adapting and modifying the previous unit test code.
   
   Hmmm, I think the new unit test cases means that is it possible to adding new UTs to demonstrate the issues you described in your PR? Since concurrency issues are happened in production env, they indicates that the existing UTs doesn't cover the concurrency issues. 
   
   If it's too hard or too complicated to add cases like that, you can simply state that and I think we are fine to merge it for now. Otherwise, the existing test cases cannot prove you have indeed fixed the concurrency issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "leslizhang (via GitHub)" <gi...@apache.org>.
leslizhang commented on code in PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#discussion_r1527929156


##########
server/src/main/java/org/apache/uniffle/server/ShuffleDataFlushEvent.java:
##########
@@ -40,6 +40,7 @@ public class ShuffleDataFlushEvent {
   private final long size;
   private final List<ShufflePartitionedBlock> shuffleBlocks;
   private final Supplier<Boolean> valid;
+

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2000165502

   @advancedxy PTAL again, I don't think we need to add new UTs. Because in this PR, we have already added locks into existing UTs. If anything goes wrong, we will know it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2001996224

   OK for me. Let others take another look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2002764371

   ping @zuston @advancedxy 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-2003089837

   > @zuston @advancedxy I think we can merge this for now.
   
   If you insist, I will merge this before completing the unit tests. But I hope the test cases could be added later in another PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1571] fix(server): Memory may leak when EventInvalidException occurs [incubator-uniffle]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1574:
URL: https://github.com/apache/incubator-uniffle/pull/1574#issuecomment-1999686284

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1574?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `83.33333%` with `10 lines` in your changes are missing coverage. Please review.
   > Project coverage is 54.01%. Comparing base [(`9737d57`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/9737d5790218cfc9c8be6439fd48bb7af98430c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`7d6c01c`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1574?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 5 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1574?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../org/apache/uniffle/server/ShuffleTaskManager.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1574?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9TaHVmZmxlVGFza01hbmFnZXIuamF2YQ==) | 81.81% | [4 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1574?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...he/uniffle/server/buffer/ShuffleBufferManager.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1574?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9idWZmZXIvU2h1ZmZsZUJ1ZmZlck1hbmFnZXIuamF2YQ==) | 78.94% | [3 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1574?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1574      +/-   ##
   ============================================
   + Coverage     53.96%   54.01%   +0.05%     
   + Complexity     2858     2291     -567     
   ============================================
     Files           438      360      -78     
     Lines         24793    16085    -8708     
     Branches       2109     1467     -642     
   ============================================
   - Hits          13379     8688    -4691     
   + Misses        10576     6892    -3684     
   + Partials        838      505     -333     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1574?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org