You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uniffle.apache.org by zu...@apache.org on 2024/04/10 03:14:49 UTC
(incubator-uniffle) branch master updated: [#1631] fix(server): ShuffleTaskInfo may leak when app is removed. (#1632)
This is an automated email from the ASF dual-hosted git repository.
zuston pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
The following commit(s) were added to refs/heads/master by this push:
new 1431c8a1b [#1631] fix(server): ShuffleTaskInfo may leak when app is removed. (#1632)
1431c8a1b is described below
commit 1431c8a1b8b53ffd2b9ab45372416e9842eba944
Author: zhengchenyu <zh...@163.com>
AuthorDate: Wed Apr 10 11:14:44 2024 +0800
[#1631] fix(server): ShuffleTaskInfo may leak when app is removed. (#1632)
### What changes were proposed in this pull request?
In our cluster, delete pod is denied by web hook, even though all application is deleted for long time.
When I curl http://host:ip/metrics/server, I found app_num_with_node is 1.
The problem is some application is leaked. I see many duplicated logs `[INFO] ShuffleTaskManager.checkResourceStatus - Detect expired appId[appattempt_xxx_xx_xx] according to rss.server.app.expired.withoutHeartbeat`.
When I jstack the server many times, clearResourceThread will be stuck forever, here is the call stack.
```
"clearResourceThread" #40 daemon prio=5 os_prio=0 cpu=3767.63ms elapsed=5393.50s tid=0x00007f24fe92e800 nid=0x8f waiting on condition [0x00007f24f7b33000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.22/Native Method)
- parking to wait for <0x00007f28d5e29f20> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.22/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.22/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.22/AbstractQueuedSynchronizer.java:917)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.22/AbstractQueuedSynchronizer.java:1240)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.22/ReentrantReadWriteLock.java:959)
at org.apache.uniffle.server.ShuffleTaskManager.removeResources(ShuffleTaskManager.java:756)
at org.apache.uniffle.server.ShuffleTaskManager.lambda$new$0(ShuffleTaskManager.java:183)
at org.apache.uniffle.server.ShuffleTaskManager$$Lambda$216/0x00007f24f824cc40.run(Unknown Source)
at java.lang.Thread.run(java.base@11.0.22/Thread.java:829)
```
Apparently there's a lock that's not being released. Looking at the code, it's easy to see that the read lock in the flushBuffer is not released correctly. The log ` ShuffleBufferManager.flushBuffer - Shuffle[3066071] for app[appattempt_xxx] has already been removed, no need to flush the buffer` proved it.
Fix: #1631
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
no test, obvious mistake
---
.../apache/uniffle/server/buffer/ShuffleBufferManager.java | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java b/server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java
index 8f41a0795..35ef3fa09 100644
--- a/server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java
+++ b/server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java
@@ -298,14 +298,14 @@ public class ShuffleBufferManager {
boolean isHugePartition) {
ReentrantReadWriteLock.ReadLock readLock = shuffleTaskManager.getAppReadLock(appId);
readLock.lock();
- if (!bufferPool.getOrDefault(appId, new HashMap<>()).containsKey(shuffleId)) {
- LOG.info(
- "Shuffle[{}] for app[{}] has already been removed, no need to flush the buffer",
- shuffleId,
- appId);
- return;
- }
try {
+ if (!bufferPool.getOrDefault(appId, new HashMap<>()).containsKey(shuffleId)) {
+ LOG.info(
+ "Shuffle[{}] for app[{}] has already been removed, no need to flush the buffer",
+ shuffleId,
+ appId);
+ return;
+ }
ShuffleDataFlushEvent event =
buffer.toFlushEvent(
appId,