You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "FMX (via GitHub)" <gi...@apache.org> on 2024/10/16 07:01:59 UTC
[PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
FMX opened a new pull request, #2815:
URL: https://github.com/apache/celeborn/pull/2815
### What changes were proposed in this pull request?
The read buffer dispatcher may lose its dispatcher thread which is not acceptable.
### Why are the changes needed?
1. Add a scheduler pool to ensure the dispatcher thread is alive.
2. Add an unhandled exception handler to record possible exceptions that cause the thread to be lost.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
Cluster test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805787385
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.worker.readBuffer.checkThread.interval")
Review Comment:
```suggestion
buildConf("celeborn.worker.readBufferDispatcherThreadWatchdog.checkInterval")
```
with doc "When configured, periodically check if the read buffer dispatcher thread is alive, if dead, relaunch a new one."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804623101
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -132,4 +106,70 @@ public void close() {
stopFlag = true;
requests.clear();
}
+
+ private class DispatcherRunnable implements Runnable {
+ private final MemoryManager memoryManager;
+
+ public DispatcherRunnable(MemoryManager memoryManager) {
+ this.memoryManager = memoryManager;
+ }
+
+ @Override
+ public void run() {
+ while (!stopFlag) {
+ try {
+ ReadBufferRequest request = null;
+ request = requests.poll(1000, TimeUnit.MILLISECONDS);
+ List<ByteBuf> buffers = null;
+ try {
+ if (request != null) {
+ long start = System.nanoTime();
+ int bufferSize = request.getBufferSize();
+ buffers = new ArrayList<>();
+ while (buffers.size() < request.getNumber()) {
+ if (memoryManager.readBufferAvailable(bufferSize)) {
+ ByteBuf buf = readBufferAllocator.buffer(bufferSize, bufferSize);
+ buffers.add(buf);
+ memoryManager.changeReadBufferCounter(bufferSize);
+ allocatedReadBuffers.increment();
+ } else {
+ try {
+ // If dispatcher can not allocate requested buffers, it will wait here until
+ // necessary
+ // buffers are get.
Review Comment:
nit: merge into previous line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805788335
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.worker.readBuffer.checkThread.interval")
Review Comment:
and this should be disabled by default. if user hits readBuffer DispatcherThread unexpectedly exit issues and can not be identified through logs, we can guide the user to enable it as a workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1802983473
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +33,50 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
private volatile boolean stopFlag = false;
+ private final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
this.readBufferAllocationWait = conf.readBufferAllocationWait();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ReadBufferDispatcherUnhandledExceptionHandler());
Review Comment:
new org.apache.celeborn.common.util.ThreadExceptionHandler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805946048
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.worker.readBufferDispatcherThreadWatchdog.checkInterval")
Review Comment:
```suggestion
buildConf("celeborn.worker.readBuffer.checkThreadInterval")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805813608
##########
common/src/main/scala/org/apache/celeborn/common/util/ThreadUtils.scala:
##########
@@ -392,6 +392,20 @@ object ThreadUtils {
pool.shutdownNow()
}
}
+
+ def newThreadWithDefaultUncaughtExceptionHandler(runnable: Runnable, name: String): Thread = {
+ val thread = new Thread(runnable, name)
+ thread.setUncaughtExceptionHandler(new ThreadExceptionHandler(name))
+ thread
+ }
+
+ def newDeamonThreadWithDefaultUncaughtExceptionHandler(
+ runnable: Runnable,
+ name: String): Thread = {
Review Comment:
```suggestion
def newDeamonThread(name: String, runnable: Runnable): Thread = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805804504
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.worker.readBuffer.checkThread.interval")
Review Comment:
Sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805954832
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
Review Comment:
```suggestion
val WORKER_READBUFFER_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804627847
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +33,51 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
private volatile boolean stopFlag = false;
+ private final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
this.readBufferAllocationWait = conf.readBufferAllocationWait();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
Review Comment:
creating a thread with a custom name, then setting up UncaughtExceptionHandler is quite common logic, can we extract it into a method inside ThreadUtils?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805944587
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
Review Comment:
```suggestion
val WORKER_READBUFFER_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804344927
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -132,4 +107,73 @@ public void close() {
stopFlag = true;
requests.clear();
}
+
+ private class DispatcherRunnable implements Runnable {
+ private final MemoryManager memoryManager;
+
+ public DispatcherRunnable(MemoryManager memoryManager) {
+ this.memoryManager = memoryManager;
+ }
+
+ @Override
+ public void run() {
+ while (!stopFlag) {
+ try {
+ ReadBufferRequest request = null;
+ try {
+ request = requests.poll(1000, TimeUnit.MILLISECONDS);
Review Comment:
Yes, you are right. I'll fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805953249
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +34,52 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
- private volatile boolean stopFlag = false;
+ @VisibleForTesting public volatile boolean stopFlag = false;
+ @VisibleForTesting public final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
- this.readBufferAllocationWait = conf.readBufferAllocationWait();
+ readBufferAllocationWait = conf.readBufferAllocationWait();
+ long checkThreadInterval = conf.readBufferDispatcherCheckThreadInterval();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ ThreadUtils.newThread(new DispatcherRunnable(), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+
+ if (checkThreadInterval > 0) {
+ ScheduledExecutorService checkAliveThread =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("ReadBufferDispatcherChecker");
+ checkAliveThread.scheduleWithFixedDelay(
+ () -> {
+ if (!dispatcherThread.get().isAlive()) {
+ dispatcherThread.set(
+ ThreadUtils.newThread(new DispatcherRunnable(), "ReadBufferDispatcher"));
Review Comment:
```suggestion
ThreadUtils.newThread(new DispatcherRunnable(), "read-buffer-dispatcher"));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804426858
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +33,51 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
private volatile boolean stopFlag = false;
+ private final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
this.readBufferAllocationWait = conf.readBufferAllocationWait();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
Review Comment:
Yes. But I found some thread lost for an unknown reason so I just want to ensure that I can record all possible incidents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804633465
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -132,4 +106,70 @@ public void close() {
stopFlag = true;
requests.clear();
}
+
+ private class DispatcherRunnable implements Runnable {
+ private final MemoryManager memoryManager;
+
+ public DispatcherRunnable(MemoryManager memoryManager) {
+ this.memoryManager = memoryManager;
+ }
+
+ @Override
+ public void run() {
+ while (!stopFlag) {
+ try {
+ ReadBufferRequest request = null;
Review Comment:
maybe
```
@Override
public void run() {
while (!stopFlag) {
try {
call another_method
} catch catch (Throwable e) {
logger.error("Read buffer dispatcher encountered error: {}", e.getMessage(), e);
}
}
}
void another_method() {
balabala
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804428077
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -132,4 +107,73 @@ public void close() {
stopFlag = true;
requests.clear();
}
+
+ private class DispatcherRunnable implements Runnable {
+ private final MemoryManager memoryManager;
+
+ public DispatcherRunnable(MemoryManager memoryManager) {
+ this.memoryManager = memoryManager;
+ }
+
+ @Override
+ public void run() {
+ while (!stopFlag) {
+ try {
+ ReadBufferRequest request = null;
+ try {
+ request = requests.poll(1000, TimeUnit.MILLISECONDS);
Review Comment:
@codenohup Thanks for the point. I think I fixed it. You can check again when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805980229
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.worker.readBufferDispatcherThreadWatchdog.checkInterval")
+ .categories("worker")
+ .version("0.5.2")
+ .internal
Review Comment:
This check is not necessary after we told the users to enable it. Because after this PR, this thread should not exit unexpectedly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805812792
##########
common/src/main/scala/org/apache/celeborn/common/util/ThreadUtils.scala:
##########
@@ -392,6 +392,20 @@ object ThreadUtils {
pool.shutdownNow()
}
}
+
+ def newThreadWithDefaultUncaughtExceptionHandler(runnable: Runnable, name: String): Thread = {
Review Comment:
```suggestion
def newThread(runnable: Runnable, name: String): Thread = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804280699
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +33,51 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
private volatile boolean stopFlag = false;
+ private final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
this.readBufferAllocationWait = conf.readBufferAllocationWait();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+
+ ScheduledExecutorService checkAliveThread =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("ReadBufferDispatcherChecker");
+ checkAliveThread.scheduleWithFixedDelay(
+ new Thread(
Review Comment:
Is there any reason to use Thread instead of Runnable/lambda here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805952998
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +34,52 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
- private volatile boolean stopFlag = false;
+ @VisibleForTesting public volatile boolean stopFlag = false;
+ @VisibleForTesting public final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
- this.readBufferAllocationWait = conf.readBufferAllocationWait();
+ readBufferAllocationWait = conf.readBufferAllocationWait();
+ long checkThreadInterval = conf.readBufferDispatcherCheckThreadInterval();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ ThreadUtils.newThread(new DispatcherRunnable(), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+
+ if (checkThreadInterval > 0) {
+ ScheduledExecutorService checkAliveThread =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("ReadBufferDispatcherChecker");
Review Comment:
```suggestion
ThreadUtils.newDaemonSingleThreadScheduledExecutor("read-buffer-dispatcher-checker");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805945327
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.worker.readBufferDispatcherThreadWatchdog.checkInterval")
+ .categories("worker")
+ .version("0.5.2")
+ .internal
Review Comment:
Why is this config internal? `celeborn.worker.readBuffer.allocationWait` is not internal, therefore IMO this could also be not internal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804621282
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +33,51 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
private volatile boolean stopFlag = false;
+ private final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
this.readBufferAllocationWait = conf.readBufferAllocationWait();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+
+ ScheduledExecutorService checkAliveThread =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("ReadBufferDispatcherChecker");
+ checkAliveThread.scheduleWithFixedDelay(
+ new Thread(
+ () -> {
+ if (!dispatcherThread.get().isAlive()) {
+ dispatcherThread.set(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(
+ new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+ }
+ }),
+ 50,
Review Comment:
this is more like a diagnostic approach rather than a regular logic that should run in production. for this case, we can temporarily introduce an `internal` conf.
-the `internal` conf won't be exposed to the end user, and can be removed anytime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804346374
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +33,51 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
private volatile boolean stopFlag = false;
+ private final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
this.readBufferAllocationWait = conf.readBufferAllocationWait();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+
+ ScheduledExecutorService checkAliveThread =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("ReadBufferDispatcherChecker");
+ checkAliveThread.scheduleWithFixedDelay(
+ new Thread(
+ () -> {
+ if (!dispatcherThread.get().isAlive()) {
+ dispatcherThread.set(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(
+ new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+ }
+ }),
+ 50,
Review Comment:
IMO, 50 millis check interval won`t introduce much CPU pressure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805870111
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +34,52 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
- private volatile boolean stopFlag = false;
+ @VisibleForTesting public volatile boolean stopFlag = false;
+ @VisibleForTesting public final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
- this.readBufferAllocationWait = conf.readBufferAllocationWait();
+ readBufferAllocationWait = conf.readBufferAllocationWait();
+ long checkThreadInterval = conf.readBufferDispatcherCheckThreadInterval();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ ThreadUtils.newThread(new DispatcherRunnable(), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+
+ ScheduledExecutorService checkAliveThread =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("ReadBufferDispatcherChecker");
Review Comment:
should this also be moved inside `if (checkThreadInterval > 0) { }`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#issuecomment-2421756730
Thanks, merge to main(v0.6.0) and branch-0.5(v0.5.2)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong closed pull request #2815: [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly
URL: https://github.com/apache/celeborn/pull/2815
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805889723
##########
common/src/main/scala/org/apache/celeborn/common/util/ThreadUtils.scala:
##########
@@ -392,6 +392,20 @@ object ThreadUtils {
pool.shutdownNow()
}
}
+
+ def newThread(runnable: Runnable, name: String): Thread = {
+ val thread = new Thread(runnable, name)
+ thread.setUncaughtExceptionHandler(new ThreadExceptionHandler(name))
+ thread
+ }
+
+ def newDeamonThread(
+ runnable: Runnable,
+ name: String): Thread = {
Review Comment:
```suggestion
def newDeamonThread(runnable: Runnable, name: String): Thread = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "codenohup (via GitHub)" <gi...@apache.org>.
codenohup commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1804302634
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +33,51 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
private volatile boolean stopFlag = false;
+ private final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
this.readBufferAllocationWait = conf.readBufferAllocationWait();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+
+ ScheduledExecutorService checkAliveThread =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("ReadBufferDispatcherChecker");
+ checkAliveThread.scheduleWithFixedDelay(
+ new Thread(
Review Comment:
Runnable?
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -132,4 +107,73 @@ public void close() {
stopFlag = true;
requests.clear();
}
+
+ private class DispatcherRunnable implements Runnable {
+ private final MemoryManager memoryManager;
+
+ public DispatcherRunnable(MemoryManager memoryManager) {
+ this.memoryManager = memoryManager;
+ }
+
+ @Override
+ public void run() {
+ while (!stopFlag) {
+ try {
+ ReadBufferRequest request = null;
+ try {
+ request = requests.poll(1000, TimeUnit.MILLISECONDS);
Review Comment:
If the DispatcherRunnable thread has polled a request, and is terminated before notifying the listener about the request, the request may be lost, potentially resulting in the reader waiting indefinitely.
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +33,51 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
private volatile boolean stopFlag = false;
+ private final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
this.readBufferAllocationWait = conf.readBufferAllocationWait();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
Review Comment:
The ThreadExceptionHandler may unnecessary, because the DispatcherRunnable#run will catch Throwable.
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -31,24 +33,51 @@
import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.network.util.NettyUtils;
import org.apache.celeborn.common.network.util.TransportConf;
+import org.apache.celeborn.common.util.ThreadExceptionHandler;
+import org.apache.celeborn.common.util.ThreadUtils;
-public class ReadBufferDispatcher extends Thread {
+public class ReadBufferDispatcher {
private final Logger logger = LoggerFactory.getLogger(ReadBufferDispatcher.class);
private final LinkedBlockingQueue<ReadBufferRequest> requests = new LinkedBlockingQueue<>();
private final MemoryManager memoryManager;
private final PooledByteBufAllocator readBufferAllocator;
private final LongAdder allocatedReadBuffers = new LongAdder();
private final long readBufferAllocationWait;
private volatile boolean stopFlag = false;
+ private final AtomicReference<Thread> dispatcherThread;
public ReadBufferDispatcher(MemoryManager memoryManager, CelebornConf conf) {
this.readBufferAllocationWait = conf.readBufferAllocationWait();
// readBuffer is not a module name, it's a placeholder.
readBufferAllocator =
NettyUtils.getPooledByteBufAllocator(new TransportConf("readBuffer", conf), null, true);
this.memoryManager = memoryManager;
- this.setName("Read-Buffer-Dispatcher");
- this.start();
+ dispatcherThread =
+ new AtomicReference<>(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+
+ ScheduledExecutorService checkAliveThread =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("ReadBufferDispatcherChecker");
+ checkAliveThread.scheduleWithFixedDelay(
+ new Thread(
+ () -> {
+ if (!dispatcherThread.get().isAlive()) {
+ dispatcherThread.set(
+ new Thread(new DispatcherRunnable(memoryManager), "ReadBufferDispatcher"));
+ dispatcherThread
+ .get()
+ .setUncaughtExceptionHandler(
+ new ThreadExceptionHandler("ReadBufferDispatcher"));
+ dispatcherThread.get().start();
+ }
+ }),
+ 50,
Review Comment:
Checking every 50 milliseconds may be too frequent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805945327
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.worker.readBufferDispatcherThreadWatchdog.checkInterval")
+ .categories("worker")
+ .version("0.5.2")
+ .internal
Review Comment:
Why is this config internal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805812792
##########
common/src/main/scala/org/apache/celeborn/common/util/ThreadUtils.scala:
##########
@@ -392,6 +392,20 @@ object ThreadUtils {
pool.shutdownNow()
}
}
+
+ def newThreadWithDefaultUncaughtExceptionHandler(runnable: Runnable, name: String): Thread = {
Review Comment:
```suggestion
def newThread(name: String, runnable: Runnable): Thread = {
```
##########
common/src/main/scala/org/apache/celeborn/common/util/ThreadUtils.scala:
##########
@@ -392,6 +392,20 @@ object ThreadUtils {
pool.shutdownNow()
}
}
+
+ def newThreadWithDefaultUncaughtExceptionHandler(runnable: Runnable, name: String): Thread = {
+ val thread = new Thread(runnable, name)
+ thread.setUncaughtExceptionHandler(new ThreadExceptionHandler(name))
+ thread
+ }
+
+ def newDeamonThreadWithDefaultUncaughtExceptionHandler(
Review Comment:
```suggestion
def newDeamonThread(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805812232
##########
common/src/main/scala/org/apache/celeborn/common/util/ThreadUtils.scala:
##########
@@ -392,6 +392,20 @@ object ThreadUtils {
pool.shutdownNow()
}
}
+
+ def newThreadWithDefaultUncaughtExceptionHandler(runnable: Runnable, name: String): Thread = {
+ val thread = new Thread(runnable, name)
+ thread.setUncaughtExceptionHandler(new ThreadExceptionHandler(name))
+ thread
+ }
+
+ def newDeamonThreadWithDefaultUncaughtExceptionHandler(
Review Comment:
```suggestion
def newDeamonThread(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805870912
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.worker.readBufferDispatcherThreadWatchdog.checkInterval")
+ .categories("worker")
+ .version("0.5.2")
+ .internal
+ .doc("The interval for worker to check read buffer dispatcher thread. 0 means disable.")
+ .timeConf(TimeUnit.MILLISECONDS)
+ .createWithDefaultString("50ms")
Review Comment:
I think we should disable it by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#issuecomment-2421121479
Ping @pan3793. Thanks for your review. Comments are addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805950379
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/ReadBufferDispatcher.java:
##########
@@ -132,4 +109,66 @@ public void close() {
stopFlag = true;
requests.clear();
}
+
+ private class DispatcherRunnable implements Runnable {
+
+ public DispatcherRunnable() {}
+
+ @Override
+ public void run() {
+ while (!stopFlag) {
+ try {
+ ReadBufferRequest request;
+ request = requests.poll(1000, TimeUnit.MILLISECONDS);
+ List<ByteBuf> buffers = new ArrayList<>();
+ try {
+ if (request != null) {
+ processBufferRequest(request, buffers);
+ } else {
+ // Free buffer pool memory to main direct memory when dispatcher is idle.
+ readBufferAllocator.trimCurrentThreadCache();
+ }
+ } catch (Throwable e) {
+ logger.error(e.getMessage(), e);
+ try {
+ // recycle all allocated buffers
+ for (ByteBuf buffer : buffers) {
+ recycle(buffer);
+ }
+ } catch (Throwable e1) {
+ logger.error("Recycle read buffer failed.", e1);
+ }
+ request.getBufferListener().notifyBuffers(null, e);
+ }
+ } catch (Throwable e) {
+ logger.error("Read buffer dispatcher encountered error: {}", e.getMessage(), e);
+ }
+ }
+ }
+
+ void processBufferRequest(ReadBufferRequest request, List<ByteBuf> buffers) {
+ long start = System.nanoTime();
+ int bufferSize = request.getBufferSize();
+ while (buffers.size() < request.getNumber()) {
+ if (memoryManager.readBufferAvailable(bufferSize)) {
+ ByteBuf buf = readBufferAllocator.buffer(bufferSize, bufferSize);
+ buffers.add(buf);
+ memoryManager.changeReadBufferCounter(bufferSize);
+ allocatedReadBuffers.increment();
+ } else {
+ try {
+ // If dispatcher can not allocate requested buffers, it will wait here until
+ // necessary buffers are get.
+ Thread.sleep(readBufferAllocationWait);
+ } catch (InterruptedException e) {
+ logger.info("Buffer dispatcher is closing");
Review Comment:
```suggestion
logger.warn("Buffer dispatcher is closing");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805944587
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
Review Comment:
```suggestion
val WORKER_READBUFFER_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2815:
URL: https://github.com/apache/celeborn/pull/2815#discussion_r1805975711
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4012,6 +4013,15 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(32)
+ val WORKER_READBUFFERS_CHECK_THREAD_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.worker.readBufferDispatcherThreadWatchdog.checkInterval")
Review Comment:
I think the config name "watchdog" is more understandable to the end users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org