You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lucasbru (via GitHub)" <gi...@apache.org> on 2023/08/09 21:45:53 UTC
[GitHub] [kafka] lucasbru opened a new pull request, #14180: Kafka Streams Threading: Non-busy wait for processable tasks
lucasbru opened a new pull request, #14180:
URL: https://github.com/apache/kafka/pull/14180
Avoid busy waiting for processable tasks. We need to be a bit careful here to not have the task executors to sleep when work is available. We have to make sure to signal on the condition variable any time a task becomes "processable". Here are some situations where a task becomes processable:
- Task is unassigned from another `TaskExecutor`.
- Task state is changed (should only happen inside when a task is locked inside the polling phase).
- When tasks are unlocked.
- When tasks are added.
- New records available.
- A task is resumed.
So in summary, we
- We should probably lock tasks when they are paused and unlock them when they are resumed. We should also wake the task executors after every polling phase. This belongs to the StreamThread integration work (separate PR). We add `DefaultTaskManager.signalProcessableTasks` for this.
- We need to awake the task executors in `DefaultTaskManager.unassignTask`, `DefaultTaskManager.unlockTasks` and `DefaultTaskManager.add`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #14180: KAFKA-15326: [7/N] Processing thread non-busy waiting
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on code in PR #14180:
URL: https://github.com/apache/kafka/pull/14180#discussion_r1316995442
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java:
##########
@@ -94,6 +103,114 @@ public void shouldAssignTaskThatCanBeProcessed() {
assertNull(taskManager.assignNextTask(taskExecutor));
}
+ private class AwaitingRunnable implements Runnable {
+ private final CountDownLatch awaitDone = new CountDownLatch(1);
+ private final AtomicBoolean shutdownRequested = new AtomicBoolean(false);
+ @Override
+ public void run() {
+ while (!shutdownRequested.get()) {
+ try {
+ taskManager.awaitProcessableTasks();
+ } catch (final InterruptedException ignored) {
+ }
+ awaitDone.countDown();
+ }
+ }
+
+ public void shutdown() {
+ shutdownRequested.set(true);
+ taskManager.signalProcessableTasks();
+ }
+ }
+
+ @Test
+ public void shouldBlockOnAwait() throws InterruptedException {
Review Comment:
Waiting for the whole verification timeout for a successful test result seems long to me.
Does `shouldReturnFromAwaitOnInterruption()` not imply that the thread is waiting? Do we miss anything if we remove `shouldBlockOnAwait()`?
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskExecutorTest.java:
##########
@@ -85,6 +85,16 @@ public void shouldShutdownTaskExecutor() {
assertNull(taskExecutor.currentTask(), "Have task assigned after shutdown");
}
+ @Test
+ public void shouldAwaitProcessableTasksIfNoneAssignable() throws InterruptedException {
+ assertNull(taskExecutor.currentTask(), "Have task assigned before startup");
+ when(task.isProcessable(anyLong())).thenReturn(false);
+
+ taskExecutor.start();
+
+ verify(taskManager, timeout(VERIFICATION_TIMEOUT).atLeastOnce()).awaitProcessableTasks();
+ }
+
Review Comment:
This test is a bit hard to follow. I think the test would be easier to understand if you specify the simpler stub `when(taskManager.assignNextTask(taskExecutor)).thenReturn(null);` here in the test instead of using the stub specified in `setUp()`.
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java:
##########
@@ -73,6 +80,8 @@ public void setUp() {
when(task.id()).thenReturn(new TaskId(0, 0, "A"));
Review Comment:
nit:
You could reuse `org.apache.kafka.test.StreamsTestUtils.TaskBuilder.statelessTask` here instead of creating your own stubs.
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java:
##########
@@ -94,6 +103,114 @@ public void shouldAssignTaskThatCanBeProcessed() {
assertNull(taskManager.assignNextTask(taskExecutor));
}
+ private class AwaitingRunnable implements Runnable {
+ private final CountDownLatch awaitDone = new CountDownLatch(1);
+ private final AtomicBoolean shutdownRequested = new AtomicBoolean(false);
+ @Override
+ public void run() {
+ while (!shutdownRequested.get()) {
+ try {
+ taskManager.awaitProcessableTasks();
+ } catch (final InterruptedException ignored) {
+ }
+ awaitDone.countDown();
+ }
+ }
+
+ public void shutdown() {
+ shutdownRequested.set(true);
+ taskManager.signalProcessableTasks();
+ }
+ }
+
+ @Test
+ public void shouldBlockOnAwait() throws InterruptedException {
+ final AwaitingRunnable awaitingRunnable = new AwaitingRunnable();
+ final Thread awaitingThread = new Thread(awaitingRunnable);
+ awaitingThread.start();
+
+ assertFalse(awaitingRunnable.awaitDone.await(VERIFICATION_TIMEOUT, TimeUnit.MILLISECONDS));
+
+ awaitingRunnable.shutdown();
+ }
+
+ @Test
+ public void shouldReturnFromAwaitOnInterruption() throws InterruptedException {
+ final AwaitingRunnable awaitingRunnable = new AwaitingRunnable();
+ final Thread awaitingThread = new Thread(awaitingRunnable);
+ awaitingThread.start();
+ verify(tasks, timeout(VERIFICATION_TIMEOUT).atLeastOnce()).activeTasks();
+
+ awaitingThread.interrupt();
+
+ assertTrue(awaitingRunnable.awaitDone.await(VERIFICATION_TIMEOUT, TimeUnit.MILLISECONDS));
+
+ awaitingRunnable.shutdown();
+ }
+
+ @Test
+ public void shouldReturnFromAwaitOnSignalProcessableTasks() throws InterruptedException {
+ final AwaitingRunnable awaitingRunnable = new AwaitingRunnable();
+ final Thread awaitingThread = new Thread(awaitingRunnable);
+ awaitingThread.start();
+ verify(tasks, timeout(VERIFICATION_TIMEOUT).atLeastOnce()).activeTasks();
+
+ taskManager.signalProcessableTasks();
+
+ assertTrue(awaitingRunnable.awaitDone.await(VERIFICATION_TIMEOUT, TimeUnit.MILLISECONDS));
+
+ awaitingRunnable.shutdown();
+ }
+
+ @Test
+ public void shouldReturnFromAwaitOnUnassignment() throws InterruptedException {
+ taskManager.add(Collections.singleton(task));
+ when(taskExecutionMetadata.canProcessTask(eq(task), anyLong())).thenReturn(true);
+
+ final StreamTask t = taskManager.assignNextTask(taskExecutor);
+ assertNotNull(t);
Review Comment:
nit:
```suggestion
final StreamTask task = taskManager.assignNextTask(taskExecutor);
assertNotNull(task);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lucasbru commented on a diff in pull request #14180: KAFKA-15326: [7/N] Processing thread non-busy waiting
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #14180:
URL: https://github.com/apache/kafka/pull/14180#discussion_r1319560316
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskExecutorTest.java:
##########
@@ -85,6 +85,16 @@ public void shouldShutdownTaskExecutor() {
assertNull(taskExecutor.currentTask(), "Have task assigned after shutdown");
}
+ @Test
+ public void shouldAwaitProcessableTasksIfNoneAssignable() throws InterruptedException {
+ assertNull(taskExecutor.currentTask(), "Have task assigned before startup");
+ when(task.isProcessable(anyLong())).thenReturn(false);
+
+ taskExecutor.start();
+
+ verify(taskManager, timeout(VERIFICATION_TIMEOUT).atLeastOnce()).awaitProcessableTasks();
+ }
+
Review Comment:
Done
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java:
##########
@@ -73,6 +80,8 @@ public void setUp() {
when(task.id()).thenReturn(new TaskId(0, 0, "A"));
Review Comment:
Done
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java:
##########
@@ -94,6 +103,114 @@ public void shouldAssignTaskThatCanBeProcessed() {
assertNull(taskManager.assignNextTask(taskExecutor));
}
+ private class AwaitingRunnable implements Runnable {
+ private final CountDownLatch awaitDone = new CountDownLatch(1);
+ private final AtomicBoolean shutdownRequested = new AtomicBoolean(false);
+ @Override
+ public void run() {
+ while (!shutdownRequested.get()) {
+ try {
+ taskManager.awaitProcessableTasks();
+ } catch (final InterruptedException ignored) {
+ }
+ awaitDone.countDown();
+ }
+ }
+
+ public void shutdown() {
+ shutdownRequested.set(true);
+ taskManager.signalProcessableTasks();
+ }
+ }
+
+ @Test
+ public void shouldBlockOnAwait() throws InterruptedException {
Review Comment:
I think it would be nice to keep it. But yeah, now I realize why I set this to 100. How about keeping it with 100? `shouldReturnFromAwaitOnInterruption` could pass if we always return immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #14180: KAFKA-15326: [7/N] Processing thread non-busy waiting
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on PR #14180:
URL: https://github.com/apache/kafka/pull/14180#issuecomment-1713360668
Build failures are unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna merged pull request #14180: KAFKA-15326: [7/N] Processing thread non-busy waiting
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna merged PR #14180:
URL: https://github.com/apache/kafka/pull/14180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wcarlson5 commented on a diff in pull request #14180: KAFKA-15326: [7/N] Processing thread non-busy waiting
Posted by "wcarlson5 (via GitHub)" <gi...@apache.org>.
wcarlson5 commented on code in PR #14180:
URL: https://github.com/apache/kafka/pull/14180#discussion_r1312206725
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java:
##########
@@ -94,6 +103,114 @@ public void shouldAssignTaskThatCanBeProcessed() {
assertNull(taskManager.assignNextTask(taskExecutor));
}
+ private class AwaitingRunnable implements Runnable {
+ private final CountDownLatch awaitDone = new CountDownLatch(1);
+ private final AtomicBoolean shutdownRequested = new AtomicBoolean(false);
+ @Override
+ public void run() {
+ while (!shutdownRequested.get()) {
+ try {
+ taskManager.awaitProcessableTasks();
+ } catch (final InterruptedException ignored) {
+ }
+ awaitDone.countDown();
+ }
+ }
+
+ public void shutdown() {
+ shutdownRequested.set(true);
+ taskManager.signalProcessableTasks();
+ }
+ }
+
+ @Test
+ public void shouldBlockOnAwait() throws InterruptedException {
+ final AwaitingRunnable awaitingRunnable = new AwaitingRunnable();
+ final Thread awaitingThread = new Thread(awaitingRunnable);
+ awaitingThread.start();
+
+ assertFalse(awaitingRunnable.awaitDone.await(100, TimeUnit.MILLISECONDS));
Review Comment:
Do you want to use the `VERIFICATION_TIMEOUT` here?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManager.java:
##########
@@ -118,6 +120,41 @@ public StreamTask assignNextTask(final TaskExecutor executor) {
});
}
+ @Override
+ public void awaitProcessableTasks() throws InterruptedException {
+ final boolean interrupted = returnWithTasksLocked(() -> {
+ for (final Task task : tasks.activeTasks()) {
+ if (!assignedTasks.containsKey(task.id()) &&
+ !lockedTasks.contains(task.id()) &&
+ canProgress((StreamTask) task, time.milliseconds())
+ ) {
+ log.info("Await unblocked: returning early from await since a processable task {} was found", task.id());
Review Comment:
I think info might be too high of a log level. This should be called frequently right?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/TaskManager.java:
##########
@@ -121,4 +119,14 @@ public interface TaskManager {
*/
Map<TaskId, StreamsException> drainUncaughtExceptions();
+ /**
+ * Signals that at least one task has become processable, e.g. because it was resumed or new records may be available.
Review Comment:
I assume this method with be used in later commits?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lucasbru commented on a diff in pull request #14180: KAFKA-15326: [7/N] Processing thread non-busy waiting
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #14180:
URL: https://github.com/apache/kafka/pull/14180#discussion_r1315783830
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java:
##########
@@ -94,6 +103,114 @@ public void shouldAssignTaskThatCanBeProcessed() {
assertNull(taskManager.assignNextTask(taskExecutor));
}
+ private class AwaitingRunnable implements Runnable {
+ private final CountDownLatch awaitDone = new CountDownLatch(1);
+ private final AtomicBoolean shutdownRequested = new AtomicBoolean(false);
+ @Override
+ public void run() {
+ while (!shutdownRequested.get()) {
+ try {
+ taskManager.awaitProcessableTasks();
+ } catch (final InterruptedException ignored) {
+ }
+ awaitDone.countDown();
+ }
+ }
+
+ public void shutdown() {
+ shutdownRequested.set(true);
+ taskManager.signalProcessableTasks();
+ }
+ }
+
+ @Test
+ public void shouldBlockOnAwait() throws InterruptedException {
+ final AwaitingRunnable awaitingRunnable = new AwaitingRunnable();
+ final Thread awaitingThread = new Thread(awaitingRunnable);
+ awaitingThread.start();
+
+ assertFalse(awaitingRunnable.awaitDone.await(100, TimeUnit.MILLISECONDS));
Review Comment:
Thanks, yes.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/TaskManager.java:
##########
@@ -121,4 +119,14 @@ public interface TaskManager {
*/
Map<TaskId, StreamsException> drainUncaughtExceptions();
+ /**
+ * Signals that at least one task has become processable, e.g. because it was resumed or new records may be available.
Review Comment:
Yes!
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManager.java:
##########
@@ -118,6 +120,41 @@ public StreamTask assignNextTask(final TaskExecutor executor) {
});
}
+ @Override
+ public void awaitProcessableTasks() throws InterruptedException {
+ final boolean interrupted = returnWithTasksLocked(() -> {
+ for (final Task task : tasks.activeTasks()) {
+ if (!assignedTasks.containsKey(task.id()) &&
+ !lockedTasks.contains(task.id()) &&
+ canProgress((StreamTask) task, time.milliseconds())
+ ) {
+ log.info("Await unblocked: returning early from await since a processable task {} was found", task.id());
Review Comment:
Yes, you are right. I changed it in a later PR, but I pulled it back into this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org