You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "frankvicky (via GitHub)" <gi...@apache.org> on 2024/08/20 16:16:26 UTC
[PR] KAFKA-17371: Flaky test in DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired [kafka]
frankvicky opened a new pull request, #16941:
URL: https://github.com/apache/kafka/pull/16941
JIRA: [KAFKA-17371](https://issues.apache.org/jira/browse/KAFKA-17371)
> The root cause of the failure is that `currentTask = taskManager.assignNextTask(DefaultTaskExecutor.this);` is not an atomic operation. This means that calling `taskManager.assignNextTask` will unblock the `verify(taskManager, timeout(VERIFICATION_TIMEOUT)).assignNextTask(taskExecutor);` statement in the test method.
If `assertNotNull(taskExecutor.currentTask());` is executed before the assignment `currentTaks = [...]` the test will fail.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17371: Flaky test in DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired [kafka]
Posted by "aoli-al (via GitHub)" <gi...@apache.org>.
aoli-al commented on PR #16941:
URL: https://github.com/apache/kafka/pull/16941#issuecomment-2311310562
This makes sense. Thanks for clarifying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17371: Flaky test in DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #16941:
URL: https://github.com/apache/kafka/pull/16941#issuecomment-2309001082
> I'm not entirely sure how this method will be used. From the docs, it seems that client code can invoke it since it is a public method declared in the interface.
It's all internal. Note the package name `...internal...`. So if there is any incorrect usage, it's a bug in Kafka Streams itself. A Kafka Streams user, would never call it.
Having said this, it could still make sense to refactor to avoid that we introduce bug accidentally. However, I am not super familiar with this part of the code and thus cannot judge... Maybe @cadonna or @lucasbru can comment on this question?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17371: Flaky test in DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #16941:
URL: https://github.com/apache/kafka/pull/16941#issuecomment-2306021927
Thanks for the PR. Merged to `trunk`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17371: Flaky test in DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired [kafka]
Posted by "aoli-al (via GitHub)" <gi...@apache.org>.
aoli-al commented on PR #16941:
URL: https://github.com/apache/kafka/pull/16941#issuecomment-2306057184
Thanks for the PR fixing the flaky test.
I'm wondering if the `currentTask` method should be redesigned. Interleaving it with other methods causes at least two test failures.
The root cause is that `currentTask` is not thread-safe and may return a stale/intermediate value. I'm not entirely sure how this method will be used. From the docs, it seems that client code can invoke it since it is a public method declared in the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17371: Flaky test in DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #16941:
URL: https://github.com/apache/kafka/pull/16941
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17371: Flaky test in DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #16941:
URL: https://github.com/apache/kafka/pull/16941#issuecomment-2310032674
The method is internal and only indended for testing. I wouldn't add synchronization which in the end affects production code, where the synchronization isn't needed.
The 'perfect' way is probably to try to rewrite the unit tests to get by without `currentTask`, but it will probably make the tests more convoluted.
The one thing I would change is probaly remove it from the `TaskExecutor` interface and make it non-public, if possible. We also sometimes put `// intended for testing` next to the method, but not everybody likes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17371: Flaky test in DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on PR #16941:
URL: https://github.com/apache/kafka/pull/16941#issuecomment-2303712028
`assignNextTask` is designed to return null if no suitable task is found, so I believe this issue could be resolved by ensuring that a task is assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17371: Flaky test in DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on PR #16941:
URL: https://github.com/apache/kafka/pull/16941#issuecomment-2299281422
Loop comment:
```
I=0; while ./gradlew streams:test --tests DefaultTaskExecutorTest.shouldUnassignTaskWhenRequired --rerun --fail-fast; do (( I=$I+1 )); echo "Completed run: $I"; sleep 1; done
```

I believe the method’s functionality is normal, and the flakiness is caused by a race condition. Sometimes, the main thread checks for null before the worker thread assigns the value, causing the test to be flaky. However, I might have missed something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org