You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/06/11 02:25:08 UTC
[GitHub] [kafka] ableegoldman opened a new pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
ableegoldman opened a new pull request #8849:
URL: https://github.com/apache/kafka/pull/8849
We need to make sure that corrupted standby tasks are actually cleaned up upon a TaskCorruptedException. However due to the `commit` prior to invoking `handleCorruption`, it's possible to throw a TaskMigratedException before actually cleaning up any of the corrupted tasks.
This is fine for active tasks since `handleLostAll` will finish up the job, but it does nothing with standby tasks. We should make sure that standby tasks are handled before attempting to commit (which we can do, since we don't need to commit anything for the corrupted standbys)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642391327
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642391773
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643488438
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642391433
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642391750
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642388044
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642992126
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643490477
@guozhangwang one of the failing tests is pretty much permanently broken, still waiting on this PR to be merged: https://github.com/apache/kafka/pull/8853
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman edited a comment on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman edited a comment on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643518513
Alright, ran the full suite locally. The one test still fail (as expected) but otherwise green @guozhangwang :
```
kafka.admin.ReassignPartitionsUnitTest > testModifyBrokerThrottles FAILED
java.lang.NullPointerException
at kafka.admin.ReassignPartitionsUnitTest.verifyBrokerThrottleResults(ReassignPartitionsUnitTest.scala:577)
at kafka.admin.ReassignPartitionsUnitTest.testModifyBrokerThrottles(ReassignPartitionsUnitTest.scala:540)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643488134
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643519678
Cherry-pick to 2.6.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643022251
Java 11 and 8 failed with (extremely) flaky `ReassignPartitionsUnitTest.testModifyBrokerThrottles` -- https://issues.apache.org/jira/browse/KAFKA-10155
Java 14 failed with `SslSelectorTest.testCloseOldestConnection`-- https://issues.apache.org/jira/browse/KAFKA-10156
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642394105
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643488049
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang removed a comment on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang removed a comment on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643488049
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642374804
Tests are still a WIP. Just wanted to open it up for input on the problem/fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643518513
Alright, ran the full suite locally. The one test still fail (as expected) but otherwise green @guozhangwang :
```
kafka.admin.ReassignPartitionsUnitTest > testModifyBrokerThrottles FAILED
java.lang.NullPointerException
at kafka.admin.ReassignPartitionsUnitTest.verifyBrokerThrottleResults(ReassignPartitionsUnitTest.scala:577)
at kafka.admin.ReassignPartitionsUnitTest.testModifyBrokerThrottles(ReassignPartitionsUnitTest.scala:540)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643509124
Will do
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang merged pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang merged pull request #8849:
URL: https://github.com/apache/kafka/pull/8849
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642381041
Btw: checkstyle failed :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642386255
Alright, tests/checkstyle/etc are all ready for the builds to be triggered
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642388373
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643488996
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643488618
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642373459
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642984741
Unrelated flaky test failed: `kafka.admin.ReassignPartitionsUnitTest.testModifyBrokerThrottles`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#discussion_r438524798
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StandbyTask.java
##########
@@ -137,7 +137,7 @@ public void resume() {
public Map<TopicPartition, OffsetAndMetadata> prepareCommit() {
if (state() == State.RUNNING || state() == State.SUSPENDED) {
stateMgr.flush();
- log.info("Task ready for committing");
+ log.debug("Prepared task for committing");
Review comment:
Just tried to consolidate the log messages between active/standby tasks (and demoted these to debug)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642387910
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#discussion_r438514181
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java
##########
@@ -92,7 +92,7 @@ public void createTopics() throws Exception {
}
@Test
- public void surviveWithOneTaskAsStandby() throws ExecutionException, InterruptedException, IOException {
+ public void surviveWithOneTaskAsStandby() throws InterruptedException, IOException {
Review comment:
nit: just simplify to `throws Exception`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642640021
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642485477
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642387968
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642968031
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642391252
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643007836
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643007754
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman edited a comment on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
ableegoldman edited a comment on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-643490477
@guozhangwang one of the failing tests is pretty much permanently broken, still waiting on this PR to be merged: https://github.com/apache/kafka/pull/8853
Until then, I wouldn't expect `ReassignPartitionsUnitTest.testModifyBrokerThrottles` to pass
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8849: KAFKA-10144: clean up corrupted standby tasks before attempting a commit
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8849:
URL: https://github.com/apache/kafka/pull/8849#issuecomment-642392428
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org