You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/05/01 21:04:42 UTC
[GitHub] [kafka] mjsax opened a new pull request #8600: KAFKA-9928: Fix flaky GlobalKTableEOSIntegrationTest
mjsax opened a new pull request #8600:
URL: https://github.com/apache/kafka/pull/8600
Most changes thus improve the error message output in case a test fails.
Potential fix: remove producer config `retries=1`
Call for review @guozhangwang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #8600: KAFKA-9928: Fix flaky GlobalKTableEOSIntegrationTest
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #8600:
URL: https://github.com/apache/kafka/pull/8600#discussion_r418737835
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/GlobalKTableEOSIntegrationTest.java
##########
@@ -304,59 +382,64 @@ private void startStreams() {
private void produceTopicValues(final String topic) {
IntegrationTestUtils.produceKeyValuesSynchronously(
- topic,
- Arrays.asList(
- new KeyValue<>("a", 1L),
- new KeyValue<>("b", 2L),
- new KeyValue<>("c", 3L),
- new KeyValue<>("d", 4L),
- new KeyValue<>("e", 5L)),
- TestUtils.producerConfig(
- CLUSTER.bootstrapServers(),
- StringSerializer.class,
- LongSerializer.class,
- new Properties()),
- mockTime);
+ topic,
+ Arrays.asList(
+ new KeyValue<>("a", 1L),
+ new KeyValue<>("b", 2L),
+ new KeyValue<>("c", 3L),
+ new KeyValue<>("d", 4L),
+ new KeyValue<>("e", 5L)
+ ),
+ TestUtils.producerConfig(
+ CLUSTER.bootstrapServers(),
+ StringSerializer.class,
+ LongSerializer.class,
+ new Properties()
+ ),
+ mockTime
+ );
}
private void produceAbortedMessages() throws Exception {
final Properties properties = new Properties();
properties.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "someid");
- properties.put(ProducerConfig.RETRIES_CONFIG, 1);
Review comment:
This might be the actually fix. Not sure why we set retries to one, but if we would loose input data, we would never complete the result and the test would time out. (Maybe not relevant for aborted message, but same below)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8600: KAFKA-9928: Fix flaky GlobalKTableEOSIntegrationTest
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8600:
URL: https://github.com/apache/kafka/pull/8600#issuecomment-624284868
I still see the following issue locally:
```
java.lang.AssertionError: Condition not met within timeout 30000. waiting for final values
expected: {a=1+F, b=2+G, c=3+H, d=4+I, e=5+J}
received: {a=1+A, b=2+G, c=3+H, d=4+I, e=5+J}
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:26)
at org.apache.kafka.test.TestUtils.lambda$waitForCondition$17(TestUtils.java:381)
at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:429)
at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:397)
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:378)
at org.apache.kafka.streams.integration.GlobalKTableEOSIntegrationTest.shouldKStreamGlobalKTableLeftJoin(GlobalKTableEOSIntegrationTest.java:205)
```
In addition, sometimes the test will hang as well (i.e. the above verification would not fail, the test just runs forever); I tried using different assignor via `INTERNAL_TASK_ASSIGNOR_CLASS` but the same hanging issue still exists.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #8600: KAFKA-9928: Fix flaky GlobalKTableEOSIntegrationTest
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #8600:
URL: https://github.com/apache/kafka/pull/8600#issuecomment-623238235
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #8600: KAFKA-9928: Fix flaky GlobalKTableEOSIntegrationTest
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #8600:
URL: https://github.com/apache/kafka/pull/8600#issuecomment-624199715
Java 14 passed.
Java 8:
```
org.apache.kafka.streams.integration.EosIntegrationTest.shouldNotViolateEosIfOneTaskFailsWithState[exactly_once]
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowConcurrentAccesses
```
Java 11: `org.apache.kafka.streams.integration.GlobalKTableIntegrationTest.shouldKStreamGlobalKTableLeftJoin`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #8600: KAFKA-9928: Fix flaky GlobalKTableEOSIntegrationTest
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #8600:
URL: https://github.com/apache/kafka/pull/8600#issuecomment-622653176
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8600: KAFKA-9928: Fix flaky GlobalKTableEOSIntegrationTest
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8600:
URL: https://github.com/apache/kafka/pull/8600#issuecomment-625340637
LGTM! Let's merge to trunk now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #8600: KAFKA-9928: Fix flaky GlobalKTableEOSIntegrationTest
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #8600:
URL: https://github.com/apache/kafka/pull/8600#issuecomment-623612153
All three runs timed out.
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #8600: KAFKA-9928: Fix flaky GlobalKTableEOSIntegrationTest
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #8600:
URL: https://github.com/apache/kafka/pull/8600#issuecomment-623237799
Java 8 passed.
Java 11: `org.apache.kafka.streams.integration.GlobalKTableIntegrationTest.shouldKStreamGlobalKTableLeftJoin` (note it's not the EOS test)
Java 14:
```
org.apache.kafka.streams.integration.GlobalKTableEOSIntegrationTest.shouldKStreamGlobalKTableLeftJoin[exactly_once_beta]
org.apache.kafka.streams.integration.GlobalKTableIntegrationTest.shouldKStreamGlobalKTableLeftJoin
```
Java 14 EOS test failed with:
```
java.lang.AssertionError: Condition not met within timeout 30000. waiting for final values
expected: {a=1+F, b=2+G, c=3+H, d=4+I, e=5+J}
received: {a=1+F, b=2+G, c=3+C, d=4+I, e=5+J}
```
It seem we are missing one update, but it's unclear why/how an input record could get dropped... Will investigate further.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org