You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "gharris1727 (via GitHub)" <gi...@apache.org> on 2023/06/09 23:17:05 UTC
[GitHub] [kafka] gharris1727 opened a new pull request, #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
gharris1727 opened a new pull request, #13838:
URL: https://github.com/apache/kafka/pull/13838
The MirrorIntegrationBaseTest-derived suites sometimes fail with the following error message:
> java.lang.AssertionError: Connector MirrorCheckpointConnector tasks did not start in time on cluster: backup-connect-cluster
This assertion fails because the MirrorCheckpointConnector never generates any task configurations:
> [2023-06-09 21:36:43,037] INFO GET response for URL=http://localhost:37695/connectors/MirrorCheckpointConnector/status is {"name":"MirrorCheckpointConnector","connector":{"state":"RUNNING","worker_id":"localhost:35597"},"tasks":[],"type":"source"} (org.apache.kafka.connect.util.clusters.EmbeddedConnectCluster:905)
The connector generates non-empty task configurations because there are no consumer groups to distribute to the tasks, because the only consumer group available is filtered out:
> [2023-06-09 21:34:39,370] DEBUG [MirrorCheckpointConnector|worker] Ignoring the following groups which do not have any offsets for topics that are accepted by the topic filter: [consumer-group-dummy] (org.apache.kafka.connect.mirror.MirrorCheckpointConnector:190)
This is because the consumer-group-dummy did not commit any offsets for any topics, and doesn't satisfy the filtering logic added in #13446 . This is because the dummy group commits offsets in warmUpConsumer: https://github.com/apache/kafka/blob/7eea2a3908fdcee1627c18827e6dcb5ed0089fdd/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/integration/MirrorConnectorsIntegrationBaseTest.java#L1189-L1198
If the single invocation of poll times out before the consumer can retrieve the metadata for the partition, then the commitSync does not commit any offsets for the consumer, leaving no offsets instead of the expected offset 0. This poll duration is 500ms, which is regularly exceeded in my 30% CPU de-flaking environment.
To fix this, increase the poll timeout from 500ms to 5s, to make consumer call-sites less flaky.
Unfortunately this strategy is not ideal for the warmUpConsumer function itself, which is always called on an empty topic and will always block for the whole duration, lengthening the runtime of the test.
So for warmUpConsumer, replace the empty poll() with Admin calls which retrieve the set of partitions and then commit offset 0 for all partitions.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] gharris1727 commented on pull request #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on PR #13838:
URL: https://github.com/apache/kafka/pull/13838#issuecomment-1591734588
Yep, I was able to reproduce some flaky failures in testReplication, and they appear to have a lot of failures in CI. I'll debug those further and remediate them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison merged pull request #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison merged PR #13838:
URL: https://github.com/apache/kafka/pull/13838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on a diff in pull request #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #13838:
URL: https://github.com/apache/kafka/pull/13838#discussion_r1229669181
##########
connect/mirror/src/test/java/org/apache/kafka/connect/mirror/integration/MirrorConnectorsIntegrationBaseTest.java:
##########
@@ -1183,17 +1187,31 @@ private void createTopics() {
}
}
- /*
- * Generate some consumer activity on both clusters to ensure the checkpoint connector always starts promptly
+ /**
+ * Commit offset 0 for all partitions of test-topic-1 for the specified consumer groups on primary and backup clusters.
+ * <p>This is done to force the MirrorCheckpointConnector to start at a task which checkpoints this group.
+ * Must be called before {@link #waitUntilMirrorMakerIsRunning} to prevent that method from timing out.
*/
- protected void warmUpConsumer(Map<String, Object> consumerProps) {
- try (Consumer<byte[], byte[]> dummyConsumer = primary.kafka().createConsumerAndSubscribeTo(consumerProps, "test-topic-1")) {
- dummyConsumer.poll(CONSUMER_POLL_TIMEOUT_MS);
- dummyConsumer.commitSync();
- }
- try (Consumer<byte[], byte[]> dummyConsumer = backup.kafka().createConsumerAndSubscribeTo(consumerProps, "test-topic-1")) {
- dummyConsumer.poll(CONSUMER_POLL_TIMEOUT_MS);
- dummyConsumer.commitSync();
+ protected void prepareConsumerGroup(Map<String, Object> consumerProps) {
+ prepareConsumerGroup(primary.kafka(), consumerProps, "test-topic-1");
+ prepareConsumerGroup(backup.kafka(), consumerProps, "test-topic-1");
+ }
+
+ private void prepareConsumerGroup(EmbeddedKafkaCluster cluster, Map<String, Object> consumerProps, String topic) {
+ try (Admin client = cluster.createAdminClient()) {
+ Map<String, TopicDescription> topics = client.describeTopics(Collections.singleton(topic))
+ .allTopicNames()
+ .get(REQUEST_TIMEOUT_DURATION_MS, TimeUnit.MILLISECONDS);
+ Map<TopicPartition, OffsetAndMetadata> collect = topics.get(topic)
+ .partitions()
+ .stream()
+ .collect(Collectors.toMap(
+ tpi -> new TopicPartition(topic, tpi.partition()),
+ ignored -> new OffsetAndMetadata(0L)));
+ AlterConsumerGroupOffsetsResult alterResult = client.alterConsumerGroupOffsets((String) consumerProps.get("group.id"), collect);
+ alterResult.all().get(REQUEST_TIMEOUT_DURATION_MS, TimeUnit.MILLISECONDS);
+ } catch (ExecutionException | InterruptedException | TimeoutException e) {
Review Comment:
Is there a reason we're catching and rethrowing as RuntimeException here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #13838:
URL: https://github.com/apache/kafka/pull/13838#issuecomment-1598185695
> Reverted in #13864 until we can address the issues with the `MirrorConnectorsIntegrationTransactionsTest::testReplication` test case.
Thank you, @C0urante !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on a diff in pull request #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #13838:
URL: https://github.com/apache/kafka/pull/13838#discussion_r1226492269
##########
connect/mirror/src/test/java/org/apache/kafka/connect/mirror/integration/MirrorConnectorsIntegrationBaseTest.java:
##########
@@ -1187,13 +1191,25 @@ private void createTopics() {
* Generate some consumer activity on both clusters to ensure the checkpoint connector always starts promptly
*/
protected void warmUpConsumer(Map<String, Object> consumerProps) {
- try (Consumer<byte[], byte[]> dummyConsumer = primary.kafka().createConsumerAndSubscribeTo(consumerProps, "test-topic-1")) {
- dummyConsumer.poll(CONSUMER_POLL_TIMEOUT_MS);
- dummyConsumer.commitSync();
- }
- try (Consumer<byte[], byte[]> dummyConsumer = backup.kafka().createConsumerAndSubscribeTo(consumerProps, "test-topic-1")) {
- dummyConsumer.poll(CONSUMER_POLL_TIMEOUT_MS);
- dummyConsumer.commitSync();
+ warmUpConsumer(primary.kafka(), consumerProps, "test-topic-1");
Review Comment:
Considering this does not use a consumer client anymore, is `warmUpConsumer()` still the best name for these methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] gharris1727 commented on a diff in pull request #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on code in PR #13838:
URL: https://github.com/apache/kafka/pull/13838#discussion_r1229900453
##########
connect/mirror/src/test/java/org/apache/kafka/connect/mirror/integration/MirrorConnectorsIntegrationBaseTest.java:
##########
@@ -1183,17 +1187,31 @@ private void createTopics() {
}
}
- /*
- * Generate some consumer activity on both clusters to ensure the checkpoint connector always starts promptly
+ /**
+ * Commit offset 0 for all partitions of test-topic-1 for the specified consumer groups on primary and backup clusters.
+ * <p>This is done to force the MirrorCheckpointConnector to start at a task which checkpoints this group.
+ * Must be called before {@link #waitUntilMirrorMakerIsRunning} to prevent that method from timing out.
*/
- protected void warmUpConsumer(Map<String, Object> consumerProps) {
- try (Consumer<byte[], byte[]> dummyConsumer = primary.kafka().createConsumerAndSubscribeTo(consumerProps, "test-topic-1")) {
- dummyConsumer.poll(CONSUMER_POLL_TIMEOUT_MS);
- dummyConsumer.commitSync();
- }
- try (Consumer<byte[], byte[]> dummyConsumer = backup.kafka().createConsumerAndSubscribeTo(consumerProps, "test-topic-1")) {
- dummyConsumer.poll(CONSUMER_POLL_TIMEOUT_MS);
- dummyConsumer.commitSync();
+ protected void prepareConsumerGroup(Map<String, Object> consumerProps) {
+ prepareConsumerGroup(primary.kafka(), consumerProps, "test-topic-1");
+ prepareConsumerGroup(backup.kafka(), consumerProps, "test-topic-1");
+ }
+
+ private void prepareConsumerGroup(EmbeddedKafkaCluster cluster, Map<String, Object> consumerProps, String topic) {
+ try (Admin client = cluster.createAdminClient()) {
+ Map<String, TopicDescription> topics = client.describeTopics(Collections.singleton(topic))
+ .allTopicNames()
+ .get(REQUEST_TIMEOUT_DURATION_MS, TimeUnit.MILLISECONDS);
+ Map<TopicPartition, OffsetAndMetadata> collect = topics.get(topic)
+ .partitions()
+ .stream()
+ .collect(Collectors.toMap(
+ tpi -> new TopicPartition(topic, tpi.partition()),
+ ignored -> new OffsetAndMetadata(0L)));
+ AlterConsumerGroupOffsetsResult alterResult = client.alterConsumerGroupOffsets((String) consumerProps.get("group.id"), collect);
+ alterResult.all().get(REQUEST_TIMEOUT_DURATION_MS, TimeUnit.MILLISECONDS);
+ } catch (ExecutionException | InterruptedException | TimeoutException e) {
Review Comment:
I was just avoiding changing the throws signatures, but I see now that it's just 3 sites. I'll change this to avoid wrapping the exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] gharris1727 commented on pull request #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on PR #13838:
URL: https://github.com/apache/kafka/pull/13838#issuecomment-1591647109
> This change seems to help with MirrorCheckpoint failing to start. I now get a lot of failures due to MirrorHeartbeat.
I'll try to reproduce this locally, I hadn't noticed that.
> Looking at the logs, it seems a significant amount of time is spent failing to load faulty test plugins, for example:
This log message is from #13182 and the faulty plugins started appearing after #13467. I think that the 14 seconds might be a little exaggerated as the scanning is re-done for each connect worker, and this test instantiates 6 workers across 2 clusters. I'll look into the performance impact of the new plugin instantiations to see if it's more than I expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #13838:
URL: https://github.com/apache/kafka/pull/13838#issuecomment-1594623484
@gharris1727 , this change seems to break the `MirrorConnectorsIntegrationTransactionsTest.testReplication` test. Could you please have a look? Thanks.
https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/1926/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] C0urante commented on pull request #13838: MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #13838:
URL: https://github.com/apache/kafka/pull/13838#issuecomment-1594927539
Reverted in https://github.com/apache/kafka/pull/13864 until we can address the issues with the `MirrorConnectorsIntegrationTransactionsTest::testReplication` test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org