You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "C0urante (via GitHub)" <gi...@apache.org> on 2024/07/31 18:40:48 UTC
[PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
C0urante opened a new pull request, #16757:
URL: https://github.com/apache/kafka/pull/16757
We're still seeing some flaky test failures for the `OffsetsApiIntegrationTest` suite, but in a much smaller subset of cases:
- testAlterSinkConnectorOffsetsDifferentKafkaClusterTargeted: [1](https://ge.apache.org/s/ydrj6vsi2t5mw/tests/task/:connect:runtime:test/details/org.apache.kafka.connect.integration.OffsetsApiIntegrationTest/testAlterSinkConnectorOffsetsDifferentKafkaClusterTargeted()?top-execution=1), [2](https://ge.apache.org/s/ndddgm7c6ma3c/tests/task/:connect:runtime:test/details/org.apache.kafka.connect.integration.OffsetsApiIntegrationTest/testAlterSinkConnectorOffsetsDifferentKafkaClusterTargeted()?top-execution=1)
- testResetSinkConnectorOffsetsDifferentKafkaClusterTargeted: [1](https://ge.apache.org/s/ydrj6vsi2t5mw/tests/task/:connect:runtime:test/details/org.apache.kafka.connect.integration.OffsetsApiIntegrationTest/testResetSinkConnectorOffsetsDifferentKafkaClusterTargeted()?top-execution=1), [2](https://ge.apache.org/s/ndddgm7c6ma3c/tests/task/:connect:runtime:test/details/org.apache.kafka.connect.integration.OffsetsApiIntegrationTest/testResetSinkConnectorOffsetsDifferentKafkaClusterTargeted()?top-execution=1)
- testGetSinkConnectorOffsetsDifferentKafkaClusterTargeted: [1](https://ge.apache.org/s/ydrj6vsi2t5mw/tests/task/:connect:runtime:test/details/org.apache.kafka.connect.integration.OffsetsApiIntegrationTest/testGetSinkConnectorOffsetsDifferentKafkaClusterTargeted()?top-execution=1)
After examining log files, it looks like this is a genuine case of timeouts being too low; in the three failures I examined, the sink connector's consumer group was never able to form or handle offset commits because the separate Kafka cluster it targeted didn't have a group coordinator and was still creating the internal offsets topic.
Instead of just increasing timeouts, I'd like to add an enhanced cluster readiness check for our `EmbeddedKafkaCluster` class that's automatically performed on startup. This way, not only do we give the test cases above more time to run (assuming that a significant amount of their runtime is currently taken up by bringing up the separate Kafka cluster), we also have better insight into whether future failures are caused by broker startup issues or something else.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #16757:
URL: https://github.com/apache/kafka/pull/16757#issuecomment-2261965565
I'm optimistic that this won't have a serious impact on test runtime, since most of the operations added in this PR are already performed by Connect clusters anyways: forming a group (of workers, not necessarily consumers), creating internal topics, and writing to and reading from them.
This doesn't apply to cases where we spin up an embedded Kafka cluster separately from a Connect cluster, but based on our current tests, those are the cases that would benefit the most from this change.
All that said, definitely open to refining this logic further in the future if it does turn out that we see a significant bump in test runtime with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #16757:
URL: https://github.com/apache/kafka/pull/16757#issuecomment-2261273250
I also believe this would fix the recent flaky failure we saw for the `RestExtensionIntegrationTest::testRestExtensionApi` test case [here](https://ge.apache.org/s/ndddgm7c6ma3c/tests/task/:connect:runtime:test/details/org.apache.kafka.connect.integration.RestExtensionIntegrationTest/testRestExtensionApi()?top-execution=1). The test creates an embedded Connect cluster with a REST extension that tries to read the list of connectors on the cluster while it's being registered. The failure I linked for the test was caused because the request to list the connectors timed out, because the Connect worker was still starting, which in turn happened because the underlying Kafka cluster appeared to still be starting. This test case wasn't improved by our health check endpoint and the way that we use it to ensure that our embedded Connect clusters are healthy after they've started because REST extension registration takes place during cluster startup, before we start querying the health check
endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #16757:
URL: https://github.com/apache/kafka/pull/16757#discussion_r1700371150
##########
connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/EmbeddedKafkaCluster.java:
##########
@@ -153,6 +154,49 @@ public void start() {
producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
}
producer = new KafkaProducer<>(producerProps, new ByteArraySerializer(), new ByteArraySerializer());
+
+ verifyClusterReadiness();
+ }
+
+ /**
+ * Perform an extended check to ensure that the primary APIs of the cluster are available, including:
+ * <ul>
+ * <li>Ability to create a topic</li>
+ * <li>Ability to produce to a topic</li>
+ * <li>Ability to form a consumer group</li>
+ * <li>Ability to consume from a topic</li>
+ * </ul>
+ * If this method completes successfully, all resources created to verify the cluster health
+ * (such as topics and consumer groups) will be cleaned up before it returns.
+ * <p>
+ * This provides extra guarantees compared to other cluster readiness checks such as
+ * {@link ConnectAssertions#assertExactlyNumBrokersAreUp(int, String)} and
+ * {@link KafkaClusterTestKit#waitForReadyBrokers()}, which verify that brokers have
+ * completed startup and joined the cluster, but do not verify that the internal consumer
+ * offsets topic has been created or that it's actually possible for users to create and
+ * interact with topics.
+ */
+ public void verifyClusterReadiness() {
+ String consumerGroupId = UUID.randomUUID().toString();
+ Map<String, Object> consumerConfig = Collections.singletonMap(GROUP_ID_CONFIG, consumerGroupId);
+ String topic = "consumer-warmup-" + consumerGroupId;
+
+ createTopic(topic);
+ produce(topic, "warmup message key", "warmup message value");
+
+ try (Consumer<?, ?> consumer = createConsumerAndSubscribeTo(consumerConfig, topic)) {
+ ConsumerRecords<?, ?> records = consumer.poll(Duration.ofMillis(GROUP_COORDINATOR_AVAILABILITY_DURATION_MS));
+ if (records.isEmpty()) {
+ throw new AssertionError("Failed to verify availability of group coordinator and produce/consume APIs on Kafka cluster in time");
+ }
+ }
+
+ try (Admin admin = createAdminClient()) {
+ admin.deleteConsumerGroups(Collections.singleton(consumerGroupId)).all().get(30, TimeUnit.SECONDS);
+ admin.deleteTopics(Collections.singleton(topic)).all().get(30, TimeUnit.SECONDS);
+ } catch (InterruptedException | ExecutionException | TimeoutException e) {
+ throw new AssertionError("Failed to clean up cluster health check resource(s)", e);
Review Comment:
Any reason why not to use `Assertions.assertDoesNotThrow`?
```
try (Admin admin = createAdminClient()) {
Assertions.assertDoesNotThrow(() -> admin.deleteConsumerGroups(Collections.singleton(consumerGroupId)).all().get(30, TimeUnit.SECONDS), "Failed to clean up consumer groups during cluster health check resource(s)");
Assertions.assertDoesNotThrow(() -> admin.deleteTopics(Collections.singleton(topic)).all().get(30, TimeUnit.SECONDS), "Failed to clean up topics during cluster health check resource(s)");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #16757:
URL: https://github.com/apache/kafka/pull/16757#issuecomment-2263898865
I've examined the test failures and filed a couple of follow-up PRs to address them. I don't believe this PR directly induces any of them (though it may make some more likely due to shifts in timing), so I'll merge now, but if we see an uptick in failures, we can revert.
Follow-up PRs (both of which are currently drafts):
- https://github.com/apache/kafka/pull/16771
- https://github.com/apache/kafka/pull/16772
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #16757:
URL: https://github.com/apache/kafka/pull/16757#discussion_r1700375840
##########
connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/EmbeddedKafkaCluster.java:
##########
@@ -153,6 +154,49 @@ public void start() {
producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
}
producer = new KafkaProducer<>(producerProps, new ByteArraySerializer(), new ByteArraySerializer());
+
+ verifyClusterReadiness();
+ }
+
+ /**
+ * Perform an extended check to ensure that the primary APIs of the cluster are available, including:
+ * <ul>
+ * <li>Ability to create a topic</li>
+ * <li>Ability to produce to a topic</li>
+ * <li>Ability to form a consumer group</li>
+ * <li>Ability to consume from a topic</li>
+ * </ul>
+ * If this method completes successfully, all resources created to verify the cluster health
+ * (such as topics and consumer groups) will be cleaned up before it returns.
+ * <p>
+ * This provides extra guarantees compared to other cluster readiness checks such as
+ * {@link ConnectAssertions#assertExactlyNumBrokersAreUp(int, String)} and
+ * {@link KafkaClusterTestKit#waitForReadyBrokers()}, which verify that brokers have
+ * completed startup and joined the cluster, but do not verify that the internal consumer
+ * offsets topic has been created or that it's actually possible for users to create and
+ * interact with topics.
+ */
+ public void verifyClusterReadiness() {
+ String consumerGroupId = UUID.randomUUID().toString();
+ Map<String, Object> consumerConfig = Collections.singletonMap(GROUP_ID_CONFIG, consumerGroupId);
+ String topic = "consumer-warmup-" + consumerGroupId;
+
+ createTopic(topic);
+ produce(topic, "warmup message key", "warmup message value");
+
+ try (Consumer<?, ?> consumer = createConsumerAndSubscribeTo(consumerConfig, topic)) {
+ ConsumerRecords<?, ?> records = consumer.poll(Duration.ofMillis(GROUP_COORDINATOR_AVAILABILITY_DURATION_MS));
+ if (records.isEmpty()) {
+ throw new AssertionError("Failed to verify availability of group coordinator and produce/consume APIs on Kafka cluster in time");
+ }
+ }
+
+ try (Admin admin = createAdminClient()) {
+ admin.deleteConsumerGroups(Collections.singleton(consumerGroupId)).all().get(30, TimeUnit.SECONDS);
+ admin.deleteTopics(Collections.singleton(topic)).all().get(30, TimeUnit.SECONDS);
+ } catch (InterruptedException | ExecutionException | TimeoutException e) {
+ throw new AssertionError("Failed to clean up cluster health check resource(s)", e);
Review Comment:
Same btw can be done for `produce` `assertDoesNotThrow(() -> producer.send(msg).get());` we are doing something similar in tools
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante merged PR #16757:
URL: https://github.com/apache/kafka/pull/16757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #16757:
URL: https://github.com/apache/kafka/pull/16757#discussion_r1700375840
##########
connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/EmbeddedKafkaCluster.java:
##########
@@ -153,6 +154,49 @@ public void start() {
producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
}
producer = new KafkaProducer<>(producerProps, new ByteArraySerializer(), new ByteArraySerializer());
+
+ verifyClusterReadiness();
+ }
+
+ /**
+ * Perform an extended check to ensure that the primary APIs of the cluster are available, including:
+ * <ul>
+ * <li>Ability to create a topic</li>
+ * <li>Ability to produce to a topic</li>
+ * <li>Ability to form a consumer group</li>
+ * <li>Ability to consume from a topic</li>
+ * </ul>
+ * If this method completes successfully, all resources created to verify the cluster health
+ * (such as topics and consumer groups) will be cleaned up before it returns.
+ * <p>
+ * This provides extra guarantees compared to other cluster readiness checks such as
+ * {@link ConnectAssertions#assertExactlyNumBrokersAreUp(int, String)} and
+ * {@link KafkaClusterTestKit#waitForReadyBrokers()}, which verify that brokers have
+ * completed startup and joined the cluster, but do not verify that the internal consumer
+ * offsets topic has been created or that it's actually possible for users to create and
+ * interact with topics.
+ */
+ public void verifyClusterReadiness() {
+ String consumerGroupId = UUID.randomUUID().toString();
+ Map<String, Object> consumerConfig = Collections.singletonMap(GROUP_ID_CONFIG, consumerGroupId);
+ String topic = "consumer-warmup-" + consumerGroupId;
+
+ createTopic(topic);
+ produce(topic, "warmup message key", "warmup message value");
+
+ try (Consumer<?, ?> consumer = createConsumerAndSubscribeTo(consumerConfig, topic)) {
+ ConsumerRecords<?, ?> records = consumer.poll(Duration.ofMillis(GROUP_COORDINATOR_AVAILABILITY_DURATION_MS));
+ if (records.isEmpty()) {
+ throw new AssertionError("Failed to verify availability of group coordinator and produce/consume APIs on Kafka cluster in time");
+ }
+ }
+
+ try (Admin admin = createAdminClient()) {
+ admin.deleteConsumerGroups(Collections.singleton(consumerGroupId)).all().get(30, TimeUnit.SECONDS);
+ admin.deleteTopics(Collections.singleton(topic)).all().get(30, TimeUnit.SECONDS);
+ } catch (InterruptedException | ExecutionException | TimeoutException e) {
+ throw new AssertionError("Failed to clean up cluster health check resource(s)", e);
Review Comment:
Same btw can be done for `produce` as well `assertDoesNotThrow(() -> producer.send(msg).get());` we are doing something similar in tools
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #16757:
URL: https://github.com/apache/kafka/pull/16757#discussion_r1700375840
##########
connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/EmbeddedKafkaCluster.java:
##########
@@ -153,6 +154,49 @@ public void start() {
producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
}
producer = new KafkaProducer<>(producerProps, new ByteArraySerializer(), new ByteArraySerializer());
+
+ verifyClusterReadiness();
+ }
+
+ /**
+ * Perform an extended check to ensure that the primary APIs of the cluster are available, including:
+ * <ul>
+ * <li>Ability to create a topic</li>
+ * <li>Ability to produce to a topic</li>
+ * <li>Ability to form a consumer group</li>
+ * <li>Ability to consume from a topic</li>
+ * </ul>
+ * If this method completes successfully, all resources created to verify the cluster health
+ * (such as topics and consumer groups) will be cleaned up before it returns.
+ * <p>
+ * This provides extra guarantees compared to other cluster readiness checks such as
+ * {@link ConnectAssertions#assertExactlyNumBrokersAreUp(int, String)} and
+ * {@link KafkaClusterTestKit#waitForReadyBrokers()}, which verify that brokers have
+ * completed startup and joined the cluster, but do not verify that the internal consumer
+ * offsets topic has been created or that it's actually possible for users to create and
+ * interact with topics.
+ */
+ public void verifyClusterReadiness() {
+ String consumerGroupId = UUID.randomUUID().toString();
+ Map<String, Object> consumerConfig = Collections.singletonMap(GROUP_ID_CONFIG, consumerGroupId);
+ String topic = "consumer-warmup-" + consumerGroupId;
+
+ createTopic(topic);
+ produce(topic, "warmup message key", "warmup message value");
+
+ try (Consumer<?, ?> consumer = createConsumerAndSubscribeTo(consumerConfig, topic)) {
+ ConsumerRecords<?, ?> records = consumer.poll(Duration.ofMillis(GROUP_COORDINATOR_AVAILABILITY_DURATION_MS));
+ if (records.isEmpty()) {
+ throw new AssertionError("Failed to verify availability of group coordinator and produce/consume APIs on Kafka cluster in time");
+ }
+ }
+
+ try (Admin admin = createAdminClient()) {
+ admin.deleteConsumerGroups(Collections.singleton(consumerGroupId)).all().get(30, TimeUnit.SECONDS);
+ admin.deleteTopics(Collections.singleton(topic)).all().get(30, TimeUnit.SECONDS);
+ } catch (InterruptedException | ExecutionException | TimeoutException e) {
+ throw new AssertionError("Failed to clean up cluster health check resource(s)", e);
Review Comment:
Same btw can be done for `produce` similar to `DeleteOffsetsConsumerGroupCommandIntegrationTest::produceRecord`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on code in PR #16757:
URL: https://github.com/apache/kafka/pull/16757#discussion_r1700785965
##########
connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/EmbeddedKafkaCluster.java:
##########
@@ -153,6 +154,49 @@ public void start() {
producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
}
producer = new KafkaProducer<>(producerProps, new ByteArraySerializer(), new ByteArraySerializer());
+
+ verifyClusterReadiness();
+ }
+
+ /**
+ * Perform an extended check to ensure that the primary APIs of the cluster are available, including:
+ * <ul>
+ * <li>Ability to create a topic</li>
+ * <li>Ability to produce to a topic</li>
+ * <li>Ability to form a consumer group</li>
+ * <li>Ability to consume from a topic</li>
+ * </ul>
+ * If this method completes successfully, all resources created to verify the cluster health
+ * (such as topics and consumer groups) will be cleaned up before it returns.
+ * <p>
+ * This provides extra guarantees compared to other cluster readiness checks such as
+ * {@link ConnectAssertions#assertExactlyNumBrokersAreUp(int, String)} and
+ * {@link KafkaClusterTestKit#waitForReadyBrokers()}, which verify that brokers have
+ * completed startup and joined the cluster, but do not verify that the internal consumer
+ * offsets topic has been created or that it's actually possible for users to create and
+ * interact with topics.
+ */
+ public void verifyClusterReadiness() {
+ String consumerGroupId = UUID.randomUUID().toString();
+ Map<String, Object> consumerConfig = Collections.singletonMap(GROUP_ID_CONFIG, consumerGroupId);
+ String topic = "consumer-warmup-" + consumerGroupId;
+
+ createTopic(topic);
+ produce(topic, "warmup message key", "warmup message value");
+
+ try (Consumer<?, ?> consumer = createConsumerAndSubscribeTo(consumerConfig, topic)) {
+ ConsumerRecords<?, ?> records = consumer.poll(Duration.ofMillis(GROUP_COORDINATOR_AVAILABILITY_DURATION_MS));
+ if (records.isEmpty()) {
+ throw new AssertionError("Failed to verify availability of group coordinator and produce/consume APIs on Kafka cluster in time");
+ }
+ }
+
+ try (Admin admin = createAdminClient()) {
+ admin.deleteConsumerGroups(Collections.singleton(consumerGroupId)).all().get(30, TimeUnit.SECONDS);
+ admin.deleteTopics(Collections.singleton(topic)).all().get(30, TimeUnit.SECONDS);
+ } catch (InterruptedException | ExecutionException | TimeoutException e) {
+ throw new AssertionError("Failed to clean up cluster health check resource(s)", e);
Review Comment:
For the first comment, we already wrap things in a try-with-resources block in order to automatically close the consumer when the scope is exited and it keeps things simpler to add a corresponding catch clause.
I guess we could do this for produce but I'd rather not touch on that logic since it's not modified by this PR. If you or someone else wants to do a comprehensive refactor for our tests to adapt this style I'd be happy to take a look, though!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #16757:
URL: https://github.com/apache/kafka/pull/16757#issuecomment-2261436134
I'm less certain of this, but it also might be a fix for the recent flaky we saw for the `ConnectWorkerIntegrationTest:: testRequestTimeouts` test case [here](https://ge.apache.org/s/ydrj6vsi2t5mw/tests/task/:connect:runtime:test/details/org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest/testRequestTimeouts()?top-execution=1). And, if not, it would again at least help us distinguish between broker errors and other ones, so I've added the readiness check to `EmbeddedKafkaCluster::restartOnlyBrokers`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #16757:
URL: https://github.com/apache/kafka/pull/16757#issuecomment-2261175308
@gharris1727 If you have a moment could you take a look? Trying to continue on my war against flaky tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add extended cluster readiness check for Connect's embedded Kafka cluster [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #16757:
URL: https://github.com/apache/kafka/pull/16757#discussion_r1700371150
##########
connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/EmbeddedKafkaCluster.java:
##########
@@ -153,6 +154,49 @@ public void start() {
producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
}
producer = new KafkaProducer<>(producerProps, new ByteArraySerializer(), new ByteArraySerializer());
+
+ verifyClusterReadiness();
+ }
+
+ /**
+ * Perform an extended check to ensure that the primary APIs of the cluster are available, including:
+ * <ul>
+ * <li>Ability to create a topic</li>
+ * <li>Ability to produce to a topic</li>
+ * <li>Ability to form a consumer group</li>
+ * <li>Ability to consume from a topic</li>
+ * </ul>
+ * If this method completes successfully, all resources created to verify the cluster health
+ * (such as topics and consumer groups) will be cleaned up before it returns.
+ * <p>
+ * This provides extra guarantees compared to other cluster readiness checks such as
+ * {@link ConnectAssertions#assertExactlyNumBrokersAreUp(int, String)} and
+ * {@link KafkaClusterTestKit#waitForReadyBrokers()}, which verify that brokers have
+ * completed startup and joined the cluster, but do not verify that the internal consumer
+ * offsets topic has been created or that it's actually possible for users to create and
+ * interact with topics.
+ */
+ public void verifyClusterReadiness() {
+ String consumerGroupId = UUID.randomUUID().toString();
+ Map<String, Object> consumerConfig = Collections.singletonMap(GROUP_ID_CONFIG, consumerGroupId);
+ String topic = "consumer-warmup-" + consumerGroupId;
+
+ createTopic(topic);
+ produce(topic, "warmup message key", "warmup message value");
+
+ try (Consumer<?, ?> consumer = createConsumerAndSubscribeTo(consumerConfig, topic)) {
+ ConsumerRecords<?, ?> records = consumer.poll(Duration.ofMillis(GROUP_COORDINATOR_AVAILABILITY_DURATION_MS));
+ if (records.isEmpty()) {
+ throw new AssertionError("Failed to verify availability of group coordinator and produce/consume APIs on Kafka cluster in time");
+ }
+ }
+
+ try (Admin admin = createAdminClient()) {
+ admin.deleteConsumerGroups(Collections.singleton(consumerGroupId)).all().get(30, TimeUnit.SECONDS);
+ admin.deleteTopics(Collections.singleton(topic)).all().get(30, TimeUnit.SECONDS);
+ } catch (InterruptedException | ExecutionException | TimeoutException e) {
+ throw new AssertionError("Failed to clean up cluster health check resource(s)", e);
Review Comment:
Any reason why not using `Assertions.assertDoesNotThrow`?
```
try (Admin admin = createAdminClient()) {
Assertions.assertDoesNotThrow(() -> admin.deleteConsumerGroups(Collections.singleton(consumerGroupId)).all().get(30, TimeUnit.SECONDS), "Failed to clean up consumer groups during cluster health check resource(s)");
Assertions.assertDoesNotThrow(() -> admin.deleteTopics(Collections.singleton(topic)).all().get(30, TimeUnit.SECONDS), "Failed to clean up topics during cluster health check resource(s)");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org