You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "Shekharrajak (via GitHub)" <gi...@apache.org> on 2026/05/09 17:57:24 UTC
[PR] Feature/kafka client 4.2.0 upgrade (druid)
Shekharrajak opened a new pull request, #19441:
URL: https://github.com/apache/druid/pull/19441
Ref https://github.com/apache/druid/issues/19322
### Description
* Kafka client to 4.2.0
* EmbeddedKafkaBroker test helper using Testcontainers
* kafka-indexing-service tests to EmbeddedKafkaBroker; drop TestBroker
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
<hr>
##### Key changed/added classes in this PR
* `MyFoo`
* `OurBar`
* `TheirBaz`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19441:
URL: https://github.com/apache/druid/pull/19441#issuecomment-4413305514
This will be the first step to improve & simplify the kafka integration testing , version upgrade to open the path for Queue Semantics Kafka Ingestion https://github.com/apache/druid/pull/19311
cc @gianm @a2l007 @clintropolis @FrankChen021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19441:
URL: https://github.com/apache/druid/pull/19441#discussion_r3213664116
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/test/EmbeddedKafkaBroker.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka.test;
+
+import org.apache.druid.indexing.kafka.KafkaConsumerConfigs;
+import org.apache.kafka.clients.admin.Admin;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.testcontainers.kafka.KafkaContainer;
+
+import java.io.Closeable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Embedded Kafka broker for unit tests, backed by Testcontainers {@link KafkaContainer}.
+ */
+public class EmbeddedKafkaBroker implements Closeable
+{
+ private static final String KAFKA_IMAGE =
+ System.getProperty("druid.testing.kafka.image", "apache/kafka:4.2.0");
+
+ private final KafkaContainer container;
+
+ public EmbeddedKafkaBroker()
+ {
+ this(Map.of());
+ }
+
+ /**
+ * @param brokerEnv extra env vars forwarded to the container; the apache/kafka
+ * image maps {@code KAFKA_FOO_BAR} to broker config {@code foo.bar}.
+ */
+ public EmbeddedKafkaBroker(Map<String, String> brokerEnv)
+ {
+ this.container = new KafkaContainer(KAFKA_IMAGE);
+ if (brokerEnv != null && !brokerEnv.isEmpty()) {
+ container.withEnv(brokerEnv);
+ }
+ }
+
+ public void start()
+ {
+ container.start();
+ }
+
+ public String getBootstrapServerUrl()
+ {
+ return container.getBootstrapServers();
+ }
+
+ public int getPort()
+ {
+ final String url = getBootstrapServerUrl();
+ final int colon = url.lastIndexOf(':');
+ return Integer.parseInt(url.substring(colon + 1));
Review Comment:
## CodeQL / Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/druid/security/code-scanning/11189)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19441:
URL: https://github.com/apache/druid/pull/19441#discussion_r3213610302
##########
extensions-core/kafka-extraction-namespace/pom.xml:
##########
@@ -114,17 +114,6 @@
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>org.apache.curator</groupId>
- <artifactId>curator-test</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.kafka</groupId>
Review Comment:
Druid only ever depended on it from test scope, and only because TestBroker needed kafka.server.KafkaServer. Once TestBroker is gone, nothing imports anything from kafka.* (Scala namespace)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19441:
URL: https://github.com/apache/druid/pull/19441#discussion_r3213617553
##########
extensions-core/kafka-extraction-namespace/pom.xml:
##########
@@ -114,17 +114,6 @@
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>org.apache.curator</groupId>
Review Comment:
this was only used to embed ZooKeeper for TestBroker/TestKafkaExtractionCluster; KRaft mode means no ZooKeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19441:
URL: https://github.com/apache/druid/pull/19441#issuecomment-4413375576
looks like a flaky test failing in CI https://github.com/apache/druid/actions/runs/25607936768/job/75172943327?pr=19441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19441:
URL: https://github.com/apache/druid/pull/19441#issuecomment-4427205942
@FrankChen021 thanks for checking, CI checks are looking good now.
Once we have these changes ready the Queue implementation of phase 1 will be easier to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19441:
URL: https://github.com/apache/druid/pull/19441#issuecomment-4430754924
Please re-run the CI check https://github.com/apache/druid/actions/runs/25716029780/job/75506218870?pr=19441 - it is not looking related to our changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 merged PR #19441:
URL: https://github.com/apache/druid/pull/19441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19441:
URL: https://github.com/apache/druid/pull/19441#discussion_r3213630383
##########
extensions-core/kafka-extraction-namespace/pom.xml:
##########
@@ -136,10 +125,15 @@
<artifactId>easymock</artifactId>
<scope>test</scope>
</dependency>
- <dependency><!-- Required by dependency check, used by kafka_2.13 -->
- <groupId>org.scala-lang</groupId>
- <artifactId>scala-library</artifactId>
- <version>${scala.library.version}</version>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers-kafka</artifactId>
Review Comment:
kafka container specific - that will help in integration testing and running containers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19441:
URL: https://github.com/apache/druid/pull/19441#issuecomment-4413490024
Please help in triggering the failing CI checks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19441:
URL: https://github.com/apache/druid/pull/19441#discussion_r3213638167
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/supervisor/KafkaSupervisorTest.java:
##########
@@ -209,22 +206,19 @@ public KafkaSupervisorTest(int numThreads)
@BeforeClass
public static void setupClass() throws Exception
{
- zkServer = new TestingCluster(1);
Review Comment:
KRaft mode - hence no zk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19441:
URL: https://github.com/apache/druid/pull/19441#discussion_r3213613581
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/supervisor/KafkaSupervisorTest.java:
##########
@@ -50,7 +49,7 @@
import org.apache.druid.indexing.kafka.KafkaIndexTaskRunner;
import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
import org.apache.druid.indexing.kafka.KafkaRecordSupplier;
-import org.apache.druid.indexing.kafka.test.TestBroker;
+import org.apache.druid.indexing.kafka.test.EmbeddedKafkaBroker;
Review Comment:
TestBroker no longer compiles (Scala kafka.server.KafkaServer removed in Kafka 4.0); embedded ZooKeeper mode no longer supported by Kafka 4.x brokers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] build(deps): Kafka client 4.2.0 upgrade (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on PR #19441:
URL: https://github.com/apache/druid/pull/19441#issuecomment-4417042947
please check the error of this check: [Static Checks CI / strict-compilation / strict-compilation (pull_request)](https://github.com/apache/druid/actions/runs/25607936712/job/75172943143?pr=19441)Failing after 39m
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org