You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mumrah (via GitHub)" <gi...@apache.org> on 2025/01/17 20:46:35 UTC
[PR] KAFKA-17587 Refactor test infrastructure [kafka]
mumrah opened a new pull request, #18602:
URL: https://github.com/apache/kafka/pull/18602
This patch reorganizes our test infrastructure into three Gradle modules:
":test-common:test-common-api" is now a minimal dependency which exposes interfaces and annotations only. It has one project dependency on server-common to expose commonly used data classes (MetadataVersion, Feature, etc). Since this pulls in server-common, this module is Java 17+. It cannot be used by ":clients" or other Java 11 modules.
":test-common:test-common-junit" includes the auto-quarantined JUnit extension. The `@Flaky` annotation has been moved here. Since this module has no project dependencies, we can add it to the Java 11 list so that ":clients" and others can utilize the `@Flaky` annotation
":test-common:test-common-runtime" now includes all of the test infrastructure code (TestKitNodes, etc).
In addition to this reorganization, this patch leverages JUnit SPI service discovery so that modules can utilize the integration test framework without depending on ":core". This will allow us to start moving integration tests out of core and into the appropriate sub-module. This is done by adding ":test-common:test-common-runtime" as a testRuntimeOnly dependency rather than as a testImplementation dependency. A trivial example was added to QuorumControllerTest to illustrate this.
This patch does not expose ClusterInstance through the SPI since it has many direct dependencies on core. A stripped-down version of ClusterInstance that can be used by non-core modules remains as future work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2599756402
Hm, yea.. I see what you mean. Here are some ideas for alternatives to test-common-api
* test-common-private-api
* test-common-defs
* test-common-lib
* test-common-spi (sort of a misnomer)
* test-common
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #18602:
URL: https://github.com/apache/kafka/pull/18602#discussion_r1920951057
##########
core/src/test/scala/unit/kafka/server/SaslApiVersionsRequestTest.scala:
##########
@@ -19,13 +19,15 @@ package kafka.server
import kafka.api.{KafkaSasl, SaslSetup}
import kafka.security.JaasTestUtils
import kafka.server.SaslApiVersionsRequestTest.{kafkaClientSaslMechanism, kafkaServerSaslMechanisms}
-import org.apache.kafka.common.test.api.{ClusterTemplate, Type, ClusterTestExtensions, ClusterConfig, ClusterInstance}
+import org.apache.kafka.common.test.api.{ClusterConfig, ClusterTemplate, Type}
import org.apache.kafka.common.config.SaslConfigs
import org.apache.kafka.common.config.internals.BrokerSecurityConfigs
import org.apache.kafka.common.message.SaslHandshakeRequestData
import org.apache.kafka.common.protocol.{ApiKeys, Errors}
import org.apache.kafka.common.requests.{ApiVersionsRequest, ApiVersionsResponse, SaslHandshakeRequest, SaslHandshakeResponse}
import org.apache.kafka.common.security.auth.SecurityProtocol
+import org.apache.kafka.common.test.ClusterInstance
+import org.apache.kafka.common.test.junit.ClusterTestExtensions
Review Comment:
Wow, good catch 😄 I left this one here intentionally to make sure it still worked if people explicitly add the annotation in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #18602:
URL: https://github.com/apache/kafka/pull/18602#discussion_r1920952622
##########
test-common/test-common-api/src/main/java/org/apache/kafka/common/test/api/README.md:
##########
@@ -79,7 +79,7 @@ a JUnit extension called `ClusterTestExtensions` which knows how to process thes
invocations. Test classes that wish to make use of these annotations need to explicitly register this extension:
```scala
-import org.apache.kafka.common.test.api.ClusterTestExtensions
+import org.apache.kafka.common.test.junit.ClusterTestExtensions
@ExtendWith(value = Array(classOf[ClusterTestExtensions]))
Review Comment:
Correct. I'll update these docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #18602:
URL: https://github.com/apache/kafka/pull/18602#discussion_r1923894424
##########
build.gradle:
##########
@@ -1102,8 +1103,9 @@ project(':core') {
testImplementation project(':server-common').sourceSets.test.output
testImplementation project(':storage:storage-api').sourceSets.test.output
testImplementation project(':server').sourceSets.test.output
- testImplementation project(':test-common')
- testImplementation project(':test-common:test-common-api')
+ testImplementation project(':test-common:test-common-runtime')
+ testImplementation project(':test-common:test-common-internal-api')
+ testImplementation project(':test-common:test-common-util')
Review Comment:
> Since test-common-internal-api is meant to be the abstraction for the test infrastructure offered by test-common-runtime, it doesn't make sense to include one without the other. It might be confusing for developers if @ClusterTest was available to import, but was not actually usable.
Yes, you're right. Let's keep the current structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2610904191
@chia7712 yea I can backport it. That will make it easier to backport test changes during the ongoing 4.0.0 release and down the road in 4.0.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2599735849
> Do we intend to have a common test module for clients and streams (aside from junit specific classes)?
I think that would make sense. There's quite a bit of utility test code that could be kept at Java 11. In this case we might consider `test-common-util` instead of `test-common-junit`
> If we were to provide a test module for external usage, what would we call it?
I was thinking `testkit` or `test-kit` (so `kafka-testkit.jar` or `kafka-test-kit.jar`). Are you thinking we go ahead and establish a public-friendly name for the `test-common-api` module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #18602:
URL: https://github.com/apache/kafka/pull/18602#discussion_r1923835005
##########
build.gradle:
##########
@@ -1102,8 +1103,9 @@ project(':core') {
testImplementation project(':server-common').sourceSets.test.output
testImplementation project(':storage:storage-api').sourceSets.test.output
testImplementation project(':server').sourceSets.test.output
- testImplementation project(':test-common')
- testImplementation project(':test-common:test-common-api')
+ testImplementation project(':test-common:test-common-runtime')
+ testImplementation project(':test-common:test-common-internal-api')
+ testImplementation project(':test-common:test-common-util')
Review Comment:
Let me see if I understand your proposal
* Refactor "test-common-internal-api" to not depend on ":server-common"
* Move "test-common-internal-api" to Java 11
* Move `@Flaky` to the "test-common-internal-api"
* Remove "test-common-util"
I think this would make sense, except that Java 11 modules like ":clients" cannot bring a runtime dependency with a newer java version. I tried this and got:
```
* What went wrong:
Could not determine the dependencies of task ':clients:test'.
> Could not resolve all dependencies for configuration ':clients:testRuntimeClasspath'.
> Could not resolve project :test-common:test-common-runtime.
Required by:
project :clients
> Dependency resolution is looking for a library compatible with JVM runtime version 11, but 'project :test-common:test-common-runtime' is only compatible with JVM runtime version 17 or newer.
```
Do you know of a way around this?
---
Since test-common-internal-api is meant to be the abstraction for the test infrastructure offered by test-common-runtime, it doesn't make sense to include one without the other. It might be confusing for developers if `@ClusterTest` was available to import, but was not actually usable.
If we want the client integration tests to be moved out of core, we should create a ":clients:integration-tests" module that is Java 17 and move the tests there (similar to what streams does).
I expect "test-common-util" will grow to include helper/common test methods. We can probably combine the most of the various TestUtils into a single Java 11 utils class under this module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #18602:
URL: https://github.com/apache/kafka/pull/18602#discussion_r1920938027
##########
build.gradle:
##########
@@ -1364,6 +1365,9 @@ project(':metadata') {
testImplementation project(':raft').sourceSets.test.output
testImplementation project(':server-common').sourceSets.test.output
+ testImplementation project(':test-common:test-common-api')
+ testRuntimeOnly project(':test-common:test-common-runtime')
Review Comment:
why only `metadata` module declares `test-common:test-common-runtime` as `testRuntimeOnly`?
##########
core/src/test/scala/unit/kafka/server/SaslApiVersionsRequestTest.scala:
##########
@@ -19,13 +19,15 @@ package kafka.server
import kafka.api.{KafkaSasl, SaslSetup}
import kafka.security.JaasTestUtils
import kafka.server.SaslApiVersionsRequestTest.{kafkaClientSaslMechanism, kafkaServerSaslMechanisms}
-import org.apache.kafka.common.test.api.{ClusterTemplate, Type, ClusterTestExtensions, ClusterConfig, ClusterInstance}
+import org.apache.kafka.common.test.api.{ClusterConfig, ClusterTemplate, Type}
import org.apache.kafka.common.config.SaslConfigs
import org.apache.kafka.common.config.internals.BrokerSecurityConfigs
import org.apache.kafka.common.message.SaslHandshakeRequestData
import org.apache.kafka.common.protocol.{ApiKeys, Errors}
import org.apache.kafka.common.requests.{ApiVersionsRequest, ApiVersionsResponse, SaslHandshakeRequest, SaslHandshakeResponse}
import org.apache.kafka.common.security.auth.SecurityProtocol
+import org.apache.kafka.common.test.ClusterInstance
+import org.apache.kafka.common.test.junit.ClusterTestExtensions
Review Comment:
Should we remove `@ExtendWith(value = Array(classOf[ClusterTestExtensions]))` from this class?
##########
test-common/test-common-api/src/main/java/org/apache/kafka/common/test/api/README.md:
##########
@@ -79,7 +79,7 @@ a JUnit extension called `ClusterTestExtensions` which knows how to process thes
invocations. Test classes that wish to make use of these annotations need to explicitly register this extension:
```scala
-import org.apache.kafka.common.test.api.ClusterTestExtensions
+import org.apache.kafka.common.test.junit.ClusterTestExtensions
@ExtendWith(value = Array(classOf[ClusterTestExtensions]))
Review Comment:
we don't need it, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2599781560
I like "internal-api" 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2608120440
@ijuma @chia7712 did you have any further comments on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #18602:
URL: https://github.com/apache/kafka/pull/18602#discussion_r1921090955
##########
core/src/test/scala/unit/kafka/server/SaslApiVersionsRequestTest.scala:
##########
@@ -19,13 +19,15 @@ package kafka.server
import kafka.api.{KafkaSasl, SaslSetup}
import kafka.security.JaasTestUtils
import kafka.server.SaslApiVersionsRequestTest.{kafkaClientSaslMechanism, kafkaServerSaslMechanisms}
-import org.apache.kafka.common.test.api.{ClusterTemplate, Type, ClusterTestExtensions, ClusterConfig, ClusterInstance}
+import org.apache.kafka.common.test.api.{ClusterConfig, ClusterTemplate, Type}
import org.apache.kafka.common.config.SaslConfigs
import org.apache.kafka.common.config.internals.BrokerSecurityConfigs
import org.apache.kafka.common.message.SaslHandshakeRequestData
import org.apache.kafka.common.protocol.{ApiKeys, Errors}
import org.apache.kafka.common.requests.{ApiVersionsRequest, ApiVersionsResponse, SaslHandshakeRequest, SaslHandshakeResponse}
import org.apache.kafka.common.security.auth.SecurityProtocol
+import org.apache.kafka.common.test.ClusterInstance
+import org.apache.kafka.common.test.junit.ClusterTestExtensions
Review Comment:
Well this was a bad choice for a test since this one is disabled. I tried it locally on another test and verified keeping the annotation is harmless
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #18602:
URL: https://github.com/apache/kafka/pull/18602#discussion_r1920950898
##########
build.gradle:
##########
@@ -1364,6 +1365,9 @@ project(':metadata') {
testImplementation project(':raft').sourceSets.test.output
testImplementation project(':server-common').sourceSets.test.output
+ testImplementation project(':test-common:test-common-api')
+ testRuntimeOnly project(':test-common:test-common-runtime')
Review Comment:
This was just done to illustrate the approach (and test it in the build). I'll revert this bit before merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2599751995
Makes sense to use the `testkit` name for the external dependency and going with `util` for a more general module name that can include shared junit stuff but also other useful things needed by all test modules.
The only open question for me is the `test-common-api` name. Even though it makes sense, we typically use `api` for modules that expose apis externally (`group-coordinator-api`, `storage-api`, `tools-api`, etc.). In this case, it's a purely internal module, so I wonder if we can find a different name to describe it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2599771553
Since you're calling one `test-common-runtime`, would `test-common-compile` be appropriate? Or perhaps `test-common-internal-api` (we already use `internals` in our package names). The latter could be used for any module that exposed internal apis, so perhaps it a reasonable pattern to establish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2599523365
@chia7712 I think this commit should fix the initialization issue. Since SPI is being used, it loads the extension for all tests. We need some additional filtering to not apply the ClusterTest logic to regular tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2599721489
Thanks for the PR. A couple of questions:
1. Do we intend to have a common test module for clients and streams (aside from junit specific classes)?
2. If we were to provide a test module for external usage, what would we call it?
These questions will help validate the names we have chosen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17587 Refactor test infrastructure [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18602:
URL: https://github.com/apache/kafka/pull/18602#issuecomment-2610520998
@mumrah could you please fix the conflicts? Additionally, should we backport it to 4.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org