You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/07/26 06:31:04 UTC
[GitHub] [flink] PatrickRen opened a new pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
PatrickRen opened a new pull request #16590:
URL: https://github.com/apache/flink/pull/16590
## What is the purpose of the change
This pull request implements a unified testing framework for Flink connectors.
```README.md``` provides description of abstractions and usage of the framework.
## Brief change log
- Basic abstractions / interfaces of testing framework
- Framework support based on JUnit 5
- Base test class with fundamental test cases
- Kafka E2E test case based on testing framework
## Verifying this change
This change added tests and can be verified as follows:
- Kafka E2E test that can be triggered by using "-Prun-testing-framework" when verifying
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (JavaDocs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
}, {
"hash" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978",
"triggerID" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"triggerType" : "PUSH"
}, {
"hash" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22053",
"triggerID" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6705cab068becca06bae6aaa50258a08bbfbbd3d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978)
* de029efd3d3ff874bf43cc8b5a39df36c0181359 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22053)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-898500155
Thanks @AHeise for the approval! The NOTICE issue has been solved as suggested by @zentol. Hope the PR is good to be merged now.
Also welcome to have a try on the brand new connector testing framework!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38c3cd15cf9b06253125b57322247f2d1d7bf95e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722)
* 9b830de58276846f46b302298a4f5be6853fc645 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685078812
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/e2e/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.e2e;
+
+import org.apache.flink.connector.kafka.source.e2e.resources.KafkaContainerizedExternalSystem;
+import org.apache.flink.connector.kafka.source.e2e.resources.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.e2e.resources.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.environment.FlinkContainersTestEnvironment;
+import org.apache.flink.connectors.test.common.environment.MiniClusterTestEnvironment;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithTestEnvironment;
+import org.apache.flink.connectors.test.common.testsuites.TestSuiteBase;
+import org.apache.flink.connectors.test.common.utils.ConnectorJarUtils;
+
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Tag;
+
+/**
+ * End-to-end test for {@link org.apache.flink.connector.kafka.source.KafkaSource} using testing
+ * framework, based on JUnit 5.
+ *
+ * <p>This class uses {@link Nested} classes to test functionality of KafkaSource under different
+ * Flink environments (MiniCluster and Containers). Each nested class extends {@link TestSuiteBase}
+ * for reusing test cases already defined by testing framework.
+ */
+@Disabled
+@Tag("TestingFramework")
+@DisplayName("Kafka Source E2E Test")
+public class KafkaSourceE2ECase {
+
+ @Nested
+ @DisplayName("On MiniCluster")
+ class OnMiniCluster extends TestSuiteBase<String> {
Review comment:
I split IT and E2E cases into two class. IT case (on MiniCluster) stays in ```flink-connector-kafka``` module and E2E case (on Docker) goes to ```flink-end-to-end-tests-common-kafka```. This also solves the Flink image building issues above since E2E case can reused the existing ```FlinkContainer``` that can build the Flink image based on flink-dist on the fly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
}, {
"hash" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b830de58276846f46b302298a4f5be6853fc645 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760)
* 6705cab068becca06bae6aaa50258a08bbfbbd3d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688388087
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
If you delete the NOTICE file _and_ add the module to `modules-skipping-deployment.modulelist` then the checker should not complain.
(by being excluded we don't require a NOTICE file, and by not having a NOTICE file subsequent checks aren't done)
If it still does, please ping me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685072304
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
Review comment:
```DataStream#executeAndCollect``` doesn't return ```JobClient```, which is required in ```triggerTaskManagerFailure```. There's a method ```DataStream#executeAndCollectWithClient``` that can fulfill the requirement, but unfortunately this method is not public. Maybe we can make it public in another ticket (or FLIP since ```DataStream``` is public API)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685065126
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/junit/extensions/TestingFrameworkExtension.java
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.junit.extensions;
+
+import org.apache.flink.connectors.test.common.TestResource;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.ExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithTestEnvironment;
+
+import org.junit.jupiter.api.extension.AfterAllCallback;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.platform.commons.support.AnnotationSupport;
+
+import java.lang.annotation.Annotation;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * A JUnit 5 {@link Extension} for supporting running of testing framework.
+ *
+ * <p>This extension is responsible for searching test resources annotated by {@link
+ * WithTestEnvironment}, {@link WithExternalSystem} and {@link WithExternalContextFactory}, storing
+ * them into storage provided by JUnit, and manage lifecycle of these resources.
+ *
+ * <p>The extension uses {@link ExtensionContext.Store} for handing over test resources to {@link
+ * TestCaseInvocationContextProvider}, which will inject these resources into test cases.
+ */
+public class TestingFrameworkExtension implements BeforeAllCallback, AfterAllCallback {
Review comment:
Renamed to ```ConnectorTestingExtension```. I'd like to use it just for connector testing framework because this extension needs to work together with ```TestCaseInvocationContextProvider```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
}, {
"hash" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978",
"triggerID" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"triggerType" : "PUSH"
}, {
"hash" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6705cab068becca06bae6aaa50258a08bbfbbd3d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978)
* de029efd3d3ff874bf43cc8b5a39df36c0181359 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686663605
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
Review comment:
I agree that the method may be too complex for end-users. I think we should leave things as is until we have a better idea. Adding `@Internal` on 1 method in a public API feels weird to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686943185
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.kafka;
+
+import org.apache.flink.connector.kafka.source.testutils.KafkaContainerizedExternalSystem;
+import org.apache.flink.connector.kafka.source.testutils.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.testutils.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithTestEnvironment;
+import org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase;
+import org.apache.flink.tests.util.TestUtils;
+import org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment;
+
+/** Kafka E2E test based on connector testing framework. */
+public class KafkaSourceE2ECase extends SourceTestSuiteBase<String> {
+
+ // Defines TestEnvironment
+ @WithTestEnvironment public FlinkContainerTestEnvironment flink;
+
+ // Defines ExternalSystem
+ @WithExternalSystem public KafkaContainerizedExternalSystem kafka;
+
+ // Defines 2 External context Factories, so test cases will be invoked twice using these two
+ // kinds of external contexts.
+ @WithExternalContextFactory public KafkaSingleTopicExternalContext.Factory singleTopic;
+ @WithExternalContextFactory public KafkaMultipleTopicExternalContext.Factory multipleTopic;
+
+ /** Instantiate and preparing test resources. */
+ public KafkaSourceE2ECase() {
Review comment:
I think constructor _should_ be the place doing test resource initialization.
By adding abstract method in ```SourceTestSuiteBase``` do you mean something like "initialize()"? I think it just has the same semantic as using the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93776a6d6881dc65de98a7c7337ebd6189b61414 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
}, {
"hash" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978",
"triggerID" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"triggerType" : "PUSH"
}, {
"hash" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22053",
"triggerID" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"triggerType" : "PUSH"
}, {
"hash" : "0244b788e6bba53190e6049e17db7efc4907654f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22100",
"triggerID" : "0244b788e6bba53190e6049e17db7efc4907654f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0244b788e6bba53190e6049e17db7efc4907654f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22100)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685810346
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
+
+ checkSingleSplitRecords(
+ testRecordsBeforeFailure.iterator(), iterator, testRecordsBeforeFailure.size());
+
+ // -------------------------------- Trigger failover ---------------------------------------
+ final ClusterControllable controller = (ClusterControllable) testEnv;
+ controller.triggerTaskManagerFailover(jobClient, () -> {});
+
+ JobStatusUtils.waitForJobStatus(
+ jobClient, Collections.singletonList(JobStatus.RUNNING), Duration.ofSeconds(30));
+
+ final Collection<T> testRecordsAfterFailure = externalContext.generateTestData();
Review comment:
I think `generateTestData` should take a seed as a parameter so that we are sure that the implementation does not generate the same data. The test may be a false positive otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688385319
##########
File path: flink-test-utils-parent/flink-connector-testing/README.md
##########
@@ -0,0 +1,44 @@
+# Flink Connector Testing Framework
Review comment:
We should eventually add this (and expand) to docs. However, I prefer to first use it internally for a while until we advertise it. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688017469
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.kafka;
+
+import org.apache.flink.connector.kafka.source.testutils.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.testutils.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.external.DefaultContainerizedExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.ExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.ExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.TestEnv;
+import org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase;
+import org.apache.flink.tests.util.TestUtils;
+import org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment;
+
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** Kafka E2E test based on connector testing framework. */
+public class KafkaSourceE2ECase extends SourceTestSuiteBase<String> {
Review comment:
This should be
```
public class KafkaSourceE2ECase extends SourceTestSuiteBase<String> {
private static final String KAFKA_HOSTNAME = "kafka";
private static final String KAFKA_IMAGE_NAME = "confluentinc/cp-kafka:5.5.2";
// Defines TestEnvironment
@TestEnv FlinkContainerTestEnvironment flink =
new FlinkContainerTestEnvironment(
1,
6,
TestUtils.getResource("kafka-connector.jar").toAbsolutePath().toString(),
TestUtils.getResource("kafka-clients.jar").toAbsolutePath().toString());
// Defines ConnectorExternalSystem
@ExternalSystem DefaultContainerizedExternalSystem<KafkaContainer> kafka =
DefaultContainerizedExternalSystem.builder()
.fromContainer(new KafkaContainer(DockerImageName.parse(KAFKA_IMAGE_NAME))
.withNetworkAliases(KAFKA_HOSTNAME))
.bindWithFlinkContainer(flink.getFlinkContainer())
.build();
// Defines 2 External context Factories, so test cases will be invoked twice using these two
// kinds of external contexts.
@ExternalContextFactory KafkaSingleTopicExternalContext.Factory singleTopic = new KafkaSingleTopicExternalContext.Factory(kafka.getContainer()); // or even just kafka
@ExternalContextFactory KafkaMultipleTopicExternalContext.Factory multipleTopic = new KafkaMultipleTopicExternalContext.Factory(kafka.getContainer());
}
```
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/external/DefaultContainerizedExternalSystem.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.external;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.connectors.test.common.TestResource;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+
+/**
+ * Default implementation of external system based on container.
+ *
+ * @param <C> Type of underlying container
+ */
+@Experimental
+public class DefaultContainerizedExternalSystem<C extends GenericContainer<C>>
+ implements TestResource {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DefaultContainerizedExternalSystem.class);
+
+ private final GenericContainer<C> container;
Review comment:
```suggestion
private final C container;
```
Same with getter.
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/testutils/KafkaMultipleTopicExternalContext.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.testutils;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.connector.kafka.source.KafkaSource;
+import org.apache.flink.connector.kafka.source.KafkaSourceBuilder;
+import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
+import org.apache.flink.connector.kafka.source.reader.deserializer.KafkaRecordDeserializationSchema;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.regex.Pattern;
+
+/**
+ * Kafka external context that will create multiple topics with only one partitions as source
+ * splits.
+ */
+public class KafkaMultipleTopicExternalContext extends KafkaSingleTopicExternalContext {
+
+ private int numTopics = 0;
+
+ private final String topicPattern;
+
+ private final Map<String, SourceSplitDataWriter<String>> topicNameToSplitWriters =
+ new HashMap<>();
+
+ public KafkaMultipleTopicExternalContext(String bootstrapServers) {
+ super(bootstrapServers);
+ this.topicPattern =
+ "kafka-multiple-topic-[0-9]+-"
+ + ThreadLocalRandom.current().nextLong(Long.MAX_VALUE);
+ }
+
+ @Override
+ public SourceSplitDataWriter<String> createSourceSplitDataWriter() {
+ String topicName = getTopicName();
+ createTopic(topicName, 1, (short) 1);
+ final KafkaPartitionDataWriter splitWriter =
+ new KafkaPartitionDataWriter(
+ getKafkaProducerProperties(numTopics), new TopicPartition(topicName, 0));
+ topicNameToSplitWriters.put(topicName, splitWriter);
+ numTopics++;
+ return splitWriter;
+ }
+
+ @Override
+ public Source<String, ?, ?> createSource(Boundedness boundedness) {
+ KafkaSourceBuilder<String> builder = KafkaSource.builder();
+
+ if (boundedness == Boundedness.BOUNDED) {
+ builder = builder.setBounded(OffsetsInitializer.latest());
+ }
+
+ return builder.setGroupId("flink-kafka-multiple-topic-test")
+ .setBootstrapServers(bootstrapServers)
+ .setTopicPattern(Pattern.compile(topicPattern))
+ .setDeserializer(
+ KafkaRecordDeserializationSchema.valueOnly(StringDeserializer.class))
+ .build();
+ }
+
+ @Override
+ public void close() {
+ topicNameToSplitWriters.forEach(
+ (topicName, splitWriter) -> {
+ try {
+ splitWriter.close();
+ deleteTopic(topicName);
+ } catch (Exception e) {
+ kafkaAdminClient.close();
+ throw new RuntimeException("Cannot close split writer", e);
+ }
+ });
+ topicNameToSplitWriters.clear();
+ kafkaAdminClient.close();
+ }
+
+ private String getTopicName() {
+ return topicPattern.replace("[0-9]+", String.valueOf(numTopics));
+ }
+
+ @Override
+ public String toString() {
+ return "Multiple-topics Kafka";
+ }
+
+ /** . */
+ public static class Factory implements ExternalContext.Factory<String> {
+
+ private final String bootstrapServers;
+
+ public Factory(String bootstrapServers) {
Review comment:
Here we should just get the `KafkaContainer` (or even `ExternalResource<KafkaContainer>`).
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.kafka;
+
+import org.apache.flink.connector.kafka.source.testutils.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.testutils.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.external.DefaultContainerizedExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.ExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.ExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.TestEnv;
+import org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase;
+import org.apache.flink.tests.util.TestUtils;
+import org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment;
+
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** Kafka E2E test based on connector testing framework. */
+public class KafkaSourceE2ECase extends SourceTestSuiteBase<String> {
+ private static final String KAFKA_HOSTNAME = "kafka";
+ private static final String KAFKA_IMAGE_NAME = "confluentinc/cp-kafka:5.5.2";
+
+ // Defines TestEnvironment
+ @TestEnv FlinkContainerTestEnvironment flink;
+
+ // Defines ConnectorExternalSystem
+ @ExternalSystem DefaultContainerizedExternalSystem<KafkaContainer> kafka;
+
+ // Defines 2 External context Factories, so test cases will be invoked twice using these two
+ // kinds of external contexts.
+ @ExternalContextFactory KafkaSingleTopicExternalContext.Factory singleTopic;
+ @ExternalContextFactory KafkaMultipleTopicExternalContext.Factory multipleTopic;
+
+ /** Instantiate and preparing test resources. */
+ public KafkaSourceE2ECase() throws Exception {
+
+ // Flink on containers
+ flink =
+ new FlinkContainerTestEnvironment(
+ 1,
+ 6,
+ TestUtils.getResource("kafka-connector.jar").toAbsolutePath().toString(),
+ TestUtils.getResource("kafka-clients.jar").toAbsolutePath().toString());
+
+ // Kafka on container has to be bound with Flink containers so that they can access each
+ // other through internal network, so we have to launch Flink containers here.
+ flink.startUp();
Review comment:
This should be unnecessary. Isn't the dependsOn in `Builder#bindWithFlinkContainer` enough?
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/testutils/KafkaMultipleTopicExternalContext.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.testutils;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.connector.kafka.source.KafkaSource;
+import org.apache.flink.connector.kafka.source.KafkaSourceBuilder;
+import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
+import org.apache.flink.connector.kafka.source.reader.deserializer.KafkaRecordDeserializationSchema;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.regex.Pattern;
+
+/**
+ * Kafka external context that will create multiple topics with only one partitions as source
+ * splits.
+ */
+public class KafkaMultipleTopicExternalContext extends KafkaSingleTopicExternalContext {
+
+ private int numTopics = 0;
+
+ private final String topicPattern;
+
+ private final Map<String, SourceSplitDataWriter<String>> topicNameToSplitWriters =
+ new HashMap<>();
+
+ public KafkaMultipleTopicExternalContext(String bootstrapServers) {
+ super(bootstrapServers);
+ this.topicPattern =
+ "kafka-multiple-topic-[0-9]+-"
+ + ThreadLocalRandom.current().nextLong(Long.MAX_VALUE);
+ }
+
+ @Override
+ public SourceSplitDataWriter<String> createSourceSplitDataWriter() {
+ String topicName = getTopicName();
+ createTopic(topicName, 1, (short) 1);
+ final KafkaPartitionDataWriter splitWriter =
+ new KafkaPartitionDataWriter(
+ getKafkaProducerProperties(numTopics), new TopicPartition(topicName, 0));
+ topicNameToSplitWriters.put(topicName, splitWriter);
+ numTopics++;
+ return splitWriter;
+ }
+
+ @Override
+ public Source<String, ?, ?> createSource(Boundedness boundedness) {
+ KafkaSourceBuilder<String> builder = KafkaSource.builder();
+
+ if (boundedness == Boundedness.BOUNDED) {
+ builder = builder.setBounded(OffsetsInitializer.latest());
+ }
+
+ return builder.setGroupId("flink-kafka-multiple-topic-test")
+ .setBootstrapServers(bootstrapServers)
+ .setTopicPattern(Pattern.compile(topicPattern))
+ .setDeserializer(
+ KafkaRecordDeserializationSchema.valueOnly(StringDeserializer.class))
+ .build();
+ }
+
+ @Override
+ public void close() {
+ topicNameToSplitWriters.forEach(
+ (topicName, splitWriter) -> {
+ try {
+ splitWriter.close();
+ deleteTopic(topicName);
+ } catch (Exception e) {
+ kafkaAdminClient.close();
+ throw new RuntimeException("Cannot close split writer", e);
+ }
+ });
+ topicNameToSplitWriters.clear();
+ kafkaAdminClient.close();
+ }
+
+ private String getTopicName() {
+ return topicPattern.replace("[0-9]+", String.valueOf(numTopics));
+ }
+
+ @Override
+ public String toString() {
+ return "Multiple-topics Kafka";
+ }
+
+ /** . */
+ public static class Factory implements ExternalContext.Factory<String> {
+
+ private final String bootstrapServers;
+
+ public Factory(String bootstrapServers) {
+ this.bootstrapServers = bootstrapServers;
+ }
+
+ @Override
+ public ExternalContext<String> createExternalContext() {
+ return new KafkaMultipleTopicExternalContext(bootstrapServers);
Review comment:
Here we should extract the bootstrap server. (I'm assuming now that the external resources are running or we can easily force it while calling this factory)
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/external/DefaultContainerizedExternalSystem.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.external;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.connectors.test.common.TestResource;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+
+/**
+ * Default implementation of external system based on container.
+ *
+ * @param <C> Type of underlying container
+ */
+@Experimental
+public class DefaultContainerizedExternalSystem<C extends GenericContainer<C>>
+ implements TestResource {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DefaultContainerizedExternalSystem.class);
+
+ private final GenericContainer<C> container;
+
+ /**
+ * Get a builder for {@link DefaultContainerizedExternalSystem}.
+ *
+ * @param <C> Type of underlying container
+ * @return An instance of builder
+ */
+ public static <C extends GenericContainer<C>> Builder<C> builder() {
+ return new Builder<>();
+ }
+
+ private DefaultContainerizedExternalSystem(GenericContainer<C> container) {
+ this.container = container;
+ }
+
+ @Override
+ public void startUp() throws Exception {
+ if (container.isRunning()) {
+ return;
+ }
+ container.start();
+ }
+
+ @Override
+ public void tearDown() throws Exception {
+ if (!container.isRunning()) {
+ return;
+ }
+ container.stop();
+ }
+
+ public GenericContainer<C> getContainer() {
+ return this.container;
+ }
+
+ /**
+ * Builder for {@link DefaultContainerizedExternalSystem}.
+ *
+ * @param <C> Type of underlying container
+ */
+ public static class Builder<C extends GenericContainer<C>> {
+
+ private GenericContainer<C> container;
+ private GenericContainer<?> flinkContainer;
+
+ public Builder<C> fromContainer(GenericContainer<C> container) {
+ this.container = container;
+ return this;
+ }
Review comment:
```suggestion
public <T extends GenericContainer<T>> Builder<T> fromContainer(
GenericContainer<T> container) {
@SuppressWarnings("unchecked")
Builder<T> self = (Builder<T>) this;
self.container = container;
return self;
}
```
This allows us to define kafka without `<KafkaContainer>builder`
```
DefaultContainerizedExternalSystem<KafkaContainer> kafka =
DefaultContainerizedExternalSystem.builder()
.fromContainer(
new KafkaContainer(DockerImageName.parse(KAFKA_IMAGE_NAME))
.withNetworkAliases(KAFKA_HOSTNAME)) // captures the type here!
.bindWithFlinkContainer(flink.getFlinkContainer())
.build();
```
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.kafka;
+
+import org.apache.flink.connector.kafka.source.testutils.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.testutils.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.external.DefaultContainerizedExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.ExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.ExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.TestEnv;
+import org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase;
+import org.apache.flink.tests.util.TestUtils;
+import org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment;
+
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** Kafka E2E test based on connector testing framework. */
+public class KafkaSourceE2ECase extends SourceTestSuiteBase<String> {
+ private static final String KAFKA_HOSTNAME = "kafka";
+ private static final String KAFKA_IMAGE_NAME = "confluentinc/cp-kafka:5.5.2";
+
+ // Defines TestEnvironment
+ @TestEnv FlinkContainerTestEnvironment flink;
+
+ // Defines ConnectorExternalSystem
+ @ExternalSystem DefaultContainerizedExternalSystem<KafkaContainer> kafka;
+
+ // Defines 2 External context Factories, so test cases will be invoked twice using these two
+ // kinds of external contexts.
+ @ExternalContextFactory KafkaSingleTopicExternalContext.Factory singleTopic;
+ @ExternalContextFactory KafkaMultipleTopicExternalContext.Factory multipleTopic;
+
+ /** Instantiate and preparing test resources. */
+ public KafkaSourceE2ECase() throws Exception {
+
+ // Flink on containers
+ flink =
+ new FlinkContainerTestEnvironment(
+ 1,
+ 6,
+ TestUtils.getResource("kafka-connector.jar").toAbsolutePath().toString(),
+ TestUtils.getResource("kafka-clients.jar").toAbsolutePath().toString());
+
+ // Kafka on container has to be bound with Flink containers so that they can access each
+ // other through internal network, so we have to launch Flink containers here.
+ flink.startUp();
+
+ final KafkaContainer kafkaContainer =
+ new KafkaContainer(DockerImageName.parse(KAFKA_IMAGE_NAME))
+ .withNetworkAliases(KAFKA_HOSTNAME);
+ kafka =
+ DefaultContainerizedExternalSystem.<KafkaContainer>builder()
+ .fromContainer(kafkaContainer)
+ .bindWithFlinkContainer(flink.getFlinkContainer())
+ .build();
+
+ // The construction of Kafka external context requires bootstrap servers of Kafka, so we
+ // start Kafka cluster here to get bootstrap servers for the external context.
+ // Kafka on container
+ kafka.startUp();
Review comment:
This should be unnecessary as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686960360
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/testutils/KafkaSourceSplitDataWriter.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.testutils;
+
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.TopicPartition;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Properties;
+
+/** Source split data writer for writing test data into Kafka topic partitions. */
+public class KafkaSourceSplitDataWriter implements SourceSplitDataWriter<String> {
Review comment:
```KafkaPartitionDataWriter``` is a good choice. I'll change to it.
For ```SourceSplitDataWriter``` I'd like to keep its name. I think ```Partition``` is too "Kafka-specific". As users of connector testing framework are actually developers of connector, I assume they have to know the concept of ```SourceSplit```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688379592
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
The notice checker _does not_ use the output from the dependency plugin, but the output of the shade-plugin. If the notice checker complainted, please link a build where this happened.
Note: The _scala suffix checker_ uses the dependency plugin output.
If a given module is not deployed but does bundled dependencies, then you can exclude it from the notice checks in `tools/ci/java-ci-tools/src/main/resources/modules-skipping-deployment.modulelist`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 027d4ab260b3914c6847704641229f13686e7ffe Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948)
* 8ba223523eb708232b5fe86eb162fb1f36e85ffc UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688379592
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
The notice checker _does not_ use the output from the dependency plugin, but the output of the shade-plugin. If the notice checker complained, please link a build where this happened.
Note: The _scala suffix checker_ uses the dependency plugin output.
If a given module is not deployed but does bundled dependencies, then you can exclude it from the notice checks in `tools/ci/java-ci-tools/src/main/resources/modules-skipping-deployment.modulelist`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688009385
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.kafka;
+
+import org.apache.flink.connector.kafka.source.testutils.KafkaContainerizedExternalSystem;
+import org.apache.flink.connector.kafka.source.testutils.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.testutils.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithTestEnvironment;
+import org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase;
+import org.apache.flink.tests.util.TestUtils;
+import org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment;
+
+/** Kafka E2E test based on connector testing framework. */
+public class KafkaSourceE2ECase extends SourceTestSuiteBase<String> {
+
+ // Defines TestEnvironment
+ @WithTestEnvironment public FlinkContainerTestEnvironment flink;
+
+ // Defines ExternalSystem
+ @WithExternalSystem public KafkaContainerizedExternalSystem kafka;
+
+ // Defines 2 External context Factories, so test cases will be invoked twice using these two
+ // kinds of external contexts.
+ @WithExternalContextFactory public KafkaSingleTopicExternalContext.Factory singleTopic;
+ @WithExternalContextFactory public KafkaMultipleTopicExternalContext.Factory multipleTopic;
+
+ /** Instantiate and preparing test resources. */
+ public KafkaSourceE2ECase() {
Review comment:
You have a fair point, but now I know exactly why this looks very confusing to me: usually rules etc are initialized directly on the fields. And indeed no values are injected such that all initialization could already happen on the fields which I would very much prefer.
That we need to initialize these things manually in a certain order indicates to me that we still have not abstracted correctly. Test resources, rules, and contexts should be defined decoratively but now we have to resort to procedural definition again. I'll add annotations to the specific points.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688558323
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
Thanks @zentol for the solution! I put ```flink-end-to-end-tests-common-kafka``` into ```modules-skipping-deployment.modulelist``` and the compile phase has succeeded 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886418776
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 027d4ab260b3914c6847704641229f13686e7ffe (Mon Jul 26 06:34:38 UTC 2021)
**Warnings:**
* **14 pom.xml files were touched**: Check for build and licensing issues.
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-19554).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686955548
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
```KafkaSourceE2ECase``` requires a JAR of ```kafka-clients``` in the target/dependencies, so I added it in the ```maven-dependency-plugin```.
Actually it is not packaged into our JAR, but our ```NoticeFileChecker``` is validating NOTICE file according to the output of ```maven-dependency-plugin```, even the dependency is not packaged in the JAR.
Usage of ```kafka-clients``` in ```KafkaSourceE2ECase```:
https://github.com/apache/flink/pull/16590/files#diff-87acbe4433c51168a661d68dd2c639d22dada8da8b8b890e395e61c4a5ad28eaR54
Usage of ```kafka-clients``` in ```maven-dependency-plugin```:
https://github.com/apache/flink/pull/16590/files#diff-5ed1afac5906e91c880091dee5a33edb9d53aeb0f1f94a88d67319a3844cb43dR225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ba223523eb708232b5fe86eb162fb1f36e85ffc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709)
* 93776a6d6881dc65de98a7c7337ebd6189b61414 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686932759
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/junit/annotations/WithTestEnvironment.java
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.junit.annotations;
+
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Marks the field in test class defining {@link TestEnvironment}.
+ *
+ * <p>Only one field can be annotated as test environment in a test class.
+ *
+ * <p>The lifecycle of {@link TestEnvironment} will be PER-CLASS for performance, because launching
+ * and tearing down Flink cluster could be relatively a heavy operation.
+ */
+@Target(ElementType.FIELD)
+@Retention(RetentionPolicy.RUNTIME)
+public @interface WithTestEnvironment {}
Review comment:
Changed to ```@TestEnv```, ```@ExternalSystem``` and ```@ExternalContextFactory```. "With" was used for avoiding name collision between annotations and interfaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688384203
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
The notice checker only partially ignores modules that are listed in modules-skipping-deployment.modulelist.
Let me think about it a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688005830
##########
File path: flink-end-to-end-tests/flink-streaming-kinesis-test/pom.xml
##########
@@ -151,6 +151,23 @@ under the License.
</systemPropertyVariables>
</configuration>
</plugin>
+
+ <!-- Skip dependency convergence check because of guava version -->
+ <plugin>
Review comment:
Okay yes, all good. We are already skipping it in all other e2e tests of connectors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688385319
##########
File path: flink-test-utils-parent/flink-connector-testing/README.md
##########
@@ -0,0 +1,44 @@
+# Flink Connector Testing Framework
Review comment:
We should eventually have add this (and expand) to docs. However, I prefer to first use it internally for a while until we advertise it. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685812389
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/e2e/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.e2e;
+
+import org.apache.flink.connector.kafka.source.e2e.resources.KafkaContainerizedExternalSystem;
+import org.apache.flink.connector.kafka.source.e2e.resources.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.e2e.resources.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.environment.FlinkContainersTestEnvironment;
+import org.apache.flink.connectors.test.common.environment.MiniClusterTestEnvironment;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithTestEnvironment;
+import org.apache.flink.connectors.test.common.testsuites.TestSuiteBase;
+import org.apache.flink.connectors.test.common.utils.ConnectorJarUtils;
+
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Tag;
+
+/**
+ * End-to-end test for {@link org.apache.flink.connector.kafka.source.KafkaSource} using testing
+ * framework, based on JUnit 5.
+ *
+ * <p>This class uses {@link Nested} classes to test functionality of KafkaSource under different
+ * Flink environments (MiniCluster and Containers). Each nested class extends {@link TestSuiteBase}
+ * for reusing test cases already defined by testing framework.
+ */
+@Disabled
+@Tag("TestingFramework")
+@DisplayName("Kafka Source E2E Test")
+public class KafkaSourceE2ECase {
+
+ @Nested
+ @DisplayName("On MiniCluster")
+ class OnMiniCluster extends TestSuiteBase<String> {
Review comment:
I think this is a good solution for now but I'd like to revisit it later; it would be lead to a module explosion if we add a new e2e module for all connectors under test. But for Kafka, this works well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
}, {
"hash" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978",
"triggerID" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"triggerType" : "PUSH"
}, {
"hash" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22053",
"triggerID" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"triggerType" : "PUSH"
}, {
"hash" : "0244b788e6bba53190e6049e17db7efc4907654f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0244b788e6bba53190e6049e17db7efc4907654f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de029efd3d3ff874bf43cc8b5a39df36c0181359 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22053)
* 0244b788e6bba53190e6049e17db7efc4907654f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93776a6d6881dc65de98a7c7337ebd6189b61414 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717)
* 38c3cd15cf9b06253125b57322247f2d1d7bf95e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-897511579
Thanks again for the review @AHeise ! I made another push based on your comments. Do you mind having another review on the latest push? Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685067902
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
Review comment:
Actually these cases are test templates instead of tests, so we can't use ```@Test``` here. ```@Case``` is a wrapper for ```@TestTemplate``` and ```@ExtendWith(TestCaseInvocationContextProvider.class)```. Using ```@Case``` can avoid all test cases being annotated with these two long annotations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688388087
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
If you delete the NOTICE file _and_ add the module to `modules-skipping-deployment.modulelist` then the checker should not complain.
(by being excluded we don't require a NOTICE file, and by not having a NOTICE file subsequent checks aren't done)
If it still complains, please ping me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
}, {
"hash" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978",
"triggerID" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"triggerType" : "PUSH"
}, {
"hash" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22053",
"triggerID" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"triggerType" : "PUSH"
}, {
"hash" : "0244b788e6bba53190e6049e17db7efc4907654f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22100",
"triggerID" : "0244b788e6bba53190e6049e17db7efc4907654f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de029efd3d3ff874bf43cc8b5a39df36c0181359 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22053)
* 0244b788e6bba53190e6049e17db7efc4907654f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22100)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise merged pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise merged pull request #16590:
URL: https://github.com/apache/flink/pull/16590
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685814603
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/pom.xml
##########
@@ -49,8 +49,15 @@ under the License.
<scope>compile</scope>
</dependency>
<dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
+ <groupId>org.junit.jupiter</groupId>
Review comment:
Don't forget to remove this PR once #16551 is merged. (It still uses migrationsupport)
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/TestResource.java
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common;
+
+/** Test resources. */
+public interface TestResource {
Review comment:
Please add documentation to this and all other classes that are supposed to be used/extended by connector developers.
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/external/ExternalContext.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.external;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.Source;
+
+import java.io.Serializable;
+import java.util.Collection;
+
+/**
+ * Context of the test interacting with external system.
+ *
+ * <p>This context is responsible for providing:
+ *
+ * <ul>
+ * <li>Instance of sources connecting to external system
+ * <li>{@link SourceSplitDataWriter} for creating splits and writing test data into the created
+ * split
+ * <li>Test data to write into the external system
+ * </ul>
+ *
+ * @param <T> Type of elements after deserialization by source
+ */
+public interface ExternalContext<T> extends Serializable, AutoCloseable {
+
+ /**
+ * Create a new instance of connector source implemented in {@link Source}.
+ *
+ * @return A new instance of Source
+ */
+ Source<T, ?, ?> createSource(Boundedness boundedness);
+
+ /**
+ * Create a new split in the external system.
+ *
+ * @return A data writer for the created split.
+ */
+ SourceSplitDataWriter<T> createSourceSplit();
+
+ /**
+ * Generate test data.
+ *
+ * @return Collection of generated test data.
+ */
+ Collection<T> generateTestData();
Review comment:
Maybe add seed or even `Random` as a parameter? That could also help us to have reproducible tests in case of issues with data generation or handling of special cases.
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/TestResource.java
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common;
+
+/** Test resources. */
+public interface TestResource {
+
+ void startUp() throws Exception;
+
+ void tearDown() throws Exception;
+
+ boolean isStarted();
Review comment:
Do we actually need this method? We could define `startUp` as idempotent and allow `tearDown` without `startUp`. `MiniClusterTestEnvironment` and `RemoteClusterTestEnvironment` don't really need it.
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/external/ContainerizedExternalSystem.java
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.external;
+
+import org.testcontainers.containers.GenericContainer;
+
+/** External system using <a href="https://www.testcontainers.org/">Testcontainers</a>. */
+public interface ContainerizedExternalSystem<C extends ContainerizedExternalSystem<?>>
Review comment:
I think we should also provide some `AbstractContainerizedExternalSystem`. I see lots of things in `KafkaContainerizedExternalSystem` probably need to be replicated for other containers.
Maybe it would even be better to just use one `DefaultContainerizedExternalSystem` and let the connectors use a builder to configure that. Except for `getBootstrapServer` the kafka container is not adding any specific behavior and that could probably also be moved in `ExternalContext`.
I guess it depends on how you want to position the two concepts `ExternalSystem` and `ExternalContext`. I think a natural separation would be that `ExternalSystem` is only responsible for life-cycle and `ExternalContext` for anything that you can do with a live system.
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/external/ExternalSystem.java
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.external;
+
+import org.apache.flink.connectors.test.common.TestResource;
+
+/** General external system for end-to-end testing. */
+public interface ExternalSystem extends TestResource {}
Review comment:
Link to annotation or else this interface looks quite useless.
##########
File path: flink-end-to-end-tests/flink-streaming-kinesis-test/pom.xml
##########
@@ -151,6 +151,23 @@ under the License.
</systemPropertyVariables>
</configuration>
</plugin>
+
+ <!-- Skip dependency convergence check because of guava version -->
+ <plugin>
Review comment:
I'm assuming we still want to converge here or at least only exclude Guava.
Could you please check if this legit? @zentol
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common/pom.xml
##########
@@ -65,6 +65,11 @@ under the License.
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
+ <dependency>
Review comment:
Could you please update the commit message?
I'd also splits the container modifications and the test environments into 2 commits.
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/external/ExternalContext.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.external;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.Source;
+
+import java.io.Serializable;
+import java.util.Collection;
+
+/**
+ * Context of the test interacting with external system.
+ *
+ * <p>This context is responsible for providing:
+ *
+ * <ul>
+ * <li>Instance of sources connecting to external system
+ * <li>{@link SourceSplitDataWriter} for creating splits and writing test data into the created
+ * split
+ * <li>Test data to write into the external system
+ * </ul>
+ *
+ * @param <T> Type of elements after deserialization by source
+ */
+public interface ExternalContext<T> extends Serializable, AutoCloseable {
+
+ /**
+ * Create a new instance of connector source implemented in {@link Source}.
+ *
+ * @return A new instance of Source
+ */
+ Source<T, ?, ?> createSource(Boundedness boundedness);
+
+ /**
+ * Create a new split in the external system.
+ *
+ * @return A data writer for the created split.
+ */
+ SourceSplitDataWriter<T> createSourceSplit();
Review comment:
`createSourceSplitDataWriter`? Or maybe it's actually more about writing partitions.
`SourceSplit`s don't make any sense outside of the `Source`.
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/testsuites/SourceTestSuiteBase.java
##########
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.ConnectorTestingExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({ConnectorTestingExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class SourceTestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SourceTestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test")) {
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 4 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test")) {
+ // Check test result
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with an idle reader.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testIdleReader(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ try (CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test")) {
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
Review comment:
constant for tag?
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.kafka;
+
+import org.apache.flink.connector.kafka.source.testutils.KafkaContainerizedExternalSystem;
+import org.apache.flink.connector.kafka.source.testutils.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.testutils.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithTestEnvironment;
+import org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase;
+import org.apache.flink.tests.util.TestUtils;
+import org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment;
+
+/** Kafka E2E test based on connector testing framework. */
+public class KafkaSourceE2ECase extends SourceTestSuiteBase<String> {
+
+ // Defines TestEnvironment
+ @WithTestEnvironment public FlinkContainerTestEnvironment flink;
+
+ // Defines ExternalSystem
+ @WithExternalSystem public KafkaContainerizedExternalSystem kafka;
+
+ // Defines 2 External context Factories, so test cases will be invoked twice using these two
+ // kinds of external contexts.
+ @WithExternalContextFactory public KafkaSingleTopicExternalContext.Factory singleTopic;
+ @WithExternalContextFactory public KafkaMultipleTopicExternalContext.Factory multipleTopic;
+
+ /** Instantiate and preparing test resources. */
+ public KafkaSourceE2ECase() {
Review comment:
I'm wondering if initialization in ctor is really the best option (In general, I like to initialize as much as possible in ctor and make the fields final).
We could also have an abstract method in `SourceTestSuiteBase` that needs to be overwritten.
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.kafka;
+
+import org.apache.flink.connector.kafka.source.testutils.KafkaContainerizedExternalSystem;
+import org.apache.flink.connector.kafka.source.testutils.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.testutils.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithTestEnvironment;
+import org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase;
+import org.apache.flink.tests.util.TestUtils;
+import org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment;
+
+/** Kafka E2E test based on connector testing framework. */
+public class KafkaSourceE2ECase extends SourceTestSuiteBase<String> {
+
+ // Defines TestEnvironment
+ @WithTestEnvironment public FlinkContainerTestEnvironment flink;
Review comment:
Afaik these fields don't need to be public anymore in JUnit5. Could you double-check and adjust?
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
Are we bundling `kafka-clients` anywhere? (Do we take any part of the jar and put it into ours?)
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/testutils/KafkaSourceSplitDataWriter.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.testutils;
+
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.TopicPartition;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Properties;
+
+/** Source split data writer for writing test data into Kafka topic partitions. */
+public class KafkaSourceSplitDataWriter implements SourceSplitDataWriter<String> {
Review comment:
Yay I think `PartitionDataWriter` and `KafkaParittionDataWriter` make more sense.
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/junit/annotations/WithTestEnvironment.java
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.junit.annotations;
+
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Marks the field in test class defining {@link TestEnvironment}.
+ *
+ * <p>Only one field can be annotated as test environment in a test class.
+ *
+ * <p>The lifecycle of {@link TestEnvironment} will be PER-CLASS for performance, because launching
+ * and tearing down Flink cluster could be relatively a heavy operation.
+ */
+@Target(ElementType.FIELD)
+@Retention(RetentionPolicy.RUNTIME)
+public @interface WithTestEnvironment {}
Review comment:
In the actual test, `WithTestEnvironment` reads weird to me.
`@WithTestEnvironment public FlinkContainerTestEnvironment flink;`
I have also not found any popular extension that uses `With...` but I have also only limited experience.
Ideally we would just call it `TestEnvironment`.
`@TestEnvironment public FlinkContainerTestEnvironment flink;`
However, that may be confusing. Another Option inspired by `@TempDir` would be
`@TestEnv public FlinkContainerTestEnvironment flink;`
I have the same concerns for the other annotations where I'd drop the `With`.
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/KafkaSourceITCase.java
##########
@@ -197,6 +207,45 @@ public void testRedundantParallelism() throws Exception {
executeAndVerify(env, stream);
}
+ /** Integration test based on connector testing framework. */
+ @Nested
+ class OnConnectorTestingFramework extends SourceTestSuiteBase<String> {
Review comment:
The name here is weird. Could we simply call the class `IntegrationTests`?
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/testsuites/SourceTestSuiteBase.java
##########
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.ConnectorTestingExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({ConnectorTestingExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class SourceTestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SourceTestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test")) {
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 4 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test")) {
+ // Check test result
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with an idle reader.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testIdleReader(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ try (CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test")) {
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(
+ TestEnvironment testEnv,
+ ExternalContext<T> externalContext,
+ ClusterControllable controller)
+ throws Exception {
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
+
+ checkSingleSplitRecords(
+ testRecordsBeforeFailure.iterator(), iterator, testRecordsBeforeFailure.size());
+
+ // -------------------------------- Trigger failover ---------------------------------------
+ controller.triggerTaskManagerFailover(jobClient, () -> {});
+
+ CommonTestUtils.waitForJobStatus(
+ jobClient,
+ Collections.singletonList(JobStatus.RUNNING),
+ Deadline.fromNow(Duration.ofSeconds(30)));
+
+ final Collection<T> testRecordsAfterFailure = externalContext.generateTestData();
+ sourceSplitDataWriter.writeRecords(testRecordsAfterFailure);
+ checkSingleSplitRecords(
+ testRecordsAfterFailure.iterator(), iterator, testRecordsAfterFailure.size());
+
+ iterator.close();
+ CommonTestUtils.terminateJob(jobClient, Duration.ofSeconds(30));
+ CommonTestUtils.waitForJobStatus(
+ jobClient,
+ Collections.singletonList(JobStatus.CANCELED),
+ Deadline.fromNow(Duration.ofSeconds(30)));
+ }
+
+ // ----------------------------- Helper Functions ---------------------------------
+
+ /**
+ * Generate a set of test records and write it to the given split writer.
+ *
+ * @param externalContext External context
+ * @return Collection of generated test records
+ */
+ protected Collection<T> generateAndWriteTestData(ExternalContext<T> externalContext) {
+ final Collection<T> testRecordCollection = externalContext.generateTestData();
+ LOG.debug("Writing {} records to external system", testRecordCollection.size());
+ externalContext.createSourceSplit().writeRecords(testRecordCollection);
+ return testRecordCollection;
+ }
+
+ /**
+ * Check if records consumed by Flink are identical to test records written into the single
+ * split within the limit of record number.
+ *
+ * @param testRecordIterator Iterator of test records
+ * @param resultRecordIterator Iterator of result records consumed by Flink
+ * @param limit Number of records to check
+ */
+ protected void checkSingleSplitRecords(
+ Iterator<T> testRecordIterator, Iterator<T> resultRecordIterator, int limit) {
+ for (int i = 0; i < limit; i++) {
+ T testRecord = testRecordIterator.next();
+ T resultRecord = resultRecordIterator.next();
+ assertEquals(testRecord, resultRecord);
+ }
+ LOG.debug("{} records are validated", limit);
+ }
+
+ /**
+ * Check if records consumed by Flink are identical to test records written into the single
+ * split.
+ *
+ * @param testRecordIterator Iterator of test records
+ * @param resultRecordIterator Iterator of result records consumed by Flink
+ */
+ protected void checkSingleSplitRecords(
+ Iterator<T> testRecordIterator, Iterator<T> resultRecordIterator) {
+ int recordCounter = 0;
+
+ while (testRecordIterator.hasNext()) {
+ T testRecord = testRecordIterator.next();
+ T resultRecord = resultRecordIterator.next();
+ assertEquals(testRecord, resultRecord);
+ recordCounter++;
+ }
+
+ assertFalse(resultRecordIterator.hasNext());
+ LOG.debug("{} records are validated", recordCounter);
+ }
+
+ /**
+ * Check if records consumed by Flink is identical to test records written into multiple splits.
+ *
+ * <p>The order of records across different splits can be arbitrary, but should be identical
+ * within a split.
+ *
+ * @param testData Test data in all splits
+ * @param resultRecordIterator Iterator of result records consumed by Flink
+ */
+ protected void checkMultipleSplitRecords(
+ MultipleSplitTestData testData, Iterator<T> resultRecordIterator) {
+ int recordCounter = 0;
+
+ while (resultRecordIterator.hasNext()) {
+ testData.matchThenNext(resultRecordIterator.next());
+ recordCounter++;
+ }
+
+ assertTrue(
+ testData.hasReachedEnd(),
+ "Records received by Flink is less than records sent to external system.");
+
+ LOG.debug("{} records are validated", recordCounter);
+ }
+
+ /**
+ * Wrapper class for validating test data in multiple splits.
+ *
+ * <p>Each collection has a pointer (iterator) pointing to current checking record. When a
+ * record is received in the stream, it will be compared to all current pointing records in
+ * collections, and the pointer to the identical record will move forward.
+ *
+ * <p>If the stream preserves the correctness and order of records in all splits, all pointers
+ * should reach the end of the collection finally.
+ */
+ public class MultipleSplitTestData {
Review comment:
I think this should be a matcher instead. Then we could have
`assertThat(resultIterator, matchesSplits(testRecordCollections));`
If you change it, then it would be nice to have it on top-level for easier reuse.
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/testutils/KafkaContainerizedExternalSystem.java
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.testutils;
+
+import org.apache.flink.connectors.test.common.external.ContainerizedExternalSystem;
+
+import org.apache.kafka.clients.admin.AdminClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+import java.util.Properties;
+
+/**
+ * Kafka external system based on {@link KafkaContainer} from <a
+ * href="https://www.testcontainers.org/">Testcontainers</a>.
+ *
+ * <p>This external system is also integrated with a Kafka admin client for topic management.
+ */
+public class KafkaContainerizedExternalSystem
+ implements ContainerizedExternalSystem<KafkaContainerizedExternalSystem> {
+
+ private GenericContainer<?> flink;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(KafkaContainerizedExternalSystem.class);
+
+ // Kafka container and its related AdminClient
+ private final KafkaContainer kafka;
+ private AdminClient kafkaAdminClient;
Review comment:
This seems completely unused.
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/testsuites/SourceTestSuiteBase.java
##########
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.ConnectorTestingExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({ConnectorTestingExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class SourceTestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SourceTestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test")) {
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 4 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test")) {
+ // Check test result
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with an idle reader.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testIdleReader(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ try (CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test")) {
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(
+ TestEnvironment testEnv,
+ ExternalContext<T> externalContext,
+ ClusterControllable controller)
+ throws Exception {
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
+
+ checkSingleSplitRecords(
+ testRecordsBeforeFailure.iterator(), iterator, testRecordsBeforeFailure.size());
+
+ // -------------------------------- Trigger failover ---------------------------------------
+ controller.triggerTaskManagerFailover(jobClient, () -> {});
+
+ CommonTestUtils.waitForJobStatus(
+ jobClient,
+ Collections.singletonList(JobStatus.RUNNING),
+ Deadline.fromNow(Duration.ofSeconds(30)));
+
+ final Collection<T> testRecordsAfterFailure = externalContext.generateTestData();
+ sourceSplitDataWriter.writeRecords(testRecordsAfterFailure);
+ checkSingleSplitRecords(
+ testRecordsAfterFailure.iterator(), iterator, testRecordsAfterFailure.size());
+
+ iterator.close();
+ CommonTestUtils.terminateJob(jobClient, Duration.ofSeconds(30));
+ CommonTestUtils.waitForJobStatus(
+ jobClient,
+ Collections.singletonList(JobStatus.CANCELED),
+ Deadline.fromNow(Duration.ofSeconds(30)));
+ }
+
+ // ----------------------------- Helper Functions ---------------------------------
+
+ /**
+ * Generate a set of test records and write it to the given split writer.
+ *
+ * @param externalContext External context
+ * @return Collection of generated test records
+ */
+ protected Collection<T> generateAndWriteTestData(ExternalContext<T> externalContext) {
+ final Collection<T> testRecordCollection = externalContext.generateTestData();
+ LOG.debug("Writing {} records to external system", testRecordCollection.size());
+ externalContext.createSourceSplit().writeRecords(testRecordCollection);
+ return testRecordCollection;
+ }
+
+ /**
+ * Check if records consumed by Flink are identical to test records written into the single
+ * split within the limit of record number.
+ *
+ * @param testRecordIterator Iterator of test records
+ * @param resultRecordIterator Iterator of result records consumed by Flink
+ * @param limit Number of records to check
+ */
+ protected void checkSingleSplitRecords(
+ Iterator<T> testRecordIterator, Iterator<T> resultRecordIterator, int limit) {
+ for (int i = 0; i < limit; i++) {
+ T testRecord = testRecordIterator.next();
+ T resultRecord = resultRecordIterator.next();
+ assertEquals(testRecord, resultRecord);
+ }
+ LOG.debug("{} records are validated", limit);
+ }
+
+ /**
+ * Check if records consumed by Flink are identical to test records written into the single
+ * split.
+ *
+ * @param testRecordIterator Iterator of test records
+ * @param resultRecordIterator Iterator of result records consumed by Flink
+ */
+ protected void checkSingleSplitRecords(
+ Iterator<T> testRecordIterator, Iterator<T> resultRecordIterator) {
+ int recordCounter = 0;
+
+ while (testRecordIterator.hasNext()) {
+ T testRecord = testRecordIterator.next();
+ T resultRecord = resultRecordIterator.next();
+ assertEquals(testRecord, resultRecord);
+ recordCounter++;
+ }
+
+ assertFalse(resultRecordIterator.hasNext());
+ LOG.debug("{} records are validated", recordCounter);
+ }
+
+ /**
+ * Check if records consumed by Flink is identical to test records written into multiple splits.
+ *
+ * <p>The order of records across different splits can be arbitrary, but should be identical
+ * within a split.
+ *
+ * @param testData Test data in all splits
+ * @param resultRecordIterator Iterator of result records consumed by Flink
+ */
+ protected void checkMultipleSplitRecords(
+ MultipleSplitTestData testData, Iterator<T> resultRecordIterator) {
+ int recordCounter = 0;
+
+ while (resultRecordIterator.hasNext()) {
+ testData.matchThenNext(resultRecordIterator.next());
+ recordCounter++;
+ }
+
+ assertTrue(
+ testData.hasReachedEnd(),
+ "Records received by Flink is less than records sent to external system.");
+
+ LOG.debug("{} records are validated", recordCounter);
+ }
+
+ /**
+ * Wrapper class for validating test data in multiple splits.
+ *
+ * <p>Each collection has a pointer (iterator) pointing to current checking record. When a
+ * record is received in the stream, it will be compared to all current pointing records in
+ * collections, and the pointer to the identical record will move forward.
+ *
+ * <p>If the stream preserves the correctness and order of records in all splits, all pointers
+ * should reach the end of the collection finally.
+ */
+ public class MultipleSplitTestData {
Review comment:
static
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r679265526
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/pom.xml
##########
@@ -0,0 +1,119 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-test-utils-parent</artifactId>
+ <version>1.14-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <artifactId>flink-connector-testing-framework_${scala.binary.version}</artifactId>
+ <name>Flink : Test utils : Testing Framework</name>
+
+ <properties>
+ <testcontainers.version>1.15.1</testcontainers.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ <version>${testcontainers.version}</version>
+ </dependency>
+
+ <!-- JUnit 5 -->
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>compile</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+
+ <dependencyManagement>
+ <dependencies>
+ <!-- We pick an arbitrary version of net.java.dev.jna:jna to satisfy dependency
+ convergence for org.testcontainers:testcontainers which transitively depends on
+ two different versions.-->
+ <dependency>
+ <groupId>net.java.dev.jna</groupId>
+ <artifactId>jna</artifactId>
+ <version>5.5.0</version>
+ </dependency>
+ </dependencies>
+ </dependencyManagement>
+
+ <profiles>
+ <profile>
+ <!--
+ This profile is for building Docker image based on latest Flink code.
+ Docker environment should be installed before using this profile.
+ To trigger this profile, use "-Pbuild-flink-image" option when running maven.
+ -->
+ <id>build-flink-image</id>
Review comment:
I'm not sure if this is the correct project for that. I'm assuming that this is for `FlinkContainer`. Do we have a better way to (automatically) create the docker image? @zentol
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/pom.xml
##########
@@ -0,0 +1,119 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-test-utils-parent</artifactId>
+ <version>1.14-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <artifactId>flink-connector-testing-framework_${scala.binary.version}</artifactId>
Review comment:
I'd probably remove the framework from the name. We have a lot of other (testing) frameworks without the explicit lateral in the name.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/pom.xml
##########
@@ -0,0 +1,119 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-test-utils-parent</artifactId>
+ <version>1.14-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <artifactId>flink-connector-testing-framework_${scala.binary.version}</artifactId>
+ <name>Flink : Test utils : Testing Framework</name>
+
+ <properties>
+ <testcontainers.version>1.15.1</testcontainers.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ <version>${testcontainers.version}</version>
+ </dependency>
+
+ <!-- JUnit 5 -->
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>compile</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+
+ <dependencyManagement>
Review comment:
Does it make sense to have that convergence in the root pom? It feels like we have more and more projects that depend on testcontainer and they could all use that.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/utils/FlinkContainers.java
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.utils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.connectors.test.common.TestResource;
+
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Flink cluster running on <a href="https://www.testcontainers.org/">Testcontainers</a>.
+ *
+ * <p>This cluster is integrated with components below:
+ * <li>Job manager and task managers
+ * <li>REST cluster client for job status tracking
+ * <li>Workspace directory for storing files generated in flink jobs
+ * <li>Checkpoint directory for storing checkpoints
+ * <li>Job directory for storing Flink job JAR files
+ */
+public class FlinkContainers implements TestResource {
Review comment:
Is there a particular reason why we cannot reuse `org.apache.flink.tests.util.flink.FlinkContainer`?
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/environment/ClusterControllable.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.environment;
+
+import org.apache.flink.core.execution.JobClient;
+
+/** Interface for triggering failover in a Flink cluster. */
+public interface ClusterControllable {
Review comment:
Wouldn't that be something that we should offer outside of the source testing framework? I'm also fine with moving it later.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/environment/MiniClusterTestEnvironment.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.environment;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.highavailability.nonha.embedded.HaLeadershipControl;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Optional;
+import java.util.concurrent.ExecutionException;
+
+/** Test environment for running jobs on Flink mini-cluster. */
+public class MiniClusterTestEnvironment implements TestEnvironment, ClusterControllable {
+
+ private final MiniClusterWithClientResource miniCluster;
+
+ private int latestTMIndex = 0;
+
+ private static final Logger LOG = LoggerFactory.getLogger(MiniClusterTestEnvironment.class);
+
+ private boolean isStarted = false;
+
+ public MiniClusterTestEnvironment() {
+ this.miniCluster =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(6)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+ }
+
+ @Override
+ public StreamExecutionEnvironment createExecutionEnvironment() {
+ return StreamExecutionEnvironment.getExecutionEnvironment();
+ }
+
+ @Override
+ public void triggerJobManagerFailover(JobClient jobClient, Runnable afterFailAction)
+ throws ExecutionException, InterruptedException {
+ final Optional<HaLeadershipControl> controlOptional =
+ miniCluster.getMiniCluster().getHaLeadershipControl();
+ if (!controlOptional.isPresent()) {
+ throw new UnsupportedOperationException(
+ "This MiniCluster does not support JobManager HA");
+ }
+ final HaLeadershipControl haLeadershipControl = controlOptional.get();
+ haLeadershipControl.revokeJobMasterLeadership(jobClient.getJobID()).get();
+ afterFailAction.run();
+ haLeadershipControl.grantJobMasterLeadership(jobClient.getJobID()).get();
+ }
+
+ @Override
+ public void triggerTaskManagerFailover(JobClient jobClient, Runnable afterFailAction)
+ throws Exception {
+ terminateTaskManager();
+ JobStatusUtils.waitForJobStatus(
+ jobClient,
+ Arrays.asList(JobStatus.FAILING, JobStatus.FAILED, JobStatus.RESTARTING),
+ Duration.ofSeconds(30));
+ afterFailAction.run();
+ startTaskManager();
+ }
+
+ @Override
+ public void isolateNetwork(JobClient jobClient, Runnable afterFailAction) {
+ throw new UnsupportedOperationException("Cannot isolate network in a MiniCluster");
+ }
Review comment:
I like the way to induce errors. That would be handy for all kinds of tests. I could also think that we could bake it into `MiniCluster`.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/junit/extensions/TestLoggerExtension.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.junit.extensions;
+
+import org.junit.jupiter.api.extension.BeforeEachCallback;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.TestWatcher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.PrintWriter;
+import java.io.StringWriter;
+
+/** A JUnit-5-style test logger. */
+public class TestLoggerExtension implements TestWatcher, BeforeEachCallback {
Review comment:
We should probably go with a global `TestExecutionListener`, see https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom . We can defer it to fat junit5 refactor commit.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
Review comment:
`Flink job with 4 parallelism`
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
Review comment:
Shouldn't we stick to `@Test` and just annotate `TestSuiteBase` with the extension? Is there any benefit in treating test with injected parameters different from test without such injections?
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
Review comment:
I don't get why you can't use `DataStream#executeAndCollect`. Could you elaborate?
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/utils/ConnectorJarUtils.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.utils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.util.Objects;
+
+/** Utilities for searching connector JARs. */
+public class ConnectorJarUtils {
+
+ private static File searchedJarFile = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(ConnectorJarUtils.class);
+
+ /**
+ * Search JAR file in target folder.
+ *
+ * @return JAR file
+ * @throws FileNotFoundException if JAR file is not found in target folder
+ */
+ public static File searchConnectorJar() throws FileNotFoundException {
Review comment:
I think we have similar functionality already [here](https://github.com/apache/flink/blob/e32f0f82164512f632a533fad01ead6a12ac8152/flink-end-to-end-tests/flink-end-to-end-tests-common/src/main/java/org/apache/flink/tests/util/flink/FlinkResourceSetup.java#L75-L75). However, I'm not sure if the spec can be also applied to the `FlinkContainer`. @zentol ?
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/e2e/resources/KafkaSingleTopicExternalContext.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.e2e.resources;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.connector.kafka.source.KafkaSource;
+import org.apache.flink.connector.kafka.source.KafkaSourceBuilder;
+import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
+import org.apache.flink.connector.kafka.source.reader.deserializer.KafkaRecordDeserializationSchema;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.admin.NewPartitions;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.UnknownTopicOrPartitionException;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A Kafka external context that will create only one topic and use partitions in that topic as
+ * source splits.
+ */
+public class KafkaSingleTopicExternalContext implements ExternalContext<String> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(KafkaSingleTopicExternalContext.class);
+
+ protected String bootstrapServers;
+ private static final String TOPIC_NAME_PREFIX = "kafka-single-topic";
+
+ private final String topicName;
+
+ private static final int DEFAULT_TIMEOUT = 30;
+
+ private final Map<Integer, SourceSplitDataWriter<String>> partitionToSplitWriter =
+ new HashMap<>();
+
+ private int numSplits = 0;
+
+ protected final AdminClient kafkaAdminClient;
+
+ public KafkaSingleTopicExternalContext(String bootstrapServers) {
+ this.bootstrapServers = bootstrapServers;
+ this.topicName =
+ TOPIC_NAME_PREFIX + "-" + ThreadLocalRandom.current().nextLong(Long.MAX_VALUE);
+ kafkaAdminClient = createAdminClient();
+ }
+
+ protected void createTopic(String topicName, int numPartitions, short replicationFactor) {
+ LOG.debug(
+ "Creating new Kafka topic {} with {} partitions and {} replicas",
+ topicName,
+ numPartitions,
+ replicationFactor);
+ NewTopic newTopic = new NewTopic(topicName, numPartitions, replicationFactor);
+ try {
+ kafkaAdminClient
+ .createTopics(Collections.singletonList(newTopic))
+ .all()
+ .get(DEFAULT_TIMEOUT, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ throw new RuntimeException(String.format("Cannot create topic '%s'", topicName), e);
+ }
+ }
+
+ protected void deleteTopic(String topicName) {
+ LOG.debug("Deleting Kafka topic {}", topicName);
+ try {
+ kafkaAdminClient
+ .deleteTopics(Collections.singletonList(topicName))
+ .all()
+ .get(DEFAULT_TIMEOUT, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ if (ExceptionUtils.getRootCause(e) instanceof UnknownTopicOrPartitionException) {
+ throw new RuntimeException(String.format("Cannot delete topic '%s'", topicName), e);
+ }
+ }
+ }
Review comment:
Can we get rid of the replication in respect to `KafkaContainerizedExternalSystem`?
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/e2e/resources/KafkaContainerizedExternalSystem.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.e2e.resources;
+
+import org.apache.flink.connectors.test.common.external.ContainerizedExternalSystem;
+import org.apache.flink.connectors.test.common.utils.FlinkContainers;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+import java.util.Collections;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Kafka external system based on {@link KafkaContainer} from <a
+ * href="https://www.testcontainers.org/">Testcontainers</a>.
+ *
+ * <p>This external system is also integrated with a Kafka admin client for topic management.
+ */
+public class KafkaContainerizedExternalSystem
+ implements ContainerizedExternalSystem<KafkaContainerizedExternalSystem> {
Review comment:
I'd convert `ContainerizedExternalSystem` into a abstract base class where the docker specific stuff and life-cycle is already implemented.
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/e2e/KafkaSourceE2ECase.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.e2e;
+
+import org.apache.flink.connector.kafka.source.e2e.resources.KafkaContainerizedExternalSystem;
+import org.apache.flink.connector.kafka.source.e2e.resources.KafkaMultipleTopicExternalContext;
+import org.apache.flink.connector.kafka.source.e2e.resources.KafkaSingleTopicExternalContext;
+import org.apache.flink.connectors.test.common.environment.FlinkContainersTestEnvironment;
+import org.apache.flink.connectors.test.common.environment.MiniClusterTestEnvironment;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithTestEnvironment;
+import org.apache.flink.connectors.test.common.testsuites.TestSuiteBase;
+import org.apache.flink.connectors.test.common.utils.ConnectorJarUtils;
+
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Tag;
+
+/**
+ * End-to-end test for {@link org.apache.flink.connector.kafka.source.KafkaSource} using testing
+ * framework, based on JUnit 5.
+ *
+ * <p>This class uses {@link Nested} classes to test functionality of KafkaSource under different
+ * Flink environments (MiniCluster and Containers). Each nested class extends {@link TestSuiteBase}
+ * for reusing test cases already defined by testing framework.
+ */
+@Disabled
+@Tag("TestingFramework")
+@DisplayName("Kafka Source E2E Test")
+public class KafkaSourceE2ECase {
+
+ @Nested
+ @DisplayName("On MiniCluster")
+ class OnMiniCluster extends TestSuiteBase<String> {
Review comment:
I'd probably rather go for a parameterized method that supplies a collection of TestEnvironments. Then we could even use a different set of `TestEnvironments` depending on where it's executed. It's also significantly fewer LOC.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
+
+ checkSingleSplitRecords(
+ testRecordsBeforeFailure.iterator(), iterator, testRecordsBeforeFailure.size());
+
+ // -------------------------------- Trigger failover ---------------------------------------
+ final ClusterControllable controller = (ClusterControllable) testEnv;
+ controller.triggerTaskManagerFailover(jobClient, () -> {});
+
+ JobStatusUtils.waitForJobStatus(
+ jobClient, Collections.singletonList(JobStatus.RUNNING), Duration.ofSeconds(30));
+
+ final Collection<T> testRecordsAfterFailure = externalContext.generateTestData();
+ sourceSplitDataWriter.writeRecords(testRecordsAfterFailure);
+ checkSingleSplitRecords(
+ testRecordsAfterFailure.iterator(), iterator, testRecordsAfterFailure.size());
+
+ iterator.close();
+ JobStatusUtils.terminateJob(jobClient, Duration.ofSeconds(30));
+ JobStatusUtils.waitForJobStatus(
+ jobClient, Collections.singletonList(JobStatus.CANCELED), Duration.ofSeconds(30));
+ }
+
+ // ----------------------------- Helper Functions ---------------------------------
+
+ /**
+ * Generate a set of test records and write it to the given split writer.
+ *
+ * @param externalContext External context
+ * @return Collection of generated test records
+ */
+ protected Collection<T> generateAndWriteTestData(ExternalContext<T> externalContext) {
+ final Collection<T> testRecordCollection = externalContext.generateTestData();
+ LOG.debug("Writing {} records to external system", testRecordCollection.size());
+ externalContext.createSourceSplit().writeRecords(testRecordCollection);
+ return testRecordCollection;
+ }
+
+ /**
+ * Check if the given test environment is controllable (can trigger failover or network
+ * isolation).
+ *
+ * @param testEnvironment Test environment being checked
+ * @throws IllegalArgumentException if the test environment is not controllable
+ */
+ protected void checkEnvironmentIsControllable(TestEnvironment testEnvironment)
Review comment:
Not necessary if you make the controller an explicit parameter.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/utils/JobStatusUtils.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.utils;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.core.execution.JobClient;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+/** Helper components for checking Flink job status. */
+public class JobStatusUtils {
Review comment:
We can probably reuse some parts of `org.apache.flink.runtime.testutils.CommonTestUtils` and even move the functions to it.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
Review comment:
`testIdleReader`
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
Review comment:
We probably should add a way for users to explicitly disable test on bounded/unbounded jobs by a simple annotation. (e.g. a source doesn't support different boundedness)
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
Review comment:
try around resultIterator - for all tests.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
Review comment:
`SourceTestSuiteBase`.
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
+
+ checkSingleSplitRecords(
+ testRecordsBeforeFailure.iterator(), iterator, testRecordsBeforeFailure.size());
+
+ // -------------------------------- Trigger failover ---------------------------------------
+ final ClusterControllable controller = (ClusterControllable) testEnv;
+ controller.triggerTaskManagerFailover(jobClient, () -> {});
+
+ JobStatusUtils.waitForJobStatus(
+ jobClient, Collections.singletonList(JobStatus.RUNNING), Duration.ofSeconds(30));
+
+ final Collection<T> testRecordsAfterFailure = externalContext.generateTestData();
Review comment:
this could be the exact same data as before right? couldn't the test then pass by just rereading the first data points?
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
+
+ checkSingleSplitRecords(
+ testRecordsBeforeFailure.iterator(), iterator, testRecordsBeforeFailure.size());
+
+ // -------------------------------- Trigger failover ---------------------------------------
+ final ClusterControllable controller = (ClusterControllable) testEnv;
+ controller.triggerTaskManagerFailover(jobClient, () -> {});
+
+ JobStatusUtils.waitForJobStatus(
+ jobClient, Collections.singletonList(JobStatus.RUNNING), Duration.ofSeconds(30));
+
+ final Collection<T> testRecordsAfterFailure = externalContext.generateTestData();
+ sourceSplitDataWriter.writeRecords(testRecordsAfterFailure);
+ checkSingleSplitRecords(
+ testRecordsAfterFailure.iterator(), iterator, testRecordsAfterFailure.size());
+
+ iterator.close();
+ JobStatusUtils.terminateJob(jobClient, Duration.ofSeconds(30));
+ JobStatusUtils.waitForJobStatus(
+ jobClient, Collections.singletonList(JobStatus.CANCELED), Duration.ofSeconds(30));
+ }
+
+ // ----------------------------- Helper Functions ---------------------------------
+
+ /**
+ * Generate a set of test records and write it to the given split writer.
+ *
+ * @param externalContext External context
+ * @return Collection of generated test records
+ */
+ protected Collection<T> generateAndWriteTestData(ExternalContext<T> externalContext) {
+ final Collection<T> testRecordCollection = externalContext.generateTestData();
+ LOG.debug("Writing {} records to external system", testRecordCollection.size());
+ externalContext.createSourceSplit().writeRecords(testRecordCollection);
+ return testRecordCollection;
+ }
+
+ /**
+ * Check if the given test environment is controllable (can trigger failover or network
+ * isolation).
+ *
+ * @param testEnvironment Test environment being checked
+ * @throws IllegalArgumentException if the test environment is not controllable
+ */
+ protected void checkEnvironmentIsControllable(TestEnvironment testEnvironment)
+ throws IllegalArgumentException {
+ assumeTrue(
+ ClusterControllable.class.isAssignableFrom(testEnvironment.getClass()),
+ "Provided test environment should be controllable.");
+ }
+
+ /**
+ * Check if records consumed by Flink are identical to test records written into the single
+ * split within the limit of record number.
+ *
+ * @param testRecordIterator Iterator of test records
+ * @param resultRecordIterator Iterator of result records consumed by Flink
+ * @param limit Number of records to check
+ */
+ protected void checkSingleSplitRecords(
+ Iterator<T> testRecordIterator, Iterator<T> resultRecordIterator, int limit) {
+ for (int i = 0; i < limit; i++) {
+ T testRecord = testRecordIterator.next();
+ T resultRecord = resultRecordIterator.next();
+ assertEquals(testRecord, resultRecord);
+ }
+ LOG.debug("{} records are validated", limit);
+ }
+
+ /**
+ * Check if records consumed by Flink are identical to test records written into the single
+ * split.
+ *
+ * @param testRecordIterator Iterator of test records
+ * @param resultRecordIterator Iterator of result records consumed by Flink
+ */
+ protected void checkSingleSplitRecords(
+ Iterator<T> testRecordIterator, Iterator<T> resultRecordIterator) {
+ int recordCounter = 0;
+
+ while (testRecordIterator.hasNext()) {
+ T testRecord = testRecordIterator.next();
+ T resultRecord = resultRecordIterator.next();
+ assertEquals(testRecord, resultRecord);
+ recordCounter++;
+ }
+
+ assertFalse(resultRecordIterator.hasNext());
+ LOG.debug("{} records are validated", recordCounter);
+ }
+
+ /**
+ * Check if records consumed by Flink is identical to test records written into multiple splits.
+ *
+ * <p>The order of records across different splits can be arbitrary, but should be identical
+ * within a split.
+ *
+ * @param testRecordIterators Collection of iterators for records in different splits
+ * @param resultRecordIterator Iterator of result records consumed by Flink
+ */
+ protected void checkMultipleSplitRecords(
+ Collection<Iterator<T>> testRecordIterators, Iterator<T> resultRecordIterator) {
Review comment:
I'd probably materialize all results and get rid of iterators. It's just so much harder to read...
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
+
+ checkSingleSplitRecords(
+ testRecordsBeforeFailure.iterator(), iterator, testRecordsBeforeFailure.size());
+
+ // -------------------------------- Trigger failover ---------------------------------------
+ final ClusterControllable controller = (ClusterControllable) testEnv;
Review comment:
Also make `ClusterControllable` injectable.
##########
File path: flink-connectors/flink-connector-kafka/pom.xml
##########
@@ -262,10 +270,116 @@ under the License.
<configuration>
<!-- Enforce single fork execution due to heavy mini cluster use in the tests -->
<forkCount>1</forkCount>
- <argLine>-Xms256m -Xmx2048m -Dmvn.forkNumber=${surefire.forkNumber} -XX:-UseGCOverheadLimit</argLine>
+ <argLine>-Xms256m -Xmx2048m -Dmvn.forkNumber=${surefire.forkNumber}
+ -XX:-UseGCOverheadLimit
+ </argLine>
</configuration>
+ <executions>
+ <execution>
+ <id>integration-tests</id>
+ <configuration>
+ <excludes>
+ <exclude>${test.unit.pattern}</exclude>
+ <exclude>**/*E2ECase*</exclude>
+ </excludes>
+ </configuration>
+ </execution>
+ </executions>
</plugin>
</plugins>
</build>
+ <!-- Profile for running testing framework cases -->
+ <profiles>
+ <profile>
+ <id>run-testing-framework</id>
Review comment:
Not a massive fan that this is a) optional and b) is quite a bit of boilerplate that would need to be replicated across several connectors.
Ideally, we should execute all connector tests as ITCases with minicluster in the IDE and as E2E with container on AZP. Then we would just need a way to re-execute the same tests with a different TestEnvironment. @zentol WDYT?
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/junit/extensions/TestingFrameworkExtension.java
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.junit.extensions;
+
+import org.apache.flink.connectors.test.common.TestResource;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.ExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalContextFactory;
+import org.apache.flink.connectors.test.common.junit.annotations.WithExternalSystem;
+import org.apache.flink.connectors.test.common.junit.annotations.WithTestEnvironment;
+
+import org.junit.jupiter.api.extension.AfterAllCallback;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.platform.commons.support.AnnotationSupport;
+
+import java.lang.annotation.Annotation;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * A JUnit 5 {@link Extension} for supporting running of testing framework.
+ *
+ * <p>This extension is responsible for searching test resources annotated by {@link
+ * WithTestEnvironment}, {@link WithExternalSystem} and {@link WithExternalContextFactory}, storing
+ * them into storage provided by JUnit, and manage lifecycle of these resources.
+ *
+ * <p>The extension uses {@link ExtensionContext.Store} for handing over test resources to {@link
+ * TestCaseInvocationContextProvider}, which will inject these resources into test cases.
+ */
+public class TestingFrameworkExtension implements BeforeAllCallback, AfterAllCallback {
Review comment:
Are you expecting usage beyond connectors? If not, we could name it `ConnectorTestingExtension` (again dropping framework). If it's general, then we should find a new package/module name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685063053
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/pom.xml
##########
@@ -0,0 +1,119 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-test-utils-parent</artifactId>
+ <version>1.14-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <artifactId>flink-connector-testing-framework_${scala.binary.version}</artifactId>
+ <name>Flink : Test utils : Testing Framework</name>
+
+ <properties>
+ <testcontainers.version>1.15.1</testcontainers.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ <version>${testcontainers.version}</version>
+ </dependency>
+
+ <!-- JUnit 5 -->
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>compile</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+
+ <dependencyManagement>
Review comment:
I've tried to do so, but it caused a license issue in Kinesis connector. I'll just keep it for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 027d4ab260b3914c6847704641229f13686e7ffe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38c3cd15cf9b06253125b57322247f2d1d7bf95e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685803188
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
Review comment:
Okay I'm not deep enough in JUnit5 yet to propose an alternative. I had assumed that we could but the `@ExtendWith` at the class and then just use `@TestTemplate`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
}, {
"hash" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978",
"triggerID" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6705cab068becca06bae6aaa50258a08bbfbbd3d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688388087
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
If you delete the NOTICE file _and_ add the module to `modules-skipping-deployment.modulelist` then the checker should not complain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b830de58276846f46b302298a4f5be6853fc645 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688022693
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/testutils/KafkaSourceSplitDataWriter.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.testutils;
+
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.TopicPartition;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Properties;
+
+/** Source split data writer for writing test data into Kafka topic partitions. */
+public class KafkaSourceSplitDataWriter implements SourceSplitDataWriter<String> {
Review comment:
Hm I see your point and I am a bit torn. On the one hand, I'd argue that partitions is such a wide-spread concept in data storage and probably better known than splits which is just a concept in distributed data ingestion/processing. On the other hand, is this very specific to source developers. So I'm probably fine either way for now. Let's make sure everything is `@Experimental` so that we could change that in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686665030
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/testsuites/SourceTestSuiteBase.java
##########
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.ConnectorTestingExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({ConnectorTestingExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class SourceTestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SourceTestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test")) {
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 4 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test")) {
+ // Check test result
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with an idle reader.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testIdleReader(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ try (CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test")) {
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
Review comment:
I don't mind that `@Tag` - I was just wondering if we could extract a constant. But I think we are not doing it on other Test tags as well. So just leave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688388087
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
If you delete the NOTICE file _and_ add the module to `modules-skipping-deployment.modulelist` then the checker should not complain.
If it still does, please ping me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-895116020
Thanks @AHeise for the review~ I changed my implementations based on your comments. Could you please have another round of review? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688400408
##########
File path: flink-test-utils-parent/flink-connector-testing/README.md
##########
@@ -0,0 +1,44 @@
+# Flink Connector Testing Framework
Review comment:
Agree. We can keep it internal for now and publish it when stable in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686965676
##########
File path: flink-end-to-end-tests/flink-streaming-kinesis-test/pom.xml
##########
@@ -151,6 +151,23 @@ under the License.
</systemPropertyVariables>
</configuration>
</plugin>
+
+ <!-- Skip dependency convergence check because of guava version -->
+ <plugin>
Review comment:
It's a conflict between
```
+-org.apache.flink:flink-streaming-kinesis-test_2.11:1.14-SNAPSHOT
+-org.apache.flink:flink-connector-kinesis_2.11:1.14-SNAPSHOT
+-com.google.guava:guava:29.0-jre
```
and
```
+-org.apache.flink:flink-streaming-kinesis-test_2.11:1.14-SNAPSHOT
+-org.apache.flink:flink-end-to-end-tests-common:1.14-SNAPSHOT
+-org.apache.flink:flink-connector-testing_2.11:1.14-SNAPSHOT
+-org.apache.flink:flink-test-utils_2.11:1.14-SNAPSHOT
+-org.apache.curator:curator-test:2.12.0
+-com.google.guava:guava:16.0.1
```
Do you have any suggestions for solving this? 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688021055
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
🤯 @rmetzger could you please help us?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688381618
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
oh snap it does look at the output of the dependency plugin as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686070524
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
Review comment:
Maybe mark it as `@Internal`? We kept it package-private because we didn't believe it was a method any normal user should use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 027d4ab260b3914c6847704641229f13686e7ffe UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688379592
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
The notice checker _does not_ use the output from the dependency plugin, but the output of the shade-plugin. If the notice checker complainted, please link a build where this happened.
Note: The _scala suffix checker_ uses the dependency plugin output.
If a given module is not deployed, then you can exclude it from the notice checks in `tools/ci/java-ci-tools/src/main/resources/modules-skipping-deployment.modulelist`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ba223523eb708232b5fe86eb162fb1f36e85ffc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686930290
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/external/ExternalSystem.java
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.external;
+
+import org.apache.flink.connectors.test.common.TestResource;
+
+/** General external system for end-to-end testing. */
+public interface ExternalSystem extends TestResource {}
Review comment:
I removed this interface and use annotation ```@ExternalSystem``` instead. Implementations of external system can directly extend the base interface ```TestResource```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r686592908
##########
File path: flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/testsuites/SourceTestSuiteBase.java
##########
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.ConnectorTestingExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({ConnectorTestingExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class SourceTestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SourceTestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test")) {
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 4 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+
+ try (final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test")) {
+ // Check test result
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with an idle reader.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testIdleReader(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ try (CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test")) {
+ checkMultipleSplitRecords(
+ new MultipleSplitTestData(testRecordCollections), resultIterator);
+ }
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
Review comment:
I was planning to use this tag to mark cases with failover scenarios, so that user can ignore this cases if their test environment doesn't support failover triggering.
I'll just remove it for now, and use other annotations to support skipping failover cases in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38c3cd15cf9b06253125b57322247f2d1d7bf95e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722)
* 9b830de58276846f46b302298a4f5be6853fc645 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
}, {
"hash" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978",
"triggerID" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"triggerType" : "PUSH"
}, {
"hash" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22053",
"triggerID" : "de029efd3d3ff874bf43cc8b5a39df36c0181359",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de029efd3d3ff874bf43cc8b5a39df36c0181359 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22053)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685814603
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/pom.xml
##########
@@ -49,8 +49,15 @@ under the License.
<scope>compile</scope>
</dependency>
<dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
+ <groupId>org.junit.jupiter</groupId>
Review comment:
Don't forget to remove this commit once #16551 is merged. (It still uses migrationsupport)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 027d4ab260b3914c6847704641229f13686e7ffe Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685075294
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
+
+ checkSingleSplitRecords(
+ testRecordsBeforeFailure.iterator(), iterator, testRecordsBeforeFailure.size());
+
+ // -------------------------------- Trigger failover ---------------------------------------
+ final ClusterControllable controller = (ClusterControllable) testEnv;
+ controller.triggerTaskManagerFailover(jobClient, () -> {});
+
+ JobStatusUtils.waitForJobStatus(
+ jobClient, Collections.singletonList(JobStatus.RUNNING), Duration.ofSeconds(30));
+
+ final Collection<T> testRecordsAfterFailure = externalContext.generateTestData();
Review comment:
Depends on implementation of ```generateTestData```. For example in Kafka implementations it'll generate different random strings in different method calls. I'd like to call this method again to generate a new dataset after triggering failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93776a6d6881dc65de98a7c7337ebd6189b61414 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717)
* 38c3cd15cf9b06253125b57322247f2d1d7bf95e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688258957
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/KafkaSourceITCase.java
##########
@@ -26,6 +26,15 @@
import org.apache.flink.configuration.Configuration;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
import org.apache.flink.connector.kafka.source.reader.deserializer.KafkaRecordDeserializationSchema;
+import org.apache.flink.connector.kafka.source.testutils.KafkaMultipleTopicExternalContext;
Review comment:
The existing KafkaSourceITCase uses ```@BeforeAll``` and ```@AfterAll``` to startup a in-memory Kafka broker. If I only nest the testing framework cases, that in-memory Kafka broker will be running until all cases including testing framework cases finish.
So I use the ```KafkaSpecificTests``` nested class to wrap all existing IT cases and destroy the in-memory Kafka before running testing framework cases. "KafkaSpecific" means these cases are testing Kafka source features like start timestamp, serializer etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688382377
##########
File path: flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,9 @@
+flink-end-to-end-test-common-kafka
Review comment:
Ah right, this is for cases where we bundle certain things by manually modifying the target/ directory, e.g., for java version-specific dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r688026116
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/KafkaSourceITCase.java
##########
@@ -26,6 +26,15 @@
import org.apache.flink.configuration.Configuration;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
import org.apache.flink.connector.kafka.source.reader.deserializer.KafkaRecordDeserializationSchema;
+import org.apache.flink.connector.kafka.source.testutils.KafkaMultipleTopicExternalContext;
Review comment:
I don't quite understand the changes in this file. This looks to me if you have "only" converted the tests from JUnit4 to JUnit5. If so please move in some extra commit. I'd also like to see the reason of moving the tests inside a nested test class. Aren't all tests `KafkaSpecificTests`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16590:
URL: https://github.com/apache/flink/pull/16590#discussion_r685809592
##########
File path: flink-test-utils-parent/flink-connector-testing-framework/src/main/java/org/apache/flink/connectors/test/common/testsuites/TestSuiteBase.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.test.common.testsuites;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connectors.test.common.environment.ClusterControllable;
+import org.apache.flink.connectors.test.common.environment.TestEnvironment;
+import org.apache.flink.connectors.test.common.external.ExternalContext;
+import org.apache.flink.connectors.test.common.external.SourceSplitDataWriter;
+import org.apache.flink.connectors.test.common.junit.annotations.Case;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.connectors.test.common.junit.extensions.TestingFrameworkExtension;
+import org.apache.flink.connectors.test.common.utils.JobStatusUtils;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Base class for all test suites.
+ *
+ * <p>All cases should have well-descriptive JavaDoc, including:
+ *
+ * <ul>
+ * <li>What's the purpose of this case
+ * <li>Simple description of how this case works
+ * <li>Condition to fulfill in order to pass this case
+ * <li>Requirement of running this case
+ * </ul>
+ */
+@ExtendWith({TestingFrameworkExtension.class, TestLoggerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class TestSuiteBase<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestSuiteBase.class);
+
+ // ----------------------------- Basic test cases ---------------------------------
+
+ /**
+ * Test connector source with only one split in the external system.
+ *
+ * <p>This test will create one split in the external system, write test data into it, and
+ * consume back via a Flink job with 1 parallelism.
+ *
+ * <p>The number and order of records consumed by Flink need to be identical to the test data
+ * written to the external system in order to pass this test.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with single split")
+ public void testSourceSingleSplit(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ // Write test data to external system
+ final Collection<T> testRecords = generateAndWriteTestData(externalContext);
+
+ // Build and execute Flink job
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1)
+ .executeAndCollect("Source Single Split Test");
+
+ // Check test result
+ checkSingleSplitRecords(testRecords.iterator(), resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with multiple splits in the external system
+ *
+ * <p>This test will create 4 splits in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with multiple splits")
+ public void testMultipleSplits(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ LOG.debug("Build and execute Flink job");
+ StreamExecutionEnvironment execEnv = testEnv.createExecutionEnvironment();
+ final CloseableIterator<T> resultIterator =
+ execEnv.fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber)
+ .executeAndCollect("Source Multiple Split Test");
+
+ // Check test result
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ resultIterator.close();
+ }
+
+ /**
+ * Test connector source with a redundant parallelism.
+ *
+ * <p>This test will create 4 split in the external system, write test data to all splits, and
+ * consume back via a Flink job with 5 parallelism, so at least one parallelism / source reader
+ * will be idle (assigned with no splits). If the split enumerator of the source doesn't signal
+ * NoMoreSplitsEvent to the idle source reader, the Flink job will never spin to FINISHED state.
+ *
+ * <p>The number and order of records in each split consumed by Flink need to be identical to
+ * the test data written into the external system to pass this test. There's no requirement for
+ * record order across splits.
+ *
+ * <p>A bounded source is required for this test.
+ */
+ @Case
+ @DisplayName("Test source with at least one idle parallelism")
+ public void testRedundantParallelism(
+ TestEnvironment testEnv, ExternalContext<T> externalContext) throws Exception {
+
+ final int splitNumber = 4;
+ final List<Collection<T>> testRecordCollections = new ArrayList<>();
+ for (int i = 0; i < splitNumber; i++) {
+ testRecordCollections.add(generateAndWriteTestData(externalContext));
+ }
+
+ final CloseableIterator<T> resultIterator =
+ testEnv.createExecutionEnvironment()
+ .fromSource(
+ externalContext.createSource(Boundedness.BOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(splitNumber + 1)
+ .executeAndCollect("Redundant Parallelism Test");
+
+ checkMultipleSplitRecords(
+ testRecordCollections.stream()
+ .map(Collection::iterator)
+ .collect(Collectors.toList()),
+ resultIterator);
+ }
+
+ /**
+ * Test connector source with task manager failover.
+ *
+ * <p>This test will create 1 split in the external system, write test record set A into the
+ * split, restart task manager to trigger job failover, write test record set B into the split,
+ * and terminate the Flink job finally.
+ *
+ * <p>The number and order of records consumed by Flink should be identical to A before the
+ * failover and B after the failover in order to pass the test.
+ *
+ * <p>An unbounded source is required for this test, since TaskManager failover will be
+ * triggered in the middle of the test.
+ */
+ @Case
+ @Tag("failover")
+ @DisplayName("Test TaskManager failure")
+ public void testTaskManagerFailure(TestEnvironment testEnv, ExternalContext<T> externalContext)
+ throws Exception {
+
+ checkEnvironmentIsControllable(testEnv);
+
+ final Collection<T> testRecordsBeforeFailure = externalContext.generateTestData();
+ final SourceSplitDataWriter<T> sourceSplitDataWriter = externalContext.createSourceSplit();
+ sourceSplitDataWriter.writeRecords(testRecordsBeforeFailure);
+
+ final StreamExecutionEnvironment env = testEnv.createExecutionEnvironment();
+
+ env.enableCheckpointing(50);
+ final DataStreamSource<T> dataStreamSource =
+ env.fromSource(
+ externalContext.createSource(Boundedness.CONTINUOUS_UNBOUNDED),
+ WatermarkStrategy.noWatermarks(),
+ "Tested Source")
+ .setParallelism(1);
+
+ // Since DataStream API doesn't expose job client for executeAndCollect(), we have
+ // to reuse these part of code to get both job client and result iterator :-(
+ // ------------------------------------ START ---------------------------------------------
+ TypeSerializer<T> serializer = dataStreamSource.getType().createSerializer(env.getConfig());
+ String accumulatorName = "dataStreamCollect_" + UUID.randomUUID();
+ CollectSinkOperatorFactory<T> factory =
+ new CollectSinkOperatorFactory<>(serializer, accumulatorName);
+ CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
+ CollectResultIterator<T> iterator =
+ new CollectResultIterator<>(
+ operator.getOperatorIdFuture(),
+ serializer,
+ accumulatorName,
+ env.getCheckpointConfig());
+ CollectStreamSink<T> sink = new CollectStreamSink<>(dataStreamSource, factory);
+ sink.name("Data stream collect sink");
+ env.addOperator(sink.getTransformation());
+ final JobClient jobClient = env.executeAsync("TaskManager Failover Test");
+ iterator.setJobClient(jobClient);
+ // -------------------------------------- END ---------------------------------------------
Review comment:
I think we should make it public in this PR in a separate commit. @sjwiesman , do you know why this is currently not exposed?
I'm assuming a FLIP is not needed as the other executeAndCollect methods have also been added without FLIP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16590: [FLINK-19554][connector/testing-framework] Connector Testing Framework
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16590:
URL: https://github.com/apache/flink/pull/16590#issuecomment-886445550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "027d4ab260b3914c6847704641229f13686e7ffe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20948",
"triggerID" : "027d4ab260b3914c6847704641229f13686e7ffe",
"triggerType" : "PUSH"
}, {
"hash" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21709",
"triggerID" : "8ba223523eb708232b5fe86eb162fb1f36e85ffc",
"triggerType" : "PUSH"
}, {
"hash" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21717",
"triggerID" : "93776a6d6881dc65de98a7c7337ebd6189b61414",
"triggerType" : "PUSH"
}, {
"hash" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21722",
"triggerID" : "38c3cd15cf9b06253125b57322247f2d1d7bf95e",
"triggerType" : "PUSH"
}, {
"hash" : "9b830de58276846f46b302298a4f5be6853fc645",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760",
"triggerID" : "9b830de58276846f46b302298a4f5be6853fc645",
"triggerType" : "PUSH"
}, {
"hash" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978",
"triggerID" : "6705cab068becca06bae6aaa50258a08bbfbbd3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b830de58276846f46b302298a4f5be6853fc645 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21760)
* 6705cab068becca06bae6aaa50258a08bbfbbd3d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21978)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org