You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/11/29 04:39:20 UTC
[GitHub] [flink] imaffe opened a new pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
imaffe opened a new pull request #17937:
URL: https://github.com/apache/flink/pull/17937
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Enhance the pulsar source unit test for the following classes
- PulsarSourceReader
- PulsarPartitionSplitReader
- PulsarSourceEnumerator
- PulsarSourceBuilder
## Brief change log
One bug fix with this test enhancement:
1. [BUG FIX] Fix overflow when stopCursor is latest and use exclusive start cursor .
2. Added tests for Ordered and Unordered version of PulsarSourceReader and PulsarPartitionSplitReader
3. Improved tests for PulsarSourceEnumerator and PulsarSourceBuilder
## Verifying this change
Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (100MB)*
- *Extended integration test for recovery after master (JobManager) failure*
- *Added test that validates that TaskInfo is transferred only once across recoveries*
- *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes / no / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
- The S3 file system connector: (yes / no / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r779560085
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
Review comment:
For the general design guideline, do we prefer not using protected methods. For example, in design method X is designed to be exposed to its subclasses, but currenly subclass is not using that method. Do we keep method X as protected, or we change X to private, and only change X to protected again when we needed to use that method in the subclasses ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* c1728e5765b33b6ba1140a2f313687eb3bbbaf5f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398)
* c1728e5765b33b6ba1140a2f313687eb3bbbaf5f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r779552658
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
@RegisterExtension
PulsarSplitReaderInvocationContextProvider provider =
new PulsarSplitReaderInvocationContextProvider();
+ /** Default reader config: max message 1, fetch timeout 1s. */
protected Configuration readerConfig() {
Configuration config = operator().config();
config.set(PULSAR_MAX_FETCH_RECORDS, 1);
config.set(PULSAR_MAX_FETCH_TIME, 1000L);
config.set(PULSAR_SUBSCRIPTION_NAME, randomAlphabetic(10));
config.set(PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE, true);
-
return config;
}
protected SourceConfiguration sourceConfig() {
return new SourceConfiguration(readerConfig());
}
- protected SplitsAddition<PulsarPartitionSplit> createSplit(String topicName, int partitionId) {
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ handleSplit(reader, topicName, partitionId, null);
+ }
+
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
+ TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), startPosition, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+ reader.handleSplitsChanges(addition);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ seekStartPositionAndHandleSplit(reader, topicName, partitionId, MessageId.latest);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
- PulsarPartitionSplit split = new PulsarPartitionSplit(partition, never());
- return new SplitsAddition<>(singletonList(split));
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), null, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+
+ // create consumer and seek before split changes
+ try (Consumer<byte[]> consumer = reader.createPulsarConsumer(partition)) {
+ // inclusive messageId
+ StartCursor startCursor = StartCursor.fromMessageId(startPosition);
+ startCursor.seekPosition(partition.getTopic(), partition.getPartitionId(), consumer);
+ } catch (PulsarClientException e) {
+ sneakyThrow(e);
+ }
+
+ reader.handleSplitsChanges(addition);
}
protected <T> PulsarMessage<T> fetchedMessage(PulsarPartitionSplitReaderBase<T> splitReader) {
- try {
- RecordsWithSplitIds<PulsarMessage<T>> records = splitReader.fetch();
- if (records.nextSplit() != null) {
- return records.nextRecordFromSplit();
+ return fetchedMessages(splitReader, 1, false).stream().findFirst().orElse(null);
+ }
+
+ protected <T> List<PulsarMessage<T>> fetchedMessages(
+ PulsarPartitionSplitReaderBase<T> splitReader, int expectedCount, boolean verify) {
+ return fetchedMessages(
+ splitReader, expectedCount, verify, Boundedness.CONTINUOUS_UNBOUNDED);
+ }
+
+ protected <T> List<PulsarMessage<T>> fetchedMessages(
Review comment:
I think here these methodsare provided potentially for the subclasses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
}, {
"hash" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48c44601bb0c15a3975d8ad41249231f06cbced2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073)
* e45e95974b2d7229be6f04f51828d09b50a5486d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r763142827
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/enumerator/PulsarSourceEnumeratorTest.java
##########
@@ -112,64 +119,270 @@ void startWithPeriodicPartitionDiscovery() throws Exception {
}
}
- private PulsarSourceEnumerator createEnumerator(
+ @ParameterizedTest
+ @EnumSource(
+ value = SubscriptionType.class,
+ names = {"Failover", "Shared"})
+ void discoverPartitionsTriggersAssignments(SubscriptionType subscriptionType) throws Throwable {
+ Set<String> prexistingTopics = setupPreexistingTopics();
+ try (MockSplitEnumeratorContext<PulsarPartitionSplit> context =
+ new MockSplitEnumeratorContext<>(NUM_SUBTASKS);
+ PulsarSourceEnumerator enumerator =
+ createEnumerator(
+ subscriptionType,
+ prexistingTopics,
+ context,
+ DISABLE_PERIODIC_PARTITION_DISCOVERY)) {
+
+ enumerator.start();
+
+ // register reader 0, 1
+ registerReader(context, enumerator, READER0);
+ registerReader(context, enumerator, READER1);
+ assertTrue(context.getSplitsAssignmentSequence().isEmpty());
+
+ // Run the partition discover callable and check the partition assignment.
+ runOneTimePartitionDiscovery(context);
+ verifyLastReadersAssignments(subscriptionType, context, prexistingTopics, 1);
+ }
+ }
+
+ @ParameterizedTest
+ @EnumSource(
+ value = SubscriptionType.class,
+ names = {"Failover", "Shared"})
+ @Timeout(value = 60)
Review comment:
We usually discourage using timeouts because they can easily make the test unstable due to different execution durations in different environments. Please try to remove it.
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assertions.fail;
+
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ /** Expected the add or override config entries in inherited classes. */
+ protected abstract void customConfig(Configuration config);
+
+ @BeforeEach
+ void beforeEach(TestInfo testInfo) {
+ Random random = new Random(System.currentTimeMillis());
+ String topicName = uniqueTestName(testInfo);
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(TestInfo testInfo) {
+ String topicName = uniqueTestName(testInfo);
+ operator().deleteTopic(topicName, true);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void assignZeroSplitsCreatesZeroSubscription(
+ boolean autoAcknowledgementEnabled, TestInfo testInfo) throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void assigningEmptySplits(boolean autoAcknowledgementEnabled, TestInfo testInfo)
+ throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // we perform assertion when add splits task
+ // is dequeued but not executed.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
Review comment:
Can you explain why we need this? This also seems like a source of flakiness.
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -126,63 +196,116 @@ void pollMessageAfterTimeout(PulsarPartitionSplitReaderBase<String> splitReader)
"Couldn't poll message from Pulsar.");
}
- /** Create a split reader with max message 1, fetch timeout 1s. */
- protected abstract PulsarPartitionSplitReaderBase<String> splitReader();
+ @Test
+ void consumeMessageCreatedAfterHandleSplitChangesAndFetch() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0);
+ operator().sendMessage(topicNameWithPartition(topicName, 0), STRING, randomAlphabetic(10));
+ fetchedMessages(splitReader, 1, true);
+ }
- /** JUnit5 extension for all the TestTemplate methods in this class. */
- public class PulsarSplitReaderInvocationContextProvider
- implements TestTemplateInvocationContextProvider {
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChanges() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
- @Override
- public boolean supportsTestTemplate(ExtensionContext context) {
- return true;
- }
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0, MessageId.earliest);
+ fetchedMessages(splitReader, NUM_RECORDS_PER_PARTITION, true);
+ }
- @Override
- public Stream<TestTemplateInvocationContext> provideTestTemplateInvocationContexts(
- ExtensionContext context) {
- return Stream.of(new PulsarSplitReaderInvocationContext(splitReader()));
- }
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndResetToLatestPosition() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0, MessageId.latest);
+ fetchedMessages(splitReader, 0, true);
}
- /** Parameter resolver for Split Reader. */
- public static class PulsarSplitReaderInvocationContext
- implements TestTemplateInvocationContext {
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndUseSecondLastMessageIdCursor() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ MessageIdImpl lastMessageId =
+ (MessageIdImpl)
+ sneakyAdmin(
+ () ->
+ operator()
+ .admin()
+ .topics()
+ .getLastMessageId(
+ topicNameWithPartition(topicName, 0)));
+ // when doing seek directly on consumer, by default it includes the specified messageId
+ seekStartPositionAndHandleSplit(
+ splitReader,
+ topicName,
+ 0,
+ new MessageIdImpl(
+ lastMessageId.getLedgerId(),
+ lastMessageId.getEntryId() - 1,
+ lastMessageId.getPartitionIndex()));
+ fetchedMessages(splitReader, 2, true);
+ }
- private final PulsarPartitionSplitReaderBase<?> splitReader;
+ @Test
+ void emptyTopic() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().createTopic(topicName, DEFAULT_PARTITIONS);
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
- public PulsarSplitReaderInvocationContext(PulsarPartitionSplitReaderBase<?> splitReader) {
- this.splitReader = checkNotNull(splitReader);
- }
+ @Test
+ void emptyTopicWithoutSeek() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().createTopic(topicName, DEFAULT_PARTITIONS);
+ handleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
- @Override
- public String getDisplayName(int invocationIndex) {
- return splitReader.getClass().getSimpleName();
+ @Test
+ void wakeupSplitReaderShouldNotCauseException() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ handleSplit(splitReader, "non-exist", 0);
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ Thread t =
+ new Thread(
+ () -> {
+ try {
+ splitReader.fetch();
+ } catch (Throwable e) {
+ error.set(e);
+ }
+ },
+ "testWakeUp-thread");
+ t.start();
+ long deadline = System.currentTimeMillis() + 5000L;
+ while (t.isAlive() && System.currentTimeMillis() < deadline) {
+ splitReader.wakeUp();
+ sleepUninterruptibly(10, TimeUnit.MILLISECONDS);
}
+ assertNull(error.get());
+ }
- @Override
- public List<Extension> getAdditionalExtensions() {
- return Collections.singletonList(
- new ParameterResolver() {
- @Override
- public boolean supportsParameter(
- ParameterContext parameterContext,
- ExtensionContext extensionContext)
- throws ParameterResolutionException {
- return parameterContext
- .getParameter()
- .getType()
- .equals(PulsarPartitionSplitReaderBase.class);
- }
-
- @Override
- public Object resolveParameter(
- ParameterContext parameterContext,
- ExtensionContext extensionContext)
- throws ParameterResolutionException {
- return splitReader;
- }
- });
- }
+ @Test
+ void assignNoSplits() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ assertNull(fetchedMessage(splitReader));
}
+
+ /** Create a split reader with max message 1, fetch timeout 1s. */
+ protected abstract PulsarPartitionSplitReaderBase<String> splitReader();
Review comment:
You can implement an extension to instantiate the test SplitReader in the subclasses and directly passes it to the test without the need of an abstract method. You can take a look at how the `SourceTestSuiteBase` does it with the `TestEnvironment` [1]
[1] https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-connector-testing/src/main/java/org/apache/flink/connectors/test/common/testsuites/SourceTestSuiteBase.java
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarOrderedSourceReaderTest.java
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.PulsarSourceOptions;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplits;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@ExtendWith(TestLoggerExtension.class)
+class PulsarOrderedSourceReaderTest extends PulsarSourceReaderTestBase {
+
+ private static final int MAX_EMPTY_POLLING_TIMES = 10;
+
+ @Override
+ protected void customConfig(Configuration config) {
+ config.set(PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE, SubscriptionType.Failover);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void pollMessageWithIntervalLongerThanTimeout(
+ boolean autoAcknowledgementEnabled, TestInfo testInfo) throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ Boundedness boundedness = Boundedness.CONTINUOUS_UNBOUNDED;
+ String topicName = uniqueTestName(testInfo);
Review comment:
Isn't this topic created before every test in `PulsarSourceReaderTestBase#before`?
In general, I'd discourage sharing test setup code through inheritance because it becomes unmaintainable very fast and it is hard to extend. I prefer to use extensions that I can use in any test.
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarOrderedPartitionSplitReaderTest.java
##########
@@ -34,4 +43,57 @@
sourceConfig(),
flinkSchema(new SimpleStringSchema()));
}
+
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesWithoutSeek() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ handleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
+
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndUseLatestStartCursorWithoutSeek() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ handleSplit(splitReader, topicName, 0, MessageId.latest);
+ fetchedMessages(splitReader, 0, true);
+ }
+
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndUseEarliestStartCursorWithoutSeek() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ handleSplit(splitReader, topicName, 0, MessageId.earliest);
+ fetchedMessages(splitReader, NUM_RECORDS_PER_PARTITION, true);
+ }
Review comment:
Nit: Maybe think about using a parameterized test to avoid the code duplication
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -18,95 +18,165 @@
package org.apache.flink.connector.pulsar.source.reader.split;
+import org.apache.flink.api.connector.source.Boundedness;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.connector.base.source.reader.RecordsWithSplitIds;
import org.apache.flink.connector.base.source.reader.splitreader.SplitsAddition;
import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.cursor.StartCursor;
+import org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor;
import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicPartition;
import org.apache.flink.connector.pulsar.source.reader.message.PulsarMessage;
import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
-import org.junit.jupiter.api.DisplayName;
-import org.junit.jupiter.api.TestTemplate;
-import org.junit.jupiter.api.extension.Extension;
-import org.junit.jupiter.api.extension.ExtensionContext;
-import org.junit.jupiter.api.extension.ParameterContext;
-import org.junit.jupiter.api.extension.ParameterResolutionException;
-import org.junit.jupiter.api.extension.ParameterResolver;
-import org.junit.jupiter.api.extension.RegisterExtension;
-import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
-import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.junit.jupiter.api.Test;
import java.io.IOException;
-import java.util.Collections;
+import java.util.ArrayList;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
-import java.util.stream.Stream;
+import java.util.concurrent.atomic.AtomicReference;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
-import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.pulsar.client.api.Schema.STRING;
+import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertNull;
/** Test utils for split readers. */
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
- @RegisterExtension
- PulsarSplitReaderInvocationContextProvider provider =
- new PulsarSplitReaderInvocationContextProvider();
-
+ /** Default reader config: max message 1, fetch timeout 1s. */
protected Configuration readerConfig() {
Configuration config = operator().config();
config.set(PULSAR_MAX_FETCH_RECORDS, 1);
config.set(PULSAR_MAX_FETCH_TIME, 1000L);
config.set(PULSAR_SUBSCRIPTION_NAME, randomAlphabetic(10));
config.set(PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE, true);
-
return config;
}
protected SourceConfiguration sourceConfig() {
return new SourceConfiguration(readerConfig());
}
- protected SplitsAddition<PulsarPartitionSplit> createSplit(String topicName, int partitionId) {
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ handleSplit(reader, topicName, partitionId, null);
+ }
+
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
- PulsarPartitionSplit split = new PulsarPartitionSplit(partition, never());
- return new SplitsAddition<>(singletonList(split));
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), startPosition, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+ reader.handleSplitsChanges(addition);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ seekStartPositionAndHandleSplit(reader, topicName, partitionId, MessageId.latest);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
+ TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), null, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+
+ // create consumer and seek before split changes
+ try (Consumer<byte[]> consumer = reader.createPulsarConsumer(partition)) {
+ // inclusive messageId
+ StartCursor startCursor = StartCursor.fromMessageId(startPosition);
+ startCursor.seekPosition(partition.getTopic(), partition.getPartitionId(), consumer);
+ } catch (PulsarClientException e) {
+ sneakyThrow(e);
+ }
+
+ reader.handleSplitsChanges(addition);
}
protected <T> PulsarMessage<T> fetchedMessage(PulsarPartitionSplitReaderBase<T> splitReader) {
- try {
- RecordsWithSplitIds<PulsarMessage<T>> records = splitReader.fetch();
- if (records.nextSplit() != null) {
- return records.nextRecordFromSplit();
+ return fetchedMessages(splitReader, 1, false).stream().findFirst().orElse(null);
+ }
+
+ protected <T> List<PulsarMessage<T>> fetchedMessages(
+ PulsarPartitionSplitReaderBase<T> splitReader, int expectedCount, boolean verify) {
+ return fetchedMessages(
+ splitReader, expectedCount, verify, Boundedness.CONTINUOUS_UNBOUNDED);
+ }
+
+ protected <T> List<PulsarMessage<T>> fetchedMessages(
Review comment:
Please move these methods to a util class or test extension
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assertions.fail;
+
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ /** Expected the add or override config entries in inherited classes. */
+ protected abstract void customConfig(Configuration config);
+
+ @BeforeEach
+ void beforeEach(TestInfo testInfo) {
+ Random random = new Random(System.currentTimeMillis());
+ String topicName = uniqueTestName(testInfo);
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(TestInfo testInfo) {
+ String topicName = uniqueTestName(testInfo);
+ operator().deleteTopic(topicName, true);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void assignZeroSplitsCreatesZeroSubscription(
+ boolean autoAcknowledgementEnabled, TestInfo testInfo) throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void assigningEmptySplits(boolean autoAcknowledgementEnabled, TestInfo testInfo)
+ throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // we perform assertion when add splits task
+ // is dequeued but not executed.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ assertEquals(status, InputStatus.NOTHING_AVAILABLE);
+ reader.close();
+ }
+
+ // ---------------------
+ protected String uniqueTestName(TestInfo testInfo) {
Review comment:
Please move all util methods into either a dedicated helper class if they can be static or introduce a new test extension for pulsar
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarOrderedSourceReaderTest.java
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.PulsarSourceOptions;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplits;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@ExtendWith(TestLoggerExtension.class)
+class PulsarOrderedSourceReaderTest extends PulsarSourceReaderTestBase {
+
+ private static final int MAX_EMPTY_POLLING_TIMES = 10;
+
+ @Override
+ protected void customConfig(Configuration config) {
+ config.set(PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE, SubscriptionType.Failover);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void pollMessageWithIntervalLongerThanTimeout(
+ boolean autoAcknowledgementEnabled, TestInfo testInfo) throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ Boundedness boundedness = Boundedness.CONTINUOUS_UNBOUNDED;
+ String topicName = uniqueTestName(testInfo);
+ setupSourceReader(reader, topicName, 0, boundedness);
+ sleepUninterruptibly(15, TimeUnit.SECONDS);
+ operator()
+ .sendMessage(TopicNameUtils.topicNameWithPartition(topicName, 0), Schema.INT32, 5);
+
+ // execute
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ int expectedRecords = NUM_RECORDS_PER_PARTITION + 1;
+ pollUtilReadExpectedNumberOfRecordsAndValidate(
+ reader,
+ output,
+ expectedRecords,
+ TopicNameUtils.topicNameWithPartition(topicName, 0));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void consumeMessagesAndCommitOffsets(boolean autoAcknowledgementEnabled, TestInfo testInfo)
+ throws Exception {
+ // set up the partition
+ PulsarOrderedSourceReader<Integer> reader =
+ (PulsarOrderedSourceReader<Integer>) sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ setupSourceReader(reader, topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED);
+
+ // waiting for results
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ pollUntil(
+ reader,
+ output,
+ () -> output.getEmittedRecords().size() == NUM_RECORDS_PER_PARTITION,
+ "The output didn't poll enough records before timeout.");
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ pollUntil(
+ reader,
+ output,
+ reader.cursorsToCommit::isEmpty,
+ "The offset commit did not finish before timeout.");
+
+ // verify consumption
+ reader.close();
+ verifyAllMessageAcknowledged(
+ NUM_RECORDS_PER_PARTITION, TopicNameUtils.topicNameWithPartition(topicName, 0));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void offsetCommitOnCheckpointComplete(boolean autoAcknowledgementEnabled, TestInfo testInfo)
+ throws Exception {
+ PulsarOrderedSourceReader<Integer> reader =
+ (PulsarOrderedSourceReader<Integer>) sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ // consume more than 1 partition
+ reader.addSplits(
+ createPartitionSplits(
+ topicName, DEFAULT_PARTITIONS, Boundedness.CONTINUOUS_UNBOUNDED));
+ reader.notifyNoMoreSplits();
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ long checkpointId = 0;
+ int emptyResultTime = 0;
+ InputStatus status;
+ do {
+ checkpointId++;
+ status = reader.pollNext(output);
+ // Create a checkpoint for each message consumption, but not complete them.
+ reader.snapshotState(checkpointId);
+ // the first couple of pollNext() might return NOTHING_AVAILABLE before data appears
+ if (InputStatus.NOTHING_AVAILABLE == status) {
+ emptyResultTime++;
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ }
+
+ } while (emptyResultTime < MAX_EMPTY_POLLING_TIMES
+ && status != InputStatus.END_OF_INPUT
+ && output.getEmittedRecords().size()
+ < NUM_RECORDS_PER_PARTITION * DEFAULT_PARTITIONS);
+
+ // The completion of the last checkpoint should subsume all previous checkpoints.
+ assertEquals(checkpointId, reader.cursorsToCommit.size());
+
+ long lastCheckpointId = checkpointId;
+ CommonTestUtils.waitUtil(
+ () -> {
+ try {
+ reader.notifyCheckpointComplete(lastCheckpointId);
Review comment:
Why do we need to call `notifyCheckpointComplete` multiple times? AFAICT the first call should already subsume all checkpoints.
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/PulsarSourceBuilderTest.java
##########
@@ -67,4 +71,60 @@ void rangeGeneratorRequiresKeyShared() {
IllegalArgumentException.class,
() -> builder.setRangeGenerator(new UniformRangeGenerator()));
}
+
+ @Test
+ void missingRequiredField() {
+ PulsarSourceBuilder<String> builder = new PulsarSourceBuilder<>();
+ assertThrows(IllegalArgumentException.class, builder::build);
Review comment:
Recently the community decided to use assertj [1] for new assertions if you rebase your branch to master you can immediately start using it.
[1]
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/enumerator/cursor/start/MessageIdStartCursor.java
##########
@@ -53,9 +53,13 @@ public MessageIdStartCursor(MessageId messageId, boolean inclusive) {
messageId instanceof MessageIdImpl,
"We only support normal message id and batch message id.");
MessageIdImpl id = (MessageIdImpl) messageId;
- this.messageId =
- new MessageIdImpl(
- id.getLedgerId(), id.getEntryId() + 1, id.getPartitionIndex());
+ if (MessageId.earliest.equals(messageId) || MessageId.latest.equals(messageId)) {
+ this.messageId = messageId;
+ } else {
+ this.messageId =
+ new MessageIdImpl(
+ id.getLedgerId(), id.getEntryId() + 1, id.getPartitionIndex());
+ }
Review comment:
I suspect that this is the bug fix? Please use a separate commit with a short explanation of what this change is fixing for the bug fix and the related test to make the git history more comprehensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981288048
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 (Mon Nov 29 04:44:08 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-25044).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r779538374
##########
File path: flink-test-utils-parent/flink-test-utils-junit/pom.xml
##########
@@ -21,6 +21,10 @@ under the License.
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
+ <properties>
+ <maven.compiler.source>8</maven.compiler.source>
+ <maven.compiler.target>8</maven.compiler.target>
+ </properties>
Review comment:
This should be deleted (I added this line for local debugging purposes), sorry for importing this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r779542238
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // The fetcher could become idle upon entry because :
+ // 1. the AddSplitsTask is dequeued from taskQueue,
+ // 2. The dequeued splits is not assigned to assignedSplits.
+ // And if both assignedSplits is null and taskQueue is empty, the fetcher could become idle,
+ // and return NOTHING_AVAILABLE upon entry. This happened multiple times during local test.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ assertThat(status).isEqualTo(InputStatus.NOTHING_AVAILABLE);
+ reader.close();
+ }
+
+ protected void setupSourceReader(
Review comment:
I'll try to avoid using this sleep and see if we have other choices
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r779543145
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
Review comment:
Basically the new test cases covers this feature change. Without this fix, one of the newly added test will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul merged pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
fapaul merged pull request #17937:
URL: https://github.com/apache/flink/pull/17937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
}, {
"hash" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220",
"triggerID" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e45e95974b2d7229be6f04f51828d09b50a5486d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r763963921
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -126,63 +196,116 @@ void pollMessageAfterTimeout(PulsarPartitionSplitReaderBase<String> splitReader)
"Couldn't poll message from Pulsar.");
}
- /** Create a split reader with max message 1, fetch timeout 1s. */
- protected abstract PulsarPartitionSplitReaderBase<String> splitReader();
+ @Test
+ void consumeMessageCreatedAfterHandleSplitChangesAndFetch() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0);
+ operator().sendMessage(topicNameWithPartition(topicName, 0), STRING, randomAlphabetic(10));
+ fetchedMessages(splitReader, 1, true);
+ }
- /** JUnit5 extension for all the TestTemplate methods in this class. */
- public class PulsarSplitReaderInvocationContextProvider
- implements TestTemplateInvocationContextProvider {
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChanges() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
- @Override
- public boolean supportsTestTemplate(ExtensionContext context) {
- return true;
- }
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0, MessageId.earliest);
+ fetchedMessages(splitReader, NUM_RECORDS_PER_PARTITION, true);
+ }
- @Override
- public Stream<TestTemplateInvocationContext> provideTestTemplateInvocationContexts(
- ExtensionContext context) {
- return Stream.of(new PulsarSplitReaderInvocationContext(splitReader()));
- }
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndResetToLatestPosition() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0, MessageId.latest);
+ fetchedMessages(splitReader, 0, true);
}
- /** Parameter resolver for Split Reader. */
- public static class PulsarSplitReaderInvocationContext
- implements TestTemplateInvocationContext {
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndUseSecondLastMessageIdCursor() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ MessageIdImpl lastMessageId =
+ (MessageIdImpl)
+ sneakyAdmin(
+ () ->
+ operator()
+ .admin()
+ .topics()
+ .getLastMessageId(
+ topicNameWithPartition(topicName, 0)));
+ // when doing seek directly on consumer, by default it includes the specified messageId
+ seekStartPositionAndHandleSplit(
+ splitReader,
+ topicName,
+ 0,
+ new MessageIdImpl(
+ lastMessageId.getLedgerId(),
+ lastMessageId.getEntryId() - 1,
+ lastMessageId.getPartitionIndex()));
+ fetchedMessages(splitReader, 2, true);
+ }
- private final PulsarPartitionSplitReaderBase<?> splitReader;
+ @Test
+ void emptyTopic() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().createTopic(topicName, DEFAULT_PARTITIONS);
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
- public PulsarSplitReaderInvocationContext(PulsarPartitionSplitReaderBase<?> splitReader) {
- this.splitReader = checkNotNull(splitReader);
- }
+ @Test
+ void emptyTopicWithoutSeek() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().createTopic(topicName, DEFAULT_PARTITIONS);
+ handleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
- @Override
- public String getDisplayName(int invocationIndex) {
- return splitReader.getClass().getSimpleName();
+ @Test
+ void wakeupSplitReaderShouldNotCauseException() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ handleSplit(splitReader, "non-exist", 0);
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ Thread t =
+ new Thread(
+ () -> {
+ try {
+ splitReader.fetch();
+ } catch (Throwable e) {
+ error.set(e);
+ }
+ },
+ "testWakeUp-thread");
+ t.start();
+ long deadline = System.currentTimeMillis() + 5000L;
+ while (t.isAlive() && System.currentTimeMillis() < deadline) {
+ splitReader.wakeUp();
+ sleepUninterruptibly(10, TimeUnit.MILLISECONDS);
}
+ assertNull(error.get());
+ }
- @Override
- public List<Extension> getAdditionalExtensions() {
- return Collections.singletonList(
- new ParameterResolver() {
- @Override
- public boolean supportsParameter(
- ParameterContext parameterContext,
- ExtensionContext extensionContext)
- throws ParameterResolutionException {
- return parameterContext
- .getParameter()
- .getType()
- .equals(PulsarPartitionSplitReaderBase.class);
- }
-
- @Override
- public Object resolveParameter(
- ParameterContext parameterContext,
- ExtensionContext extensionContext)
- throws ParameterResolutionException {
- return splitReader;
- }
- });
- }
+ @Test
+ void assignNoSplits() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ assertNull(fetchedMessage(splitReader));
}
+
+ /** Create a split reader with max message 1, fetch timeout 1s. */
+ protected abstract PulsarPartitionSplitReaderBase<String> splitReader();
Review comment:
@fapaul Do you mean use @TestTemplate and a custom TestInvocationContextProvider ? Me and @syhily used @TestTemplate and TestInvocationContext and then refactored to what is looks like now (cause we thought they were an overkill in our case). But sure I can switch back to use the TestInvocationContextProvider for extensibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r763938676
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarOrderedSourceReaderTest.java
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.PulsarSourceOptions;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplits;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@ExtendWith(TestLoggerExtension.class)
+class PulsarOrderedSourceReaderTest extends PulsarSourceReaderTestBase {
+
+ private static final int MAX_EMPTY_POLLING_TIMES = 10;
+
+ @Override
+ protected void customConfig(Configuration config) {
+ config.set(PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE, SubscriptionType.Failover);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void pollMessageWithIntervalLongerThanTimeout(
+ boolean autoAcknowledgementEnabled, TestInfo testInfo) throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ Boundedness boundedness = Boundedness.CONTINUOUS_UNBOUNDED;
+ String topicName = uniqueTestName(testInfo);
+ setupSourceReader(reader, topicName, 0, boundedness);
+ sleepUninterruptibly(15, TimeUnit.SECONDS);
+ operator()
+ .sendMessage(TopicNameUtils.topicNameWithPartition(topicName, 0), Schema.INT32, 5);
+
+ // execute
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ int expectedRecords = NUM_RECORDS_PER_PARTITION + 1;
+ pollUtilReadExpectedNumberOfRecordsAndValidate(
+ reader,
+ output,
+ expectedRecords,
+ TopicNameUtils.topicNameWithPartition(topicName, 0));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void consumeMessagesAndCommitOffsets(boolean autoAcknowledgementEnabled, TestInfo testInfo)
+ throws Exception {
+ // set up the partition
+ PulsarOrderedSourceReader<Integer> reader =
+ (PulsarOrderedSourceReader<Integer>) sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ setupSourceReader(reader, topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED);
+
+ // waiting for results
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ pollUntil(
+ reader,
+ output,
+ () -> output.getEmittedRecords().size() == NUM_RECORDS_PER_PARTITION,
+ "The output didn't poll enough records before timeout.");
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ pollUntil(
+ reader,
+ output,
+ reader.cursorsToCommit::isEmpty,
+ "The offset commit did not finish before timeout.");
+
+ // verify consumption
+ reader.close();
+ verifyAllMessageAcknowledged(
+ NUM_RECORDS_PER_PARTITION, TopicNameUtils.topicNameWithPartition(topicName, 0));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void offsetCommitOnCheckpointComplete(boolean autoAcknowledgementEnabled, TestInfo testInfo)
+ throws Exception {
+ PulsarOrderedSourceReader<Integer> reader =
+ (PulsarOrderedSourceReader<Integer>) sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ // consume more than 1 partition
+ reader.addSplits(
+ createPartitionSplits(
+ topicName, DEFAULT_PARTITIONS, Boundedness.CONTINUOUS_UNBOUNDED));
+ reader.notifyNoMoreSplits();
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ long checkpointId = 0;
+ int emptyResultTime = 0;
+ InputStatus status;
+ do {
+ checkpointId++;
+ status = reader.pollNext(output);
+ // Create a checkpoint for each message consumption, but not complete them.
+ reader.snapshotState(checkpointId);
+ // the first couple of pollNext() might return NOTHING_AVAILABLE before data appears
+ if (InputStatus.NOTHING_AVAILABLE == status) {
+ emptyResultTime++;
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ }
+
+ } while (emptyResultTime < MAX_EMPTY_POLLING_TIMES
+ && status != InputStatus.END_OF_INPUT
+ && output.getEmittedRecords().size()
+ < NUM_RECORDS_PER_PARTITION * DEFAULT_PARTITIONS);
+
+ // The completion of the last checkpoint should subsume all previous checkpoints.
+ assertEquals(checkpointId, reader.cursorsToCommit.size());
+
+ long lastCheckpointId = checkpointId;
+ CommonTestUtils.waitUtil(
+ () -> {
+ try {
+ reader.notifyCheckpointComplete(lastCheckpointId);
Review comment:
I think the logic is when the checkpoint complete and the cursorsToCommit is empty, then it means the checkpoint is OK. It will run multiple times if there is an exception. Ideally after one run the cursorsToCommit should become empty. But I do agree maybe we can optimize this so that if one exception is thrown, we consider the test as fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-1000620891
@fapaul Hi Fabian, I've updated this PR, would you mind take a look ? Thanks ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-998834424
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r783144119
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarOrderedSourceReaderTest.java
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.testutils.extension.SubType;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.core.io.InputStatus;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.junit.jupiter.api.TestTemplate;
+
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplits;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
+
+class PulsarOrderedSourceReaderTest extends PulsarSourceReaderTestBase {
+
+ private static final int MAX_EMPTY_POLLING_TIMES = 10;
+
+ @SubType SubscriptionType subscriptionType = SubscriptionType.Failover;
+
+ @TestTemplate
+ void pollMessageWithIntervalLongerThanTimeout(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ setupSourceReader(reader, topicName, 0, boundedness);
+ sleepUninterruptibly(15, TimeUnit.SECONDS);
Review comment:
@imaffe can you have a look at this?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
Review comment:
seems not addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-1012970412
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r779540715
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/extension/PulsarSourceOrderlinessExtension.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.testutils.extension;
+
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.platform.commons.support.AnnotationSupport;
+
+import java.lang.annotation.Annotation;
+import java.util.Collection;
+import java.util.List;
+
+/** An extension for subclasses to specify {@link org.apache.pulsar.client.api.SubscriptionType}. */
+public class PulsarSourceOrderlinessExtension implements BeforeAllCallback {
Review comment:
The word choice is a bit weird, but what I want to express here is this extension passes either ordered or unordered as a parameter to the tes. I'll try to rephrase it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398)
* c1728e5765b33b6ba1140a2f313687eb3bbbaf5f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-999984059
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1728e5765b33b6ba1140a2f313687eb3bbbaf5f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-987754418
> Thanks for the additional tests it will definitely help to increase the stability of the connector 🙏
>
> I left a few remarks regarding the test setup and had some questions. Can you also try to split the PR in separate commits?
>
> 1. Bug fix code + test
> 2. More tests (Maybe split again to ease review i.e. one commit for splitreader one for sourcereader)
Great, I'll be working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r763936530
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/enumerator/PulsarSourceEnumeratorTest.java
##########
@@ -112,64 +119,270 @@ void startWithPeriodicPartitionDiscovery() throws Exception {
}
}
- private PulsarSourceEnumerator createEnumerator(
+ @ParameterizedTest
+ @EnumSource(
+ value = SubscriptionType.class,
+ names = {"Failover", "Shared"})
+ void discoverPartitionsTriggersAssignments(SubscriptionType subscriptionType) throws Throwable {
+ Set<String> prexistingTopics = setupPreexistingTopics();
+ try (MockSplitEnumeratorContext<PulsarPartitionSplit> context =
+ new MockSplitEnumeratorContext<>(NUM_SUBTASKS);
+ PulsarSourceEnumerator enumerator =
+ createEnumerator(
+ subscriptionType,
+ prexistingTopics,
+ context,
+ DISABLE_PERIODIC_PARTITION_DISCOVERY)) {
+
+ enumerator.start();
+
+ // register reader 0, 1
+ registerReader(context, enumerator, READER0);
+ registerReader(context, enumerator, READER1);
+ assertTrue(context.getSplitsAssignmentSequence().isEmpty());
+
+ // Run the partition discover callable and check the partition assignment.
+ runOneTimePartitionDiscovery(context);
+ verifyLastReadersAssignments(subscriptionType, context, prexistingTopics, 1);
+ }
+ }
+
+ @ParameterizedTest
+ @EnumSource(
+ value = SubscriptionType.class,
+ names = {"Failover", "Shared"})
+ @Timeout(value = 60)
Review comment:
Yeah I recall there is a azure timeout for each test. If that is the case I don't need the timeout on JUnit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
}, {
"hash" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220",
"triggerID" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"triggerType" : "PUSH"
}, {
"hash" : "248894b6ec7935722944edf641753b99f3a0ec38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29394",
"triggerID" : "248894b6ec7935722944edf641753b99f3a0ec38",
"triggerType" : "PUSH"
}, {
"hash" : "5bffc54d9cfaa9245563f623b76b98cf35e1ff81",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1012970412",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1012970412",
"triggerType" : "MANUAL"
}, {
"hash" : "5bffc54d9cfaa9245563f623b76b98cf35e1ff81",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5bffc54d9cfaa9245563f623b76b98cf35e1ff81",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 5bffc54d9cfaa9245563f623b76b98cf35e1ff81 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-1011826129
> I think two comments are not addressed but everything else looks good now, thanks for the patience. Can you squash all the commits together that the PR only includes two commits one for the bug fix and one for the tests? Nit: Next time it would be great if you mark the comments you have addressed :)
The first case can be deleted. The edge case it tries to test is addressed in the SplitReaderTest. Basically what we want to do here is to wait for a certain timeout between two messages in the pulsar client and see how it behaves. Do we have suggestions on how to test these kinds of behaviour? (needs to wait for an external system to timeout, feels we can't use a blocking queue to signal a event complete here).
The second place, I'll set the polling timeout to a large number. And rely on the global timeout to terminate execution if the end condition is not met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r781783619
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
Review comment:
I changed private and moved methods to subclasses ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r763940584
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assertions.fail;
+
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ /** Expected the add or override config entries in inherited classes. */
+ protected abstract void customConfig(Configuration config);
+
+ @BeforeEach
+ void beforeEach(TestInfo testInfo) {
+ Random random = new Random(System.currentTimeMillis());
+ String topicName = uniqueTestName(testInfo);
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(TestInfo testInfo) {
+ String topicName = uniqueTestName(testInfo);
+ operator().deleteTopic(topicName, true);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void assignZeroSplitsCreatesZeroSubscription(
+ boolean autoAcknowledgementEnabled, TestInfo testInfo) throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void assigningEmptySplits(boolean autoAcknowledgementEnabled, TestInfo testInfo)
+ throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // we perform assertion when add splits task
+ // is dequeued but not executed.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
Review comment:
Oh, there is an edge case. Basically the fetcher could become idle upon entry because : 1. the AddSplitsTask is dequeued from taskQueue, 2. The dequeued splits is not assigned to `assignedSplits`.And if both `assignedSplits` is null and taskQueue is empty, the fetcher could become idle, and return NOTHING_AVAILABLE upon entry. This happened multiple times during local test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-1012304383
Looks good please squash and rebase then I'll take care of merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
}, {
"hash" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220",
"triggerID" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"triggerType" : "PUSH"
}, {
"hash" : "248894b6ec7935722944edf641753b99f3a0ec38",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29394",
"triggerID" : "248894b6ec7935722944edf641753b99f3a0ec38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e45e95974b2d7229be6f04f51828d09b50a5486d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220)
* 248894b6ec7935722944edf641753b99f3a0ec38 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29394)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-1012110340
It includes 3 commits, if the newest is ok, I'll squash to 2 commits ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
}, {
"hash" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220",
"triggerID" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48c44601bb0c15a3975d8ad41249231f06cbced2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073)
* e45e95974b2d7229be6f04f51828d09b50a5486d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* c1728e5765b33b6ba1140a2f313687eb3bbbaf5f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r778886160
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // The fetcher could become idle upon entry because :
+ // 1. the AddSplitsTask is dequeued from taskQueue,
+ // 2. The dequeued splits is not assigned to assignedSplits.
+ // And if both assignedSplits is null and taskQueue is empty, the fetcher could become idle,
+ // and return NOTHING_AVAILABLE upon entry. This happened multiple times during local test.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ assertThat(status).isEqualTo(InputStatus.NOTHING_AVAILABLE);
+ reader.close();
+ }
+
+ protected void setupSourceReader(
Review comment:
private?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
Review comment:
Nit: why the split?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // The fetcher could become idle upon entry because :
+ // 1. the AddSplitsTask is dequeued from taskQueue,
+ // 2. The dequeued splits is not assigned to assignedSplits.
+ // And if both assignedSplits is null and taskQueue is empty, the fetcher could become idle,
+ // and return NOTHING_AVAILABLE upon entry. This happened multiple times during local test.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ assertThat(status).isEqualTo(InputStatus.NOTHING_AVAILABLE);
+ reader.close();
+ }
+
+ protected void setupSourceReader(
+ PulsarSourceReaderBase<Integer> reader,
+ String topicName,
+ int partitionId,
+ Boundedness boundedness) {
+ PulsarPartitionSplit split = createPartitionSplit(topicName, partitionId, boundedness);
+ reader.addSplits(Collections.singletonList(split));
+ reader.notifyNoMoreSplits();
+ }
+
+ protected void pollUtilReadExpectedNumberOfRecordsAndValidate(
+ PulsarSourceReaderBase<Integer> reader,
+ TestingReaderOutput<Integer> output,
+ int expectedRecords,
+ String topicNameWithPartition)
+ throws Exception {
+ pollUntil(
+ reader,
+ output,
+ () -> output.getEmittedRecords().size() == expectedRecords,
+ "The output didn't poll enough records before timeout.");
+ reader.close();
+ verifyAllMessageAcknowledged(expectedRecords, topicNameWithPartition);
+ assertThat(output.getEmittedRecords()).hasSize(expectedRecords);
+ }
+
+ protected void pollUntil(
Review comment:
This can be a static util I do not an instance reference
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // The fetcher could become idle upon entry because :
+ // 1. the AddSplitsTask is dequeued from taskQueue,
+ // 2. The dequeued splits is not assigned to assignedSplits.
+ // And if both assignedSplits is null and taskQueue is empty, the fetcher could become idle,
+ // and return NOTHING_AVAILABLE upon entry. This happened multiple times during local test.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ assertThat(status).isEqualTo(InputStatus.NOTHING_AVAILABLE);
+ reader.close();
+ }
+
+ protected void setupSourceReader(
+ PulsarSourceReaderBase<Integer> reader,
+ String topicName,
+ int partitionId,
+ Boundedness boundedness) {
+ PulsarPartitionSplit split = createPartitionSplit(topicName, partitionId, boundedness);
+ reader.addSplits(Collections.singletonList(split));
+ reader.notifyNoMoreSplits();
+ }
+
+ protected void pollUtilReadExpectedNumberOfRecordsAndValidate(
+ PulsarSourceReaderBase<Integer> reader,
+ TestingReaderOutput<Integer> output,
+ int expectedRecords,
+ String topicNameWithPartition)
+ throws Exception {
+ pollUntil(
+ reader,
+ output,
+ () -> output.getEmittedRecords().size() == expectedRecords,
+ "The output didn't poll enough records before timeout.");
+ reader.close();
+ verifyAllMessageAcknowledged(expectedRecords, topicNameWithPartition);
+ assertThat(output.getEmittedRecords()).hasSize(expectedRecords);
+ }
+
+ protected void pollUntil(
+ PulsarSourceReaderBase<Integer> reader,
+ ReaderOutput<Integer> output,
+ Supplier<Boolean> condition,
+ String errorMessage)
+ throws InterruptedException, TimeoutException {
+ CommonTestUtils.waitUtil(
+ () -> {
+ try {
+ reader.pollNext(output);
+ } catch (Exception exception) {
+ throw new RuntimeException(
+ "Caught unexpected exception when polling from the reader",
+ exception);
+ }
+ return condition.get();
+ },
+ Duration.ofSeconds(DEFAULT_POLLING_TIMEOUT_SECONDS),
+ errorMessage);
+ }
+
+ protected void verifyAllMessageAcknowledged(int expectedMessages, String partitionName)
Review comment:
I wonder whether we can introduce an assertion helper class that is constructed with the `operator`. So we do not need to use inheritance to share utils
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarOrderedSourceReaderTest.java
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.testutils.extension.SubType;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.core.io.InputStatus;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.junit.jupiter.api.TestTemplate;
+
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplits;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
+
+class PulsarOrderedSourceReaderTest extends PulsarSourceReaderTestBase {
+
+ private static final int MAX_EMPTY_POLLING_TIMES = 10;
+
+ @SubType SubscriptionType subscriptionType = SubscriptionType.Failover;
+
+ @TestTemplate
+ void pollMessageWithIntervalLongerThanTimeout(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ setupSourceReader(reader, topicName, 0, boundedness);
+ sleepUninterruptibly(15, TimeUnit.SECONDS);
Review comment:
Can you make the test faster than 15 seconds?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // The fetcher could become idle upon entry because :
+ // 1. the AddSplitsTask is dequeued from taskQueue,
+ // 2. The dequeued splits is not assigned to assignedSplits.
+ // And if both assignedSplits is null and taskQueue is empty, the fetcher could become idle,
+ // and return NOTHING_AVAILABLE upon entry. This happened multiple times during local test.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ assertThat(status).isEqualTo(InputStatus.NOTHING_AVAILABLE);
+ reader.close();
+ }
+
+ protected void setupSourceReader(
+ PulsarSourceReaderBase<Integer> reader,
+ String topicName,
+ int partitionId,
+ Boundedness boundedness) {
+ PulsarPartitionSplit split = createPartitionSplit(topicName, partitionId, boundedness);
+ reader.addSplits(Collections.singletonList(split));
+ reader.notifyNoMoreSplits();
+ }
+
+ protected void pollUtilReadExpectedNumberOfRecordsAndValidate(
Review comment:
Move to PulsarOrderedSourceReaderTest?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
@RegisterExtension
PulsarSplitReaderInvocationContextProvider provider =
new PulsarSplitReaderInvocationContextProvider();
+ /** Default reader config: max message 1, fetch timeout 1s. */
protected Configuration readerConfig() {
Configuration config = operator().config();
config.set(PULSAR_MAX_FETCH_RECORDS, 1);
config.set(PULSAR_MAX_FETCH_TIME, 1000L);
config.set(PULSAR_SUBSCRIPTION_NAME, randomAlphabetic(10));
config.set(PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE, true);
-
return config;
}
protected SourceConfiguration sourceConfig() {
return new SourceConfiguration(readerConfig());
}
- protected SplitsAddition<PulsarPartitionSplit> createSplit(String topicName, int partitionId) {
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ handleSplit(reader, topicName, partitionId, null);
+ }
+
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
+ TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), startPosition, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+ reader.handleSplitsChanges(addition);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ seekStartPositionAndHandleSplit(reader, topicName, partitionId, MessageId.latest);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
Review comment:
private?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
@RegisterExtension
PulsarSplitReaderInvocationContextProvider provider =
new PulsarSplitReaderInvocationContextProvider();
+ /** Default reader config: max message 1, fetch timeout 1s. */
protected Configuration readerConfig() {
Review comment:
private?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
@RegisterExtension
PulsarSplitReaderInvocationContextProvider provider =
new PulsarSplitReaderInvocationContextProvider();
+ /** Default reader config: max message 1, fetch timeout 1s. */
protected Configuration readerConfig() {
Configuration config = operator().config();
config.set(PULSAR_MAX_FETCH_RECORDS, 1);
config.set(PULSAR_MAX_FETCH_TIME, 1000L);
config.set(PULSAR_SUBSCRIPTION_NAME, randomAlphabetic(10));
config.set(PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE, true);
-
return config;
}
protected SourceConfiguration sourceConfig() {
Review comment:
private?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // The fetcher could become idle upon entry because :
+ // 1. the AddSplitsTask is dequeued from taskQueue,
+ // 2. The dequeued splits is not assigned to assignedSplits.
+ // And if both assignedSplits is null and taskQueue is empty, the fetcher could become idle,
+ // and return NOTHING_AVAILABLE upon entry. This happened multiple times during local test.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ assertThat(status).isEqualTo(InputStatus.NOTHING_AVAILABLE);
+ reader.close();
+ }
+
+ protected void setupSourceReader(
+ PulsarSourceReaderBase<Integer> reader,
+ String topicName,
+ int partitionId,
+ Boundedness boundedness) {
+ PulsarPartitionSplit split = createPartitionSplit(topicName, partitionId, boundedness);
+ reader.addSplits(Collections.singletonList(split));
+ reader.notifyNoMoreSplits();
+ }
+
+ protected void pollUtilReadExpectedNumberOfRecordsAndValidate(
+ PulsarSourceReaderBase<Integer> reader,
+ TestingReaderOutput<Integer> output,
+ int expectedRecords,
+ String topicNameWithPartition)
+ throws Exception {
+ pollUntil(
+ reader,
+ output,
+ () -> output.getEmittedRecords().size() == expectedRecords,
+ "The output didn't poll enough records before timeout.");
+ reader.close();
+ verifyAllMessageAcknowledged(expectedRecords, topicNameWithPartition);
+ assertThat(output.getEmittedRecords()).hasSize(expectedRecords);
+ }
+
+ protected void pollUntil(
+ PulsarSourceReaderBase<Integer> reader,
+ ReaderOutput<Integer> output,
+ Supplier<Boolean> condition,
+ String errorMessage)
+ throws InterruptedException, TimeoutException {
+ CommonTestUtils.waitUtil(
+ () -> {
+ try {
+ reader.pollNext(output);
+ } catch (Exception exception) {
+ throw new RuntimeException(
+ "Caught unexpected exception when polling from the reader",
+ exception);
+ }
+ return condition.get();
+ },
+ Duration.ofSeconds(DEFAULT_POLLING_TIMEOUT_SECONDS),
+ errorMessage);
+ }
+
+ protected void verifyAllMessageAcknowledged(int expectedMessages, String partitionName)
+ throws PulsarAdminException {
+ TopicStats topicStats = operator().admin().topics().getStats(partitionName, true, true);
+ // verify if the messages has been consumed
+ Map<String, ? extends SubscriptionStats> subscriptionStats = topicStats.getSubscriptions();
+ assertThat(subscriptionStats).hasSizeGreaterThan(0);
+ subscriptionStats.forEach(
+ (subscription, stats) -> {
+ assertThat(stats.getUnackedMessages()).isZero();
+ assertThat(stats.getMsgOutCounter()).isEqualTo(expectedMessages);
+ });
+ }
+
+ protected void verifyNoSubscriptionCreated(String partitionName) throws PulsarAdminException {
Review comment:
private
##########
File path: flink-test-utils-parent/flink-test-utils-junit/pom.xml
##########
@@ -21,6 +21,10 @@ under the License.
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
+ <properties>
+ <maven.compiler.source>8</maven.compiler.source>
+ <maven.compiler.target>8</maven.compiler.target>
+ </properties>
Review comment:
Why is this needed? I am not sure whether we want to do this because usually all modules should work with java 8 and java 11
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assertions.fail;
+
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ /** Expected the add or override config entries in inherited classes. */
+ protected abstract void customConfig(Configuration config);
+
+ @BeforeEach
+ void beforeEach(TestInfo testInfo) {
+ Random random = new Random(System.currentTimeMillis());
+ String topicName = uniqueTestName(testInfo);
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(TestInfo testInfo) {
+ String topicName = uniqueTestName(testInfo);
+ operator().deleteTopic(topicName, true);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void assignZeroSplitsCreatesZeroSubscription(
+ boolean autoAcknowledgementEnabled, TestInfo testInfo) throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void assigningEmptySplits(boolean autoAcknowledgementEnabled, TestInfo testInfo)
+ throws Exception {
+ PulsarSourceReaderBase<Integer> reader = sourceReader(autoAcknowledgementEnabled);
+ String topicName = uniqueTestName(testInfo);
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // we perform assertion when add splits task
+ // is dequeued but not executed.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
Review comment:
I still do not understand this point if `pollNext` returns we already have the inputStatus and it is not requested again. So why the sleep?
In general, the situation you described cannot be reliably solved with a sleep because it may take longer than one second until the fetcher it idle.
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
@RegisterExtension
PulsarSplitReaderInvocationContextProvider provider =
new PulsarSplitReaderInvocationContextProvider();
+ /** Default reader config: max message 1, fetch timeout 1s. */
protected Configuration readerConfig() {
Configuration config = operator().config();
config.set(PULSAR_MAX_FETCH_RECORDS, 1);
config.set(PULSAR_MAX_FETCH_TIME, 1000L);
config.set(PULSAR_SUBSCRIPTION_NAME, randomAlphabetic(10));
config.set(PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE, true);
-
return config;
}
protected SourceConfiguration sourceConfig() {
return new SourceConfiguration(readerConfig());
}
- protected SplitsAddition<PulsarPartitionSplit> createSplit(String topicName, int partitionId) {
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ handleSplit(reader, topicName, partitionId, null);
+ }
+
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
+ TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), startPosition, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+ reader.handleSplitsChanges(addition);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ seekStartPositionAndHandleSplit(reader, topicName, partitionId, MessageId.latest);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
- PulsarPartitionSplit split = new PulsarPartitionSplit(partition, never());
- return new SplitsAddition<>(singletonList(split));
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), null, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+
+ // create consumer and seek before split changes
+ try (Consumer<byte[]> consumer = reader.createPulsarConsumer(partition)) {
+ // inclusive messageId
+ StartCursor startCursor = StartCursor.fromMessageId(startPosition);
+ startCursor.seekPosition(partition.getTopic(), partition.getPartitionId(), consumer);
+ } catch (PulsarClientException e) {
+ sneakyThrow(e);
+ }
+
+ reader.handleSplitsChanges(addition);
}
protected <T> PulsarMessage<T> fetchedMessage(PulsarPartitionSplitReaderBase<T> splitReader) {
Review comment:
private?
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/enumerator/cursor/start/MessageIdStartCursor.java
##########
@@ -53,9 +53,13 @@ public MessageIdStartCursor(MessageId messageId, boolean inclusive) {
messageId instanceof MessageIdImpl,
"We only support normal message id and batch message id.");
MessageIdImpl id = (MessageIdImpl) messageId;
- this.messageId =
- new MessageIdImpl(
- id.getLedgerId(), id.getEntryId() + 1, id.getPartitionIndex());
+ if (MessageId.earliest.equals(messageId) || MessageId.latest.equals(messageId)) {
+ this.messageId = messageId;
+ } else {
+ this.messageId =
+ new MessageIdImpl(
+ id.getLedgerId(), id.getEntryId() + 1, id.getPartitionIndex());
+ }
Review comment:
Maybe I am missing something on this commit but I see you have fixed the production but I would also expect to see a test case on this commit to verify the fix.
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
@RegisterExtension
PulsarSplitReaderInvocationContextProvider provider =
new PulsarSplitReaderInvocationContextProvider();
+ /** Default reader config: max message 1, fetch timeout 1s. */
protected Configuration readerConfig() {
Configuration config = operator().config();
config.set(PULSAR_MAX_FETCH_RECORDS, 1);
config.set(PULSAR_MAX_FETCH_TIME, 1000L);
config.set(PULSAR_SUBSCRIPTION_NAME, randomAlphabetic(10));
config.set(PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE, true);
-
return config;
}
protected SourceConfiguration sourceConfig() {
return new SourceConfiguration(readerConfig());
}
- protected SplitsAddition<PulsarPartitionSplit> createSplit(String topicName, int partitionId) {
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ handleSplit(reader, topicName, partitionId, null);
+ }
+
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
+ TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), startPosition, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+ reader.handleSplitsChanges(addition);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
Review comment:
private?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/extension/PulsarSourceOrderlinessExtension.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.testutils.extension;
+
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.platform.commons.support.AnnotationSupport;
+
+import java.lang.annotation.Annotation;
+import java.util.Collection;
+import java.util.List;
+
+/** An extension for subclasses to specify {@link org.apache.pulsar.client.api.SubscriptionType}. */
+public class PulsarSourceOrderlinessExtension implements BeforeAllCallback {
Review comment:
Is orderliness the same as ordered here?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
Review comment:
private
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
@RegisterExtension
PulsarSplitReaderInvocationContextProvider provider =
new PulsarSplitReaderInvocationContextProvider();
+ /** Default reader config: max message 1, fetch timeout 1s. */
protected Configuration readerConfig() {
Configuration config = operator().config();
config.set(PULSAR_MAX_FETCH_RECORDS, 1);
config.set(PULSAR_MAX_FETCH_TIME, 1000L);
config.set(PULSAR_SUBSCRIPTION_NAME, randomAlphabetic(10));
config.set(PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE, true);
-
return config;
}
protected SourceConfiguration sourceConfig() {
return new SourceConfiguration(readerConfig());
}
- protected SplitsAddition<PulsarPartitionSplit> createSplit(String topicName, int partitionId) {
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ handleSplit(reader, topicName, partitionId, null);
+ }
+
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
+ TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), startPosition, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+ reader.handleSplitsChanges(addition);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ seekStartPositionAndHandleSplit(reader, topicName, partitionId, MessageId.latest);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
- PulsarPartitionSplit split = new PulsarPartitionSplit(partition, never());
- return new SplitsAddition<>(singletonList(split));
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), null, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+
+ // create consumer and seek before split changes
+ try (Consumer<byte[]> consumer = reader.createPulsarConsumer(partition)) {
+ // inclusive messageId
+ StartCursor startCursor = StartCursor.fromMessageId(startPosition);
+ startCursor.seekPosition(partition.getTopic(), partition.getPartitionId(), consumer);
+ } catch (PulsarClientException e) {
+ sneakyThrow(e);
+ }
+
+ reader.handleSplitsChanges(addition);
}
protected <T> PulsarMessage<T> fetchedMessage(PulsarPartitionSplitReaderBase<T> splitReader) {
- try {
- RecordsWithSplitIds<PulsarMessage<T>> records = splitReader.fetch();
- if (records.nextSplit() != null) {
- return records.nextRecordFromSplit();
+ return fetchedMessages(splitReader, 1, false).stream().findFirst().orElse(null);
+ }
+
+ protected <T> List<PulsarMessage<T>> fetchedMessages(
+ PulsarPartitionSplitReaderBase<T> splitReader, int expectedCount, boolean verify) {
+ return fetchedMessages(
+ splitReader, expectedCount, verify, Boundedness.CONTINUOUS_UNBOUNDED);
+ }
+
+ protected <T> List<PulsarMessage<T>> fetchedMessages(
Review comment:
private?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
Review comment:
A lot of the methods of this class seem to unnecessarily be protected either they can be private or even better provide util outside if they do not work with an instance reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1728e5765b33b6ba1140a2f313687eb3bbbaf5f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460)
* 48c44601bb0c15a3975d8ad41249231f06cbced2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r781031713
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
Review comment:
In general, I think having less visibility is preferable so using private where ever possible. Furthermore, there is also a saying that one should use composition over inheritance. Every time I want to use inheritance I think for a moment if there is no other way before implementing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r763972317
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -18,95 +18,165 @@
package org.apache.flink.connector.pulsar.source.reader.split;
+import org.apache.flink.api.connector.source.Boundedness;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.connector.base.source.reader.RecordsWithSplitIds;
import org.apache.flink.connector.base.source.reader.splitreader.SplitsAddition;
import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.cursor.StartCursor;
+import org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor;
import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicPartition;
import org.apache.flink.connector.pulsar.source.reader.message.PulsarMessage;
import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
-import org.junit.jupiter.api.DisplayName;
-import org.junit.jupiter.api.TestTemplate;
-import org.junit.jupiter.api.extension.Extension;
-import org.junit.jupiter.api.extension.ExtensionContext;
-import org.junit.jupiter.api.extension.ParameterContext;
-import org.junit.jupiter.api.extension.ParameterResolutionException;
-import org.junit.jupiter.api.extension.ParameterResolver;
-import org.junit.jupiter.api.extension.RegisterExtension;
-import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
-import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.junit.jupiter.api.Test;
import java.io.IOException;
-import java.util.Collections;
+import java.util.ArrayList;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
-import java.util.stream.Stream;
+import java.util.concurrent.atomic.AtomicReference;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
-import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.pulsar.client.api.Schema.STRING;
+import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertNull;
/** Test utils for split readers. */
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
- @RegisterExtension
- PulsarSplitReaderInvocationContextProvider provider =
- new PulsarSplitReaderInvocationContextProvider();
-
+ /** Default reader config: max message 1, fetch timeout 1s. */
protected Configuration readerConfig() {
Configuration config = operator().config();
config.set(PULSAR_MAX_FETCH_RECORDS, 1);
config.set(PULSAR_MAX_FETCH_TIME, 1000L);
config.set(PULSAR_SUBSCRIPTION_NAME, randomAlphabetic(10));
config.set(PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE, true);
-
return config;
}
protected SourceConfiguration sourceConfig() {
return new SourceConfiguration(readerConfig());
}
- protected SplitsAddition<PulsarPartitionSplit> createSplit(String topicName, int partitionId) {
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ handleSplit(reader, topicName, partitionId, null);
+ }
+
+ protected void handleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
- PulsarPartitionSplit split = new PulsarPartitionSplit(partition, never());
- return new SplitsAddition<>(singletonList(split));
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), startPosition, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+ reader.handleSplitsChanges(addition);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId) {
+ seekStartPositionAndHandleSplit(reader, topicName, partitionId, MessageId.latest);
+ }
+
+ protected void seekStartPositionAndHandleSplit(
+ PulsarPartitionSplitReaderBase<String> reader,
+ String topicName,
+ int partitionId,
+ MessageId startPosition) {
+ TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
+ PulsarPartitionSplit split =
+ new PulsarPartitionSplit(partition, StopCursor.never(), null, null);
+ SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
+
+ // create consumer and seek before split changes
+ try (Consumer<byte[]> consumer = reader.createPulsarConsumer(partition)) {
+ // inclusive messageId
+ StartCursor startCursor = StartCursor.fromMessageId(startPosition);
+ startCursor.seekPosition(partition.getTopic(), partition.getPartitionId(), consumer);
+ } catch (PulsarClientException e) {
+ sneakyThrow(e);
+ }
+
+ reader.handleSplitsChanges(addition);
}
protected <T> PulsarMessage<T> fetchedMessage(PulsarPartitionSplitReaderBase<T> splitReader) {
- try {
- RecordsWithSplitIds<PulsarMessage<T>> records = splitReader.fetch();
- if (records.nextSplit() != null) {
- return records.nextRecordFromSplit();
+ return fetchedMessages(splitReader, 1, false).stream().findFirst().orElse(null);
+ }
+
+ protected <T> List<PulsarMessage<T>> fetchedMessages(
+ PulsarPartitionSplitReaderBase<T> splitReader, int expectedCount, boolean verify) {
+ return fetchedMessages(
+ splitReader, expectedCount, verify, Boundedness.CONTINUOUS_UNBOUNDED);
+ }
+
+ protected <T> List<PulsarMessage<T>> fetchedMessages(
Review comment:
These methods are created to avoid code duplication and only used in this test class. The logic here is tightly coupled with the testing logic. Moving these methods to a common test util class might lose its context when reading it first time. In this case do we still need to move to util class or use test extension (e.g. call fetchedMessages () in BeforeCallback).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* c1728e5765b33b6ba1140a2f313687eb3bbbaf5f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r781784568
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
Review comment:
For composition over inheritance, I totally understand the design considerations, but I kinda feel currently using composition here is a little bit overkill from readability perspective. I think once the testing methods continues to grow (if needed) we can consider using a class to place those methods later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48c44601bb0c15a3975d8ad41249231f06cbced2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r779543145
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
Review comment:
new test cases covers this feature change. Without this fix, one of the newly added test will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r763963921
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -126,63 +196,116 @@ void pollMessageAfterTimeout(PulsarPartitionSplitReaderBase<String> splitReader)
"Couldn't poll message from Pulsar.");
}
- /** Create a split reader with max message 1, fetch timeout 1s. */
- protected abstract PulsarPartitionSplitReaderBase<String> splitReader();
+ @Test
+ void consumeMessageCreatedAfterHandleSplitChangesAndFetch() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0);
+ operator().sendMessage(topicNameWithPartition(topicName, 0), STRING, randomAlphabetic(10));
+ fetchedMessages(splitReader, 1, true);
+ }
- /** JUnit5 extension for all the TestTemplate methods in this class. */
- public class PulsarSplitReaderInvocationContextProvider
- implements TestTemplateInvocationContextProvider {
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChanges() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
- @Override
- public boolean supportsTestTemplate(ExtensionContext context) {
- return true;
- }
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0, MessageId.earliest);
+ fetchedMessages(splitReader, NUM_RECORDS_PER_PARTITION, true);
+ }
- @Override
- public Stream<TestTemplateInvocationContext> provideTestTemplateInvocationContexts(
- ExtensionContext context) {
- return Stream.of(new PulsarSplitReaderInvocationContext(splitReader()));
- }
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndResetToLatestPosition() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0, MessageId.latest);
+ fetchedMessages(splitReader, 0, true);
}
- /** Parameter resolver for Split Reader. */
- public static class PulsarSplitReaderInvocationContext
- implements TestTemplateInvocationContext {
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndUseSecondLastMessageIdCursor() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ MessageIdImpl lastMessageId =
+ (MessageIdImpl)
+ sneakyAdmin(
+ () ->
+ operator()
+ .admin()
+ .topics()
+ .getLastMessageId(
+ topicNameWithPartition(topicName, 0)));
+ // when doing seek directly on consumer, by default it includes the specified messageId
+ seekStartPositionAndHandleSplit(
+ splitReader,
+ topicName,
+ 0,
+ new MessageIdImpl(
+ lastMessageId.getLedgerId(),
+ lastMessageId.getEntryId() - 1,
+ lastMessageId.getPartitionIndex()));
+ fetchedMessages(splitReader, 2, true);
+ }
- private final PulsarPartitionSplitReaderBase<?> splitReader;
+ @Test
+ void emptyTopic() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().createTopic(topicName, DEFAULT_PARTITIONS);
+ seekStartPositionAndHandleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
- public PulsarSplitReaderInvocationContext(PulsarPartitionSplitReaderBase<?> splitReader) {
- this.splitReader = checkNotNull(splitReader);
- }
+ @Test
+ void emptyTopicWithoutSeek() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().createTopic(topicName, DEFAULT_PARTITIONS);
+ handleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
- @Override
- public String getDisplayName(int invocationIndex) {
- return splitReader.getClass().getSimpleName();
+ @Test
+ void wakeupSplitReaderShouldNotCauseException() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ handleSplit(splitReader, "non-exist", 0);
+ AtomicReference<Throwable> error = new AtomicReference<>();
+ Thread t =
+ new Thread(
+ () -> {
+ try {
+ splitReader.fetch();
+ } catch (Throwable e) {
+ error.set(e);
+ }
+ },
+ "testWakeUp-thread");
+ t.start();
+ long deadline = System.currentTimeMillis() + 5000L;
+ while (t.isAlive() && System.currentTimeMillis() < deadline) {
+ splitReader.wakeUp();
+ sleepUninterruptibly(10, TimeUnit.MILLISECONDS);
}
+ assertNull(error.get());
+ }
- @Override
- public List<Extension> getAdditionalExtensions() {
- return Collections.singletonList(
- new ParameterResolver() {
- @Override
- public boolean supportsParameter(
- ParameterContext parameterContext,
- ExtensionContext extensionContext)
- throws ParameterResolutionException {
- return parameterContext
- .getParameter()
- .getType()
- .equals(PulsarPartitionSplitReaderBase.class);
- }
-
- @Override
- public Object resolveParameter(
- ParameterContext parameterContext,
- ExtensionContext extensionContext)
- throws ParameterResolutionException {
- return splitReader;
- }
- });
- }
+ @Test
+ void assignNoSplits() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ assertNull(fetchedMessage(splitReader));
}
+
+ /** Create a split reader with max message 1, fetch timeout 1s. */
+ protected abstract PulsarPartitionSplitReaderBase<String> splitReader();
Review comment:
Do you mean use @TestTemplate and a custom TestInvocationContextProvider ? Me and @syhily used @TestTemplate and TestInvocationContext and then refactored to what is looks like now (cause we thought they were an overkill in our case). But sure I can switch back to use the TestInvocationContextProvider for extensibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r781787236
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
Review comment:
And this new fix should have covered the comments ~ Let me know if we need other fixes or I missed anything ~ Thank you so much @fapaul for valuable feedbacks ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
}, {
"hash" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220",
"triggerID" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"triggerType" : "PUSH"
}, {
"hash" : "248894b6ec7935722944edf641753b99f3a0ec38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29394",
"triggerID" : "248894b6ec7935722944edf641753b99f3a0ec38",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1012970412",
"triggerType" : "MANUAL"
}, {
"hash" : "5bffc54d9cfaa9245563f623b76b98cf35e1ff81",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29411",
"triggerID" : "1012970412",
"triggerType" : "MANUAL"
}, {
"hash" : "5bffc54d9cfaa9245563f623b76b98cf35e1ff81",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29411",
"triggerID" : "5bffc54d9cfaa9245563f623b76b98cf35e1ff81",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 5bffc54d9cfaa9245563f623b76b98cf35e1ff81 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29411)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
}, {
"hash" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220",
"triggerID" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"triggerType" : "PUSH"
}, {
"hash" : "248894b6ec7935722944edf641753b99f3a0ec38",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "248894b6ec7935722944edf641753b99f3a0ec38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e45e95974b2d7229be6f04f51828d09b50a5486d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220)
* 248894b6ec7935722944edf641753b99f3a0ec38 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
}, {
"hash" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29220",
"triggerID" : "e45e95974b2d7229be6f04f51828d09b50a5486d",
"triggerType" : "PUSH"
}, {
"hash" : "248894b6ec7935722944edf641753b99f3a0ec38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29394",
"triggerID" : "248894b6ec7935722944edf641753b99f3a0ec38",
"triggerType" : "PUSH"
}, {
"hash" : "5bffc54d9cfaa9245563f623b76b98cf35e1ff81",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29411",
"triggerID" : "1012970412",
"triggerType" : "MANUAL"
}, {
"hash" : "5bffc54d9cfaa9245563f623b76b98cf35e1ff81",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29411",
"triggerID" : "5bffc54d9cfaa9245563f623b76b98cf35e1ff81",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1012970412",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5bffc54d9cfaa9245563f623b76b98cf35e1ff81 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29411)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r763942770
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarOrderedPartitionSplitReaderTest.java
##########
@@ -34,4 +43,57 @@
sourceConfig(),
flinkSchema(new SimpleStringSchema()));
}
+
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesWithoutSeek() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ handleSplit(splitReader, topicName, 0);
+ fetchedMessages(splitReader, 0, true);
+ }
+
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndUseLatestStartCursorWithoutSeek() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ handleSplit(splitReader, topicName, 0, MessageId.latest);
+ fetchedMessages(splitReader, 0, true);
+ }
+
+ @Test
+ void consumeMessageCreatedBeforeHandleSplitsChangesAndUseEarliestStartCursorWithoutSeek() {
+ PulsarPartitionSplitReaderBase<String> splitReader = splitReader();
+ String topicName = randomAlphabetic(10);
+ operator().setupTopic(topicName, STRING, () -> randomAlphabetic(10));
+ handleSplit(splitReader, topicName, 0, MessageId.earliest);
+ fetchedMessages(splitReader, NUM_RECORDS_PER_PARTITION, true);
+ }
Review comment:
Oh they are different test cases. Using latest and latest - 1 has different semantics (and different code paths). But sure Parameterized test will also be feasible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fb86a6233a0fab5a0ef858b40566c3cc76f1a82 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187)
* b409bcdc3fc8d43d9bd669e0232749d1368a916a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382)
* 0a4f850a137668ceea146a4fa61803cadcdd755c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4f850a137668ceea146a4fa61803cadcdd755c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386)
* d8be6d890df2b2aca793b411313adad98d47ac11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390)
* 463b2c0160ba8892fa654a7689f9a58d516583e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r780005813
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/source/PulsarSourceReaderTestBase.java
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.source.reader.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.pulsar.source.config.SourceConfiguration;
+import org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils;
+import org.apache.flink.connector.pulsar.source.reader.PulsarSourceReaderFactory;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema;
+import org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchemaInitializationContext;
+import org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
+import org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderOutput;
+import org.apache.flink.connectors.test.common.junit.extensions.TestLoggerExtension;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.testutils.CommonTestUtils;
+
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.policies.data.SubscriptionStats;
+import org.apache.pulsar.common.policies.data.TopicStats;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.ParameterResolver;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
+import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_TYPE;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.pulsarSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.createPartitionSplit;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
+abstract class PulsarSourceReaderTestBase extends PulsarTestSuiteBase {
+
+ protected static final int DEFAULT_POLLING_TIMEOUT_SECONDS = 120;
+
+ @RegisterExtension
+ PulsarSourceReaderInvocationContextProvider provider =
+ new PulsarSourceReaderInvocationContextProvider();
+
+ @BeforeEach
+ void beforeEach(String topicName) {
+ Random random = new Random(System.currentTimeMillis());
+ operator().setupTopic(topicName, Schema.INT32, () -> random.nextInt(20));
+ }
+
+ @AfterEach
+ void afterEach(String topicName) {
+ operator().deleteTopic(topicName, true);
+ }
+
+ @TestTemplate
+ void assignZeroSplitsCreatesZeroSubscription(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ reader.snapshotState(100L);
+ reader.notifyCheckpointComplete(100L);
+ // Verify the committed offsets.
+ reader.close();
+ for (int i = 0; i < PulsarRuntimeOperator.DEFAULT_PARTITIONS; i++) {
+ verifyNoSubscriptionCreated(TopicNameUtils.topicNameWithPartition(topicName, i));
+ }
+ }
+
+ @TestTemplate
+ void assigningEmptySplits(
+ PulsarSourceReaderBase<Integer> reader, Boundedness boundedness, String topicName)
+ throws Exception {
+ final PulsarPartitionSplit emptySplit =
+ createPartitionSplit(
+ topicName, 0, Boundedness.CONTINUOUS_UNBOUNDED, MessageId.latest);
+
+ reader.addSplits(Collections.singletonList(emptySplit));
+
+ TestingReaderOutput<Integer> output = new TestingReaderOutput<>();
+ InputStatus status;
+ status = reader.pollNext(output);
+ // We sleep for 1 sec to avoid entering false negative case:
+ // The fetcher could become idle upon entry because :
+ // 1. the AddSplitsTask is dequeued from taskQueue,
+ // 2. The dequeued splits is not assigned to assignedSplits.
+ // And if both assignedSplits is null and taskQueue is empty, the fetcher could become idle,
+ // and return NOTHING_AVAILABLE upon entry. This happened multiple times during local test.
+ sleepUninterruptibly(1, TimeUnit.SECONDS);
+ assertThat(status).isEqualTo(InputStatus.NOTHING_AVAILABLE);
+ reader.close();
+ }
+
+ protected void setupSourceReader(
Review comment:
This sleep can be deleted here as this comment is not relevant. The relevant test case was deleted because that test case is covered in other test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r779541764
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
Review comment:
Many of the methods might be used by the subclaseses. I'll look at these methods to see which can be private or put them in util classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17937:
URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27187",
"triggerID" : "1fb86a6233a0fab5a0ef858b40566c3cc76f1a82",
"triggerType" : "PUSH"
}, {
"hash" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28382",
"triggerID" : "b409bcdc3fc8d43d9bd669e0232749d1368a916a",
"triggerType" : "PUSH"
}, {
"hash" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28386",
"triggerID" : "0a4f850a137668ceea146a4fa61803cadcdd755c",
"triggerType" : "PUSH"
}, {
"hash" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28390",
"triggerID" : "d8be6d890df2b2aca793b411313adad98d47ac11",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"triggerType" : "PUSH"
}, {
"hash" : "463b2c0160ba8892fa654a7689f9a58d516583e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398",
"triggerID" : "998834424",
"triggerType" : "MANUAL"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"triggerType" : "PUSH"
}, {
"hash" : "c1728e5765b33b6ba1140a2f313687eb3bbbaf5f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460",
"triggerID" : "999984059",
"triggerType" : "MANUAL"
}, {
"hash" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073",
"triggerID" : "48c44601bb0c15a3975d8ad41249231f06cbced2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1728e5765b33b6ba1140a2f313687eb3bbbaf5f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460)
* 48c44601bb0c15a3975d8ad41249231f06cbced2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29073)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
Posted by GitBox <gi...@apache.org>.
imaffe commented on a change in pull request #17937:
URL: https://github.com/apache/flink/pull/17937#discussion_r781783619
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderTestBase.java
##########
@@ -39,78 +50,164 @@
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import static java.time.Duration.ofSeconds;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_MAX_FETCH_TIME;
import static org.apache.flink.connector.pulsar.source.PulsarSourceOptions.PULSAR_SUBSCRIPTION_NAME;
-import static org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor.never;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicNameWithPartition;
import static org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange.createFullRange;
+import static org.apache.flink.connector.pulsar.source.reader.deserializer.PulsarDeserializationSchema.flinkSchema;
+import static org.apache.flink.connector.pulsar.testutils.PulsarTestCommonUtils.isAssignableFromParameterContext;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_SOURCE_READER_SUBSCRIPTION_TYPE_STORE_KEY;
+import static org.apache.flink.connector.pulsar.testutils.extension.PulsarSourceOrderlinessExtension.PULSAR_TEST_RESOURCE_NAMESPACE;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.DEFAULT_PARTITIONS;
+import static org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator.NUM_RECORDS_PER_PARTITION;
import static org.apache.flink.core.testutils.CommonTestUtils.waitUtil;
+import static org.apache.flink.shaded.guava30.com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.client.api.Schema.STRING;
-import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.assertj.core.api.Assertions.assertThat;
/** Test utils for split readers. */
+@ExtendWith({
+ PulsarSourceOrderlinessExtension.class,
+ TestLoggerExtension.class,
+})
public abstract class PulsarPartitionSplitReaderTestBase extends PulsarTestSuiteBase {
Review comment:
I used private and moved methods to subclasses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org