You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/08/19 19:20:54 UTC
[GitHub] [flink] syhily opened a new pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
syhily opened a new pull request #16900:
URL: https://github.com/apache/flink/pull/16900
## What is the purpose of the change
The pulsar connector use TestContainers for all the pulsar related tests. We should have a lightweight embedded pulsar backend for performing all the Unit tests.
## Brief change log
- Add mocked `MockPulsarService` and `NonClosableMockBookKeeper`.
- Design a `PulsarRuntimeProvider` for user to choose different pulsar backend.
- Modify the test code for matching the new pulsar test tools.
## Verifying this change
This change is already covered by existing tests, such as *PulsarSourceITCase*.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes)
- We add the `org.apache.pulsar:pulsar-broker` and `org.apache.pulsar:testmocks` in test scope.
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r693123793
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/mock/PulsarMockProvider.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.testutils.runtime.mock;
+
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeProvider;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.client.admin.PulsarAdmin;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.common.policies.data.ClusterData;
+import org.apache.pulsar.common.policies.data.RetentionPolicies;
+import org.apache.pulsar.common.policies.data.TenantInfo;
+
+import java.util.Optional;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.testutils.runtime.mock.PortBindingUtils.findAvailablePort;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** Providing a mocked pulsar server. */
+public class PulsarMockProvider implements PulsarRuntimeProvider {
+
+ private static final String CLUSTER_NAME = "mock-pulsar-" + randomAlphanumeric(6);
+ private final MockPulsarService pulsarService;
+ private PulsarRuntimeOperator operator;
+
+ public PulsarMockProvider() {
+ this(createConfig());
+ }
+
+ public PulsarMockProvider(ServiceConfiguration configuration) {
+ this.pulsarService = new MockPulsarService(configuration);
+ }
+
+ @Override
+ public void startUp() {
+ try {
+ pulsarService.start();
+ } catch (PulsarServerException e) {
+ throw new IllegalStateException(e);
+ }
+ this.operator =
+ new PulsarRuntimeOperator(
+ pulsarService.getBrokerServiceUrl(), pulsarService.getWebServiceAddress());
+
+ // Successfully start a pulsar broker, we have to create the required resources.
+ sneakyAdmin(this::createTestResource);
+ }
+
+ @Override
+ public void tearDown() {
+ try {
+ pulsarService.close();
+ operator.close();
+ this.operator = null;
+ } catch (Exception e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ @Override
+ public PulsarRuntimeOperator operator() {
+ return checkNotNull(operator, "You should start this mock pulsar first.");
+ }
+
+ private void createTestResource() throws PulsarAdminException {
+ PulsarAdmin admin = operator().admin();
+ if (!admin.clusters().getClusters().contains(CLUSTER_NAME)) {
+ // Make clients can test short names
+ ClusterData data =
+ ClusterData.builder()
+ .serviceUrl("http://127.0.0.1:" + pulsarService.getBrokerServicePort())
+ .build();
+ admin.clusters().createCluster(CLUSTER_NAME, data);
+ }
+
+ TenantInfo info =
+ TenantInfo.builder()
+ .adminRoles(ImmutableSet.of("appid1", "appid2"))
+ .allowedClusters(ImmutableSet.of(CLUSTER_NAME))
+ .build();
+ if (!admin.tenants().getTenants().contains("public")) {
+ admin.tenants().createTenant("public", info);
+ } else {
+ admin.tenants().updateTenant("public", info);
+ }
+
+ if (!admin.namespaces().getNamespaces("public").contains("public/default")) {
+ admin.namespaces().createNamespace("public/default");
+ admin.namespaces().setRetention("public/default", new RetentionPolicies(60, 1000));
+ }
+ }
+
+ private static ServiceConfiguration createConfig() {
+ ServiceConfiguration configuration = new ServiceConfiguration();
+
+ configuration.setAdvertisedAddress("localhost");
+ configuration.setClusterName(CLUSTER_NAME);
+
+ configuration.setManagedLedgerCacheSizeMB(8);
+ configuration.setActiveConsumerFailoverDelayTimeMillis(0);
+ configuration.setDefaultRetentionTimeInMinutes(7);
+ configuration.setDefaultNumberOfNamespaceBundles(1);
+ configuration.setZookeeperServers("localhost:2181");
+ configuration.setConfigurationStoreServers("localhost:3181");
+
+ configuration.setAuthenticationEnabled(false);
+ configuration.setAuthorizationEnabled(false);
+ configuration.setAllowAutoTopicCreation(true);
+ configuration.setBrokerDeleteInactiveTopicsEnabled(false);
+
+ configuration.setWebSocketServiceEnabled(false);
+ // Binding Ports
+ configuration.setBrokerServicePort(Optional.of(findAvailablePort()));
+ configuration.setWebServicePort(Optional.of(findAvailablePort()));
Review comment:
Can't we use port 0 here and let the underlying system find open random ports itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r694562472
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/config/PulsarSourceConfigUtils.java
##########
@@ -234,25 +239,87 @@ public static void checkConfigurations(Configuration configuration) {
PulsarClient client, Schema<T> schema, ConsumerConfigurationData<T> config)
throws PulsarClientException {
// ConsumerBuilder don't support using the given ConsumerConfigurationData directly.
- ConsumerBuilder<T> consumerBuilder = client.newConsumer(schema).loadConf(configMap(config));
+ ConsumerBuilder<T> builder = new ConsumerBuilderImpl<>((PulsarClientImpl) client, schema);
- // Set some non-serializable fields.
- if (config.getMessageListener() != null) {
- consumerBuilder.messageListener(config.getMessageListener());
- }
- if (config.getConsumerEventListener() != null) {
- consumerBuilder.consumerEventListener(config.getConsumerEventListener());
- }
- if (config.getCryptoKeyReader() != null) {
- consumerBuilder.cryptoKeyReader(config.getCryptoKeyReader());
- }
- if (config.getMessageCrypto() != null) {
- consumerBuilder.messageCrypto(config.getMessageCrypto());
- }
- if (config.getBatchReceivePolicy() != null) {
- consumerBuilder.batchReceivePolicy(config.getBatchReceivePolicy());
+ // Since pulsar don't expose the config constructor.
+ // We have to set the builder fields one by one.
+ setConfig(config.getTopicNames(), topics -> builder.topics(new ArrayList<>(topics)));
+ setConfig(config.getTopicsPattern(), builder::topicsPattern);
+ setConfig(config.getSubscriptionName(), builder::subscriptionName);
+ setConfig(
+ config.getAckTimeoutMillis(),
+ millis -> builder.ackTimeout(millis, TimeUnit.MILLISECONDS));
+ setConfig(config.isAckReceiptEnabled(), builder::isAckReceiptEnabled);
+ setConfig(
+ config.getTickDurationMillis(),
+ millis -> builder.ackTimeoutTickTime(millis, TimeUnit.MILLISECONDS));
+ setConfig(
+ config.getNegativeAckRedeliveryDelayMicros(),
+ micros -> builder.negativeAckRedeliveryDelay(micros, TimeUnit.MICROSECONDS));
+ setConfig(config.getSubscriptionType(), builder::subscriptionType);
+ setConfig(config.getSubscriptionMode(), builder::subscriptionMode);
+ setConfig(config.getMessageListener(), builder::messageListener);
+ setConfig(config.getConsumerEventListener(), builder::consumerEventListener);
+ setConfig(config.getCryptoKeyReader(), builder::cryptoKeyReader);
+ setConfig(config.getMessageCrypto(), builder::messageCrypto);
+ setConfig(config.getCryptoFailureAction(), builder::cryptoFailureAction);
+ setConfig(config.getReceiverQueueSize(), builder::receiverQueueSize);
+ setConfig(
+ config.getAcknowledgementsGroupTimeMicros(),
+ micros -> builder.acknowledgmentGroupTime(micros, TimeUnit.MICROSECONDS));
+ setConfig(config.getConsumerName(), builder::consumerName);
+ setConfig(config.getPriorityLevel(), builder::priorityLevel);
+ setConfig(config.getMaxPendingChunkedMessage(), builder::maxPendingChunkedMessage);
+ setConfig(
+ config.isAutoAckOldestChunkedMessageOnQueueFull(),
+ builder::autoAckOldestChunkedMessageOnQueueFull);
+ setConfig(config.getProperties(), builder::properties);
+ setConfig(
+ config.getMaxTotalReceiverQueueSizeAcrossPartitions(),
+ builder::maxTotalReceiverQueueSizeAcrossPartitions);
+ setConfig(config.isReadCompacted(), builder::readCompacted);
+ setConfig(config.getPatternAutoDiscoveryPeriod(), builder::patternAutoDiscoveryPeriod);
+ setConfig(config.getPatternAutoDiscoveryPeriod(), builder::patternAutoDiscoveryPeriod);
+ setConfig(config.getSubscriptionInitialPosition(), builder::subscriptionInitialPosition);
+ setConfig(config.getRegexSubscriptionMode(), builder::subscriptionTopicsMode);
+ setConfig(config.isReplicateSubscriptionState(), builder::replicateSubscriptionState);
+ setConfig(config.getDeadLetterPolicy(), builder::deadLetterPolicy);
+ setConfig(config.isAutoUpdatePartitions(), builder::autoUpdatePartitions);
+ setConfig(
+ config.getAutoUpdatePartitionsIntervalSeconds(),
+ seconds ->
+ builder.autoUpdatePartitionsInterval(
+ Math.toIntExact(seconds), TimeUnit.SECONDS));
+ if (config.isResetIncludeHead()) {
+ builder.startMessageIdInclusive();
}
+ setConfig(config.getBatchReceivePolicy(), builder::batchReceivePolicy);
+ setConfig(config.getKeySharedPolicy(), builder::keySharedPolicy);
+ setConfig(config.isRetryEnable(), builder::enableRetry);
+ setConfig(config.isBatchIndexAckEnabled(), builder::enableBatchIndexAcknowledgment);
+ setConfig(
+ config.getExpireTimeOfIncompleteChunkedMessageMillis(),
+ millis ->
+ builder.expireTimeOfIncompleteChunkedMessage(
+ millis, TimeUnit.MILLISECONDS));
+ setConfig(config.isPoolMessages(), builder::poolMessages);
- return consumerBuilder.subscribe();
+ return builder.subscribe();
+ }
+
+ private static <T> void setConfig(T value, java.util.function.Consumer<T> consumer) {
+ if (value != null) {
+ if (value instanceof Collection) {
+ if (!((Collection<?>) value).isEmpty()) {
+ consumer.accept(value);
+ }
+ } else if (value instanceof Map) {
+ if (!((Map<?, ?>) value).isEmpty()) {
+ consumer.accept(value);
+ }
+ } else {
+ consumer.accept(value);
+ }
Review comment:
I would be fine with a separate commit in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-905865179
> LGTM % suggestion. For future contribution, we need to strive for smaller self-contained commits. In this case, I can still see it as an amendment of the first initial big commit.
>
> So we could have 1 commit that just deals with removing `ClientConfigurationData`. Then 1 commit improves documentation. 1 commit for improved configuration validation etc.
>
> That makes reviewing much easier and helps future maintainers (including you) to understand why certain changes happened in the past.
Thanks for this detailed instruction. I have split the second big commit into small commits. This makes me think twice before pushing the code. Why I need this modification? Is it related with current issue? I still have a long way to go on how to contribute code in the open source community. But I would try my best to learn these knowledge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93c302d1b404337b8a5e509ff0b740c9c7d3e0a7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773)
* 60a73ab57a72105c906ac533cabe256e04bd821d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r694016919
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/config/PulsarConfigUtils.java
##########
@@ -217,26 +217,7 @@ public static PulsarClient createClient(
* except the authentication.
*/
public static PulsarClient createClient(ClientConfigurationData config) {
- // PulsarClientBuilder don't support using the given ClientConfigurationData directly.
- Map<String, Object> configMap = configMap(config);
-
- // These two auth config would be serialized to a secret information by default.
- configMap.put("authParamMap", config.getAuthParamMap());
- configMap.put("authParams", config.getAuthParams());
-
- ClientBuilder clientBuilder = PulsarClient.builder().loadConf(configMap);
-
- // Set some non-serializable fields.
- if (config.getAuthentication() != null) {
- clientBuilder.authentication(config.getAuthentication());
- }
- if (config.getClock() != null) {
- clientBuilder.clock(config.getClock());
- }
- if (config.getServiceUrlProvider() != null) {
- clientBuilder.serviceUrlProvider(config.getServiceUrlProvider());
- }
-
+ ClientBuilder clientBuilder = new ClientBuilderImpl(config);
Review comment:
I'm not a big fan of using internal API. Should we try to get some of your changes into Pulsar upstream?
I could imagine that such a Test client would be beneficial also for other Pulsar users.
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/config/PulsarSourceConfigUtils.java
##########
@@ -234,25 +239,87 @@ public static void checkConfigurations(Configuration configuration) {
PulsarClient client, Schema<T> schema, ConsumerConfigurationData<T> config)
throws PulsarClientException {
// ConsumerBuilder don't support using the given ConsumerConfigurationData directly.
- ConsumerBuilder<T> consumerBuilder = client.newConsumer(schema).loadConf(configMap(config));
+ ConsumerBuilder<T> builder = new ConsumerBuilderImpl<>((PulsarClientImpl) client, schema);
- // Set some non-serializable fields.
- if (config.getMessageListener() != null) {
- consumerBuilder.messageListener(config.getMessageListener());
- }
- if (config.getConsumerEventListener() != null) {
- consumerBuilder.consumerEventListener(config.getConsumerEventListener());
- }
- if (config.getCryptoKeyReader() != null) {
- consumerBuilder.cryptoKeyReader(config.getCryptoKeyReader());
- }
- if (config.getMessageCrypto() != null) {
- consumerBuilder.messageCrypto(config.getMessageCrypto());
- }
- if (config.getBatchReceivePolicy() != null) {
- consumerBuilder.batchReceivePolicy(config.getBatchReceivePolicy());
+ // Since pulsar don't expose the config constructor.
+ // We have to set the builder fields one by one.
+ setConfig(config.getTopicNames(), topics -> builder.topics(new ArrayList<>(topics)));
+ setConfig(config.getTopicsPattern(), builder::topicsPattern);
+ setConfig(config.getSubscriptionName(), builder::subscriptionName);
+ setConfig(
+ config.getAckTimeoutMillis(),
+ millis -> builder.ackTimeout(millis, TimeUnit.MILLISECONDS));
+ setConfig(config.isAckReceiptEnabled(), builder::isAckReceiptEnabled);
+ setConfig(
+ config.getTickDurationMillis(),
+ millis -> builder.ackTimeoutTickTime(millis, TimeUnit.MILLISECONDS));
+ setConfig(
+ config.getNegativeAckRedeliveryDelayMicros(),
+ micros -> builder.negativeAckRedeliveryDelay(micros, TimeUnit.MICROSECONDS));
+ setConfig(config.getSubscriptionType(), builder::subscriptionType);
+ setConfig(config.getSubscriptionMode(), builder::subscriptionMode);
+ setConfig(config.getMessageListener(), builder::messageListener);
+ setConfig(config.getConsumerEventListener(), builder::consumerEventListener);
+ setConfig(config.getCryptoKeyReader(), builder::cryptoKeyReader);
+ setConfig(config.getMessageCrypto(), builder::messageCrypto);
+ setConfig(config.getCryptoFailureAction(), builder::cryptoFailureAction);
+ setConfig(config.getReceiverQueueSize(), builder::receiverQueueSize);
+ setConfig(
+ config.getAcknowledgementsGroupTimeMicros(),
+ micros -> builder.acknowledgmentGroupTime(micros, TimeUnit.MICROSECONDS));
+ setConfig(config.getConsumerName(), builder::consumerName);
+ setConfig(config.getPriorityLevel(), builder::priorityLevel);
+ setConfig(config.getMaxPendingChunkedMessage(), builder::maxPendingChunkedMessage);
+ setConfig(
+ config.isAutoAckOldestChunkedMessageOnQueueFull(),
+ builder::autoAckOldestChunkedMessageOnQueueFull);
+ setConfig(config.getProperties(), builder::properties);
+ setConfig(
+ config.getMaxTotalReceiverQueueSizeAcrossPartitions(),
+ builder::maxTotalReceiverQueueSizeAcrossPartitions);
+ setConfig(config.isReadCompacted(), builder::readCompacted);
+ setConfig(config.getPatternAutoDiscoveryPeriod(), builder::patternAutoDiscoveryPeriod);
+ setConfig(config.getPatternAutoDiscoveryPeriod(), builder::patternAutoDiscoveryPeriod);
+ setConfig(config.getSubscriptionInitialPosition(), builder::subscriptionInitialPosition);
+ setConfig(config.getRegexSubscriptionMode(), builder::subscriptionTopicsMode);
+ setConfig(config.isReplicateSubscriptionState(), builder::replicateSubscriptionState);
+ setConfig(config.getDeadLetterPolicy(), builder::deadLetterPolicy);
+ setConfig(config.isAutoUpdatePartitions(), builder::autoUpdatePartitions);
+ setConfig(
+ config.getAutoUpdatePartitionsIntervalSeconds(),
+ seconds ->
+ builder.autoUpdatePartitionsInterval(
+ Math.toIntExact(seconds), TimeUnit.SECONDS));
+ if (config.isResetIncludeHead()) {
+ builder.startMessageIdInclusive();
}
+ setConfig(config.getBatchReceivePolicy(), builder::batchReceivePolicy);
+ setConfig(config.getKeySharedPolicy(), builder::keySharedPolicy);
+ setConfig(config.isRetryEnable(), builder::enableRetry);
+ setConfig(config.isBatchIndexAckEnabled(), builder::enableBatchIndexAcknowledgment);
+ setConfig(
+ config.getExpireTimeOfIncompleteChunkedMessageMillis(),
+ millis ->
+ builder.expireTimeOfIncompleteChunkedMessage(
+ millis, TimeUnit.MILLISECONDS));
+ setConfig(config.isPoolMessages(), builder::poolMessages);
- return consumerBuilder.subscribe();
+ return builder.subscribe();
+ }
+
+ private static <T> void setConfig(T value, java.util.function.Consumer<T> consumer) {
+ if (value != null) {
+ if (value instanceof Collection) {
+ if (!((Collection<?>) value).isEmpty()) {
+ consumer.accept(value);
+ }
+ } else if (value instanceof Map) {
+ if (!((Map<?, ?>) value).isEmpty()) {
+ consumer.accept(value);
+ }
+ } else {
+ consumer.accept(value);
+ }
Review comment:
I don't understand why this change is needed at all.
Why isn't `ConsumerBuilder<T> consumerBuilder = client.newConsumer(schema).loadConf(configMap(config));` working anymore?
Again would it be easier to remove `ConsumerConfigurationData` and just use `Configuration`?
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/config/PulsarConfigUtils.java
##########
@@ -217,26 +217,7 @@ public static PulsarClient createClient(
* except the authentication.
*/
public static PulsarClient createClient(ClientConfigurationData config) {
- // PulsarClientBuilder don't support using the given ClientConfigurationData directly.
- Map<String, Object> configMap = configMap(config);
-
- // These two auth config would be serialized to a secret information by default.
- configMap.put("authParamMap", config.getAuthParamMap());
- configMap.put("authParams", config.getAuthParams());
-
- ClientBuilder clientBuilder = PulsarClient.builder().loadConf(configMap);
-
- // Set some non-serializable fields.
- if (config.getAuthentication() != null) {
- clientBuilder.authentication(config.getAuthentication());
- }
- if (config.getClock() != null) {
- clientBuilder.clock(config.getClock());
- }
- if (config.getServiceUrlProvider() != null) {
- clientBuilder.serviceUrlProvider(config.getServiceUrlProvider());
- }
-
+ ClientBuilder clientBuilder = new ClientBuilderImpl(config);
Review comment:
Is this just a way to get rid of the `configMap` method? If so, wouldn't it make sense to rather remove `ClientConfigurationData` from user-facing and just go with `Configuration` all the way?
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/PulsarTestEnvironment.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.testutils;
+
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntime;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeProvider;
+import org.apache.flink.connectors.test.common.TestResource;
+import org.apache.flink.connectors.test.common.junit.annotations.ExternalSystem;
+
+import org.junit.jupiter.api.extension.AfterAllCallback;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runners.model.MultipleFailureException;
+import org.junit.runners.model.Statement;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * A JUnit 5 {@link Extension} for supporting running a pulsar instance before executing tests. This
+ * class is also a {@link ExternalSystem} for {@code flink-connector-testing} tools.
+ *
+ * <p>Some old flink tests are based on JUint 4, this class is also support it. The follow code
+ * snippet shows how to use this class in JUnit 4.
Review comment:
Did you see anywhere the need for JUnit4? We actually want to forbid writing new tests with JUnit4 soonish, so I don't think this hybrid support is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r695687321
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/config/PulsarOptions.java
##########
@@ -230,89 +296,172 @@ private PulsarOptions() {
Description.builder()
.text(
"Duration of waiting for a connection to a broker to be established.")
+ .text("It uses milliseconds as the unit of time.")
Review comment:
```suggestion
.text(
"Duration (in ms) of waiting for a connection to a broker to be established.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily removed a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily removed a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-906027971
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93c302d1b404337b8a5e509ff0b740c9c7d3e0a7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773)
* 60a73ab57a72105c906ac533cabe256e04bd821d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r693138134
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/mock/PulsarMockProvider.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.testutils.runtime.mock;
+
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeProvider;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.client.admin.PulsarAdmin;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.common.policies.data.ClusterData;
+import org.apache.pulsar.common.policies.data.RetentionPolicies;
+import org.apache.pulsar.common.policies.data.TenantInfo;
+
+import java.util.Optional;
+
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
+import static org.apache.flink.connector.pulsar.testutils.runtime.mock.PortBindingUtils.findAvailablePort;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** Providing a mocked pulsar server. */
+public class PulsarMockProvider implements PulsarRuntimeProvider {
+
+ private static final String CLUSTER_NAME = "mock-pulsar-" + randomAlphanumeric(6);
+ private final MockPulsarService pulsarService;
+ private PulsarRuntimeOperator operator;
+
+ public PulsarMockProvider() {
+ this(createConfig());
+ }
+
+ public PulsarMockProvider(ServiceConfiguration configuration) {
+ this.pulsarService = new MockPulsarService(configuration);
+ }
+
+ @Override
+ public void startUp() {
+ try {
+ pulsarService.start();
+ } catch (PulsarServerException e) {
+ throw new IllegalStateException(e);
+ }
+ this.operator =
+ new PulsarRuntimeOperator(
+ pulsarService.getBrokerServiceUrl(), pulsarService.getWebServiceAddress());
+
+ // Successfully start a pulsar broker, we have to create the required resources.
+ sneakyAdmin(this::createTestResource);
+ }
+
+ @Override
+ public void tearDown() {
+ try {
+ pulsarService.close();
+ operator.close();
+ this.operator = null;
+ } catch (Exception e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ @Override
+ public PulsarRuntimeOperator operator() {
+ return checkNotNull(operator, "You should start this mock pulsar first.");
+ }
+
+ private void createTestResource() throws PulsarAdminException {
+ PulsarAdmin admin = operator().admin();
+ if (!admin.clusters().getClusters().contains(CLUSTER_NAME)) {
+ // Make clients can test short names
+ ClusterData data =
+ ClusterData.builder()
+ .serviceUrl("http://127.0.0.1:" + pulsarService.getBrokerServicePort())
+ .build();
+ admin.clusters().createCluster(CLUSTER_NAME, data);
+ }
+
+ TenantInfo info =
+ TenantInfo.builder()
+ .adminRoles(ImmutableSet.of("appid1", "appid2"))
+ .allowedClusters(ImmutableSet.of(CLUSTER_NAME))
+ .build();
+ if (!admin.tenants().getTenants().contains("public")) {
+ admin.tenants().createTenant("public", info);
+ } else {
+ admin.tenants().updateTenant("public", info);
+ }
+
+ if (!admin.namespaces().getNamespaces("public").contains("public/default")) {
+ admin.namespaces().createNamespace("public/default");
+ admin.namespaces().setRetention("public/default", new RetentionPolicies(60, 1000));
+ }
+ }
+
+ private static ServiceConfiguration createConfig() {
+ ServiceConfiguration configuration = new ServiceConfiguration();
+
+ configuration.setAdvertisedAddress("localhost");
+ configuration.setClusterName(CLUSTER_NAME);
+
+ configuration.setManagedLedgerCacheSizeMB(8);
+ configuration.setActiveConsumerFailoverDelayTimeMillis(0);
+ configuration.setDefaultRetentionTimeInMinutes(7);
+ configuration.setDefaultNumberOfNamespaceBundles(1);
+ configuration.setZookeeperServers("localhost:2181");
+ configuration.setConfigurationStoreServers("localhost:3181");
+
+ configuration.setAuthenticationEnabled(false);
+ configuration.setAuthorizationEnabled(false);
+ configuration.setAllowAutoTopicCreation(true);
+ configuration.setBrokerDeleteInactiveTopicsEnabled(false);
+
+ configuration.setWebSocketServiceEnabled(false);
+ // Binding Ports
+ configuration.setBrokerServicePort(Optional.of(findAvailablePort()));
+ configuration.setWebServicePort(Optional.of(findAvailablePort()));
Review comment:
No, this config option is `Optional`, set it to empty would disable this endpoint. Otherwise, you have to give a available port.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* db84a292597428f82df1688e87787a6cc2591cfd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22876",
"triggerID" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"triggerType" : "PUSH"
}, {
"hash" : "958dbafc8a14a0f46e54db04fd9cbd0f21f5d57f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "958dbafc8a14a0f46e54db04fd9cbd0f21f5d57f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3180e650ed121eb4706dffbd4584d2a08189e24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22876)
* 958dbafc8a14a0f46e54db04fd9cbd0f21f5d57f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93ea63d1795facc130ecb8c08a6a07562cfc22d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-906144704
[FAIL] 'Run kubernetes pyflink application test' failed after 4 minutes and 21 seconds! Test exited with exit code 1
I don't know how to deal with this test failure. https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22876&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=070ff179-953e-5bda-71fa-d6599415701c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22876",
"triggerID" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60a73ab57a72105c906ac533cabe256e04bd821d Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860)
* f3180e650ed121eb4706dffbd4584d2a08189e24 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22876)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93ea63d1795facc130ecb8c08a6a07562cfc22d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530)
* 91e338034e41b317aaa4427332faac129a338701 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r694109706
##########
File path: flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/PulsarTestEnvironment.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.pulsar.testutils;
+
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntime;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
+import org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeProvider;
+import org.apache.flink.connectors.test.common.TestResource;
+import org.apache.flink.connectors.test.common.junit.annotations.ExternalSystem;
+
+import org.junit.jupiter.api.extension.AfterAllCallback;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.Extension;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runners.model.MultipleFailureException;
+import org.junit.runners.model.Statement;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * A JUnit 5 {@link Extension} for supporting running a pulsar instance before executing tests. This
+ * class is also a {@link ExternalSystem} for {@code flink-connector-testing} tools.
+ *
+ * <p>Some old flink tests are based on JUint 4, this class is also support it. The follow code
+ * snippet shows how to use this class in JUnit 4.
Review comment:
`flink-connector-test-utils` provide the basic source reader test, it's based on JUnit 4. I have to hold this support until everything is based on JUnit 5 in flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-906194276
These typo have been fixed. I'm writing document on top of this PR. I hope it can be merged soon or later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91e338034e41b317aaa4427332faac129a338701 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691)
* 93c302d1b404337b8a5e509ff0b740c9c7d3e0a7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r694060240
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/config/PulsarSourceConfigUtils.java
##########
@@ -234,25 +239,87 @@ public static void checkConfigurations(Configuration configuration) {
PulsarClient client, Schema<T> schema, ConsumerConfigurationData<T> config)
throws PulsarClientException {
// ConsumerBuilder don't support using the given ConsumerConfigurationData directly.
- ConsumerBuilder<T> consumerBuilder = client.newConsumer(schema).loadConf(configMap(config));
+ ConsumerBuilder<T> builder = new ConsumerBuilderImpl<>((PulsarClientImpl) client, schema);
- // Set some non-serializable fields.
- if (config.getMessageListener() != null) {
- consumerBuilder.messageListener(config.getMessageListener());
- }
- if (config.getConsumerEventListener() != null) {
- consumerBuilder.consumerEventListener(config.getConsumerEventListener());
- }
- if (config.getCryptoKeyReader() != null) {
- consumerBuilder.cryptoKeyReader(config.getCryptoKeyReader());
- }
- if (config.getMessageCrypto() != null) {
- consumerBuilder.messageCrypto(config.getMessageCrypto());
- }
- if (config.getBatchReceivePolicy() != null) {
- consumerBuilder.batchReceivePolicy(config.getBatchReceivePolicy());
+ // Since pulsar don't expose the config constructor.
+ // We have to set the builder fields one by one.
+ setConfig(config.getTopicNames(), topics -> builder.topics(new ArrayList<>(topics)));
+ setConfig(config.getTopicsPattern(), builder::topicsPattern);
+ setConfig(config.getSubscriptionName(), builder::subscriptionName);
+ setConfig(
+ config.getAckTimeoutMillis(),
+ millis -> builder.ackTimeout(millis, TimeUnit.MILLISECONDS));
+ setConfig(config.isAckReceiptEnabled(), builder::isAckReceiptEnabled);
+ setConfig(
+ config.getTickDurationMillis(),
+ millis -> builder.ackTimeoutTickTime(millis, TimeUnit.MILLISECONDS));
+ setConfig(
+ config.getNegativeAckRedeliveryDelayMicros(),
+ micros -> builder.negativeAckRedeliveryDelay(micros, TimeUnit.MICROSECONDS));
+ setConfig(config.getSubscriptionType(), builder::subscriptionType);
+ setConfig(config.getSubscriptionMode(), builder::subscriptionMode);
+ setConfig(config.getMessageListener(), builder::messageListener);
+ setConfig(config.getConsumerEventListener(), builder::consumerEventListener);
+ setConfig(config.getCryptoKeyReader(), builder::cryptoKeyReader);
+ setConfig(config.getMessageCrypto(), builder::messageCrypto);
+ setConfig(config.getCryptoFailureAction(), builder::cryptoFailureAction);
+ setConfig(config.getReceiverQueueSize(), builder::receiverQueueSize);
+ setConfig(
+ config.getAcknowledgementsGroupTimeMicros(),
+ micros -> builder.acknowledgmentGroupTime(micros, TimeUnit.MICROSECONDS));
+ setConfig(config.getConsumerName(), builder::consumerName);
+ setConfig(config.getPriorityLevel(), builder::priorityLevel);
+ setConfig(config.getMaxPendingChunkedMessage(), builder::maxPendingChunkedMessage);
+ setConfig(
+ config.isAutoAckOldestChunkedMessageOnQueueFull(),
+ builder::autoAckOldestChunkedMessageOnQueueFull);
+ setConfig(config.getProperties(), builder::properties);
+ setConfig(
+ config.getMaxTotalReceiverQueueSizeAcrossPartitions(),
+ builder::maxTotalReceiverQueueSizeAcrossPartitions);
+ setConfig(config.isReadCompacted(), builder::readCompacted);
+ setConfig(config.getPatternAutoDiscoveryPeriod(), builder::patternAutoDiscoveryPeriod);
+ setConfig(config.getPatternAutoDiscoveryPeriod(), builder::patternAutoDiscoveryPeriod);
+ setConfig(config.getSubscriptionInitialPosition(), builder::subscriptionInitialPosition);
+ setConfig(config.getRegexSubscriptionMode(), builder::subscriptionTopicsMode);
+ setConfig(config.isReplicateSubscriptionState(), builder::replicateSubscriptionState);
+ setConfig(config.getDeadLetterPolicy(), builder::deadLetterPolicy);
+ setConfig(config.isAutoUpdatePartitions(), builder::autoUpdatePartitions);
+ setConfig(
+ config.getAutoUpdatePartitionsIntervalSeconds(),
+ seconds ->
+ builder.autoUpdatePartitionsInterval(
+ Math.toIntExact(seconds), TimeUnit.SECONDS));
+ if (config.isResetIncludeHead()) {
+ builder.startMessageIdInclusive();
}
+ setConfig(config.getBatchReceivePolicy(), builder::batchReceivePolicy);
+ setConfig(config.getKeySharedPolicy(), builder::keySharedPolicy);
+ setConfig(config.isRetryEnable(), builder::enableRetry);
+ setConfig(config.isBatchIndexAckEnabled(), builder::enableBatchIndexAcknowledgment);
+ setConfig(
+ config.getExpireTimeOfIncompleteChunkedMessageMillis(),
+ millis ->
+ builder.expireTimeOfIncompleteChunkedMessage(
+ millis, TimeUnit.MILLISECONDS));
+ setConfig(config.isPoolMessages(), builder::poolMessages);
- return consumerBuilder.subscribe();
+ return builder.subscribe();
+ }
+
+ private static <T> void setConfig(T value, java.util.function.Consumer<T> consumer) {
+ if (value != null) {
+ if (value instanceof Collection) {
+ if (!((Collection<?>) value).isEmpty()) {
+ consumer.accept(value);
+ }
+ } else if (value instanceof Map) {
+ if (!((Map<?, ?>) value).isEmpty()) {
+ consumer.accept(value);
+ }
+ } else {
+ consumer.accept(value);
+ }
Review comment:
> We still have the opportunity to change API until the final release. Would it help to get rid of `ClientConfigurationData`? It feels like there is quite a bit friction involved. And since you replicated all options to `ConfigOption`s, it would even be better to go `Configuration` all the way for Flink users.
Drop the `ClientConfigurationData` and `ConsumerConfigurationData` would be cool. I think we could just use the `Configuration` all the way. Do we need a separate PR for this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 60a73ab57a72105c906ac533cabe256e04bd821d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902874170
> Before doing a full-fledged review, I'd like to understand the scope a bit better since the PR is rather big as well.
> Do you want to contribute the test utilities to improve
>
> * Test performance (avoiding unnecessary container spawns)
> * Improve test stability (why would it improve stability?)
> * Add more control over the system under test? (for example, to test more edge cases/failures)
>
> Are parts of these sources already present in the Pulsar repo? (= have they already matured?)
I would like to answer these questions. Use an embedded pulsar is almost the same as using a container pulsar for tests. Therefore it can significantly improve the performance of the tests, avoid unnecessary container spawns.
Some pulsar tests failure in Flink Azure CI is caused by container hanging. I think this embedded server may improve the test stability.
Using a mocked pulsar broker can be operated by test. We truly add more control ability.
These codes don't existed in Pulsar repo, the mock test on Pulsar side have to use powermock and manually config the pulsar broker. I have used the common pulsar `testmocks` module for mocking bookie and zookeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902630512
@AHeise Can you review this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902185763
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 3e61d9e95580e703ce5ad6812677c58e42309ef8 (Thu Aug 19 19:34:38 UTC 2021)
**Warnings:**
* **1 pom.xml files were touched**: Check for build and licensing issues.
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-23877).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93ea63d1795facc130ecb8c08a6a07562cfc22d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530)
* 91e338034e41b317aaa4427332faac129a338701 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-906027971
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db84a292597428f82df1688e87787a6cc2591cfd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r696344008
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/PulsarSourceOptions.java
##########
@@ -429,21 +513,28 @@ private PulsarSourceOptions() {
public static final ConfigOption<Integer> PULSAR_MAX_REDELIVER_COUNT =
ConfigOptions.key(CONSUMER_CONFIG_PREFIX + "deadLetterPolicy.maxRedeliverCount")
.intType()
- .defaultValue(0);
+ .defaultValue(0)
+ .withDescription(
+ "Maximum number of times that a message will be redelivered before being sent to the dead letter queue.");
public static final ConfigOption<String> PULSAR_RETRY_LETTER_TOPIC =
ConfigOptions.key(CONSUMER_CONFIG_PREFIX + "deadLetterPolicy.retryLetterTopic")
.stringType()
- .noDefaultValue();
+ .noDefaultValue()
+ .withDescription(
+ "Name of the retry topic where the failing messages will be sent.");
public static final ConfigOption<String> PULSAR_DEAD_LETTER_TOPIC =
ConfigOptions.key(CONSUMER_CONFIG_PREFIX + "deadLetterPolicy.deadLetterTopic")
.stringType()
- .noDefaultValue();
+ .noDefaultValue()
+ .withDescription(
+ "Name of the dead topic where the failing messages will be sent.");
public static final ConfigOption<Boolean> PULSAR_RETRY_ENABLE =
ConfigOptions.key(CONSUMER_CONFIG_PREFIX + "retryEnable")
.booleanType()
- .defaultValue(false);
+ .defaultValue(false)
+ .withDescription("If enabled, the consumer will auto retry message.");
Review comment:
```suggestion
.withDescription("If enabled, the consumer will auto retry messages.");
```
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/PulsarSourceBuilder.java
##########
@@ -360,64 +359,52 @@
/**
* Set an arbitrary property for the PulsarSource and PulsarConsumer. The valid keys can be
- * found in {@link PulsarSourceOptions}.
+ * found in {@link PulsarSourceOptions} and {@link PulsarOptions}.
*
- * @param key the key of the property.
- * @param value the value of the property.
- * @return this PulsarSourceBuilder.
- */
- public PulsarSourceBuilder<OUT> setProperty(String key, String value) {
- checkNotNull(key);
- checkNotNull(value);
- if (configuration.containsKey(key)) {
- ConfigOption<String> rawOption = ConfigOptions.key(key).stringType().noDefaultValue();
- LOG.warn(
- "Config option {} already has a value {}, override to new value {}",
- key,
- configuration.getString(rawOption),
- value);
- }
- configuration.setString(key, value);
- return this;
- }
-
- /**
- * Set an arbitrary property for the PulsarSource and PulsarConsumer. The valid keys can be
- * found in {@link PulsarSourceOptions}.
+ * <p>Make sure the option could be set only once or with same value.
*
* @param key the key of the property.
* @param value the value of the property.
* @return this PulsarSourceBuilder.
*/
- public <T> PulsarSourceBuilder<OUT> setProperty(ConfigOption<T> key, T value) {
+ public <T> PulsarSourceBuilder<OUT> setConfig(ConfigOption<T> key, T value) {
checkNotNull(key);
checkNotNull(value);
if (configuration.contains(key)) {
T oldValue = configuration.get(key);
- LOG.warn(
- "Config option {} already has a value {}, override to new value {}",
- key,
- oldValue,
- value);
+ checkArgument(
+ Objects.equals(oldValue, value),
+ "This option %s has been set to value %s.",
Review comment:
```suggestion
"This option %s has been already set to value %s.",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r694833397
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/config/PulsarSourceConfigUtils.java
##########
@@ -234,25 +239,87 @@ public static void checkConfigurations(Configuration configuration) {
PulsarClient client, Schema<T> schema, ConsumerConfigurationData<T> config)
throws PulsarClientException {
// ConsumerBuilder don't support using the given ConsumerConfigurationData directly.
- ConsumerBuilder<T> consumerBuilder = client.newConsumer(schema).loadConf(configMap(config));
+ ConsumerBuilder<T> builder = new ConsumerBuilderImpl<>((PulsarClientImpl) client, schema);
- // Set some non-serializable fields.
- if (config.getMessageListener() != null) {
- consumerBuilder.messageListener(config.getMessageListener());
- }
- if (config.getConsumerEventListener() != null) {
- consumerBuilder.consumerEventListener(config.getConsumerEventListener());
- }
- if (config.getCryptoKeyReader() != null) {
- consumerBuilder.cryptoKeyReader(config.getCryptoKeyReader());
- }
- if (config.getMessageCrypto() != null) {
- consumerBuilder.messageCrypto(config.getMessageCrypto());
- }
- if (config.getBatchReceivePolicy() != null) {
- consumerBuilder.batchReceivePolicy(config.getBatchReceivePolicy());
+ // Since pulsar don't expose the config constructor.
+ // We have to set the builder fields one by one.
+ setConfig(config.getTopicNames(), topics -> builder.topics(new ArrayList<>(topics)));
+ setConfig(config.getTopicsPattern(), builder::topicsPattern);
+ setConfig(config.getSubscriptionName(), builder::subscriptionName);
+ setConfig(
+ config.getAckTimeoutMillis(),
+ millis -> builder.ackTimeout(millis, TimeUnit.MILLISECONDS));
+ setConfig(config.isAckReceiptEnabled(), builder::isAckReceiptEnabled);
+ setConfig(
+ config.getTickDurationMillis(),
+ millis -> builder.ackTimeoutTickTime(millis, TimeUnit.MILLISECONDS));
+ setConfig(
+ config.getNegativeAckRedeliveryDelayMicros(),
+ micros -> builder.negativeAckRedeliveryDelay(micros, TimeUnit.MICROSECONDS));
+ setConfig(config.getSubscriptionType(), builder::subscriptionType);
+ setConfig(config.getSubscriptionMode(), builder::subscriptionMode);
+ setConfig(config.getMessageListener(), builder::messageListener);
+ setConfig(config.getConsumerEventListener(), builder::consumerEventListener);
+ setConfig(config.getCryptoKeyReader(), builder::cryptoKeyReader);
+ setConfig(config.getMessageCrypto(), builder::messageCrypto);
+ setConfig(config.getCryptoFailureAction(), builder::cryptoFailureAction);
+ setConfig(config.getReceiverQueueSize(), builder::receiverQueueSize);
+ setConfig(
+ config.getAcknowledgementsGroupTimeMicros(),
+ micros -> builder.acknowledgmentGroupTime(micros, TimeUnit.MICROSECONDS));
+ setConfig(config.getConsumerName(), builder::consumerName);
+ setConfig(config.getPriorityLevel(), builder::priorityLevel);
+ setConfig(config.getMaxPendingChunkedMessage(), builder::maxPendingChunkedMessage);
+ setConfig(
+ config.isAutoAckOldestChunkedMessageOnQueueFull(),
+ builder::autoAckOldestChunkedMessageOnQueueFull);
+ setConfig(config.getProperties(), builder::properties);
+ setConfig(
+ config.getMaxTotalReceiverQueueSizeAcrossPartitions(),
+ builder::maxTotalReceiverQueueSizeAcrossPartitions);
+ setConfig(config.isReadCompacted(), builder::readCompacted);
+ setConfig(config.getPatternAutoDiscoveryPeriod(), builder::patternAutoDiscoveryPeriod);
+ setConfig(config.getPatternAutoDiscoveryPeriod(), builder::patternAutoDiscoveryPeriod);
+ setConfig(config.getSubscriptionInitialPosition(), builder::subscriptionInitialPosition);
+ setConfig(config.getRegexSubscriptionMode(), builder::subscriptionTopicsMode);
+ setConfig(config.isReplicateSubscriptionState(), builder::replicateSubscriptionState);
+ setConfig(config.getDeadLetterPolicy(), builder::deadLetterPolicy);
+ setConfig(config.isAutoUpdatePartitions(), builder::autoUpdatePartitions);
+ setConfig(
+ config.getAutoUpdatePartitionsIntervalSeconds(),
+ seconds ->
+ builder.autoUpdatePartitionsInterval(
+ Math.toIntExact(seconds), TimeUnit.SECONDS));
+ if (config.isResetIncludeHead()) {
+ builder.startMessageIdInclusive();
}
+ setConfig(config.getBatchReceivePolicy(), builder::batchReceivePolicy);
+ setConfig(config.getKeySharedPolicy(), builder::keySharedPolicy);
+ setConfig(config.isRetryEnable(), builder::enableRetry);
+ setConfig(config.isBatchIndexAckEnabled(), builder::enableBatchIndexAcknowledgment);
+ setConfig(
+ config.getExpireTimeOfIncompleteChunkedMessageMillis(),
+ millis ->
+ builder.expireTimeOfIncompleteChunkedMessage(
+ millis, TimeUnit.MILLISECONDS));
+ setConfig(config.isPoolMessages(), builder::poolMessages);
- return consumerBuilder.subscribe();
+ return builder.subscribe();
+ }
+
+ private static <T> void setConfig(T value, java.util.function.Consumer<T> consumer) {
+ if (value != null) {
+ if (value instanceof Collection) {
+ if (!((Collection<?>) value).isEmpty()) {
+ consumer.accept(value);
+ }
+ } else if (value instanceof Map) {
+ if (!((Map<?, ?>) value).isEmpty()) {
+ consumer.accept(value);
+ }
+ } else {
+ consumer.accept(value);
+ }
Review comment:
Seems like we have to combine this mock server with config refactor in one PR. The tests on mock server need the removal of PulsarJsonUtils. I would divide them in two commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93c302d1b404337b8a5e509ff0b740c9c7d3e0a7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60a73ab57a72105c906ac533cabe256e04bd821d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860)
* f3180e650ed121eb4706dffbd4584d2a08189e24 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91e338034e41b317aaa4427332faac129a338701 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691)
* 93c302d1b404337b8a5e509ff0b740c9c7d3e0a7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902382011
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* db84a292597428f82df1688e87787a6cc2591cfd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db84a292597428f82df1688e87787a6cc2591cfd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16900:
URL: https://github.com/apache/flink/pull/16900#discussion_r696341613
##########
File path: flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/PulsarSourceBuilder.java
##########
@@ -360,64 +359,52 @@
/**
* Set an arbitrary property for the PulsarSource and PulsarConsumer. The valid keys can be
- * found in {@link PulsarSourceOptions}.
+ * found in {@link PulsarSourceOptions} and {@link PulsarOptions}.
*
- * @param key the key of the property.
- * @param value the value of the property.
- * @return this PulsarSourceBuilder.
- */
- public PulsarSourceBuilder<OUT> setProperty(String key, String value) {
- checkNotNull(key);
- checkNotNull(value);
- if (configuration.containsKey(key)) {
- ConfigOption<String> rawOption = ConfigOptions.key(key).stringType().noDefaultValue();
- LOG.warn(
- "Config option {} already has a value {}, override to new value {}",
- key,
- configuration.getString(rawOption),
- value);
- }
- configuration.setString(key, value);
- return this;
- }
-
- /**
- * Set an arbitrary property for the PulsarSource and PulsarConsumer. The valid keys can be
- * found in {@link PulsarSourceOptions}.
+ * <p>Make sure the option could be set only once or with same value.
*
* @param key the key of the property.
* @param value the value of the property.
* @return this PulsarSourceBuilder.
*/
- public <T> PulsarSourceBuilder<OUT> setProperty(ConfigOption<T> key, T value) {
+ public <T> PulsarSourceBuilder<OUT> setConfig(ConfigOption<T> key, T value) {
checkNotNull(key);
checkNotNull(value);
if (configuration.contains(key)) {
T oldValue = configuration.get(key);
- LOG.warn(
- "Config option {} already has a value {}, override to new value {}",
- key,
- oldValue,
- value);
+ checkArgument(
+ Objects.equals(oldValue, value),
+ "This option %s has been set to value %s.",
Review comment:
```suggestion
"This option %s has been already set to value %s.",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 60a73ab57a72105c906ac533cabe256e04bd821d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db84a292597428f82df1688e87787a6cc2591cfd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510)
* 93ea63d1795facc130ecb8c08a6a07562cfc22d7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-906320840
I merged it. Are you further improving the test stability or should I close FLINK-23877 for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-906320840
I merged it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise merged pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise merged pull request #16900:
URL: https://github.com/apache/flink/pull/16900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22876",
"triggerID" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3180e650ed121eb4706dffbd4584d2a08189e24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22876)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] syhily removed a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
syhily removed a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902382011
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91e338034e41b317aaa4427332faac129a338701 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
}, {
"hash" : "91e338034e41b317aaa4427332faac129a338701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22691",
"triggerID" : "91e338034e41b317aaa4427332faac129a338701",
"triggerType" : "PUSH"
}, {
"hash" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22773",
"triggerID" : "93c302d1b404337b8a5e509ff0b740c9c7d3e0a7",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"triggerType" : "PUSH"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22864",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "60a73ab57a72105c906ac533cabe256e04bd821d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22860",
"triggerID" : "906027971",
"triggerType" : "MANUAL"
}, {
"hash" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22876",
"triggerID" : "f3180e650ed121eb4706dffbd4584d2a08189e24",
"triggerType" : "PUSH"
}, {
"hash" : "958dbafc8a14a0f46e54db04fd9cbd0f21f5d57f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22882",
"triggerID" : "958dbafc8a14a0f46e54db04fd9cbd0f21f5d57f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3180e650ed121eb4706dffbd4584d2a08189e24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22876)
* 958dbafc8a14a0f46e54db04fd9cbd0f21f5d57f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22882)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
AHeise edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-906320840
I merged it. Thank you very much for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db84a292597428f82df1688e87787a6cc2591cfd UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16900: [FLINK-23877][connector/pulsar] Provide an embedded pulsar broker for testing.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16900:
URL: https://github.com/apache/flink/pull/16900#issuecomment-902199681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "db84a292597428f82df1688e87787a6cc2591cfd",
"triggerType" : "PUSH"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "db84a292597428f82df1688e87787a6cc2591cfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510",
"triggerID" : "902382011",
"triggerType" : "MANUAL"
}, {
"hash" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530",
"triggerID" : "93ea63d1795facc130ecb8c08a6a07562cfc22d7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db84a292597428f82df1688e87787a6cc2591cfd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22519) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22510)
* 93ea63d1795facc130ecb8c08a6a07562cfc22d7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22530)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org