You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ti...@apache.org on 2023/02/08 10:13:46 UTC
[flink-connector-pulsar] branch main updated: [FLINK-30689][Connector/Pulsar] Support sending message byte with extra schema check. (#20)
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-pulsar.git
The following commit(s) were added to refs/heads/main by this push:
new dfc2c54 [FLINK-30689][Connector/Pulsar] Support sending message byte with extra schema check. (#20)
dfc2c54 is described below
commit dfc2c54aaa1202bca35cc8fcab051b318e5bd074
Author: Yufan Sheng <yu...@streamnative.io>
AuthorDate: Wed Feb 8 18:13:40 2023 +0800
[FLINK-30689][Connector/Pulsar] Support sending message byte with extra schema check. (#20)
---
.../docs/connectors/datastream/pulsar.md | 34 ++++
docs/content/docs/connectors/datastream/pulsar.md | 43 +++++
.../generated/pulsar_sink_configuration.html | 10 +-
.../pulsar/common/crypto/DefaultPulsarCrypto.java | 3 +-
.../pulsar/common/crypto/PulsarCryptoDisabled.java | 1 +
.../pulsar/common/schema/PulsarSchemaUtils.java | 11 ++
.../common/utils/PulsarTransactionUtils.java | 26 +--
.../connector/pulsar/sink/PulsarSinkOptions.java | 16 +-
.../pulsar/sink/committer/PulsarCommitter.java | 119 ++++++-------
.../pulsar/sink/config/SinkConfiguration.java | 14 +-
.../connector/pulsar/sink/writer/PulsarWriter.java | 2 +-
.../pulsar/sink/writer/message/PulsarMessage.java | 47 ++++-
.../sink/writer/message/PulsarMessageBuilder.java | 45 +++--
.../writer/serializer/PulsarSchemaWrapper.java | 14 +-
.../serializer/PulsarSerializationSchema.java | 2 +
.../PulsarSerializationSchemaWrapper.java | 8 +-
.../sink/writer/topic/TopicProducerRegister.java | 189 ++++++++++++---------
.../pulsar/sink/writer/PulsarWriterTest.java | 5 +
.../writer/topic/TopicProducerRegisterTest.java | 34 +++-
.../testutils/runtime/PulsarRuntimeOperator.java | 4 +-
20 files changed, 413 insertions(+), 214 deletions(-)
diff --git a/docs/content.zh/docs/connectors/datastream/pulsar.md b/docs/content.zh/docs/connectors/datastream/pulsar.md
index ad10d39..2f732bd 100644
--- a/docs/content.zh/docs/connectors/datastream/pulsar.md
+++ b/docs/content.zh/docs/connectors/datastream/pulsar.md
@@ -698,6 +698,8 @@ PulsarSink.builder().set_topics(["topic-a-partition-0", "topic-a-partition-2", "
{{< /tab >}}
{{< /tabs >}}
+#### Schema evolution
+
同时使用 `PulsarSerializationSchema.pulsarSchema()` 以及在 builder 中指定 `PulsarSinkBuilder.enableSchemaEvolution()` 可以启用 [Schema evolution][schema-evolution] 特性。该特性会使用 Pulsar Broker 端提供的 Schema 版本兼容性检测以及 Schema 版本演进。下列示例展示了如何启用 Schema Evolution。
```java
@@ -717,6 +719,37 @@ PulsarSink<String> sink = PulsarSink.builder()
例如,如果使用 `PulsarSerializationSchema.pulsarSchema(Schema.STRING)` 而不使用 `PulsarSinkBuilder.enableSchemaEvolution()`。那么在写入 Topic 中所记录的消息 Schema 将会是 `Schema.BYTES`。
{{< /hint >}}
+#### PulsarMessage<byte[]> 类型的消息的校验
+
+Pulsar 的 topic 至少会包含一种 Schema,`Schema.BYTES` 是默认的 Schema 类型并常作为没有 Schema 的 topic 的 Schema 类型。使用 `Schema.BYTES` 发送消息将会跳过类型检测,这意味着使用 `SerializationSchema` 和没有启用 Schema evolution 的 `Schema` 所发送的消息并不安全。
+
+可以启用 `pulsar.sink.validateSinkMessageBytes` 选项来让链接器使用 Pulsar 提供的 `Schema.AUTO_PRODUCE_BYTES()` 发送消息。它会在发送字节数组消息时额外进行校验,与 topic 上最新版本的 Schema 进行对比,保证消息内容能正确。
+
+但是,并非所有的 Pulsar 的 Schema 都支持校验字符串,所以在默认情况下我们禁用了此选项。可以按需启用。
+
+#### 自定义序列化器
+
+可以通过继承 `PulsarSerializationSchema` 接口来实现自定义的序列化逻辑。接口需要返回一个类型为 `PulsarMessage` 的消息,此类型实例无法被直接创建,连接器提供了构造方法并定义了三种消息类型的构建。
+
+- 使用 Pulsar 的 `Scheme` 来构建消息,常用于你知道 topic 对应的 schema 是什么的时候。我们会检查你提供的 `Schema` 在 topic 上是否兼容。
+ ```java
+ PulsarMessage.builder(Schema<M> schema, M message)
+ ...
+ .build();
+ ```
+- 创建一个消息类型为字节数组的消息。默认情况下不进行 Schema 检查。
+ ```java
+ PulsarMessage.builder(byte[] bytes)
+ ...
+ .build();
+ ```
+- 创建一个消息体为空的墓碑消息。[墓碑][tombstone-data-store] 是一种特殊的消息,并在 Pulsar 中提供了支持。
+ ```java
+ PulsarMessage.builder()
+ ...
+ .build();
+ ```
+
### 消息路由策略
在 Pulsar Sink 中,消息路由发生在于分区之间,而非上层 Topic。对于给定 Topic 的情况,路由算法会首先会查询出 Topic 之上所有的分区信息,并在这些分区上实现消息的路由。Pulsar Sink 默认提供 2 种路由策略的实现。
@@ -1063,3 +1096,4 @@ Pulsar 事务机制仍在积极发展中,当前版本并不稳定。 Pulsar 2.
[schema-evolution]: https://pulsar.apache.org/docs/2.10.x/schema-evolution-compatibility/#schema-evolution
[standard-metrics]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
+[tombstone-data-store]: https://en.wikipedia.org/wiki/Tombstone_(data_store)
\ No newline at end of file
diff --git a/docs/content/docs/connectors/datastream/pulsar.md b/docs/content/docs/connectors/datastream/pulsar.md
index 1d0f851..78bfb7a 100644
--- a/docs/content/docs/connectors/datastream/pulsar.md
+++ b/docs/content/docs/connectors/datastream/pulsar.md
@@ -800,6 +800,8 @@ you can use the predefined `PulsarSerializationSchema`. The Pulsar sink provides
{{< /tab >}}
{{< /tabs >}}
+#### Schema evolution
+
[Schema evolution][schema-evolution] can be enabled by users using `PulsarSerializationSchema.pulsarSchema()` and
`PulsarSinkBuilder.enableSchemaEvolution()`. This means that any broker schema validation is in place.
@@ -821,6 +823,46 @@ For example, if you set `PulsarSerializationSchema.pulsarSchema(Schema.STRING)`
the schema stored in Pulsar topics is `Schema.BYTES`.
{{< /hint >}}
+#### PulsarMessage<byte[]> validation
+
+Pulsar topic always has at least one schema. The `Schema.BYTES` is the default one for any topic without schema being set.
+But sending messages bytes with `Schema.BYTES` bypass the schema validate. So the message sent with `SerializationSchema`
+and `Schema` which doesn't enable the schema evolution may save the invalid messages in the topic.
+
+You can enable the `pulsar.sink.validateSinkMessageBytes` option to let the connector use the Pulsar's `Schema.AUTO_PRODUCE_BYTES()`
+which supports extra check for the message bytes before sending. It will query the latest schema in topic and use it to
+validate the message bytes.
+
+But some schemas in Pulsar don't support validation, so we disable this option by default. you should use it at your own risk.
+
+#### Custom serializer
+
+You can have your own serialization logic by implementing the `PulsarSerializationSchema` interface. The return type
+for this interface is `PulsarMessage` which you can't create it directly. Instead, we use builder method for creating
+three types of the Pulsar messages.
+
+- Create a message with a Pulsar `Scheme`. This is always when you know the schema in topic.
+ We will check if the given schema is compatible in the correspond topic.
+ ```java
+ PulsarMessage.builder(Schema<M> schema, M message)
+ ...
+ .build();
+ ```
+- Create the message without any Pulsar `Scheme`. The message type can only be the byte array.
+ It won't validate the message bytes by default.
+ ```java
+ PulsarMessage.builder(byte[] bytes)
+ ...
+ .build();
+ ```
+- Create a tombstone message with empty payloads. [Tombstone][tombstone-data-store] is a special message which is
+ supported in Pulsar.
+ ```java
+ PulsarMessage.builder()
+ ...
+ .build();
+ ```
+
### Message Routing
Routing in Pulsar Sink is operated on the partition level. For a list of partitioned topics,
@@ -1225,3 +1267,4 @@ You can use the latest `pulsar-client-all` release to resolve this issue.
[schema-evolution]: https://pulsar.apache.org/docs/2.10.x/schema-evolution-compatibility/#schema-evolution
[standard-metrics]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
+[tombstone-data-store]: https://en.wikipedia.org/wiki/Tombstone_(data_store)
\ No newline at end of file
diff --git a/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html b/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html
index a40d3c0..fe5f66c 100644
--- a/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html
+++ b/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html
@@ -18,7 +18,7 @@
<td><h5>pulsar.sink.enableMetrics</h5></td>
<td style="word-wrap: break-word;">true</td>
<td>Boolean</td>
- <td>The metrics from Pulsar Producer are only exposed if you enable this option.You should set the <code class="highlighter-rouge">pulsar.client.statsIntervalSeconds</code> to a positive value if you enable this option.</td>
+ <td>The metrics from Pulsar Producer are only exposed if you enable this option. You should set the <code class="highlighter-rouge">pulsar.client.statsIntervalSeconds</code> to a positive value if you enable this option.</td>
</tr>
<tr>
<td><h5>pulsar.sink.enableSchemaEvolution</h5></td>
@@ -48,7 +48,13 @@
<td><h5>pulsar.sink.transactionTimeoutMillis</h5></td>
<td style="word-wrap: break-word;">10800000</td>
<td>Long</td>
- <td>This option is used when the user require the <code class="highlighter-rouge">DeliveryGuarantee.EXACTLY_ONCE</code> semantic.We would use transaction for making sure the message could be write only once.</td>
+ <td>This option is used when the user require the <code class="highlighter-rouge">DeliveryGuarantee.EXACTLY_ONCE</code> semantic. We would use transaction for making sure the message could be write only once.</td>
+ </tr>
+ <tr>
+ <td><h5>pulsar.sink.validateSinkMessageBytes</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>Pulsar client can validate the raw message bytes with the latest topic schema. This can make sure your serialized messages bytes is valid for consumer.</td>
</tr>
</tbody>
</table>
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/DefaultPulsarCrypto.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/DefaultPulsarCrypto.java
index 97c84d2..39ca287 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/DefaultPulsarCrypto.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/DefaultPulsarCrypto.java
@@ -37,6 +37,7 @@ import static org.apache.flink.util.Preconditions.checkNotNull;
/** The default implementation for {@link PulsarCrypto}. */
@PublicEvolving
public class DefaultPulsarCrypto implements PulsarCrypto {
+ private static final long serialVersionUID = 2246425411998723487L;
private final CryptoKeyReader cryptoKeyReader;
private final Set<String> encryptKeys;
@@ -93,7 +94,7 @@ public class DefaultPulsarCrypto implements PulsarCrypto {
public DefaultPulsarCryptoBuilder messageCrypto(
SerializableSupplier<MessageCrypto<MessageMetadata, MessageMetadata>>
messageCryptoSupplier) {
- this.messageCryptoSupplier = messageCryptoSupplier;
+ this.messageCryptoSupplier = checkNotNull(messageCryptoSupplier);
return this;
}
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/PulsarCryptoDisabled.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/PulsarCryptoDisabled.java
index 7f6902f..fb6c0ec 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/PulsarCryptoDisabled.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/PulsarCryptoDisabled.java
@@ -28,6 +28,7 @@ import java.util.Set;
/** A default implementation, we will use it if the user didn't enable end-to-end encryption. */
@Internal
public class PulsarCryptoDisabled implements PulsarCrypto {
+ private static final long serialVersionUID = 7735076959011427862L;
@Override
public CryptoKeyReader cryptoKeyReader() {
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/schema/PulsarSchemaUtils.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/schema/PulsarSchemaUtils.java
index 00bf2cf..b346006 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/schema/PulsarSchemaUtils.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/schema/PulsarSchemaUtils.java
@@ -30,6 +30,8 @@ import org.apache.flink.connector.pulsar.common.schema.factories.StringSchemaFac
import com.google.protobuf.Message;
import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.impl.schema.AutoConsumeSchema;
+import org.apache.pulsar.client.impl.schema.AutoProduceBytesSchema;
import org.apache.pulsar.client.impl.schema.BooleanSchema;
import org.apache.pulsar.client.impl.schema.ByteSchema;
import org.apache.pulsar.client.impl.schema.BytesSchema;
@@ -46,6 +48,7 @@ import org.apache.pulsar.client.impl.schema.SchemaInfoImpl;
import org.apache.pulsar.client.impl.schema.ShortSchema;
import org.apache.pulsar.client.impl.schema.TimeSchema;
import org.apache.pulsar.client.impl.schema.TimestampSchema;
+import org.apache.pulsar.common.protocol.schema.SchemaHash;
import org.apache.pulsar.common.schema.SchemaInfo;
import org.apache.pulsar.common.schema.SchemaType;
@@ -204,4 +207,12 @@ public final class PulsarSchemaUtils {
throw new IllegalStateException("Couldn't find the schema class " + className);
}
}
+
+ public static SchemaHash hash(Schema<?> schema) {
+ if (schema instanceof AutoProduceBytesSchema || schema instanceof AutoConsumeSchema) {
+ return SchemaHash.of(Schema.BYTES);
+ }
+
+ return SchemaHash.of(schema);
+ }
}
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarTransactionUtils.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarTransactionUtils.java
index c8ffe63..fbb597d 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarTransactionUtils.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarTransactionUtils.java
@@ -23,14 +23,17 @@ import org.apache.flink.util.FlinkRuntimeException;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.transaction.Transaction;
-import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientException;
+import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyClient;
-import static org.apache.flink.util.ExceptionUtils.findThrowable;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientException.unwrap;
/** A suit of workarounds for the Pulsar Transaction. */
@Internal
@@ -53,18 +56,21 @@ public final class PulsarTransactionUtils {
Thread.currentThread().interrupt();
throw new IllegalStateException(e);
} catch (ExecutionException e) {
- throw new FlinkRuntimeException(e);
+ throw new FlinkRuntimeException(unwrap(e));
}
}
/**
- * This is a bug in original {@link TransactionCoordinatorClientException#unwrap(Throwable)}
- * method. Pulsar wraps the {@link ExecutionException} which hides the real execution exception.
- *
- * <p>This bug should be fixed after the 2.10.0 release. We just keep this for safety.
+ * {@link PulsarClient} didn't expose the internal {@link TransactionCoordinatorClient} to the
+ * end user. But the connector needs it to manually commit/abort the transaction by {@link
+ * TxnID}.
*/
- public static TransactionCoordinatorClientException unwrap(
- TransactionCoordinatorClientException e) {
- return findThrowable(e.getCause(), TransactionCoordinatorClientException.class).orElse(e);
+ public static TransactionCoordinatorClient getTcClient(PulsarClient pulsarClient) {
+ TransactionCoordinatorClient coordinatorClient =
+ ((PulsarClientImpl) pulsarClient).getTcClient();
+ // Ensure you have enabled transaction.
+ checkNotNull(coordinatorClient, "You haven't enable transaction in Pulsar client.");
+
+ return coordinatorClient;
}
}
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/PulsarSinkOptions.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/PulsarSinkOptions.java
index dbc1026..a5bb977 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/PulsarSinkOptions.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/PulsarSinkOptions.java
@@ -88,7 +88,7 @@ public final class PulsarSinkOptions {
.withDescription(
Description.builder()
.text(
- "This option is used when the user require the %s semantic.",
+ "This option is used when the user require the %s semantic. ",
code("DeliveryGuarantee.EXACTLY_ONCE"))
.text(
"We would use transaction for making sure the message could be write only once.")
@@ -137,12 +137,24 @@ public final class PulsarSinkOptions {
.withDescription(
Description.builder()
.text(
- "The metrics from Pulsar Producer are only exposed if you enable this option.")
+ "The metrics from Pulsar Producer are only exposed if you enable this option. ")
.text(
"You should set the %s to a positive value if you enable this option.",
code(PULSAR_STATS_INTERVAL_SECONDS.key()))
.build());
+ public static final ConfigOption<Boolean> PULSAR_VALIDATE_SINK_MESSAGE_BYTES =
+ ConfigOptions.key(SINK_CONFIG_PREFIX + "validateSinkMessageBytes")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ Description.builder()
+ .text(
+ "Pulsar client can validate the raw message bytes with the latest topic schema. ")
+ .text(
+ "This can make sure your serialized messages bytes is valid for consumer.")
+ .build());
+
///////////////////////////////////////////////////////////////////////////////
//
// The configuration for ProducerConfigurationData part.
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/committer/PulsarCommitter.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/committer/PulsarCommitter.java
index 8389bdc..11e80c7 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/committer/PulsarCommitter.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/committer/PulsarCommitter.java
@@ -21,7 +21,6 @@ package org.apache.flink.connector.pulsar.sink.committer;
import org.apache.flink.annotation.Internal;
import org.apache.flink.api.connector.sink2.Committer;
import org.apache.flink.connector.base.DeliveryGuarantee;
-import org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils;
import org.apache.flink.connector.pulsar.sink.PulsarSink;
import org.apache.flink.connector.pulsar.sink.config.SinkConfiguration;
import org.apache.flink.util.FlinkRuntimeException;
@@ -34,7 +33,6 @@ import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientExce
import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientException.MetaStoreHandlerNotExistsException;
import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientException.TransactionNotFoundException;
import org.apache.pulsar.client.api.transaction.TxnID;
-import org.apache.pulsar.client.impl.PulsarClientImpl;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -43,6 +41,7 @@ import java.io.IOException;
import java.util.Collection;
import static org.apache.flink.connector.pulsar.common.config.PulsarClientFactory.createClient;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils.getTcClient;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.pulsar.common.naming.TopicName.TRANSACTION_COORDINATOR_ASSIGN;
@@ -65,8 +64,8 @@ public class PulsarCommitter implements Committer<PulsarCommittable>, Closeable
}
@Override
- public void commit(Collection<CommitRequest<PulsarCommittable>> requests)
- throws IOException, InterruptedException {
+ @SuppressWarnings("java:S3776")
+ public void commit(Collection<CommitRequest<PulsarCommittable>> requests) {
TransactionCoordinatorClient client = transactionCoordinatorClient();
for (CommitRequest<PulsarCommittable> request : requests) {
@@ -77,66 +76,61 @@ public class PulsarCommitter implements Committer<PulsarCommittable>, Closeable
LOG.debug("Start committing the Pulsar transaction {} for topic {}", txnID, topic);
try {
client.commit(txnID);
- } catch (TransactionCoordinatorClientException e) {
- // This is a known bug for Pulsar Transaction.
- // We have to use instanceof instead of catching them.
- TransactionCoordinatorClientException ex = PulsarTransactionUtils.unwrap(e);
- if (ex instanceof CoordinatorNotFoundException) {
- LOG.error(
- "We couldn't find the Transaction Coordinator from Pulsar broker {}. "
- + "Check your broker configuration.",
- committable,
- ex);
- request.signalFailedWithKnownReason(ex);
- } else if (ex instanceof InvalidTxnStatusException) {
+ } catch (CoordinatorNotFoundException e) {
+ LOG.error(
+ "We couldn't find the Transaction Coordinator from Pulsar broker {}. "
+ + "Check your broker configuration.",
+ committable,
+ e);
+ request.signalFailedWithKnownReason(e);
+ } catch (InvalidTxnStatusException e) {
+ LOG.error(
+ "Unable to commit transaction ({}) because it's in an invalid state. "
+ + "Most likely the transaction has been aborted for some reason. "
+ + "Please check the Pulsar broker logs for more details.",
+ committable,
+ e);
+ request.signalAlreadyCommitted();
+ } catch (TransactionNotFoundException e) {
+ if (request.getNumberOfRetries() == 0) {
LOG.error(
- "Unable to commit transaction ({}) because it's in an invalid state. "
- + "Most likely the transaction has been aborted for some reason. "
- + "Please check the Pulsar broker logs for more details.",
+ "Unable to commit transaction ({}) because it's not found on Pulsar broker. "
+ + "Most likely the checkpoint interval exceed the transaction timeout.",
committable,
- ex);
+ e);
+ request.signalFailedWithKnownReason(e);
+ } else {
+ LOG.warn(
+ "We can't find the transaction {} after {} retry committing. "
+ + "This may mean that the transaction have been committed in previous but failed with timeout. "
+ + "So we just mark it as committed.",
+ txnID,
+ request.getNumberOfRetries());
request.signalAlreadyCommitted();
- } else if (ex instanceof TransactionNotFoundException) {
- if (request.getNumberOfRetries() == 0) {
- LOG.error(
- "Unable to commit transaction ({}) because it's not found on Pulsar broker. "
- + "Most likely the checkpoint interval exceed the transaction timeout.",
- committable,
- ex);
- request.signalFailedWithKnownReason(ex);
- } else {
- LOG.warn(
- "We can't find the transaction {} after {} retry committing. "
- + "This may mean that the transaction have been committed in previous but failed with timeout. "
- + "So we just mark it as committed.",
- txnID,
- request.getNumberOfRetries());
- request.signalAlreadyCommitted();
- }
- } else if (ex instanceof MetaStoreHandlerNotExistsException) {
- LOG.error(
- "We can't find the meta store handler by the mostSigBits from TxnID {}. "
- + "Did you change the metadata for topic {}?",
- committable,
- TRANSACTION_COORDINATOR_ASSIGN,
- ex);
- request.signalFailedWithKnownReason(ex);
+ }
+ } catch (MetaStoreHandlerNotExistsException e) {
+ LOG.error(
+ "We can't find the meta store handler by the mostSigBits from TxnID {}. "
+ + "Did you change the metadata for topic {}?",
+ committable,
+ TRANSACTION_COORDINATOR_ASSIGN,
+ e);
+ request.signalFailedWithKnownReason(e);
+ } catch (TransactionCoordinatorClientException e) {
+ LOG.error(
+ "Encountered retriable exception while committing transaction {} for topic {}.",
+ committable,
+ topic,
+ e);
+ int maxRecommitTimes = sinkConfiguration.getMaxRecommitTimes();
+ if (request.getNumberOfRetries() < maxRecommitTimes) {
+ request.retryLater();
} else {
- LOG.error(
- "Encountered retriable exception while committing transaction {} for topic {}.",
- committable,
- topic,
- ex);
- int maxRecommitTimes = sinkConfiguration.getMaxRecommitTimes();
- if (request.getNumberOfRetries() < maxRecommitTimes) {
- request.retryLater();
- } else {
- String message =
- String.format(
- "Failed to commit transaction %s after retrying %d times",
- txnID, maxRecommitTimes);
- request.signalFailedWithKnownReason(new FlinkRuntimeException(message, ex));
- }
+ String message =
+ String.format(
+ "Failed to commit transaction %s after retrying %d times",
+ txnID, maxRecommitTimes);
+ request.signalFailedWithKnownReason(new FlinkRuntimeException(message, e));
}
} catch (Exception e) {
LOG.error(
@@ -156,10 +150,7 @@ public class PulsarCommitter implements Committer<PulsarCommittable>, Closeable
private TransactionCoordinatorClient transactionCoordinatorClient() {
if (coordinatorClient == null) {
this.pulsarClient = createClient(sinkConfiguration);
- this.coordinatorClient = ((PulsarClientImpl) pulsarClient).getTcClient();
-
- // Ensure you have enabled transaction.
- checkNotNull(coordinatorClient, "You haven't enable transaction in Pulsar client.");
+ this.coordinatorClient = getTcClient(pulsarClient);
}
return coordinatorClient;
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/config/SinkConfiguration.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/config/SinkConfiguration.java
index 0a56ff3..d041bde 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/config/SinkConfiguration.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/config/SinkConfiguration.java
@@ -37,6 +37,7 @@ import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_EN
import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_MAX_RECOMMIT_TIMES;
import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_MESSAGE_KEY_HASH;
import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_TOPIC_METADATA_REFRESH_INTERVAL;
+import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_VALIDATE_SINK_MESSAGE_BYTES;
import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_DELIVERY_GUARANTEE;
import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_SCHEMA_EVOLUTION;
import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_TRANSACTION_TIMEOUT;
@@ -54,6 +55,7 @@ public class SinkConfiguration extends PulsarConfiguration {
private final boolean enableSchemaEvolution;
private final int maxRecommitTimes;
private final boolean enableMetrics;
+ private final boolean validateSinkMessageBytes;
public SinkConfiguration(Configuration configuration) {
super(configuration);
@@ -67,6 +69,7 @@ public class SinkConfiguration extends PulsarConfiguration {
this.maxRecommitTimes = get(PULSAR_MAX_RECOMMIT_TIMES);
this.enableMetrics =
get(PULSAR_ENABLE_SINK_METRICS) && get(PULSAR_STATS_INTERVAL_SECONDS) > 0;
+ this.validateSinkMessageBytes = get(PULSAR_VALIDATE_SINK_MESSAGE_BYTES);
}
/** The delivery guarantee changes the behavior of {@link PulsarWriter}. */
@@ -123,6 +126,11 @@ public class SinkConfiguration extends PulsarConfiguration {
return enableMetrics;
}
+ /** Whether to add extra schema check for byte messages. */
+ public boolean isValidateSinkMessageBytes() {
+ return validateSinkMessageBytes;
+ }
+
@Override
public boolean equals(Object o) {
if (this == o) {
@@ -141,7 +149,8 @@ public class SinkConfiguration extends PulsarConfiguration {
&& enableSchemaEvolution == that.enableSchemaEvolution
&& messageKeyHash == that.messageKeyHash
&& maxRecommitTimes == that.maxRecommitTimes
- && enableMetrics == that.enableMetrics;
+ && enableMetrics == that.enableMetrics
+ && validateSinkMessageBytes == that.validateSinkMessageBytes;
}
@Override
@@ -154,6 +163,7 @@ public class SinkConfiguration extends PulsarConfiguration {
messageKeyHash,
enableSchemaEvolution,
maxRecommitTimes,
- enableMetrics);
+ enableMetrics,
+ validateSinkMessageBytes);
}
}
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriter.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriter.java
index 0d3009c..f5a4bb7 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriter.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriter.java
@@ -141,7 +141,7 @@ public class PulsarWriter<IN> implements PrecommittingSinkWriter<IN, PulsarCommi
List<String> availableTopics = metadataListener.availableTopics();
String topic = topicRouter.route(element, key, availableTopics, sinkContext);
- // Create message builder for sending message.
+ // Create message builder for sending messages.
TypedMessageBuilder<?> builder = createMessageBuilder(topic, context, message);
// Message Delay delivery.
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessage.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessage.java
index 0c45763..dc01908 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessage.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessage.java
@@ -28,6 +28,8 @@ import javax.annotation.Nullable;
import java.util.List;
import java.util.Map;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
/**
* The message instance would be used for {@link TypedMessageBuilder}. We create this class because
* the Pulsar lacks such kind of POJO class.
@@ -38,19 +40,25 @@ public class PulsarMessage<T> {
@Nullable private final byte[] orderingKey;
@Nullable private final String key;
private final long eventTime;
- private final Schema<T> schema;
+ @Nullable private final Schema<T> schema;
@Nullable private final T value;
@Nullable private final Map<String, String> properties;
@Nullable private final Long sequenceId;
@Nullable private final List<String> replicationClusters;
private final boolean disableReplication;
- /** Package private for building this class only in {@link PulsarMessageBuilder}. */
+ /**
+ * Package private for building this class only in {@link PulsarMessageBuilder}. Use the {@link
+ * #builder(byte[])}, {@link #builder(Schema, Object)} methods for creating the builder. And
+ * {@link #builder()} is only used for creating a special tombstone message, use it as you
+ * needs.
+ */
+ @SuppressWarnings("java:S107")
PulsarMessage(
@Nullable byte[] orderingKey,
@Nullable String key,
long eventTime,
- Schema<T> schema,
+ @Nullable Schema<T> schema,
@Nullable T value,
@Nullable Map<String, String> properties,
@Nullable Long sequenceId,
@@ -67,6 +75,38 @@ public class PulsarMessage<T> {
this.disableReplication = disableReplication;
}
+ /**
+ * Method wrapper of {@link TypedMessageBuilder#value(Object)}. You can pass any schema for
+ * validating it on Pulsar. This is called schema evolution. But the topic on Pulsar should bind
+ * to a fixed {@link Schema}. You may not have multiple schemas on the same topic unless it's
+ * compatible with each other. This is determined by the configured <a
+ * href="https://pulsar.apache.org/docs/2.10.x/schema-evolution-compatibility/#schema-compatibility-check-strategy">schema
+ * evolution policy</a> on corresponding topic.
+ */
+ public static <M> PulsarMessageBuilder<M> builder(Schema<M> schema, M message) {
+ checkNotNull(schema, "Schema should be provided.");
+ checkNotNull(message, "Message should be provided.");
+ return new PulsarMessageBuilder<>(schema, message);
+ }
+
+ /**
+ * Create a message with a pre-serialized byte array. This message can be sent to any topic with
+ * any schema. We will just bypass the Pulsar's schema evolution check. But if the target topic
+ * does not exist on Pulsar. The auto-created topic won't have any schema in this way.
+ */
+ public static PulsarMessageBuilder<byte[]> builder(byte[] bytes) {
+ checkNotNull(bytes, "Message bytes should be provided.");
+ return new PulsarMessageBuilder<>(Schema.BYTES, bytes);
+ }
+
+ /**
+ * Create a tombstone message with empty payloads. It will be skipped and considered deleted
+ * (akin to the concept of tombstones in key-value databases).
+ */
+ public static PulsarMessageBuilder<byte[]> builder() {
+ return new PulsarMessageBuilder<>(null, null);
+ }
+
@Nullable
public byte[] getOrderingKey() {
return orderingKey;
@@ -81,6 +121,7 @@ public class PulsarMessage<T> {
return eventTime;
}
+ @Nullable
public Schema<T> getSchema() {
return schema;
}
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessageBuilder.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessageBuilder.java
index 5185932..95fb5cb 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessageBuilder.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessageBuilder.java
@@ -24,10 +24,13 @@ import org.apache.flink.connector.pulsar.sink.writer.router.KeyHashTopicRouter;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.TypedMessageBuilder;
+import javax.annotation.Nullable;
+
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import static org.apache.flink.util.Preconditions.checkArgument;
import static org.apache.flink.util.Preconditions.checkNotNull;
/** {@link TypedMessageBuilder} wrapper for providing the required method for end-users. */
@@ -37,13 +40,22 @@ public class PulsarMessageBuilder<T> {
private byte[] orderingKey;
private String key;
private long eventTime;
- Schema<T> schema;
- private T value;
- private Map<String, String> properties = new HashMap<>();
+ private final Schema<T> schema;
+ private final T value;
+ private final Map<String, String> properties = new HashMap<>();
private Long sequenceId;
private List<String> replicationClusters;
private boolean disableReplication = false;
+ /**
+ * Make this constructor package private for in favor of the {@link PulsarMessage#builder()}
+ * method in {@link PulsarMessage}.
+ */
+ PulsarMessageBuilder(Schema<T> schema, @Nullable T value) {
+ this.schema = schema;
+ this.value = value;
+ }
+
/** Method wrapper of {@link TypedMessageBuilder#orderingKey(byte[])}. */
public PulsarMessageBuilder<T> orderingKey(byte[] orderingKey) {
this.orderingKey = checkNotNull(orderingKey);
@@ -51,32 +63,21 @@ public class PulsarMessageBuilder<T> {
}
/**
- * Property {@link TypedMessageBuilder#key(String)}. This property would also be used in {@link
- * KeyHashTopicRouter}.
+ * Method wrapper of {@link TypedMessageBuilder#key(String)}. This key would also be used in
+ * {@link KeyHashTopicRouter}.
*/
public PulsarMessageBuilder<T> key(String key) {
this.key = checkNotNull(key);
return this;
}
- /** Method wrapper of {@link TypedMessageBuilder#eventTime(long)}. */
- public PulsarMessageBuilder<T> eventTime(long eventTime) {
- this.eventTime = eventTime;
- return this;
- }
-
/**
- * Method wrapper of {@link TypedMessageBuilder#value(Object)}. You can pass any schema for
- * validating it on Pulsar. This is called schema evolution. But the topic on Pulsar should bind
- * to a fixed {@link Schema}. You can't have multiple schemas on the same topic unless it's
- * compatible with each other.
- *
- * @param value The value could be null, which is called tombstones message in Pulsar. (It will
- * be skipped and considered deleted.)
+ * Method wrapper of {@link TypedMessageBuilder#eventTime(long)}. If you don't provide the event
+ * time, we will try to use Flink's sink context time instead.
*/
- public PulsarMessageBuilder<T> value(Schema<T> schema, T value) {
- this.schema = checkNotNull(schema);
- this.value = value;
+ public PulsarMessageBuilder<T> eventTime(long eventTime) {
+ checkArgument(eventTime > 0, "The given event time should above 0.");
+ this.eventTime = eventTime;
return this;
}
@@ -111,8 +112,6 @@ public class PulsarMessageBuilder<T> {
}
public PulsarMessage<T> build() {
- checkNotNull(schema, "Schema should be provided.");
-
return new PulsarMessage<>(
orderingKey,
key,
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSchemaWrapper.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSchemaWrapper.java
index 0d508e5..7c6aef9 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSchemaWrapper.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSchemaWrapper.java
@@ -22,7 +22,6 @@ import org.apache.flink.annotation.Internal;
import org.apache.flink.connector.pulsar.common.schema.PulsarSchema;
import org.apache.flink.connector.pulsar.sink.writer.context.PulsarSinkContext;
import org.apache.flink.connector.pulsar.sink.writer.message.PulsarMessage;
-import org.apache.flink.connector.pulsar.sink.writer.message.PulsarMessageBuilder;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.common.schema.KeyValue;
@@ -56,18 +55,11 @@ public class PulsarSchemaWrapper<IN> implements PulsarSerializationSchema<IN> {
@Override
public PulsarMessage<?> serialize(IN element, PulsarSinkContext sinkContext) {
- Schema<IN> schema = this.pulsarSchema.getPulsarSchema();
+ Schema<IN> schema = pulsarSchema.getPulsarSchema();
if (sinkContext.isEnableSchemaEvolution()) {
- PulsarMessageBuilder<IN> builder = new PulsarMessageBuilder<>();
- builder.value(schema, element);
-
- return builder.build();
+ return PulsarMessage.builder(schema, element).build();
} else {
- PulsarMessageBuilder<byte[]> builder = new PulsarMessageBuilder<>();
- byte[] bytes = schema.encode(element);
- builder.value(Schema.BYTES, bytes);
-
- return builder.build();
+ return PulsarMessage.builder(schema.encode(element)).build();
}
}
}
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchema.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchema.java
index 45e3279..a61f7b8 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchema.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchema.java
@@ -53,6 +53,7 @@ public interface PulsarSerializationSchema<IN> extends Serializable {
* @param sinkConfiguration All the configure options for the Pulsar sink. You can add custom
* options.
*/
+ @SuppressWarnings("java:S112")
default void open(
InitializationContext initializationContext,
PulsarSinkContext sinkContext,
@@ -70,6 +71,7 @@ public interface PulsarSerializationSchema<IN> extends Serializable {
* @param element Element to be serialized.
* @param sinkContext Context to provide extra information.
*/
+ @SuppressWarnings("java:S1452")
PulsarMessage<?> serialize(IN element, PulsarSinkContext sinkContext);
/**
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchemaWrapper.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchemaWrapper.java
index 716d2db..2f24f94 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchemaWrapper.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchemaWrapper.java
@@ -24,9 +24,6 @@ import org.apache.flink.api.common.serialization.SerializationSchema.Initializat
import org.apache.flink.connector.pulsar.sink.config.SinkConfiguration;
import org.apache.flink.connector.pulsar.sink.writer.context.PulsarSinkContext;
import org.apache.flink.connector.pulsar.sink.writer.message.PulsarMessage;
-import org.apache.flink.connector.pulsar.sink.writer.message.PulsarMessageBuilder;
-
-import org.apache.pulsar.client.api.Schema;
/** Wrap the Flink's SerializationSchema into PulsarSerializationSchema. */
@Internal
@@ -50,10 +47,7 @@ public class PulsarSerializationSchemaWrapper<IN> implements PulsarSerialization
@Override
public PulsarMessage<?> serialize(IN element, PulsarSinkContext sinkContext) {
- PulsarMessageBuilder<byte[]> builder = new PulsarMessageBuilder<>();
byte[] value = serializationSchema.serialize(element);
- builder.value(Schema.BYTES, value);
-
- return builder.build();
+ return PulsarMessage.builder(value).build();
}
}
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegister.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegister.java
index ae99527..3b32467 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegister.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegister.java
@@ -22,6 +22,7 @@ import org.apache.flink.annotation.Internal;
import org.apache.flink.connector.base.DeliveryGuarantee;
import org.apache.flink.connector.pulsar.common.crypto.PulsarCrypto;
import org.apache.flink.connector.pulsar.common.metrics.ProducerMetricsInterceptor;
+import org.apache.flink.connector.pulsar.common.schema.PulsarSchemaUtils;
import org.apache.flink.connector.pulsar.sink.committer.PulsarCommittable;
import org.apache.flink.connector.pulsar.sink.config.SinkConfiguration;
import org.apache.flink.metrics.MetricGroup;
@@ -41,17 +42,21 @@ import org.apache.pulsar.client.api.TypedMessageBuilder;
import org.apache.pulsar.client.api.transaction.Transaction;
import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient;
import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.ProducerBase;
import org.apache.pulsar.client.impl.ProducerBuilderImpl;
-import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.client.impl.TypedMessageBuilderImpl;
import org.apache.pulsar.client.impl.conf.ProducerConfigurationData;
+import org.apache.pulsar.client.impl.transaction.TransactionImpl;
import org.apache.pulsar.common.api.proto.MessageMetadata;
-import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.protocol.schema.SchemaHash;
+import org.apache.pulsar.common.schema.SchemaType;
import org.apache.pulsar.shade.com.google.common.base.Strings;
+import javax.annotation.Nullable;
+
import java.io.Closeable;
import java.io.IOException;
import java.util.ArrayList;
-import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -79,23 +84,24 @@ import static org.apache.flink.connector.pulsar.common.metrics.MetricNames.TOTAL
import static org.apache.flink.connector.pulsar.common.metrics.MetricNames.TOTAL_SEND_FAILED;
import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyClient;
import static org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils.createTransaction;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils.getTcClient;
import static org.apache.flink.connector.pulsar.sink.config.PulsarSinkConfigUtils.createProducerBuilder;
-import static org.apache.flink.util.Preconditions.checkArgument;
-import static org.apache.flink.util.Preconditions.checkNotNull;
/**
- * All the Pulsar Producers share the same Client, but self hold the queue for a specified topic. So
+ * All the Pulsar Producers share the same Client, but self-hold the queue for a specified topic. So
* we have to create different instances for different topics.
*/
@Internal
public class TopicProducerRegister implements Closeable {
private final PulsarClient pulsarClient;
+ @Nullable private final TransactionCoordinatorClient coordinatorClient;
private final SinkConfiguration sinkConfiguration;
private final PulsarCrypto pulsarCrypto;
private final SinkWriterMetricGroup metricGroup;
- private final Map<String, Map<SchemaInfo, Producer<?>>> producerRegister;
- private final Map<String, Transaction> transactionRegister;
+ private final Map<String, Schema<byte[]>> schemas;
+ private final Map<String, Map<SchemaHash, Producer<?>>> producers;
+ private final Map<String, Transaction> transactions;
public TopicProducerRegister(
SinkConfiguration sinkConfiguration,
@@ -105,30 +111,46 @@ public class TopicProducerRegister implements Closeable {
this.sinkConfiguration = sinkConfiguration;
this.pulsarCrypto = pulsarCrypto;
this.metricGroup = metricGroup;
- this.producerRegister = new HashMap<>();
- this.transactionRegister = new HashMap<>();
+ this.schemas = new HashMap<>();
+ this.producers = new HashMap<>();
+ this.transactions = new HashMap<>();
if (sinkConfiguration.isEnableMetrics()) {
metricGroup.setCurrentSendTimeGauge(this::currentSendTimeGauge);
}
+
+ // Check if we have enabled the transaction in the exactly-once delivery guarantee.
+ if (sinkConfiguration.getDeliveryGuarantee() == DeliveryGuarantee.EXACTLY_ONCE) {
+ this.coordinatorClient = getTcClient(pulsarClient);
+ } else {
+ this.coordinatorClient = null;
+ }
}
/**
* Create a TypedMessageBuilder which could be sent to Pulsar directly. First, we would create a
- * topic-related producer or use a cached instead. Then we would try to find a topic-related
+ * topic-related producer or use a cached one instead. Then we would try to find a topic-related
* transaction. We would generate a transaction instance if there is no transaction. Finally, we
- * create the message builder and put the element into it.
+ * created the message builder and put the element into it.
+ *
+ * <p>Pulsar's Producer doesn't have {@code producer.newMessage(schema, transaction)} method. We
+ * have to manually create it.
*/
- public <T> TypedMessageBuilder<T> createMessageBuilder(String topic, Schema<T> schema) {
- Producer<T> producer = getOrCreateProducer(topic, schema);
- DeliveryGuarantee deliveryGuarantee = sinkConfiguration.getDeliveryGuarantee();
+ @SuppressWarnings("unchecked")
+ public <T> TypedMessageBuilder<T> createMessageBuilder(
+ String topic, @Nullable Schema<?> schema) {
+ if (schema == null || schema.getSchemaInfo().getType() == SchemaType.BYTES) {
+ schema = getBytesSchema(topic);
+ }
+ ProducerBase<?> producer = (ProducerBase<?>) getOrCreateProducer(topic, schema);
+ TransactionImpl transaction = null;
- if (deliveryGuarantee == DeliveryGuarantee.EXACTLY_ONCE) {
- Transaction transaction = getOrCreateTransaction(topic);
- return producer.newMessage(transaction);
- } else {
- return producer.newMessage();
+ if (sinkConfiguration.getDeliveryGuarantee() == DeliveryGuarantee.EXACTLY_ONCE) {
+ transaction = (TransactionImpl) getOrCreateTransaction(topic);
}
+
+ return (TypedMessageBuilder<T>)
+ new TypedMessageBuilderImpl<>(producer, schema, transaction);
}
/**
@@ -136,15 +158,16 @@ public class TopicProducerRegister implements Closeable {
* be removed until Flink triggered a checkpoint.
*/
public List<PulsarCommittable> prepareCommit() {
- List<PulsarCommittable> committables = new ArrayList<>(transactionRegister.size());
- transactionRegister.forEach(
- (topic, transaction) -> {
- TxnID txnID = transaction.getTxnID();
- PulsarCommittable committable = new PulsarCommittable(txnID, topic);
- committables.add(committable);
- });
+ List<PulsarCommittable> committables = new ArrayList<>(transactions.size());
+ for (Map.Entry<String, Transaction> entry : transactions.entrySet()) {
+ String topic = entry.getKey();
+ Transaction transaction = entry.getValue();
+ TxnID txnID = transaction.getTxnID();
+
+ committables.add(new PulsarCommittable(txnID, topic));
+ }
+ transactions.clear();
- clearTransactions();
return committables;
}
@@ -153,9 +176,8 @@ public class TopicProducerRegister implements Closeable {
* successfully persisted.
*/
public void flush() throws IOException {
- Collection<Map<SchemaInfo, Producer<?>>> collection = producerRegister.values();
- for (Map<SchemaInfo, Producer<?>> producers : collection) {
- for (Producer<?> producer : producers.values()) {
+ for (Map<SchemaHash, Producer<?>> set : producers.values()) {
+ for (Producer<?> producer : set.values()) {
producer.flush();
}
}
@@ -171,9 +193,9 @@ public class TopicProducerRegister implements Closeable {
closer.register(this::abortTransactions);
// Remove all the producers.
- closer.register(producerRegister::clear);
+ closer.register(producers::clear);
- // All the producers would be closed by this method.
+ // This method would close all the producers.
// We would block until all the producers have been successfully closed.
closer.register(pulsarClient);
}
@@ -182,39 +204,16 @@ public class TopicProducerRegister implements Closeable {
/** Create or return the cached topic-related producer. */
@SuppressWarnings("unchecked")
private <T> Producer<T> getOrCreateProducer(String topic, Schema<T> schema) {
- Map<SchemaInfo, Producer<?>> producers =
- producerRegister.computeIfAbsent(topic, key -> new HashMap<>());
- SchemaInfo schemaInfo = schema.getSchemaInfo();
-
- if (producers.containsKey(schemaInfo)) {
- return (Producer<T>) producers.get(schemaInfo);
+ Map<SchemaHash, Producer<?>> set = producers.computeIfAbsent(topic, t -> new HashMap<>());
+ SchemaHash hash = PulsarSchemaUtils.hash(schema);
+ if (set.containsKey(hash)) {
+ return (Producer<T>) set.get(hash);
}
ProducerBuilder<T> builder = createProducerBuilder(pulsarClient, schema, sinkConfiguration);
- // Set the message crypto key reader.
- CryptoKeyReader cryptoKeyReader = pulsarCrypto.cryptoKeyReader();
- if (cryptoKeyReader != null) {
- builder.cryptoKeyReader(cryptoKeyReader);
-
- // Set the encrypt keys.
- Set<String> encryptKeys = pulsarCrypto.encryptKeys();
- checkArgument(
- encryptKeys != null && !encryptKeys.isEmpty(),
- "You should provide encryptKeys in PulsarCrypto");
- encryptKeys.forEach(builder::addEncryptionKey);
-
- // Set the message crypto if provided.
- // Pulsar forgets to expose the config in producer builder.
- // See issue https://github.com/apache/pulsar/issues/19139
- MessageCrypto<MessageMetadata, MessageMetadata> messageCrypto =
- pulsarCrypto.messageCrypto();
- if (messageCrypto != null) {
- ProducerConfigurationData producerConfig =
- ((ProducerBuilderImpl<?>) builder).getConf();
- producerConfig.setMessageCrypto(messageCrypto);
- }
- }
+ // Enable end-to-end encryption if provided.
+ configPulsarCrypto(builder);
// Set the required topic name.
builder.topic(topic);
@@ -225,16 +224,43 @@ public class TopicProducerRegister implements Closeable {
// Expose the stats for calculating and monitoring.
exposeProducerMetrics(producer);
- producers.put(schemaInfo, producer);
+ set.put(hash, producer);
return producer;
}
+ private void configPulsarCrypto(ProducerBuilder<?> builder) {
+ CryptoKeyReader cryptoKeyReader = pulsarCrypto.cryptoKeyReader();
+ if (cryptoKeyReader == null) {
+ return;
+ }
+
+ // Set the message crypto key reader.
+ builder.cryptoKeyReader(cryptoKeyReader);
+
+ // Set the encrypt keys.
+ Set<String> encryptKeys = pulsarCrypto.encryptKeys();
+ if (encryptKeys == null || encryptKeys.isEmpty()) {
+ throw new IllegalArgumentException("You should provide encryptKeys in PulsarCrypto");
+ }
+ encryptKeys.forEach(builder::addEncryptionKey);
+
+ // Set the message crypto if provided.
+ // Pulsar forgets to expose the config in producer builder.
+ // See issue https://github.com/apache/pulsar/issues/19139
+ MessageCrypto<MessageMetadata, MessageMetadata> messageCrypto =
+ pulsarCrypto.messageCrypto();
+ if (messageCrypto != null) {
+ ProducerConfigurationData producerConfig = ((ProducerBuilderImpl<?>) builder).getConf();
+ producerConfig.setMessageCrypto(messageCrypto);
+ }
+ }
+
/**
* Get the cached topic-related transaction. Or create a new transaction after checkpointing.
*/
private Transaction getOrCreateTransaction(String topic) {
- return transactionRegister.computeIfAbsent(
+ return transactions.computeIfAbsent(
topic,
t -> {
long timeoutMillis = sinkConfiguration.getTransactionTimeoutMillis();
@@ -242,41 +268,40 @@ public class TopicProducerRegister implements Closeable {
});
}
+ /**
+ * {@link Schema#AUTO_PRODUCE_BYTES} is used for extra validation. But it should be initialized
+ * with extra info in the Pulsar client. So it can't be reused and will be cached here.
+ */
+ private Schema<byte[]> getBytesSchema(String topic) {
+ if (sinkConfiguration.isValidateSinkMessageBytes()) {
+ return schemas.computeIfAbsent(topic, t -> Schema.AUTO_PRODUCE_BYTES());
+ } else {
+ return Schema.BYTES;
+ }
+ }
+
/** Abort the existed transactions. This method would be used when closing PulsarWriter. */
private void abortTransactions() {
- if (transactionRegister.isEmpty()) {
+ if (coordinatorClient == null || transactions.isEmpty()) {
return;
}
- TransactionCoordinatorClient coordinatorClient =
- ((PulsarClientImpl) pulsarClient).getTcClient();
- // This null check is used for making sure transaction is enabled in client.
- checkNotNull(coordinatorClient);
-
try (Closer closer = Closer.create()) {
- for (Transaction transaction : transactionRegister.values()) {
+ for (Transaction transaction : transactions.values()) {
TxnID txnID = transaction.getTxnID();
closer.register(() -> coordinatorClient.abort(txnID));
}
- clearTransactions();
+ transactions.clear();
} catch (IOException e) {
throw new FlinkRuntimeException(e);
}
}
- /**
- * Clean these transactions. All transactions should be passed to Pulsar committer, we would
- * create new transaction when new message comes.
- */
- private void clearTransactions() {
- transactionRegister.clear();
- }
-
private Long currentSendTimeGauge() {
double sendTime =
- producerRegister.values().stream()
- .flatMap(v -> v.values().stream())
+ producers.values().stream()
+ .flatMap(set -> set.values().stream())
.map(Producer::getStats)
.mapToDouble(ProducerStats::getSendLatencyMillis50pct)
.average()
diff --git a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriterTest.java b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriterTest.java
index 9b949eb..d7a64fa 100644
--- a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriterTest.java
+++ b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriterTest.java
@@ -161,6 +161,11 @@ class PulsarWriterTest extends PulsarTestSuiteBase {
return 1;
}
+ @Override
+ public int getAttemptNumber() {
+ return 0;
+ }
+
@Override
public SinkWriterMetricGroup metricGroup() {
return metricGroup;
diff --git a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegisterTest.java b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegisterTest.java
index be4b964..00deda7 100644
--- a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegisterTest.java
+++ b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegisterTest.java
@@ -18,6 +18,7 @@
package org.apache.flink.connector.pulsar.sink.writer.topic;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.connector.base.DeliveryGuarantee;
import org.apache.flink.connector.pulsar.common.crypto.PulsarCrypto;
import org.apache.flink.connector.pulsar.sink.committer.PulsarCommittable;
@@ -26,19 +27,25 @@ import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.TypedMessageBuilder;
import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient;
import org.apache.pulsar.client.api.transaction.TxnID;
+import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
import java.io.IOException;
import java.util.List;
+import java.util.concurrent.ThreadLocalRandom;
import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
import static org.apache.flink.connector.base.DeliveryGuarantee.EXACTLY_ONCE;
+import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_VALIDATE_SINK_MESSAGE_BYTES;
import static org.apache.flink.metrics.groups.UnregisteredMetricsGroup.createSinkWriterMetricGroup;
import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
/** Unit tests for {@link TopicProducerRegister}. */
class TopicProducerRegisterTest extends PulsarTestSuiteBase {
@@ -50,7 +57,8 @@ class TopicProducerRegisterTest extends PulsarTestSuiteBase {
String topic = randomAlphabetic(10);
operator().createTopic(topic, 8);
- SinkConfiguration configuration = sinkConfiguration(deliveryGuarantee);
+ SinkConfiguration configuration =
+ new SinkConfiguration(operator().sinkConfig(deliveryGuarantee));
TopicProducerRegister register =
new TopicProducerRegister(
configuration, PulsarCrypto.disabled(), createSinkWriterMetricGroup());
@@ -79,7 +87,8 @@ class TopicProducerRegisterTest extends PulsarTestSuiteBase {
String topic = randomAlphabetic(10);
operator().createTopic(topic, 8);
- SinkConfiguration configuration = sinkConfiguration(deliveryGuarantee);
+ SinkConfiguration configuration =
+ new SinkConfiguration(operator().sinkConfig(deliveryGuarantee));
TopicProducerRegister register =
new TopicProducerRegister(
configuration, PulsarCrypto.disabled(), createSinkWriterMetricGroup());
@@ -91,7 +100,24 @@ class TopicProducerRegisterTest extends PulsarTestSuiteBase {
assertThat(committables).isEmpty();
}
- private SinkConfiguration sinkConfiguration(DeliveryGuarantee deliveryGuarantee) {
- return new SinkConfiguration(operator().sinkConfig(deliveryGuarantee));
+ @Test
+ void sendMessageBytesWithWrongSchemaAndEnableCheck() throws Exception {
+ String topic = randomAlphabetic(10);
+ operator().createTopic(topic, 8);
+ operator().admin().schemas().createSchema(topic, Schema.INT16.getSchemaInfo());
+
+ Configuration configuration = operator().sinkConfig(DeliveryGuarantee.AT_LEAST_ONCE);
+ configuration.set(PULSAR_VALIDATE_SINK_MESSAGE_BYTES, true);
+ SinkConfiguration sinkConfiguration = new SinkConfiguration(configuration);
+ TopicProducerRegister register =
+ new TopicProducerRegister(
+ sinkConfiguration, PulsarCrypto.disabled(), createSinkWriterMetricGroup());
+
+ long message = ThreadLocalRandom.current().nextLong();
+ TypedMessageBuilder<byte[]> builder = register.createMessageBuilder(topic, Schema.BYTES);
+
+ assertThrows(
+ SchemaSerializationException.class,
+ () -> builder.value(Schema.INT64.encode(message)));
}
}
diff --git a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntimeOperator.java b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntimeOperator.java
index c76bb2f..f975cbe 100644
--- a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntimeOperator.java
+++ b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntimeOperator.java
@@ -42,7 +42,6 @@ import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.TypedMessageBuilder;
import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient;
import org.apache.pulsar.client.api.transaction.TxnID;
-import org.apache.pulsar.client.impl.PulsarClientImpl;
import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.partition.PartitionedTopicMetadata;
@@ -68,6 +67,7 @@ import static org.apache.flink.connector.pulsar.common.config.PulsarOptions.PULS
import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyClient;
import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils.getTcClient;
import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_SEND_TIMEOUT_MS;
import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_DELIVERY_GUARANTEE;
import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_TRANSACTION_TIMEOUT;
@@ -396,7 +396,7 @@ public class PulsarRuntimeOperator implements Closeable {
/** Return the transaction coordinator client for operating {@link TxnID}. */
public TransactionCoordinatorClient coordinatorClient() {
- return ((PulsarClientImpl) client()).getTcClient();
+ return getTcClient(client());
}
/**