You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ti...@apache.org on 2023/02/08 10:13:46 UTC

[flink-connector-pulsar] branch main updated: [FLINK-30689][Connector/Pulsar] Support sending message byte with extra schema check. (#20)

This is an automated email from the ASF dual-hosted git repository.

tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-pulsar.git


The following commit(s) were added to refs/heads/main by this push:
     new dfc2c54  [FLINK-30689][Connector/Pulsar] Support sending message byte with extra schema check. (#20)
dfc2c54 is described below

commit dfc2c54aaa1202bca35cc8fcab051b318e5bd074
Author: Yufan Sheng <yu...@streamnative.io>
AuthorDate: Wed Feb 8 18:13:40 2023 +0800

    [FLINK-30689][Connector/Pulsar] Support sending message byte with extra schema check. (#20)
---
 .../docs/connectors/datastream/pulsar.md           |  34 ++++
 docs/content/docs/connectors/datastream/pulsar.md  |  43 +++++
 .../generated/pulsar_sink_configuration.html       |  10 +-
 .../pulsar/common/crypto/DefaultPulsarCrypto.java  |   3 +-
 .../pulsar/common/crypto/PulsarCryptoDisabled.java |   1 +
 .../pulsar/common/schema/PulsarSchemaUtils.java    |  11 ++
 .../common/utils/PulsarTransactionUtils.java       |  26 +--
 .../connector/pulsar/sink/PulsarSinkOptions.java   |  16 +-
 .../pulsar/sink/committer/PulsarCommitter.java     | 119 ++++++-------
 .../pulsar/sink/config/SinkConfiguration.java      |  14 +-
 .../connector/pulsar/sink/writer/PulsarWriter.java |   2 +-
 .../pulsar/sink/writer/message/PulsarMessage.java  |  47 ++++-
 .../sink/writer/message/PulsarMessageBuilder.java  |  45 +++--
 .../writer/serializer/PulsarSchemaWrapper.java     |  14 +-
 .../serializer/PulsarSerializationSchema.java      |   2 +
 .../PulsarSerializationSchemaWrapper.java          |   8 +-
 .../sink/writer/topic/TopicProducerRegister.java   | 189 ++++++++++++---------
 .../pulsar/sink/writer/PulsarWriterTest.java       |   5 +
 .../writer/topic/TopicProducerRegisterTest.java    |  34 +++-
 .../testutils/runtime/PulsarRuntimeOperator.java   |   4 +-
 20 files changed, 413 insertions(+), 214 deletions(-)

diff --git a/docs/content.zh/docs/connectors/datastream/pulsar.md b/docs/content.zh/docs/connectors/datastream/pulsar.md
index ad10d39..2f732bd 100644
--- a/docs/content.zh/docs/connectors/datastream/pulsar.md
+++ b/docs/content.zh/docs/connectors/datastream/pulsar.md
@@ -698,6 +698,8 @@ PulsarSink.builder().set_topics(["topic-a-partition-0", "topic-a-partition-2", "
   {{< /tab >}}
   {{< /tabs >}}
 
+#### Schema evolution
+
 同时使用 `PulsarSerializationSchema.pulsarSchema()` 以及在 builder 中指定 `PulsarSinkBuilder.enableSchemaEvolution()` 可以启用 [Schema evolution][schema-evolution] 特性。该特性会使用 Pulsar Broker 端提供的 Schema 版本兼容性检测以及 Schema 版本演进。下列示例展示了如何启用 Schema Evolution。
 
 ```java
@@ -717,6 +719,37 @@ PulsarSink<String> sink = PulsarSink.builder()
 例如,如果使用 `PulsarSerializationSchema.pulsarSchema(Schema.STRING)` 而不使用 `PulsarSinkBuilder.enableSchemaEvolution()`。那么在写入 Topic 中所记录的消息 Schema 将会是 `Schema.BYTES`。
 {{< /hint >}}
 
+#### PulsarMessage<byte[]> 类型的消息的校验
+
+Pulsar 的 topic 至少会包含一种 Schema,`Schema.BYTES` 是默认的 Schema 类型并常作为没有 Schema 的 topic 的 Schema 类型。使用 `Schema.BYTES` 发送消息将会跳过类型检测,这意味着使用 `SerializationSchema` 和没有启用 Schema evolution 的 `Schema` 所发送的消息并不安全。
+
+可以启用 `pulsar.sink.validateSinkMessageBytes` 选项来让链接器使用 Pulsar 提供的 `Schema.AUTO_PRODUCE_BYTES()` 发送消息。它会在发送字节数组消息时额外进行校验,与 topic 上最新版本的 Schema 进行对比,保证消息内容能正确。
+
+但是,并非所有的 Pulsar 的 Schema 都支持校验字符串,所以在默认情况下我们禁用了此选项。可以按需启用。
+
+#### 自定义序列化器
+
+可以通过继承 `PulsarSerializationSchema` 接口来实现自定义的序列化逻辑。接口需要返回一个类型为 `PulsarMessage` 的消息,此类型实例无法被直接创建,连接器提供了构造方法并定义了三种消息类型的构建。
+
+- 使用 Pulsar 的 `Scheme` 来构建消息,常用于你知道 topic 对应的 schema 是什么的时候。我们会检查你提供的 `Schema` 在 topic 上是否兼容。
+  ```java
+  PulsarMessage.builder(Schema<M> schema, M message)
+      ...
+      .build();
+  ```
+- 创建一个消息类型为字节数组的消息。默认情况下不进行 Schema 检查。
+  ```java
+  PulsarMessage.builder(byte[] bytes)
+      ...
+      .build();
+  ```
+- 创建一个消息体为空的墓碑消息。[墓碑][tombstone-data-store] 是一种特殊的消息,并在 Pulsar 中提供了支持。
+  ```java
+  PulsarMessage.builder()
+      ...
+      .build();
+  ```
+
 ### 消息路由策略
 
 在 Pulsar Sink 中,消息路由发生在于分区之间,而非上层 Topic。对于给定 Topic 的情况,路由算法会首先会查询出 Topic 之上所有的分区信息,并在这些分区上实现消息的路由。Pulsar Sink 默认提供 2 种路由策略的实现。
@@ -1063,3 +1096,4 @@ Pulsar 事务机制仍在积极发展中,当前版本并不稳定。 Pulsar 2.
 
 [schema-evolution]: https://pulsar.apache.org/docs/2.10.x/schema-evolution-compatibility/#schema-evolution
 [standard-metrics]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
+[tombstone-data-store]: https://en.wikipedia.org/wiki/Tombstone_(data_store)
\ No newline at end of file
diff --git a/docs/content/docs/connectors/datastream/pulsar.md b/docs/content/docs/connectors/datastream/pulsar.md
index 1d0f851..78bfb7a 100644
--- a/docs/content/docs/connectors/datastream/pulsar.md
+++ b/docs/content/docs/connectors/datastream/pulsar.md
@@ -800,6 +800,8 @@ you can use the predefined `PulsarSerializationSchema`. The Pulsar sink provides
   {{< /tab >}}
   {{< /tabs >}}
 
+#### Schema evolution
+
 [Schema evolution][schema-evolution] can be enabled by users using `PulsarSerializationSchema.pulsarSchema()` and
 `PulsarSinkBuilder.enableSchemaEvolution()`. This means that any broker schema validation is in place.
 
@@ -821,6 +823,46 @@ For example, if you set `PulsarSerializationSchema.pulsarSchema(Schema.STRING)`
 the schema stored in Pulsar topics is `Schema.BYTES`.
 {{< /hint >}}
 
+#### PulsarMessage<byte[]> validation
+
+Pulsar topic always has at least one schema. The `Schema.BYTES` is the default one for any topic without schema being set.
+But sending messages bytes with `Schema.BYTES` bypass the schema validate. So the message sent with `SerializationSchema`
+and `Schema` which doesn't enable the schema evolution may save the invalid messages in the topic.
+
+You can enable the `pulsar.sink.validateSinkMessageBytes` option to let the connector use the Pulsar's `Schema.AUTO_PRODUCE_BYTES()`
+which supports extra check for the message bytes before sending. It will query the latest schema in topic and use it to
+validate the message bytes.
+
+But some schemas in Pulsar don't support validation, so we disable this option by default. you should use it at your own risk.
+
+#### Custom serializer
+
+You can have your own serialization logic by implementing the `PulsarSerializationSchema` interface. The return type
+for this interface is `PulsarMessage` which you can't create it directly. Instead, we use builder method for creating
+three types of the Pulsar messages.
+
+- Create a message with a Pulsar `Scheme`. This is always when you know the schema in topic.
+  We will check if the given schema is compatible in the correspond topic.
+  ```java
+  PulsarMessage.builder(Schema<M> schema, M message)
+      ...
+      .build();
+  ```
+- Create the message without any Pulsar `Scheme`. The message type can only be the byte array.
+  It won't validate the message bytes by default.
+  ```java
+  PulsarMessage.builder(byte[] bytes)
+      ...
+      .build();
+  ```
+- Create a tombstone message with empty payloads. [Tombstone][tombstone-data-store] is a special message which is
+  supported in Pulsar.
+  ```java
+  PulsarMessage.builder()
+      ...
+      .build();
+  ```
+
 ### Message Routing
 
 Routing in Pulsar Sink is operated on the partition level. For a list of partitioned topics,
@@ -1225,3 +1267,4 @@ You can use the latest `pulsar-client-all` release to resolve this issue.
 
 [schema-evolution]: https://pulsar.apache.org/docs/2.10.x/schema-evolution-compatibility/#schema-evolution
 [standard-metrics]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
+[tombstone-data-store]: https://en.wikipedia.org/wiki/Tombstone_(data_store)
\ No newline at end of file
diff --git a/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html b/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html
index a40d3c0..fe5f66c 100644
--- a/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html
+++ b/docs/layouts/shortcodes/generated/pulsar_sink_configuration.html
@@ -18,7 +18,7 @@
             <td><h5>pulsar.sink.enableMetrics</h5></td>
             <td style="word-wrap: break-word;">true</td>
             <td>Boolean</td>
-            <td>The metrics from Pulsar Producer are only exposed if you enable this option.You should set the <code class="highlighter-rouge">pulsar.client.statsIntervalSeconds</code> to a positive value if you enable this option.</td>
+            <td>The metrics from Pulsar Producer are only exposed if you enable this option. You should set the <code class="highlighter-rouge">pulsar.client.statsIntervalSeconds</code> to a positive value if you enable this option.</td>
         </tr>
         <tr>
             <td><h5>pulsar.sink.enableSchemaEvolution</h5></td>
@@ -48,7 +48,13 @@
             <td><h5>pulsar.sink.transactionTimeoutMillis</h5></td>
             <td style="word-wrap: break-word;">10800000</td>
             <td>Long</td>
-            <td>This option is used when the user require the <code class="highlighter-rouge">DeliveryGuarantee.EXACTLY_ONCE</code> semantic.We would use transaction for making sure the message could be write only once.</td>
+            <td>This option is used when the user require the <code class="highlighter-rouge">DeliveryGuarantee.EXACTLY_ONCE</code> semantic. We would use transaction for making sure the message could be write only once.</td>
+        </tr>
+        <tr>
+            <td><h5>pulsar.sink.validateSinkMessageBytes</h5></td>
+            <td style="word-wrap: break-word;">false</td>
+            <td>Boolean</td>
+            <td>Pulsar client can validate the raw message bytes with the latest topic schema. This can make sure your serialized messages bytes is valid for consumer.</td>
         </tr>
     </tbody>
 </table>
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/DefaultPulsarCrypto.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/DefaultPulsarCrypto.java
index 97c84d2..39ca287 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/DefaultPulsarCrypto.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/DefaultPulsarCrypto.java
@@ -37,6 +37,7 @@ import static org.apache.flink.util.Preconditions.checkNotNull;
 /** The default implementation for {@link PulsarCrypto}. */
 @PublicEvolving
 public class DefaultPulsarCrypto implements PulsarCrypto {
+    private static final long serialVersionUID = 2246425411998723487L;
 
     private final CryptoKeyReader cryptoKeyReader;
     private final Set<String> encryptKeys;
@@ -93,7 +94,7 @@ public class DefaultPulsarCrypto implements PulsarCrypto {
         public DefaultPulsarCryptoBuilder messageCrypto(
                 SerializableSupplier<MessageCrypto<MessageMetadata, MessageMetadata>>
                         messageCryptoSupplier) {
-            this.messageCryptoSupplier = messageCryptoSupplier;
+            this.messageCryptoSupplier = checkNotNull(messageCryptoSupplier);
             return this;
         }
 
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/PulsarCryptoDisabled.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/PulsarCryptoDisabled.java
index 7f6902f..fb6c0ec 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/PulsarCryptoDisabled.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/crypto/PulsarCryptoDisabled.java
@@ -28,6 +28,7 @@ import java.util.Set;
 /** A default implementation, we will use it if the user didn't enable end-to-end encryption. */
 @Internal
 public class PulsarCryptoDisabled implements PulsarCrypto {
+    private static final long serialVersionUID = 7735076959011427862L;
 
     @Override
     public CryptoKeyReader cryptoKeyReader() {
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/schema/PulsarSchemaUtils.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/schema/PulsarSchemaUtils.java
index 00bf2cf..b346006 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/schema/PulsarSchemaUtils.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/schema/PulsarSchemaUtils.java
@@ -30,6 +30,8 @@ import org.apache.flink.connector.pulsar.common.schema.factories.StringSchemaFac
 
 import com.google.protobuf.Message;
 import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.impl.schema.AutoConsumeSchema;
+import org.apache.pulsar.client.impl.schema.AutoProduceBytesSchema;
 import org.apache.pulsar.client.impl.schema.BooleanSchema;
 import org.apache.pulsar.client.impl.schema.ByteSchema;
 import org.apache.pulsar.client.impl.schema.BytesSchema;
@@ -46,6 +48,7 @@ import org.apache.pulsar.client.impl.schema.SchemaInfoImpl;
 import org.apache.pulsar.client.impl.schema.ShortSchema;
 import org.apache.pulsar.client.impl.schema.TimeSchema;
 import org.apache.pulsar.client.impl.schema.TimestampSchema;
+import org.apache.pulsar.common.protocol.schema.SchemaHash;
 import org.apache.pulsar.common.schema.SchemaInfo;
 import org.apache.pulsar.common.schema.SchemaType;
 
@@ -204,4 +207,12 @@ public final class PulsarSchemaUtils {
             throw new IllegalStateException("Couldn't find the schema class " + className);
         }
     }
+
+    public static SchemaHash hash(Schema<?> schema) {
+        if (schema instanceof AutoProduceBytesSchema || schema instanceof AutoConsumeSchema) {
+            return SchemaHash.of(Schema.BYTES);
+        }
+
+        return SchemaHash.of(schema);
+    }
 }
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarTransactionUtils.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarTransactionUtils.java
index c8ffe63..fbb597d 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarTransactionUtils.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarTransactionUtils.java
@@ -23,14 +23,17 @@ import org.apache.flink.util.FlinkRuntimeException;
 
 import org.apache.pulsar.client.api.PulsarClient;
 import org.apache.pulsar.client.api.transaction.Transaction;
-import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientException;
+import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
 
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 
 import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyClient;
-import static org.apache.flink.util.ExceptionUtils.findThrowable;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientException.unwrap;
 
 /** A suit of workarounds for the Pulsar Transaction. */
 @Internal
@@ -53,18 +56,21 @@ public final class PulsarTransactionUtils {
             Thread.currentThread().interrupt();
             throw new IllegalStateException(e);
         } catch (ExecutionException e) {
-            throw new FlinkRuntimeException(e);
+            throw new FlinkRuntimeException(unwrap(e));
         }
     }
 
     /**
-     * This is a bug in original {@link TransactionCoordinatorClientException#unwrap(Throwable)}
-     * method. Pulsar wraps the {@link ExecutionException} which hides the real execution exception.
-     *
-     * <p>This bug should be fixed after the 2.10.0 release. We just keep this for safety.
+     * {@link PulsarClient} didn't expose the internal {@link TransactionCoordinatorClient} to the
+     * end user. But the connector needs it to manually commit/abort the transaction by {@link
+     * TxnID}.
      */
-    public static TransactionCoordinatorClientException unwrap(
-            TransactionCoordinatorClientException e) {
-        return findThrowable(e.getCause(), TransactionCoordinatorClientException.class).orElse(e);
+    public static TransactionCoordinatorClient getTcClient(PulsarClient pulsarClient) {
+        TransactionCoordinatorClient coordinatorClient =
+                ((PulsarClientImpl) pulsarClient).getTcClient();
+        // Ensure you have enabled transaction.
+        checkNotNull(coordinatorClient, "You haven't enable transaction in Pulsar client.");
+
+        return coordinatorClient;
     }
 }
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/PulsarSinkOptions.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/PulsarSinkOptions.java
index dbc1026..a5bb977 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/PulsarSinkOptions.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/PulsarSinkOptions.java
@@ -88,7 +88,7 @@ public final class PulsarSinkOptions {
                     .withDescription(
                             Description.builder()
                                     .text(
-                                            "This option is used when the user require the %s semantic.",
+                                            "This option is used when the user require the %s semantic. ",
                                             code("DeliveryGuarantee.EXACTLY_ONCE"))
                                     .text(
                                             "We would use transaction for making sure the message could be write only once.")
@@ -137,12 +137,24 @@ public final class PulsarSinkOptions {
                     .withDescription(
                             Description.builder()
                                     .text(
-                                            "The metrics from Pulsar Producer are only exposed if you enable this option.")
+                                            "The metrics from Pulsar Producer are only exposed if you enable this option. ")
                                     .text(
                                             "You should set the %s to a positive value if you enable this option.",
                                             code(PULSAR_STATS_INTERVAL_SECONDS.key()))
                                     .build());
 
+    public static final ConfigOption<Boolean> PULSAR_VALIDATE_SINK_MESSAGE_BYTES =
+            ConfigOptions.key(SINK_CONFIG_PREFIX + "validateSinkMessageBytes")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            Description.builder()
+                                    .text(
+                                            "Pulsar client can validate the raw message bytes with the latest topic schema. ")
+                                    .text(
+                                            "This can make sure your serialized messages bytes is valid for consumer.")
+                                    .build());
+
     ///////////////////////////////////////////////////////////////////////////////
     //
     // The configuration for ProducerConfigurationData part.
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/committer/PulsarCommitter.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/committer/PulsarCommitter.java
index 8389bdc..11e80c7 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/committer/PulsarCommitter.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/committer/PulsarCommitter.java
@@ -21,7 +21,6 @@ package org.apache.flink.connector.pulsar.sink.committer;
 import org.apache.flink.annotation.Internal;
 import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.connector.base.DeliveryGuarantee;
-import org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils;
 import org.apache.flink.connector.pulsar.sink.PulsarSink;
 import org.apache.flink.connector.pulsar.sink.config.SinkConfiguration;
 import org.apache.flink.util.FlinkRuntimeException;
@@ -34,7 +33,6 @@ import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientExce
 import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientException.MetaStoreHandlerNotExistsException;
 import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClientException.TransactionNotFoundException;
 import org.apache.pulsar.client.api.transaction.TxnID;
-import org.apache.pulsar.client.impl.PulsarClientImpl;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -43,6 +41,7 @@ import java.io.IOException;
 import java.util.Collection;
 
 import static org.apache.flink.connector.pulsar.common.config.PulsarClientFactory.createClient;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils.getTcClient;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 import static org.apache.pulsar.common.naming.TopicName.TRANSACTION_COORDINATOR_ASSIGN;
 
@@ -65,8 +64,8 @@ public class PulsarCommitter implements Committer<PulsarCommittable>, Closeable
     }
 
     @Override
-    public void commit(Collection<CommitRequest<PulsarCommittable>> requests)
-            throws IOException, InterruptedException {
+    @SuppressWarnings("java:S3776")
+    public void commit(Collection<CommitRequest<PulsarCommittable>> requests) {
         TransactionCoordinatorClient client = transactionCoordinatorClient();
 
         for (CommitRequest<PulsarCommittable> request : requests) {
@@ -77,66 +76,61 @@ public class PulsarCommitter implements Committer<PulsarCommittable>, Closeable
             LOG.debug("Start committing the Pulsar transaction {} for topic {}", txnID, topic);
             try {
                 client.commit(txnID);
-            } catch (TransactionCoordinatorClientException e) {
-                // This is a known bug for Pulsar Transaction.
-                // We have to use instanceof instead of catching them.
-                TransactionCoordinatorClientException ex = PulsarTransactionUtils.unwrap(e);
-                if (ex instanceof CoordinatorNotFoundException) {
-                    LOG.error(
-                            "We couldn't find the Transaction Coordinator from Pulsar broker {}. "
-                                    + "Check your broker configuration.",
-                            committable,
-                            ex);
-                    request.signalFailedWithKnownReason(ex);
-                } else if (ex instanceof InvalidTxnStatusException) {
+            } catch (CoordinatorNotFoundException e) {
+                LOG.error(
+                        "We couldn't find the Transaction Coordinator from Pulsar broker {}. "
+                                + "Check your broker configuration.",
+                        committable,
+                        e);
+                request.signalFailedWithKnownReason(e);
+            } catch (InvalidTxnStatusException e) {
+                LOG.error(
+                        "Unable to commit transaction ({}) because it's in an invalid state. "
+                                + "Most likely the transaction has been aborted for some reason. "
+                                + "Please check the Pulsar broker logs for more details.",
+                        committable,
+                        e);
+                request.signalAlreadyCommitted();
+            } catch (TransactionNotFoundException e) {
+                if (request.getNumberOfRetries() == 0) {
                     LOG.error(
-                            "Unable to commit transaction ({}) because it's in an invalid state. "
-                                    + "Most likely the transaction has been aborted for some reason. "
-                                    + "Please check the Pulsar broker logs for more details.",
+                            "Unable to commit transaction ({}) because it's not found on Pulsar broker. "
+                                    + "Most likely the checkpoint interval exceed the transaction timeout.",
                             committable,
-                            ex);
+                            e);
+                    request.signalFailedWithKnownReason(e);
+                } else {
+                    LOG.warn(
+                            "We can't find the transaction {} after {} retry committing. "
+                                    + "This may mean that the transaction have been committed in previous but failed with timeout. "
+                                    + "So we just mark it as committed.",
+                            txnID,
+                            request.getNumberOfRetries());
                     request.signalAlreadyCommitted();
-                } else if (ex instanceof TransactionNotFoundException) {
-                    if (request.getNumberOfRetries() == 0) {
-                        LOG.error(
-                                "Unable to commit transaction ({}) because it's not found on Pulsar broker. "
-                                        + "Most likely the checkpoint interval exceed the transaction timeout.",
-                                committable,
-                                ex);
-                        request.signalFailedWithKnownReason(ex);
-                    } else {
-                        LOG.warn(
-                                "We can't find the transaction {} after {} retry committing. "
-                                        + "This may mean that the transaction have been committed in previous but failed with timeout. "
-                                        + "So we just mark it as committed.",
-                                txnID,
-                                request.getNumberOfRetries());
-                        request.signalAlreadyCommitted();
-                    }
-                } else if (ex instanceof MetaStoreHandlerNotExistsException) {
-                    LOG.error(
-                            "We can't find the meta store handler by the mostSigBits from TxnID {}. "
-                                    + "Did you change the metadata for topic {}?",
-                            committable,
-                            TRANSACTION_COORDINATOR_ASSIGN,
-                            ex);
-                    request.signalFailedWithKnownReason(ex);
+                }
+            } catch (MetaStoreHandlerNotExistsException e) {
+                LOG.error(
+                        "We can't find the meta store handler by the mostSigBits from TxnID {}. "
+                                + "Did you change the metadata for topic {}?",
+                        committable,
+                        TRANSACTION_COORDINATOR_ASSIGN,
+                        e);
+                request.signalFailedWithKnownReason(e);
+            } catch (TransactionCoordinatorClientException e) {
+                LOG.error(
+                        "Encountered retriable exception while committing transaction {} for topic {}.",
+                        committable,
+                        topic,
+                        e);
+                int maxRecommitTimes = sinkConfiguration.getMaxRecommitTimes();
+                if (request.getNumberOfRetries() < maxRecommitTimes) {
+                    request.retryLater();
                 } else {
-                    LOG.error(
-                            "Encountered retriable exception while committing transaction {} for topic {}.",
-                            committable,
-                            topic,
-                            ex);
-                    int maxRecommitTimes = sinkConfiguration.getMaxRecommitTimes();
-                    if (request.getNumberOfRetries() < maxRecommitTimes) {
-                        request.retryLater();
-                    } else {
-                        String message =
-                                String.format(
-                                        "Failed to commit transaction %s after retrying %d times",
-                                        txnID, maxRecommitTimes);
-                        request.signalFailedWithKnownReason(new FlinkRuntimeException(message, ex));
-                    }
+                    String message =
+                            String.format(
+                                    "Failed to commit transaction %s after retrying %d times",
+                                    txnID, maxRecommitTimes);
+                    request.signalFailedWithKnownReason(new FlinkRuntimeException(message, e));
                 }
             } catch (Exception e) {
                 LOG.error(
@@ -156,10 +150,7 @@ public class PulsarCommitter implements Committer<PulsarCommittable>, Closeable
     private TransactionCoordinatorClient transactionCoordinatorClient() {
         if (coordinatorClient == null) {
             this.pulsarClient = createClient(sinkConfiguration);
-            this.coordinatorClient = ((PulsarClientImpl) pulsarClient).getTcClient();
-
-            // Ensure you have enabled transaction.
-            checkNotNull(coordinatorClient, "You haven't enable transaction in Pulsar client.");
+            this.coordinatorClient = getTcClient(pulsarClient);
         }
 
         return coordinatorClient;
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/config/SinkConfiguration.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/config/SinkConfiguration.java
index 0a56ff3..d041bde 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/config/SinkConfiguration.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/config/SinkConfiguration.java
@@ -37,6 +37,7 @@ import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_EN
 import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_MAX_RECOMMIT_TIMES;
 import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_MESSAGE_KEY_HASH;
 import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_TOPIC_METADATA_REFRESH_INTERVAL;
+import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_VALIDATE_SINK_MESSAGE_BYTES;
 import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_DELIVERY_GUARANTEE;
 import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_SCHEMA_EVOLUTION;
 import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_TRANSACTION_TIMEOUT;
@@ -54,6 +55,7 @@ public class SinkConfiguration extends PulsarConfiguration {
     private final boolean enableSchemaEvolution;
     private final int maxRecommitTimes;
     private final boolean enableMetrics;
+    private final boolean validateSinkMessageBytes;
 
     public SinkConfiguration(Configuration configuration) {
         super(configuration);
@@ -67,6 +69,7 @@ public class SinkConfiguration extends PulsarConfiguration {
         this.maxRecommitTimes = get(PULSAR_MAX_RECOMMIT_TIMES);
         this.enableMetrics =
                 get(PULSAR_ENABLE_SINK_METRICS) && get(PULSAR_STATS_INTERVAL_SECONDS) > 0;
+        this.validateSinkMessageBytes = get(PULSAR_VALIDATE_SINK_MESSAGE_BYTES);
     }
 
     /** The delivery guarantee changes the behavior of {@link PulsarWriter}. */
@@ -123,6 +126,11 @@ public class SinkConfiguration extends PulsarConfiguration {
         return enableMetrics;
     }
 
+    /** Whether to add extra schema check for byte messages. */
+    public boolean isValidateSinkMessageBytes() {
+        return validateSinkMessageBytes;
+    }
+
     @Override
     public boolean equals(Object o) {
         if (this == o) {
@@ -141,7 +149,8 @@ public class SinkConfiguration extends PulsarConfiguration {
                 && enableSchemaEvolution == that.enableSchemaEvolution
                 && messageKeyHash == that.messageKeyHash
                 && maxRecommitTimes == that.maxRecommitTimes
-                && enableMetrics == that.enableMetrics;
+                && enableMetrics == that.enableMetrics
+                && validateSinkMessageBytes == that.validateSinkMessageBytes;
     }
 
     @Override
@@ -154,6 +163,7 @@ public class SinkConfiguration extends PulsarConfiguration {
                 messageKeyHash,
                 enableSchemaEvolution,
                 maxRecommitTimes,
-                enableMetrics);
+                enableMetrics,
+                validateSinkMessageBytes);
     }
 }
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriter.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriter.java
index 0d3009c..f5a4bb7 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriter.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriter.java
@@ -141,7 +141,7 @@ public class PulsarWriter<IN> implements PrecommittingSinkWriter<IN, PulsarCommi
         List<String> availableTopics = metadataListener.availableTopics();
         String topic = topicRouter.route(element, key, availableTopics, sinkContext);
 
-        // Create message builder for sending message.
+        // Create message builder for sending messages.
         TypedMessageBuilder<?> builder = createMessageBuilder(topic, context, message);
 
         // Message Delay delivery.
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessage.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessage.java
index 0c45763..dc01908 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessage.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessage.java
@@ -28,6 +28,8 @@ import javax.annotation.Nullable;
 import java.util.List;
 import java.util.Map;
 
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
 /**
  * The message instance would be used for {@link TypedMessageBuilder}. We create this class because
  * the Pulsar lacks such kind of POJO class.
@@ -38,19 +40,25 @@ public class PulsarMessage<T> {
     @Nullable private final byte[] orderingKey;
     @Nullable private final String key;
     private final long eventTime;
-    private final Schema<T> schema;
+    @Nullable private final Schema<T> schema;
     @Nullable private final T value;
     @Nullable private final Map<String, String> properties;
     @Nullable private final Long sequenceId;
     @Nullable private final List<String> replicationClusters;
     private final boolean disableReplication;
 
-    /** Package private for building this class only in {@link PulsarMessageBuilder}. */
+    /**
+     * Package private for building this class only in {@link PulsarMessageBuilder}. Use the {@link
+     * #builder(byte[])}, {@link #builder(Schema, Object)} methods for creating the builder. And
+     * {@link #builder()} is only used for creating a special tombstone message, use it as you
+     * needs.
+     */
+    @SuppressWarnings("java:S107")
     PulsarMessage(
             @Nullable byte[] orderingKey,
             @Nullable String key,
             long eventTime,
-            Schema<T> schema,
+            @Nullable Schema<T> schema,
             @Nullable T value,
             @Nullable Map<String, String> properties,
             @Nullable Long sequenceId,
@@ -67,6 +75,38 @@ public class PulsarMessage<T> {
         this.disableReplication = disableReplication;
     }
 
+    /**
+     * Method wrapper of {@link TypedMessageBuilder#value(Object)}. You can pass any schema for
+     * validating it on Pulsar. This is called schema evolution. But the topic on Pulsar should bind
+     * to a fixed {@link Schema}. You may not have multiple schemas on the same topic unless it's
+     * compatible with each other. This is determined by the configured <a
+     * href="https://pulsar.apache.org/docs/2.10.x/schema-evolution-compatibility/#schema-compatibility-check-strategy">schema
+     * evolution policy</a> on corresponding topic.
+     */
+    public static <M> PulsarMessageBuilder<M> builder(Schema<M> schema, M message) {
+        checkNotNull(schema, "Schema should be provided.");
+        checkNotNull(message, "Message should be provided.");
+        return new PulsarMessageBuilder<>(schema, message);
+    }
+
+    /**
+     * Create a message with a pre-serialized byte array. This message can be sent to any topic with
+     * any schema. We will just bypass the Pulsar's schema evolution check. But if the target topic
+     * does not exist on Pulsar. The auto-created topic won't have any schema in this way.
+     */
+    public static PulsarMessageBuilder<byte[]> builder(byte[] bytes) {
+        checkNotNull(bytes, "Message bytes should be provided.");
+        return new PulsarMessageBuilder<>(Schema.BYTES, bytes);
+    }
+
+    /**
+     * Create a tombstone message with empty payloads. It will be skipped and considered deleted
+     * (akin to the concept of tombstones in key-value databases).
+     */
+    public static PulsarMessageBuilder<byte[]> builder() {
+        return new PulsarMessageBuilder<>(null, null);
+    }
+
     @Nullable
     public byte[] getOrderingKey() {
         return orderingKey;
@@ -81,6 +121,7 @@ public class PulsarMessage<T> {
         return eventTime;
     }
 
+    @Nullable
     public Schema<T> getSchema() {
         return schema;
     }
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessageBuilder.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessageBuilder.java
index 5185932..95fb5cb 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessageBuilder.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/message/PulsarMessageBuilder.java
@@ -24,10 +24,13 @@ import org.apache.flink.connector.pulsar.sink.writer.router.KeyHashTopicRouter;
 import org.apache.pulsar.client.api.Schema;
 import org.apache.pulsar.client.api.TypedMessageBuilder;
 
+import javax.annotation.Nullable;
+
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 
+import static org.apache.flink.util.Preconditions.checkArgument;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /** {@link TypedMessageBuilder} wrapper for providing the required method for end-users. */
@@ -37,13 +40,22 @@ public class PulsarMessageBuilder<T> {
     private byte[] orderingKey;
     private String key;
     private long eventTime;
-    Schema<T> schema;
-    private T value;
-    private Map<String, String> properties = new HashMap<>();
+    private final Schema<T> schema;
+    private final T value;
+    private final Map<String, String> properties = new HashMap<>();
     private Long sequenceId;
     private List<String> replicationClusters;
     private boolean disableReplication = false;
 
+    /**
+     * Make this constructor package private for in favor of the {@link PulsarMessage#builder()}
+     * method in {@link PulsarMessage}.
+     */
+    PulsarMessageBuilder(Schema<T> schema, @Nullable T value) {
+        this.schema = schema;
+        this.value = value;
+    }
+
     /** Method wrapper of {@link TypedMessageBuilder#orderingKey(byte[])}. */
     public PulsarMessageBuilder<T> orderingKey(byte[] orderingKey) {
         this.orderingKey = checkNotNull(orderingKey);
@@ -51,32 +63,21 @@ public class PulsarMessageBuilder<T> {
     }
 
     /**
-     * Property {@link TypedMessageBuilder#key(String)}. This property would also be used in {@link
-     * KeyHashTopicRouter}.
+     * Method wrapper of {@link TypedMessageBuilder#key(String)}. This key would also be used in
+     * {@link KeyHashTopicRouter}.
      */
     public PulsarMessageBuilder<T> key(String key) {
         this.key = checkNotNull(key);
         return this;
     }
 
-    /** Method wrapper of {@link TypedMessageBuilder#eventTime(long)}. */
-    public PulsarMessageBuilder<T> eventTime(long eventTime) {
-        this.eventTime = eventTime;
-        return this;
-    }
-
     /**
-     * Method wrapper of {@link TypedMessageBuilder#value(Object)}. You can pass any schema for
-     * validating it on Pulsar. This is called schema evolution. But the topic on Pulsar should bind
-     * to a fixed {@link Schema}. You can't have multiple schemas on the same topic unless it's
-     * compatible with each other.
-     *
-     * @param value The value could be null, which is called tombstones message in Pulsar. (It will
-     *     be skipped and considered deleted.)
+     * Method wrapper of {@link TypedMessageBuilder#eventTime(long)}. If you don't provide the event
+     * time, we will try to use Flink's sink context time instead.
      */
-    public PulsarMessageBuilder<T> value(Schema<T> schema, T value) {
-        this.schema = checkNotNull(schema);
-        this.value = value;
+    public PulsarMessageBuilder<T> eventTime(long eventTime) {
+        checkArgument(eventTime > 0, "The given event time should above 0.");
+        this.eventTime = eventTime;
         return this;
     }
 
@@ -111,8 +112,6 @@ public class PulsarMessageBuilder<T> {
     }
 
     public PulsarMessage<T> build() {
-        checkNotNull(schema, "Schema should be provided.");
-
         return new PulsarMessage<>(
                 orderingKey,
                 key,
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSchemaWrapper.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSchemaWrapper.java
index 0d508e5..7c6aef9 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSchemaWrapper.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSchemaWrapper.java
@@ -22,7 +22,6 @@ import org.apache.flink.annotation.Internal;
 import org.apache.flink.connector.pulsar.common.schema.PulsarSchema;
 import org.apache.flink.connector.pulsar.sink.writer.context.PulsarSinkContext;
 import org.apache.flink.connector.pulsar.sink.writer.message.PulsarMessage;
-import org.apache.flink.connector.pulsar.sink.writer.message.PulsarMessageBuilder;
 
 import org.apache.pulsar.client.api.Schema;
 import org.apache.pulsar.common.schema.KeyValue;
@@ -56,18 +55,11 @@ public class PulsarSchemaWrapper<IN> implements PulsarSerializationSchema<IN> {
 
     @Override
     public PulsarMessage<?> serialize(IN element, PulsarSinkContext sinkContext) {
-        Schema<IN> schema = this.pulsarSchema.getPulsarSchema();
+        Schema<IN> schema = pulsarSchema.getPulsarSchema();
         if (sinkContext.isEnableSchemaEvolution()) {
-            PulsarMessageBuilder<IN> builder = new PulsarMessageBuilder<>();
-            builder.value(schema, element);
-
-            return builder.build();
+            return PulsarMessage.builder(schema, element).build();
         } else {
-            PulsarMessageBuilder<byte[]> builder = new PulsarMessageBuilder<>();
-            byte[] bytes = schema.encode(element);
-            builder.value(Schema.BYTES, bytes);
-
-            return builder.build();
+            return PulsarMessage.builder(schema.encode(element)).build();
         }
     }
 }
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchema.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchema.java
index 45e3279..a61f7b8 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchema.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchema.java
@@ -53,6 +53,7 @@ public interface PulsarSerializationSchema<IN> extends Serializable {
      * @param sinkConfiguration All the configure options for the Pulsar sink. You can add custom
      *     options.
      */
+    @SuppressWarnings("java:S112")
     default void open(
             InitializationContext initializationContext,
             PulsarSinkContext sinkContext,
@@ -70,6 +71,7 @@ public interface PulsarSerializationSchema<IN> extends Serializable {
      * @param element Element to be serialized.
      * @param sinkContext Context to provide extra information.
      */
+    @SuppressWarnings("java:S1452")
     PulsarMessage<?> serialize(IN element, PulsarSinkContext sinkContext);
 
     /**
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchemaWrapper.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchemaWrapper.java
index 716d2db..2f24f94 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchemaWrapper.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/serializer/PulsarSerializationSchemaWrapper.java
@@ -24,9 +24,6 @@ import org.apache.flink.api.common.serialization.SerializationSchema.Initializat
 import org.apache.flink.connector.pulsar.sink.config.SinkConfiguration;
 import org.apache.flink.connector.pulsar.sink.writer.context.PulsarSinkContext;
 import org.apache.flink.connector.pulsar.sink.writer.message.PulsarMessage;
-import org.apache.flink.connector.pulsar.sink.writer.message.PulsarMessageBuilder;
-
-import org.apache.pulsar.client.api.Schema;
 
 /** Wrap the Flink's SerializationSchema into PulsarSerializationSchema. */
 @Internal
@@ -50,10 +47,7 @@ public class PulsarSerializationSchemaWrapper<IN> implements PulsarSerialization
 
     @Override
     public PulsarMessage<?> serialize(IN element, PulsarSinkContext sinkContext) {
-        PulsarMessageBuilder<byte[]> builder = new PulsarMessageBuilder<>();
         byte[] value = serializationSchema.serialize(element);
-        builder.value(Schema.BYTES, value);
-
-        return builder.build();
+        return PulsarMessage.builder(value).build();
     }
 }
diff --git a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegister.java b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegister.java
index ae99527..3b32467 100644
--- a/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegister.java
+++ b/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegister.java
@@ -22,6 +22,7 @@ import org.apache.flink.annotation.Internal;
 import org.apache.flink.connector.base.DeliveryGuarantee;
 import org.apache.flink.connector.pulsar.common.crypto.PulsarCrypto;
 import org.apache.flink.connector.pulsar.common.metrics.ProducerMetricsInterceptor;
+import org.apache.flink.connector.pulsar.common.schema.PulsarSchemaUtils;
 import org.apache.flink.connector.pulsar.sink.committer.PulsarCommittable;
 import org.apache.flink.connector.pulsar.sink.config.SinkConfiguration;
 import org.apache.flink.metrics.MetricGroup;
@@ -41,17 +42,21 @@ import org.apache.pulsar.client.api.TypedMessageBuilder;
 import org.apache.pulsar.client.api.transaction.Transaction;
 import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient;
 import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.ProducerBase;
 import org.apache.pulsar.client.impl.ProducerBuilderImpl;
-import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.client.impl.TypedMessageBuilderImpl;
 import org.apache.pulsar.client.impl.conf.ProducerConfigurationData;
+import org.apache.pulsar.client.impl.transaction.TransactionImpl;
 import org.apache.pulsar.common.api.proto.MessageMetadata;
-import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.protocol.schema.SchemaHash;
+import org.apache.pulsar.common.schema.SchemaType;
 import org.apache.pulsar.shade.com.google.common.base.Strings;
 
+import javax.annotation.Nullable;
+
 import java.io.Closeable;
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Collection;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
@@ -79,23 +84,24 @@ import static org.apache.flink.connector.pulsar.common.metrics.MetricNames.TOTAL
 import static org.apache.flink.connector.pulsar.common.metrics.MetricNames.TOTAL_SEND_FAILED;
 import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyClient;
 import static org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils.createTransaction;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils.getTcClient;
 import static org.apache.flink.connector.pulsar.sink.config.PulsarSinkConfigUtils.createProducerBuilder;
-import static org.apache.flink.util.Preconditions.checkArgument;
-import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
- * All the Pulsar Producers share the same Client, but self hold the queue for a specified topic. So
+ * All the Pulsar Producers share the same Client, but self-hold the queue for a specified topic. So
  * we have to create different instances for different topics.
  */
 @Internal
 public class TopicProducerRegister implements Closeable {
 
     private final PulsarClient pulsarClient;
+    @Nullable private final TransactionCoordinatorClient coordinatorClient;
     private final SinkConfiguration sinkConfiguration;
     private final PulsarCrypto pulsarCrypto;
     private final SinkWriterMetricGroup metricGroup;
-    private final Map<String, Map<SchemaInfo, Producer<?>>> producerRegister;
-    private final Map<String, Transaction> transactionRegister;
+    private final Map<String, Schema<byte[]>> schemas;
+    private final Map<String, Map<SchemaHash, Producer<?>>> producers;
+    private final Map<String, Transaction> transactions;
 
     public TopicProducerRegister(
             SinkConfiguration sinkConfiguration,
@@ -105,30 +111,46 @@ public class TopicProducerRegister implements Closeable {
         this.sinkConfiguration = sinkConfiguration;
         this.pulsarCrypto = pulsarCrypto;
         this.metricGroup = metricGroup;
-        this.producerRegister = new HashMap<>();
-        this.transactionRegister = new HashMap<>();
+        this.schemas = new HashMap<>();
+        this.producers = new HashMap<>();
+        this.transactions = new HashMap<>();
 
         if (sinkConfiguration.isEnableMetrics()) {
             metricGroup.setCurrentSendTimeGauge(this::currentSendTimeGauge);
         }
+
+        // Check if we have enabled the transaction in the exactly-once delivery guarantee.
+        if (sinkConfiguration.getDeliveryGuarantee() == DeliveryGuarantee.EXACTLY_ONCE) {
+            this.coordinatorClient = getTcClient(pulsarClient);
+        } else {
+            this.coordinatorClient = null;
+        }
     }
 
     /**
      * Create a TypedMessageBuilder which could be sent to Pulsar directly. First, we would create a
-     * topic-related producer or use a cached instead. Then we would try to find a topic-related
+     * topic-related producer or use a cached one instead. Then we would try to find a topic-related
      * transaction. We would generate a transaction instance if there is no transaction. Finally, we
-     * create the message builder and put the element into it.
+     * created the message builder and put the element into it.
+     *
+     * <p>Pulsar's Producer doesn't have {@code producer.newMessage(schema, transaction)} method. We
+     * have to manually create it.
      */
-    public <T> TypedMessageBuilder<T> createMessageBuilder(String topic, Schema<T> schema) {
-        Producer<T> producer = getOrCreateProducer(topic, schema);
-        DeliveryGuarantee deliveryGuarantee = sinkConfiguration.getDeliveryGuarantee();
+    @SuppressWarnings("unchecked")
+    public <T> TypedMessageBuilder<T> createMessageBuilder(
+            String topic, @Nullable Schema<?> schema) {
+        if (schema == null || schema.getSchemaInfo().getType() == SchemaType.BYTES) {
+            schema = getBytesSchema(topic);
+        }
+        ProducerBase<?> producer = (ProducerBase<?>) getOrCreateProducer(topic, schema);
+        TransactionImpl transaction = null;
 
-        if (deliveryGuarantee == DeliveryGuarantee.EXACTLY_ONCE) {
-            Transaction transaction = getOrCreateTransaction(topic);
-            return producer.newMessage(transaction);
-        } else {
-            return producer.newMessage();
+        if (sinkConfiguration.getDeliveryGuarantee() == DeliveryGuarantee.EXACTLY_ONCE) {
+            transaction = (TransactionImpl) getOrCreateTransaction(topic);
         }
+
+        return (TypedMessageBuilder<T>)
+                new TypedMessageBuilderImpl<>(producer, schema, transaction);
     }
 
     /**
@@ -136,15 +158,16 @@ public class TopicProducerRegister implements Closeable {
      * be removed until Flink triggered a checkpoint.
      */
     public List<PulsarCommittable> prepareCommit() {
-        List<PulsarCommittable> committables = new ArrayList<>(transactionRegister.size());
-        transactionRegister.forEach(
-                (topic, transaction) -> {
-                    TxnID txnID = transaction.getTxnID();
-                    PulsarCommittable committable = new PulsarCommittable(txnID, topic);
-                    committables.add(committable);
-                });
+        List<PulsarCommittable> committables = new ArrayList<>(transactions.size());
+        for (Map.Entry<String, Transaction> entry : transactions.entrySet()) {
+            String topic = entry.getKey();
+            Transaction transaction = entry.getValue();
+            TxnID txnID = transaction.getTxnID();
+
+            committables.add(new PulsarCommittable(txnID, topic));
+        }
+        transactions.clear();
 
-        clearTransactions();
         return committables;
     }
 
@@ -153,9 +176,8 @@ public class TopicProducerRegister implements Closeable {
      * successfully persisted.
      */
     public void flush() throws IOException {
-        Collection<Map<SchemaInfo, Producer<?>>> collection = producerRegister.values();
-        for (Map<SchemaInfo, Producer<?>> producers : collection) {
-            for (Producer<?> producer : producers.values()) {
+        for (Map<SchemaHash, Producer<?>> set : producers.values()) {
+            for (Producer<?> producer : set.values()) {
                 producer.flush();
             }
         }
@@ -171,9 +193,9 @@ public class TopicProducerRegister implements Closeable {
             closer.register(this::abortTransactions);
 
             // Remove all the producers.
-            closer.register(producerRegister::clear);
+            closer.register(producers::clear);
 
-            // All the producers would be closed by this method.
+            // This method would close all the producers.
             // We would block until all the producers have been successfully closed.
             closer.register(pulsarClient);
         }
@@ -182,39 +204,16 @@ public class TopicProducerRegister implements Closeable {
     /** Create or return the cached topic-related producer. */
     @SuppressWarnings("unchecked")
     private <T> Producer<T> getOrCreateProducer(String topic, Schema<T> schema) {
-        Map<SchemaInfo, Producer<?>> producers =
-                producerRegister.computeIfAbsent(topic, key -> new HashMap<>());
-        SchemaInfo schemaInfo = schema.getSchemaInfo();
-
-        if (producers.containsKey(schemaInfo)) {
-            return (Producer<T>) producers.get(schemaInfo);
+        Map<SchemaHash, Producer<?>> set = producers.computeIfAbsent(topic, t -> new HashMap<>());
+        SchemaHash hash = PulsarSchemaUtils.hash(schema);
+        if (set.containsKey(hash)) {
+            return (Producer<T>) set.get(hash);
         }
 
         ProducerBuilder<T> builder = createProducerBuilder(pulsarClient, schema, sinkConfiguration);
 
-        // Set the message crypto key reader.
-        CryptoKeyReader cryptoKeyReader = pulsarCrypto.cryptoKeyReader();
-        if (cryptoKeyReader != null) {
-            builder.cryptoKeyReader(cryptoKeyReader);
-
-            // Set the encrypt keys.
-            Set<String> encryptKeys = pulsarCrypto.encryptKeys();
-            checkArgument(
-                    encryptKeys != null && !encryptKeys.isEmpty(),
-                    "You should provide encryptKeys in PulsarCrypto");
-            encryptKeys.forEach(builder::addEncryptionKey);
-
-            // Set the message crypto if provided.
-            // Pulsar forgets to expose the config in producer builder.
-            // See issue https://github.com/apache/pulsar/issues/19139
-            MessageCrypto<MessageMetadata, MessageMetadata> messageCrypto =
-                    pulsarCrypto.messageCrypto();
-            if (messageCrypto != null) {
-                ProducerConfigurationData producerConfig =
-                        ((ProducerBuilderImpl<?>) builder).getConf();
-                producerConfig.setMessageCrypto(messageCrypto);
-            }
-        }
+        // Enable end-to-end encryption if provided.
+        configPulsarCrypto(builder);
 
         // Set the required topic name.
         builder.topic(topic);
@@ -225,16 +224,43 @@ public class TopicProducerRegister implements Closeable {
 
         // Expose the stats for calculating and monitoring.
         exposeProducerMetrics(producer);
-        producers.put(schemaInfo, producer);
+        set.put(hash, producer);
 
         return producer;
     }
 
+    private void configPulsarCrypto(ProducerBuilder<?> builder) {
+        CryptoKeyReader cryptoKeyReader = pulsarCrypto.cryptoKeyReader();
+        if (cryptoKeyReader == null) {
+            return;
+        }
+
+        // Set the message crypto key reader.
+        builder.cryptoKeyReader(cryptoKeyReader);
+
+        // Set the encrypt keys.
+        Set<String> encryptKeys = pulsarCrypto.encryptKeys();
+        if (encryptKeys == null || encryptKeys.isEmpty()) {
+            throw new IllegalArgumentException("You should provide encryptKeys in PulsarCrypto");
+        }
+        encryptKeys.forEach(builder::addEncryptionKey);
+
+        // Set the message crypto if provided.
+        // Pulsar forgets to expose the config in producer builder.
+        // See issue https://github.com/apache/pulsar/issues/19139
+        MessageCrypto<MessageMetadata, MessageMetadata> messageCrypto =
+                pulsarCrypto.messageCrypto();
+        if (messageCrypto != null) {
+            ProducerConfigurationData producerConfig = ((ProducerBuilderImpl<?>) builder).getConf();
+            producerConfig.setMessageCrypto(messageCrypto);
+        }
+    }
+
     /**
      * Get the cached topic-related transaction. Or create a new transaction after checkpointing.
      */
     private Transaction getOrCreateTransaction(String topic) {
-        return transactionRegister.computeIfAbsent(
+        return transactions.computeIfAbsent(
                 topic,
                 t -> {
                     long timeoutMillis = sinkConfiguration.getTransactionTimeoutMillis();
@@ -242,41 +268,40 @@ public class TopicProducerRegister implements Closeable {
                 });
     }
 
+    /**
+     * {@link Schema#AUTO_PRODUCE_BYTES} is used for extra validation. But it should be initialized
+     * with extra info in the Pulsar client. So it can't be reused and will be cached here.
+     */
+    private Schema<byte[]> getBytesSchema(String topic) {
+        if (sinkConfiguration.isValidateSinkMessageBytes()) {
+            return schemas.computeIfAbsent(topic, t -> Schema.AUTO_PRODUCE_BYTES());
+        } else {
+            return Schema.BYTES;
+        }
+    }
+
     /** Abort the existed transactions. This method would be used when closing PulsarWriter. */
     private void abortTransactions() {
-        if (transactionRegister.isEmpty()) {
+        if (coordinatorClient == null || transactions.isEmpty()) {
             return;
         }
 
-        TransactionCoordinatorClient coordinatorClient =
-                ((PulsarClientImpl) pulsarClient).getTcClient();
-        // This null check is used for making sure transaction is enabled in client.
-        checkNotNull(coordinatorClient);
-
         try (Closer closer = Closer.create()) {
-            for (Transaction transaction : transactionRegister.values()) {
+            for (Transaction transaction : transactions.values()) {
                 TxnID txnID = transaction.getTxnID();
                 closer.register(() -> coordinatorClient.abort(txnID));
             }
 
-            clearTransactions();
+            transactions.clear();
         } catch (IOException e) {
             throw new FlinkRuntimeException(e);
         }
     }
 
-    /**
-     * Clean these transactions. All transactions should be passed to Pulsar committer, we would
-     * create new transaction when new message comes.
-     */
-    private void clearTransactions() {
-        transactionRegister.clear();
-    }
-
     private Long currentSendTimeGauge() {
         double sendTime =
-                producerRegister.values().stream()
-                        .flatMap(v -> v.values().stream())
+                producers.values().stream()
+                        .flatMap(set -> set.values().stream())
                         .map(Producer::getStats)
                         .mapToDouble(ProducerStats::getSendLatencyMillis50pct)
                         .average()
diff --git a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriterTest.java b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriterTest.java
index 9b949eb..d7a64fa 100644
--- a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriterTest.java
+++ b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriterTest.java
@@ -161,6 +161,11 @@ class PulsarWriterTest extends PulsarTestSuiteBase {
             return 1;
         }
 
+        @Override
+        public int getAttemptNumber() {
+            return 0;
+        }
+
         @Override
         public SinkWriterMetricGroup metricGroup() {
             return metricGroup;
diff --git a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegisterTest.java b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegisterTest.java
index be4b964..00deda7 100644
--- a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegisterTest.java
+++ b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegisterTest.java
@@ -18,6 +18,7 @@
 
 package org.apache.flink.connector.pulsar.sink.writer.topic;
 
+import org.apache.flink.configuration.Configuration;
 import org.apache.flink.connector.base.DeliveryGuarantee;
 import org.apache.flink.connector.pulsar.common.crypto.PulsarCrypto;
 import org.apache.flink.connector.pulsar.sink.committer.PulsarCommittable;
@@ -26,19 +27,25 @@ import org.apache.flink.connector.pulsar.testutils.PulsarTestSuiteBase;
 
 import org.apache.pulsar.client.api.Message;
 import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.TypedMessageBuilder;
 import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient;
 import org.apache.pulsar.client.api.transaction.TxnID;
+import org.junit.jupiter.api.Test;
 import org.junit.jupiter.params.ParameterizedTest;
 import org.junit.jupiter.params.provider.EnumSource;
 
 import java.io.IOException;
 import java.util.List;
+import java.util.concurrent.ThreadLocalRandom;
 
 import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
 import static org.apache.flink.connector.base.DeliveryGuarantee.EXACTLY_ONCE;
+import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_VALIDATE_SINK_MESSAGE_BYTES;
 import static org.apache.flink.metrics.groups.UnregisteredMetricsGroup.createSinkWriterMetricGroup;
 import static org.assertj.core.api.Assertions.assertThat;
 import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 
 /** Unit tests for {@link TopicProducerRegister}. */
 class TopicProducerRegisterTest extends PulsarTestSuiteBase {
@@ -50,7 +57,8 @@ class TopicProducerRegisterTest extends PulsarTestSuiteBase {
         String topic = randomAlphabetic(10);
         operator().createTopic(topic, 8);
 
-        SinkConfiguration configuration = sinkConfiguration(deliveryGuarantee);
+        SinkConfiguration configuration =
+                new SinkConfiguration(operator().sinkConfig(deliveryGuarantee));
         TopicProducerRegister register =
                 new TopicProducerRegister(
                         configuration, PulsarCrypto.disabled(), createSinkWriterMetricGroup());
@@ -79,7 +87,8 @@ class TopicProducerRegisterTest extends PulsarTestSuiteBase {
         String topic = randomAlphabetic(10);
         operator().createTopic(topic, 8);
 
-        SinkConfiguration configuration = sinkConfiguration(deliveryGuarantee);
+        SinkConfiguration configuration =
+                new SinkConfiguration(operator().sinkConfig(deliveryGuarantee));
         TopicProducerRegister register =
                 new TopicProducerRegister(
                         configuration, PulsarCrypto.disabled(), createSinkWriterMetricGroup());
@@ -91,7 +100,24 @@ class TopicProducerRegisterTest extends PulsarTestSuiteBase {
         assertThat(committables).isEmpty();
     }
 
-    private SinkConfiguration sinkConfiguration(DeliveryGuarantee deliveryGuarantee) {
-        return new SinkConfiguration(operator().sinkConfig(deliveryGuarantee));
+    @Test
+    void sendMessageBytesWithWrongSchemaAndEnableCheck() throws Exception {
+        String topic = randomAlphabetic(10);
+        operator().createTopic(topic, 8);
+        operator().admin().schemas().createSchema(topic, Schema.INT16.getSchemaInfo());
+
+        Configuration configuration = operator().sinkConfig(DeliveryGuarantee.AT_LEAST_ONCE);
+        configuration.set(PULSAR_VALIDATE_SINK_MESSAGE_BYTES, true);
+        SinkConfiguration sinkConfiguration = new SinkConfiguration(configuration);
+        TopicProducerRegister register =
+                new TopicProducerRegister(
+                        sinkConfiguration, PulsarCrypto.disabled(), createSinkWriterMetricGroup());
+
+        long message = ThreadLocalRandom.current().nextLong();
+        TypedMessageBuilder<byte[]> builder = register.createMessageBuilder(topic, Schema.BYTES);
+
+        assertThrows(
+                SchemaSerializationException.class,
+                () -> builder.value(Schema.INT64.encode(message)));
     }
 }
diff --git a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntimeOperator.java b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntimeOperator.java
index c76bb2f..f975cbe 100644
--- a/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntimeOperator.java
+++ b/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntimeOperator.java
@@ -42,7 +42,6 @@ import org.apache.pulsar.client.api.Schema;
 import org.apache.pulsar.client.api.TypedMessageBuilder;
 import org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient;
 import org.apache.pulsar.client.api.transaction.TxnID;
-import org.apache.pulsar.client.impl.PulsarClientImpl;
 import org.apache.pulsar.common.naming.TopicName;
 import org.apache.pulsar.common.partition.PartitionedTopicMetadata;
 
@@ -68,6 +67,7 @@ import static org.apache.flink.connector.pulsar.common.config.PulsarOptions.PULS
 import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyAdmin;
 import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyClient;
 import static org.apache.flink.connector.pulsar.common.utils.PulsarExceptionUtils.sneakyThrow;
+import static org.apache.flink.connector.pulsar.common.utils.PulsarTransactionUtils.getTcClient;
 import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_SEND_TIMEOUT_MS;
 import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_DELIVERY_GUARANTEE;
 import static org.apache.flink.connector.pulsar.sink.PulsarSinkOptions.PULSAR_WRITE_TRANSACTION_TIMEOUT;
@@ -396,7 +396,7 @@ public class PulsarRuntimeOperator implements Closeable {
 
     /** Return the transaction coordinator client for operating {@link TxnID}. */
     public TransactionCoordinatorClient coordinatorClient() {
-        return ((PulsarClientImpl) client()).getTcClient();
+        return getTcClient(client());
     }
 
     /**