You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by pe...@apache.org on 2021/12/30 03:56:08 UTC

[pulsar] branch branch-2.8 updated (1a32edc -> 3644311)

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a change to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git.


    from 1a32edc  Fixes the NPE in system topics policies service (#13469)
     new 7aac8b8  [Issue 9888] add python3.9 on manylinux2014 build support (#10954)
     new 4f876678 Add log error tracking for semaphore count leak (#12410)
     new 1a1c886  Apply clang-format check for python wrapper (#13418)
     new b763f83  [Producer] Change the time units from ns to ms (#13057)
     new 25c3ae3  Fix dead loop in BacklogQuotaManager.dropBacklogForTimeLimit (#13194) (#13249)
     new ef9e773  Fix semaphore and memory leak when chunks failed to enqueue (#13454)
     new c9c48d2  Fix getting last message id from empty compact ledger (#13476)
     new cac6942  fix: bug when allAll bucket (#13467)
     new 77c66b3  Fix reousrce leak when create producer failed (#13505)
     new 2a73cdf  Use PulsarByteBufAllocator to allocate buffer for chunks at consumer side (#13536)
     new 078c8db  [Pulsar SQL] support protobuf/timestamp (#13287)
     new bdfa14d  Return message ID from compacted ledger while the compaction cursor reach the end of the topic (#13533)
     new 3644311  [Security] Upgrade Log4j to 2.17.1 (#13552)

The 13 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 ...entos7.yaml => ci-python-build-3.9-client.yaml} |  11 +-
 buildtools/pom.xml                                 |   2 +-
 distribution/server/src/assemble/LICENSE.bin.txt   |  10 +-
 .../bookkeeper/mledger/util/StatsBuckets.java      |   2 +-
 .../bookkeeper/mledger/util/TestStatsBuckets.java  |  13 +
 pom.xml                                            |   2 +-
 .../pulsar/broker/service/BacklogQuotaManager.java |  28 +-
 .../apache/pulsar/broker/service/ServerCnx.java    |   5 +-
 .../apache/pulsar/compaction/CompactedTopic.java   |   2 +
 .../pulsar/compaction/CompactedTopicImpl.java      |  21 +-
 .../broker/service/BacklogQuotaManagerTest.java    |  48 ++
 .../broker/service/ExclusiveProducerTest.java      |  28 +
 .../broker/service/TopicTerminationTest.java       |  29 +
 .../pulsar/client/impl/MessageChunkingTest.java    |  43 +
 .../pulsar/compaction/CompactedTopicTest.java      |  86 +-
 pulsar-client-cpp/CMakeLists.txt                   |   6 +-
 ...-build-centos7.sh => docker-build-python3.9.sh} |  27 +-
 pulsar-client-cpp/docker/build-wheels.sh           |   1 +
 pulsar-client-cpp/docker/create-images.sh          |  17 +-
 .../docker/{ => manylinux1}/Dockerfile             |  21 +-
 .../docker/{ => manylinux2014}/Dockerfile          |  23 +-
 pulsar-client-cpp/python/src/authentication.cc     |  44 +-
 pulsar-client-cpp/python/src/client.cc             |  63 +-
 pulsar-client-cpp/python/src/config.cc             | 256 +++---
 pulsar-client-cpp/python/src/consumer.cc           |  81 +-
 pulsar-client-cpp/python/src/cryptoKeyReader.cc    |   2 +-
 pulsar-client-cpp/python/src/enums.cc              | 162 ++--
 pulsar-client-cpp/python/src/exceptions.cc         |  49 +-
 pulsar-client-cpp/python/src/message.cc            | 147 ++--
 pulsar-client-cpp/python/src/producer.cc           |  71 +-
 pulsar-client-cpp/python/src/pulsar.cc             |   4 +-
 pulsar-client-cpp/python/src/reader.cc             |  48 +-
 pulsar-client-cpp/python/src/schema.cc             |  10 +-
 pulsar-client-cpp/python/src/utils.h               |   3 +-
 .../apache/pulsar/client/impl/ConsumerImpl.java    |   3 +-
 .../apache/pulsar/client/impl/ProducerImpl.java    |  83 +-
 .../pulsar/common/util/RelativeTimeUtil.java       |  12 +
 .../PulsarProtobufNativeColumnDecoder.java         |  15 +-
 .../PulsarProtobufNativeRowDecoderFactory.java     |  18 +-
 .../sql/presto/decoder/protobufnative/TestMsg.java | 917 ++++++++++++++++-----
 .../presto/decoder/protobufnative/TestMsg.proto    |   2 +
 .../protobufnative/TestProtobufNativeDecoder.java  |  14 +
 .../pulsar/websocket/stats/StatsBuckets.java       |   2 +-
 43 files changed, 1620 insertions(+), 811 deletions(-)
 copy .github/workflows/{ci-cpp-build-centos7.yaml => ci-python-build-3.9-client.yaml} (88%)
 copy pulsar-client-cpp/{docker-build-centos7.sh => docker-build-python3.9.sh} (62%)
 copy pulsar-client-cpp/docker/{ => manylinux1}/Dockerfile (97%)
 rename pulsar-client-cpp/docker/{ => manylinux2014}/Dockerfile (96%)

[pulsar] 07/13: Fix getting last message id from empty compact ledger (#13476)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit c9c48d26e99d80bd9165e5478edcd1a1a7b8acab
Author: ran <ga...@126.com>
AuthorDate: Mon Dec 27 09:45:38 2021 +0800

    Fix getting last message id from empty compact ledger (#13476)
    
    ### Motivation
    
    Currently, if the last confirmed entry is an empty ledger, getting the last message-id operation will get data from the compact ledger, if the compact ledger is also an empty ledger, it will encounter `IncorrectParameterException`.
    
    **Broker error message**
    ```
    [pulsar-io-29-9] ERROR org.apache.bookkeeper.client.LedgerHandle - IncorrectParameterException on ledgerId:617 firstEntry:-1 lastEntry:-1
    ```
    
    **Client error log**
    ```
    Exception in thread "main" org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription reader-bf9246cfcb of the topic persistent://public/ns-test/t1 gets the last message id was failed
    {"errorMsg":"Failed to read last entry of the compacted Ledger Incorrect parameter input","reqId":79405902881798690, "remote":"localhost/127.0.0.1:6650", "local":"/127.0.0.1:55207"}
    	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1034)
    	at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:2001)
    	at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:181)
    	at org.apache.pulsar.compaction.CompactedTopicTest.main(CompactedTopicTest.java:730)
    ```
    
    ### Modifications
    
    Check the compact ledger entry id before reading an entry from the compact ledger, if there is no entry, return a null value.
    
    (cherry picked from commit 8136762e743fe1bc4be2adeb08631a8c44719d37)
---
 .../pulsar/compaction/CompactedTopicImpl.java      | 15 ++++++---
 .../pulsar/compaction/CompactedTopicTest.java      | 38 ++++++++++++++++++++++
 2 files changed, 48 insertions(+), 5 deletions(-)

diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
index 4577540..a6d6fc9 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
@@ -293,11 +293,16 @@ public class CompactedTopicImpl implements CompactedTopic {
         if (compactionHorizon == null) {
             return CompletableFuture.completedFuture(null);
         }
-        return compactedTopicContext.thenCompose(context ->
-                readEntries(context.ledger, context.ledger.getLastAddConfirmed(), context.ledger.getLastAddConfirmed())
-                        .thenCompose(entries -> entries.size() > 0
-                                ? CompletableFuture.completedFuture(entries.get(0))
-                                : CompletableFuture.completedFuture(null)));
+        return compactedTopicContext.thenCompose(context -> {
+            if (context.ledger.getLastAddConfirmed() == -1) {
+                return CompletableFuture.completedFuture(null);
+            }
+            return readEntries(
+                    context.ledger, context.ledger.getLastAddConfirmed(), context.ledger.getLastAddConfirmed())
+                    .thenCompose(entries -> entries.size() > 0
+                            ? CompletableFuture.completedFuture(entries.get(0))
+                            : CompletableFuture.completedFuture(null));
+        });
     }
 
     private static int comparePositionAndMessageId(PositionImpl p, MessageIdData m) {
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java
index 69d66d9..d44868a 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java
@@ -650,4 +650,42 @@ public class CompactedTopicTest extends MockedPulsarServiceBaseTest {
         reader.close();
         producer.close();
     }
+
+    @Test(timeOut = 1000 * 30)
+    public void testReader() throws Exception {
+        final String ns = "my-property/use/my-ns";
+        String topic = "persistent://" + ns + "/t1";
+
+        @Cleanup
+        Producer<byte[]> producer = pulsarClient.newProducer()
+                .topic(topic)
+                .create();
+
+        producer.newMessage().key("k").value(("value").getBytes()).send();
+        producer.newMessage().key("k").value(null).send();
+        pulsar.getCompactor().compact(topic).get();
+
+        Awaitility.await()
+                .pollInterval(3, TimeUnit.SECONDS)
+                .atMost(30, TimeUnit.SECONDS).untilAsserted(() -> {
+            admin.topics().unload(topic);
+            Thread.sleep(100);
+            Assert.assertTrue(admin.topics().getInternalStats(topic).lastConfirmedEntry.endsWith("-1"));
+        });
+        // Make sure the last confirm entry is -1, then get last message id from compact ledger
+        PersistentTopicInternalStats internalStats = admin.topics().getInternalStats(topic);
+        Assert.assertTrue(internalStats.lastConfirmedEntry.endsWith("-1"));
+        // Because the latest value of the key `k` is null, so there is no data in compact ledger.
+        Assert.assertEquals(internalStats.compactedLedger.size, 0);
+
+        @Cleanup
+        Reader<byte[]> reader = pulsarClient.newReader()
+                .topic(topic)
+                .startMessageIdInclusive()
+                .startMessageId(MessageId.earliest)
+                .readCompacted(true)
+                .create();
+        Assert.assertFalse(reader.hasMessageAvailable());
+    }
+
 }

[pulsar] 08/13: fix: bug when allAll bucket (#13467)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit cac69427ec10af90ae82957b0d26cc95a7701a39
Author: Lei Zhiyuan <le...@gmail.com>
AuthorDate: Tue Dec 28 19:33:01 2021 +0800

    fix: bug when allAll bucket (#13467)
    
    (cherry picked from commit 1235162cbe91fe04d0014d7ccd9672a7c68f4c36)
---
 .../org/apache/bookkeeper/mledger/util/StatsBuckets.java    |  2 +-
 .../apache/bookkeeper/mledger/util/TestStatsBuckets.java    | 13 +++++++++++++
 .../org/apache/pulsar/websocket/stats/StatsBuckets.java     |  2 +-
 3 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/StatsBuckets.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/StatsBuckets.java
index 6d08bf4..dd77988 100644
--- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/StatsBuckets.java
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/StatsBuckets.java
@@ -110,7 +110,7 @@ public class StatsBuckets {
             buckets[i].add(other.values[i]);
         }
 
-        sumCounter.add(other.count);
+        sumCounter.add(other.sum);
     }
 
     private boolean isSorted(long[] array) {
diff --git a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/TestStatsBuckets.java b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/TestStatsBuckets.java
index 0236663..12efbb0 100644
--- a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/TestStatsBuckets.java
+++ b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/TestStatsBuckets.java
@@ -99,4 +99,17 @@ public class TestStatsBuckets {
         assertEquals(stats.getCount(), 3);
         assertEquals(stats.getBuckets(), new long[] { 1, 0, 1, 1 });
     }
+
+    @Test
+    public void testAddAll() {
+        StatsBuckets stats = new StatsBuckets(10, 20, 30);
+        stats.addValue(1);
+        stats.addValue(2);
+        stats.refresh();
+        StatsBuckets stats2 = new StatsBuckets(10, 20, 30);
+        stats2.addAll(stats);
+        stats2.refresh();
+        assertEquals(stats2.getSum(),3);
+        assertEquals(stats2.getCount(),2);
+    }
 }
diff --git a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/stats/StatsBuckets.java b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/stats/StatsBuckets.java
index 82f34d1..8625146 100644
--- a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/stats/StatsBuckets.java
+++ b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/stats/StatsBuckets.java
@@ -61,7 +61,7 @@ public class StatsBuckets {
         for (int i = 0; i < buckets.length; i++) {
             buckets[i].add(other.values[i]);
         }
-        sumCounter.add(other.count);
+        sumCounter.add(other.sum);
     }
     
     

[pulsar] 09/13: Fix reousrce leak when create producer failed (#13505)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 77c66b3911e261c40eab86b7890dede6b736559e
Author: Aloys <lo...@gmail.com>
AuthorDate: Tue Dec 28 19:06:43 2021 +0800

    Fix reousrce leak when create producer failed (#13505)
    
    Fixes #13214
    
    When client create producer failed caused by connection failed, topic terminated, or produce fenced. There are some resources that are not released in the client.
    
    When creating producer failed.
    1. stop the sendTimout  task
    2. cancel the batchTimerTask
    3. cancel the keyGeneratorTask
    4. cancel the statTimeout task
    
    (cherry picked from commit 57eccf48e418e0243f586a945134785dbe9b5d08)
---
 .../broker/service/ExclusiveProducerTest.java      | 28 +++++++++++++++
 .../broker/service/TopicTerminationTest.java       | 29 +++++++++++++++
 .../apache/pulsar/client/impl/ProducerImpl.java    | 42 +++++++++++++---------
 3 files changed, 82 insertions(+), 17 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ExclusiveProducerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ExclusiveProducerTest.java
index 04da55d..4d2b16a 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ExclusiveProducerTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ExclusiveProducerTest.java
@@ -25,6 +25,7 @@ import java.util.Optional;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.TimeUnit;
 
+import io.netty.util.HashedWheelTimer;
 import lombok.Cleanup;
 
 import org.apache.pulsar.client.api.Producer;
@@ -33,8 +34,11 @@ import org.apache.pulsar.client.api.PulsarClient;
 import org.apache.pulsar.client.api.PulsarClientException.ProducerBusyException;
 import org.apache.pulsar.client.api.PulsarClientException.ProducerFencedException;
 import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
 import org.apache.pulsar.common.naming.TopicName;
+import org.awaitility.Awaitility;
 import org.powermock.reflect.Whitebox;
+import org.testng.Assert;
 import org.testng.annotations.AfterClass;
 import org.testng.annotations.BeforeClass;
 import org.testng.annotations.DataProvider;
@@ -118,6 +122,30 @@ public class ExclusiveProducerTest extends BrokerTestBase {
     }
 
     @Test(dataProvider = "topics")
+    public void testProducerTasksCleanupWhenUsingExclusiveProducers(String type, boolean partitioned) throws Exception {
+        String topic = newTopic(type, partitioned);
+        Producer<String> p1 = pulsarClient.newProducer(Schema.STRING)
+                .topic(topic)
+                .accessMode(ProducerAccessMode.Exclusive)
+                .create();
+
+        try {
+            pulsarClient.newProducer(Schema.STRING)
+                    .topic(topic)
+                    .accessMode(ProducerAccessMode.Exclusive)
+                    .create();
+            fail("Should have failed");
+        } catch (ProducerFencedException e) {
+            // Expected
+        }
+
+        p1.close();
+
+        HashedWheelTimer timer = (HashedWheelTimer) ((PulsarClientImpl) pulsarClient).timer();
+        Awaitility.await().untilAsserted(() -> Assert.assertEquals(timer.pendingTimeouts(), 0));
+    }
+
+    @Test(dataProvider = "topics")
     public void existingSharedProducer(String type, boolean partitioned) throws Exception {
         String topic = newTopic(type, partitioned);
 
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicTerminationTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicTerminationTest.java
index a7a1111..8499524 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicTerminationTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicTerminationTest.java
@@ -31,6 +31,8 @@ import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.CyclicBarrier;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
+
+import io.netty.util.HashedWheelTimer;
 import org.apache.pulsar.client.admin.PulsarAdminException.NotAllowedException;
 import org.apache.pulsar.client.api.Consumer;
 import org.apache.pulsar.client.api.Message;
@@ -41,7 +43,10 @@ import org.apache.pulsar.client.api.Producer;
 import org.apache.pulsar.client.api.PulsarClientException;
 import org.apache.pulsar.client.api.Reader;
 import org.apache.pulsar.client.api.ReaderListener;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
 import org.apache.pulsar.common.util.FutureUtil;
+import org.awaitility.Awaitility;
+import org.testng.Assert;
 import org.testng.annotations.AfterMethod;
 import org.testng.annotations.BeforeMethod;
 import org.testng.annotations.Test;
@@ -107,6 +112,30 @@ public class TopicTerminationTest extends BrokerTestBase {
         }
     }
 
+    public void testCreatingProducerTasksCleanupWhenOnTerminatedTopic() throws Exception {
+        Producer<byte[]> producer = pulsarClient.newProducer().topic(topicName)
+                .enableBatching(false)
+                .messageRoutingMode(MessageRoutingMode.SinglePartition)
+                .create();
+
+        producer.send("msg-1".getBytes());
+        producer.send("msg-2".getBytes());
+        MessageId msgId3 = producer.send("msg-3".getBytes());
+
+        MessageId lastMessageId = admin.topics().terminateTopicAsync(topicName).get();
+        assertEquals(lastMessageId, msgId3);
+        producer.close();
+
+        try {
+            pulsarClient.newProducer().topic(topicName).create();
+            fail("Should have thrown exception");
+        } catch (PulsarClientException.TopicTerminatedException e) {
+            // Expected
+        }
+        HashedWheelTimer timer = (HashedWheelTimer) ((PulsarClientImpl) pulsarClient).timer();
+        Awaitility.await().untilAsserted(() -> Assert.assertEquals(timer.pendingTimeouts(), 0));
+    }
+
     @Test(timeOut = 20000)
     public void testTerminateWhilePublishing() throws Exception {
         Producer<byte[]> producer = pulsarClient.newProducer().topic(topicName)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
index a5baba7..78aaf3c 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
@@ -866,23 +866,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
             return CompletableFuture.completedFuture(null);
         }
 
-        Timeout timeout = sendTimeout;
-        if (timeout != null) {
-            timeout.cancel();
-            sendTimeout = null;
-        }
-
-        ScheduledFuture<?> batchTimerTask = this.batchTimerTask;
-        if (batchTimerTask != null) {
-            batchTimerTask.cancel(false);
-            this.batchTimerTask = null;
-        }
-
-        if (keyGeneratorTask != null && !keyGeneratorTask.isCancelled()) {
-            keyGeneratorTask.cancel(false);
-        }
-
-        stats.cancelStatsTimeout();
+        closeProducerTasks();
 
         ClientCnx cnx = cnx();
         if (cnx == null || currentState != State.Ready) {
@@ -1544,6 +1528,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
                             failPendingMessages(cnx(), (PulsarClientException) cause);
                         }
                         producerCreatedFuture.completeExceptionally(cause);
+                        closeProducerTasks();
                         client.cleanupProducer(this);
                     } else if (cause instanceof PulsarClientException.ProducerFencedException) {
                         setState(State.ProducerFenced);
@@ -1551,6 +1536,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
                             failPendingMessages(cnx(), (PulsarClientException) cause);
                         }
                         producerCreatedFuture.completeExceptionally(cause);
+                        closeProducerTasks();
                         client.cleanupProducer(this);
                     } else if (producerCreatedFuture.isDone() || //
                     (cause instanceof PulsarClientException && PulsarClientException.isRetriableError(cause)
@@ -1561,6 +1547,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
                     } else {
                         setState(State.Failed);
                         producerCreatedFuture.completeExceptionally(cause);
+                        closeProducerTasks();
                         client.cleanupProducer(this);
                         Timeout timeout = sendTimeout;
                         if (timeout != null) {
@@ -1583,11 +1570,32 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
             } else {
                 log.info("[{}] Producer creation failed for producer {} after producerTimeout", topic, producerId);
             }
+            closeProducerTasks();
             setState(State.Failed);
             client.cleanupProducer(this);
         }
     }
 
+    private void closeProducerTasks() {
+        Timeout timeout = sendTimeout;
+        if (timeout != null) {
+            timeout.cancel();
+            sendTimeout = null;
+        }
+
+        ScheduledFuture<?> batchTimerTask = this.batchTimerTask;
+        if (batchTimerTask != null) {
+            batchTimerTask.cancel(false);
+            this.batchTimerTask = null;
+        }
+
+        if (keyGeneratorTask != null && !keyGeneratorTask.isCancelled()) {
+            keyGeneratorTask.cancel(false);
+        }
+
+        stats.cancelStatsTimeout();
+    }
+
     private void resendMessages(ClientCnx cnx, long expectedEpoch) {
         cnx.ctx().channel().eventLoop().execute(() -> {
             synchronized (this) {

[pulsar] 11/13: [Pulsar SQL] support protobuf/timestamp (#13287)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 078c8db6fd526021a1d2138c0fce74defb29d7d7
Author: Tao Jiuming <95...@users.noreply.github.com>
AuthorDate: Wed Dec 29 09:38:38 2021 +0800

    [Pulsar SQL] support protobuf/timestamp (#13287)
    
    (cherry picked from commit 1ea4ad814f5f30b8c371db2a86626cd568ace553)
---
 .../PulsarProtobufNativeColumnDecoder.java         |  15 +-
 .../PulsarProtobufNativeRowDecoderFactory.java     |  18 +-
 .../sql/presto/decoder/protobufnative/TestMsg.java | 917 ++++++++++++++++-----
 .../presto/decoder/protobufnative/TestMsg.proto    |   2 +
 .../protobufnative/TestProtobufNativeDecoder.java  |  14 +
 5 files changed, 769 insertions(+), 197 deletions(-)

diff --git a/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/decoder/protobufnative/PulsarProtobufNativeColumnDecoder.java b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/decoder/protobufnative/PulsarProtobufNativeColumnDecoder.java
index 56e9b77..c71e7ca 100644
--- a/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/decoder/protobufnative/PulsarProtobufNativeColumnDecoder.java
+++ b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/decoder/protobufnative/PulsarProtobufNativeColumnDecoder.java
@@ -70,7 +70,8 @@ public class PulsarProtobufNativeColumnDecoder {
             BigintType.BIGINT,
             RealType.REAL,
             DoubleType.DOUBLE,
-            VarbinaryType.VARBINARY);
+            VarbinaryType.VARBINARY,
+            TimestampType.TIMESTAMP);
 
     private final Type columnType;
     private final String columnMapping;
@@ -193,6 +194,15 @@ public class PulsarProtobufNativeColumnDecoder {
                 return floatToIntBits((Float) value);
             }
 
+            //return millisecond which parsed from protobuf/timestamp
+            if (columnType instanceof TimestampType && value instanceof DynamicMessage) {
+                DynamicMessage message = (DynamicMessage) value;
+                int nanos = (int) message.getField(message.getDescriptorForType().findFieldByName("nanos"));
+                long seconds = (long) message.getField(message.getDescriptorForType().findFieldByName("seconds"));
+                //maybe an exception here, but seems will never happen in hundred years.
+                return seconds * MILLIS_PER_SECOND + nanos / NANOS_PER_MILLISECOND;
+            }
+
             throw new PrestoException(DECODER_CONVERSION_NOT_SUPPORTED,
                     format("cannot decode object of '%s' as '%s' for column '%s'",
                             value.getClass(), columnType, columnName));
@@ -377,5 +387,6 @@ public class PulsarProtobufNativeColumnDecoder {
 
     protected static final String PROTOBUF_MAP_KEY_NAME = "key";
     protected static final String PROTOBUF_MAP_VALUE_NAME = "value";
-
+    private static final long MILLIS_PER_SECOND = 1000;
+    private static final long NANOS_PER_MILLISECOND = 1000000;
 }
diff --git a/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/decoder/protobufnative/PulsarProtobufNativeRowDecoderFactory.java b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/decoder/protobufnative/PulsarProtobufNativeRowDecoderFactory.java
index 3a415b3..2d0f9af 100644
--- a/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/decoder/protobufnative/PulsarProtobufNativeRowDecoderFactory.java
+++ b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/decoder/protobufnative/PulsarProtobufNativeRowDecoderFactory.java
@@ -25,6 +25,7 @@ import static java.util.stream.Collectors.toList;
 
 import com.google.common.collect.ImmutableList;
 import com.google.protobuf.Descriptors;
+import com.google.protobuf.TimestampProto;
 import io.airlift.log.Logger;
 import io.prestosql.decoder.DecoderColumnHandle;
 import io.prestosql.spi.PrestoException;
@@ -37,11 +38,13 @@ import io.prestosql.spi.type.IntegerType;
 import io.prestosql.spi.type.RealType;
 import io.prestosql.spi.type.RowType;
 import io.prestosql.spi.type.StandardTypes;
+import io.prestosql.spi.type.TimestampType;
 import io.prestosql.spi.type.Type;
 import io.prestosql.spi.type.TypeManager;
 import io.prestosql.spi.type.TypeSignature;
 import io.prestosql.spi.type.TypeSignatureParameter;
 import io.prestosql.spi.type.VarbinaryType;
+
 import java.util.List;
 import java.util.Optional;
 import java.util.Set;
@@ -142,11 +145,16 @@ public class PulsarProtobufNativeRowDecoderFactory implements PulsarRowDecoderFa
                             ImmutableList.of(TypeSignatureParameter.typeParameter(keyType),
                                     TypeSignatureParameter.typeParameter(valueType)));
                 } else {
-                    //row
-                    dataType = RowType.from(msg.getFields().stream()
-                            .map(rowField -> new RowType.Field(Optional.of(rowField.getName()),
-                                    parseProtobufPrestoType(rowField)))
-                            .collect(toImmutableList()));
+                    if (TimestampProto.getDescriptor().toProto().getName().equals(msg.getFile().toProto().getName())) {
+                        //if msg type is protobuf/timestamp
+                        dataType = TimestampType.TIMESTAMP;
+                    } else {
+                        //row
+                        dataType = RowType.from(msg.getFields().stream()
+                                .map(rowField -> new RowType.Field(Optional.of(rowField.getName()),
+                                        parseProtobufPrestoType(rowField)))
+                                .collect(toImmutableList()));
+                    }
                 }
                 break;
             default:
diff --git a/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.java b/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.java
index 3065caf..401ded5 100644
--- a/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.java
+++ b/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.java
@@ -67,6 +67,8 @@ public final class TestMsg {
     }
 
     /**
+     * @param value The numeric wire value of the corresponding enum entry.
+     * @return The enum associated with the given numeric wire value.
      * @deprecated Use {@link #forNumber(int)} instead.
      */
     @java.lang.Deprecated
@@ -74,6 +76,10 @@ public final class TestMsg {
       return forNumber(value);
     }
 
+    /**
+     * @param value The numeric wire value of the corresponding enum entry.
+     * @return The enum associated with the given numeric wire value.
+     */
     public static TestEnum forNumber(int value) {
       switch (value) {
         case 0: return SHARED;
@@ -96,6 +102,10 @@ public final class TestMsg {
 
     public final com.google.protobuf.Descriptors.EnumValueDescriptor
         getValueDescriptor() {
+      if (this == UNRECOGNIZED) {
+        throw new java.lang.IllegalStateException(
+            "Can't get the descriptor of an unrecognized enum value.");
+      }
       return getDescriptor().getValues().get(ordinal());
     }
     public final com.google.protobuf.Descriptors.EnumDescriptor
@@ -136,25 +146,30 @@ public final class TestMsg {
 
     /**
      * <code>string foo = 1;</code>
+     * @return The foo.
      */
     java.lang.String getFoo();
     /**
      * <code>string foo = 1;</code>
+     * @return The bytes for foo.
      */
     com.google.protobuf.ByteString
         getFooBytes();
 
     /**
      * <code>double bar = 2;</code>
+     * @return The bar.
      */
     double getBar();
 
     /**
      * <code>.proto.SubMessage.NestedMessage nestedMessage = 3;</code>
+     * @return Whether the nestedMessage field is set.
      */
     boolean hasNestedMessage();
     /**
      * <code>.proto.SubMessage.NestedMessage nestedMessage = 3;</code>
+     * @return The nestedMessage.
      */
     org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage getNestedMessage();
     /**
@@ -165,7 +180,7 @@ public final class TestMsg {
   /**
    * Protobuf type {@code proto.SubMessage}
    */
-  public  static final class SubMessage extends
+  public static final class SubMessage extends
       com.google.protobuf.GeneratedMessageV3 implements
       // @@protoc_insertion_point(message_implements:proto.SubMessage)
       SubMessageOrBuilder {
@@ -176,7 +191,13 @@ public final class TestMsg {
     }
     private SubMessage() {
       foo_ = "";
-      bar_ = 0D;
+    }
+
+    @java.lang.Override
+    @SuppressWarnings({"unused"})
+    protected java.lang.Object newInstance(
+        UnusedPrivateParameter unused) {
+      return new SubMessage();
     }
 
     @java.lang.Override
@@ -192,7 +213,6 @@ public final class TestMsg {
       if (extensionRegistry == null) {
         throw new java.lang.NullPointerException();
       }
-      int mutable_bitField0_ = 0;
       com.google.protobuf.UnknownFieldSet.Builder unknownFields =
           com.google.protobuf.UnknownFieldSet.newBuilder();
       try {
@@ -203,13 +223,6 @@ public final class TestMsg {
             case 0:
               done = true;
               break;
-            default: {
-              if (!parseUnknownFieldProto3(
-                  input, unknownFields, extensionRegistry, tag)) {
-                done = true;
-              }
-              break;
-            }
             case 10: {
               java.lang.String s = input.readStringRequireUtf8();
 
@@ -234,6 +247,13 @@ public final class TestMsg {
 
               break;
             }
+            default: {
+              if (!parseUnknownField(
+                  input, unknownFields, extensionRegistry, tag)) {
+                done = true;
+              }
+              break;
+            }
           }
         }
       } catch (com.google.protobuf.InvalidProtocolBufferException e) {
@@ -251,6 +271,7 @@ public final class TestMsg {
       return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_descriptor;
     }
 
+    @java.lang.Override
     protected com.google.protobuf.GeneratedMessageV3.FieldAccessorTable
         internalGetFieldAccessorTable() {
       return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_fieldAccessorTable
@@ -264,29 +285,37 @@ public final class TestMsg {
 
       /**
        * <code>string title = 1;</code>
+       * @return The title.
        */
       java.lang.String getTitle();
       /**
        * <code>string title = 1;</code>
+       * @return The bytes for title.
        */
       com.google.protobuf.ByteString
           getTitleBytes();
 
       /**
        * <code>repeated string urls = 2;</code>
+       * @return A list containing the urls.
        */
       java.util.List<java.lang.String>
           getUrlsList();
       /**
        * <code>repeated string urls = 2;</code>
+       * @return The count of urls.
        */
       int getUrlsCount();
       /**
        * <code>repeated string urls = 2;</code>
+       * @param index The index of the element to return.
+       * @return The urls at the given index.
        */
       java.lang.String getUrls(int index);
       /**
        * <code>repeated string urls = 2;</code>
+       * @param index The index of the value to return.
+       * @return The bytes of the urls at the given index.
        */
       com.google.protobuf.ByteString
           getUrlsBytes(int index);
@@ -294,7 +323,7 @@ public final class TestMsg {
     /**
      * Protobuf type {@code proto.SubMessage.NestedMessage}
      */
-    public  static final class NestedMessage extends
+    public static final class NestedMessage extends
         com.google.protobuf.GeneratedMessageV3 implements
         // @@protoc_insertion_point(message_implements:proto.SubMessage.NestedMessage)
         NestedMessageOrBuilder {
@@ -309,6 +338,13 @@ public final class TestMsg {
       }
 
       @java.lang.Override
+      @SuppressWarnings({"unused"})
+      protected java.lang.Object newInstance(
+          UnusedPrivateParameter unused) {
+        return new NestedMessage();
+      }
+
+      @java.lang.Override
       public final com.google.protobuf.UnknownFieldSet
       getUnknownFields() {
         return this.unknownFields;
@@ -332,13 +368,6 @@ public final class TestMsg {
               case 0:
                 done = true;
                 break;
-              default: {
-                if (!parseUnknownFieldProto3(
-                    input, unknownFields, extensionRegistry, tag)) {
-                  done = true;
-                }
-                break;
-              }
               case 10: {
                 java.lang.String s = input.readStringRequireUtf8();
 
@@ -347,13 +376,20 @@ public final class TestMsg {
               }
               case 18: {
                 java.lang.String s = input.readStringRequireUtf8();
-                if (!((mutable_bitField0_ & 0x00000002) == 0x00000002)) {
+                if (!((mutable_bitField0_ & 0x00000001) != 0)) {
                   urls_ = new com.google.protobuf.LazyStringArrayList();
-                  mutable_bitField0_ |= 0x00000002;
+                  mutable_bitField0_ |= 0x00000001;
                 }
                 urls_.add(s);
                 break;
               }
+              default: {
+                if (!parseUnknownField(
+                    input, unknownFields, extensionRegistry, tag)) {
+                  done = true;
+                }
+                break;
+              }
             }
           }
         } catch (com.google.protobuf.InvalidProtocolBufferException e) {
@@ -362,7 +398,7 @@ public final class TestMsg {
           throw new com.google.protobuf.InvalidProtocolBufferException(
               e).setUnfinishedMessage(this);
         } finally {
-          if (((mutable_bitField0_ & 0x00000002) == 0x00000002)) {
+          if (((mutable_bitField0_ & 0x00000001) != 0)) {
             urls_ = urls_.getUnmodifiableView();
           }
           this.unknownFields = unknownFields.build();
@@ -374,6 +410,7 @@ public final class TestMsg {
         return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_NestedMessage_descriptor;
       }
 
+      @java.lang.Override
       protected com.google.protobuf.GeneratedMessageV3.FieldAccessorTable
           internalGetFieldAccessorTable() {
         return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_NestedMessage_fieldAccessorTable
@@ -381,12 +418,13 @@ public final class TestMsg {
                 org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage.class, org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage.Builder.class);
       }
 
-      private int bitField0_;
       public static final int TITLE_FIELD_NUMBER = 1;
       private volatile java.lang.Object title_;
       /**
        * <code>string title = 1;</code>
+       * @return The title.
        */
+      @java.lang.Override
       public java.lang.String getTitle() {
         java.lang.Object ref = title_;
         if (ref instanceof java.lang.String) {
@@ -401,7 +439,9 @@ public final class TestMsg {
       }
       /**
        * <code>string title = 1;</code>
+       * @return The bytes for title.
        */
+      @java.lang.Override
       public com.google.protobuf.ByteString
           getTitleBytes() {
         java.lang.Object ref = title_;
@@ -420,6 +460,7 @@ public final class TestMsg {
       private com.google.protobuf.LazyStringList urls_;
       /**
        * <code>repeated string urls = 2;</code>
+       * @return A list containing the urls.
        */
       public com.google.protobuf.ProtocolStringList
           getUrlsList() {
@@ -427,18 +468,23 @@ public final class TestMsg {
       }
       /**
        * <code>repeated string urls = 2;</code>
+       * @return The count of urls.
        */
       public int getUrlsCount() {
         return urls_.size();
       }
       /**
        * <code>repeated string urls = 2;</code>
+       * @param index The index of the element to return.
+       * @return The urls at the given index.
        */
       public java.lang.String getUrls(int index) {
         return urls_.get(index);
       }
       /**
        * <code>repeated string urls = 2;</code>
+       * @param index The index of the value to return.
+       * @return The bytes of the urls at the given index.
        */
       public com.google.protobuf.ByteString
           getUrlsBytes(int index) {
@@ -446,6 +492,7 @@ public final class TestMsg {
       }
 
       private byte memoizedIsInitialized = -1;
+      @java.lang.Override
       public final boolean isInitialized() {
         byte isInitialized = memoizedIsInitialized;
         if (isInitialized == 1) return true;
@@ -455,6 +502,7 @@ public final class TestMsg {
         return true;
       }
 
+      @java.lang.Override
       public void writeTo(com.google.protobuf.CodedOutputStream output)
                           throws java.io.IOException {
         if (!getTitleBytes().isEmpty()) {
@@ -466,6 +514,7 @@ public final class TestMsg {
         unknownFields.writeTo(output);
       }
 
+      @java.lang.Override
       public int getSerializedSize() {
         int size = memoizedSize;
         if (size != -1) return size;
@@ -497,13 +546,12 @@ public final class TestMsg {
         }
         org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage other = (org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage) obj;
 
-        boolean result = true;
-        result = result && getTitle()
-            .equals(other.getTitle());
-        result = result && getUrlsList()
-            .equals(other.getUrlsList());
-        result = result && unknownFields.equals(other.unknownFields);
-        return result;
+        if (!getTitle()
+            .equals(other.getTitle())) return false;
+        if (!getUrlsList()
+            .equals(other.getUrlsList())) return false;
+        if (!unknownFields.equals(other.unknownFields)) return false;
+        return true;
       }
 
       @java.lang.Override
@@ -594,6 +642,7 @@ public final class TestMsg {
             .parseWithIOException(PARSER, input, extensionRegistry);
       }
 
+      @java.lang.Override
       public Builder newBuilderForType() { return newBuilder(); }
       public static Builder newBuilder() {
         return DEFAULT_INSTANCE.toBuilder();
@@ -601,6 +650,7 @@ public final class TestMsg {
       public static Builder newBuilder(org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage prototype) {
         return DEFAULT_INSTANCE.toBuilder().mergeFrom(prototype);
       }
+      @java.lang.Override
       public Builder toBuilder() {
         return this == DEFAULT_INSTANCE
             ? new Builder() : new Builder().mergeFrom(this);
@@ -624,6 +674,7 @@ public final class TestMsg {
           return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_NestedMessage_descriptor;
         }
 
+        @java.lang.Override
         protected com.google.protobuf.GeneratedMessageV3.FieldAccessorTable
             internalGetFieldAccessorTable() {
           return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_NestedMessage_fieldAccessorTable
@@ -646,24 +697,28 @@ public final class TestMsg {
                   .alwaysUseFieldBuilders) {
           }
         }
+        @java.lang.Override
         public Builder clear() {
           super.clear();
           title_ = "";
 
           urls_ = com.google.protobuf.LazyStringArrayList.EMPTY;
-          bitField0_ = (bitField0_ & ~0x00000002);
+          bitField0_ = (bitField0_ & ~0x00000001);
           return this;
         }
 
+        @java.lang.Override
         public com.google.protobuf.Descriptors.Descriptor
             getDescriptorForType() {
           return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_NestedMessage_descriptor;
         }
 
+        @java.lang.Override
         public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage getDefaultInstanceForType() {
           return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage.getDefaultInstance();
         }
 
+        @java.lang.Override
         public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage build() {
           org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage result = buildPartial();
           if (!result.isInitialized()) {
@@ -672,47 +727,53 @@ public final class TestMsg {
           return result;
         }
 
+        @java.lang.Override
         public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage buildPartial() {
           org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage result = new org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage(this);
           int from_bitField0_ = bitField0_;
-          int to_bitField0_ = 0;
           result.title_ = title_;
-          if (((bitField0_ & 0x00000002) == 0x00000002)) {
+          if (((bitField0_ & 0x00000001) != 0)) {
             urls_ = urls_.getUnmodifiableView();
-            bitField0_ = (bitField0_ & ~0x00000002);
+            bitField0_ = (bitField0_ & ~0x00000001);
           }
           result.urls_ = urls_;
-          result.bitField0_ = to_bitField0_;
           onBuilt();
           return result;
         }
 
+        @java.lang.Override
         public Builder clone() {
-          return (Builder) super.clone();
+          return super.clone();
         }
+        @java.lang.Override
         public Builder setField(
             com.google.protobuf.Descriptors.FieldDescriptor field,
             java.lang.Object value) {
-          return (Builder) super.setField(field, value);
+          return super.setField(field, value);
         }
+        @java.lang.Override
         public Builder clearField(
             com.google.protobuf.Descriptors.FieldDescriptor field) {
-          return (Builder) super.clearField(field);
+          return super.clearField(field);
         }
+        @java.lang.Override
         public Builder clearOneof(
             com.google.protobuf.Descriptors.OneofDescriptor oneof) {
-          return (Builder) super.clearOneof(oneof);
+          return super.clearOneof(oneof);
         }
+        @java.lang.Override
         public Builder setRepeatedField(
             com.google.protobuf.Descriptors.FieldDescriptor field,
             int index, java.lang.Object value) {
-          return (Builder) super.setRepeatedField(field, index, value);
+          return super.setRepeatedField(field, index, value);
         }
+        @java.lang.Override
         public Builder addRepeatedField(
             com.google.protobuf.Descriptors.FieldDescriptor field,
             java.lang.Object value) {
-          return (Builder) super.addRepeatedField(field, value);
+          return super.addRepeatedField(field, value);
         }
+        @java.lang.Override
         public Builder mergeFrom(com.google.protobuf.Message other) {
           if (other instanceof org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage) {
             return mergeFrom((org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage)other);
@@ -731,7 +792,7 @@ public final class TestMsg {
           if (!other.urls_.isEmpty()) {
             if (urls_.isEmpty()) {
               urls_ = other.urls_;
-              bitField0_ = (bitField0_ & ~0x00000002);
+              bitField0_ = (bitField0_ & ~0x00000001);
             } else {
               ensureUrlsIsMutable();
               urls_.addAll(other.urls_);
@@ -743,10 +804,12 @@ public final class TestMsg {
           return this;
         }
 
+        @java.lang.Override
         public final boolean isInitialized() {
           return true;
         }
 
+        @java.lang.Override
         public Builder mergeFrom(
             com.google.protobuf.CodedInputStream input,
             com.google.protobuf.ExtensionRegistryLite extensionRegistry)
@@ -769,6 +832,7 @@ public final class TestMsg {
         private java.lang.Object title_ = "";
         /**
          * <code>string title = 1;</code>
+         * @return The title.
          */
         public java.lang.String getTitle() {
           java.lang.Object ref = title_;
@@ -784,6 +848,7 @@ public final class TestMsg {
         }
         /**
          * <code>string title = 1;</code>
+         * @return The bytes for title.
          */
         public com.google.protobuf.ByteString
             getTitleBytes() {
@@ -800,6 +865,8 @@ public final class TestMsg {
         }
         /**
          * <code>string title = 1;</code>
+         * @param value The title to set.
+         * @return This builder for chaining.
          */
         public Builder setTitle(
             java.lang.String value) {
@@ -813,6 +880,7 @@ public final class TestMsg {
         }
         /**
          * <code>string title = 1;</code>
+         * @return This builder for chaining.
          */
         public Builder clearTitle() {
           
@@ -822,6 +890,8 @@ public final class TestMsg {
         }
         /**
          * <code>string title = 1;</code>
+         * @param value The bytes for title to set.
+         * @return This builder for chaining.
          */
         public Builder setTitleBytes(
             com.google.protobuf.ByteString value) {
@@ -837,13 +907,14 @@ public final class TestMsg {
 
         private com.google.protobuf.LazyStringList urls_ = com.google.protobuf.LazyStringArrayList.EMPTY;
         private void ensureUrlsIsMutable() {
-          if (!((bitField0_ & 0x00000002) == 0x00000002)) {
+          if (!((bitField0_ & 0x00000001) != 0)) {
             urls_ = new com.google.protobuf.LazyStringArrayList(urls_);
-            bitField0_ |= 0x00000002;
+            bitField0_ |= 0x00000001;
            }
         }
         /**
          * <code>repeated string urls = 2;</code>
+         * @return A list containing the urls.
          */
         public com.google.protobuf.ProtocolStringList
             getUrlsList() {
@@ -851,18 +922,23 @@ public final class TestMsg {
         }
         /**
          * <code>repeated string urls = 2;</code>
+         * @return The count of urls.
          */
         public int getUrlsCount() {
           return urls_.size();
         }
         /**
          * <code>repeated string urls = 2;</code>
+         * @param index The index of the element to return.
+         * @return The urls at the given index.
          */
         public java.lang.String getUrls(int index) {
           return urls_.get(index);
         }
         /**
          * <code>repeated string urls = 2;</code>
+         * @param index The index of the value to return.
+         * @return The bytes of the urls at the given index.
          */
         public com.google.protobuf.ByteString
             getUrlsBytes(int index) {
@@ -870,6 +946,9 @@ public final class TestMsg {
         }
         /**
          * <code>repeated string urls = 2;</code>
+         * @param index The index to set the value at.
+         * @param value The urls to set.
+         * @return This builder for chaining.
          */
         public Builder setUrls(
             int index, java.lang.String value) {
@@ -883,6 +962,8 @@ public final class TestMsg {
         }
         /**
          * <code>repeated string urls = 2;</code>
+         * @param value The urls to add.
+         * @return This builder for chaining.
          */
         public Builder addUrls(
             java.lang.String value) {
@@ -896,6 +977,8 @@ public final class TestMsg {
         }
         /**
          * <code>repeated string urls = 2;</code>
+         * @param values The urls to add.
+         * @return This builder for chaining.
          */
         public Builder addAllUrls(
             java.lang.Iterable<java.lang.String> values) {
@@ -907,15 +990,18 @@ public final class TestMsg {
         }
         /**
          * <code>repeated string urls = 2;</code>
+         * @return This builder for chaining.
          */
         public Builder clearUrls() {
           urls_ = com.google.protobuf.LazyStringArrayList.EMPTY;
-          bitField0_ = (bitField0_ & ~0x00000002);
+          bitField0_ = (bitField0_ & ~0x00000001);
           onChanged();
           return this;
         }
         /**
          * <code>repeated string urls = 2;</code>
+         * @param value The bytes of the urls to add.
+         * @return This builder for chaining.
          */
         public Builder addUrlsBytes(
             com.google.protobuf.ByteString value) {
@@ -928,11 +1014,13 @@ public final class TestMsg {
           onChanged();
           return this;
         }
+        @java.lang.Override
         public final Builder setUnknownFields(
             final com.google.protobuf.UnknownFieldSet unknownFields) {
-          return super.setUnknownFieldsProto3(unknownFields);
+          return super.setUnknownFields(unknownFields);
         }
 
+        @java.lang.Override
         public final Builder mergeUnknownFields(
             final com.google.protobuf.UnknownFieldSet unknownFields) {
           return super.mergeUnknownFields(unknownFields);
@@ -954,6 +1042,7 @@ public final class TestMsg {
 
       private static final com.google.protobuf.Parser<NestedMessage>
           PARSER = new com.google.protobuf.AbstractParser<NestedMessage>() {
+        @java.lang.Override
         public NestedMessage parsePartialFrom(
             com.google.protobuf.CodedInputStream input,
             com.google.protobuf.ExtensionRegistryLite extensionRegistry)
@@ -971,6 +1060,7 @@ public final class TestMsg {
         return PARSER;
       }
 
+      @java.lang.Override
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage getDefaultInstanceForType() {
         return DEFAULT_INSTANCE;
       }
@@ -981,7 +1071,9 @@ public final class TestMsg {
     private volatile java.lang.Object foo_;
     /**
      * <code>string foo = 1;</code>
+     * @return The foo.
      */
+    @java.lang.Override
     public java.lang.String getFoo() {
       java.lang.Object ref = foo_;
       if (ref instanceof java.lang.String) {
@@ -996,7 +1088,9 @@ public final class TestMsg {
     }
     /**
      * <code>string foo = 1;</code>
+     * @return The bytes for foo.
      */
+    @java.lang.Override
     public com.google.protobuf.ByteString
         getFooBytes() {
       java.lang.Object ref = foo_;
@@ -1015,7 +1109,9 @@ public final class TestMsg {
     private double bar_;
     /**
      * <code>double bar = 2;</code>
+     * @return The bar.
      */
+    @java.lang.Override
     public double getBar() {
       return bar_;
     }
@@ -1024,24 +1120,30 @@ public final class TestMsg {
     private org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage nestedMessage_;
     /**
      * <code>.proto.SubMessage.NestedMessage nestedMessage = 3;</code>
+     * @return Whether the nestedMessage field is set.
      */
+    @java.lang.Override
     public boolean hasNestedMessage() {
       return nestedMessage_ != null;
     }
     /**
      * <code>.proto.SubMessage.NestedMessage nestedMessage = 3;</code>
+     * @return The nestedMessage.
      */
+    @java.lang.Override
     public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage getNestedMessage() {
       return nestedMessage_ == null ? org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage.getDefaultInstance() : nestedMessage_;
     }
     /**
      * <code>.proto.SubMessage.NestedMessage nestedMessage = 3;</code>
      */
+    @java.lang.Override
     public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessageOrBuilder getNestedMessageOrBuilder() {
       return getNestedMessage();
     }
 
     private byte memoizedIsInitialized = -1;
+    @java.lang.Override
     public final boolean isInitialized() {
       byte isInitialized = memoizedIsInitialized;
       if (isInitialized == 1) return true;
@@ -1051,6 +1153,7 @@ public final class TestMsg {
       return true;
     }
 
+    @java.lang.Override
     public void writeTo(com.google.protobuf.CodedOutputStream output)
                         throws java.io.IOException {
       if (!getFooBytes().isEmpty()) {
@@ -1065,6 +1168,7 @@ public final class TestMsg {
       unknownFields.writeTo(output);
     }
 
+    @java.lang.Override
     public int getSerializedSize() {
       int size = memoizedSize;
       if (size != -1) return size;
@@ -1096,20 +1200,18 @@ public final class TestMsg {
       }
       org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage other = (org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage) obj;
 
-      boolean result = true;
-      result = result && getFoo()
-          .equals(other.getFoo());
-      result = result && (
-          java.lang.Double.doubleToLongBits(getBar())
-          == java.lang.Double.doubleToLongBits(
-              other.getBar()));
-      result = result && (hasNestedMessage() == other.hasNestedMessage());
+      if (!getFoo()
+          .equals(other.getFoo())) return false;
+      if (java.lang.Double.doubleToLongBits(getBar())
+          != java.lang.Double.doubleToLongBits(
+              other.getBar())) return false;
+      if (hasNestedMessage() != other.hasNestedMessage()) return false;
       if (hasNestedMessage()) {
-        result = result && getNestedMessage()
-            .equals(other.getNestedMessage());
+        if (!getNestedMessage()
+            .equals(other.getNestedMessage())) return false;
       }
-      result = result && unknownFields.equals(other.unknownFields);
-      return result;
+      if (!unknownFields.equals(other.unknownFields)) return false;
+      return true;
     }
 
     @java.lang.Override
@@ -1203,6 +1305,7 @@ public final class TestMsg {
           .parseWithIOException(PARSER, input, extensionRegistry);
     }
 
+    @java.lang.Override
     public Builder newBuilderForType() { return newBuilder(); }
     public static Builder newBuilder() {
       return DEFAULT_INSTANCE.toBuilder();
@@ -1210,6 +1313,7 @@ public final class TestMsg {
     public static Builder newBuilder(org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage prototype) {
       return DEFAULT_INSTANCE.toBuilder().mergeFrom(prototype);
     }
+    @java.lang.Override
     public Builder toBuilder() {
       return this == DEFAULT_INSTANCE
           ? new Builder() : new Builder().mergeFrom(this);
@@ -1233,6 +1337,7 @@ public final class TestMsg {
         return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_descriptor;
       }
 
+      @java.lang.Override
       protected com.google.protobuf.GeneratedMessageV3.FieldAccessorTable
           internalGetFieldAccessorTable() {
         return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_fieldAccessorTable
@@ -1255,6 +1360,7 @@ public final class TestMsg {
                 .alwaysUseFieldBuilders) {
         }
       }
+      @java.lang.Override
       public Builder clear() {
         super.clear();
         foo_ = "";
@@ -1270,15 +1376,18 @@ public final class TestMsg {
         return this;
       }
 
+      @java.lang.Override
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
         return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_SubMessage_descriptor;
       }
 
+      @java.lang.Override
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage getDefaultInstanceForType() {
         return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.getDefaultInstance();
       }
 
+      @java.lang.Override
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage build() {
         org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage result = buildPartial();
         if (!result.isInitialized()) {
@@ -1287,6 +1396,7 @@ public final class TestMsg {
         return result;
       }
 
+      @java.lang.Override
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage buildPartial() {
         org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage result = new org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage(this);
         result.foo_ = foo_;
@@ -1300,32 +1410,39 @@ public final class TestMsg {
         return result;
       }
 
+      @java.lang.Override
       public Builder clone() {
-        return (Builder) super.clone();
+        return super.clone();
       }
+      @java.lang.Override
       public Builder setField(
           com.google.protobuf.Descriptors.FieldDescriptor field,
           java.lang.Object value) {
-        return (Builder) super.setField(field, value);
+        return super.setField(field, value);
       }
+      @java.lang.Override
       public Builder clearField(
           com.google.protobuf.Descriptors.FieldDescriptor field) {
-        return (Builder) super.clearField(field);
+        return super.clearField(field);
       }
+      @java.lang.Override
       public Builder clearOneof(
           com.google.protobuf.Descriptors.OneofDescriptor oneof) {
-        return (Builder) super.clearOneof(oneof);
+        return super.clearOneof(oneof);
       }
+      @java.lang.Override
       public Builder setRepeatedField(
           com.google.protobuf.Descriptors.FieldDescriptor field,
           int index, java.lang.Object value) {
-        return (Builder) super.setRepeatedField(field, index, value);
+        return super.setRepeatedField(field, index, value);
       }
+      @java.lang.Override
       public Builder addRepeatedField(
           com.google.protobuf.Descriptors.FieldDescriptor field,
           java.lang.Object value) {
-        return (Builder) super.addRepeatedField(field, value);
+        return super.addRepeatedField(field, value);
       }
+      @java.lang.Override
       public Builder mergeFrom(com.google.protobuf.Message other) {
         if (other instanceof org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage) {
           return mergeFrom((org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage)other);
@@ -1352,10 +1469,12 @@ public final class TestMsg {
         return this;
       }
 
+      @java.lang.Override
       public final boolean isInitialized() {
         return true;
       }
 
+      @java.lang.Override
       public Builder mergeFrom(
           com.google.protobuf.CodedInputStream input,
           com.google.protobuf.ExtensionRegistryLite extensionRegistry)
@@ -1377,6 +1496,7 @@ public final class TestMsg {
       private java.lang.Object foo_ = "";
       /**
        * <code>string foo = 1;</code>
+       * @return The foo.
        */
       public java.lang.String getFoo() {
         java.lang.Object ref = foo_;
@@ -1392,6 +1512,7 @@ public final class TestMsg {
       }
       /**
        * <code>string foo = 1;</code>
+       * @return The bytes for foo.
        */
       public com.google.protobuf.ByteString
           getFooBytes() {
@@ -1408,6 +1529,8 @@ public final class TestMsg {
       }
       /**
        * <code>string foo = 1;</code>
+       * @param value The foo to set.
+       * @return This builder for chaining.
        */
       public Builder setFoo(
           java.lang.String value) {
@@ -1421,6 +1544,7 @@ public final class TestMsg {
       }
       /**
        * <code>string foo = 1;</code>
+       * @return This builder for chaining.
        */
       public Builder clearFoo() {
         
@@ -1430,6 +1554,8 @@ public final class TestMsg {
       }
       /**
        * <code>string foo = 1;</code>
+       * @param value The bytes for foo to set.
+       * @return This builder for chaining.
        */
       public Builder setFooBytes(
           com.google.protobuf.ByteString value) {
@@ -1446,12 +1572,16 @@ public final class TestMsg {
       private double bar_ ;
       /**
        * <code>double bar = 2;</code>
+       * @return The bar.
        */
+      @java.lang.Override
       public double getBar() {
         return bar_;
       }
       /**
        * <code>double bar = 2;</code>
+       * @param value The bar to set.
+       * @return This builder for chaining.
        */
       public Builder setBar(double value) {
         
@@ -1461,6 +1591,7 @@ public final class TestMsg {
       }
       /**
        * <code>double bar = 2;</code>
+       * @return This builder for chaining.
        */
       public Builder clearBar() {
         
@@ -1469,17 +1600,19 @@ public final class TestMsg {
         return this;
       }
 
-      private org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage nestedMessage_ = null;
+      private org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage nestedMessage_;
       private com.google.protobuf.SingleFieldBuilderV3<
           org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage, org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage.Builder, org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessageOrBuilder> nestedMessageBuilder_;
       /**
        * <code>.proto.SubMessage.NestedMessage nestedMessage = 3;</code>
+       * @return Whether the nestedMessage field is set.
        */
       public boolean hasNestedMessage() {
         return nestedMessageBuilder_ != null || nestedMessage_ != null;
       }
       /**
        * <code>.proto.SubMessage.NestedMessage nestedMessage = 3;</code>
+       * @return The nestedMessage.
        */
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.NestedMessage getNestedMessage() {
         if (nestedMessageBuilder_ == null) {
@@ -1585,11 +1718,13 @@ public final class TestMsg {
         }
         return nestedMessageBuilder_;
       }
+      @java.lang.Override
       public final Builder setUnknownFields(
           final com.google.protobuf.UnknownFieldSet unknownFields) {
-        return super.setUnknownFieldsProto3(unknownFields);
+        return super.setUnknownFields(unknownFields);
       }
 
+      @java.lang.Override
       public final Builder mergeUnknownFields(
           final com.google.protobuf.UnknownFieldSet unknownFields) {
         return super.mergeUnknownFields(unknownFields);
@@ -1611,6 +1746,7 @@ public final class TestMsg {
 
     private static final com.google.protobuf.Parser<SubMessage>
         PARSER = new com.google.protobuf.AbstractParser<SubMessage>() {
+      @java.lang.Override
       public SubMessage parsePartialFrom(
           com.google.protobuf.CodedInputStream input,
           com.google.protobuf.ExtensionRegistryLite extensionRegistry)
@@ -1628,6 +1764,7 @@ public final class TestMsg {
       return PARSER;
     }
 
+    @java.lang.Override
     public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage getDefaultInstanceForType() {
       return DEFAULT_INSTANCE;
     }
@@ -1640,99 +1777,119 @@ public final class TestMsg {
 
     /**
      * <code>string stringField = 1;</code>
+     * @return The stringField.
      */
     java.lang.String getStringField();
     /**
      * <code>string stringField = 1;</code>
+     * @return The bytes for stringField.
      */
     com.google.protobuf.ByteString
         getStringFieldBytes();
 
     /**
      * <code>double doubleField = 2;</code>
+     * @return The doubleField.
      */
     double getDoubleField();
 
     /**
      * <code>float floatField = 3;</code>
+     * @return The floatField.
      */
     float getFloatField();
 
     /**
      * <code>int32 int32Field = 4;</code>
+     * @return The int32Field.
      */
     int getInt32Field();
 
     /**
      * <code>int64 int64Field = 5;</code>
+     * @return The int64Field.
      */
     long getInt64Field();
 
     /**
      * <code>uint32 uint32Field = 6;</code>
+     * @return The uint32Field.
      */
     int getUint32Field();
 
     /**
      * <code>uint64 uint64Field = 7;</code>
+     * @return The uint64Field.
      */
     long getUint64Field();
 
     /**
      * <code>sint32 sint32Field = 8;</code>
+     * @return The sint32Field.
      */
     int getSint32Field();
 
     /**
      * <code>sint64 sint64Field = 9;</code>
+     * @return The sint64Field.
      */
     long getSint64Field();
 
     /**
      * <code>fixed32 fixed32Field = 10;</code>
+     * @return The fixed32Field.
      */
     int getFixed32Field();
 
     /**
      * <code>fixed64 fixed64Field = 11;</code>
+     * @return The fixed64Field.
      */
     long getFixed64Field();
 
     /**
      * <code>sfixed32 sfixed32Field = 12;</code>
+     * @return The sfixed32Field.
      */
     int getSfixed32Field();
 
     /**
      * <code>sfixed64 sfixed64Field = 13;</code>
+     * @return The sfixed64Field.
      */
     long getSfixed64Field();
 
     /**
      * <code>bool boolField = 14;</code>
+     * @return The boolField.
      */
     boolean getBoolField();
 
     /**
      * <code>bytes bytesField = 15;</code>
+     * @return The bytesField.
      */
     com.google.protobuf.ByteString getBytesField();
 
     /**
      * <code>.proto.TestEnum testEnum = 16;</code>
+     * @return The enum numeric value on the wire for testEnum.
      */
     int getTestEnumValue();
     /**
      * <code>.proto.TestEnum testEnum = 16;</code>
+     * @return The testEnum.
      */
     org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum getTestEnum();
 
     /**
      * <code>.proto.SubMessage subMessage = 17;</code>
+     * @return Whether the subMessage field is set.
      */
     boolean hasSubMessage();
     /**
      * <code>.proto.SubMessage subMessage = 17;</code>
+     * @return The subMessage.
      */
     org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage getSubMessage();
     /**
@@ -1742,19 +1899,25 @@ public final class TestMsg {
 
     /**
      * <code>repeated string repeatedField = 18;</code>
+     * @return A list containing the repeatedField.
      */
     java.util.List<java.lang.String>
         getRepeatedFieldList();
     /**
      * <code>repeated string repeatedField = 18;</code>
+     * @return The count of repeatedField.
      */
     int getRepeatedFieldCount();
     /**
      * <code>repeated string repeatedField = 18;</code>
+     * @param index The index of the element to return.
+     * @return The repeatedField at the given index.
      */
     java.lang.String getRepeatedField(int index);
     /**
      * <code>repeated string repeatedField = 18;</code>
+     * @param index The index of the value to return.
+     * @return The bytes of the repeatedField at the given index.
      */
     com.google.protobuf.ByteString
         getRepeatedFieldBytes(int index);
@@ -1792,11 +1955,26 @@ public final class TestMsg {
 
     double getMapFieldOrThrow(
         java.lang.String key);
+
+    /**
+     * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+     * @return Whether the timestampField field is set.
+     */
+    boolean hasTimestampField();
+    /**
+     * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+     * @return The timestampField.
+     */
+    com.google.protobuf.Timestamp getTimestampField();
+    /**
+     * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+     */
+    com.google.protobuf.TimestampOrBuilder getTimestampFieldOrBuilder();
   }
   /**
    * Protobuf type {@code proto.TestMessage}
    */
-  public  static final class TestMessage extends
+  public static final class TestMessage extends
       com.google.protobuf.GeneratedMessageV3 implements
       // @@protoc_insertion_point(message_implements:proto.TestMessage)
       TestMessageOrBuilder {
@@ -1807,25 +1985,19 @@ public final class TestMsg {
     }
     private TestMessage() {
       stringField_ = "";
-      doubleField_ = 0D;
-      floatField_ = 0F;
-      int32Field_ = 0;
-      int64Field_ = 0L;
-      uint32Field_ = 0;
-      uint64Field_ = 0L;
-      sint32Field_ = 0;
-      sint64Field_ = 0L;
-      fixed32Field_ = 0;
-      fixed64Field_ = 0L;
-      sfixed32Field_ = 0;
-      sfixed64Field_ = 0L;
-      boolField_ = false;
       bytesField_ = com.google.protobuf.ByteString.EMPTY;
       testEnum_ = 0;
       repeatedField_ = com.google.protobuf.LazyStringArrayList.EMPTY;
     }
 
     @java.lang.Override
+    @SuppressWarnings({"unused"})
+    protected java.lang.Object newInstance(
+        UnusedPrivateParameter unused) {
+      return new TestMessage();
+    }
+
+    @java.lang.Override
     public final com.google.protobuf.UnknownFieldSet
     getUnknownFields() {
       return this.unknownFields;
@@ -1849,13 +2021,6 @@ public final class TestMsg {
             case 0:
               done = true;
               break;
-            default: {
-              if (!parseUnknownFieldProto3(
-                  input, unknownFields, extensionRegistry, tag)) {
-                done = true;
-              }
-              break;
-            }
             case 10: {
               java.lang.String s = input.readStringRequireUtf8();
 
@@ -1953,18 +2118,18 @@ public final class TestMsg {
             }
             case 146: {
               java.lang.String s = input.readStringRequireUtf8();
-              if (!((mutable_bitField0_ & 0x00020000) == 0x00020000)) {
+              if (!((mutable_bitField0_ & 0x00000001) != 0)) {
                 repeatedField_ = new com.google.protobuf.LazyStringArrayList();
-                mutable_bitField0_ |= 0x00020000;
+                mutable_bitField0_ |= 0x00000001;
               }
               repeatedField_.add(s);
               break;
             }
             case 154: {
-              if (!((mutable_bitField0_ & 0x00040000) == 0x00040000)) {
+              if (!((mutable_bitField0_ & 0x00000002) != 0)) {
                 mapField_ = com.google.protobuf.MapField.newMapField(
                     MapFieldDefaultEntryHolder.defaultEntry);
-                mutable_bitField0_ |= 0x00040000;
+                mutable_bitField0_ |= 0x00000002;
               }
               com.google.protobuf.MapEntry<java.lang.String, java.lang.Double>
               mapField__ = input.readMessage(
@@ -1973,6 +2138,26 @@ public final class TestMsg {
                   mapField__.getKey(), mapField__.getValue());
               break;
             }
+            case 162: {
+              com.google.protobuf.Timestamp.Builder subBuilder = null;
+              if (timestampField_ != null) {
+                subBuilder = timestampField_.toBuilder();
+              }
+              timestampField_ = input.readMessage(com.google.protobuf.Timestamp.parser(), extensionRegistry);
+              if (subBuilder != null) {
+                subBuilder.mergeFrom(timestampField_);
+                timestampField_ = subBuilder.buildPartial();
+              }
+
+              break;
+            }
+            default: {
+              if (!parseUnknownField(
+                  input, unknownFields, extensionRegistry, tag)) {
+                done = true;
+              }
+              break;
+            }
           }
         }
       } catch (com.google.protobuf.InvalidProtocolBufferException e) {
@@ -1981,7 +2166,7 @@ public final class TestMsg {
         throw new com.google.protobuf.InvalidProtocolBufferException(
             e).setUnfinishedMessage(this);
       } finally {
-        if (((mutable_bitField0_ & 0x00020000) == 0x00020000)) {
+        if (((mutable_bitField0_ & 0x00000001) != 0)) {
           repeatedField_ = repeatedField_.getUnmodifiableView();
         }
         this.unknownFields = unknownFields.build();
@@ -1994,6 +2179,7 @@ public final class TestMsg {
     }
 
     @SuppressWarnings({"rawtypes"})
+    @java.lang.Override
     protected com.google.protobuf.MapField internalGetMapField(
         int number) {
       switch (number) {
@@ -2004,6 +2190,7 @@ public final class TestMsg {
               "Invalid map field number: " + number);
       }
     }
+    @java.lang.Override
     protected com.google.protobuf.GeneratedMessageV3.FieldAccessorTable
         internalGetFieldAccessorTable() {
       return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_TestMessage_fieldAccessorTable
@@ -2011,12 +2198,13 @@ public final class TestMsg {
               org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage.class, org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage.Builder.class);
     }
 
-    private int bitField0_;
     public static final int STRINGFIELD_FIELD_NUMBER = 1;
     private volatile java.lang.Object stringField_;
     /**
      * <code>string stringField = 1;</code>
+     * @return The stringField.
      */
+    @java.lang.Override
     public java.lang.String getStringField() {
       java.lang.Object ref = stringField_;
       if (ref instanceof java.lang.String) {
@@ -2031,7 +2219,9 @@ public final class TestMsg {
     }
     /**
      * <code>string stringField = 1;</code>
+     * @return The bytes for stringField.
      */
+    @java.lang.Override
     public com.google.protobuf.ByteString
         getStringFieldBytes() {
       java.lang.Object ref = stringField_;
@@ -2050,7 +2240,9 @@ public final class TestMsg {
     private double doubleField_;
     /**
      * <code>double doubleField = 2;</code>
+     * @return The doubleField.
      */
+    @java.lang.Override
     public double getDoubleField() {
       return doubleField_;
     }
@@ -2059,7 +2251,9 @@ public final class TestMsg {
     private float floatField_;
     /**
      * <code>float floatField = 3;</code>
+     * @return The floatField.
      */
+    @java.lang.Override
     public float getFloatField() {
       return floatField_;
     }
@@ -2068,7 +2262,9 @@ public final class TestMsg {
     private int int32Field_;
     /**
      * <code>int32 int32Field = 4;</code>
+     * @return The int32Field.
      */
+    @java.lang.Override
     public int getInt32Field() {
       return int32Field_;
     }
@@ -2077,7 +2273,9 @@ public final class TestMsg {
     private long int64Field_;
     /**
      * <code>int64 int64Field = 5;</code>
+     * @return The int64Field.
      */
+    @java.lang.Override
     public long getInt64Field() {
       return int64Field_;
     }
@@ -2086,7 +2284,9 @@ public final class TestMsg {
     private int uint32Field_;
     /**
      * <code>uint32 uint32Field = 6;</code>
+     * @return The uint32Field.
      */
+    @java.lang.Override
     public int getUint32Field() {
       return uint32Field_;
     }
@@ -2095,7 +2295,9 @@ public final class TestMsg {
     private long uint64Field_;
     /**
      * <code>uint64 uint64Field = 7;</code>
+     * @return The uint64Field.
      */
+    @java.lang.Override
     public long getUint64Field() {
       return uint64Field_;
     }
@@ -2104,7 +2306,9 @@ public final class TestMsg {
     private int sint32Field_;
     /**
      * <code>sint32 sint32Field = 8;</code>
+     * @return The sint32Field.
      */
+    @java.lang.Override
     public int getSint32Field() {
       return sint32Field_;
     }
@@ -2113,7 +2317,9 @@ public final class TestMsg {
     private long sint64Field_;
     /**
      * <code>sint64 sint64Field = 9;</code>
+     * @return The sint64Field.
      */
+    @java.lang.Override
     public long getSint64Field() {
       return sint64Field_;
     }
@@ -2122,7 +2328,9 @@ public final class TestMsg {
     private int fixed32Field_;
     /**
      * <code>fixed32 fixed32Field = 10;</code>
+     * @return The fixed32Field.
      */
+    @java.lang.Override
     public int getFixed32Field() {
       return fixed32Field_;
     }
@@ -2131,7 +2339,9 @@ public final class TestMsg {
     private long fixed64Field_;
     /**
      * <code>fixed64 fixed64Field = 11;</code>
+     * @return The fixed64Field.
      */
+    @java.lang.Override
     public long getFixed64Field() {
       return fixed64Field_;
     }
@@ -2140,7 +2350,9 @@ public final class TestMsg {
     private int sfixed32Field_;
     /**
      * <code>sfixed32 sfixed32Field = 12;</code>
+     * @return The sfixed32Field.
      */
+    @java.lang.Override
     public int getSfixed32Field() {
       return sfixed32Field_;
     }
@@ -2149,7 +2361,9 @@ public final class TestMsg {
     private long sfixed64Field_;
     /**
      * <code>sfixed64 sfixed64Field = 13;</code>
+     * @return The sfixed64Field.
      */
+    @java.lang.Override
     public long getSfixed64Field() {
       return sfixed64Field_;
     }
@@ -2158,7 +2372,9 @@ public final class TestMsg {
     private boolean boolField_;
     /**
      * <code>bool boolField = 14;</code>
+     * @return The boolField.
      */
+    @java.lang.Override
     public boolean getBoolField() {
       return boolField_;
     }
@@ -2167,7 +2383,9 @@ public final class TestMsg {
     private com.google.protobuf.ByteString bytesField_;
     /**
      * <code>bytes bytesField = 15;</code>
+     * @return The bytesField.
      */
+    @java.lang.Override
     public com.google.protobuf.ByteString getBytesField() {
       return bytesField_;
     }
@@ -2176,14 +2394,17 @@ public final class TestMsg {
     private int testEnum_;
     /**
      * <code>.proto.TestEnum testEnum = 16;</code>
+     * @return The enum numeric value on the wire for testEnum.
      */
-    public int getTestEnumValue() {
+    @java.lang.Override public int getTestEnumValue() {
       return testEnum_;
     }
     /**
      * <code>.proto.TestEnum testEnum = 16;</code>
+     * @return The testEnum.
      */
-    public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum getTestEnum() {
+    @java.lang.Override public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum getTestEnum() {
+      @SuppressWarnings("deprecation")
       org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum result = org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum.valueOf(testEnum_);
       return result == null ? org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum.UNRECOGNIZED : result;
     }
@@ -2192,19 +2413,24 @@ public final class TestMsg {
     private org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage subMessage_;
     /**
      * <code>.proto.SubMessage subMessage = 17;</code>
+     * @return Whether the subMessage field is set.
      */
+    @java.lang.Override
     public boolean hasSubMessage() {
       return subMessage_ != null;
     }
     /**
      * <code>.proto.SubMessage subMessage = 17;</code>
+     * @return The subMessage.
      */
+    @java.lang.Override
     public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage getSubMessage() {
       return subMessage_ == null ? org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.getDefaultInstance() : subMessage_;
     }
     /**
      * <code>.proto.SubMessage subMessage = 17;</code>
      */
+    @java.lang.Override
     public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessageOrBuilder getSubMessageOrBuilder() {
       return getSubMessage();
     }
@@ -2213,6 +2439,7 @@ public final class TestMsg {
     private com.google.protobuf.LazyStringList repeatedField_;
     /**
      * <code>repeated string repeatedField = 18;</code>
+     * @return A list containing the repeatedField.
      */
     public com.google.protobuf.ProtocolStringList
         getRepeatedFieldList() {
@@ -2220,18 +2447,23 @@ public final class TestMsg {
     }
     /**
      * <code>repeated string repeatedField = 18;</code>
+     * @return The count of repeatedField.
      */
     public int getRepeatedFieldCount() {
       return repeatedField_.size();
     }
     /**
      * <code>repeated string repeatedField = 18;</code>
+     * @param index The index of the element to return.
+     * @return The repeatedField at the given index.
      */
     public java.lang.String getRepeatedField(int index) {
       return repeatedField_.get(index);
     }
     /**
      * <code>repeated string repeatedField = 18;</code>
+     * @param index The index of the value to return.
+     * @return The bytes of the repeatedField at the given index.
      */
     public com.google.protobuf.ByteString
         getRepeatedFieldBytes(int index) {
@@ -2268,6 +2500,7 @@ public final class TestMsg {
      * <code>map&lt;string, double&gt; mapField = 19;</code>
      */
 
+    @java.lang.Override
     public boolean containsMapField(
         java.lang.String key) {
       if (key == null) { throw new java.lang.NullPointerException(); }
@@ -2276,6 +2509,7 @@ public final class TestMsg {
     /**
      * Use {@link #getMapFieldMap()} instead.
      */
+    @java.lang.Override
     @java.lang.Deprecated
     public java.util.Map<java.lang.String, java.lang.Double> getMapField() {
       return getMapFieldMap();
@@ -2283,6 +2517,7 @@ public final class TestMsg {
     /**
      * <code>map&lt;string, double&gt; mapField = 19;</code>
      */
+    @java.lang.Override
 
     public java.util.Map<java.lang.String, java.lang.Double> getMapFieldMap() {
       return internalGetMapField().getMap();
@@ -2290,6 +2525,7 @@ public final class TestMsg {
     /**
      * <code>map&lt;string, double&gt; mapField = 19;</code>
      */
+    @java.lang.Override
 
     public double getMapFieldOrDefault(
         java.lang.String key,
@@ -2302,6 +2538,7 @@ public final class TestMsg {
     /**
      * <code>map&lt;string, double&gt; mapField = 19;</code>
      */
+    @java.lang.Override
 
     public double getMapFieldOrThrow(
         java.lang.String key) {
@@ -2314,7 +2551,34 @@ public final class TestMsg {
       return map.get(key);
     }
 
+    public static final int TIMESTAMPFIELD_FIELD_NUMBER = 20;
+    private com.google.protobuf.Timestamp timestampField_;
+    /**
+     * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+     * @return Whether the timestampField field is set.
+     */
+    @java.lang.Override
+    public boolean hasTimestampField() {
+      return timestampField_ != null;
+    }
+    /**
+     * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+     * @return The timestampField.
+     */
+    @java.lang.Override
+    public com.google.protobuf.Timestamp getTimestampField() {
+      return timestampField_ == null ? com.google.protobuf.Timestamp.getDefaultInstance() : timestampField_;
+    }
+    /**
+     * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+     */
+    @java.lang.Override
+    public com.google.protobuf.TimestampOrBuilder getTimestampFieldOrBuilder() {
+      return getTimestampField();
+    }
+
     private byte memoizedIsInitialized = -1;
+    @java.lang.Override
     public final boolean isInitialized() {
       byte isInitialized = memoizedIsInitialized;
       if (isInitialized == 1) return true;
@@ -2324,6 +2588,7 @@ public final class TestMsg {
       return true;
     }
 
+    @java.lang.Override
     public void writeTo(com.google.protobuf.CodedOutputStream output)
                         throws java.io.IOException {
       if (!getStringFieldBytes().isEmpty()) {
@@ -2386,9 +2651,13 @@ public final class TestMsg {
           internalGetMapField(),
           MapFieldDefaultEntryHolder.defaultEntry,
           19);
+      if (timestampField_ != null) {
+        output.writeMessage(20, getTimestampField());
+      }
       unknownFields.writeTo(output);
     }
 
+    @java.lang.Override
     public int getSerializedSize() {
       int size = memoizedSize;
       if (size != -1) return size;
@@ -2479,6 +2748,10 @@ public final class TestMsg {
         size += com.google.protobuf.CodedOutputStream
             .computeMessageSize(19, mapField__);
       }
+      if (timestampField_ != null) {
+        size += com.google.protobuf.CodedOutputStream
+          .computeMessageSize(20, getTimestampField());
+      }
       size += unknownFields.getSerializedSize();
       memoizedSize = size;
       return size;
@@ -2494,53 +2767,55 @@ public final class TestMsg {
       }
       org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage other = (org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage) obj;
 
-      boolean result = true;
-      result = result && getStringField()
-          .equals(other.getStringField());
-      result = result && (
-          java.lang.Double.doubleToLongBits(getDoubleField())
-          == java.lang.Double.doubleToLongBits(
-              other.getDoubleField()));
-      result = result && (
-          java.lang.Float.floatToIntBits(getFloatField())
-          == java.lang.Float.floatToIntBits(
-              other.getFloatField()));
-      result = result && (getInt32Field()
-          == other.getInt32Field());
-      result = result && (getInt64Field()
-          == other.getInt64Field());
-      result = result && (getUint32Field()
-          == other.getUint32Field());
-      result = result && (getUint64Field()
-          == other.getUint64Field());
-      result = result && (getSint32Field()
-          == other.getSint32Field());
-      result = result && (getSint64Field()
-          == other.getSint64Field());
-      result = result && (getFixed32Field()
-          == other.getFixed32Field());
-      result = result && (getFixed64Field()
-          == other.getFixed64Field());
-      result = result && (getSfixed32Field()
-          == other.getSfixed32Field());
-      result = result && (getSfixed64Field()
-          == other.getSfixed64Field());
-      result = result && (getBoolField()
-          == other.getBoolField());
-      result = result && getBytesField()
-          .equals(other.getBytesField());
-      result = result && testEnum_ == other.testEnum_;
-      result = result && (hasSubMessage() == other.hasSubMessage());
+      if (!getStringField()
+          .equals(other.getStringField())) return false;
+      if (java.lang.Double.doubleToLongBits(getDoubleField())
+          != java.lang.Double.doubleToLongBits(
+              other.getDoubleField())) return false;
+      if (java.lang.Float.floatToIntBits(getFloatField())
+          != java.lang.Float.floatToIntBits(
+              other.getFloatField())) return false;
+      if (getInt32Field()
+          != other.getInt32Field()) return false;
+      if (getInt64Field()
+          != other.getInt64Field()) return false;
+      if (getUint32Field()
+          != other.getUint32Field()) return false;
+      if (getUint64Field()
+          != other.getUint64Field()) return false;
+      if (getSint32Field()
+          != other.getSint32Field()) return false;
+      if (getSint64Field()
+          != other.getSint64Field()) return false;
+      if (getFixed32Field()
+          != other.getFixed32Field()) return false;
+      if (getFixed64Field()
+          != other.getFixed64Field()) return false;
+      if (getSfixed32Field()
+          != other.getSfixed32Field()) return false;
+      if (getSfixed64Field()
+          != other.getSfixed64Field()) return false;
+      if (getBoolField()
+          != other.getBoolField()) return false;
+      if (!getBytesField()
+          .equals(other.getBytesField())) return false;
+      if (testEnum_ != other.testEnum_) return false;
+      if (hasSubMessage() != other.hasSubMessage()) return false;
       if (hasSubMessage()) {
-        result = result && getSubMessage()
-            .equals(other.getSubMessage());
-      }
-      result = result && getRepeatedFieldList()
-          .equals(other.getRepeatedFieldList());
-      result = result && internalGetMapField().equals(
-          other.internalGetMapField());
-      result = result && unknownFields.equals(other.unknownFields);
-      return result;
+        if (!getSubMessage()
+            .equals(other.getSubMessage())) return false;
+      }
+      if (!getRepeatedFieldList()
+          .equals(other.getRepeatedFieldList())) return false;
+      if (!internalGetMapField().equals(
+          other.internalGetMapField())) return false;
+      if (hasTimestampField() != other.hasTimestampField()) return false;
+      if (hasTimestampField()) {
+        if (!getTimestampField()
+            .equals(other.getTimestampField())) return false;
+      }
+      if (!unknownFields.equals(other.unknownFields)) return false;
+      return true;
     }
 
     @java.lang.Override
@@ -2602,6 +2877,10 @@ public final class TestMsg {
         hash = (37 * hash) + MAPFIELD_FIELD_NUMBER;
         hash = (53 * hash) + internalGetMapField().hashCode();
       }
+      if (hasTimestampField()) {
+        hash = (37 * hash) + TIMESTAMPFIELD_FIELD_NUMBER;
+        hash = (53 * hash) + getTimestampField().hashCode();
+      }
       hash = (29 * hash) + unknownFields.hashCode();
       memoizedHashCode = hash;
       return hash;
@@ -2677,6 +2956,7 @@ public final class TestMsg {
           .parseWithIOException(PARSER, input, extensionRegistry);
     }
 
+    @java.lang.Override
     public Builder newBuilderForType() { return newBuilder(); }
     public static Builder newBuilder() {
       return DEFAULT_INSTANCE.toBuilder();
@@ -2684,6 +2964,7 @@ public final class TestMsg {
     public static Builder newBuilder(org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage prototype) {
       return DEFAULT_INSTANCE.toBuilder().mergeFrom(prototype);
     }
+    @java.lang.Override
     public Builder toBuilder() {
       return this == DEFAULT_INSTANCE
           ? new Builder() : new Builder().mergeFrom(this);
@@ -2729,6 +3010,7 @@ public final class TestMsg {
                 "Invalid map field number: " + number);
         }
       }
+      @java.lang.Override
       protected com.google.protobuf.GeneratedMessageV3.FieldAccessorTable
           internalGetFieldAccessorTable() {
         return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_TestMessage_fieldAccessorTable
@@ -2751,6 +3033,7 @@ public final class TestMsg {
                 .alwaysUseFieldBuilders) {
         }
       }
+      @java.lang.Override
       public Builder clear() {
         super.clear();
         stringField_ = "";
@@ -2792,20 +3075,29 @@ public final class TestMsg {
           subMessageBuilder_ = null;
         }
         repeatedField_ = com.google.protobuf.LazyStringArrayList.EMPTY;
-        bitField0_ = (bitField0_ & ~0x00020000);
+        bitField0_ = (bitField0_ & ~0x00000001);
         internalGetMutableMapField().clear();
+        if (timestampFieldBuilder_ == null) {
+          timestampField_ = null;
+        } else {
+          timestampField_ = null;
+          timestampFieldBuilder_ = null;
+        }
         return this;
       }
 
+      @java.lang.Override
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
         return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.internal_static_proto_TestMessage_descriptor;
       }
 
+      @java.lang.Override
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage getDefaultInstanceForType() {
         return org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage.getDefaultInstance();
       }
 
+      @java.lang.Override
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage build() {
         org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage result = buildPartial();
         if (!result.isInitialized()) {
@@ -2814,10 +3106,10 @@ public final class TestMsg {
         return result;
       }
 
+      @java.lang.Override
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage buildPartial() {
         org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage result = new org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage(this);
         int from_bitField0_ = bitField0_;
-        int to_bitField0_ = 0;
         result.stringField_ = stringField_;
         result.doubleField_ = doubleField_;
         result.floatField_ = floatField_;
@@ -2839,44 +3131,55 @@ public final class TestMsg {
         } else {
           result.subMessage_ = subMessageBuilder_.build();
         }
-        if (((bitField0_ & 0x00020000) == 0x00020000)) {
+        if (((bitField0_ & 0x00000001) != 0)) {
           repeatedField_ = repeatedField_.getUnmodifiableView();
-          bitField0_ = (bitField0_ & ~0x00020000);
+          bitField0_ = (bitField0_ & ~0x00000001);
         }
         result.repeatedField_ = repeatedField_;
         result.mapField_ = internalGetMapField();
         result.mapField_.makeImmutable();
-        result.bitField0_ = to_bitField0_;
+        if (timestampFieldBuilder_ == null) {
+          result.timestampField_ = timestampField_;
+        } else {
+          result.timestampField_ = timestampFieldBuilder_.build();
+        }
         onBuilt();
         return result;
       }
 
+      @java.lang.Override
       public Builder clone() {
-        return (Builder) super.clone();
+        return super.clone();
       }
+      @java.lang.Override
       public Builder setField(
           com.google.protobuf.Descriptors.FieldDescriptor field,
           java.lang.Object value) {
-        return (Builder) super.setField(field, value);
+        return super.setField(field, value);
       }
+      @java.lang.Override
       public Builder clearField(
           com.google.protobuf.Descriptors.FieldDescriptor field) {
-        return (Builder) super.clearField(field);
+        return super.clearField(field);
       }
+      @java.lang.Override
       public Builder clearOneof(
           com.google.protobuf.Descriptors.OneofDescriptor oneof) {
-        return (Builder) super.clearOneof(oneof);
+        return super.clearOneof(oneof);
       }
+      @java.lang.Override
       public Builder setRepeatedField(
           com.google.protobuf.Descriptors.FieldDescriptor field,
           int index, java.lang.Object value) {
-        return (Builder) super.setRepeatedField(field, index, value);
+        return super.setRepeatedField(field, index, value);
       }
+      @java.lang.Override
       public Builder addRepeatedField(
           com.google.protobuf.Descriptors.FieldDescriptor field,
           java.lang.Object value) {
-        return (Builder) super.addRepeatedField(field, value);
+        return super.addRepeatedField(field, value);
       }
+      @java.lang.Override
       public Builder mergeFrom(com.google.protobuf.Message other) {
         if (other instanceof org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage) {
           return mergeFrom((org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage)other);
@@ -2943,7 +3246,7 @@ public final class TestMsg {
         if (!other.repeatedField_.isEmpty()) {
           if (repeatedField_.isEmpty()) {
             repeatedField_ = other.repeatedField_;
-            bitField0_ = (bitField0_ & ~0x00020000);
+            bitField0_ = (bitField0_ & ~0x00000001);
           } else {
             ensureRepeatedFieldIsMutable();
             repeatedField_.addAll(other.repeatedField_);
@@ -2952,15 +3255,20 @@ public final class TestMsg {
         }
         internalGetMutableMapField().mergeFrom(
             other.internalGetMapField());
+        if (other.hasTimestampField()) {
+          mergeTimestampField(other.getTimestampField());
+        }
         this.mergeUnknownFields(other.unknownFields);
         onChanged();
         return this;
       }
 
+      @java.lang.Override
       public final boolean isInitialized() {
         return true;
       }
 
+      @java.lang.Override
       public Builder mergeFrom(
           com.google.protobuf.CodedInputStream input,
           com.google.protobuf.ExtensionRegistryLite extensionRegistry)
@@ -2983,6 +3291,7 @@ public final class TestMsg {
       private java.lang.Object stringField_ = "";
       /**
        * <code>string stringField = 1;</code>
+       * @return The stringField.
        */
       public java.lang.String getStringField() {
         java.lang.Object ref = stringField_;
@@ -2998,6 +3307,7 @@ public final class TestMsg {
       }
       /**
        * <code>string stringField = 1;</code>
+       * @return The bytes for stringField.
        */
       public com.google.protobuf.ByteString
           getStringFieldBytes() {
@@ -3014,6 +3324,8 @@ public final class TestMsg {
       }
       /**
        * <code>string stringField = 1;</code>
+       * @param value The stringField to set.
+       * @return This builder for chaining.
        */
       public Builder setStringField(
           java.lang.String value) {
@@ -3027,6 +3339,7 @@ public final class TestMsg {
       }
       /**
        * <code>string stringField = 1;</code>
+       * @return This builder for chaining.
        */
       public Builder clearStringField() {
         
@@ -3036,6 +3349,8 @@ public final class TestMsg {
       }
       /**
        * <code>string stringField = 1;</code>
+       * @param value The bytes for stringField to set.
+       * @return This builder for chaining.
        */
       public Builder setStringFieldBytes(
           com.google.protobuf.ByteString value) {
@@ -3052,12 +3367,16 @@ public final class TestMsg {
       private double doubleField_ ;
       /**
        * <code>double doubleField = 2;</code>
+       * @return The doubleField.
        */
+      @java.lang.Override
       public double getDoubleField() {
         return doubleField_;
       }
       /**
        * <code>double doubleField = 2;</code>
+       * @param value The doubleField to set.
+       * @return This builder for chaining.
        */
       public Builder setDoubleField(double value) {
         
@@ -3067,6 +3386,7 @@ public final class TestMsg {
       }
       /**
        * <code>double doubleField = 2;</code>
+       * @return This builder for chaining.
        */
       public Builder clearDoubleField() {
         
@@ -3078,12 +3398,16 @@ public final class TestMsg {
       private float floatField_ ;
       /**
        * <code>float floatField = 3;</code>
+       * @return The floatField.
        */
+      @java.lang.Override
       public float getFloatField() {
         return floatField_;
       }
       /**
        * <code>float floatField = 3;</code>
+       * @param value The floatField to set.
+       * @return This builder for chaining.
        */
       public Builder setFloatField(float value) {
         
@@ -3093,6 +3417,7 @@ public final class TestMsg {
       }
       /**
        * <code>float floatField = 3;</code>
+       * @return This builder for chaining.
        */
       public Builder clearFloatField() {
         
@@ -3104,12 +3429,16 @@ public final class TestMsg {
       private int int32Field_ ;
       /**
        * <code>int32 int32Field = 4;</code>
+       * @return The int32Field.
        */
+      @java.lang.Override
       public int getInt32Field() {
         return int32Field_;
       }
       /**
        * <code>int32 int32Field = 4;</code>
+       * @param value The int32Field to set.
+       * @return This builder for chaining.
        */
       public Builder setInt32Field(int value) {
         
@@ -3119,6 +3448,7 @@ public final class TestMsg {
       }
       /**
        * <code>int32 int32Field = 4;</code>
+       * @return This builder for chaining.
        */
       public Builder clearInt32Field() {
         
@@ -3130,12 +3460,16 @@ public final class TestMsg {
       private long int64Field_ ;
       /**
        * <code>int64 int64Field = 5;</code>
+       * @return The int64Field.
        */
+      @java.lang.Override
       public long getInt64Field() {
         return int64Field_;
       }
       /**
        * <code>int64 int64Field = 5;</code>
+       * @param value The int64Field to set.
+       * @return This builder for chaining.
        */
       public Builder setInt64Field(long value) {
         
@@ -3145,6 +3479,7 @@ public final class TestMsg {
       }
       /**
        * <code>int64 int64Field = 5;</code>
+       * @return This builder for chaining.
        */
       public Builder clearInt64Field() {
         
@@ -3156,12 +3491,16 @@ public final class TestMsg {
       private int uint32Field_ ;
       /**
        * <code>uint32 uint32Field = 6;</code>
+       * @return The uint32Field.
        */
+      @java.lang.Override
       public int getUint32Field() {
         return uint32Field_;
       }
       /**
        * <code>uint32 uint32Field = 6;</code>
+       * @param value The uint32Field to set.
+       * @return This builder for chaining.
        */
       public Builder setUint32Field(int value) {
         
@@ -3171,6 +3510,7 @@ public final class TestMsg {
       }
       /**
        * <code>uint32 uint32Field = 6;</code>
+       * @return This builder for chaining.
        */
       public Builder clearUint32Field() {
         
@@ -3182,12 +3522,16 @@ public final class TestMsg {
       private long uint64Field_ ;
       /**
        * <code>uint64 uint64Field = 7;</code>
+       * @return The uint64Field.
        */
+      @java.lang.Override
       public long getUint64Field() {
         return uint64Field_;
       }
       /**
        * <code>uint64 uint64Field = 7;</code>
+       * @param value The uint64Field to set.
+       * @return This builder for chaining.
        */
       public Builder setUint64Field(long value) {
         
@@ -3197,6 +3541,7 @@ public final class TestMsg {
       }
       /**
        * <code>uint64 uint64Field = 7;</code>
+       * @return This builder for chaining.
        */
       public Builder clearUint64Field() {
         
@@ -3208,12 +3553,16 @@ public final class TestMsg {
       private int sint32Field_ ;
       /**
        * <code>sint32 sint32Field = 8;</code>
+       * @return The sint32Field.
        */
+      @java.lang.Override
       public int getSint32Field() {
         return sint32Field_;
       }
       /**
        * <code>sint32 sint32Field = 8;</code>
+       * @param value The sint32Field to set.
+       * @return This builder for chaining.
        */
       public Builder setSint32Field(int value) {
         
@@ -3223,6 +3572,7 @@ public final class TestMsg {
       }
       /**
        * <code>sint32 sint32Field = 8;</code>
+       * @return This builder for chaining.
        */
       public Builder clearSint32Field() {
         
@@ -3234,12 +3584,16 @@ public final class TestMsg {
       private long sint64Field_ ;
       /**
        * <code>sint64 sint64Field = 9;</code>
+       * @return The sint64Field.
        */
+      @java.lang.Override
       public long getSint64Field() {
         return sint64Field_;
       }
       /**
        * <code>sint64 sint64Field = 9;</code>
+       * @param value The sint64Field to set.
+       * @return This builder for chaining.
        */
       public Builder setSint64Field(long value) {
         
@@ -3249,6 +3603,7 @@ public final class TestMsg {
       }
       /**
        * <code>sint64 sint64Field = 9;</code>
+       * @return This builder for chaining.
        */
       public Builder clearSint64Field() {
         
@@ -3260,12 +3615,16 @@ public final class TestMsg {
       private int fixed32Field_ ;
       /**
        * <code>fixed32 fixed32Field = 10;</code>
+       * @return The fixed32Field.
        */
+      @java.lang.Override
       public int getFixed32Field() {
         return fixed32Field_;
       }
       /**
        * <code>fixed32 fixed32Field = 10;</code>
+       * @param value The fixed32Field to set.
+       * @return This builder for chaining.
        */
       public Builder setFixed32Field(int value) {
         
@@ -3275,6 +3634,7 @@ public final class TestMsg {
       }
       /**
        * <code>fixed32 fixed32Field = 10;</code>
+       * @return This builder for chaining.
        */
       public Builder clearFixed32Field() {
         
@@ -3286,12 +3646,16 @@ public final class TestMsg {
       private long fixed64Field_ ;
       /**
        * <code>fixed64 fixed64Field = 11;</code>
+       * @return The fixed64Field.
        */
+      @java.lang.Override
       public long getFixed64Field() {
         return fixed64Field_;
       }
       /**
        * <code>fixed64 fixed64Field = 11;</code>
+       * @param value The fixed64Field to set.
+       * @return This builder for chaining.
        */
       public Builder setFixed64Field(long value) {
         
@@ -3301,6 +3665,7 @@ public final class TestMsg {
       }
       /**
        * <code>fixed64 fixed64Field = 11;</code>
+       * @return This builder for chaining.
        */
       public Builder clearFixed64Field() {
         
@@ -3312,12 +3677,16 @@ public final class TestMsg {
       private int sfixed32Field_ ;
       /**
        * <code>sfixed32 sfixed32Field = 12;</code>
+       * @return The sfixed32Field.
        */
+      @java.lang.Override
       public int getSfixed32Field() {
         return sfixed32Field_;
       }
       /**
        * <code>sfixed32 sfixed32Field = 12;</code>
+       * @param value The sfixed32Field to set.
+       * @return This builder for chaining.
        */
       public Builder setSfixed32Field(int value) {
         
@@ -3327,6 +3696,7 @@ public final class TestMsg {
       }
       /**
        * <code>sfixed32 sfixed32Field = 12;</code>
+       * @return This builder for chaining.
        */
       public Builder clearSfixed32Field() {
         
@@ -3338,12 +3708,16 @@ public final class TestMsg {
       private long sfixed64Field_ ;
       /**
        * <code>sfixed64 sfixed64Field = 13;</code>
+       * @return The sfixed64Field.
        */
+      @java.lang.Override
       public long getSfixed64Field() {
         return sfixed64Field_;
       }
       /**
        * <code>sfixed64 sfixed64Field = 13;</code>
+       * @param value The sfixed64Field to set.
+       * @return This builder for chaining.
        */
       public Builder setSfixed64Field(long value) {
         
@@ -3353,6 +3727,7 @@ public final class TestMsg {
       }
       /**
        * <code>sfixed64 sfixed64Field = 13;</code>
+       * @return This builder for chaining.
        */
       public Builder clearSfixed64Field() {
         
@@ -3364,12 +3739,16 @@ public final class TestMsg {
       private boolean boolField_ ;
       /**
        * <code>bool boolField = 14;</code>
+       * @return The boolField.
        */
+      @java.lang.Override
       public boolean getBoolField() {
         return boolField_;
       }
       /**
        * <code>bool boolField = 14;</code>
+       * @param value The boolField to set.
+       * @return This builder for chaining.
        */
       public Builder setBoolField(boolean value) {
         
@@ -3379,6 +3758,7 @@ public final class TestMsg {
       }
       /**
        * <code>bool boolField = 14;</code>
+       * @return This builder for chaining.
        */
       public Builder clearBoolField() {
         
@@ -3390,12 +3770,16 @@ public final class TestMsg {
       private com.google.protobuf.ByteString bytesField_ = com.google.protobuf.ByteString.EMPTY;
       /**
        * <code>bytes bytesField = 15;</code>
+       * @return The bytesField.
        */
+      @java.lang.Override
       public com.google.protobuf.ByteString getBytesField() {
         return bytesField_;
       }
       /**
        * <code>bytes bytesField = 15;</code>
+       * @param value The bytesField to set.
+       * @return This builder for chaining.
        */
       public Builder setBytesField(com.google.protobuf.ByteString value) {
         if (value == null) {
@@ -3408,6 +3792,7 @@ public final class TestMsg {
       }
       /**
        * <code>bytes bytesField = 15;</code>
+       * @return This builder for chaining.
        */
       public Builder clearBytesField() {
         
@@ -3419,27 +3804,36 @@ public final class TestMsg {
       private int testEnum_ = 0;
       /**
        * <code>.proto.TestEnum testEnum = 16;</code>
+       * @return The enum numeric value on the wire for testEnum.
        */
-      public int getTestEnumValue() {
+      @java.lang.Override public int getTestEnumValue() {
         return testEnum_;
       }
       /**
        * <code>.proto.TestEnum testEnum = 16;</code>
+       * @param value The enum numeric value on the wire for testEnum to set.
+       * @return This builder for chaining.
        */
       public Builder setTestEnumValue(int value) {
+        
         testEnum_ = value;
         onChanged();
         return this;
       }
       /**
        * <code>.proto.TestEnum testEnum = 16;</code>
+       * @return The testEnum.
        */
+      @java.lang.Override
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum getTestEnum() {
+        @SuppressWarnings("deprecation")
         org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum result = org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum.valueOf(testEnum_);
         return result == null ? org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum.UNRECOGNIZED : result;
       }
       /**
        * <code>.proto.TestEnum testEnum = 16;</code>
+       * @param value The testEnum to set.
+       * @return This builder for chaining.
        */
       public Builder setTestEnum(org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestEnum value) {
         if (value == null) {
@@ -3452,6 +3846,7 @@ public final class TestMsg {
       }
       /**
        * <code>.proto.TestEnum testEnum = 16;</code>
+       * @return This builder for chaining.
        */
       public Builder clearTestEnum() {
         
@@ -3460,17 +3855,19 @@ public final class TestMsg {
         return this;
       }
 
-      private org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage subMessage_ = null;
+      private org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage subMessage_;
       private com.google.protobuf.SingleFieldBuilderV3<
           org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage, org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage.Builder, org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessageOrBuilder> subMessageBuilder_;
       /**
        * <code>.proto.SubMessage subMessage = 17;</code>
+       * @return Whether the subMessage field is set.
        */
       public boolean hasSubMessage() {
         return subMessageBuilder_ != null || subMessage_ != null;
       }
       /**
        * <code>.proto.SubMessage subMessage = 17;</code>
+       * @return The subMessage.
        */
       public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.SubMessage getSubMessage() {
         if (subMessageBuilder_ == null) {
@@ -3579,13 +3976,14 @@ public final class TestMsg {
 
       private com.google.protobuf.LazyStringList repeatedField_ = com.google.protobuf.LazyStringArrayList.EMPTY;
       private void ensureRepeatedFieldIsMutable() {
-        if (!((bitField0_ & 0x00020000) == 0x00020000)) {
+        if (!((bitField0_ & 0x00000001) != 0)) {
           repeatedField_ = new com.google.protobuf.LazyStringArrayList(repeatedField_);
-          bitField0_ |= 0x00020000;
+          bitField0_ |= 0x00000001;
          }
       }
       /**
        * <code>repeated string repeatedField = 18;</code>
+       * @return A list containing the repeatedField.
        */
       public com.google.protobuf.ProtocolStringList
           getRepeatedFieldList() {
@@ -3593,18 +3991,23 @@ public final class TestMsg {
       }
       /**
        * <code>repeated string repeatedField = 18;</code>
+       * @return The count of repeatedField.
        */
       public int getRepeatedFieldCount() {
         return repeatedField_.size();
       }
       /**
        * <code>repeated string repeatedField = 18;</code>
+       * @param index The index of the element to return.
+       * @return The repeatedField at the given index.
        */
       public java.lang.String getRepeatedField(int index) {
         return repeatedField_.get(index);
       }
       /**
        * <code>repeated string repeatedField = 18;</code>
+       * @param index The index of the value to return.
+       * @return The bytes of the repeatedField at the given index.
        */
       public com.google.protobuf.ByteString
           getRepeatedFieldBytes(int index) {
@@ -3612,6 +4015,9 @@ public final class TestMsg {
       }
       /**
        * <code>repeated string repeatedField = 18;</code>
+       * @param index The index to set the value at.
+       * @param value The repeatedField to set.
+       * @return This builder for chaining.
        */
       public Builder setRepeatedField(
           int index, java.lang.String value) {
@@ -3625,6 +4031,8 @@ public final class TestMsg {
       }
       /**
        * <code>repeated string repeatedField = 18;</code>
+       * @param value The repeatedField to add.
+       * @return This builder for chaining.
        */
       public Builder addRepeatedField(
           java.lang.String value) {
@@ -3638,6 +4046,8 @@ public final class TestMsg {
       }
       /**
        * <code>repeated string repeatedField = 18;</code>
+       * @param values The repeatedField to add.
+       * @return This builder for chaining.
        */
       public Builder addAllRepeatedField(
           java.lang.Iterable<java.lang.String> values) {
@@ -3649,15 +4059,18 @@ public final class TestMsg {
       }
       /**
        * <code>repeated string repeatedField = 18;</code>
+       * @return This builder for chaining.
        */
       public Builder clearRepeatedField() {
         repeatedField_ = com.google.protobuf.LazyStringArrayList.EMPTY;
-        bitField0_ = (bitField0_ & ~0x00020000);
+        bitField0_ = (bitField0_ & ~0x00000001);
         onChanged();
         return this;
       }
       /**
        * <code>repeated string repeatedField = 18;</code>
+       * @param value The bytes of the repeatedField to add.
+       * @return This builder for chaining.
        */
       public Builder addRepeatedFieldBytes(
           com.google.protobuf.ByteString value) {
@@ -3701,6 +4114,7 @@ public final class TestMsg {
        * <code>map&lt;string, double&gt; mapField = 19;</code>
        */
 
+      @java.lang.Override
       public boolean containsMapField(
           java.lang.String key) {
         if (key == null) { throw new java.lang.NullPointerException(); }
@@ -3709,6 +4123,7 @@ public final class TestMsg {
       /**
        * Use {@link #getMapFieldMap()} instead.
        */
+      @java.lang.Override
       @java.lang.Deprecated
       public java.util.Map<java.lang.String, java.lang.Double> getMapField() {
         return getMapFieldMap();
@@ -3716,6 +4131,7 @@ public final class TestMsg {
       /**
        * <code>map&lt;string, double&gt; mapField = 19;</code>
        */
+      @java.lang.Override
 
       public java.util.Map<java.lang.String, java.lang.Double> getMapFieldMap() {
         return internalGetMapField().getMap();
@@ -3723,6 +4139,7 @@ public final class TestMsg {
       /**
        * <code>map&lt;string, double&gt; mapField = 19;</code>
        */
+      @java.lang.Override
 
       public double getMapFieldOrDefault(
           java.lang.String key,
@@ -3735,6 +4152,7 @@ public final class TestMsg {
       /**
        * <code>map&lt;string, double&gt; mapField = 19;</code>
        */
+      @java.lang.Override
 
       public double getMapFieldOrThrow(
           java.lang.String key) {
@@ -3793,11 +4211,132 @@ public final class TestMsg {
             .putAll(values);
         return this;
       }
+
+      private com.google.protobuf.Timestamp timestampField_;
+      private com.google.protobuf.SingleFieldBuilderV3<
+          com.google.protobuf.Timestamp, com.google.protobuf.Timestamp.Builder, com.google.protobuf.TimestampOrBuilder> timestampFieldBuilder_;
+      /**
+       * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+       * @return Whether the timestampField field is set.
+       */
+      public boolean hasTimestampField() {
+        return timestampFieldBuilder_ != null || timestampField_ != null;
+      }
+      /**
+       * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+       * @return The timestampField.
+       */
+      public com.google.protobuf.Timestamp getTimestampField() {
+        if (timestampFieldBuilder_ == null) {
+          return timestampField_ == null ? com.google.protobuf.Timestamp.getDefaultInstance() : timestampField_;
+        } else {
+          return timestampFieldBuilder_.getMessage();
+        }
+      }
+      /**
+       * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+       */
+      public Builder setTimestampField(com.google.protobuf.Timestamp value) {
+        if (timestampFieldBuilder_ == null) {
+          if (value == null) {
+            throw new NullPointerException();
+          }
+          timestampField_ = value;
+          onChanged();
+        } else {
+          timestampFieldBuilder_.setMessage(value);
+        }
+
+        return this;
+      }
+      /**
+       * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+       */
+      public Builder setTimestampField(
+          com.google.protobuf.Timestamp.Builder builderForValue) {
+        if (timestampFieldBuilder_ == null) {
+          timestampField_ = builderForValue.build();
+          onChanged();
+        } else {
+          timestampFieldBuilder_.setMessage(builderForValue.build());
+        }
+
+        return this;
+      }
+      /**
+       * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+       */
+      public Builder mergeTimestampField(com.google.protobuf.Timestamp value) {
+        if (timestampFieldBuilder_ == null) {
+          if (timestampField_ != null) {
+            timestampField_ =
+              com.google.protobuf.Timestamp.newBuilder(timestampField_).mergeFrom(value).buildPartial();
+          } else {
+            timestampField_ = value;
+          }
+          onChanged();
+        } else {
+          timestampFieldBuilder_.mergeFrom(value);
+        }
+
+        return this;
+      }
+      /**
+       * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+       */
+      public Builder clearTimestampField() {
+        if (timestampFieldBuilder_ == null) {
+          timestampField_ = null;
+          onChanged();
+        } else {
+          timestampField_ = null;
+          timestampFieldBuilder_ = null;
+        }
+
+        return this;
+      }
+      /**
+       * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+       */
+      public com.google.protobuf.Timestamp.Builder getTimestampFieldBuilder() {
+        
+        onChanged();
+        return getTimestampFieldFieldBuilder().getBuilder();
+      }
+      /**
+       * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+       */
+      public com.google.protobuf.TimestampOrBuilder getTimestampFieldOrBuilder() {
+        if (timestampFieldBuilder_ != null) {
+          return timestampFieldBuilder_.getMessageOrBuilder();
+        } else {
+          return timestampField_ == null ?
+              com.google.protobuf.Timestamp.getDefaultInstance() : timestampField_;
+        }
+      }
+      /**
+       * <code>.google.protobuf.Timestamp timestampField = 20;</code>
+       */
+      private com.google.protobuf.SingleFieldBuilderV3<
+          com.google.protobuf.Timestamp, com.google.protobuf.Timestamp.Builder, com.google.protobuf.TimestampOrBuilder> 
+          getTimestampFieldFieldBuilder() {
+        if (timestampFieldBuilder_ == null) {
+          timestampFieldBuilder_ = new com.google.protobuf.SingleFieldBuilderV3<
+              com.google.protobuf.Timestamp, com.google.protobuf.Timestamp.Builder, com.google.protobuf.TimestampOrBuilder>(
+                  getTimestampField(),
+                  getParentForChildren(),
+                  isClean());
+          timestampField_ = null;
+        }
+        return timestampFieldBuilder_;
+      }
+      @java.lang.Override
       public final Builder setUnknownFields(
           final com.google.protobuf.UnknownFieldSet unknownFields) {
-        return super.setUnknownFieldsProto3(unknownFields);
+        return super.setUnknownFields(unknownFields);
       }
 
+      @java.lang.Override
       public final Builder mergeUnknownFields(
           final com.google.protobuf.UnknownFieldSet unknownFields) {
         return super.mergeUnknownFields(unknownFields);
@@ -3819,6 +4358,7 @@ public final class TestMsg {
 
     private static final com.google.protobuf.Parser<TestMessage>
         PARSER = new com.google.protobuf.AbstractParser<TestMessage>() {
+      @java.lang.Override
       public TestMessage parsePartialFrom(
           com.google.protobuf.CodedInputStream input,
           com.google.protobuf.ExtensionRegistryLite extensionRegistry)
@@ -3836,6 +4376,7 @@ public final class TestMsg {
       return PARSER;
     }
 
+    @java.lang.Override
     public org.apache.pulsar.sql.presto.decoder.protobufnative.TestMsg.TestMessage getDefaultInstanceForType() {
       return DEFAULT_INSTANCE;
     }
@@ -3871,40 +4412,35 @@ public final class TestMsg {
       descriptor;
   static {
     java.lang.String[] descriptorData = {
-      "\n\rTestMsg.proto\022\005proto\"\214\001\n\nSubMessage\022\013\n" +
-      "\003foo\030\001 \001(\t\022\013\n\003bar\030\002 \001(\001\0226\n\rnestedMessage" +
-      "\030\003 \001(\0132\037.proto.SubMessage.NestedMessage\032" +
-      ",\n\rNestedMessage\022\r\n\005title\030\001 \001(\t\022\014\n\004urls\030" +
-      "\002 \003(\t\"\216\004\n\013TestMessage\022\023\n\013stringField\030\001 \001" +
-      "(\t\022\023\n\013doubleField\030\002 \001(\001\022\022\n\nfloatField\030\003 " +
-      "\001(\002\022\022\n\nint32Field\030\004 \001(\005\022\022\n\nint64Field\030\005 " +
-      "\001(\003\022\023\n\013uint32Field\030\006 \001(\r\022\023\n\013uint64Field\030" +
-      "\007 \001(\004\022\023\n\013sint32Field\030\010 \001(\021\022\023\n\013sint64Fiel" +
-      "d\030\t \001(\022\022\024\n\014fixed32Field\030\n \001(\007\022\024\n\014fixed64" +
-      "Field\030\013 \001(\006\022\025\n\rsfixed32Field\030\014 \001(\017\022\025\n\rsf" +
-      "ixed64Field\030\r \001(\020\022\021\n\tboolField\030\016 \001(\010\022\022\n\n" +
-      "bytesField\030\017 \001(\014\022!\n\010testEnum\030\020 \001(\0162\017.pro" +
-      "to.TestEnum\022%\n\nsubMessage\030\021 \001(\0132\021.proto." +
-      "SubMessage\022\025\n\rrepeatedField\030\022 \003(\t\0222\n\010map" +
-      "Field\030\023 \003(\0132 .proto.TestMessage.MapField" +
-      "Entry\032/\n\rMapFieldEntry\022\013\n\003key\030\001 \001(\t\022\r\n\005v" +
-      "alue\030\002 \001(\001:\0028\001*$\n\010TestEnum\022\n\n\006SHARED\020\000\022\014" +
-      "\n\010FAILOVER\020\001B>\n3org.apache.pulsar.sql.pr" +
-      "esto.decoder.protobufnativeB\007TestMsgb\006pr" +
-      "oto3"
+      "\n\rTestMsg.proto\022\005proto\032\037google/protobuf/" +
+      "timestamp.proto\"\214\001\n\nSubMessage\022\013\n\003foo\030\001 " +
+      "\001(\t\022\013\n\003bar\030\002 \001(\001\0226\n\rnestedMessage\030\003 \001(\0132" +
+      "\037.proto.SubMessage.NestedMessage\032,\n\rNest" +
+      "edMessage\022\r\n\005title\030\001 \001(\t\022\014\n\004urls\030\002 \003(\t\"\302" +
+      "\004\n\013TestMessage\022\023\n\013stringField\030\001 \001(\t\022\023\n\013d" +
+      "oubleField\030\002 \001(\001\022\022\n\nfloatField\030\003 \001(\002\022\022\n\n" +
+      "int32Field\030\004 \001(\005\022\022\n\nint64Field\030\005 \001(\003\022\023\n\013" +
+      "uint32Field\030\006 \001(\r\022\023\n\013uint64Field\030\007 \001(\004\022\023" +
+      "\n\013sint32Field\030\010 \001(\021\022\023\n\013sint64Field\030\t \001(\022" +
+      "\022\024\n\014fixed32Field\030\n \001(\007\022\024\n\014fixed64Field\030\013" +
+      " \001(\006\022\025\n\rsfixed32Field\030\014 \001(\017\022\025\n\rsfixed64F" +
+      "ield\030\r \001(\020\022\021\n\tboolField\030\016 \001(\010\022\022\n\nbytesFi" +
+      "eld\030\017 \001(\014\022!\n\010testEnum\030\020 \001(\0162\017.proto.Test" +
+      "Enum\022%\n\nsubMessage\030\021 \001(\0132\021.proto.SubMess" +
+      "age\022\025\n\rrepeatedField\030\022 \003(\t\0222\n\010mapField\030\023" +
+      " \003(\0132 .proto.TestMessage.MapFieldEntry\0222" +
+      "\n\016timestampField\030\024 \001(\0132\032.google.protobuf" +
+      ".Timestamp\032/\n\rMapFieldEntry\022\013\n\003key\030\001 \001(\t" +
+      "\022\r\n\005value\030\002 \001(\001:\0028\001*$\n\010TestEnum\022\n\n\006SHARE" +
+      "D\020\000\022\014\n\010FAILOVER\020\001B>\n3org.apache.pulsar.s" +
+      "ql.presto.decoder.protobufnativeB\007TestMs" +
+      "gP\000b\006proto3"
     };
-    com.google.protobuf.Descriptors.FileDescriptor.InternalDescriptorAssigner assigner =
-        new com.google.protobuf.Descriptors.FileDescriptor.    InternalDescriptorAssigner() {
-          public com.google.protobuf.ExtensionRegistry assignDescriptors(
-              com.google.protobuf.Descriptors.FileDescriptor root) {
-            descriptor = root;
-            return null;
-          }
-        };
-    com.google.protobuf.Descriptors.FileDescriptor
+    descriptor = com.google.protobuf.Descriptors.FileDescriptor
       .internalBuildGeneratedFileFrom(descriptorData,
         new com.google.protobuf.Descriptors.FileDescriptor[] {
-        }, assigner);
+          com.google.protobuf.TimestampProto.getDescriptor(),
+        });
     internal_static_proto_SubMessage_descriptor =
       getDescriptor().getMessageTypes().get(0);
     internal_static_proto_SubMessage_fieldAccessorTable = new
@@ -3922,13 +4458,14 @@ public final class TestMsg {
     internal_static_proto_TestMessage_fieldAccessorTable = new
       com.google.protobuf.GeneratedMessageV3.FieldAccessorTable(
         internal_static_proto_TestMessage_descriptor,
-        new java.lang.String[] { "StringField", "DoubleField", "FloatField", "Int32Field", "Int64Field", "Uint32Field", "Uint64Field", "Sint32Field", "Sint64Field", "Fixed32Field", "Fixed64Field", "Sfixed32Field", "Sfixed64Field", "BoolField", "BytesField", "TestEnum", "SubMessage", "RepeatedField", "MapField", });
+        new java.lang.String[] { "StringField", "DoubleField", "FloatField", "Int32Field", "Int64Field", "Uint32Field", "Uint64Field", "Sint32Field", "Sint64Field", "Fixed32Field", "Fixed64Field", "Sfixed32Field", "Sfixed64Field", "BoolField", "BytesField", "TestEnum", "SubMessage", "RepeatedField", "MapField", "TimestampField", });
     internal_static_proto_TestMessage_MapFieldEntry_descriptor =
       internal_static_proto_TestMessage_descriptor.getNestedTypes().get(0);
     internal_static_proto_TestMessage_MapFieldEntry_fieldAccessorTable = new
       com.google.protobuf.GeneratedMessageV3.FieldAccessorTable(
         internal_static_proto_TestMessage_MapFieldEntry_descriptor,
         new java.lang.String[] { "Key", "Value", });
+    com.google.protobuf.TimestampProto.getDescriptor();
   }
 
   // @@protoc_insertion_point(outer_class_scope)
diff --git a/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto b/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto
index 0a0f489..fd522bd 100644
--- a/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto
+++ b/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto
@@ -19,6 +19,7 @@
 syntax = "proto3";
 package proto;
 
+import public "google/protobuf/timestamp.proto";
 
 option java_package = "org.apache.pulsar.sql.presto.decoder.protobufnative";
 option java_outer_classname = "TestMsg";
@@ -58,4 +59,5 @@ message TestMessage {
     SubMessage subMessage = 17;
     repeated string repeatedField = 18;
     map<string, double> mapField = 19;
+    google.protobuf.Timestamp timestampField = 20;
 }
\ No newline at end of file
diff --git a/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestProtobufNativeDecoder.java b/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestProtobufNativeDecoder.java
index fc1834c..c4c6cb2 100644
--- a/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestProtobufNativeDecoder.java
+++ b/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestProtobufNativeDecoder.java
@@ -20,6 +20,7 @@ package org.apache.pulsar.sql.presto.decoder.protobufnative;
 
 import com.google.common.collect.ImmutableList;
 import com.google.protobuf.ByteString;
+import com.google.protobuf.Timestamp;
 import io.netty.buffer.ByteBuf;
 import io.prestosql.decoder.DecoderColumnHandle;
 import io.prestosql.decoder.FieldValueProvider;
@@ -43,6 +44,7 @@ import static io.prestosql.spi.type.BigintType.BIGINT;
 import static io.prestosql.spi.type.BooleanType.BOOLEAN;
 import static io.prestosql.spi.type.DoubleType.DOUBLE;
 import static io.prestosql.spi.type.IntegerType.INTEGER;
+import static io.prestosql.spi.type.TimestampType.TIMESTAMP;
 import static io.prestosql.spi.type.VarbinaryType.VARBINARY;
 import static io.prestosql.spi.type.VarcharType.VARCHAR;
 import static org.apache.pulsar.sql.presto.TestPulsarConnector.getPulsarConnectorId;
@@ -65,6 +67,12 @@ public class TestProtobufNativeDecoder extends AbstractDecoderTester {
 
     @Test
     public void testPrimitiveType() {
+        //Time: 2921-1-1
+        long mills = 30010669261001L;
+        Timestamp timestamp = Timestamp.newBuilder()
+                .setSeconds(mills / 1000)
+                .setNanos((int) (mills % 1000) * 1000000)
+                .build();
 
         TestMsg.TestMessage testMessage = TestMsg.TestMessage.newBuilder()
                 .setStringField("aaa")
@@ -83,6 +91,7 @@ public class TestProtobufNativeDecoder extends AbstractDecoderTester {
                 .setBoolField(true)
                 .setBytesField(ByteString.copyFrom("abc".getBytes()))
                 .setTestEnum(TestMsg.TestEnum.FAILOVER)
+                .setTimestampField(timestamp)
                 .build();
 
         ByteBuf payload = io.netty.buffer.Unpooled
@@ -163,6 +172,11 @@ public class TestProtobufNativeDecoder extends AbstractDecoderTester {
                 PulsarColumnHandle.HandleKeyValueType.NONE);
         checkValue(decodedRow, enumFieldColumnHandle, testMessage.getTestEnum().name());
 
+        PulsarColumnHandle timestampFieldColumnHandle = new PulsarColumnHandle(getPulsarConnectorId().toString(),
+                "timestampField", TIMESTAMP,false,false,"timestampField",null,null,
+                PulsarColumnHandle.HandleKeyValueType.NONE);
+        checkValue(decodedRow, timestampFieldColumnHandle, mills);
+
     }
 
     @Test

[pulsar] 04/13: [Producer] Change the time units from ns to ms (#13057)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit b763f835068e23b0ddbd49871531d72cf660577d
Author: litao <to...@gmail.com>
AuthorDate: Thu Dec 23 22:08:20 2021 +0800

    [Producer] Change the time units from ns to ms (#13057)
    
    ### Motivation
    The time unit in this exception message is ns, which is not very readable. We can change it from ns to ms.
    ```
    org.apache.pulsar.client.api.PulsarClientException$TimeoutException:
    The producer xxx can not send message to the topic xxx within given timeout : createdAt 461913074 ns ago, firstSentAt 29545553038276935 ns ago, lastSentAt 29545553038276935 ns ago, retryCount 0 at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:916)
    at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.send(TypedMessageBuilderImpl.java:93)
    at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:63)
    at com.yum.boh.oh.service.impl.StoreOrderPostServiceImpl.generalProcessing(StoreOrderPostServiceImpl.java:272)
    at com.yum.boh.oh.service.impl.StoreOrderPostServiceImpl.saveThirdOrder(StoreOrderPostServiceImpl.java:72)
    at com.yum.boh.oh.controller.StoreOrderController.postOrderInfo$original$T8425mfx(StoreOrderController.java:39)
    at com.yum.boh.oh.controller.StoreOrderController.postOrderInfo$original$T8425mfx$accessor$vJljNzML(StoreOrderController.java)
    at com.yum.boh.oh.controller.StoreOrderController$auxiliary$nysalhgy.call(Unknown Source)
    at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
    ```
    
    ### Modifications
    Change the time units from ns to ms for ProducerImpl#OpSendMsg.
    
    (cherry picked from commit 891660e396ad2831ec2bc94b535fc5d0c9543ec7)
---
 .../java/org/apache/pulsar/client/impl/ProducerImpl.java     |  8 +++++---
 .../java/org/apache/pulsar/common/util/RelativeTimeUtil.java | 12 ++++++++++++
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
index 1bd0977..3afd660 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
@@ -87,6 +87,7 @@ import org.apache.pulsar.common.schema.SchemaInfo;
 import org.apache.pulsar.common.schema.SchemaType;
 import org.apache.pulsar.common.util.DateFormatter;
 import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.RelativeTimeUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -1234,9 +1235,9 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
                     String errMsg = String.format(
                         "%s : createdAt %s ns ago, firstSentAt %s ns ago, lastSentAt %s ns ago, retryCount %s",
                         te.getMessage(),
-                        ns - this.createdAt,
-                        this.firstSentAt <= 0 ? ns - this.lastSentAt : ns - this.firstSentAt,
-                        ns - this.lastSentAt,
+                        RelativeTimeUtil.nsToSeconds(ns - this.createdAt),
+                        RelativeTimeUtil.nsToSeconds(this.firstSentAt <= 0 ? ns - this.lastSentAt : ns - this.firstSentAt),
+                        RelativeTimeUtil.nsToSeconds(ns - this.lastSentAt),
                         retryCount
                     );
 
@@ -1299,6 +1300,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
         };
     }
 
+
     /**
      * Queue implementation that is used as the pending messages queue.
      *
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/RelativeTimeUtil.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/RelativeTimeUtil.java
index 36e6adf..454cfda 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/RelativeTimeUtil.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/RelativeTimeUtil.java
@@ -18,6 +18,7 @@
  */
 package org.apache.pulsar.common.util;
 
+import java.math.BigDecimal;
 import java.util.concurrent.TimeUnit;
 import lombok.experimental.UtilityClass;
 
@@ -63,4 +64,15 @@ public class RelativeTimeUtil {
             throw new IllegalArgumentException("Invalid time unit '" + lastChar + "'");
         }
     }
+
+    /**
+     * Convert nanoseconds to seconds and keep three decimal places.
+     * @param ns
+     * @return seconds
+     */
+    public static double nsToSeconds(long ns) {
+        double seconds = (double) ns / 1_000_000_000;
+        BigDecimal bd = new BigDecimal(seconds);
+        return bd.setScale(3, BigDecimal.ROUND_HALF_UP).doubleValue();
+    }
 }

[pulsar] 06/13: Fix semaphore and memory leak when chunks failed to enqueue (#13454)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit ef9e773ef454db3db4ed41de20858c2c0b2dadaf
Author: Yunze Xu <xy...@163.com>
AuthorDate: Fri Dec 24 10:56:17 2021 +0800

    Fix semaphore and memory leak when chunks failed to enqueue (#13454)
    
    ### Motivation
    
    When a large message is sent by chunks, each chunk needs to reserve a spot of the semaphore. However, when it failed, the already reserved memory from limiter and spots from semaphore are not released.
    
    ### Modifications
    
    - Release the semaphore and memory when `canEnqueueRequest` returns false for chunks.
    - Add `testChunksEnqueueFailed` to cover this case. It sends a large message whose number of chunks is greater than the `maxPendingMessages` so that the first time `canEnqueueRequest` returns true while the following `canEnqueueRequest` calls will return false.
    
    (cherry picked from commit 2e2cd57f984b601a878dd11e1d27f4c169a84e5b)
---
 .../pulsar/client/impl/MessageChunkingTest.java    | 43 ++++++++++++++++++++++
 .../apache/pulsar/client/impl/ProducerImpl.java    |  2 +
 2 files changed, 45 insertions(+)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/MessageChunkingTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/MessageChunkingTest.java
index d4eab77..40191ef 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/MessageChunkingTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/MessageChunkingTest.java
@@ -20,6 +20,7 @@ package org.apache.pulsar.client.impl;
 
 import static org.testng.Assert.assertEquals;
 import static org.testng.Assert.assertNotEquals;
+import static org.testng.Assert.assertNotNull;
 import static org.testng.Assert.assertTrue;
 import static org.testng.Assert.fail;
 import com.google.common.collect.Lists;
@@ -27,17 +28,20 @@ import com.google.common.collect.Sets;
 import io.netty.buffer.ByteBuf;
 import io.netty.buffer.Unpooled;
 import java.lang.reflect.Field;
+import java.nio.charset.StandardCharsets;
 import java.util.List;
 import java.util.Random;
 import java.util.Set;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 import lombok.Cleanup;
 import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
 import org.apache.bookkeeper.mledger.impl.PositionImpl;
 import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.client.api.ClientBuilder;
 import org.apache.pulsar.client.api.CompressionType;
 import org.apache.pulsar.client.api.Consumer;
 import org.apache.pulsar.client.api.Message;
@@ -47,6 +51,7 @@ import org.apache.pulsar.client.api.Producer;
 import org.apache.pulsar.client.api.ProducerBuilder;
 import org.apache.pulsar.client.api.ProducerConsumerBase;
 import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.client.api.SizeUnit;
 import org.apache.pulsar.client.impl.MessageImpl.SchemaState;
 import org.apache.pulsar.client.impl.ProducerImpl.OpSendMsg;
 import org.apache.pulsar.common.api.proto.MessageMetadata;
@@ -369,6 +374,44 @@ public class MessageChunkingTest extends ProducerConsumerBase {
         producer = null; // clean reference of mocked producer
     }
 
+    @Test
+    public void testChunksEnqueueFailed() throws Exception {
+        final String topicName = "persistent://my-property/my-ns/test-chunks-enqueue-failed";
+        log.info("-- Starting {} test --", methodName);
+        this.conf.setMaxMessageSize(5);
+
+        final MemoryLimitController controller = ((PulsarClientImpl) pulsarClient).getMemoryLimitController();
+        assertEquals(controller.currentUsage(), 0);
+
+        final int maxPendingMessages = 10;
+
+        @Cleanup
+        Producer<byte[]> producer = pulsarClient.newProducer()
+                .topic(topicName)
+                .maxPendingMessages(maxPendingMessages)
+                .enableChunking(true)
+                .enableBatching(false)
+                .create();
+        assertTrue(producer instanceof ProducerImpl);
+        Semaphore semaphore = ((ProducerImpl<byte[]>) producer).getSemaphore().orElse(null);
+        assertNotNull(semaphore);
+        assertEquals(semaphore.availablePermits(), maxPendingMessages);
+        producer.send(createMessagePayload(1).getBytes());
+        try {
+            producer.send(createMessagePayload(100).getBytes(StandardCharsets.UTF_8));
+            fail("It should fail with ProducerQueueIsFullError");
+        } catch (PulsarClientException e) {
+            assertTrue(e instanceof PulsarClientException.ProducerQueueIsFullError);
+            assertEquals(controller.currentUsage(), 0);
+            assertEquals(semaphore.availablePermits(), maxPendingMessages);
+        }
+    }
+
+    @Override
+    protected void customizeNewPulsarClientBuilder(ClientBuilder clientBuilder) {
+        clientBuilder.memoryLimit(10000L, SizeUnit.BYTES);
+    }
+
     private String createMessagePayload(int size) {
         StringBuilder str = new StringBuilder();
         Random rand = new Random();
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
index 3afd660..a5baba7 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
@@ -456,6 +456,8 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
         // chunked message also sent individually so, try to acquire send-permits
         for (int i = 0; i < (totalChunks - 1); i++) {
             if (!canEnqueueRequest(callback, message.getSequenceId(), 0 /* The memory was already reserved */)) {
+                client.getMemoryLimitController().releaseMemory(uncompressedSize);
+                semaphoreRelease(i + 1);
                 return;
             }
         }

[pulsar] 05/13: Fix dead loop in BacklogQuotaManager.dropBacklogForTimeLimit (#13194) (#13249)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 25c3ae3c575469e5ad115f71b54291a1b680dff5
Author: 萧易客 <km...@live.com>
AuthorDate: Fri Dec 24 00:46:31 2021 +0800

    Fix dead loop in BacklogQuotaManager.dropBacklogForTimeLimit (#13194) (#13249)
    
    Fixes #13194
    
    ### Motivation
    https://github.com/apache/pulsar/blob/38fb839154462fc5c6b0b4293f02762ed4021cd9/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java#L200-L219
    BacklogQuotaManager.dropBacklogForTimeLimit may fall into dead loop in some conditions, e.g.
    `backlogQuotaDefaultLimitSecond` is enabled
    1. producer stop produce after produced some messages, current ledger is A
    2. times up, triggered ledger rollover, a new ledger B created which is empty (no entries)
    3. now lastConfirmedEntry is `A:last-entry-id`
    4. after `backlogQuotaDefaultLimitSecond` times up, it'll reset cursor to position `A:last-entry-id+1` which is only valid, so loop begin until the producer resume produce
    
    ### Modifications
    
    Record the previous slowestReaderPosition, if it is same with newer slowestReaderPosition after `resetCursor`, then exit loop.
    
    (cherry picked from commit 021409b6e1d3c910afe8e05d51a536cee647cb90)
---
 .../pulsar/broker/service/BacklogQuotaManager.java | 28 ++++++++-----
 .../broker/service/BacklogQuotaManagerTest.java    | 48 ++++++++++++++++++++++
 2 files changed, 65 insertions(+), 11 deletions(-)

diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java
index c9cd15e..65ffaab 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java
@@ -26,9 +26,10 @@ import java.util.Optional;
 import java.util.concurrent.CompletableFuture;
 import org.apache.bookkeeper.mledger.ManagedCursor;
 import org.apache.bookkeeper.mledger.ManagedCursor.IndividualDeletedEntries;
+import org.apache.bookkeeper.mledger.Position;
 import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
 import org.apache.bookkeeper.mledger.impl.PositionImpl;
-import org.apache.bookkeeper.mledger.proto.MLDataFormats;
+import org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo;
 import org.apache.pulsar.broker.PulsarService;
 import org.apache.pulsar.broker.admin.AdminResource;
 import org.apache.pulsar.broker.service.persistent.PersistentTopic;
@@ -241,17 +242,22 @@ public class BacklogQuotaManager {
             Long currentMillis = ((ManagedLedgerImpl) persistentTopic.getManagedLedger()).getClock().millis();
             ManagedLedgerImpl mLedger = (ManagedLedgerImpl) persistentTopic.getManagedLedger();
             try {
-                Long ledgerId =  mLedger.getCursors().getSlowestReaderPosition().getLedgerId();
-                MLDataFormats.ManagedLedgerInfo.LedgerInfo  ledgerInfo = mLedger.getLedgerInfo(ledgerId).get();
-                // Timestamp only > 0 if ledger has been closed
-                while (ledgerInfo.getTimestamp() > 0
-                        && currentMillis - ledgerInfo.getTimestamp() > quota.getLimitTime()) {
+                for (;;) {
                     ManagedCursor slowestConsumer = mLedger.getSlowestConsumer();
-                    // skip whole ledger for the slowest cursor
-                    slowestConsumer.resetCursor(mLedger.getNextValidPosition(
-                            PositionImpl.get(ledgerInfo.getLedgerId(), ledgerInfo.getEntries() - 1)));
-                    ledgerId =  mLedger.getCursors().getSlowestReaderPosition().getLedgerId();
-                    ledgerInfo = mLedger.getLedgerInfo(ledgerId).get();
+                    Position oldestPosition = slowestConsumer.getMarkDeletedPosition();
+                    ManagedLedgerInfo.LedgerInfo ledgerInfo = mLedger.getLedgerInfo(oldestPosition.getLedgerId()).get();
+                    // Timestamp only > 0 if ledger has been closed
+                    if (ledgerInfo.getTimestamp() > 0
+                            && currentMillis - ledgerInfo.getTimestamp() > quota.getLimitTime()) {
+                        // skip whole ledger for the slowest cursor
+                        PositionImpl nextPosition = mLedger.getNextValidPosition(
+                                PositionImpl.get(ledgerInfo.getLedgerId(), ledgerInfo.getEntries() - 1));
+                        if (!nextPosition.equals(oldestPosition)) {
+                            slowestConsumer.resetCursor(nextPosition);
+                            continue;
+                        }
+                    }
+                    break;
                 }
             } catch (Exception e) {
                 log.error("[{}] Error resetting cursor for slowest consumer [{}]", persistentTopic.getName(),
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BacklogQuotaManagerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BacklogQuotaManagerTest.java
index 560cd9e..c0fc189 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BacklogQuotaManagerTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BacklogQuotaManagerTest.java
@@ -477,6 +477,54 @@ public class BacklogQuotaManagerTest {
     }
 
     @Test
+    public void testConsumerBacklogEvictionTimeQuotaWithEmptyLedger() throws Exception {
+        assertEquals(admin.namespaces().getBacklogQuotaMap("prop/ns-quota"),
+                Maps.newHashMap());
+        admin.namespaces().setBacklogQuota("prop/ns-quota",
+                BacklogQuota.builder()
+                        .limitTime(TIME_TO_CHECK_BACKLOG_QUOTA)
+                        .retentionPolicy(BacklogQuota.RetentionPolicy.consumer_backlog_eviction)
+                        .build(), BacklogQuota.BacklogQuotaType.message_age);
+        PulsarClient client = PulsarClient.builder().serviceUrl(adminUrl.toString()).statsInterval(0, TimeUnit.SECONDS)
+                .build();
+
+        final String topic = "persistent://prop/ns-quota/topic4";
+        final String subName = "c1";
+
+        Consumer<byte[]> consumer = client.newConsumer().topic(topic).subscriptionName(subName).subscribe();
+        org.apache.pulsar.client.api.Producer<byte[]> producer = createProducer(client, topic);
+        producer.send(new byte[1024]);
+        consumer.receive();
+
+        admin.topics().unload(topic);
+        PersistentTopicInternalStats internalStats = admin.topics().getInternalStats(topic);
+        assertEquals(internalStats.ledgers.size(), 2);
+        assertEquals(internalStats.ledgers.get(1).entries, 0);
+
+        TopicStats stats = admin.topics().getStats(topic);
+        assertEquals(stats.getSubscriptions().get(subName).getMsgBacklog(), 1);
+
+        TimeUnit.SECONDS.sleep(TIME_TO_CHECK_BACKLOG_QUOTA);
+
+        Awaitility.await()
+                .pollInterval(Duration.ofSeconds(1))
+                .atMost(Duration.ofSeconds(TIME_TO_CHECK_BACKLOG_QUOTA))
+                .untilAsserted(() -> {
+                    rolloverStats();
+
+                    // Cause the last ledger is empty, it is not possible to skip first ledger,
+                    // so the number of ledgers will keep unchanged, and backlog is clear
+                    PersistentTopicInternalStats latestInternalStats = admin.topics().getInternalStats(topic);
+                    assertEquals(latestInternalStats.ledgers.size(), 2);
+                    assertEquals(latestInternalStats.ledgers.get(1).entries, 0);
+                    TopicStats latestStats = admin.topics().getStats(topic);
+                    assertEquals(latestStats.getSubscriptions().get(subName).getMsgBacklog(), 0);
+                });
+
+        client.close();
+    }
+
+    @Test
     public void testConsumerBacklogEvictionWithAckSizeQuota() throws Exception {
         assertEquals(admin.namespaces().getBacklogQuotaMap("prop/ns-quota"),
                 Maps.newHashMap());

[pulsar] 10/13: Use PulsarByteBufAllocator to allocate buffer for chunks at consumer side (#13536)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 2a73cdfd763efcc27c2ba0b26d755b5ed5331f39
Author: Yunze Xu <xy...@163.com>
AuthorDate: Tue Dec 28 20:35:21 2021 +0800

    Use PulsarByteBufAllocator to allocate buffer for chunks at consumer side (#13536)
    
    ### Motivation
    
    Currently Pulsar consumer allocates memory from direct memory via `Unpooled.directBuffer` directly, which doesn't make use of the widely used allocator in Pulsar.
    
    ### Modifications
    
    Use `PulsarByteBufAllocator` as the memory allocator for chunks buffer.
    
    (cherry picked from commit a5d347370286cb67ffa4ee179ec6fb5117546a4e)
---
 .../src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java      | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
index f3e6b99..3ddc5d6 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
@@ -26,7 +26,6 @@ import com.google.common.collect.Iterables;
 import com.scurrilous.circe.checksum.Crc32cIntChecksum;
 
 import io.netty.buffer.ByteBuf;
-import io.netty.buffer.Unpooled;
 import io.netty.channel.ChannelHandlerContext;
 import io.netty.util.Recycler;
 import io.netty.util.Recycler.Handle;
@@ -1113,7 +1112,7 @@ public class ConsumerImpl<T> extends ConsumerBase<T> implements ConnectionHandle
         }
 
         if (msgMetadata.getChunkId() == 0) {
-            ByteBuf chunkedMsgBuffer = Unpooled.directBuffer(msgMetadata.getTotalChunkMsgSize(),
+            ByteBuf chunkedMsgBuffer = PulsarByteBufAllocator.DEFAULT.buffer(msgMetadata.getTotalChunkMsgSize(),
                     msgMetadata.getTotalChunkMsgSize());
             int totalChunks = msgMetadata.getNumChunksFromMsg();
             chunkedMessagesMap.computeIfAbsent(msgMetadata.getUuid(),

[pulsar] 01/13: [Issue 9888] add python3.9 on manylinux2014 build support (#10954)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 7aac8b806159a53a9f3e9f195fe2f2d214926934
Author: Livio Benčik <lb...@gmail.com>
AuthorDate: Sun Jun 20 10:54:36 2021 +0200

    [Issue 9888] add python3.9 on manylinux2014 build support (#10954)
    
    * add python3.9 on manylinux2014 build support
    
    * add python3.9 client build CI
    
    * fix python3.9 client build CI
    
    Co-authored-by: livio <li...@mindsmiths.com>
    (cherry picked from commit 825187f3b2785b3d2e940920ab1e8fc1db92906f)
---
 .github/workflows/ci-python-build-3.9-client.yaml  | 62 ++++++++++++++++++++++
 .../create-images.sh => docker-build-python3.9.sh} | 50 +++++++++--------
 pulsar-client-cpp/docker/build-wheels.sh           |  1 +
 pulsar-client-cpp/docker/create-images.sh          | 17 +++---
 .../docker/{ => manylinux1}/Dockerfile             | 21 ++++----
 .../docker/{ => manylinux2014}/Dockerfile          | 23 ++++----
 6 files changed, 115 insertions(+), 59 deletions(-)

diff --git a/.github/workflows/ci-python-build-3.9-client.yaml b/.github/workflows/ci-python-build-3.9-client.yaml
new file mode 100644
index 0000000..f8b04a7
--- /dev/null
+++ b/.github/workflows/ci-python-build-3.9-client.yaml
@@ -0,0 +1,62 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: CI - Python - Build 3.9 client
+on:
+  pull_request:
+    branches:
+      - master
+    paths:
+      - '.github/workflows/**'
+      - 'pulsar-client-cpp/**'
+  push:
+    branches:
+      - branch-*
+    paths:
+      - '.github/workflows/**'
+      - 'pulsar-client-cpp/**'
+jobs:
+
+  build-wheel:
+    name:
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+
+    steps:
+      - name: checkout
+        uses: actions/checkout@v2
+
+      - name: Tune Runner VM
+        uses: ./.github/actions/tune-runner-vm
+
+      - name: Detect changed files
+        id:   changes
+        uses: apache/pulsar-test-infra/paths-filter@master
+        with:
+          filters: .github/changes-filter.yaml
+
+      - name: Check changed files
+        id: check_changes
+        run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"
+
+      - name: build python3.9 client
+        if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+        run: |
+          echo "Build Python3.9 client library"
+          pulsar-client-cpp/docker-build-python3.9.sh
\ No newline at end of file
diff --git a/pulsar-client-cpp/docker/create-images.sh b/pulsar-client-cpp/docker-build-python3.9.sh
similarity index 52%
copy from pulsar-client-cpp/docker/create-images.sh
copy to pulsar-client-cpp/docker-build-python3.9.sh
index 568f5f0..15cc2fd 100755
--- a/pulsar-client-cpp/docker/create-images.sh
+++ b/pulsar-client-cpp/docker-build-python3.9.sh
@@ -18,32 +18,30 @@
 # under the License.
 #
 
-
-# Create all the Docker images for variations of Python versions
+# Build Pulsar Python3.9 client
 
 set -e
 
-PYTHON_VERSIONS=(
-   '2.7 cp27-cp27mu'
-   '2.7 cp27-cp27m'
-   '3.5 cp35-cp35m'
-   '3.6 cp36-cp36m'
-   '3.7 cp37-cp37m'
-   '3.8 cp38-cp38'
-   '3.9 cp39-cp39'
-)
-
-for line in "${PYTHON_VERSIONS[@]}"; do
-    read -r -a PY <<< "$line"
-    PYTHON_VERSION=${PY[0]}
-    PYTHON_SPEC=${PY[1]}
-    echo "--------- Build Docker image for $PYTHON_VERSION -- $PYTHON_SPEC"
-
-    IMAGE_NAME=pulsar-build:manylinux-$PYTHON_SPEC
-
-    docker build -t $IMAGE_NAME . \
-            --build-arg PYTHON_VERSION=$PYTHON_VERSION \
-            --build-arg PYTHON_SPEC=$PYTHON_SPEC
-
-    echo "==== Successfully built image $IMAGE_NAME"
-done
+ROOT_DIR=$(git rev-parse --show-toplevel)
+cd $ROOT_DIR/pulsar-client-cpp
+
+
+# Build manylinux2014 build image
+PYTHON_VERSION="3.9"
+PYTHON_SPEC="cp39-cp39"
+IMAGE_NAME=pulsar-build:manylinux-$PYTHON_SPEC
+
+docker build -t $IMAGE_NAME ./docker/manylinux2014 \
+        --build-arg PYTHON_VERSION=$PYTHON_VERSION \
+        --build-arg PYTHON_SPEC=$PYTHON_SPEC
+
+
+# Build wheel file
+BUILD_IMAGE_NAME="${BUILD_IMAGE_NAME:-pulsar-build}"
+IMAGE=$BUILD_IMAGE_NAME:manylinux-$PYTHON_SPEC
+
+VOLUME_OPTION=${VOLUME_OPTION:-"-v $ROOT_DIR:/pulsar"}
+COMMAND="/pulsar/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh"
+DOCKER_CMD="docker run -i ${VOLUME_OPTION} -e USE_FULL_POM_NAME -e NAME_POSTFIX ${IMAGE}"
+
+$DOCKER_CMD bash -c "${COMMAND}"
diff --git a/pulsar-client-cpp/docker/build-wheels.sh b/pulsar-client-cpp/docker/build-wheels.sh
index 5371f85..a61f215 100755
--- a/pulsar-client-cpp/docker/build-wheels.sh
+++ b/pulsar-client-cpp/docker/build-wheels.sh
@@ -34,6 +34,7 @@ PYTHON_VERSIONS=(
    '3.6 cp36-cp36m'
    '3.7 cp37-cp37m'
    '3.8 cp38-cp38'
+   '3.9 cp39-cp39'
 )
 
 function contains() {
diff --git a/pulsar-client-cpp/docker/create-images.sh b/pulsar-client-cpp/docker/create-images.sh
index 568f5f0..6aa1d69 100755
--- a/pulsar-client-cpp/docker/create-images.sh
+++ b/pulsar-client-cpp/docker/create-images.sh
@@ -24,24 +24,25 @@
 set -e
 
 PYTHON_VERSIONS=(
-   '2.7 cp27-cp27mu'
-   '2.7 cp27-cp27m'
-   '3.5 cp35-cp35m'
-   '3.6 cp36-cp36m'
-   '3.7 cp37-cp37m'
-   '3.8 cp38-cp38'
-   '3.9 cp39-cp39'
+   '2.7 cp27-cp27mu manylinux1'
+   '2.7 cp27-cp27m manylinux1'
+   '3.5 cp35-cp35m manylinux2014'
+   '3.6 cp36-cp36m manylinux2014'
+   '3.7 cp37-cp37m manylinux2014'
+   '3.8 cp38-cp38 manylinux2014'
+   '3.9 cp39-cp39 manylinux2014'
 )
 
 for line in "${PYTHON_VERSIONS[@]}"; do
     read -r -a PY <<< "$line"
     PYTHON_VERSION=${PY[0]}
     PYTHON_SPEC=${PY[1]}
+    BASE_IMAGE=${PY[2]}
     echo "--------- Build Docker image for $PYTHON_VERSION -- $PYTHON_SPEC"
 
     IMAGE_NAME=pulsar-build:manylinux-$PYTHON_SPEC
 
-    docker build -t $IMAGE_NAME . \
+    docker build -t $IMAGE_NAME $BASE_IMAGE \
             --build-arg PYTHON_VERSION=$PYTHON_VERSION \
             --build-arg PYTHON_SPEC=$PYTHON_SPEC
 
diff --git a/pulsar-client-cpp/docker/Dockerfile b/pulsar-client-cpp/docker/manylinux1/Dockerfile
similarity index 97%
copy from pulsar-client-cpp/docker/Dockerfile
copy to pulsar-client-cpp/docker/manylinux1/Dockerfile
index 93f38b8..0df9240 100644
--- a/pulsar-client-cpp/docker/Dockerfile
+++ b/pulsar-client-cpp/docker/manylinux1/Dockerfile
@@ -62,7 +62,7 @@ RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_0j.tar.gz
     rm -rf /OpenSSL_1_1_0j.tar.gz /openssl-OpenSSL_1_1_0j
 
 # Download and compile boost
-RUN curl -O -L https://dl.bintray.com/boostorg/release/1.68.0/source/boost_1_68_0.tar.gz && \
+RUN curl -O -L https://boostorg.jfrog.io/artifactory/main/release/1.68.0/source/boost_1_68_0.tar.gz && \
     tar xvfz boost_1_68_0.tar.gz && \
     cd /boost_1_68_0 && \
     ./bootstrap.sh --with-libraries=program_options,filesystem,regex,thread,system,python && \
@@ -126,14 +126,6 @@ RUN curl -O -L https://github.com/Kitware/CMake/archive/v3.12.1.tar.gz && \
     make && make install && \
     rm -rf /v3.12.1.tar.gz /CMake-3.12.1
 
-# LibCurl
-RUN curl -O -L  https://github.com/curl/curl/releases/download/curl-7_61_0/curl-7.61.0.tar.gz && \
-    tar xvfz curl-7.61.0.tar.gz && \
-    cd curl-7.61.0 && \
-    CFLAGS=-fPIC ./configure --with-ssl=/usr/local/ssl/ && \
-    make && make install && \
-    rm -rf /curl-7.61.0.tar.gz /curl-7.61.0
-
 # Zstandard
 RUN curl -O -L https://github.com/facebook/zstd/releases/download/v1.3.7/zstd-1.3.7.tar.gz && \
     tar xvfz zstd-1.3.7.tar.gz && \
@@ -150,6 +142,14 @@ RUN curl -O -L https://github.com/google/snappy/releases/download/1.1.3/snappy-1
     make && make install && \
     rm -rf /snappy-1.1.3 /snappy-1.1.3.tar.gz
 
+# LibCurl
+RUN curl -O -L  https://github.com/curl/curl/releases/download/curl-7_61_0/curl-7.61.0.tar.gz && \
+    tar xvfz curl-7.61.0.tar.gz && \
+    cd curl-7.61.0 && \
+    CFLAGS=-fPIC ./configure --with-ssl=/usr/local/ssl/ && \
+    make && make install && \
+    rm -rf /curl-7.61.0.tar.gz /curl-7.61.0
+
 RUN pip install twine
 RUN pip install fastavro
 RUN pip install six
@@ -159,6 +159,3 @@ RUN pip install enum34
 ENV PYTHON_INCLUDE_DIR /opt/python/${PYTHON_SPEC}/include
 ENV PYTHON_LIBRARIES   /opt/python/${PYTHON_SPEC}/lib/python${PYTHON_VERSION}
 ENV OPENSSL_ROOT_DIR   /usr/local/ssl/
-
-COPY build-wheel-file-within-docker.sh /
-COPY build-client-lib-within-docker.sh /
diff --git a/pulsar-client-cpp/docker/Dockerfile b/pulsar-client-cpp/docker/manylinux2014/Dockerfile
similarity index 96%
rename from pulsar-client-cpp/docker/Dockerfile
rename to pulsar-client-cpp/docker/manylinux2014/Dockerfile
index 93f38b8..20a7247 100644
--- a/pulsar-client-cpp/docker/Dockerfile
+++ b/pulsar-client-cpp/docker/manylinux2014/Dockerfile
@@ -18,7 +18,7 @@
 #
 
 
-FROM quay.io/pypa/manylinux1_x86_64
+FROM quay.io/pypa/manylinux2014_x86_64
 
 RUN yum install -y gtest-devel
 
@@ -62,7 +62,7 @@ RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_0j.tar.gz
     rm -rf /OpenSSL_1_1_0j.tar.gz /openssl-OpenSSL_1_1_0j
 
 # Download and compile boost
-RUN curl -O -L https://dl.bintray.com/boostorg/release/1.68.0/source/boost_1_68_0.tar.gz && \
+RUN curl -O -L https://boostorg.jfrog.io/artifactory/main/release/1.68.0/source/boost_1_68_0.tar.gz && \
     tar xvfz boost_1_68_0.tar.gz && \
     cd /boost_1_68_0 && \
     ./bootstrap.sh --with-libraries=program_options,filesystem,regex,thread,system,python && \
@@ -126,14 +126,6 @@ RUN curl -O -L https://github.com/Kitware/CMake/archive/v3.12.1.tar.gz && \
     make && make install && \
     rm -rf /v3.12.1.tar.gz /CMake-3.12.1
 
-# LibCurl
-RUN curl -O -L  https://github.com/curl/curl/releases/download/curl-7_61_0/curl-7.61.0.tar.gz && \
-    tar xvfz curl-7.61.0.tar.gz && \
-    cd curl-7.61.0 && \
-    CFLAGS=-fPIC ./configure --with-ssl=/usr/local/ssl/ && \
-    make && make install && \
-    rm -rf /curl-7.61.0.tar.gz /curl-7.61.0
-
 # Zstandard
 RUN curl -O -L https://github.com/facebook/zstd/releases/download/v1.3.7/zstd-1.3.7.tar.gz && \
     tar xvfz zstd-1.3.7.tar.gz && \
@@ -150,6 +142,14 @@ RUN curl -O -L https://github.com/google/snappy/releases/download/1.1.3/snappy-1
     make && make install && \
     rm -rf /snappy-1.1.3 /snappy-1.1.3.tar.gz
 
+# LibCurl
+RUN curl -O -L  https://github.com/curl/curl/releases/download/curl-7_61_0/curl-7.61.0.tar.gz && \
+    tar xvfz curl-7.61.0.tar.gz && \
+    cd curl-7.61.0 && \
+    CFLAGS=-fPIC ./configure --with-ssl=/usr/local/ssl/ && \
+    make && make install && \
+    rm -rf /curl-7.61.0.tar.gz /curl-7.61.0
+
 RUN pip install twine
 RUN pip install fastavro
 RUN pip install six
@@ -159,6 +159,3 @@ RUN pip install enum34
 ENV PYTHON_INCLUDE_DIR /opt/python/${PYTHON_SPEC}/include
 ENV PYTHON_LIBRARIES   /opt/python/${PYTHON_SPEC}/lib/python${PYTHON_VERSION}
 ENV OPENSSL_ROOT_DIR   /usr/local/ssl/
-
-COPY build-wheel-file-within-docker.sh /
-COPY build-client-lib-within-docker.sh /

[pulsar] 12/13: Return message ID from compacted ledger while the compaction cursor reach the end of the topic (#13533)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit bdfa14d529dd2b8f0a3ac8219a815426938cf147
Author: lipenghui <pe...@apache.org>
AuthorDate: Wed Dec 29 13:26:57 2021 +0800

    Return message ID from compacted ledger while the compaction cursor reach the end of the topic (#13533)
    
    ### Motivation
    
    The problem happens when the compaction cursor reaches the end of the topic but the tail messages
    of the topic have been removed by producer writes null value messages during the topic compaction.
    
    For example:
    
    - 5 messages in the original topic with key: 0,1,2,3,4
    - the corresponding message IDs are: 1:0, 1:1, 1:2, 1:3, 1:4
    - producer send null value messages for key 3 and 4
    - trigger the topic compaction task
    
    After the compaction task complete,
    
    - 5 messages in the original topic: 1:0, 1:1, 1:2, 1:3, 1:4
    - 3 messages in the compacted ledger: 1:0, 1:1, 1:2
    
    At this moment, if the reader tries to get the last message ID of the topic,
    we should return `1:2` not `1:4`, because the reader is not able to read the message
    with keys `3` and `4` from the compacted topic, otherwise, the `reader.readNext()` method
    will be blocked until a new message written to the topic.
    
    ### Modifications
    
    The fix is straightforward, when the broker receives a get last message ID request,
    the broker will check if the compaction cursor reaches the end of the original topic.
    If yes, respond last message ID from the compacted ledger.
    
    ### Verifying this change
    
    New test added `testHasMessageAvailableWithNullValueMessage` which ensure the `hasMessageAvailable()`
    return false no more messages from the compacted topic if the compaction cursor reaches the end of the topic.
    
    (cherry picked from commit d49415e5a558ea9a82c93e55e77b2c3542eacb10)
---
 .../apache/pulsar/broker/service/ServerCnx.java    |  5 ++-
 .../apache/pulsar/compaction/CompactedTopic.java   |  2 +
 .../pulsar/compaction/CompactedTopicImpl.java      |  6 +--
 .../pulsar/compaction/CompactedTopicTest.java      | 48 +++++++++++++++++++++-
 4 files changed, 55 insertions(+), 6 deletions(-)

diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
index 9044488..824a2e8 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
@@ -1706,7 +1706,10 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
         ManagedLedgerImpl ml = (ManagedLedgerImpl) persistentTopic.getManagedLedger();
 
         // If it's not pointing to a valid entry, respond messageId of the current position.
-        if (lastPosition.getEntryId() == -1) {
+        // If the compaction cursor reach the end of the topic, respond messageId from compacted ledger
+        Optional<Position> compactionHorizon = persistentTopic.getCompactedTopic().getCompactionHorizon();
+        if (lastPosition.getEntryId() == -1 || (compactionHorizon.isPresent()
+                        && lastPosition.compareTo((PositionImpl) compactionHorizon.get()) <= 0)) {
             handleLastMessageIdFromCompactedLedger(persistentTopic, requestId, partitionIndex,
                     markDeletePosition);
             return;
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopic.java
index 31955a5..9e50fc0 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopic.java
@@ -18,6 +18,7 @@
  */
 package org.apache.pulsar.compaction;
 
+import java.util.Optional;
 import java.util.concurrent.CompletableFuture;
 import org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback;
 import org.apache.bookkeeper.mledger.Entry;
@@ -34,4 +35,5 @@ public interface CompactedTopic {
                                 ReadEntriesCallback callback,
                                 Consumer consumer);
     CompletableFuture<Entry> readLastEntryOfCompactedLedger();
+    Optional<Position> getCompactionHorizon();
 }
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
index a6d6fc9..aac213f 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
@@ -20,7 +20,6 @@ package org.apache.pulsar.compaction;
 
 import com.github.benmanes.caffeine.cache.AsyncLoadingCache;
 import com.github.benmanes.caffeine.cache.Caffeine;
-import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.ComparisonChain;
 import io.netty.buffer.ByteBuf;
 import java.util.ArrayList;
@@ -311,9 +310,8 @@ public class CompactedTopicImpl implements CompactedTopic {
             .compare(p.getEntryId(), m.getEntryId()).result();
     }
 
-    @VisibleForTesting
-    PositionImpl getCompactionHorizon() {
-        return this.compactionHorizon;
+    public synchronized Optional<Position> getCompactionHorizon() {
+        return Optional.ofNullable(this.compactionHorizon);
     }
     private static final Logger log = LoggerFactory.getLogger(CompactedTopicImpl.class);
 }
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java
index d44868a..4d00d28 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java
@@ -254,7 +254,8 @@ public class CompactedTopicTest extends MockedPulsarServiceBaseTest {
         Assert.assertTrue(compactedTopic.getCompactedTopicContext().isPresent());
         Assert.assertEquals(compactedTopic.getCompactedTopicContext().get().getLedger().getId(),
                 newCompactedLedger.getId());
-        Assert.assertEquals(compactedTopic.getCompactionHorizon(), newHorizon);
+        Assert.assertTrue(compactedTopic.getCompactionHorizon().isPresent());
+        Assert.assertEquals(compactedTopic.getCompactionHorizon().get(), newHorizon);
         compactedTopic.deleteCompactedLedger(oldCompactedLedger.getId()).join();
 
         // old ledger should be deleted, new still there
@@ -688,4 +689,49 @@ public class CompactedTopicTest extends MockedPulsarServiceBaseTest {
         Assert.assertFalse(reader.hasMessageAvailable());
     }
 
+    @Test
+    public void testHasMessageAvailableWithNullValueMessage() throws Exception {
+        String topic = "persistent://my-property/use/my-ns/testHasMessageAvailable-" +
+                UUID.randomUUID();
+        final int numMessages = 10;
+        @Cleanup
+        Producer<String> producer = pulsarClient.newProducer(Schema.STRING)
+                .topic(topic)
+                .blockIfQueueFull(true)
+                .enableBatching(false)
+                .create();
+        CompletableFuture<MessageId> lastMessage = null;
+        for (int i = 0; i < numMessages; ++i) {
+            lastMessage = producer.newMessage().key(i + "").value(String.format("msg [%d]", i)).sendAsync();
+        }
+
+        for (int i = numMessages / 2; i < numMessages; ++i) {
+            lastMessage = producer.newMessage().key(i + "").value(null).sendAsync();
+        }
+        producer.flush();
+        lastMessage.join();
+        admin.topics().triggerCompaction(topic);
+        Awaitility.await().untilAsserted(() -> {
+            PersistentTopicInternalStats stats = admin.topics().getInternalStats(topic);
+            Assert.assertNotEquals(stats.compactedLedger.ledgerId, -1);
+            Assert.assertEquals(stats.compactedLedger.entries, numMessages / 2);
+            Assert.assertEquals(admin.topics().getStats(topic)
+                    .getSubscriptions().get(COMPACTION_SUBSCRIPTION).getConsumers().size(), 0);
+            Assert.assertEquals(stats.lastConfirmedEntry, stats.cursors.get(COMPACTION_SUBSCRIPTION).markDeletePosition);
+        });
+
+        @Cleanup
+        Reader<byte[]> reader = pulsarClient.newReader()
+                .topic(topic)
+                .startMessageIdInclusive()
+                .startMessageId(MessageId.earliest)
+                .readCompacted(true)
+                .create();
+        for (int i = numMessages / 2; i < numMessages; ++i) {
+            reader.readNext();
+        }
+        Assert.assertFalse(reader.hasMessageAvailable());
+        Assert.assertNull(reader.readNext(3, TimeUnit.SECONDS));
+    }
+
 }

[pulsar] 03/13: Apply clang-format check for python wrapper (#13418)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 1a1c886d26c1648657981ff950807ae9bc491956
Author: Yunze Xu <xy...@163.com>
AuthorDate: Tue Dec 21 23:51:16 2021 +0800

    Apply clang-format check for python wrapper (#13418)
    
    (cherry picked from commit 636d5b4c2133bcdf8c1988bbe82eb1064565d5dc)
---
 pulsar-client-cpp/CMakeLists.txt                |   6 +-
 pulsar-client-cpp/python/src/authentication.cc  |  44 ++--
 pulsar-client-cpp/python/src/client.cc          |  63 +++---
 pulsar-client-cpp/python/src/config.cc          | 256 ++++++++++++------------
 pulsar-client-cpp/python/src/consumer.cc        |  81 ++++----
 pulsar-client-cpp/python/src/cryptoKeyReader.cc |   2 +-
 pulsar-client-cpp/python/src/enums.cc           | 162 +++++++--------
 pulsar-client-cpp/python/src/exceptions.cc      |  49 +++--
 pulsar-client-cpp/python/src/message.cc         | 147 ++++++--------
 pulsar-client-cpp/python/src/producer.cc        |  71 ++++---
 pulsar-client-cpp/python/src/pulsar.cc          |   4 +-
 pulsar-client-cpp/python/src/reader.cc          |  48 ++---
 pulsar-client-cpp/python/src/schema.cc          |  10 +-
 pulsar-client-cpp/python/src/utils.h            |   3 +-
 14 files changed, 441 insertions(+), 505 deletions(-)

diff --git a/pulsar-client-cpp/CMakeLists.txt b/pulsar-client-cpp/CMakeLists.txt
index 22d5605..51f0352 100644
--- a/pulsar-client-cpp/CMakeLists.txt
+++ b/pulsar-client-cpp/CMakeLists.txt
@@ -444,7 +444,8 @@ add_custom_target(format python ${BUILD_SUPPORT_DIR}/run_clang_format.py
         ${CMAKE_SOURCE_DIR}/perf
         ${CMAKE_SOURCE_DIR}/examples
         ${CMAKE_SOURCE_DIR}/tests
-        ${CMAKE_SOURCE_DIR}/include)
+        ${CMAKE_SOURCE_DIR}/include
+        ${CMAKE_SOURCE_DIR}/python/src)
 
 # `make check-format` option (for CI test)
 add_custom_target(check-format python ${BUILD_SUPPORT_DIR}/run_clang_format.py
@@ -455,4 +456,5 @@ add_custom_target(check-format python ${BUILD_SUPPORT_DIR}/run_clang_format.py
         ${CMAKE_SOURCE_DIR}/perf
         ${CMAKE_SOURCE_DIR}/examples
         ${CMAKE_SOURCE_DIR}/tests
-        ${CMAKE_SOURCE_DIR}/include)
+        ${CMAKE_SOURCE_DIR}/include
+        ${CMAKE_SOURCE_DIR}/python/src)
\ No newline at end of file
diff --git a/pulsar-client-cpp/python/src/authentication.cc b/pulsar-client-cpp/python/src/authentication.cc
index e236c7e..920a717 100644
--- a/pulsar-client-cpp/python/src/authentication.cc
+++ b/pulsar-client-cpp/python/src/authentication.cc
@@ -26,8 +26,8 @@ AuthenticationWrapper::AuthenticationWrapper(const std::string& dynamicLibPath,
 }
 
 struct AuthenticationTlsWrapper : public AuthenticationWrapper {
-    AuthenticationTlsWrapper(const std::string& certificatePath, const std::string& privateKeyPath) :
-            AuthenticationWrapper() {
+    AuthenticationTlsWrapper(const std::string& certificatePath, const std::string& privateKeyPath)
+        : AuthenticationWrapper() {
         this->auth = AuthTls::create(certificatePath, privateKeyPath);
     }
 };
@@ -35,13 +35,10 @@ struct AuthenticationTlsWrapper : public AuthenticationWrapper {
 struct TokenSupplierWrapper {
     PyObject* _pySupplier;
 
-    TokenSupplierWrapper(py::object pySupplier) :
-        _pySupplier(pySupplier.ptr()) {
-        Py_XINCREF(_pySupplier);
-    }
+    TokenSupplierWrapper(py::object pySupplier) : _pySupplier(pySupplier.ptr()) { Py_XINCREF(_pySupplier); }
 
     TokenSupplierWrapper(const TokenSupplierWrapper& other) {
-        _pySupplier= other._pySupplier;
+        _pySupplier = other._pySupplier;
         Py_XINCREF(_pySupplier);
     }
 
@@ -51,9 +48,7 @@ struct TokenSupplierWrapper {
         return *this;
     }
 
-    virtual ~TokenSupplierWrapper() {
-        Py_XDECREF(_pySupplier);
-    }
+    virtual ~TokenSupplierWrapper() { Py_XDECREF(_pySupplier); }
 
     std::string operator()() {
         PyGILState_STATE state = PyGILState_Ensure();
@@ -61,7 +56,7 @@ struct TokenSupplierWrapper {
         std::string token;
         try {
             token = py::call<std::string>(_pySupplier);
-        } catch(const py::error_already_set& e) {
+        } catch (const py::error_already_set& e) {
             PyErr_Print();
         }
 
@@ -70,10 +65,8 @@ struct TokenSupplierWrapper {
     }
 };
 
-
 struct AuthenticationTokenWrapper : public AuthenticationWrapper {
-    AuthenticationTokenWrapper(py::object token) :
-            AuthenticationWrapper() {
+    AuthenticationTokenWrapper(py::object token) : AuthenticationWrapper() {
         if (py::extract<std::string>(token).check()) {
             // It's a string
             std::string tokenStr = py::extract<std::string>(token);
@@ -86,15 +79,13 @@ struct AuthenticationTokenWrapper : public AuthenticationWrapper {
 };
 
 struct AuthenticationAthenzWrapper : public AuthenticationWrapper {
-    AuthenticationAthenzWrapper(const std::string& authParamsString) :
-            AuthenticationWrapper() {
+    AuthenticationAthenzWrapper(const std::string& authParamsString) : AuthenticationWrapper() {
         this->auth = AuthAthenz::create(authParamsString);
     }
 };
 
 struct AuthenticationOauth2Wrapper : public AuthenticationWrapper {
-    AuthenticationOauth2Wrapper(const std::string& authParamsString) :
-            AuthenticationWrapper() {
+    AuthenticationOauth2Wrapper(const std::string& authParamsString) : AuthenticationWrapper() {
         this->auth = AuthOauth2::create(authParamsString);
     }
 };
@@ -102,22 +93,17 @@ struct AuthenticationOauth2Wrapper : public AuthenticationWrapper {
 void export_authentication() {
     using namespace boost::python;
 
-    class_<AuthenticationWrapper>("Authentication", init<const std::string&, const std::string&>())
-            ;
+    class_<AuthenticationWrapper>("Authentication", init<const std::string&, const std::string&>());
 
-    class_<AuthenticationTlsWrapper, bases<AuthenticationWrapper> >("AuthenticationTLS",
-                                                                    init<const std::string&, const std::string&>())
-            ;
+    class_<AuthenticationTlsWrapper, bases<AuthenticationWrapper> >(
+        "AuthenticationTLS", init<const std::string&, const std::string&>());
 
     class_<AuthenticationTokenWrapper, bases<AuthenticationWrapper> >("AuthenticationToken",
-                                                                    init<py::object>())
-            ;
+                                                                      init<py::object>());
 
     class_<AuthenticationAthenzWrapper, bases<AuthenticationWrapper> >("AuthenticationAthenz",
-                                                                       init<const std::string&>())
-            ;
+                                                                       init<const std::string&>());
 
     class_<AuthenticationOauth2Wrapper, bases<AuthenticationWrapper> >("AuthenticationOauth2",
-                                                                       init<const std::string&>())
-            ;
+                                                                       init<const std::string&>());
 }
diff --git a/pulsar-client-cpp/python/src/client.cc b/pulsar-client-cpp/python/src/client.cc
index 3dcbf7f..445a6dc 100644
--- a/pulsar-client-cpp/python/src/client.cc
+++ b/pulsar-client-cpp/python/src/client.cc
@@ -22,11 +22,10 @@ Producer Client_createProducer(Client& client, const std::string& topic, const P
     Producer producer;
     Result res;
 
-    Py_BEGIN_ALLOW_THREADS
-    res = client.createProducer(topic, conf, producer);
+    Py_BEGIN_ALLOW_THREADS res = client.createProducer(topic, conf, producer);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
     return producer;
 }
 
@@ -35,11 +34,10 @@ Consumer Client_subscribe(Client& client, const std::string& topic, const std::s
     Consumer consumer;
     Result res;
 
-    Py_BEGIN_ALLOW_THREADS
-    res = client.subscribe(topic, subscriptionName, conf, consumer);
+    Py_BEGIN_ALLOW_THREADS res = client.subscribe(topic, subscriptionName, conf, consumer);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
     return consumer;
 }
 
@@ -50,43 +48,39 @@ Consumer Client_subscribe_topics(Client& client, boost::python::list& topics,
 
     std::vector<std::string> topics_vector;
 
-    for (int i = 0; i < len(topics); i ++) {
+    for (int i = 0; i < len(topics); i++) {
         std::string content = boost::python::extract<std::string>(topics[i]);
         topics_vector.push_back(content);
     }
 
-    Py_BEGIN_ALLOW_THREADS
-        res = client.subscribe(topics_vector, subscriptionName, conf, consumer);
+    Py_BEGIN_ALLOW_THREADS res = client.subscribe(topics_vector, subscriptionName, conf, consumer);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
     return consumer;
 }
 
-Consumer Client_subscribe_pattern(Client& client, const std::string& topic_pattern, const std::string& subscriptionName,
-                                 const ConsumerConfiguration& conf) {
+Consumer Client_subscribe_pattern(Client& client, const std::string& topic_pattern,
+                                  const std::string& subscriptionName, const ConsumerConfiguration& conf) {
     Consumer consumer;
     Result res;
 
-    Py_BEGIN_ALLOW_THREADS
-        res = client.subscribeWithRegex(topic_pattern, subscriptionName, conf, consumer);
+    Py_BEGIN_ALLOW_THREADS res = client.subscribeWithRegex(topic_pattern, subscriptionName, conf, consumer);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
     return consumer;
 }
 
-Reader Client_createReader(Client& client, const std::string& topic,
-                           const MessageId& startMessageId,
+Reader Client_createReader(Client& client, const std::string& topic, const MessageId& startMessageId,
                            const ReaderConfiguration& conf) {
     Reader reader;
     Result res;
 
-    Py_BEGIN_ALLOW_THREADS
-    res = client.createReader(topic, startMessageId, conf, reader);
+    Py_BEGIN_ALLOW_THREADS res = client.createReader(topic, startMessageId, conf, reader);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
     return reader;
 }
 
@@ -94,11 +88,10 @@ boost::python::list Client_getTopicPartitions(Client& client, const std::string&
     std::vector<std::string> partitions;
     Result res;
 
-    Py_BEGIN_ALLOW_THREADS
-    res = client.getPartitionsForTopic(topic, partitions);
+    Py_BEGIN_ALLOW_THREADS res = client.getPartitionsForTopic(topic, partitions);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 
     boost::python::list pyList;
     for (int i = 0; i < partitions.size(); i++) {
@@ -111,24 +104,22 @@ boost::python::list Client_getTopicPartitions(Client& client, const std::string&
 void Client_close(Client& client) {
     Result res;
 
-    Py_BEGIN_ALLOW_THREADS
-    res = client.close();
+    Py_BEGIN_ALLOW_THREADS res = client.close();
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
 void export_client() {
     using namespace boost::python;
 
-    class_<Client>("Client", init<const std::string&, const ClientConfiguration& >())
-            .def("create_producer", &Client_createProducer)
-            .def("subscribe", &Client_subscribe)
-            .def("subscribe_topics", &Client_subscribe_topics)
-            .def("subscribe_pattern", &Client_subscribe_pattern)
-            .def("create_reader", &Client_createReader)
-            .def("get_topic_partitions", &Client_getTopicPartitions)
-            .def("close", &Client_close)
-            .def("shutdown", &Client::shutdown)
-            ;
+    class_<Client>("Client", init<const std::string&, const ClientConfiguration&>())
+        .def("create_producer", &Client_createProducer)
+        .def("subscribe", &Client_subscribe)
+        .def("subscribe_topics", &Client_subscribe_topics)
+        .def("subscribe_pattern", &Client_subscribe_pattern)
+        .def("create_reader", &Client_createReader)
+        .def("get_topic_partitions", &Client_getTopicPartitions)
+        .def("close", &Client_close)
+        .def("shutdown", &Client::shutdown);
 }
diff --git a/pulsar-client-cpp/python/src/config.cc b/pulsar-client-cpp/python/src/config.cc
index a287248..2dee1a1 100644
--- a/pulsar-client-cpp/python/src/config.cc
+++ b/pulsar-client-cpp/python/src/config.cc
@@ -21,14 +21,11 @@
 #include "lib/Utils.h"
 #include <memory>
 
-template<typename T>
+template <typename T>
 struct ListenerWrapper {
     PyObject* _pyListener;
 
-    ListenerWrapper(py::object pyListener) :
-        _pyListener(pyListener.ptr()) {
-        Py_XINCREF(_pyListener);
-    }
+    ListenerWrapper(py::object pyListener) : _pyListener(pyListener.ptr()) { Py_XINCREF(_pyListener); }
 
     ListenerWrapper(const ListenerWrapper& other) {
         _pyListener = other._pyListener;
@@ -41,9 +38,7 @@ struct ListenerWrapper {
         return *this;
     }
 
-    virtual ~ListenerWrapper() {
-        Py_XDECREF(_pyListener);
-    }
+    virtual ~ListenerWrapper() { Py_XDECREF(_pyListener); }
 
     void operator()(T consumer, const Message& msg) {
         PyGILState_STATE state = PyGILState_Ensure();
@@ -65,7 +60,7 @@ static ConsumerConfiguration& ConsumerConfiguration_setMessageListener(ConsumerC
 }
 
 static ReaderConfiguration& ReaderConfiguration_setReaderListener(ReaderConfiguration& conf,
-                                                                   py::object pyListener) {
+                                                                  py::object pyListener) {
     conf.setReaderListener(ListenerWrapper<Reader>(pyListener));
     return conf;
 }
@@ -78,41 +73,36 @@ static ClientConfiguration& ClientConfiguration_setAuthentication(ClientConfigur
 }
 
 static ConsumerConfiguration& ConsumerConfiguration_setCryptoKeyReader(ConsumerConfiguration& conf,
-                                                                        py::object cryptoKeyReader) {
+                                                                       py::object cryptoKeyReader) {
     CryptoKeyReaderWrapper cryptoKeyReaderWrapper = py::extract<CryptoKeyReaderWrapper>(cryptoKeyReader);
     conf.setCryptoKeyReader(cryptoKeyReaderWrapper.cryptoKeyReader);
     return conf;
 }
 
 static ProducerConfiguration& ProducerConfiguration_setCryptoKeyReader(ProducerConfiguration& conf,
-                                                                        py::object cryptoKeyReader) {
+                                                                       py::object cryptoKeyReader) {
     CryptoKeyReaderWrapper cryptoKeyReaderWrapper = py::extract<CryptoKeyReaderWrapper>(cryptoKeyReader);
     conf.setCryptoKeyReader(cryptoKeyReaderWrapper.cryptoKeyReader);
     return conf;
 }
 
 static ReaderConfiguration& ReaderConfiguration_setCryptoKeyReader(ReaderConfiguration& conf,
-                                                                        py::object cryptoKeyReader) {
+                                                                   py::object cryptoKeyReader) {
     CryptoKeyReaderWrapper cryptoKeyReaderWrapper = py::extract<CryptoKeyReaderWrapper>(cryptoKeyReader);
     conf.setCryptoKeyReader(cryptoKeyReaderWrapper.cryptoKeyReader);
     return conf;
 }
 
-class LoggerWrapper: public Logger {
+class LoggerWrapper : public Logger {
     PyObject* const _pyLogger;
     const int _pythonLogLevel;
     const std::unique_ptr<Logger> _fallbackLogger;
 
-    static constexpr int _getLogLevelValue(Level level) {
-        return 10 + (level * 10);
-    }
+    static constexpr int _getLogLevelValue(Level level) { return 10 + (level * 10); }
 
    public:
-
     LoggerWrapper(PyObject* pyLogger, int pythonLogLevel, Logger* fallbackLogger)
-        : _pyLogger(pyLogger),
-          _pythonLogLevel(pythonLogLevel),
-          _fallbackLogger(fallbackLogger) {
+        : _pyLogger(pyLogger), _pythonLogLevel(pythonLogLevel), _fallbackLogger(fallbackLogger) {
         Py_XINCREF(_pyLogger);
     }
 
@@ -121,13 +111,9 @@ class LoggerWrapper: public Logger {
     LoggerWrapper& operator=(const LoggerWrapper&) = delete;
     LoggerWrapper& operator=(LoggerWrapper&&) = delete;
 
-    virtual ~LoggerWrapper() {
-        Py_XDECREF(_pyLogger);
-    }
+    virtual ~LoggerWrapper() { Py_XDECREF(_pyLogger); }
 
-    bool isEnabled(Level level) {
-        return _getLogLevelValue(level) >= _pythonLogLevel;
-    }
+    bool isEnabled(Level level) { return _getLogLevelValue(level) >= _pythonLogLevel; }
 
     void log(Level level, int line, const std::string& message) {
         if (!Py_IsInitialized()) {
@@ -187,11 +173,9 @@ class LoggerWrapperFactory : public LoggerFactory {
         initializePythonLogLevel();
     }
 
-    virtual ~LoggerWrapperFactory() {
-        Py_XDECREF(_pyLogger);
-    }
+    virtual ~LoggerWrapperFactory() { Py_XDECREF(_pyLogger); }
 
-    Logger* getLogger(const std::string &fileName) {
+    Logger* getLogger(const std::string& fileName) {
         const auto fallbackLogger = _fallbackLoggerFactory->getLogger(fileName);
         if (_pythonLogLevel.is_present()) {
             return new LoggerWrapper(_pyLogger, _pythonLogLevel.value(), fallbackLogger);
@@ -206,112 +190,128 @@ static ClientConfiguration& ClientConfiguration_setLogger(ClientConfiguration& c
     return conf;
 }
 
-
 void export_config() {
     using namespace boost::python;
 
     class_<ClientConfiguration>("ClientConfiguration")
-            .def("authentication", &ClientConfiguration_setAuthentication, return_self<>())
-            .def("operation_timeout_seconds", &ClientConfiguration::getOperationTimeoutSeconds)
-            .def("operation_timeout_seconds", &ClientConfiguration::setOperationTimeoutSeconds, return_self<>())
-            .def("connection_timeout", &ClientConfiguration::getConnectionTimeout)
-            .def("connection_timeout", &ClientConfiguration::setConnectionTimeout, return_self<>())
-            .def("io_threads", &ClientConfiguration::getIOThreads)
-            .def("io_threads", &ClientConfiguration::setIOThreads, return_self<>())
-            .def("message_listener_threads", &ClientConfiguration::getMessageListenerThreads)
-            .def("message_listener_threads", &ClientConfiguration::setMessageListenerThreads, return_self<>())
-            .def("concurrent_lookup_requests", &ClientConfiguration::getConcurrentLookupRequest)
-            .def("concurrent_lookup_requests", &ClientConfiguration::setConcurrentLookupRequest, return_self<>())
-            .def("log_conf_file_path", &ClientConfiguration::getLogConfFilePath, return_value_policy<copy_const_reference>())
-            .def("log_conf_file_path", &ClientConfiguration::setLogConfFilePath, return_self<>())
-            .def("use_tls", &ClientConfiguration::isUseTls)
-            .def("use_tls", &ClientConfiguration::setUseTls, return_self<>())
-            .def("tls_trust_certs_file_path", &ClientConfiguration::getTlsTrustCertsFilePath, return_value_policy<copy_const_reference>())
-            .def("tls_trust_certs_file_path", &ClientConfiguration::setTlsTrustCertsFilePath, return_self<>())
-            .def("tls_allow_insecure_connection", &ClientConfiguration::isTlsAllowInsecureConnection)
-            .def("tls_allow_insecure_connection", &ClientConfiguration::setTlsAllowInsecureConnection, return_self<>())
-            .def("tls_validate_hostname", &ClientConfiguration::setValidateHostName, return_self<>())
-            .def("set_logger", &ClientConfiguration_setLogger, return_self<>())
-            ;
+        .def("authentication", &ClientConfiguration_setAuthentication, return_self<>())
+        .def("operation_timeout_seconds", &ClientConfiguration::getOperationTimeoutSeconds)
+        .def("operation_timeout_seconds", &ClientConfiguration::setOperationTimeoutSeconds, return_self<>())
+        .def("connection_timeout", &ClientConfiguration::getConnectionTimeout)
+        .def("connection_timeout", &ClientConfiguration::setConnectionTimeout, return_self<>())
+        .def("io_threads", &ClientConfiguration::getIOThreads)
+        .def("io_threads", &ClientConfiguration::setIOThreads, return_self<>())
+        .def("message_listener_threads", &ClientConfiguration::getMessageListenerThreads)
+        .def("message_listener_threads", &ClientConfiguration::setMessageListenerThreads, return_self<>())
+        .def("concurrent_lookup_requests", &ClientConfiguration::getConcurrentLookupRequest)
+        .def("concurrent_lookup_requests", &ClientConfiguration::setConcurrentLookupRequest, return_self<>())
+        .def("log_conf_file_path", &ClientConfiguration::getLogConfFilePath,
+             return_value_policy<copy_const_reference>())
+        .def("log_conf_file_path", &ClientConfiguration::setLogConfFilePath, return_self<>())
+        .def("use_tls", &ClientConfiguration::isUseTls)
+        .def("use_tls", &ClientConfiguration::setUseTls, return_self<>())
+        .def("tls_trust_certs_file_path", &ClientConfiguration::getTlsTrustCertsFilePath,
+             return_value_policy<copy_const_reference>())
+        .def("tls_trust_certs_file_path", &ClientConfiguration::setTlsTrustCertsFilePath, return_self<>())
+        .def("tls_allow_insecure_connection", &ClientConfiguration::isTlsAllowInsecureConnection)
+        .def("tls_allow_insecure_connection", &ClientConfiguration::setTlsAllowInsecureConnection,
+             return_self<>())
+        .def("tls_validate_hostname", &ClientConfiguration::setValidateHostName, return_self<>())
+        .def("set_logger", &ClientConfiguration_setLogger, return_self<>());
 
     class_<ProducerConfiguration>("ProducerConfiguration")
-            .def("producer_name", &ProducerConfiguration::getProducerName, return_value_policy<copy_const_reference>())
-            .def("producer_name", &ProducerConfiguration::setProducerName, return_self<>())
-            .def("schema", &ProducerConfiguration::getSchema, return_value_policy<copy_const_reference>())
-            .def("schema", &ProducerConfiguration::setSchema, return_self<>())
-            .def("send_timeout_millis", &ProducerConfiguration::getSendTimeout)
-            .def("send_timeout_millis", &ProducerConfiguration::setSendTimeout, return_self<>())
-            .def("initial_sequence_id", &ProducerConfiguration::getInitialSequenceId)
-            .def("initial_sequence_id", &ProducerConfiguration::setInitialSequenceId, return_self<>())
-            .def("compression_type", &ProducerConfiguration::getCompressionType)
-            .def("compression_type", &ProducerConfiguration::setCompressionType, return_self<>())
-            .def("max_pending_messages", &ProducerConfiguration::getMaxPendingMessages)
-            .def("max_pending_messages", &ProducerConfiguration::setMaxPendingMessages, return_self<>())
-            .def("max_pending_messages_across_partitions", &ProducerConfiguration::getMaxPendingMessagesAcrossPartitions)
-            .def("max_pending_messages_across_partitions", &ProducerConfiguration::setMaxPendingMessagesAcrossPartitions, return_self<>())
-            .def("block_if_queue_full", &ProducerConfiguration::getBlockIfQueueFull)
-            .def("block_if_queue_full", &ProducerConfiguration::setBlockIfQueueFull, return_self<>())
-            .def("partitions_routing_mode", &ProducerConfiguration::getPartitionsRoutingMode)
-            .def("partitions_routing_mode", &ProducerConfiguration::setPartitionsRoutingMode, return_self<>())
-            .def("lazy_start_partitioned_producers", &ProducerConfiguration::getLazyStartPartitionedProducers)
-            .def("lazy_start_partitioned_producers", &ProducerConfiguration::setLazyStartPartitionedProducers, return_self<>())
-            .def("batching_enabled", &ProducerConfiguration::getBatchingEnabled, return_value_policy<copy_const_reference>())
-            .def("batching_enabled", &ProducerConfiguration::setBatchingEnabled, return_self<>())
-            .def("batching_max_messages", &ProducerConfiguration::getBatchingMaxMessages, return_value_policy<copy_const_reference>())
-            .def("batching_max_messages", &ProducerConfiguration::setBatchingMaxMessages, return_self<>())
-            .def("batching_max_allowed_size_in_bytes", &ProducerConfiguration::getBatchingMaxAllowedSizeInBytes, return_value_policy<copy_const_reference>())
-            .def("batching_max_allowed_size_in_bytes", &ProducerConfiguration::setBatchingMaxAllowedSizeInBytes, return_self<>())
-            .def("batching_max_publish_delay_ms", &ProducerConfiguration::getBatchingMaxPublishDelayMs, return_value_policy<copy_const_reference>())
-            .def("batching_max_publish_delay_ms", &ProducerConfiguration::setBatchingMaxPublishDelayMs, return_self<>())
-            .def("property", &ProducerConfiguration::setProperty, return_self<>())
-            .def("batching_type", &ProducerConfiguration::setBatchingType, return_self<>())
-            .def("batching_type", &ProducerConfiguration::getBatchingType)
-            .def("encryption_key", &ProducerConfiguration::addEncryptionKey, return_self<>())
-            .def("crypto_key_reader", &ProducerConfiguration_setCryptoKeyReader, return_self<>())
-            ;
+        .def("producer_name", &ProducerConfiguration::getProducerName,
+             return_value_policy<copy_const_reference>())
+        .def("producer_name", &ProducerConfiguration::setProducerName, return_self<>())
+        .def("schema", &ProducerConfiguration::getSchema, return_value_policy<copy_const_reference>())
+        .def("schema", &ProducerConfiguration::setSchema, return_self<>())
+        .def("send_timeout_millis", &ProducerConfiguration::getSendTimeout)
+        .def("send_timeout_millis", &ProducerConfiguration::setSendTimeout, return_self<>())
+        .def("initial_sequence_id", &ProducerConfiguration::getInitialSequenceId)
+        .def("initial_sequence_id", &ProducerConfiguration::setInitialSequenceId, return_self<>())
+        .def("compression_type", &ProducerConfiguration::getCompressionType)
+        .def("compression_type", &ProducerConfiguration::setCompressionType, return_self<>())
+        .def("max_pending_messages", &ProducerConfiguration::getMaxPendingMessages)
+        .def("max_pending_messages", &ProducerConfiguration::setMaxPendingMessages, return_self<>())
+        .def("max_pending_messages_across_partitions",
+             &ProducerConfiguration::getMaxPendingMessagesAcrossPartitions)
+        .def("max_pending_messages_across_partitions",
+             &ProducerConfiguration::setMaxPendingMessagesAcrossPartitions, return_self<>())
+        .def("block_if_queue_full", &ProducerConfiguration::getBlockIfQueueFull)
+        .def("block_if_queue_full", &ProducerConfiguration::setBlockIfQueueFull, return_self<>())
+        .def("partitions_routing_mode", &ProducerConfiguration::getPartitionsRoutingMode)
+        .def("partitions_routing_mode", &ProducerConfiguration::setPartitionsRoutingMode, return_self<>())
+        .def("lazy_start_partitioned_producers", &ProducerConfiguration::getLazyStartPartitionedProducers)
+        .def("lazy_start_partitioned_producers", &ProducerConfiguration::setLazyStartPartitionedProducers,
+             return_self<>())
+        .def("batching_enabled", &ProducerConfiguration::getBatchingEnabled,
+             return_value_policy<copy_const_reference>())
+        .def("batching_enabled", &ProducerConfiguration::setBatchingEnabled, return_self<>())
+        .def("batching_max_messages", &ProducerConfiguration::getBatchingMaxMessages,
+             return_value_policy<copy_const_reference>())
+        .def("batching_max_messages", &ProducerConfiguration::setBatchingMaxMessages, return_self<>())
+        .def("batching_max_allowed_size_in_bytes", &ProducerConfiguration::getBatchingMaxAllowedSizeInBytes,
+             return_value_policy<copy_const_reference>())
+        .def("batching_max_allowed_size_in_bytes", &ProducerConfiguration::setBatchingMaxAllowedSizeInBytes,
+             return_self<>())
+        .def("batching_max_publish_delay_ms", &ProducerConfiguration::getBatchingMaxPublishDelayMs,
+             return_value_policy<copy_const_reference>())
+        .def("batching_max_publish_delay_ms", &ProducerConfiguration::setBatchingMaxPublishDelayMs,
+             return_self<>())
+        .def("property", &ProducerConfiguration::setProperty, return_self<>())
+        .def("batching_type", &ProducerConfiguration::setBatchingType, return_self<>())
+        .def("batching_type", &ProducerConfiguration::getBatchingType)
+        .def("encryption_key", &ProducerConfiguration::addEncryptionKey, return_self<>())
+        .def("crypto_key_reader", &ProducerConfiguration_setCryptoKeyReader, return_self<>());
 
     class_<ConsumerConfiguration>("ConsumerConfiguration")
-            .def("consumer_type", &ConsumerConfiguration::getConsumerType)
-            .def("consumer_type", &ConsumerConfiguration::setConsumerType, return_self<>())
-            .def("schema", &ConsumerConfiguration::getSchema, return_value_policy<copy_const_reference>())
-            .def("schema", &ConsumerConfiguration::setSchema, return_self<>())
-            .def("message_listener", &ConsumerConfiguration_setMessageListener, return_self<>())
-            .def("receiver_queue_size", &ConsumerConfiguration::getReceiverQueueSize)
-            .def("receiver_queue_size", &ConsumerConfiguration::setReceiverQueueSize)
-            .def("max_total_receiver_queue_size_across_partitions", &ConsumerConfiguration::getMaxTotalReceiverQueueSizeAcrossPartitions)
-            .def("max_total_receiver_queue_size_across_partitions", &ConsumerConfiguration::setMaxTotalReceiverQueueSizeAcrossPartitions)
-            .def("consumer_name", &ConsumerConfiguration::getConsumerName, return_value_policy<copy_const_reference>())
-            .def("consumer_name", &ConsumerConfiguration::setConsumerName)
-            .def("unacked_messages_timeout_ms", &ConsumerConfiguration::getUnAckedMessagesTimeoutMs)
-            .def("unacked_messages_timeout_ms", &ConsumerConfiguration::setUnAckedMessagesTimeoutMs)
-            .def("negative_ack_redelivery_delay_ms", &ConsumerConfiguration::getNegativeAckRedeliveryDelayMs)
-            .def("negative_ack_redelivery_delay_ms", &ConsumerConfiguration::setNegativeAckRedeliveryDelayMs)
-            .def("broker_consumer_stats_cache_time_ms", &ConsumerConfiguration::getBrokerConsumerStatsCacheTimeInMs)
-            .def("broker_consumer_stats_cache_time_ms", &ConsumerConfiguration::setBrokerConsumerStatsCacheTimeInMs)
-            .def("pattern_auto_discovery_period", &ConsumerConfiguration::getPatternAutoDiscoveryPeriod)
-            .def("pattern_auto_discovery_period", &ConsumerConfiguration::setPatternAutoDiscoveryPeriod)
-            .def("read_compacted", &ConsumerConfiguration::isReadCompacted)
-            .def("read_compacted", &ConsumerConfiguration::setReadCompacted)
-            .def("property", &ConsumerConfiguration::setProperty, return_self<>())
-            .def("subscription_initial_position", &ConsumerConfiguration::getSubscriptionInitialPosition)
-            .def("subscription_initial_position", &ConsumerConfiguration::setSubscriptionInitialPosition)
-            .def("crypto_key_reader", &ConsumerConfiguration_setCryptoKeyReader, return_self<>())
-            .def("replicate_subscription_state_enabled", &ConsumerConfiguration::setReplicateSubscriptionStateEnabled)
-            .def("replicate_subscription_state_enabled", &ConsumerConfiguration::isReplicateSubscriptionStateEnabled)
-            ;
+        .def("consumer_type", &ConsumerConfiguration::getConsumerType)
+        .def("consumer_type", &ConsumerConfiguration::setConsumerType, return_self<>())
+        .def("schema", &ConsumerConfiguration::getSchema, return_value_policy<copy_const_reference>())
+        .def("schema", &ConsumerConfiguration::setSchema, return_self<>())
+        .def("message_listener", &ConsumerConfiguration_setMessageListener, return_self<>())
+        .def("receiver_queue_size", &ConsumerConfiguration::getReceiverQueueSize)
+        .def("receiver_queue_size", &ConsumerConfiguration::setReceiverQueueSize)
+        .def("max_total_receiver_queue_size_across_partitions",
+             &ConsumerConfiguration::getMaxTotalReceiverQueueSizeAcrossPartitions)
+        .def("max_total_receiver_queue_size_across_partitions",
+             &ConsumerConfiguration::setMaxTotalReceiverQueueSizeAcrossPartitions)
+        .def("consumer_name", &ConsumerConfiguration::getConsumerName,
+             return_value_policy<copy_const_reference>())
+        .def("consumer_name", &ConsumerConfiguration::setConsumerName)
+        .def("unacked_messages_timeout_ms", &ConsumerConfiguration::getUnAckedMessagesTimeoutMs)
+        .def("unacked_messages_timeout_ms", &ConsumerConfiguration::setUnAckedMessagesTimeoutMs)
+        .def("negative_ack_redelivery_delay_ms", &ConsumerConfiguration::getNegativeAckRedeliveryDelayMs)
+        .def("negative_ack_redelivery_delay_ms", &ConsumerConfiguration::setNegativeAckRedeliveryDelayMs)
+        .def("broker_consumer_stats_cache_time_ms",
+             &ConsumerConfiguration::getBrokerConsumerStatsCacheTimeInMs)
+        .def("broker_consumer_stats_cache_time_ms",
+             &ConsumerConfiguration::setBrokerConsumerStatsCacheTimeInMs)
+        .def("pattern_auto_discovery_period", &ConsumerConfiguration::getPatternAutoDiscoveryPeriod)
+        .def("pattern_auto_discovery_period", &ConsumerConfiguration::setPatternAutoDiscoveryPeriod)
+        .def("read_compacted", &ConsumerConfiguration::isReadCompacted)
+        .def("read_compacted", &ConsumerConfiguration::setReadCompacted)
+        .def("property", &ConsumerConfiguration::setProperty, return_self<>())
+        .def("subscription_initial_position", &ConsumerConfiguration::getSubscriptionInitialPosition)
+        .def("subscription_initial_position", &ConsumerConfiguration::setSubscriptionInitialPosition)
+        .def("crypto_key_reader", &ConsumerConfiguration_setCryptoKeyReader, return_self<>())
+        .def("replicate_subscription_state_enabled",
+             &ConsumerConfiguration::setReplicateSubscriptionStateEnabled)
+        .def("replicate_subscription_state_enabled",
+             &ConsumerConfiguration::isReplicateSubscriptionStateEnabled);
 
     class_<ReaderConfiguration>("ReaderConfiguration")
-            .def("reader_listener", &ReaderConfiguration_setReaderListener, return_self<>())
-            .def("schema", &ReaderConfiguration::getSchema, return_value_policy<copy_const_reference>())
-            .def("schema", &ReaderConfiguration::setSchema, return_self<>())
-            .def("receiver_queue_size", &ReaderConfiguration::getReceiverQueueSize)
-            .def("receiver_queue_size", &ReaderConfiguration::setReceiverQueueSize)
-            .def("reader_name", &ReaderConfiguration::getReaderName, return_value_policy<copy_const_reference>())
-            .def("reader_name", &ReaderConfiguration::setReaderName)
-            .def("subscription_role_prefix", &ReaderConfiguration::getSubscriptionRolePrefix, return_value_policy<copy_const_reference>())
-            .def("subscription_role_prefix", &ReaderConfiguration::setSubscriptionRolePrefix)
-            .def("read_compacted", &ReaderConfiguration::isReadCompacted)
-            .def("read_compacted", &ReaderConfiguration::setReadCompacted)
-            .def("crypto_key_reader", &ReaderConfiguration_setCryptoKeyReader, return_self<>())
-            ;
+        .def("reader_listener", &ReaderConfiguration_setReaderListener, return_self<>())
+        .def("schema", &ReaderConfiguration::getSchema, return_value_policy<copy_const_reference>())
+        .def("schema", &ReaderConfiguration::setSchema, return_self<>())
+        .def("receiver_queue_size", &ReaderConfiguration::getReceiverQueueSize)
+        .def("receiver_queue_size", &ReaderConfiguration::setReceiverQueueSize)
+        .def("reader_name", &ReaderConfiguration::getReaderName, return_value_policy<copy_const_reference>())
+        .def("reader_name", &ReaderConfiguration::setReaderName)
+        .def("subscription_role_prefix", &ReaderConfiguration::getSubscriptionRolePrefix,
+             return_value_policy<copy_const_reference>())
+        .def("subscription_role_prefix", &ReaderConfiguration::setSubscriptionRolePrefix)
+        .def("read_compacted", &ReaderConfiguration::isReadCompacted)
+        .def("read_compacted", &ReaderConfiguration::setReadCompacted)
+        .def("crypto_key_reader", &ReaderConfiguration_setCryptoKeyReader, return_self<>());
 }
diff --git a/pulsar-client-cpp/python/src/consumer.cc b/pulsar-client-cpp/python/src/consumer.cc
index 815282d..28bedad 100644
--- a/pulsar-client-cpp/python/src/consumer.cc
+++ b/pulsar-client-cpp/python/src/consumer.cc
@@ -20,11 +20,10 @@
 
 void Consumer_unsubscribe(Consumer& consumer) {
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = consumer.unsubscribe();
+    Py_BEGIN_ALLOW_THREADS res = consumer.unsubscribe();
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
 Message Consumer_receive(Consumer& consumer) {
@@ -32,11 +31,10 @@ Message Consumer_receive(Consumer& consumer) {
     Result res;
 
     while (true) {
-        Py_BEGIN_ALLOW_THREADS
-        res = consumer.receive(msg);
+        Py_BEGIN_ALLOW_THREADS res = consumer.receive(msg);
         Py_END_ALLOW_THREADS
 
-        if (res != ResultTimeout) {
+            if (res != ResultTimeout) {
             // In case of timeout we keep calling receive() to simulate a
             // blocking call until a message is available, while breaking
             // every once in a while to check the Python signal status
@@ -56,17 +54,14 @@ Message Consumer_receive(Consumer& consumer) {
 Message Consumer_receive_timeout(Consumer& consumer, int timeoutMs) {
     Message msg;
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = consumer.receive(msg, timeoutMs);
+    Py_BEGIN_ALLOW_THREADS res = consumer.receive(msg, timeoutMs);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
     return msg;
 }
 
-void Consumer_acknowledge(Consumer& consumer, const Message& msg) {
-    consumer.acknowledgeAsync(msg, nullptr);
-}
+void Consumer_acknowledge(Consumer& consumer, const Message& msg) { consumer.acknowledgeAsync(msg, nullptr); }
 
 void Consumer_acknowledge_message_id(Consumer& consumer, const MessageId& msgId) {
     consumer.acknowledgeAsync(msgId, nullptr);
@@ -77,7 +72,7 @@ void Consumer_negative_acknowledge(Consumer& consumer, const Message& msg) {
 }
 
 void Consumer_negative_acknowledge_message_id(Consumer& consumer, const MessageId& msgId) {
-     consumer.negativeAcknowledge(msgId);
+    consumer.negativeAcknowledge(msgId);
 }
 
 void Consumer_acknowledge_cumulative(Consumer& consumer, const Message& msg) {
@@ -90,60 +85,52 @@ void Consumer_acknowledge_cumulative_message_id(Consumer& consumer, const Messag
 
 void Consumer_close(Consumer& consumer) {
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = consumer.close();
+    Py_BEGIN_ALLOW_THREADS res = consumer.close();
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
-void Consumer_pauseMessageListener(Consumer& consumer) {
-    CHECK_RESULT(consumer.pauseMessageListener());
-}
+void Consumer_pauseMessageListener(Consumer& consumer) { CHECK_RESULT(consumer.pauseMessageListener()); }
 
-void Consumer_resumeMessageListener(Consumer& consumer) {
-    CHECK_RESULT(consumer.resumeMessageListener());
-}
+void Consumer_resumeMessageListener(Consumer& consumer) { CHECK_RESULT(consumer.resumeMessageListener()); }
 
 void Consumer_seek(Consumer& consumer, const MessageId& msgId) {
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = consumer.seek(msgId);
+    Py_BEGIN_ALLOW_THREADS res = consumer.seek(msgId);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
 void Consumer_seek_timestamp(Consumer& consumer, uint64_t timestamp) {
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = consumer.seek(timestamp);
+    Py_BEGIN_ALLOW_THREADS res = consumer.seek(timestamp);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
 void export_consumer() {
     using namespace boost::python;
 
     class_<Consumer>("Consumer", no_init)
-            .def("topic", &Consumer::getTopic, "return the topic this consumer is subscribed to",
-                 return_value_policy<copy_const_reference>())
-            .def("subscription_name", &Consumer::getSubscriptionName, return_value_policy<copy_const_reference>())
-            .def("unsubscribe", &Consumer_unsubscribe)
-            .def("receive", &Consumer_receive)
-            .def("receive", &Consumer_receive_timeout)
-            .def("acknowledge", &Consumer_acknowledge)
-            .def("acknowledge", &Consumer_acknowledge_message_id)
-            .def("acknowledge_cumulative", &Consumer_acknowledge_cumulative)
-            .def("acknowledge_cumulative", &Consumer_acknowledge_cumulative_message_id)
-            .def("negative_acknowledge", &Consumer_negative_acknowledge)
-            .def("negative_acknowledge", &Consumer_negative_acknowledge_message_id)
-            .def("close", &Consumer_close)
-            .def("pause_message_listener", &Consumer_pauseMessageListener)
-            .def("resume_message_listener", &Consumer_resumeMessageListener)
-            .def("redeliver_unacknowledged_messages", &Consumer::redeliverUnacknowledgedMessages)
-            .def("seek", &Consumer_seek)
-            .def("seek", &Consumer_seek_timestamp)
-            ;
+        .def("topic", &Consumer::getTopic, "return the topic this consumer is subscribed to",
+             return_value_policy<copy_const_reference>())
+        .def("subscription_name", &Consumer::getSubscriptionName, return_value_policy<copy_const_reference>())
+        .def("unsubscribe", &Consumer_unsubscribe)
+        .def("receive", &Consumer_receive)
+        .def("receive", &Consumer_receive_timeout)
+        .def("acknowledge", &Consumer_acknowledge)
+        .def("acknowledge", &Consumer_acknowledge_message_id)
+        .def("acknowledge_cumulative", &Consumer_acknowledge_cumulative)
+        .def("acknowledge_cumulative", &Consumer_acknowledge_cumulative_message_id)
+        .def("negative_acknowledge", &Consumer_negative_acknowledge)
+        .def("negative_acknowledge", &Consumer_negative_acknowledge_message_id)
+        .def("close", &Consumer_close)
+        .def("pause_message_listener", &Consumer_pauseMessageListener)
+        .def("resume_message_listener", &Consumer_resumeMessageListener)
+        .def("redeliver_unacknowledged_messages", &Consumer::redeliverUnacknowledgedMessages)
+        .def("seek", &Consumer_seek)
+        .def("seek", &Consumer_seek_timestamp);
 }
diff --git a/pulsar-client-cpp/python/src/cryptoKeyReader.cc b/pulsar-client-cpp/python/src/cryptoKeyReader.cc
index ccefe6f..2c46b6f 100644
--- a/pulsar-client-cpp/python/src/cryptoKeyReader.cc
+++ b/pulsar-client-cpp/python/src/cryptoKeyReader.cc
@@ -21,7 +21,7 @@
 CryptoKeyReaderWrapper::CryptoKeyReaderWrapper() {}
 
 CryptoKeyReaderWrapper::CryptoKeyReaderWrapper(const std::string& publicKeyPath,
-                                             const std::string& privateKeyPath) {
+                                               const std::string& privateKeyPath) {
     this->cryptoKeyReader = DefaultCryptoKeyReader::create(publicKeyPath, privateKeyPath);
 }
 
diff --git a/pulsar-client-cpp/python/src/enums.cc b/pulsar-client-cpp/python/src/enums.cc
index c23b211..1b21af5 100644
--- a/pulsar-client-cpp/python/src/enums.cc
+++ b/pulsar-client-cpp/python/src/enums.cc
@@ -18,104 +18,96 @@
  */
 #include "utils.h"
 
-
 void export_enums() {
     using namespace boost::python;
 
     enum_<ProducerConfiguration::PartitionsRoutingMode>("PartitionsRoutingMode")
-            .value("UseSinglePartition", ProducerConfiguration::UseSinglePartition)
-            .value("RoundRobinDistribution", ProducerConfiguration::RoundRobinDistribution)
-            .value("CustomPartition", ProducerConfiguration::CustomPartition)
-            ;
+        .value("UseSinglePartition", ProducerConfiguration::UseSinglePartition)
+        .value("RoundRobinDistribution", ProducerConfiguration::RoundRobinDistribution)
+        .value("CustomPartition", ProducerConfiguration::CustomPartition);
 
     enum_<CompressionType>("CompressionType")
-            .value("NONE", CompressionNone) // Don't use 'None' since it's a keyword in py3
-            .value("LZ4", CompressionLZ4)
-            .value("ZLib", CompressionZLib)
-            .value("ZSTD", CompressionZSTD)
-            .value("SNAPPY", CompressionSNAPPY)
-            ;
+        .value("NONE", CompressionNone)  // Don't use 'None' since it's a keyword in py3
+        .value("LZ4", CompressionLZ4)
+        .value("ZLib", CompressionZLib)
+        .value("ZSTD", CompressionZSTD)
+        .value("SNAPPY", CompressionSNAPPY);
 
     enum_<ConsumerType>("ConsumerType")
-            .value("Exclusive", ConsumerExclusive)
-            .value("Shared", ConsumerShared)
-            .value("Failover", ConsumerFailover)
-            .value("KeyShared", ConsumerKeyShared)
-            ;
+        .value("Exclusive", ConsumerExclusive)
+        .value("Shared", ConsumerShared)
+        .value("Failover", ConsumerFailover)
+        .value("KeyShared", ConsumerKeyShared);
 
-    enum_<Result >("Result", "Collection of return codes")
-            .value("Ok", ResultOk)
-            .value("UnknownError", ResultUnknownError)
-            .value("InvalidConfiguration", ResultInvalidConfiguration)
-            .value("Timeout", ResultTimeout)
-            .value("LookupError", ResultLookupError)
-            .value("ConnectError", ResultConnectError)
-            .value("ReadError", ResultReadError)
-            .value("AuthenticationError", ResultAuthenticationError)
-            .value("AuthorizationError", ResultAuthorizationError)
-            .value("ErrorGettingAuthenticationData", ResultErrorGettingAuthenticationData)
-            .value("BrokerMetadataError", ResultBrokerMetadataError)
-            .value("BrokerPersistenceError", ResultBrokerPersistenceError)
-            .value("ChecksumError", ResultChecksumError)
-            .value("ConsumerBusy", ResultConsumerBusy)
-            .value("NotConnected", ResultNotConnected)
-            .value("AlreadyClosed", ResultAlreadyClosed)
-            .value("InvalidMessage", ResultInvalidMessage)
-            .value("ConsumerNotInitialized", ResultConsumerNotInitialized)
-            .value("ProducerNotInitialized", ResultProducerNotInitialized)
-            .value("ProducerBusy", ResultProducerBusy)
-            .value("TooManyLookupRequestException", ResultTooManyLookupRequestException)
-            .value("InvalidTopicName", ResultInvalidTopicName)
-            .value("InvalidUrl", ResultInvalidUrl)
-            .value("ServiceUnitNotReady", ResultServiceUnitNotReady)
-            .value("OperationNotSupported", ResultOperationNotSupported)
-            .value("ProducerBlockedQuotaExceededError", ResultProducerBlockedQuotaExceededError)
-            .value("ProducerBlockedQuotaExceededException", ResultProducerBlockedQuotaExceededException)
-            .value("ProducerQueueIsFull", ResultProducerQueueIsFull)
-            .value("MessageTooBig", ResultMessageTooBig)
-            .value("TopicNotFound", ResultTopicNotFound)
-            .value("SubscriptionNotFound", ResultSubscriptionNotFound)
-            .value("ConsumerNotFound", ResultConsumerNotFound)
-            .value("UnsupportedVersionError", ResultUnsupportedVersionError)
-            .value("TopicTerminated", ResultTopicTerminated)
-            .value("CryptoError", ResultCryptoError)
-            .value("IncompatibleSchema", ResultIncompatibleSchema)
-            .value("ConsumerAssignError", ResultConsumerAssignError)
-            .value("CumulativeAcknowledgementNotAllowedError", ResultCumulativeAcknowledgementNotAllowedError)
-            .value("TransactionCoordinatorNotFoundError", ResultTransactionCoordinatorNotFoundError)
-            .value("InvalidTxnStatusError", ResultInvalidTxnStatusError)
-            .value("NotAllowedError", ResultNotAllowedError)
-            .value("TransactionConflict", ResultTransactionConflict)
-            .value("TransactionNotFound", ResultTransactionNotFound)
-            .value("ProducerFenced", ResultProducerFenced)
-            .value("MemoryBufferIsFull", ResultMemoryBufferIsFull)
-            ;
+    enum_<Result>("Result", "Collection of return codes")
+        .value("Ok", ResultOk)
+        .value("UnknownError", ResultUnknownError)
+        .value("InvalidConfiguration", ResultInvalidConfiguration)
+        .value("Timeout", ResultTimeout)
+        .value("LookupError", ResultLookupError)
+        .value("ConnectError", ResultConnectError)
+        .value("ReadError", ResultReadError)
+        .value("AuthenticationError", ResultAuthenticationError)
+        .value("AuthorizationError", ResultAuthorizationError)
+        .value("ErrorGettingAuthenticationData", ResultErrorGettingAuthenticationData)
+        .value("BrokerMetadataError", ResultBrokerMetadataError)
+        .value("BrokerPersistenceError", ResultBrokerPersistenceError)
+        .value("ChecksumError", ResultChecksumError)
+        .value("ConsumerBusy", ResultConsumerBusy)
+        .value("NotConnected", ResultNotConnected)
+        .value("AlreadyClosed", ResultAlreadyClosed)
+        .value("InvalidMessage", ResultInvalidMessage)
+        .value("ConsumerNotInitialized", ResultConsumerNotInitialized)
+        .value("ProducerNotInitialized", ResultProducerNotInitialized)
+        .value("ProducerBusy", ResultProducerBusy)
+        .value("TooManyLookupRequestException", ResultTooManyLookupRequestException)
+        .value("InvalidTopicName", ResultInvalidTopicName)
+        .value("InvalidUrl", ResultInvalidUrl)
+        .value("ServiceUnitNotReady", ResultServiceUnitNotReady)
+        .value("OperationNotSupported", ResultOperationNotSupported)
+        .value("ProducerBlockedQuotaExceededError", ResultProducerBlockedQuotaExceededError)
+        .value("ProducerBlockedQuotaExceededException", ResultProducerBlockedQuotaExceededException)
+        .value("ProducerQueueIsFull", ResultProducerQueueIsFull)
+        .value("MessageTooBig", ResultMessageTooBig)
+        .value("TopicNotFound", ResultTopicNotFound)
+        .value("SubscriptionNotFound", ResultSubscriptionNotFound)
+        .value("ConsumerNotFound", ResultConsumerNotFound)
+        .value("UnsupportedVersionError", ResultUnsupportedVersionError)
+        .value("TopicTerminated", ResultTopicTerminated)
+        .value("CryptoError", ResultCryptoError)
+        .value("IncompatibleSchema", ResultIncompatibleSchema)
+        .value("ConsumerAssignError", ResultConsumerAssignError)
+        .value("CumulativeAcknowledgementNotAllowedError", ResultCumulativeAcknowledgementNotAllowedError)
+        .value("TransactionCoordinatorNotFoundError", ResultTransactionCoordinatorNotFoundError)
+        .value("InvalidTxnStatusError", ResultInvalidTxnStatusError)
+        .value("NotAllowedError", ResultNotAllowedError)
+        .value("TransactionConflict", ResultTransactionConflict)
+        .value("TransactionNotFound", ResultTransactionNotFound)
+        .value("ProducerFenced", ResultProducerFenced)
+        .value("MemoryBufferIsFull", ResultMemoryBufferIsFull);
 
     enum_<SchemaType>("SchemaType", "Supported schema types")
-            .value("NONE", pulsar::NONE)
-            .value("STRING", pulsar::STRING)
-            .value("INT8", pulsar::INT8)
-            .value("INT16", pulsar::INT16)
-            .value("INT32", pulsar::INT32)
-            .value("INT64", pulsar::INT64)
-            .value("FLOAT", pulsar::FLOAT)
-            .value("DOUBLE", pulsar::DOUBLE)
-            .value("BYTES", pulsar::BYTES)
-            .value("JSON", pulsar::JSON)
-            .value("PROTOBUF", pulsar::PROTOBUF)
-            .value("AVRO", pulsar::AVRO)
-            .value("AUTO_CONSUME", pulsar::AUTO_CONSUME)
-            .value("AUTO_PUBLISH", pulsar::AUTO_PUBLISH)
-            .value("KEY_VALUE", pulsar::KEY_VALUE)
-            ;
+        .value("NONE", pulsar::NONE)
+        .value("STRING", pulsar::STRING)
+        .value("INT8", pulsar::INT8)
+        .value("INT16", pulsar::INT16)
+        .value("INT32", pulsar::INT32)
+        .value("INT64", pulsar::INT64)
+        .value("FLOAT", pulsar::FLOAT)
+        .value("DOUBLE", pulsar::DOUBLE)
+        .value("BYTES", pulsar::BYTES)
+        .value("JSON", pulsar::JSON)
+        .value("PROTOBUF", pulsar::PROTOBUF)
+        .value("AVRO", pulsar::AVRO)
+        .value("AUTO_CONSUME", pulsar::AUTO_CONSUME)
+        .value("AUTO_PUBLISH", pulsar::AUTO_PUBLISH)
+        .value("KEY_VALUE", pulsar::KEY_VALUE);
 
     enum_<InitialPosition>("InitialPosition", "Supported initial position")
-            .value("Latest", InitialPositionLatest)
-            .value("Earliest", InitialPositionEarliest)
-            ;
+        .value("Latest", InitialPositionLatest)
+        .value("Earliest", InitialPositionEarliest);
 
     enum_<ProducerConfiguration::BatchingType>("BatchingType", "Supported batching types")
-            .value("Default", ProducerConfiguration::DefaultBatching)
-            .value("KeyBased", ProducerConfiguration::KeyBasedBatching)
-            ;
+        .value("Default", ProducerConfiguration::DefaultBatching)
+        .value("KeyBased", ProducerConfiguration::KeyBasedBatching);
 }
diff --git a/pulsar-client-cpp/python/src/exceptions.cc b/pulsar-client-cpp/python/src/exceptions.cc
index c39b52d..25b3bd0 100644
--- a/pulsar-client-cpp/python/src/exceptions.cc
+++ b/pulsar-client-cpp/python/src/exceptions.cc
@@ -29,16 +29,13 @@ PyObject* createExceptionClass(const char* name, PyObject* baseTypeObj = PyExc_E
     std::string fullName = "_pulsar.";
     fullName += name;
 
-    PyObject* typeObj = PyErr_NewException(const_cast<char*>(fullName.c_str()),
-                                           baseTypeObj, nullptr);
+    PyObject* typeObj = PyErr_NewException(const_cast<char*>(fullName.c_str()), baseTypeObj, nullptr);
     if (!typeObj) throw_error_already_set();
     scope().attr(name) = handle<>(borrowed(typeObj));
     return typeObj;
 }
 
-PyObject* get_exception_class(Result result) {
-    return exceptions[result];
-}
+PyObject* get_exception_class(Result result) { return exceptions[result]; }
 
 void export_exceptions() {
     using namespace boost::python;
@@ -46,44 +43,58 @@ void export_exceptions() {
     basePulsarException = createExceptionClass("PulsarException");
 
     exceptions[ResultUnknownError] = createExceptionClass("UnknownError", basePulsarException);
-    exceptions[ResultInvalidConfiguration] = createExceptionClass("InvalidConfiguration", basePulsarException);
+    exceptions[ResultInvalidConfiguration] =
+        createExceptionClass("InvalidConfiguration", basePulsarException);
     exceptions[ResultTimeout] = createExceptionClass("Timeout", basePulsarException);
     exceptions[ResultLookupError] = createExceptionClass("LookupError", basePulsarException);
     exceptions[ResultConnectError] = createExceptionClass("ConnectError", basePulsarException);
     exceptions[ResultReadError] = createExceptionClass("ReadError", basePulsarException);
     exceptions[ResultAuthenticationError] = createExceptionClass("AuthenticationError", basePulsarException);
     exceptions[ResultAuthorizationError] = createExceptionClass("AuthorizationError", basePulsarException);
-    exceptions[ResultErrorGettingAuthenticationData] = createExceptionClass("ErrorGettingAuthenticationData", basePulsarException);
+    exceptions[ResultErrorGettingAuthenticationData] =
+        createExceptionClass("ErrorGettingAuthenticationData", basePulsarException);
     exceptions[ResultBrokerMetadataError] = createExceptionClass("BrokerMetadataError", basePulsarException);
-    exceptions[ResultBrokerPersistenceError] = createExceptionClass("BrokerPersistenceError", basePulsarException);
+    exceptions[ResultBrokerPersistenceError] =
+        createExceptionClass("BrokerPersistenceError", basePulsarException);
     exceptions[ResultChecksumError] = createExceptionClass("ChecksumError", basePulsarException);
     exceptions[ResultConsumerBusy] = createExceptionClass("ConsumerBusy", basePulsarException);
     exceptions[ResultNotConnected] = createExceptionClass("NotConnected", basePulsarException);
     exceptions[ResultAlreadyClosed] = createExceptionClass("AlreadyClosed", basePulsarException);
     exceptions[ResultInvalidMessage] = createExceptionClass("InvalidMessage", basePulsarException);
-    exceptions[ResultConsumerNotInitialized] = createExceptionClass("ConsumerNotInitialized", basePulsarException);
-    exceptions[ResultProducerNotInitialized] = createExceptionClass("ProducerNotInitialized", basePulsarException);
+    exceptions[ResultConsumerNotInitialized] =
+        createExceptionClass("ConsumerNotInitialized", basePulsarException);
+    exceptions[ResultProducerNotInitialized] =
+        createExceptionClass("ProducerNotInitialized", basePulsarException);
     exceptions[ResultProducerBusy] = createExceptionClass("ProducerBusy", basePulsarException);
-    exceptions[ResultTooManyLookupRequestException] = createExceptionClass("TooManyLookupRequestException", basePulsarException);
+    exceptions[ResultTooManyLookupRequestException] =
+        createExceptionClass("TooManyLookupRequestException", basePulsarException);
     exceptions[ResultInvalidTopicName] = createExceptionClass("InvalidTopicName", basePulsarException);
     exceptions[ResultInvalidUrl] = createExceptionClass("InvalidUrl", basePulsarException);
     exceptions[ResultServiceUnitNotReady] = createExceptionClass("ServiceUnitNotReady", basePulsarException);
-    exceptions[ResultOperationNotSupported] = createExceptionClass("OperationNotSupported", basePulsarException);
-    exceptions[ResultProducerBlockedQuotaExceededError] = createExceptionClass("ProducerBlockedQuotaExceededError", basePulsarException);
-    exceptions[ResultProducerBlockedQuotaExceededException] = createExceptionClass("ProducerBlockedQuotaExceededException", basePulsarException);
+    exceptions[ResultOperationNotSupported] =
+        createExceptionClass("OperationNotSupported", basePulsarException);
+    exceptions[ResultProducerBlockedQuotaExceededError] =
+        createExceptionClass("ProducerBlockedQuotaExceededError", basePulsarException);
+    exceptions[ResultProducerBlockedQuotaExceededException] =
+        createExceptionClass("ProducerBlockedQuotaExceededException", basePulsarException);
     exceptions[ResultProducerQueueIsFull] = createExceptionClass("ProducerQueueIsFull", basePulsarException);
     exceptions[ResultMessageTooBig] = createExceptionClass("MessageTooBig", basePulsarException);
     exceptions[ResultTopicNotFound] = createExceptionClass("TopicNotFound", basePulsarException);
-    exceptions[ResultSubscriptionNotFound] = createExceptionClass("SubscriptionNotFound", basePulsarException);
+    exceptions[ResultSubscriptionNotFound] =
+        createExceptionClass("SubscriptionNotFound", basePulsarException);
     exceptions[ResultConsumerNotFound] = createExceptionClass("ConsumerNotFound", basePulsarException);
-    exceptions[ResultUnsupportedVersionError] = createExceptionClass("UnsupportedVersionError", basePulsarException);
+    exceptions[ResultUnsupportedVersionError] =
+        createExceptionClass("UnsupportedVersionError", basePulsarException);
     exceptions[ResultTopicTerminated] = createExceptionClass("TopicTerminated", basePulsarException);
     exceptions[ResultCryptoError] = createExceptionClass("CryptoError", basePulsarException);
     exceptions[ResultIncompatibleSchema] = createExceptionClass("IncompatibleSchema", basePulsarException);
     exceptions[ResultConsumerAssignError] = createExceptionClass("ConsumerAssignError", basePulsarException);
-    exceptions[ResultCumulativeAcknowledgementNotAllowedError] = createExceptionClass("CumulativeAcknowledgementNotAllowedError", basePulsarException);
-    exceptions[ResultTransactionCoordinatorNotFoundError] = createExceptionClass("TransactionCoordinatorNotFoundError", basePulsarException);
-    exceptions[ResultInvalidTxnStatusError] = createExceptionClass("InvalidTxnStatusError", basePulsarException);
+    exceptions[ResultCumulativeAcknowledgementNotAllowedError] =
+        createExceptionClass("CumulativeAcknowledgementNotAllowedError", basePulsarException);
+    exceptions[ResultTransactionCoordinatorNotFoundError] =
+        createExceptionClass("TransactionCoordinatorNotFoundError", basePulsarException);
+    exceptions[ResultInvalidTxnStatusError] =
+        createExceptionClass("InvalidTxnStatusError", basePulsarException);
     exceptions[ResultNotAllowedError] = createExceptionClass("NotAllowedError", basePulsarException);
     exceptions[ResultTransactionConflict] = createExceptionClass("TransactionConflict", basePulsarException);
     exceptions[ResultTransactionNotFound] = createExceptionClass("TransactionNotFound", basePulsarException);
diff --git a/pulsar-client-cpp/python/src/message.cc b/pulsar-client-cpp/python/src/message.cc
index 8532966..b93380b 100644
--- a/pulsar-client-cpp/python/src/message.cc
+++ b/pulsar-client-cpp/python/src/message.cc
@@ -28,34 +28,23 @@ std::string MessageId_str(const MessageId& msgId) {
     return ss.str();
 }
 
-bool MessageId_eq(const MessageId& a, const MessageId& b) {
-    return a == b;
-}
+bool MessageId_eq(const MessageId& a, const MessageId& b) { return a == b; }
 
-bool MessageId_ne(const MessageId& a, const MessageId& b) {
-    return a != b;
-}
+bool MessageId_ne(const MessageId& a, const MessageId& b) { return a != b; }
 
-bool MessageId_lt(const MessageId& a, const MessageId& b) {
-    return a < b;
-}
+bool MessageId_lt(const MessageId& a, const MessageId& b) { return a < b; }
 
-bool MessageId_le(const MessageId& a, const MessageId& b) {
-    return a <= b;
-}
+bool MessageId_le(const MessageId& a, const MessageId& b) { return a <= b; }
 
-bool MessageId_gt(const MessageId& a, const MessageId& b) {
-    return a > b;
-}
+bool MessageId_gt(const MessageId& a, const MessageId& b) { return a > b; }
 
-bool MessageId_ge(const MessageId& a, const MessageId& b) {
-    return a >= b;
-}
+bool MessageId_ge(const MessageId& a, const MessageId& b) { return a >= b; }
 
 boost::python::object MessageId_serialize(const MessageId& msgId) {
     std::string serialized;
     msgId.serialize(serialized);
-    return boost::python::object(boost::python::handle<>(PyBytes_FromStringAndSize(serialized.c_str(), serialized.length())));
+    return boost::python::object(
+        boost::python::handle<>(PyBytes_FromStringAndSize(serialized.c_str(), serialized.length())));
 }
 
 std::string Message_str(const Message& msg) {
@@ -65,7 +54,8 @@ std::string Message_str(const Message& msg) {
 }
 
 boost::python::object Message_data(const Message& msg) {
-    return boost::python::object(boost::python::handle<>(PyBytes_FromStringAndSize((const char*)msg.getData(), msg.getLength())));
+    return boost::python::object(
+        boost::python::handle<>(PyBytes_FromStringAndSize((const char*)msg.getData(), msg.getLength())));
 }
 
 boost::python::object Message_properties(const Message& msg) {
@@ -88,9 +78,7 @@ std::string schema_version_str(const Message& msg) {
     return ss.str();
 }
 
-const MessageId& Message_getMessageId(const Message& msg) {
-    return msg.getMessageId();
-}
+const MessageId& Message_getMessageId(const Message& msg) { return msg.getMessageId(); }
 
 void deliverAfter(MessageBuilder* const builder, PyObject* obj_delta) {
     PyDateTime_Delta const* pydelta = reinterpret_cast<PyDateTime_Delta*>(obj_delta);
@@ -102,12 +90,9 @@ void deliverAfter(MessageBuilder* const builder, PyObject* obj_delta) {
     }
 
     // Create chrono duration object
-    std::chrono::milliseconds
-        duration = std::chrono::duration_cast<std::chrono::milliseconds>(
-                std::chrono::hours(24)*days
-                + std::chrono::seconds(pydelta->seconds)
-                + std::chrono::microseconds(pydelta->microseconds)
-                );
+    std::chrono::milliseconds duration = std::chrono::duration_cast<std::chrono::milliseconds>(
+        std::chrono::hours(24) * days + std::chrono::seconds(pydelta->seconds) +
+        std::chrono::microseconds(pydelta->microseconds));
 
     if (is_negative) {
         duration = duration * -1;
@@ -121,70 +106,66 @@ void export_message() {
 
     PyDateTime_IMPORT;
 
-    MessageBuilder& (MessageBuilder::*MessageBuilderSetContentString)(const std::string&) = &MessageBuilder::setContent;
+    MessageBuilder& (MessageBuilder::*MessageBuilderSetContentString)(const std::string&) =
+        &MessageBuilder::setContent;
 
     class_<MessageBuilder, boost::noncopyable>("MessageBuilder")
-            .def("content", MessageBuilderSetContentString, return_self<>())
-            .def("property", &MessageBuilder::setProperty, return_self<>())
-            .def("properties", &MessageBuilder::setProperties, return_self<>())
-            .def("sequence_id", &MessageBuilder::setSequenceId, return_self<>())
-            .def("deliver_after", &deliverAfter, return_self<>())
-            .def("deliver_at", &MessageBuilder::setDeliverAt, return_self<>())
-            .def("partition_key", &MessageBuilder::setPartitionKey, return_self<>())
-            .def("event_timestamp", &MessageBuilder::setEventTimestamp, return_self<>())
-            .def("replication_clusters", &MessageBuilder::setReplicationClusters, return_self<>())
-            .def("disable_replication", &MessageBuilder::disableReplication, return_self<>())
-            .def("build", &MessageBuilder::build)
-            ;
-
-    class_<Message::StringMap>("MessageStringMap")
-            .def(map_indexing_suite<Message::StringMap>())
-            ;
+        .def("content", MessageBuilderSetContentString, return_self<>())
+        .def("property", &MessageBuilder::setProperty, return_self<>())
+        .def("properties", &MessageBuilder::setProperties, return_self<>())
+        .def("sequence_id", &MessageBuilder::setSequenceId, return_self<>())
+        .def("deliver_after", &deliverAfter, return_self<>())
+        .def("deliver_at", &MessageBuilder::setDeliverAt, return_self<>())
+        .def("partition_key", &MessageBuilder::setPartitionKey, return_self<>())
+        .def("event_timestamp", &MessageBuilder::setEventTimestamp, return_self<>())
+        .def("replication_clusters", &MessageBuilder::setReplicationClusters, return_self<>())
+        .def("disable_replication", &MessageBuilder::disableReplication, return_self<>())
+        .def("build", &MessageBuilder::build);
+
+    class_<Message::StringMap>("MessageStringMap").def(map_indexing_suite<Message::StringMap>());
 
     static const MessageId& _MessageId_earliest = MessageId::earliest();
     static const MessageId& _MessageId_latest = MessageId::latest();
 
     class_<MessageId>("MessageId")
-            .def(init<int32_t, int64_t, int64_t, int32_t>())
-            .def("__str__", &MessageId_str)
-            .def("__eq__", &MessageId_eq)
-            .def("__ne__", &MessageId_ne)
-            .def("__le__", &MessageId_le)
-            .def("__lt__", &MessageId_lt)
-            .def("__ge__", &MessageId_ge)
-            .def("__gt__", &MessageId_gt)
-            .def("ledger_id", &MessageId::ledgerId)
-            .def("entry_id", &MessageId::entryId)
-            .def("batch_index", &MessageId::batchIndex)
-            .def("partition", &MessageId::partition)
-            .add_static_property("earliest", make_getter(&_MessageId_earliest))
-            .add_static_property("latest", make_getter(&_MessageId_latest))
-            .def("serialize", &MessageId_serialize)
-            .def("deserialize", &MessageId::deserialize).staticmethod("deserialize")
-            ;
+        .def(init<int32_t, int64_t, int64_t, int32_t>())
+        .def("__str__", &MessageId_str)
+        .def("__eq__", &MessageId_eq)
+        .def("__ne__", &MessageId_ne)
+        .def("__le__", &MessageId_le)
+        .def("__lt__", &MessageId_lt)
+        .def("__ge__", &MessageId_ge)
+        .def("__gt__", &MessageId_gt)
+        .def("ledger_id", &MessageId::ledgerId)
+        .def("entry_id", &MessageId::entryId)
+        .def("batch_index", &MessageId::batchIndex)
+        .def("partition", &MessageId::partition)
+        .add_static_property("earliest", make_getter(&_MessageId_earliest))
+        .add_static_property("latest", make_getter(&_MessageId_latest))
+        .def("serialize", &MessageId_serialize)
+        .def("deserialize", &MessageId::deserialize)
+        .staticmethod("deserialize");
 
     class_<Message>("Message")
-            .def("properties", &Message_properties)
-            .def("data", &Message_data)
-            .def("length", &Message::getLength)
-            .def("partition_key", &Message::getPartitionKey, return_value_policy<copy_const_reference>())
-            .def("publish_timestamp", &Message::getPublishTimestamp)
-            .def("event_timestamp", &Message::getEventTimestamp)
-            .def("message_id", &Message_getMessageId, return_value_policy<copy_const_reference>())
-            .def("__str__", &Message_str)
-            .def("topic_name", &Topic_name_str)
-            .def("redelivery_count", &Message::getRedeliveryCount)
-            .def("schema_version", &schema_version_str)
-            ;
-
-    MessageBatch& (MessageBatch::*MessageBatchParseFromString)(const std::string& payload, uint32_t batchSize) = &MessageBatch::parseFrom;
+        .def("properties", &Message_properties)
+        .def("data", &Message_data)
+        .def("length", &Message::getLength)
+        .def("partition_key", &Message::getPartitionKey, return_value_policy<copy_const_reference>())
+        .def("publish_timestamp", &Message::getPublishTimestamp)
+        .def("event_timestamp", &Message::getEventTimestamp)
+        .def("message_id", &Message_getMessageId, return_value_policy<copy_const_reference>())
+        .def("__str__", &Message_str)
+        .def("topic_name", &Topic_name_str)
+        .def("redelivery_count", &Message::getRedeliveryCount)
+        .def("schema_version", &schema_version_str);
+
+    MessageBatch& (MessageBatch::*MessageBatchParseFromString)(const std::string& payload,
+                                                               uint32_t batchSize) = &MessageBatch::parseFrom;
 
     class_<MessageBatch>("MessageBatch")
-            .def("with_message_id", &MessageBatch::withMessageId, return_self<>())
-            .def("parse_from", MessageBatchParseFromString, return_self<>())
-            .def("messages", &MessageBatch::messages, return_value_policy<copy_const_reference>())
-            ;
+        .def("with_message_id", &MessageBatch::withMessageId, return_self<>())
+        .def("parse_from", MessageBatchParseFromString, return_self<>())
+        .def("messages", &MessageBatch::messages, return_value_policy<copy_const_reference>());
 
-    class_<std::vector<Message> >("Messages")
-        .def(vector_indexing_suite<std::vector<Message> >() );
+    class_<std::vector<Message> >("Messages").def(vector_indexing_suite<std::vector<Message> >());
 }
diff --git a/pulsar-client-cpp/python/src/producer.cc b/pulsar-client-cpp/python/src/producer.cc
index 343650f..345639e 100644
--- a/pulsar-client-cpp/python/src/producer.cc
+++ b/pulsar-client-cpp/python/src/producer.cc
@@ -25,11 +25,10 @@ extern boost::python::object MessageId_serialize(const MessageId& msgId);
 boost::python::object Producer_send(Producer& producer, const Message& message) {
     Result res;
     MessageId messageId;
-    Py_BEGIN_ALLOW_THREADS
-    res = producer.send(message, messageId);
+    Py_BEGIN_ALLOW_THREADS res = producer.send(message, messageId);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
     return MessageId_serialize(messageId);
 }
 
@@ -54,57 +53,55 @@ void Producer_sendAsync(Producer& producer, const Message& message, py::object c
     PyObject* pyCallback = callback.ptr();
     Py_XINCREF(pyCallback);
 
-    Py_BEGIN_ALLOW_THREADS
-    producer.sendAsync(message, std::bind(Producer_sendAsyncCallback, pyCallback,
-            std::placeholders::_1, std::placeholders::_2));
+    Py_BEGIN_ALLOW_THREADS producer.sendAsync(
+        message,
+        std::bind(Producer_sendAsyncCallback, pyCallback, std::placeholders::_1, std::placeholders::_2));
     Py_END_ALLOW_THREADS
 }
 
 void Producer_flush(Producer& producer) {
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = producer.flush();
+    Py_BEGIN_ALLOW_THREADS res = producer.flush();
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
 void Producer_close(Producer& producer) {
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = producer.close();
+    Py_BEGIN_ALLOW_THREADS res = producer.close();
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
 void export_producer() {
     using namespace boost::python;
 
     class_<Producer>("Producer", no_init)
-            .def("topic", &Producer::getTopic, "return the topic to which producer is publishing to",
-                 return_value_policy<copy_const_reference>())
-            .def("producer_name", &Producer::getProducerName,
-                 "return the producer name which could have been assigned by the system or specified by the client",
-                 return_value_policy<copy_const_reference>())
-            .def("last_sequence_id", &Producer::getLastSequenceId)
-            .def("send", &Producer_send,
-                 "Publish a message on the topic associated with this Producer.\n"
-                         "\n"
-                         "This method will block until the message will be accepted and persisted\n"
-                         "by the broker. In case of errors, the client library will try to\n"
-                         "automatically recover and use a different broker.\n"
-                         "\n"
-                         "If it wasn't possible to successfully publish the message within the sendTimeout,\n"
-                         "an error will be returned.\n"
-                         "\n"
-                         "This method is equivalent to asyncSend() and wait until the callback is triggered.\n"
-                         "\n"
-                         "@param msg message to publish\n")
-            .def("send_async", &Producer_sendAsync)
-            .def("flush", &Producer_flush,
-                 "Flush all the messages buffered in the client and wait until all messages have been\n"
-                         "successfully persisted\n")
-            .def("close", &Producer_close)
-            ;
+        .def("topic", &Producer::getTopic, "return the topic to which producer is publishing to",
+             return_value_policy<copy_const_reference>())
+        .def("producer_name", &Producer::getProducerName,
+             "return the producer name which could have been assigned by the system or specified by the "
+             "client",
+             return_value_policy<copy_const_reference>())
+        .def("last_sequence_id", &Producer::getLastSequenceId)
+        .def("send", &Producer_send,
+             "Publish a message on the topic associated with this Producer.\n"
+             "\n"
+             "This method will block until the message will be accepted and persisted\n"
+             "by the broker. In case of errors, the client library will try to\n"
+             "automatically recover and use a different broker.\n"
+             "\n"
+             "If it wasn't possible to successfully publish the message within the sendTimeout,\n"
+             "an error will be returned.\n"
+             "\n"
+             "This method is equivalent to asyncSend() and wait until the callback is triggered.\n"
+             "\n"
+             "@param msg message to publish\n")
+        .def("send_async", &Producer_sendAsync)
+        .def("flush", &Producer_flush,
+             "Flush all the messages buffered in the client and wait until all messages have been\n"
+             "successfully persisted\n")
+        .def("close", &Producer_close);
 }
diff --git a/pulsar-client-cpp/python/src/pulsar.cc b/pulsar-client-cpp/python/src/pulsar.cc
index a46ce53..e591b73 100644
--- a/pulsar-client-cpp/python/src/pulsar.cc
+++ b/pulsar-client-cpp/python/src/pulsar.cc
@@ -32,7 +32,6 @@ void export_exceptions();
 
 PyObject* get_exception_class(Result result);
 
-
 static void translateException(const PulsarException& ex) {
     std::string err = "Pulsar error: ";
     err += strResult(ex._result);
@@ -40,8 +39,7 @@ static void translateException(const PulsarException& ex) {
     PyErr_SetString(get_exception_class(ex._result), err.c_str());
 }
 
-BOOST_PYTHON_MODULE(_pulsar)
-{
+BOOST_PYTHON_MODULE(_pulsar) {
     py::register_exception_translator<PulsarException>(translateException);
 
     // Initialize thread support so that we can grab the GIL mutex
diff --git a/pulsar-client-cpp/python/src/reader.cc b/pulsar-client-cpp/python/src/reader.cc
index fec65da..668fb94 100644
--- a/pulsar-client-cpp/python/src/reader.cc
+++ b/pulsar-client-cpp/python/src/reader.cc
@@ -24,12 +24,12 @@ Message Reader_readNext(Reader& reader) {
 
     while (true) {
         Py_BEGIN_ALLOW_THREADS
-        // Use 100ms timeout to periodically check whether the
-        // interpreter was interrupted
-        res = reader.readNext(msg, 100);
+            // Use 100ms timeout to periodically check whether the
+            // interpreter was interrupted
+            res = reader.readNext(msg, 100);
         Py_END_ALLOW_THREADS
 
-        if (res != ResultTimeout) {
+            if (res != ResultTimeout) {
             // In case of timeout we keep calling receive() to simulate a
             // blocking call until a message is available, while breaking
             // every once in a while to check the Python signal status
@@ -49,62 +49,56 @@ Message Reader_readNext(Reader& reader) {
 Message Reader_readNextTimeout(Reader& reader, int timeoutMs) {
     Message msg;
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = reader.readNext(msg, timeoutMs);
+    Py_BEGIN_ALLOW_THREADS res = reader.readNext(msg, timeoutMs);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
     return msg;
 }
 
 bool Reader_hasMessageAvailable(Reader& reader) {
     bool available = false;
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-        res = reader.hasMessageAvailable(available);
+    Py_BEGIN_ALLOW_THREADS res = reader.hasMessageAvailable(available);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
     return available;
 }
 
 void Reader_close(Reader& reader) {
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = reader.close();
+    Py_BEGIN_ALLOW_THREADS res = reader.close();
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
 void Reader_seek(Reader& reader, const MessageId& msgId) {
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = reader.seek(msgId);
+    Py_BEGIN_ALLOW_THREADS res = reader.seek(msgId);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
 void Reader_seek_timestamp(Reader& reader, uint64_t timestamp) {
     Result res;
-    Py_BEGIN_ALLOW_THREADS
-    res = reader.seek(timestamp);
+    Py_BEGIN_ALLOW_THREADS res = reader.seek(timestamp);
     Py_END_ALLOW_THREADS
 
-    CHECK_RESULT(res);
+        CHECK_RESULT(res);
 }
 
 void export_reader() {
     using namespace boost::python;
 
     class_<Reader>("Reader", no_init)
-            .def("topic", &Reader::getTopic, return_value_policy<copy_const_reference>())
-            .def("read_next", &Reader_readNext)
-            .def("read_next", &Reader_readNextTimeout)
-            .def("has_message_available", &Reader_hasMessageAvailable)
-            .def("close", &Reader_close)
-            .def("seek", &Reader_seek)
-            .def("seek", &Reader_seek_timestamp)
-            ;
+        .def("topic", &Reader::getTopic, return_value_policy<copy_const_reference>())
+        .def("read_next", &Reader_readNext)
+        .def("read_next", &Reader_readNextTimeout)
+        .def("has_message_available", &Reader_hasMessageAvailable)
+        .def("close", &Reader_close)
+        .def("seek", &Reader_seek)
+        .def("seek", &Reader_seek_timestamp);
 }
diff --git a/pulsar-client-cpp/python/src/schema.cc b/pulsar-client-cpp/python/src/schema.cc
index 397ec65..cdfcda6 100644
--- a/pulsar-client-cpp/python/src/schema.cc
+++ b/pulsar-client-cpp/python/src/schema.cc
@@ -21,10 +21,8 @@
 void export_schema() {
     using namespace boost::python;
 
-    class_<SchemaInfo>("SchemaInfo",
-            init<SchemaType, const std::string& , const std::string&>())
-            .def("schema_type", &SchemaInfo::getSchemaType)
-            .def("name", &SchemaInfo::getName, return_value_policy<copy_const_reference>())
-            .def("schema", &SchemaInfo::getSchema, return_value_policy<copy_const_reference>())
-            ;
+    class_<SchemaInfo>("SchemaInfo", init<SchemaType, const std::string&, const std::string&>())
+        .def("schema_type", &SchemaInfo::getSchemaType)
+        .def("name", &SchemaInfo::getName, return_value_policy<copy_const_reference>())
+        .def("schema", &SchemaInfo::getSchema, return_value_policy<copy_const_reference>());
 }
diff --git a/pulsar-client-cpp/python/src/utils.h b/pulsar-client-cpp/python/src/utils.h
index 457d1f8..5be4473 100644
--- a/pulsar-client-cpp/python/src/utils.h
+++ b/pulsar-client-cpp/python/src/utils.h
@@ -27,8 +27,7 @@ namespace py = boost::python;
 
 struct PulsarException {
     Result _result;
-    PulsarException(Result res) :
-            _result(res) {}
+    PulsarException(Result res) : _result(res) {}
 };
 
 inline void CHECK_RESULT(Result res) {

[pulsar] 02/13: Add log error tracking for semaphore count leak (#12410)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 4f87667881065f8a5d039fc78536c821ae907df9
Author: Ali Ahmed <al...@gmail.com>
AuthorDate: Wed Oct 20 13:16:24 2021 -0700

    Add log error tracking for semaphore count leak (#12410)
    
    Co-authored-by: Ali Ahmed <al...@splunk.com>
    (cherry picked from commit 7c219b11966d4eb8cc20111468c3439d23f8777c)
---
 .../apache/pulsar/client/impl/ProducerImpl.java    | 31 +++++++++++++++++-----
 1 file changed, 24 insertions(+), 7 deletions(-)

diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
index bbf75ee..1bd0977 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
@@ -140,6 +140,8 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
 
     private Optional<Long> topicEpoch = Optional.empty();
 
+    private boolean errorState;
+
     @SuppressWarnings("rawtypes")
     private static final AtomicLongFieldUpdater<ProducerImpl> msgIdGeneratorUpdater = AtomicLongFieldUpdater
             .newUpdater(ProducerImpl.class, "msgIdGenerator");
@@ -250,6 +252,21 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
         grabCnx();
     }
 
+    protected void semaphoreRelease(final int releaseCountRequest) {
+        if (semaphore.isPresent()) {
+            if (!errorState) {
+                final int availablePermits = semaphore.get().availablePermits();
+                if (availablePermits - releaseCountRequest < 0) {
+                    log.error("Semaphore permit release count request greater then availablePermits" +
+                                    " : availablePermits={}, releaseCountRequest={}",
+                            availablePermits, releaseCountRequest);
+                    errorState = true;
+                }
+            }
+            semaphore.get().release(releaseCountRequest);
+        }
+    }
+
     protected OpSendMsgQueue createPendingMessagesQueue() {
         return new OpSendMsgQueue();
     }
@@ -1012,9 +1029,9 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
     }
 
     private void releaseSemaphoreForSendOp(OpSendMsg op) {
-        if (semaphore.isPresent()) {
-            semaphore.get().release(isBatchMessagingEnabled() ? op.numMessagesInBatch : 1);
-        }
+
+        semaphoreRelease(isBatchMessagingEnabled() ? op.numMessagesInBatch : 1);
+
         client.getMemoryLimitController().releaseMemory(op.uncompressedSize);
     }
 
@@ -1731,7 +1748,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
             });
 
             pendingMessages.clear();
-            semaphore.ifPresent(s -> s.release(releaseCount.get()));
+            semaphoreRelease(releaseCount.get());
             if (batchMessagingEnabled) {
                 failPendingBatchMessages(ex);
             }
@@ -1757,7 +1774,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
         }
         final int numMessagesInBatch = batchMessageContainer.getNumMessagesInBatch();
         batchMessageContainer.discard(ex);
-        semaphore.ifPresent(s -> s.release(numMessagesInBatch));
+        semaphoreRelease(numMessagesInBatch);
     }
 
     @Override
@@ -1800,9 +1817,9 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
                     processOpSendMsg(opSendMsg);
                 }
             } catch (PulsarClientException e) {
-                semaphore.ifPresent(s -> s.release(batchMessageContainer.getNumMessagesInBatch()));
+                semaphoreRelease(batchMessageContainer.getNumMessagesInBatch());
             } catch (Throwable t) {
-                semaphore.ifPresent(s -> s.release(batchMessageContainer.getNumMessagesInBatch()));
+                semaphoreRelease(batchMessageContainer.getNumMessagesInBatch());
                 log.warn("[{}] [{}] error while create opSendMsg by batch message container", topic, producerName, t);
             }
         }

[pulsar] 13/13: [Security] Upgrade Log4j to 2.17.1 (#13552)

Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 36443112c86afc296780c180154f922d611ebf25
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Wed Dec 29 12:35:57 2021 +0200

    [Security] Upgrade Log4j to 2.17.1 (#13552)
    
    - see https://logging.apache.org/log4j/2.x/security.html
    - mitigates CVE-2021-44832
    
    (cherry picked from commit 978bb7c1998acef749912075ea8f4e1e1c148e2d)
---
 buildtools/pom.xml                               |  2 +-
 distribution/server/src/assemble/LICENSE.bin.txt | 10 +++++-----
 pom.xml                                          |  2 +-
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/buildtools/pom.xml b/buildtools/pom.xml
index a4c4b6d..3eb2a91 100644
--- a/buildtools/pom.xml
+++ b/buildtools/pom.xml
@@ -39,7 +39,7 @@
     <maven.compiler.source>1.8</maven.compiler.source>
     <maven.compiler.target>1.8</maven.compiler.target>
     <surefire.version>3.0.0-M3</surefire.version>
-    <log4j2.version>2.17.0</log4j2.version>
+    <log4j2.version>2.17.1</log4j2.version>
     <slf4j.version>1.7.25</slf4j.version>
     <testng.version>7.3.0</testng.version>
     <commons-lang3.version>3.11</commons-lang3.version>
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt
index 3d08034..a7739f3 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -385,11 +385,11 @@ The Apache Software License, Version 2.0
     - jakarta.validation-jakarta.validation-api-2.0.2.jar
     - javax.validation-validation-api-1.1.0.Final.jar
  * Log4J
-    - org.apache.logging.log4j-log4j-api-2.17.0.jar
-    - org.apache.logging.log4j-log4j-core-2.17.0.jar
-    - org.apache.logging.log4j-log4j-slf4j-impl-2.17.0.jar
-    - org.apache.logging.log4j-log4j-web-2.17.0.jar
-    - org.apache.logging.log4j-log4j-1.2-api-2.17.0.jar
+    - org.apache.logging.log4j-log4j-api-2.17.1.jar
+    - org.apache.logging.log4j-log4j-core-2.17.1.jar
+    - org.apache.logging.log4j-log4j-slf4j-impl-2.17.1.jar
+    - org.apache.logging.log4j-log4j-web-2.17.1.jar
+    - org.apache.logging.log4j-log4j-1.2-api-2.17.1.jar
  * Java Native Access JNA -- net.java.dev.jna-jna-4.2.0.jar
  * BookKeeper
     - org.apache.bookkeeper-bookkeeper-common-4.14.3.jar
diff --git a/pom.xml b/pom.xml
index 1fe972f..54e1710 100644
--- a/pom.xml
+++ b/pom.xml
@@ -119,7 +119,7 @@ flexible messaging model and an intuitive client API.</description>
     <rocksdb.version>6.10.2</rocksdb.version>
     <slf4j.version>1.7.25</slf4j.version>
     <commons.collections.version>3.2.2</commons.collections.version>
-    <log4j2.version>2.17.0</log4j2.version>
+    <log4j2.version>2.17.1</log4j2.version>
     <bouncycastle.version>1.69</bouncycastle.version>
     <bouncycastlefips.version>1.0.2</bouncycastlefips.version>
     <jackson.version>2.12.3</jackson.version>