You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/11/20 00:25:32 UTC

[GitHub] [pulsar] zymap opened a new pull request #8635: Fix the pulsar sql can not query the last message issue

zymap opened a new pull request #8635:
URL: https://github.com/apache/pulsar/pull/8635


   Fixes #7811
   
   ### Motivation
   
   Currently, the bookkeeper client gets the LAC is the piggyback lac which is carried by the next message. This causes the pulsar SQL connector will always get the `message number -1` messages. If we want to query all messages, we need to use the explicit LAC to get the total number of entries. 
   BookKeeper 4.12.0 makes us can use the `useV2Protocol` to enable using v3 protocol which will get the explicit LAC.
   
   ### Modifications
   
   - Add the properties for the pulsar SQL.
   - Update the integration test to query the last message.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] zymap commented on pull request #8635: Fix the pulsar sql can not query the last message issue

Posted by GitBox <gi...@apache.org>.
zymap commented on pull request #8635:
URL: https://github.com/apache/pulsar/pull/8635#issuecomment-730729886


   @codelipenghui I addressed your comments.PTAL again. Thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] zymap commented on pull request #8635: Fix the pulsar sql can not query the last message issue

Posted by GitBox <gi...@apache.org>.
zymap commented on pull request #8635:
URL: https://github.com/apache/pulsar/pull/8635#issuecomment-731077900


   /pulsarbot run-failure-checks
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui merged pull request #8635: Fix the pulsar sql can not query the last message issue

Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #8635:
URL: https://github.com/apache/pulsar/pull/8635


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on a change in pull request #8635: Fix the pulsar sql can not query the last message issue

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #8635:
URL: https://github.com/apache/pulsar/pull/8635#discussion_r527309403



##########
File path: conf/broker.conf
##########
@@ -774,6 +774,9 @@ bookkeeperDiskWeightBasedPlacementEnabled=false
 # A value of '0' disables sending any explicit LACs. Default is 0.
 bookkeeperExplicitLacIntervalInMills=0
 
+  # Use older Bookkeeper wire protocol with bookie
+bookkeeperUseV2WireProtocol=true

Review comment:
       I think we already have this config in the broker.conf https://github.com/apache/pulsar/blob/master/conf/broker.conf#L680

##########
File path: pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarConnectorConfig.java
##########
@@ -72,6 +72,8 @@
     private int bookkeeperThrottleValue = 0;
     private int bookkeeperNumIOThreads = 2 * Runtime.getRuntime().availableProcessors();
     private int bookkeeperNumWorkerThreads = Runtime.getRuntime().availableProcessors();
+    private boolean useV2Protocol = true;

Review comment:
       ```suggestion
       private boolean bookkeeperUseV2Protocol = true;
   ```

##########
File path: pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarConnectorConfig.java
##########
@@ -72,6 +72,8 @@
     private int bookkeeperThrottleValue = 0;
     private int bookkeeperNumIOThreads = 2 * Runtime.getRuntime().availableProcessors();
     private int bookkeeperNumWorkerThreads = Runtime.getRuntime().availableProcessors();
+    private boolean useV2Protocol = true;
+    private int explicitInterval = 0;

Review comment:
       ```suggestion
       private int bookkeeperExplicitInterval = 0;
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org