You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "liangyepianzhou (via GitHub)" <gi...@apache.org> on 2023/06/11 15:09:41 UTC

[GitHub] [pulsar] liangyepianzhou opened a new pull request, #20559: [optimize][txn] Implement lazy loading for transaction buffer

liangyepianzhou opened a new pull request, #20559:
URL: https://github.com/apache/pulsar/pull/20559

   ## Motivation
   This PR introduces lazy loading for the transaction buffer, improving efficiency and resource utilization. It includes the following modifications:
   
   1. Implement lazy loading for the transaction buffer, ensuring it is loaded only when necessary.
   2. Always load the transaction buffer if it has been used before, regardless of the transaction coordinator's state. ## Modification
   1. Modify the topic metadata when the transaction buffer is used for the first time.
   2. Determine the type of transaction buffer based on whether the topic buffer has been used and whether the transaction coordinator is enabled.
   3. Remove the checkIfTBRecoverCompletely function, as it is no longer required. It eliminates the need to take a snapshot when a producer connects. These changes optimize the behavior of the transaction buffer, improving performance and resource management.
   
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: <!-- ENTER URL HERE -->
   
   <!--
   After opening this PR, the build in apache/pulsar will fail and instructions will
   be provided for opening a PR in the PR author's forked repository.
   
   apache/pulsar pull requests should be first tested in your own fork since the 
   apache/pulsar CI based on GitHub Actions has constrained resources and quota.
   GitHub Actions provides separate quota for pull requests that are executed in 
   a forked repository.
   
   The tests will be run in the forked repository until all PR review comments have
   been handled, the tests pass and the PR is approved by a reviewer.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #20559: [optimize][txn] Implement lazy loading for transaction buffer

Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #20559:
URL: https://github.com/apache/pulsar/pull/20559#discussion_r1317029743


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/TransactionBufferProvider.java:
##########
@@ -51,5 +52,5 @@ static TransactionBufferProvider newProvider(String providerClassName) throws IO
      * @param originTopic
      * @return
      */
-    TransactionBuffer newTransactionBuffer(Topic originTopic);
+    TransactionBuffer newTransactionBuffer(Topic originTopic, PositionImpl startUsedPosition);

Review Comment:
   Maybe we do not need to add the new API. We can require the startUsedPosition in the Transaction Buffer again.
   It is still a break-changing even though we added a new default method because we changed the logic in the PersistentTopic that called this method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on pull request #20559: [optimize][txn] Implement lazy loading for transaction buffer

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20559:
URL: https://github.com/apache/pulsar/pull/20559#issuecomment-1631742354

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] poorbarcode commented on a diff in pull request #20559: [optimize][txn] Implement lazy loading for transaction buffer

Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #20559:
URL: https://github.com/apache/pulsar/pull/20559#discussion_r1312688627


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/TransactionBufferProvider.java:
##########
@@ -51,5 +52,5 @@ static TransactionBufferProvider newProvider(String providerClassName) throws IO
      * @param originTopic
      * @return
      */
-    TransactionBuffer newTransactionBuffer(Topic originTopic);
+    TransactionBuffer newTransactionBuffer(Topic originTopic, PositionImpl startUsedPosition);

Review Comment:
   Since the `TransactionBufferProvider` is a public API and users might use the config `transactionBufferProviderClassName` to change the implementation to use, we can't modify the original method; adding a new default method is better.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java:
##########
@@ -250,7 +231,54 @@ public long getCommittedTxnCount() {
     }
 
     @Override
+/**
+ * If it's the first time using the transaction buffer, the method records the max read position of the topic
+ * to metadata of the topic at the time of the first write. If the transaction buffer has been used before,
+ * it writes the message after recovering the transaction buffer.
+ */
     public CompletableFuture<Position> appendBufferToTxn(TxnID txnId, long sequenceId, ByteBuf buffer) {
+        ManagedLedgerImpl managedLedger = (ManagedLedgerImpl) topic.getManagedLedger();
+        if (checkIfReady()) {
+            // Transaction buffer is ready, directly append the buffer
+            return internalAppendBufferToTxn(txnId, sequenceId, buffer);
+        } else if (changeToInitializingState()) {
+            // Transition to initializing state and record max read position if it's the first usage
+            Position firstMaxReadPosition = topic.getManagedLedger().getLastConfirmedEntry();
+            if (managedLedger.getProperties().get(MAX_READ_POSITION) == null) {
+                HashMap<String, String> hashMap = new HashMap<>();
+                hashMap.put(MAX_READ_POSITION, firstMaxReadPosition.toString());
+                managedLedger.asyncSetProperties(hashMap, new AsyncCallbacks.UpdatePropertiesCallback() {
+                    @Override
+                    public void updatePropertiesComplete(Map<String, String> properties, Object ctx) {
+                        changeToReadyState();
+                        transactionBufferFuture.complete(null);
+                    }
+
+                    @Override
+                    public void updatePropertiesFailed(ManagedLedgerException exception, Object ctx) {
+                        log.error("Failed to set first max read position to topic {}", topic.getName(), exception);
+                        changeToCloseState();
+                        transactionBufferFuture.completeExceptionally(exception);
+                    }
+                }, null);
+            }
+        }
+        // Return a CompletableFuture that will complete after recovering the transaction buffer,
+        // then append the buffer
+        return transactionBufferFuture.thenCompose(ignore -> internalAppendBufferToTxn(txnId, sequenceId, buffer));
+    }
+
+    private PositionImpl getPositionFromString(String positionStr) {

Review Comment:
   The method `getPositionFromString` is useless, should we remove it?



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -1459,61 +1459,48 @@ protected void handleProducer(final CommandProducer cmdProducer) {
                     });
 
                     schemaVersionFuture.thenAccept(schemaVersion -> {
-                        topic.checkIfTransactionBufferRecoverCompletely(isTxnEnabled).thenAccept(future -> {

Review Comment:
   The method `checkIfTransactionBufferRecoverCompletely` triggers an event to persist a snapshot once, but you did not persist a snapshot when the first sending with the transition.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java:
##########
@@ -66,6 +69,8 @@
 @Slf4j
 public class TopicTransactionBuffer extends TopicTransactionBufferState implements TransactionBuffer, TimerTask {
 
+    public static final String MAX_READ_POSITION = "maxReadPosition";

Review Comment:
   The prop added does not mean the max read position of the topic, right? It is the first position that sending with the transaction.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java:
##########
@@ -250,7 +231,54 @@ public long getCommittedTxnCount() {
     }
 
     @Override
+/**
+ * If it's the first time using the transaction buffer, the method records the max read position of the topic
+ * to metadata of the topic at the time of the first write. If the transaction buffer has been used before,
+ * it writes the message after recovering the transaction buffer.
+ */
     public CompletableFuture<Position> appendBufferToTxn(TxnID txnId, long sequenceId, ByteBuf buffer) {
+        ManagedLedgerImpl managedLedger = (ManagedLedgerImpl) topic.getManagedLedger();
+        if (checkIfReady()) {
+            // Transaction buffer is ready, directly append the buffer
+            return internalAppendBufferToTxn(txnId, sequenceId, buffer);
+        } else if (changeToInitializingState()) {
+            // Transition to initializing state and record max read position if it's the first usage
+            Position firstMaxReadPosition = topic.getManagedLedger().getLastConfirmedEntry();
+            if (managedLedger.getProperties().get(MAX_READ_POSITION) == null) {
+                HashMap<String, String> hashMap = new HashMap<>();
+                hashMap.put(MAX_READ_POSITION, firstMaxReadPosition.toString());
+                managedLedger.asyncSetProperties(hashMap, new AsyncCallbacks.UpdatePropertiesCallback() {
+                    @Override
+                    public void updatePropertiesComplete(Map<String, String> properties, Object ctx) {
+                        changeToReadyState();

Review Comment:
   Should we start the scheduled task to persist the snapshot here? And add a test for this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #20559: [optimize][txn] Implement lazy loading for transaction buffer

Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #20559:
URL: https://github.com/apache/pulsar/pull/20559#discussion_r1317031590


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java:
##########
@@ -250,7 +231,54 @@ public long getCommittedTxnCount() {
     }
 
     @Override
+/**
+ * If it's the first time using the transaction buffer, the method records the max read position of the topic
+ * to metadata of the topic at the time of the first write. If the transaction buffer has been used before,
+ * it writes the message after recovering the transaction buffer.
+ */
     public CompletableFuture<Position> appendBufferToTxn(TxnID txnId, long sequenceId, ByteBuf buffer) {
+        ManagedLedgerImpl managedLedger = (ManagedLedgerImpl) topic.getManagedLedger();
+        if (checkIfReady()) {
+            // Transaction buffer is ready, directly append the buffer
+            return internalAppendBufferToTxn(txnId, sequenceId, buffer);
+        } else if (changeToInitializingState()) {
+            // Transition to initializing state and record max read position if it's the first usage
+            Position firstMaxReadPosition = topic.getManagedLedger().getLastConfirmedEntry();
+            if (managedLedger.getProperties().get(MAX_READ_POSITION) == null) {
+                HashMap<String, String> hashMap = new HashMap<>();
+                hashMap.put(MAX_READ_POSITION, firstMaxReadPosition.toString());
+                managedLedger.asyncSetProperties(hashMap, new AsyncCallbacks.UpdatePropertiesCallback() {
+                    @Override
+                    public void updatePropertiesComplete(Map<String, String> properties, Object ctx) {
+                        changeToReadyState();
+                        transactionBufferFuture.complete(null);
+                    }
+
+                    @Override
+                    public void updatePropertiesFailed(ManagedLedgerException exception, Object ctx) {
+                        log.error("Failed to set first max read position to topic {}", topic.getName(), exception);
+                        changeToCloseState();
+                        transactionBufferFuture.completeExceptionally(exception);
+                    }
+                }, null);
+            }
+        }
+        // Return a CompletableFuture that will complete after recovering the transaction buffer,
+        // then append the buffer
+        return transactionBufferFuture.thenCompose(ignore -> internalAppendBufferToTxn(txnId, sequenceId, buffer));
+    }
+
+    private PositionImpl getPositionFromString(String positionStr) {

Review Comment:
   I remove the new adding struct method of TransactionBuffer, so it is needed in the TransactionBuffer.
   We can not make this method is public in the PersistentTopic.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java:
##########
@@ -250,7 +231,54 @@ public long getCommittedTxnCount() {
     }
 
     @Override
+/**
+ * If it's the first time using the transaction buffer, the method records the max read position of the topic
+ * to metadata of the topic at the time of the first write. If the transaction buffer has been used before,
+ * it writes the message after recovering the transaction buffer.
+ */
     public CompletableFuture<Position> appendBufferToTxn(TxnID txnId, long sequenceId, ByteBuf buffer) {
+        ManagedLedgerImpl managedLedger = (ManagedLedgerImpl) topic.getManagedLedger();
+        if (checkIfReady()) {
+            // Transaction buffer is ready, directly append the buffer
+            return internalAppendBufferToTxn(txnId, sequenceId, buffer);
+        } else if (changeToInitializingState()) {
+            // Transition to initializing state and record max read position if it's the first usage
+            Position firstMaxReadPosition = topic.getManagedLedger().getLastConfirmedEntry();
+            if (managedLedger.getProperties().get(MAX_READ_POSITION) == null) {
+                HashMap<String, String> hashMap = new HashMap<>();
+                hashMap.put(MAX_READ_POSITION, firstMaxReadPosition.toString());
+                managedLedger.asyncSetProperties(hashMap, new AsyncCallbacks.UpdatePropertiesCallback() {
+                    @Override
+                    public void updatePropertiesComplete(Map<String, String> properties, Object ctx) {
+                        changeToReadyState();
+                        transactionBufferFuture.complete(null);
+                    }
+
+                    @Override
+                    public void updatePropertiesFailed(ManagedLedgerException exception, Object ctx) {
+                        log.error("Failed to set first max read position to topic {}", topic.getName(), exception);
+                        changeToCloseState();
+                        transactionBufferFuture.completeExceptionally(exception);
+                    }
+                }, null);
+            }
+        }
+        // Return a CompletableFuture that will complete after recovering the transaction buffer,
+        // then append the buffer
+        return transactionBufferFuture.thenCompose(ignore -> internalAppendBufferToTxn(txnId, sequenceId, buffer));
+    }
+
+    private PositionImpl getPositionFromString(String positionStr) {

Review Comment:
   I remove the new adding struct method of TransactionBuffer, so it is needed in the TransactionBuffer.
   We can not make this method public in the PersistentTopic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] liangyepianzhou closed pull request #20559: [optimize][txn] Implement lazy loading for transaction buffer

Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou closed pull request #20559: [optimize][txn] Implement lazy loading for transaction buffer
URL: https://github.com/apache/pulsar/pull/20559


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #20559: [optimize][txn] Implement lazy loading for transaction buffer

Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #20559:
URL: https://github.com/apache/pulsar/pull/20559#discussion_r1316933551


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java:
##########
@@ -250,7 +231,54 @@ public long getCommittedTxnCount() {
     }
 
     @Override
+/**
+ * If it's the first time using the transaction buffer, the method records the max read position of the topic
+ * to metadata of the topic at the time of the first write. If the transaction buffer has been used before,
+ * it writes the message after recovering the transaction buffer.
+ */
     public CompletableFuture<Position> appendBufferToTxn(TxnID txnId, long sequenceId, ByteBuf buffer) {
+        ManagedLedgerImpl managedLedger = (ManagedLedgerImpl) topic.getManagedLedger();
+        if (checkIfReady()) {
+            // Transaction buffer is ready, directly append the buffer
+            return internalAppendBufferToTxn(txnId, sequenceId, buffer);
+        } else if (changeToInitializingState()) {
+            // Transition to initializing state and record max read position if it's the first usage
+            Position firstMaxReadPosition = topic.getManagedLedger().getLastConfirmedEntry();
+            if (managedLedger.getProperties().get(MAX_READ_POSITION) == null) {
+                HashMap<String, String> hashMap = new HashMap<>();
+                hashMap.put(MAX_READ_POSITION, firstMaxReadPosition.toString());
+                managedLedger.asyncSetProperties(hashMap, new AsyncCallbacks.UpdatePropertiesCallback() {
+                    @Override
+                    public void updatePropertiesComplete(Map<String, String> properties, Object ctx) {
+                        changeToReadyState();

Review Comment:
   We have a scheduled task to do a similar thing in the snapshot processor. 
   We only write the first snapshot in the managedLedger properties, and should not add more dependencies to metadata.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -1459,61 +1459,48 @@ protected void handleProducer(final CommandProducer cmdProducer) {
                     });
 
                     schemaVersionFuture.thenAccept(schemaVersion -> {
-                        topic.checkIfTransactionBufferRecoverCompletely(isTxnEnabled).thenAccept(future -> {

Review Comment:
   `checkIfTransactionBufferRecoverCompletely` is triggered when building the producer.
   We should persist a snapshot when first sending with a transaction, and that is implemented in TransactionBuffer.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java:
##########
@@ -66,6 +69,8 @@
 @Slf4j
 public class TopicTransactionBuffer extends TopicTransactionBufferState implements TransactionBuffer, TimerTask {
 
+    public static final String MAX_READ_POSITION = "maxReadPosition";

Review Comment:
   Yeah, it is the last confirmed position before the first transaction message.
   When the state of the transaction message is confirmed, this position is the max read position. 
   Or if you have any other suggestions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org