[GitHub] [pulsar] sijie commented on a change in pull request #4879: [Transaction][Buffer]make the transaction index to store in the ledger

[GitBox <gi...@apache.org>]

sijie commented on a change in pull request #4879: [Transaction][Buffer]make the transaction index to store in the ledger
URL: https://github.com/apache/pulsar/pull/4879#discussion_r310854778
 
 

 ##########
 File path: pulsar-transaction/buffer/src/main/java/org/apache/pulsar/transaction/buffer/impl/TransactionCursorImpl.java
 ##########
 @@ -123,4 +204,377 @@ private void addTxnToCommittedIndex(TxnID txnID, long committedAtLedgerId) {
 
         return removeFuture;
     }
+
+    // Take a snapshot for all indexes. We can persist the transaction meta because the indexes can be rebuilt by it.
+    // a. Create a begin block and put the current transaction log position into it.
+    // b. Create the middle  block to store the transaction meta  and the snapshot start position.
+    // c. Create the end block to say the snapshot is ending and put the sanpshot start position to get the number of
+    //    snapshot blocks  when recovering.
+    public CompletableFuture<Void> takeSnapshot(Position txnBufferPosition) {
+        return startSnapshot(txnBufferPosition)
+            .thenCompose(position -> indexSnapshot(position, txnIndex.values()))
+            .thenCompose(position -> endSnapshot(position));
+    }
+
+    private CompletableFuture<Position> startSnapshot(Position position) {
+        return record(DataFormat.startStore(position));
+    }
+
+    private CompletableFuture<Position> indexSnapshot(Position startSnapshotPos,
+                                               Collection<TransactionMetaImpl> snapshotsMeta) {
+        List<CompletableFuture<Position>> snapshot =
 
 Review comment:
   - one transaction metadata can be huge. so you need to avoid sending an entry larger than the entrySize that bookkeeper can accept. so you need to support storing multiple metadata chunks for one transaction metadata.
   
   - avoid sending all the transaction metadata as the same time. add a TODO and create an issue to implement throttling mechanism.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services