You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/08/02 08:23:27 UTC

[GitHub] [pulsar] liangyepianzhou opened a new issue, #16913: [PIP-196] TransactionBuffer Multiple-snapshots

liangyepianzhou opened a new issue, #16913:
URL: https://github.com/apache/pulsar/issues/16913

   ## ## Background
   1. We use TransactionBuffer to handle messages sent with transactions
   1. TransactionBuffer has two variables: aborts and maxReadPosition.
      1. maxReadPosition can be used to limit where is the max position that the consumer can consume.
      1. Aborts store the aborted transactions, it was used to filter the messages sent by aborted transactions.
   1. In order to avoid the need for a lot of time to recover these two data after the broker restarts, **snapshots** of these two data are taken regularly.
   1. The snapshot will be stored into a system topic that all the topics in a namespace share.
   
   More details about the current implementation of TransactionBuffer can be found in the following document:
   [A Deep-dive of Apache Pulsar transaction buffer](https://docs.google.com/document/d/1pC1xRYwdVQyO4eJ1yFYTIxbB9O-nGD5sIiC0-cSOHro/edit#heading=h.qw7r6j6d4lp7)
   
   ## ## Motivation
   In the [background](#_ln17rdvx5u9y), we know that TransactionBuffer takes snapshots regularly. And the snapshot contains a position (`maxReadPosition`) and a linkedMap (`*aborts`)* which stores all the valid aborted transactions of the topic.  
   ` `But when the manager ledger retention time is set too long and the topic has a lot of aborted transactions, a snapshot can not store all the aborted transactions. And then we need to use multiple snapshots to store the excessive aborted transactions.
   ## ## Goal
   1. Store the excessive aborted transactions by multiple snapshots.
   1. Recover on demand when recovering, not read into memory at once.
   1. Compatible with previous implementations.
   ## ## Overview
   Use the origin snapshot topic to store the snapshots containing the fixed number of transactions. The number of the transactions in a snapshot can be configured, and we hope it is small, then we can merge the small snapshots into a large snapshot when it reaches a configured number.  We add a [snapshot index class](#_to1qhl1yo8ao) to store the snapshot index information. And add a snapshot index topic to store the indexes, when the transaction buffer recovers, it only needs to read the index topic and build the index. Then use the index to read on demand instead of reading all at once.
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 001](https://user-images.githubusercontent.com/55571188/182326692-41409409-303a-4ab9-a1c2-5c7f7a792d52.png)
   
   ## ## Configuration
   
   
   |Configuration|Description|
   | :- | :- |
   |AbortedTxnNumbersInSnapshot|To determine how many aborted transactions are in a snapshot.|
   |SnapshotsNumberToDoMerge|To determine how many snapshots a merged snapshot consists of.|
   
   ## ## Implementation
   ### ### Change the data structure of storing aborted transactions
   The `*aborts`* changes from LinkedMap to ConcurrentSkipListMap<maxReadPosition, Set<TxnID>>. We store snapshots in a ` ConcurrentSkipListMap<maxReadPosition, Set<TxnID>> ` instead of a `LinkedMap`.
   
   You can see in the following example, that every Set will store aborted transactions and has a maxReadPosition as its key. When a set<TxnID> has contained the `AbortedTxnNumbersInSnapshot` number of the aborted transactions, it will increase the sequence ID and take a snapshot. If the number of small snapshots reaches `SnapshotsNumberToDoMerge`,  the small snapshot will merge into a larger snapshot.
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 002](https://user-images.githubusercontent.com/55571188/182326780-a8913597-2722-4f9d-8b25-48c6479edecc.png)
   
   ### ### Snapshot topic
   Transaction buffer snapshot contains aborted transactions and the corresponding max read position. And the snapshots are stored in the shared system topic, more details can be found in  [A Deep-dive of Apache Pulsar transaction buffer](https://docs.google.com/document/d/1pC1xRYwdVQyO4eJ1yFYTIxbB9O-nGD5sIiC0-cSOHro/edit#heading=h.qw7r6j6d4lp7)
   
   In this PIP, we still use the current implementation to store snapshots, that a namespace uses the shared system topic to store snapshots, except for the last snapshot.
   
   In the following example, we can see the system topic store 3 snapshots for 2 topics, 
   
   Topic1 has 2 snapshots: snapshot 1, and snapshot3, and topic2 has 1 snapshot: snapshot2.  
   And the latest snapshot of each snapshot is stored in the [index topic](#_bsz1dpz4rs9j).
   
   The number of the transactions in a snapshot can be [configured](#_xp5ghegqbcuj), and we hope it is small, then we can [merge](#_npeei6ibtqxb) the small snapshots into a large snapshot when it reaches a [configured](#_xp5ghegqbcuj) number. 
   
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 003](https://user-images.githubusercontent.com/55571188/182326895-073cedaf-cf16-4bf4-95dd-846c277e5101.png)
   
   ### ### SnapshotIndex Class
   ` `We add a snapshot index class to store the snapshot information,  including 
   
   1. The snapshot sequence ID.
   1. The max read position of the snapshot.
   1. whether the snapshot is a snapshot that is merged with other snapshots
   1. A sequence ID Set of  What snapshots are the snapshot contain if it is a merged snapshot?
   1. The Position where the snapshot is persistent.
   
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 004](https://user-images.githubusercontent.com/55571188/182327115-29cbde38-dda2-4f10-b8d0-ef97437b1bb6.png)
   
   ### ###  snapshot index topic
   We add a system topic to store the index of the snapshots in the snapshot topic.
   
   Just like the snapshot topic, the index topic is shared by all topics in a namespace.
   
   The index is written after each multiple-snapshot is written.
   
   In the following example, the index topic stores the index of the snapshots of the 2 topics in the above example. There are stored the position and maxReadPosition for every snapshot in the index. And the latest snapshot for each topic is stored in the index topic too.
   
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 005](https://user-images.githubusercontent.com/55571188/182327173-b3a00fc9-cd8f-4504-b963-196683b7f6dc.png)
   
   ##
   ### ### Indexes Map
   When the transaction reads the index topic, it will store the max read position and its snapshotIndex in a concurrentSkipMap. It is used for reading snapshots on demand.
   
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 006](https://user-images.githubusercontent.com/55571188/182327288-7e994640-5073-4c98-a3d7-fbe2936aa270.png)
   
   ### ### Merge snapshots
   When the number of small snapshots reaches the configuration `SnapshotsNumberToDoMerge`, these small snapshots will be merged into a large snapshot. The snapshot merge can be done like this:
   
   1. ` `Write a large snapshot into the snapshot topic.
   1. `  `Update the topic snapshot index, indicate that this snapshot is a merged snapshot, and save the sequence ID of the previous snapshot it contains in its [SnapshotIndex](#_to1qhl1yo8ao).
   1. Delete these small snapshots by compactor which key = multiple-sequenceID-topicName and value =null.
   1. Delete the indexes of these small snapshots.
   
   If these operations are failed at operation 2, the old snapshots will be covered by the new large snapshot when compact due to they have the same sequence ID.  
   If these operations are failed at operation 3, these small snapshots will be deleted after the transaction buffer is recovered. Because when the topic transaction buffer read the indexes to memory if the latest index is a merged snapshot, and its consisted snapshots still exist in the index, it will do operation3 operation4 again.
   
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 002](https://user-images.githubusercontent.com/55571188/182327362-a7aac7e1-6a14-4845-a2e0-21027d9407c3.png)
   
   ### ### Read on demand
   Implement two hashmaps for topicTransactionBuffer to store the ship of subscription cursor and maxReadPosition.
   #### #### GetMaxReadPosition (Methodparam: cursor)
   When a consumer reads messages, it will get the maxReadPosition from the transaction buffer.
   
   If the transaction buffer has stored the aborted transactions from the read position to the next max read position, the transaction buffer will return the next max read position to the consumer.
   
   ` `if not, the transaction buffer will return the max Read Position before its read position and read the next max read position snapshot into memory using the snapshot index.
   
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 007](https://user-images.githubusercontent.com/55571188/182327422-21b34307-7efe-4825-9993-75e6385db93d.png)
   
   #### #### Check aborts (Methodparam: txnID, entry position)
   When a consumer checks whether a transaction message is an aborted transaction message with the transaction ID and the message position, the transaction buffer will use the message position to get the corresponding aborted transactions in [aborts](#_b6fzlp7b8k2w) to check whether the transaction ID is an aborted transaction.
   #### #### Clean up unused aborts data
   When all subscription cursors relating to this topic transaction buffer already acknowledged a position, we can delete the cache aborted transactions whose max read position less than the smallest mark delete position. So there will be a timer to check the snapshots whose max read position is smaller than the smallest mark delete position,
   
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 008](https://user-images.githubusercontent.com/55571188/182327495-f496b8a3-3e3b-41ff-9ee5-9965efd7322c.png)
   
   ### ### compatibility
   When TransactionBuffer recovers if read the snapshot with key `topicName` without starting with `multiple`, read the snapshot into memory as the first snapshot of multiple snapshots, and then write the snapshot into snapshot topic and write an index into index topic. Regardless of whether the number of aborted transactions in this snapshot is greater or less than the number of aborted transactions that we configure for a multiple-snapshot.
   ### ### Summary
   1. We can use multiple snapshots to store the aborted transactions for this topic.
   1. Add a variable snapshot sequence ID in the transaction buffer. The sequence ID represents the number of taking snapshots. And the sequence ID will be written into the snapshot topic as a part of the key of the snapshot.
   1. The key of the snapshots can be set as  `multiple-sequenceID-topicName`. Make the multiple-sequenceID at preview, we can ignore the impact of the topicName to key. 
   1. We can add two [configurations](#_xp5ghegqbcuj) to configure the number of aborted transactions that a snapshot contains and how many snapshots should be [merged](#_npeei6ibtqxb) into a large snapshot.
   1. Add a new class `[*SnapshotIndex*](#_to1qhl1yo8ao)` to store the snapshot index messages. 
   1. Add a system topic that is shared by all topics in a namespace, and use this system topic to store the snapshot indexes of the snapshot system topic. Then we can use this index topic to recover on demand, not read all snapshots into memory at once. And the latest snapshot which has the variable number of aborted transactions is stored in the index topic.
   
   ![Aspose Words 614e497f-3373-45b5-9d97-6cafa52a0d8d 009](https://user-images.githubusercontent.com/55571188/182327775-19bd454a-abcd-43b4-bc6a-ae0880e05377.png)
   
   1. [Read multiple snapshots into memory on demand.](#_w04uhkeasji7)
   1. For [compatibility](#_9p3zggyyj7qg), when TransactionBuffer recovering, if read the snapshot with key `topicName` without starting with `multiple`, read the snapshot into memory as the first snapshot of multiple snapshots,  and then write the snapshot into snapshot topic and write an index into index topic.
   
   ### ### Workflow
   #### #### take snapshot
   ##### ##### When?
   Same as the previous implementation to take latest snapshot:
   
   1. Reach the **transactionBufferSnapshotMaxTransactionCount**
   1. Reach the **transactionBufferSnapshotMinTimeInMillis**
   
   Changes in the current implementation:
   
   1. When a Set<TxnID> reaches the maximum number of aborted transactions. Just like the aborted transactions reach 1000 in a Set<TxnID> in[ the above example](#_b6fzlp7b8k2w).
   1. When the transaction buffer recovers, the transaction buffer read a snapshot with a key not starting with `multiple`. This is for [compatibility](#_9p3zggyyj7qg).
   1. When the number of small snapshots reaches the `[SnapshotsNumberToDoMerge](#_xp5ghegqbcuj)` merge the small snapshots into a large snapshot.
   ##### ##### How?
   If the aborted transactions in a snapshot are full, increase the sequence ID, and take snapshots with key multiple-sequenceID-topicName. If the aborted transactions in a snapshot are not full, in other words, this snapshot has the latest maxReadPosition, take snapshots with key snapshot-topicName into[ the index topic](#_kqq8tleale5i).
   
   If have aborts been deleted, we can delete the snapshot according to indexes by compressing for key = multiple-sequenceID-topicName and value = null. 
   #### #### Recover
   1. Read the index topic from the beginning position to build indexes, and read the latest snapshot into memory.
   1. If the latest index is a merged class and the small snapshots that it consisted of still si in the indexes, to do delete the small snapshots again.[  More details.](#_npeei6ibtqxb)
   #### #### Support subscriptions check aborted transactions.
   You can see this in the section[ read on demand](#_w04uhkeasji7).
   ## ## Implement Step
   1. Add configurations
   1. Implement multiple snapshots and index topics.
   1. Implement merge small snapshot.
   1. Implement read multiple snapshots on demand.
   ## ## Reject implementation
   ### ### optimize the data structure that stores the aborted transactions
   The transaction buffer uses a LinkedMap<TxnID, Position> to store aborted transactions. This position is where the transaction commits or abort marker is written.
   
   We have tried using RoaringBitmap to store  TxnID, but RoaringBitmap does not compress well for huge data intervals. And transaction. In normal business scenarios, there may be an aborted transaction every 10W transactions. In this case, RoaringBitmap will use more memory. 
   
   The test report can be found [here](https://docs.google.com/document/u/0/d/1TDjWh2DmMFOes_p_b-H1CpbEs0oz_Uo5v3GhURPM7Gg/edit).
   ### ### Change to managerLedger implementation
   
   Since the shared system topic has a series of disadvantages mentioned above, we consider the implementation of one managerLedger per topic transaction buffer.
   
   In this case, the topic transaction buffer can easily delete useless data, and the two questions above can be easily resolved.
   
   But this solution will increase the metadata used by the topic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] liangyepianzhou closed issue #16913: PIP-196 Segmented transaction buffer snapshot

Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou closed issue #16913: PIP-196 Segmented transaction buffer snapshot
URL: https://github.com/apache/pulsar/issues/16913


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on issue #16913: PIP-196 Segmented transaction buffer snapshot

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16913:
URL: https://github.com/apache/pulsar/issues/16913#issuecomment-1273998490

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] RobertIndie commented on issue #16913: PIP-196 Segmented transaction buffer snapshot

Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
RobertIndie commented on issue #16913:
URL: https://github.com/apache/pulsar/issues/16913#issuecomment-1502626459

   Confirmed with @liangyepianzhou . There is still work left for metric and admin. Move this PIP to the 3.1.0 milestone to track.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] liangyepianzhou commented on issue #16913: PIP-196 Segmented transaction buffer snapshot

Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on issue #16913:
URL: https://github.com/apache/pulsar/issues/16913#issuecomment-1241414006

   The latest proposal can be found [here](https://docs.google.com/document/d/1hBk2nGcj0Os-ULi2q404gCoxIsPleGY8p5H1hqWD5kI/edit#heading=h.t85y6pbv4a48).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] RobertIndie commented on issue #16913: PIP-196 Segmented transaction buffer snapshot

Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
RobertIndie commented on issue #16913:
URL: https://github.com/apache/pulsar/issues/16913#issuecomment-1502626404

   Confirmed with @liangyepianzhou . There is still work left for metric and admin. Move this PIP to the 3.1.0 milestone to track.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org