You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/27 12:29:10 UTC

[GitHub] [pulsar] 315157973 opened a new pull request #10729: PIP 81: Split the individual acknowledgments into multiple entries

315157973 opened a new pull request #10729:
URL: https://github.com/apache/pulsar/pull/10729

### Motivation
Pulsar persists the individual acknowledgments into Ledger for each subscription.
`acknowledgment hole` uses guava range to record.
Now a single entry is used to record these holes, but a single entry can not stores more than 5MB data.
Therefore, this PIP stores these holes in different entries separately. And use LRU to ensure low memory usage.
In this way, we can support almost unlimited `acknowledgment hole`.

Doc: https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit

### Modifications
recover:
1. Added API `recoverFromMultiEntry`
2. The old logic is abstract as `recoverFromSingleEntry`

persist position to ledger：
1. Add API `persistPositionToMultiEntry` and `saveMarker`
2. The old logic is abstract as `persistPositionToSingleEntry`

when creating new ledger:
1. Add API `copyLruEntriesToNewLedger`
2. The old logic remains the same

LRU:
1. Add class RangeSetWrapper.Responsible for the loading of LRU data and elimination of LRU data.
2. The loading of LRU data is synchronous, and elimination of LRU data is asynchronous, with `RangeSetWrapper#LruTask`

### Verifying this change
persist position to ledger has 3 chain:
```
internalAsyncMarkDelete + NoLedger -> startCreatingNewMetadataLedger -> createNewMetadataLedgerAndSwitch -> doCreateNewMetadataLedger -> persistPositionToLedger -> switchToNewLedger -> flushPendingMarkDeletes -> internalMarkDelete -> persistPositionToLedger -> remove individualDeletedMessages
```

```
internalAsyncMarkDelete + Ledger is Open -> internalMarkDelete -> persistPositionToLedger -> remove individualDeletedMessages
```

```
persistPositionWhenClosing -> persistPositionToLedger -> close ledger
```

create new ledger has 1 chain:
```
initialize -> createNewMetadataLedgerAndSwitch -> doCreateNewMetadataLedger -> switchToNewLedger -> flushPendingMarkDeletes -> internalMarkDelete -> persistPositionToLedger -> remove individualDeletedMessages
```

recover has 1 chain:
```
initializeCursors -> recover -> recoverFromLedger -> recoverFromMultiEntry
```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org