You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/02 04:18:21 UTC

[GitHub] [pulsar] nodece opened a new issue #14529: PIP-146 ManagedCursorInfo compression

nodece opened a new issue #14529:
URL: https://github.com/apache/pulsar/issues/14529

<!---
Instructions for creating a PIP using this issue template:

1. The author(s) of the proposal will create a GitHub issue ticket using this template.
(Optionally, it can be helpful to send a note discussing the proposal to
dev@pulsar.apache.org mailing list before submitting this GitHub issue. This discussion can
help developers gauge interest in the proposed changes before formalizing the proposal.)
2. The author(s) will send a note to the dev@pulsar.apache.org mailing list
to start the discussion, using subject prefix `[PIP] xxx`. To determine the appropriate PIP
number `xxx`, inspect the mailing list (https://lists.apache.org/list.html?dev@pulsar.apache.org)
for the most recent PIP. Add 1 to that PIP's number to get your PIP's number.
3. Based on the discussion and feedback, some changes might be applied by
the author(s) to the text of the proposal.
4. Once some consensus is reached, there will be a vote to formally approve
the proposal. The vote will be held on the dev@pulsar.apache.org mailing list. Everyone
is welcome to vote on the proposal, though it will considered to be binding
only the vote of PMC members. It will be required to have a lazy majority of
at least 3 binding +1s votes. The vote should stay open for at least 48 hours.
5. When the vote is closed, if the outcome is positive, the state of the
proposal is updated and the Pull Requests associated with this proposal can
start to get merged into the master branch.

-->

## Motivation

The cursor data is managed by ZooKeeper/etcd metadata store. When cursor data becomes more and more, the data size will increase and will take a lot of time to pull the data. Therefore, it is necessary to add compression for the cursor, which can reduce the size of data and reduce the time of pulling data.

## Goal

Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the ManagedCursorInfo.

## Implementation

- Cursor compression format
```
[MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + [MANAGED_CURSOR_INFO_PAYLOAD]
```

- MAGIC_NUMBER
Ox4779

- METADATA
Add a named `ManagedCursorInfoMetadata` message to `MLDataFormats.proto`:

```
message ManagedCursorInfoMetadata {
required CompressionType compressionType = 1;
required int32 uncompressedSize = 2;
}
```

Currently, these compressions have been supported, we only need to deal with compression and decompression of the `ManagedCursorInfo` data:

- Get CursorInfo from the metadata store
We will check the cursor data header, if it is compressed, we will parse the bytes data by compressed format, otherwise by the original way.

- Add/Update CursorInfo to the metadata store
The default is to use compression if the compression type is specified.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nodece closed issue #14529: PIP-146: ManagedCursorInfo compression

Posted by GitBox <gi...@apache.org>.

nodece closed issue #14529:
URL: https://github.com/apache/pulsar/issues/14529


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nodece commented on issue #14529: PIP-146: ManagedCursorInfo compression

Posted by GitBox <gi...@apache.org>.

nodece commented on issue #14529:
URL: https://github.com/apache/pulsar/issues/14529#issuecomment-1068818751


   This proposal has 3 (+1) bindings and 0 (-1) and has stayed open for at least 48 hours:
   https://lists.apache.org/thread/kvpostlbqty87cdjgt7vrfozn3qz2j1t
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org