You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/23 17:57:45 UTC

[GitHub] [pulsar] BewareMyPower opened a new pull request, #16202: [feature][broker] PIP 37: Support chunking with Shared subscription

BewareMyPower opened a new pull request, #16202:
URL: https://github.com/apache/pulsar/pull/16202

### Motivation

https://github.com/apache/pulsar/wiki/PIP-37%3A-Large-message-size-handling-in-Pulsar#option-1-broker-caches-mapping-of-message-uuid-and-consumerid

The cons of option 1 described in the original proposal don't exist
for current code because broker keeps redelivered messages into
**sorted** map now.

### Modifications

First of all, to avoid too many code changes, an `EntryAndMetadata`
class is introduced to bind the `Entry` with the associated
`MessageMetadata` to avoid parsing the metadata repeatedly. It also
implements the `Entry` interface, so this PR changes some
`List<Entry>` parameters to `List<? extends Entry>` so that a
`List<EntryAndMetadata>` argument can be accepted.

Then, a `SharedConsumerAssignor` is introduced to assign a list of
entries to all shared consumers.
1. Use a default selector to select the next consumer, like
`PersistentDispatcherMultipleConsumers#getNextConsumer`,
2. Each time a consumer is chosen, assign the entries in range
[i, i+permits) to the consumer except entries that have uuid:
- If uuid is not cached, cache `uuid -> consumer` to indicate the
chunked message of this uuid must be dispatched to this consumer.
- Otherwise, assign this entry to the owner consumer of the uuid.

The `assign` method returns a map that maps `Consumer` to
`List<EntryAndMetadata>`. The following logic is similar to the
Key_Shared dispatcher.

Finally, cancel the limit in `ConsumerImpl`.

### Verifying this change

`SharedConsumerAssignorTest` is added to show how the assignor works
in detail.

`MessageChunkingSharedTest` is added to verify the Shared dispatcher
works on chunked messages, including:
- Single producer sends chunked messages with various chunk count to a
consumer has a limited permits.
- Single producer sends chunked messages to two consumers to verify
both they can receive chunked messages.
- Produce interleaved chunks via `PersistentTopic` directly to simulate
multiple producers, and verify the new consumer can receive all
unacknowledged messages received by the old consumer.

### TODO

We need to change the implementation of `ChunkMessageIdImpl` to make
it possible for consumer to acknowledge all entries of a chunked
message.

Since this PR already includes many changes, I will do that later.

### Documentation

Check the box below or label this PR directly.

Need to update docs?

- [ ] `doc-required`
(Your PR needs to update docs and you will update later)

- [x] `doc-not-needed`
(Please explain why)

- [ ] `doc`
(Your PR contains doc changes)

- [ ] `doc-complete`
(Docs have been already added)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org