New proposal for chunk messages with shared subscriptions

[Yunze Xu <yz...@streamnative.io.INVALID>]

Hi folks,

Recently I'm working on the implementation of PIP-37, see
https://github.com/apache/pulsar/wiki/PIP-37%3A-Large-message-size-handling-in-Pulsar#usecase-3-multiple-producers-with-shared-consumers 

As we can see, https://github.com/apache/pulsar/pull/4400 only
implements chunking messages with non-shared subscriptions. When I
followed the **Option 2** section, I found it works but there are many
details that need to be taken care of.

For example,
- Should we add a marker type to indicate the chunk marker?
- Normally, the markers like Transaction markers are not visible to
  the client, but we need to send the chunk marker to client.
- What's the format of the chunk marker?
- Which compatibility problems would be brought by this design?

I think we need a new proposal to explain it in details and I'm
working on that, as well as the demo.

Feel free to ping me if you have any concern.

Thanks,
Yunze

Re: New proposal for chunk messages with shared subscriptions

[Yunze Xu <yz...@streamnative.io.INVALID>]

I opened an initial PR for it: https://github.com/apache/pulsar/pull/16061 

It doesn’t adopt any option of the original PIP 37. I think we need another
proposal for it, just like the chunked message ID in PIP 107.


Thanks,
Yunze




> 2022年6月7日 22:12，Yunze Xu <yz...@streamnative.io> 写道：
> 
> Hi folks,
> 
> Recently I'm working on the implementation of PIP-37, see
> https://github.com/apache/pulsar/wiki/PIP-37%3A-Large-message-size-handling-in-Pulsar#usecase-3-multiple-producers-with-shared-consumers 
> 
> As we can see, https://github.com/apache/pulsar/pull/4400 only
> implements chunking messages with non-shared subscriptions. When I
> followed the **Option 2** section, I found it works but there are many
> details that need to be taken care of.
> 
> For example,
> - Should we add a marker type to indicate the chunk marker?
> - Normally, the markers like Transaction markers are not visible to
>  the client, but we need to send the chunk marker to client.
> - What's the format of the chunk marker?
> - Which compatibility problems would be brought by this design?
> 
> I think we need a new proposal to explain it in details and I'm
> working on that, as well as the demo.
> 
> Feel free to ping me if you have any concern.
> 
> Thanks,
> Yunze
> 
> 
> 
>