[GitHub] [pulsar] Jason918 opened a new issue #13544: [PIP 131] Resolve produce chunk messages failed when topic level maxMessageSize is set

[GitBox <gi...@apache.org>]

Jason918 opened a new issue #13544:
URL: https://github.com/apache/pulsar/issues/13544


   <!---
   Instructions for creating a PIP using this issue template:
   
    1. The author(s) of the proposal will create a GitHub issue ticket using this template.
       (Optionally, it can be helpful to send a note discussing the proposal to
       dev@pulsar.apache.org mailing list before submitting this GitHub issue. This discussion can
       help developers gauge interest in the proposed changes before formalizing the proposal.)
    2. The author(s) will send a note to the dev@pulsar.apache.org mailing list
       to start the discussion, using subject prefix `[PIP] xxx`. To determine the appropriate PIP
       number `xxx`, inspect the mailing list (https://lists.apache.org/list.html?dev@pulsar.apache.org)
       for the most recent PIP. Add 1 to that PIP's number to get your PIP's number.
    3. Based on the discussion and feedback, some changes might be applied by
       the author(s) to the text of the proposal.
    4. Once some consensus is reached, there will be a vote to formally approve
       the proposal. The vote will be held on the dev@pulsar.apache.org mailing list. Everyone
       is welcome to vote on the proposal, though it will considered to be binding
       only the vote of PMC members. It will be required to have a lazy majority of
       at least 3 binding +1s votes. The vote should stay open for at least 48 hours.
    5. When the vote is closed, if the outcome is positive, the state of the
       proposal is updated and the Pull Requests associated with this proposal can
       start to get merged into the master branch.
   
   -->
   
   ## Motivation
   
   Currently, chunk messages producing fails if topic level maxMessageSize is set [1]. The root cause of this issue is because chunk message is using broker level maxMessageSize as chunk size. And topic level maxMessageSize is always <= broker level maxMessageSize. So once it is set, the on-going chunk message producing fails.
   
   ## Goal
   
   Resolve topic level maxMessageSize compatibility issue with chunking messages.
   
   ## Implementation
   
   Current best solution would be just skipping topic level maxMessageSize check in org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize. 
   Topic level maxMessageSize is introduced in [2], for the purpose of "easier to plan resource quotas for client allocation". And IMO this change will not bring further complex into this.
   
   ## Reject Alternatives
   
   Add a client side topic level maxMessageSize and keep it synced with broker.
   
   Required changes:
   - [client] Add a new field org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this client side topic level maxMessageSize.
   - [PulsarApi.proto] Add a new field maxMessageSize in the CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
   - [PulsarApi.proto] Add a new Command like CommandUpdateClientPolicy{producerId, maxMessageSize} to update ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
   Further more, some other data consistency issues need be handled very carefully when maxMessageSize is updated.
   This alternative is complex but can also solve other topic level maxMessageSize issue [3] when batching is enabled (non-batching case is solved with PR [4]).
   
   [1] https://github.com/apache/pulsar/issues/13360
   [2] https://github.com/apache/pulsar/pull/8732
   [3] https://github.com/apache/pulsar/issues/12958
   [4] https://github.com/apache/pulsar/pull/13147


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui closed issue #13544: [PIP 131] Resolve produce chunk messages failed when topic level maxMessageSize is set

[GitBox <gi...@apache.org>]

codelipenghui closed issue #13544:
URL: https://github.com/apache/pulsar/issues/13544


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org