You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/07 11:57:04 UTC

[GitHub] [pulsar] gaozhangmin opened a new issue #13170: Add messages produced by Protocol Handler When checking maxMessagePublishBufferSizeInMB

gaozhangmin opened a new issue #13170:
URL: https://github.com/apache/pulsar/issues/13170


   
   ## Motivation
   
   MaxMessagePublishBufferSizeInMB is used to limit the direct memory usage for reading messages from producers, default value is 1/2 of direct memory. 
   The problem is:
   If we enable any Protocol Handler in broker such as KoP, the direct memory to buffer the pending messages from Kafka producers is not limited by maxMessagePublishBufferSizeInMB. And if we have Pulsar producers and Kafka producers sending message together with higher throughput than BookKeeper IO, broker OOM will happen.
   
   ## Goal
   
   When handling producer message, we could check if pending bytes from pulsar and protocol handler exceed maxMessagePublishBufferSizeInMB, if yes,  stop reading from producer both in broker and Protocol Handler side.
   
   ## API Changes
   using a LongAdder in `BrokerService` to record the current pending bytes for entire broker
   
   ## Implementation
   
   To make effective use of memory, using a LongAdder in `BrokerService` to record the current pending bytes for entire broker, include both pulsar pending messages and Protocol Handler's pending messages.
   
   
   ## Reject Alternatives
   
   No alternatives yet.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #13170: [PIP-111] Add messages produced by Protocol Handler When checking maxMessagePublishBufferSizeInMB

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13170:
URL: https://github.com/apache/pulsar/issues/13170#issuecomment-1051439857


   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaozhangmin closed issue #13170: [PIP-111] Add messages produced by Protocol Handler When checking maxMessagePublishBufferSizeInMB

Posted by GitBox <gi...@apache.org>.
gaozhangmin closed issue #13170:
URL: https://github.com/apache/pulsar/issues/13170


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org