[GitHub] [pulsar] keyboardbobo opened a new issue #14943: maxBatchSize preallocated memory may be thousands of times larger than actual message length

[GitBox <gi...@apache.org>]

keyboardbobo opened a new issue #14943:
URL: https://github.com/apache/pulsar/issues/14943


   **Is your enhancement request related to a problem? Please describe.**
   Guys, during the stress test, we found that the client's memory is very large, and even Full gc appeared. After analyzing the dump file, we found that the memory space occupied is much larger than the actual message, and the serialization of 1KB of messages takes up 1MB.
   ```
   batchedMessageMetadataAndPayload = PulsarByteBufAllocator.DEFAULT
                           .buffer(Math.min(maxBatchSize, ClientCnx.getMaxMessageSize()));
   ```
   
   `maxBatchSize = Math.max(maxBatchSize, uncompressedSize);`
   
   Debugging found that the maxbatchsize that controls the pre-allocated `ByteBufPair.b2` memory size is stateful. As the size of the largest batch or the largest single message grows, this may cause the pre-allocated `ByteBufPair.b2` memory to grow larger and larger. , which is thousands of times larger than the payload of MessageImpl
   
   **Describe the solution you'd like**
   It is best to loop through the messages to be packed to accurately calculate the memory size to be allocated
   
   **Describe alternatives you've considered**
   Let the user choose whether to precisely allocate or pre-allocate
   
   **Additional context**
   Add any other context or screenshots about the feature request here.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] keyboardbobo commented on issue #14943: maxBatchSize preallocated memory may be thousands of times larger than actual message length

[GitBox <gi...@apache.org>]

keyboardbobo commented on issue #14943:
URL: https://github.com/apache/pulsar/issues/14943#issuecomment-1084150493


   > It is best to loop through the messages to be packed to accurately calculate the memory size to be allocated
   
   Whether it is possible to loop through the messages to be packaged and add their sizes to accurately calculate the size of the memory to be allocated？


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaozhangmin commented on issue #14943: maxBatchSize preallocated memory may be thousands of times larger than actual message length

[GitBox <gi...@apache.org>]

gaozhangmin commented on issue #14943:
URL: https://github.com/apache/pulsar/issues/14943#issuecomment-1084027950


   I have a doubt, How to achieve `precisely allocate`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaozhangmin edited a comment on issue #14943: maxBatchSize preallocated memory may be thousands of times larger than actual message length

[GitBox <gi...@apache.org>]

gaozhangmin edited a comment on issue #14943:
URL: https://github.com/apache/pulsar/issues/14943#issuecomment-1084027950


   I have a doubt, How to achieve `precisely allocate`? @keyboardbobo


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org