You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/10/01 22:33:49 UTC

[GitHub] [pulsar] MrThreepwood opened a new issue #12262: Batch Receive Does not have Minimum Delivery of 1

MrThreepwood opened a new issue #12262:
URL: https://github.com/apache/pulsar/issues/12262


   **Describe the bug**
   When attempting to batch process messages from a topic, no messages are processed when a single message is larger than the max payload size.
   
   **To Reproduce**
   Steps to reproduce the behavior:
   Create a batch consumer with a batch policy that has a non zero max byte amount.
   
   **Expected behavior**
   Either a single messages is delivered or an error is thrown.
   
   **Desktop (please complete the following information):**
    - OS: linux server
   
   **Additional context**
   We're trying to find a way to safely consume multiple messages from a queue of items with highly variable size. Because the pulsar client does not allow us to determine information about the next message (or even if it exists), the best we can do is ask for the next X messages. While the batch receive looked ideal, as we could limit on number of messages and message size, we ran into an issue. If all workers attempted to grab 1MB of data across our 1kish subscriptions, we would run OOM. If we set the size smaller than that, we simply stop processing messages on some queues alltogether, with absolutely no errors/observability on why.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] MrThreepwood commented on issue #12262: Batch Receive Does not have Minimum Delivery of 1

Posted by GitBox <gi...@apache.org>.
MrThreepwood commented on issue #12262:
URL: https://github.com/apache/pulsar/issues/12262#issuecomment-1026177979


   Any thoughts/updates on this? It kinda makes it difficult/impossible to use the batch feature if your feed can just stop processing at any point with absolutely no feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] MrThreepwood commented on issue #12262: Batch Receive Does not have Minimum Delivery of 1

Posted by GitBox <gi...@apache.org>.
MrThreepwood commented on issue #12262:
URL: https://github.com/apache/pulsar/issues/12262#issuecomment-1032874243


   Thanks. Sorry I missed this earlier, this is very helpful!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui closed issue #12262: Batch Receive Does not have Minimum Delivery of 1

Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #12262:
URL: https://github.com/apache/pulsar/issues/12262


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui commented on issue #12262: Batch Receive Does not have Minimum Delivery of 1

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #12262:
URL: https://github.com/apache/pulsar/issues/12262#issuecomment-1031045887


   Nice catch!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Jason918 commented on issue #12262: Batch Receive Does not have Minimum Delivery of 1

Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #12262:
URL: https://github.com/apache/pulsar/issues/12262#issuecomment-1031031666


   @MrThreepwood Sorry for the late response. I have submitted a PR(#14139), please take a look. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org