You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2021/12/13 02:51:26 UTC

[GitHub] [rocketmq] Erik1288 opened a new issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Erik1288 opened a new issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634


   The issue tracker is **ONLY** used for bug report(feature request need to follow [RIP process](https://github.com/apache/rocketmq/wiki/RocketMQ-Improvement-Proposal)). Keep in mind, please check whether there is an existing same report before your raise a new one.
   
   Alternately (especially if your communication is not a bug report), you can send mail to our [mailing lists](http://rocketmq.apache.org/about/contact/). We welcome any friendly suggestions, bug fixes, collaboration and other improvements.
   
   Please ensure that your bug report is clear and that it is complete. Otherwise, we may be unable to understand it or to reproduce it, either of which would prevent us from fixing the bug. We strongly recommend the report(bug report or feature request) could include some hints as the following:
   
   **FEATURE REQUEST**
   
   1. Please describe the feature you are requesting.
   * Introducing BatchConsumeQueue index to improve throughput for BatchMessage.
   * Refactor consume queue interface.
   
   2. Provide any additional detail on your proposed use case for this feature.
   * When Batch Consume Queue is introduced, each BatchMessage is no longer treated as a collection of sub-messages, but as an independent message.
           * When receiving a BatchMessage, the engine treats a BatchMessage as a normal message.
           * When indexing a BatchMessage，the engine will construct one index item for a single batch.
           * When consuming a BatchMessage，the engine will access index only once for a single batch.
           * End-to-end compression/uncompression will be supported on BatchMessage on the fly.
   
   3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   It's 'nice-to-have'.
   No.
   
   4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:
   
   * Implement client-side and server-side batch consume queue in broker.
   * Implement historic consume queue and batch consume queue converter.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] Zanglei06 commented on issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Posted by GitBox <gi...@apache.org>.
Zanglei06 commented on issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634#issuecomment-994742687


   > 
   "When consuming a BatchMessage，the engine will access index only once for a single batch."
   Does it means that the new batch msg can not be consumed by current version of consumers? thus causing compatibility problems.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] duhenglucky closed issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Posted by GitBox <gi...@apache.org>.
duhenglucky closed issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] RongtongJin commented on issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Posted by GitBox <gi...@apache.org>.
RongtongJin commented on issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634#issuecomment-998855116


   refer https://github.com/apache/rocketmq/wiki/RIP-26-Improve-Batch-Message-Processing-Throughput


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] duhenglucky commented on issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Posted by GitBox <gi...@apache.org>.
duhenglucky commented on issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634#issuecomment-1078569395


   Merged


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] guyinyou commented on issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Posted by GitBox <gi...@apache.org>.
guyinyou commented on issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634#issuecomment-992288008


   I think it uses the new BatchConsumeQueue to index single messages and batch messages. Since batch messages are no longer parsed as a single message, when indexing a single message, the client should pull down and parse the batch message to single message.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] HScarb commented on issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Posted by GitBox <gi...@apache.org>.
HScarb commented on issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634#issuecomment-992280609


   You mean separate `BatchConsumeQueue` from `ConsumeQueue` and consumer will be able to consume from both `BatchConsumeQueue` and `ConsumeQueue`? Or just extend `ConsumeQueue` with a new `BatchConsumeQueue` which support batch message consuming and single message consumeing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] Zanglei06 commented on issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Posted by GitBox <gi...@apache.org>.
Zanglei06 commented on issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634#issuecomment-1000633397


   I read the Compatibility description, "old client+new broker: old clients won't make request with inner-batch flag, so broker will use old outer-batch manner."
   
   If I use new producer client and send inner-batch msgs, will old consumer client consume the msgs ? It seems the answer is NO. @RongtongJin @Erik1288 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] dongeforever commented on issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Posted by GitBox <gi...@apache.org>.
dongeforever commented on issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634#issuecomment-1022792299


   The broker side is backward compatible, but the client side is not backward compatible currently.
   Not only batch, other features such as Pop, has a similar problem too.
   Currently, the client uses the broker version to control the SQL filter feature. 
   We may need to extend this mechanism to solve this problem.
   @Erik1288 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] guyinyou commented on issue #3634: [RIP-26] Improve Batch Message Processing Throughput

Posted by GitBox <gi...@apache.org>.
guyinyou commented on issue #3634:
URL: https://github.com/apache/rocketmq/issues/3634#issuecomment-992158125


   Very much looking forward to


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org