[GitHub] [rocketmq-client-go] j2gg0s opened a new issue, #941: PushConsumer's default config easy to casue recreate `processQueue`

[GitBox <gi...@apache.org>]

j2gg0s opened a new issue, #941:
URL: https://github.com/apache/rocketmq-client-go/issues/941

   Under the following conditions, our consumer will always recreate `processQueue`.
   - default config
   - pushConsumer with no orderly consume
   - produce tps greater than consume tps
   - topic with 64 queues
   - consume 1 message took more than 1 second.
   
   The related warning log is: `delete mq from offset table` and `fetch offset of mq from broker success`.
   
   The reasons is:
   - We pull message for every (topic, queue) with  batch size controlled by `PullBatchSize`, default is 32.
   - Each batch of messages will send to `processQueue.msgCh`, which's buffer size controlled is `32`.
   - `pushConsumer.consumeMessageCurrently` send n messages to consumer's handler, n is controlled by `ConsumeMessageBatchMaxSize`, default is 1.
   - `pushConsumer.crCh` limit a maximum of `ConsumeGoroutineNums` goroutines to process messages, `ConsumeGoroutineNums` default is 20.
   -  Assume, consumer took d seconds to process `ConsumeMessageBatchMaxSize(1)` message.
   - _PullMaxIdleTime is 120 second
   So when topic has 64 queue, the pull interval is `d * 32 / ConsumeMessageBatchMaxSize(1) * 64 / ConsumeGoroutineNums`.
   To avoid processQueue be recreated, d should less than 1 second, this number is two small.
   
   
   Flow control only works in certain situations:
   - `processQueue` pull messages more than 1000 over 32 times with batch size 32, 32*32=1024
   
   Let we change default `PullBatchSize` to 64?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-client-go] j2gg0s closed issue #941: PushConsumer's default config easy to casue recreate `processQueue`

[GitBox <gi...@apache.org>]

j2gg0s closed issue #941: PushConsumer's default config easy to casue recreate `processQueue`
URL: https://github.com/apache/rocketmq-client-go/issues/941


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org