[jira] [Created] (ROCKETMQ-329) Some messages was skipped by consumerOffset when rocketMQ consuming a large number of messages.

["sypeng (JIRA)" <ji...@apache.org>]

sypeng created ROCKETMQ-329:
-------------------------------

             Summary: Some messages was skipped by consumerOffset when rocketMQ consuming a large number of messages.
                 Key: ROCKETMQ-329
                 URL: https://issues.apache.org/jira/browse/ROCKETMQ-329
             Project: Apache RocketMQ
          Issue Type: Bug
          Components: rocketmq-broker, rocketmq-client
    Affects Versions: 4.1.0-incubating
         Environment: centos 7
one mqnamesvr, one mqbroker, with default settings.
            Reporter: sypeng
            Assignee: yukon
         Attachments: 0A3A66281DB342EB89871F8DE01865DB.jpg, 343007284451C411C424FABCC52F352B.jpg, 85C2383276ECDDD9C573913186B746C8.jpg

I started one mqnamesrv and one mqbroker process on centos7 with default settings:
{{nohup bin/mqname &
nohup bin/mqbroker -n localhost:9876 &}}
and produced a large number of messages with 5 threads continuously(total 1000000 messages)，found that each consumerOffset skipped about 4500 messages when persist (as shown in the attached picture), which cause them lost while consumer consuming.  The consumer lost about 16000 messages eventually.

The speed of message production is about 5000 per second, and the speed of message consumption is 5 per second (the consumer's logic is sleep 200 millseconds and return consume success.) 

The info "the message queue not be able to consume, because it's dropped" and the warning "the pull request offset illegal" are printed in the mq's log when this condition happened. (as shown in the attached picture)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)