You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "bxfjb (via GitHub)" <gi...@apache.org> on 2024/01/29 04:04:18 UTC
[I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
bxfjb opened a new issue, #7790:
URL: https://github.com/apache/rocketmq/issues/7790
### Before Creating the Bug Report
- [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).
- [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.
- [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.
### Runtime platform environment
CentOS Linux release 7.3.1611
### RocketMQ version
4.8.0
### JDK Version
1.8
### Describe the Bug
In dledger mode, master broker which was shut down and reboot would not synchornize consume offset with the previous temporary master broker. The problem described may caused repeatedly consumption.
After reading code, one bug may exist in the synchornize logic:
BrokerController.java:

The oneway flag is set true, which means that following code in BrokerOuterAPI@registerBrokerAll will return null without waiting for the result:

which caused the following marked code didn't run actually:

Masteraddr which is null leads to a result that SlaveSynchronize@syncAll didnt't run either, including function body of SlaveSynchronize@syncConsumerOffset:

The Synchronization task was set to run every 10 seconds , but before next sync happens, the reboot broker was elected to be master again already, which means it stopped to try to get the newest consume offset, but start to push messsages from the offset before.

### Steps to Reproduce
* Run a cluster in dledger mode.
* Shut down master broker gracefully.
* Reboot previous master broker, be sure that consume offset increases during downtime.
* Repeat consumption should be observed.
### What Did You Expect to See?
Rebooted broker should synchornize the newest consume offset and consume from it correctly.
### What Did You See Instead?
Abnormal repeated consumption.
### Additional Context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7790: [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly
URL: https://github.com/apache/rocketmq/issues/7790
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-2811283221
This issue was closed because it has been inactive for 3 days since being marked as stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "cserwen (via GitHub)" <gi...@apache.org>.
cserwen commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-1972403289
Can you submit a pr to fix it @bxfjb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "lsy1990 (via GitHub)" <gi...@apache.org>.
lsy1990 commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-2051002607
when the master switches,We also encountered the same problem of repeated consumption,Is there other method to avoid this issue before patch become merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "bxfjb (via GitHub)" <gi...@apache.org>.
bxfjb commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-2051056211
> when the master switches,We also encountered the same problem of repeated consumption,Is there other method to avoid this issue before patch become merged
Nothing usable yet, you may merge the patch to your personal repository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "lsy1990 (via GitHub)" <gi...@apache.org>.
lsy1990 commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-2051003286
when the master switches,We also encountered the same problem of repeated consumption,Is there other method to avoid this issue before patch become merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "bxfjb (via GitHub)" <gi...@apache.org>.
bxfjb commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-2014291125
> Since the consumption progress is synchronized every 10 seconds, if this mechanism does not change, in theory, repeated consumption will occur as long as a master-slave switch occurs. A synchronization mechanism may be needed to ensure that the consumption progress is synchronized.
Actually the consumption synchronization period is modified to 3s in latest code, so that the repeated consumption you mentioned should decrease a lot. To reduce the remaining repeat, #7901 now explicitly call syncAll after register broker infomation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "bxfjb (via GitHub)" <gi...@apache.org>.
bxfjb commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-1990997354
> Can you submit a pr to fix it @bxfjb
Here is the pr, writing unittest is more complicate than I thought :)
https://github.com/apache/rocketmq/pull/7901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "LittleBoy18 (via GitHub)" <gi...@apache.org>.
LittleBoy18 commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-2014201450
Since the consumption progress is synchronized every 10 seconds, if this mechanism does not change, in theory, repeated consumption will occur as long as a master-slave switch occurs. A synchronization mechanism may be needed to ensure that the consumption progress is synchronized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "lsy1990 (via GitHub)" <gi...@apache.org>.
lsy1990 commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-2050958091
when the master switches,We also encountered the same problem of repeated consumption,Is there other method to avoid this issue before patch become merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Master broker in Dledger mode didn't sync consume offset in time after reboot, which caused consume repeatedly [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7790:
URL: https://github.com/apache/rocketmq/issues/7790#issuecomment-2799325346
This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org