You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/03/24 14:00:42 UTC
[GitHub] [rocketmq] HScarb opened a new issue #4033: [ScheduleMessage] Async deliver message will resend forever when service not avaliable
HScarb opened a new issue #4033:
URL: https://github.com/apache/rocketmq/issues/4033
**BUG REPORT**
1. Please describe the issue you observed:
- What did you do (The steps to reproduce)?
Send schedule messages to dledger cluster and cause message resend.
Then change master to slave.
- What is expected to see?
Message stop resending
- What did you see instead?
Message still resending while the store is available
2. Please tell us about your environment:
4.9.3 DLedger cluster
3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] Git-Yang commented on issue #4033: [ScheduleMessage] Async deliver message will resend forever when service not avaliable
Posted by GitBox <gi...@apache.org>.
Git-Yang commented on issue #4033:
URL: https://github.com/apache/rocketmq/issues/4033#issuecomment-1080307017
> @HScarb Skip the message will cause message loss.
>
> The root cause of this problem is the infinite recursion of handleResult().
>
> PutResultProcess.thenProcess is like the below:
>
> ```
> this.future.thenAccept(result -> {
> this.handleResult(result);
> });
> ```
>
> The core problem is when to call the "thenAccept ".
>
> If the "thenAccept" is called by the background thread(for example, in dledger mode, by the QuorumAckChecker) and get an error, the handleResult will resend the message and get a new future.
>
> As the future needs to be completed one by one. At this time, the old future will depend on the new future to complete, and the new future depend on the old future too.
>
> It will cause a deadlock.
>
> @Git-Yang @duhengforever Please help a check.
>
> The suggested solution is to use a new thread to execute the callback instead of in the "ack thread"(GroupCommitService or QuorumAckChecker).
The handleResult will resend the message, bug not get a new future.
I think, to avoid infinite recursion, the retry logic is no longer executed in onException() , but in HandlePutResultTask .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] Git-Yang edited a comment on issue #4033: [ScheduleMessage] Async deliver message will resend forever when service not avaliable
Posted by GitBox <gi...@apache.org>.
Git-Yang edited a comment on issue #4033:
URL: https://github.com/apache/rocketmq/issues/4033#issuecomment-1080307017
> @HScarb Skip the message will cause message loss.
>
> The root cause of this problem is the infinite recursion of handleResult().
>
> PutResultProcess.thenProcess is like the below:
>
> ```
> this.future.thenAccept(result -> {
> this.handleResult(result);
> });
> ```
>
> The core problem is when to call the "thenAccept ".
>
> If the "thenAccept" is called by the background thread(for example, in dledger mode, by the QuorumAckChecker) and get an error, the handleResult will resend the message and get a new future.
>
> As the future needs to be completed one by one. At this time, the old future will depend on the new future to complete, and the new future depend on the old future too.
>
> It will cause a deadlock.
>
> @Git-Yang @duhengforever Please help a check.
>
> The suggested solution is to use a new thread to execute the callback instead of in the "ack thread"(GroupCommitService or QuorumAckChecker).
The handleResult will resend the message, bug not get a new future. Resending is via putMessage(), so Futrues are not infinitely nested.
I think, to avoid infinite recursion, the retry logic is no longer executed in onException() , but in HandlePutResultTask .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] dongeforever edited a comment on issue #4033: [ScheduleMessage] Async deliver message will resend forever when service not avaliable
Posted by GitBox <gi...@apache.org>.
dongeforever edited a comment on issue #4033:
URL: https://github.com/apache/rocketmq/issues/4033#issuecomment-1080286798
@HScarb Skip the message will cause message loss.
The root cause of this problem is the infinite recursion of handleResult().
PutResultProcess.thenProcess is like the below:
```
this.future.thenAccept(result -> {
this.handleResult(result);
});
```
The core problem is when to call the "thenAccept ".
If the "thenAccept" is called by the background thread(for example, in dledger mode, by the QuorumAckChecker) and get an error, the handleResult will resend the message and get a new future.
As the future needs to be completed one by one.
At this time, the old future will depend on the new future to complete, and the new future depend on the old future too.
It will cause a deadlock.
@Git-Yang @duhengforever Please help a check.
The suggested solution is to use a new thread to execute the callback instead of in the "ack thread"(GroupCommitService or QuorumAckChecker).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] Git-Yang removed a comment on issue #4033: [ScheduleMessage] Async deliver message will resend forever when service not avaliable
Posted by GitBox <gi...@apache.org>.
Git-Yang removed a comment on issue #4033:
URL: https://github.com/apache/rocketmq/issues/4033#issuecomment-1080306792
>
The handleResult will resend the message, bug not get a new future.
I think, to avoid infinite recursion, the retry logic is no longer executed in onException() , but in HandlePutResultTask .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] dongeforever commented on issue #4033: [ScheduleMessage] Async deliver message will resend forever when service not avaliable
Posted by GitBox <gi...@apache.org>.
dongeforever commented on issue #4033:
URL: https://github.com/apache/rocketmq/issues/4033#issuecomment-1080316004
Yeah, Futures are not infinitely nested. The putMessage will create a future internally, and the newly created future will timeout and resend again.
It seems ok to do the retry in HandlePutResultTask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] Git-Yang commented on issue #4033: [ScheduleMessage] Async deliver message will resend forever when service not avaliable
Posted by GitBox <gi...@apache.org>.
Git-Yang commented on issue #4033:
URL: https://github.com/apache/rocketmq/issues/4033#issuecomment-1080306792
>
The handleResult will resend the message, bug not get a new future.
I think, to avoid infinite recursion, the retry logic is no longer executed in onException() , but in HandlePutResultTask .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] dongeforever commented on issue #4033: [ScheduleMessage] Async deliver message will resend forever when service not avaliable
Posted by GitBox <gi...@apache.org>.
dongeforever commented on issue #4033:
URL: https://github.com/apache/rocketmq/issues/4033#issuecomment-1080286798
@HScarb Skip the message will cause message loss.
The root cause of this problem is the infinite recursion of handleResult().
PutResultProcess.thenProcess is like the below:
```
this.future.thenAccept(result -> {
this.handleResult(result);
});
```
The core problem is when to call the "thenAccept ".
If the "thenAccepth" is called by the background thread(for example, in dledger mode, by the QuorumAckChecker) and get an error, the handleResult will resend the message and get a new future.
As the future needs to be completed one by one.
At this time, the old future will depend on the new future to complete, and the new future depend on the old future too.
It will cause a deadlock.
@Git-Yang @duhengforever Please help a check.
The suggested solution is to use a new thread to execute the callback instead of in the "ack thread"(GroupCommitService or QuorumAckChecker).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org