You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/08/30 11:33:36 UTC
[GitHub] [rocketmq] complone opened a new issue, #4933: Rocketmq producer side supports idempotency
complone opened a new issue, #4933:
URL: https://github.com/apache/rocketmq/issues/4933
In the current process of rocketmq producing messages, there is uncertainty in the network call itself, that is, the so-called processing state, so there will be repeated situations. This problem is also encountered by many other MQ products. The usual method is to require the consumer to deduplicate when consuming messages. The idempotent basis should be generated by the message producer. When sending a message, we can pass the key of the message. Set the id, the corresponding API is org.apache.rocketmq.common.message.setKeys(String keys) The logic of this part should be optimized by the producer
Please ensure that your bug report is clear and self-contained. Otherwise, it would take additional rounds of communication, thus more time, to understand the problem itself.
Generally, fixing an issue goes through the following steps:
1. Understand the issue reported;
1. Reproduce the unexpected behavior locally;
1. Perform root cause analysis to identify the underlying problem;
1. Create test cases to cover the identified problem;
1. Work out a solution to rectify the behavior and make the newly created test cases pass;
1. Make a pull request and go through peer review;
As a result, it would be very helpful yet challenging if you could provide an isolated project reproducing your reported issue. Anyway, please ensure your issue report is informative enough for the community to pick up. At a minimum, include the following hints:
**BUG REPORT**
1. Please describe the issue you observed:
- What did you do (The steps to reproduce)?
- What is expected to see?
- What did you see instead?
2. Please tell us about your environment:
3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):
**FEATURE REQUEST**
1. Please describe the feature you are requesting.
2. Provide any additional detail on your proposed use case for this feature.
2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
4. If there are some sub-tasks involved, use -[] for each sub-task and create a corresponding issue to map to the sub-task:
- [sub-task1-issue-number](example_sub_issue1_link_here): sub-task1 description here,
- [sub-task2-issue-number](example_sub_issue2_link_here): sub-task2 description here,
- ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] complone commented on issue #4933: [feature] Rocketmq producer side supports idempotency
Posted by GitBox <gi...@apache.org>.
complone commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1232505902
The details are still being sorted out and other Rocketmq issues are still in progress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] complone commented on issue #4933: [RIP-46] Rocketmq producer side supports idempotency
Posted by GitBox <gi...@apache.org>.
complone commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1264432091
see RIP-47 : https://docs.google.com/document/d/1nResLevPbeGmKwSQiId_jw0tfBhJtoPZTduOQL3qxNg/edit?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] complone commented on issue #4933: [feature] Rocketmq producer side supports idempotency
Posted by GitBox <gi...@apache.org>.
complone commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1246367514
Hello everyone, the problems we need to solve at this stage are:
In order to improve the accuracy of data and ensure the consistency between the source and the target in the end-to-end transmission process, it is necessary to do idempotent at the producer or consumer
However, if consumers cannot connect to the nameserver to obtain partition information temporarily due to network jitter, they need to implement idempotent mechanism in the producer
For this reason, our goal is to align the produerId and transactionId of kafka. There are three points to note at this stage by following the code
- First, before sending data, you need to manually create a transaction coordinator to control the one to many relationship between PID and transcationId. To this end, you need to ensure that the consumption progress of offset is notified to the tc in a timely manner and that the messages of submitted but unsettled transactions will not be deleted by mistake during the simultaneous process of the scheduled deletion task
- Whether it is necessary to judge the progress of the current offset and the global PID visible transaction offset in PullLiteProductor in order to enable PID ->transactionid to achieve cross partition transaction consistency
- When downstream consumers consume, they need to ensure that messages marked as transactionId in the upstream market are perceived by downstream consumers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] github-actions[bot] commented on issue #4933: [RIP-46] Rocketmq producer side supports idempotency
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1742248888
This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] zhouxinyu commented on issue #4933: [feature] Rocketmq producer side supports idempotency
Posted by GitBox <gi...@apache.org>.
zhouxinyu commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1232366424
> @duhenglucky please assign to me
It's a big feature, do you have any plan or proposal if you want to pick up this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] azhsmesos commented on issue #4933: [feature] Rocketmq producer side supports idempotency
Posted by GitBox <gi...@apache.org>.
azhsmesos commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1234954664
> @complone It's a great improvement proposal. hope to see your detailed design. we'd better discuss it in the community with [RIP](https://github.com/apache/rocketmq/wiki/RocketMQ-Improvement-Proposal) first, @ltamber and I are willing to join in and complete this feature together.
May I ask where is the corresponding RIP discussed? Can you give me a link
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] complone commented on issue #4933: [feature] Rocketmq producer side supports idempotency
Posted by GitBox <gi...@apache.org>.
complone commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1231564242
@duhenglucky please assign to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] complone commented on issue #4933: [feature] Rocketmq producer side supports idempotency
Posted by GitBox <gi...@apache.org>.
complone commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1240510642
> 阿
@azhsmesos I think I need to investigate the implementation mechanism of rocketmq transactions first. I am currently sorting out the related method call timings of kafka and rocketmq transactions. After sorting out, I will establish a RIP discussion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] duhenglucky commented on issue #4933: [feature] Rocketmq producer side supports idempotency
Posted by GitBox <gi...@apache.org>.
duhenglucky commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1232510025
@complone It's a great improvement proposal. hope to see your detailed design. we'd better discuss it in the community first, @ltamber and me are willing to join in and complete this feature together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] complone commented on issue #4933: [feature] Rocketmq producer side supports idempotency
Posted by GitBox <gi...@apache.org>.
complone commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1232517897
The real existence of this function I think I am referring to [kafka's inbox](https://www.archive.link/dig/detail/1652381596718757 ), etc. to send the rId that implements mq。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [RIP-46] Rocketmq producer side supports idempotency [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4933:
URL: https://github.com/apache/rocketmq/issues/4933#issuecomment-1749815601
This issue was closed because it has been inactive for 3 days since being marked as stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [RIP-46] Rocketmq producer side supports idempotency [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4933: [RIP-46] Rocketmq producer side supports idempotency
URL: https://github.com/apache/rocketmq/issues/4933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org