You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/13 01:36:28 UTC
[GitHub] [pulsar] liangyepianzhou opened a new pull request #14669: [Transaction] Fix send transaction messages not in order
liangyepianzhou opened a new pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669
### Motivation
If we use `future.thenCompose(internalSendAsync(message))`` to send transaction messages, We have no way to guarantee that the order will not change when run `internalSendAsync(message)`
### Modification
1. Add a `HashMap<Transaction, ArrayList<Pair<Message<?>, CompletableFuture<MessageId>>>> ` to store messages needed to send.
2. return messages in `HashMap<Transaction, ArrayList<Pair<Message<?>, CompletableFuture<MessageId>>>> `after registerProducedTopic
### Verifying this change
I have test it in a chaosTest, and the problem can be resolve in this way.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?
- [ ] `doc-required`
(If you need help on updating docs, create a doc issue)
- [ ] `no-need-doc`
(Please explain why)
- [ ] `doc`
(If this PR contains doc changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #14669: [Transaction] Fix send transaction messages not in order
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669#issuecomment-1066001476
@liangyepianzhou:Thanks for your contribution. For this PR, do we need to update docs?
(The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on pull request #14669: [Transaction] Fix send transaction messages not in order
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669#issuecomment-1066026048
> > If we use future.thenCompose(internalSendAsync(message))`` to send transaction messages, We have no way to guarantee that the order will not change when run internalSendAsync(message)`
>
This is exactly the logic of using transaction to send messages, the normal message is `internalSendAsync(message)`, and the `transaction` message adds a step `thenCompose(() -> internalSendAsync(message))` call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #14669: [Transaction] Fix send transaction messages not in order
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669#issuecomment-1066001674
@liangyepianzhou:Thanks for providing doc info!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on pull request #14669: [Transaction] Fix send transaction messages not in order
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669#issuecomment-1066247289
Could you add a test for this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on pull request #14669: [Transaction] Fix send transaction messages not in order
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669#issuecomment-1066428788
I've put the test code snippet that sends the transaction message into the description. Because only one message is sent per transaction. This is equivalent to the need to go to the broker to request addPublishPartitionToTxn every time. The response returned by the broker is not completely ordered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on pull request #14669: [Transaction] Fix send transaction messages not in order
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669#issuecomment-1066429593
And this is no need to fix, because users may no care this when they use a transaction to send a message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou closed pull request #14669: [Transaction] Fix send transaction messages not in order
Posted by GitBox <gi...@apache.org>.
liangyepianzhou closed pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #14669: [Transaction] Fix send transaction messages not in order
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669#issuecomment-1066017421
> If we use future.thenCompose(internalSendAsync(message))`` to send transaction messages, We have no way to guarantee that the order will not change when run internalSendAsync(message)`
It's the problem from the usage of Java Future API, not pulsar. Does the above code example is in the pulsar code base or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on pull request #14669: [Transaction] Fix send transaction messages not in order
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on pull request #14669:
URL: https://github.com/apache/pulsar/pull/14669#issuecomment-1066257094
> Could you add a test for this change?
This is a concurrent problem. In fact, it is no problem when I run the chaos test in a minutes, and the problem is happen when I run the chaos test in five minutes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org