You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "thetumbled (via GitHub)" <gi...@apache.org> on 2023/02/07 12:50:13 UTC
[GitHub] [pulsar] thetumbled opened a new pull request, #19451: [fixbug] [broker] filter messages in pending ack state.
thetumbled opened a new pull request, #19451:
URL: https://github.com/apache/pulsar/pull/19451
### Motivation
Currently, broker do not filter messages in pending ack state when dispatching messages to client. If messages in pending ack state are delivered to client, `PulsarClientException$TransactionConflictException` will be throwed.
PR #14327 has try to fix one of cases that broker will dispatch messages in pending ack state.
But, there are other cases that will trigger this problem.
For example, if producer send non batch messages, msg1 and msg2. consumer use txn1 ack msg1, use txn2 ack msg2.
then consumer abort txn2, at this time, we should redeliver only msg2. Because msg1 is in pending ack state.
### Modifications
filter non batch messages in pending ack state.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] thetumbled closed pull request #19451: [fixbug] [broker] filter messages in pending ack state.
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled closed pull request #19451: [fixbug] [broker] filter messages in pending ack state.
URL: https://github.com/apache/pulsar/pull/19451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] thetumbled commented on pull request #19451: [fixbug] [broker] filter messages in pending ack state.
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #19451:
URL: https://github.com/apache/pulsar/pull/19451#issuecomment-1421888854
PTAL, thanks. @congbobo184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] thetumbled commented on pull request #19451: [fixbug] [broker] filter messages in pending ack state.
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #19451:
URL: https://github.com/apache/pulsar/pull/19451#issuecomment-1422114703
> There have some problem:
>
> 1. use Exclusive, why not use cumulative ack?
> 2. only check the postion, how batch message can be redeliver?
>
> I don't think it's necessary to fix it, at least for now
1. i find that only in `Exclusive` mode can i recur the problem, it may be have something to do with different dispatcher. In shared mode the test code can not recur the issue.
2. batch messages in pending ack state are filtered by your PR #14327.
3. According to my understanding, PR #14327 try to filter batch messages in pending ack state. for example, producer produce a batch message consisting of two messages msg1 and msg2. consumer use txn1 ack msg1, use txn2 ack msg2, then the consumer abort txn2, at this time, the bitset hold in PendingAckHandleImpl is 01, the bitset hold in cursor is 11. So when broker redeliver msgs to consumer triggered by transaction abortion, broker will send a batch message with bitset 11. so the consumer receive msg1 and msg2 again. but msg1 should be filtered.
4. and, i do not understand what you say in #14327, that is the risk of message lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on pull request #19451: [fixbug] [broker] filter messages in pending ack state.
Posted by "congbobo184 (via GitHub)" <gi...@apache.org>.
congbobo184 commented on PR #19451:
URL: https://github.com/apache/pulsar/pull/19451#issuecomment-1422156738
> 1. i find that only in `Exclusive` mode can i recur the problem, it may be have something to do with different dispatcher. In shared mode the test code can not recur the issue.
exclusive redeliver messages from the cursor markdelete
> 4. and, i do not understand what you say in [[feature][txn] Fix individual ack batch message with txn abort redevlier duplicate messages #14327](https://github.com/apache/pulsar/pull/14327), that is the risk of message lost.
message dispatch is in order, if messages can be filtered, consumers using cumulative ack with txn will lose the messages which have been filtered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] thetumbled commented on pull request #19451: [fixbug] [broker] filter messages in pending ack state.
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #19451:
URL: https://github.com/apache/pulsar/pull/19451#issuecomment-1443419758
the problem described in this pr have been fixed in #19581, so i close this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org