You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "urbandan (via GitHub)" <gi...@apache.org> on 2023/06/01 14:41:04 UTC
[GitHub] [kafka] urbandan opened a new pull request, #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
urbandan opened a new pull request, #13796:
URL: https://github.com/apache/kafka/pull/13796
…atches to complete before resetting the sequence number
The idempotent producer resets the sequence number when a batch fails with a non-retriable error. This can violate the idempotent guarantee if there are preceding, in-flight requests which are being retried, as their producer epoch and sequence number gets rewritten, effectively granting those messages a new "identity".
Instead, the producer should wait for the preceding, retried batches to complete before resetting the sequence number. This ensures that the sequence numbers can only get reset after the preceding batches are definitely completed.
By calling markSequenceUnresolved instead of requestEpochBumpForPartition, draining batches from the partition is halted, and the epoch bump is delayed until all in-flight requests are completed.
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1573959019
> Instead, the producer should wait for the preceding, retried batches to complete before resetting the sequence number. This ensures that the sequence numbers can only get reset after the preceding batches are definitely completed.
Can we guarantee that the retrying batches will complete?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14034 Idempotent producer should wait for preceding in-flight b… [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-2567257750
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] viktorsomogyi commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "viktorsomogyi (via GitHub)" <gi...@apache.org>.
viktorsomogyi commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1578650158
I had a short chat with @urbandan yesterday to understand the scope of this fix. During the conversation we came to the conclusion that bumping the epoch without a safety check is generally unsafe as there might be requests in the queue that are timed out with request timeout yet successfully appended on the broker and still wait in the queue in the producer for a retry while their epoch is being bumped.
While this PR fixes once case, It would be good to review all usages where the epoch is being bumped. I don't insist on redoing all the other cases, so let me know Daniel if you want to expand the scope of this or do it in a follow up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14034 Idempotent producer should wait for preceding in-flight b… [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
URL: https://github.com/apache/kafka/pull/13796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] urbandan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "urbandan (via GitHub)" <gi...@apache.org>.
urbandan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1573381258
@philipnee @ijuma @hachikuji @jolshan Could you please review this fix? Based on the git history, you made changes lately in the related code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1593699560
I suppose my general my concern is along that line. If we are in a state that is non-retriable it seems like it would be more likely to not have a way to recover. That's just my general feeling though. I guess I want to confirm that this change will be better (fix more edge cases) than what we have now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] viktorsomogyi commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "viktorsomogyi (via GitHub)" <gi...@apache.org>.
viktorsomogyi commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1736950716
@urbandan do you want to continue this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] urbandan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "urbandan (via GitHub)" <gi...@apache.org>.
urbandan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1590766255
Thank you for the info. I'd say that in isolation, this specific issue (KAFKA-14034) should be resolved by retrying the in-flight batches.
Are you suggesting that until KAFKA-14359 is fixed, we shouldn't try to fix KAFKA-14034, because we increase the risk of encountering KAFKA-14359?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1590186665
With respect to the retries. There is currently a known error where if we produce using an idempotent producer we can get stuck in a retry loop.
1. Request A with sequence 0 fails with a retriable error
2. Request B with sequence 1 succeeds. Because we currently don't assert the sequence is 0 on the first records written (or any records written when we don't yet have producer state, this will succeed)
3. Request A will be retried and will always fail with a retriable OutOfOrderSequence error.
Imagine at step 2, we have a request C that fails with a non-retriable error. Will we get stuck waiting for Request A until the retries run out? The default is pretty high so this will go on for some time.
BTW this issue is https://issues.apache.org/jira/browse/KAFKA-14359, and we are still considering the best way to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "vvcephei (via GitHub)" <gi...@apache.org>.
vvcephei commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1578989020
Just checking to see if the github/jenkins trigger still works. It looks like it does not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14034 Idempotent producer should wait for preceding in-flight b… [kafka]
Posted by "urbandan (via GitHub)" <gi...@apache.org>.
urbandan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1742583164
@viktorsomogyi I would like to, yes, but my understanding is (based on the comments of @jolshan) that another loosely related bug blocks this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14034 Idempotent producer should wait for preceding in-flight b… [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-2629226267
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] urbandan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "urbandan (via GitHub)" <gi...@apache.org>.
urbandan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1572188562
@viktorsomogyi Could you please review this fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] urbandan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "urbandan (via GitHub)" <gi...@apache.org>.
urbandan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1580030486
@viktorsomogyi Since this part of the code is quite tricky, I would try to address the different issues in different PRs. I believe that the fix I'm proposing will solve the issue reported in KAFKA-14034 specifically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] urbandan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "urbandan (via GitHub)" <gi...@apache.org>.
urbandan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1576211511
> > Instead, the producer should wait for the preceding, retried batches to complete before resetting the sequence number. This ensures that the sequence numbers can only get reset after the preceding batches are definitely completed.
>
> Can we guarantee that the retrying batches will complete?
No, we can't - it all depends on the retries and the delivery timeout.
Do you think that causes a problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1581669973
> No, we can't - it all depends on the retries and the delivery timeout.
Do you think that causes a problem?
I guess I just need to clarify what retried batches are here -- is the idea that we wait for inflight batches to return a response or time out? What if the response triggers another retry? Would we prevent that from sending out?
I'm also wondering the benefit of preserving the previous batches if there is an error. How does the system recovery differently if we allow those batches to "complete". I think we could run into cases where the error causes the inflight batches to be unable to be written. Do we prefer to fail them (what we may do now) and start clean or try to write them with new sequences? I can see both scenarios causing issues.
I guess it boils down to availability of writes (rewriting the sequences allows us to continue writing) or idempotency correctness (trying to wait for them to complete with their old sequences). The sticking point I'm running into is why getting those extra inflight requests (potentially) written is better if we've hit a non-retriable error.
Maybe I just need an example :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] urbandan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "urbandan (via GitHub)" <gi...@apache.org>.
urbandan commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1582188829
> I guess I just need to clarify what retried batches are here -- is the idea that we wait for inflight batches to return a response or time out? What if the response triggers another retry? Would we prevent that from sending out?
The core idea is that we let each of the in-flight batches complete, even if they need multiple retries. This would allow the producer to
1. Avoid inconsistency - by letting in-flight batches finish, we do not run the risk of overwriting their sequence number while we are still not sure if they were appended or not.
2. Operate with best-effort - when using an idempotent producer, and encountering an error, it is costly to verify if a message was appended to the log or not (I think the "official" suggestion is to consume the topic to verify). By letting the in-flight batches finish, the idempotent producer will report fewer false positive errors.
> I'm also wondering the benefit of preserving the previous batches if there is an error. How does the system recover differently if we allow those batches to "complete". I think we could run into cases where the error causes the inflight batches to be unable to be written. Do we prefer to fail them (what we may do with this change) and start clean or try to write them with new sequences? I can see both scenarios causing issues.
I believe that produce errors should be handled separately, and should not cascade to other batches. I think most errors do not really predict the result of other produce requests.
> I guess it boils down to availability of writes (rewriting the sequences allows us to continue writing) or idempotency correctness (trying to wait for them to complete with their old sequences). The sticking point I'm running into is why getting those extra inflight requests (potentially) written is better if we've hit a non-retriable error.
My understanding is that here correctness beats availability. Are you suggesting that we should just cancel in-flight batches when encountering an error?
> Maybe I just need an example :)
I will try to write up some examples, and also write more unit tests to demonstrate those scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "vvcephei (via GitHub)" <gi...@apache.org>.
vvcephei commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1578988387
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] github-actions[bot] commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13796:
URL: https://github.com/apache/kafka/pull/13796#issuecomment-1720434737
This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch) <p> If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org