You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "davide-armand (via GitHub)" <gi...@apache.org> on 2024/12/11 16:09:59 UTC
[PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
davide-armand opened a new pull request, #18142:
URL: https://github.com/apache/kafka/pull/18142
The call to `backingStore.get()` (called by connector task threads through `OffsetStorageReaderImpl.offsets()`) can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
This change moves that call outside the synchronized clause that holds `offsetReadFutures`, so that if `backingStore.get()` hangs then it does not keep `offsetReadFutures` locked. The access to `closed` flag (`closed.get()`) is kept inside the synchronize clause to avoid race condition with `close()`.
This is important because `OffsetStorageReaderImpl.close()` needs to lock `offsetReadFutures` as well in order to cancel the futures.
Since the herder thread calls `OffsetStorageReaderImpl.close()` when attempting to stops a task, before this change this was resulting in the herder thread hanging indefinetely waiting for `backingStore.get()` to complete.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "viktorsomogyi (via GitHub)" <gi...@apache.org>.
viktorsomogyi commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-4555982010
@davide-armand thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2541110118
The CI is now green.
The additional commit that fixed the CI should be squashed before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "viktorsomogyi (via GitHub)" <gi...@apache.org>.
viktorsomogyi merged PR #18142:
URL: https://github.com/apache/kafka/pull/18142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2829817385
Rebase on latest `apache:trunk`.
Compilation is now failing, seems unrelated to the changes in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2735241489
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-3116295166
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #18142: KAFKA-17871: avoid blocking the herder thread when producer flushing hangs
URL: https://github.com/apache/kafka/pull/18142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "jlprat (via GitHub)" <gi...@apache.org>.
jlprat commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2539607705
Hi @davide-armand
CI is complaining about check style violations. Could you take a look at it?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on code in PR #18142:
URL: https://github.com/apache/kafka/pull/18142#discussion_r3287525601
##########
connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageReaderImpl.java:
##########
@@ -86,14 +86,18 @@ public <T> Map<Map<String, T>, Map<String, Object>> offsets(Collection<Map<Strin
Map<ByteBuffer, ByteBuffer> raw;
try {
Future<Map<ByteBuffer, ByteBuffer>> offsetReadFuture;
+
+ // Note: this call can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
+ offsetReadFuture = backingStore.get(serializedToOriginal.keySet());
Review Comment:
@viktorsomogyi I forced pushed the suggested change, I agree it makes sense to fail fast here.
I also pushed a separate commit to refactor `OffsetStorageReaderImpl.offsets()` into smaller methods to fix a checkstyle complexity error in the build caused by this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-4296446043
This PR is still relevant.
I rebased it on latest trunk and got a green build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2742145207
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "viktorsomogyi (via GitHub)" <gi...@apache.org>.
viktorsomogyi commented on code in PR #18142:
URL: https://github.com/apache/kafka/pull/18142#discussion_r3249357041
##########
connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageReaderImpl.java:
##########
@@ -86,14 +86,18 @@ public <T> Map<Map<String, T>, Map<String, Object>> offsets(Collection<Map<Strin
Map<ByteBuffer, ByteBuffer> raw;
try {
Future<Map<ByteBuffer, ByteBuffer>> offsetReadFuture;
+
+ // Note: this call can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
+ offsetReadFuture = backingStore.get(serializedToOriginal.keySet());
Review Comment:
(Ofc with the error message moved to a constant to avoid duplication)
##########
connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageReaderImpl.java:
##########
@@ -86,14 +86,18 @@ public <T> Map<Map<String, T>, Map<String, Object>> offsets(Collection<Map<Strin
Map<ByteBuffer, ByteBuffer> raw;
try {
Future<Map<ByteBuffer, ByteBuffer>> offsetReadFuture;
+
+ // Note: this call can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
+ offsetReadFuture = backingStore.get(serializedToOriginal.keySet());
Review Comment:
I'm wondering if we need a behavior change here, afterall the point would be to fix the race condition, but with this change we also call `backingStore.get()` regardless of the value of `closed` which may impose some extra work. Do you think it would make sense to duplicate the if condition below to fail fast?
```suggestion
if (closed.get()) {
throw new ConnectException(
"Offset reader is closed. This is likely because the task has already been "
+ "scheduled to stop but has taken longer than the graceful shutdown "
+ "period to do so.");
}
// Note: this call can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
offsetReadFuture = backingStore.get(serializedToOriginal.keySet());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-3218734480
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2771640033
> Hi @davide-armand Thanks for the fix, and thanks for the test!
>
> Are you able to remove the Thread.sleep calls from the test? These could introduce some flakiness later if run on slow machines. Additionally I think this test leaks threads through the executors. Could you ensure that the executors are closed, and the threads terminate?
@gharris1727 Pushed a fix in a separate commit, will squash before merging.
I cannot get a green build, if I'm not mistaken it fails on unrelated tests.
Should I keep restarting the CI hoping for a green build?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2552708528
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-4762468687
Hi @davide-armand @viktorsomogyi , would you mind reviewing this PR https://github.com/apache/kafka/pull/22632 when you have a chance? This test has been showing up as flaky fairly often recently, so it would be great to get your input. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org