You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "davide-armand (via GitHub)" <gi...@apache.org> on 2024/12/11 16:09:59 UTC

[PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

davide-armand opened a new pull request, #18142:
URL: https://github.com/apache/kafka/pull/18142

   The call to `backingStore.get()` (called by connector task threads through `OffsetStorageReaderImpl.offsets()`) can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
   
   This change moves that call outside the synchronized clause that holds `offsetReadFutures`, so that if `backingStore.get()` hangs then it does not keep `offsetReadFutures` locked. The access to `closed` flag (`closed.get()`) is kept inside the synchronize clause to avoid race condition with `close()`.
   
   This is important because `OffsetStorageReaderImpl.close()` needs to lock `offsetReadFutures` as well in order to cancel the futures.
   Since the herder thread calls `OffsetStorageReaderImpl.close()` when attempting to stops a task, before this change this was resulting in the herder thread hanging indefinetely waiting for `backingStore.get()` to complete.
   
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "viktorsomogyi (via GitHub)" <gi...@apache.org>.
viktorsomogyi commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-4555982010

   @davide-armand thank you for your contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2541110118

   The CI is now green.
   The additional commit that fixed the CI should be squashed before merging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "viktorsomogyi (via GitHub)" <gi...@apache.org>.
viktorsomogyi merged PR #18142:
URL: https://github.com/apache/kafka/pull/18142


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2829817385

   Rebase on latest `apache:trunk`.
   Compilation is now failing, seems unrelated to the changes in the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2735241489

   This PR is being marked as stale since it has not had any activity in 90 days. If you
   would like to keep this PR alive, please leave a comment asking for a review. If the PR has 
   merge conflicts, update it with the latest from the base branch.
   <p>
   If you are having difficulty finding a reviewer, please reach out on the 
   [mailing list](https://kafka.apache.org/contact).
   <p>
   If this PR is no longer valid or desired, please feel free to close it. If no activity
   occurs in the next 30 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-3116295166

   This PR is being marked as stale since it has not had any activity in 90 days. If you
   would like to keep this PR alive, please leave a comment asking for a review. If the PR has 
   merge conflicts, update it with the latest from the base branch.
   <p>
   If you are having difficulty finding a reviewer, please reach out on the 
   [mailing list](https://kafka.apache.org/contact).
   <p>
   If this PR is no longer valid or desired, please feel free to close it. If no activity
   occurs in the next 30 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #18142: KAFKA-17871: avoid blocking the herder thread when producer flushing hangs
URL: https://github.com/apache/kafka/pull/18142


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "jlprat (via GitHub)" <gi...@apache.org>.
jlprat commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2539607705

   Hi @davide-armand
   CI is complaining about check style violations. Could you take a look at it?
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on code in PR #18142:
URL: https://github.com/apache/kafka/pull/18142#discussion_r3287525601


##########
connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageReaderImpl.java:
##########
@@ -86,14 +86,18 @@ public <T> Map<Map<String, T>, Map<String, Object>> offsets(Collection<Map<Strin
         Map<ByteBuffer, ByteBuffer> raw;
         try {
             Future<Map<ByteBuffer, ByteBuffer>> offsetReadFuture;
+
+            // Note: this call can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
+            offsetReadFuture = backingStore.get(serializedToOriginal.keySet());

Review Comment:
   @viktorsomogyi  I forced pushed the suggested change, I agree it makes sense to fail fast here.
   I also pushed a separate commit to refactor `OffsetStorageReaderImpl.offsets()` into smaller methods to fix a checkstyle complexity error in the build caused by this change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-4296446043

   This PR is still relevant.
   I rebased it on latest trunk and got a green build.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2742145207

   A label of 'needs-attention' was automatically added to this PR in order to raise the
   attention of the committers. Once this issue has been triaged, the `triage` label
   should be removed to prevent this automation from happening again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "viktorsomogyi (via GitHub)" <gi...@apache.org>.
viktorsomogyi commented on code in PR #18142:
URL: https://github.com/apache/kafka/pull/18142#discussion_r3249357041


##########
connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageReaderImpl.java:
##########
@@ -86,14 +86,18 @@ public <T> Map<Map<String, T>, Map<String, Object>> offsets(Collection<Map<Strin
         Map<ByteBuffer, ByteBuffer> raw;
         try {
             Future<Map<ByteBuffer, ByteBuffer>> offsetReadFuture;
+
+            // Note: this call can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
+            offsetReadFuture = backingStore.get(serializedToOriginal.keySet());

Review Comment:
   (Ofc with the error message moved to a constant to avoid duplication)



##########
connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageReaderImpl.java:
##########
@@ -86,14 +86,18 @@ public <T> Map<Map<String, T>, Map<String, Object>> offsets(Collection<Map<Strin
         Map<ByteBuffer, ByteBuffer> raw;
         try {
             Future<Map<ByteBuffer, ByteBuffer>> offsetReadFuture;
+
+            // Note: this call can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
+            offsetReadFuture = backingStore.get(serializedToOriginal.keySet());

Review Comment:
   I'm wondering if we need a behavior change here, afterall the point would be to fix the race condition, but with this change we also call `backingStore.get()` regardless of the value of `closed` which may impose some extra work. Do you think it would make sense to duplicate the if condition below to fail fast?
   
   ```suggestion
               if (closed.get()) {
                   throw new ConnectException(
                           "Offset reader is closed. This is likely because the task has already been "
                               + "scheduled to stop but has taken longer than the graceful shutdown "
                               + "period to do so.");
               }
               
               // Note: this call can block for long time waiting for data flush to complete (`KafkaProducer.flush()`).
               offsetReadFuture = backingStore.get(serializedToOriginal.keySet());
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-3218734480

   This PR has been closed since it has not had any activity in 120 days. If you feel like this
   was a mistake, or you would like to continue working on it, please feel free to re-open the 
   PR and ask for a review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "davide-armand (via GitHub)" <gi...@apache.org>.
davide-armand commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2771640033

   > Hi @davide-armand Thanks for the fix, and thanks for the test!
   > 
   > Are you able to remove the Thread.sleep calls from the test? These could introduce some flakiness later if run on slow machines. Additionally I think this test leaks threads through the executors. Could you ensure that the executors are closed, and the threads terminate?
   
   @gharris1727 Pushed a fix in a separate commit, will squash before merging.
   
   I cannot get a green build, if I'm not mistaken it fails on unrelated tests.
   Should I keep restarting the CI hoping for a green build?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-2552708528

   A label of 'needs-attention' was automatically added to this PR in order to raise the
   attention of the committers. Once this issue has been triaged, the `triage` label
   should be removed to prevent this automation from happening again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17871: avoid blocking the herder thread when producer flushing hangs [kafka]

Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on PR #18142:
URL: https://github.com/apache/kafka/pull/18142#issuecomment-4762468687

   Hi @davide-armand @viktorsomogyi , would you mind reviewing this PR https://github.com/apache/kafka/pull/22632 when you have a chance? This test has been showing up as flaky fairly often recently, so it would be great to get your input. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org