You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "zheliu2 (via GitHub)" <gi...@apache.org> on 2026/03/09 15:30:59 UTC
[PR] Kafka Connect: Fix thread hanging after broker restart [iceberg]
zheliu2 opened a new pull request, #15569:
URL: https://github.com/apache/iceberg/pull/15569
## Summary
Fixes #15452
After a Kafka broker restart, threads in the Iceberg sink connector can hang and stop committing to the catalog while the data consumer group continues to progress. Two root causes:
- **Fatal producer exceptions silently swallowed**: `Coordinator.commit()` catches all exceptions including fatal Kafka producer errors (`ProducerFencedException`, `OutOfOrderSequenceException`, `InvalidProducerEpochException`). After such errors the producer is unrecoverable, but the coordinator keeps running with a broken producer, unable to send any events. This PR propagates fatal exceptions so the coordinator thread terminates and the Kafka Connect framework restarts the task.
- **Channel.stop() can hang indefinitely**: `producer.close()`, `consumer.close()`, and `admin.close()` are called without timeouts. If the broker is unavailable, these calls can block forever, preventing the coordinator thread from completing shutdown. This PR adds a 30-second close timeout and wraps each close in try-catch so one failing client does not block the others.
## Changes
- `Coordinator.java`: Add `isFatalException()` check in `commit()` to propagate fatal Kafka producer errors instead of swallowing them
- `Channel.java`: Add close timeouts (30s) and individual try-catch for producer, consumer, and admin close operations
- `TestCoordinator.java`: Add `testCommitFatalProducerError` verifying that `ProducerFencedException` propagates from `process()`
## Test plan
- [x] `testCommitFatalProducerError` - Verifies that fencing the producer causes `process()` to throw `ProducerFencedException`
- [x] All existing `TestCoordinator` tests pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Fix thread hanging after broker restart [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #15569:
URL: https://github.com/apache/iceberg/pull/15569#issuecomment-4028862447
This PR appears to be generated and submitted by AI without human review. Closing for now. If you'd like to resubmit, please follow our AI contribution guidelines: https://iceberg.apache.org/contribute/#guidelines-for-ai-assisted-contributions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Fix thread hanging after broker restart [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao closed pull request #15569: Kafka Connect: Fix thread hanging after broker restart
URL: https://github.com/apache/iceberg/pull/15569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org