Re: [I] iceberg-kafka-connect zombie coordinator thread when writing to S3 fails [iceberg]

["flysen (via GitHub)" <gi...@apache.org>]

flysen commented on issue #16016:
URL: https://github.com/apache/iceberg/issues/16016#issuecomment-5599650323

   **Independent confirmation on 1.11.0 / AWS Glue Catalog**
   
   We hit this exact zombie-coordinator signature during the same incident we described on #15293 ([our comment there](https://github.com/apache/iceberg/issues/15293#issuecomment-5598930908)) — triggered by a targeted consumer-group offset reset, not a storage failure or cluster rebuild.
   
   Across that ~17-hour incident we found the `"found no partitions assigned across all members, cannot determine leader"` WARN logged 88 times across all 6 tasks — initially dismissed as benign, but it's the exact `hasLeaderPartition()`-returns-false-during-rebalance condition described here, which skips `stopCoordinator()` in `close()`. Matches @tongwai-wong-appier's 1.11.0 confirmation: no storage failure needed, an ordinary rebalance is enough.
   
   One caveat: checking the `-coord` group's live membership mid-failure showed only one member at that moment, so we can't say this is the sole explanation for every failed round we saw — possibly a residual effect once the zombie itself is gone. Full evidence (thread dumps, ruled-out causes, timeline) is on #15293 if useful.
   
   +1 to @gdmachado's fix direction (remember leader partition at `open()`, stop unconditionally on it in `close()` rather than re-querying group state mid-rebalance).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org