[jira] [Comment Edited] (FLINK-28060) Kafka Commit on checkpointing fails repeatedly after a broker restart

["David Anderson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-28060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555527#comment-17555527 ] 

David Anderson edited comment on FLINK-28060 at 6/17/22 10:33 AM:
------------------------------------------------------------------

It doesn't seem wise to upgrade the Kafka client from 2.8.1 to 3.1.1 for Flink 1.15.1. That's a big change to make this close to our release, and we shouldn't delay this release any further. So here's a proposal:
 * We bump the Kafka Clients to 3.1.1 in master now.
 * We don't try to fix FLINK-28060 for 1.15.1.

 * We create the Flink 1.15.1 release straight away, noting that there's a known issue with Kafka (FLINK-28060).
 * We reach out to the Kafka community to see if they're willing to create a 2.8.2 release with this patch.
 * In parallel, we merge the Kafka Clients bump to 3.1.1 after release 1.15.1 is done, to see how it behaves on the CI for the next few weeks and plan a quick Flink 1.15.2 release (most likely something like a month later).


was (Author: danderson):
It doesn't seem wise to upgrade the Kafka client from 2.8.1 to 3.1.1 for Flink 1.15.1. That's a big change to make this close to our release, and we shouldn't delay this release any further. So here's a proposal:
 * We bump the Kafka Clients to 3.1.1 in master now.
 * We don't try to fix FLINK-28060 for 1.15.1.

 * We create the Flink 1.15.1 release straight away, noting that there's a known issue with Kafka (FLINK-28060).
 * We reach out to the Kafka community to see if they're willing to create a 2.8.2 release with this patch.
 * In parallel, we merge the Kafka Clients bump to 3.1.1 after release 1.15.1 is done, to see how it behaves on the CI for the next few weeks and plan a quick for Flink 1.15.2 (most likely something like a month later).

> Kafka Commit on checkpointing fails repeatedly after a broker restart
> ---------------------------------------------------------------------
>
>                 Key: FLINK-28060
>                 URL: https://issues.apache.org/jira/browse/FLINK-28060
>             Project: Flink
>          Issue Type: Bug
>          Components: API / DataStream, Connectors / Kafka
>    Affects Versions: 1.15.0
>         Environment: Reproduced on MacOS and Linux.
> Using java 8, Flink 1.15.0, Kafka 2.8.1.
>            Reporter: Christian Lorenz
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: flink-kafka-testjob.zip
>
>
> When Kafka Offset committing is enabled and done on Flinks checkpointing, an error might occur if one Kafka broker is shutdown which might be the leader of that partition in Kafkas internal __consumer_offsets topic.
> This is an expected behaviour. But once the broker is started up again, the next checkpoint issued by flink should commit the meanwhile processed offsets back to kafka. Somehow this does not seem to happen always in Flink 1.15.0 anymore and the offset committing is broken. An warning like the following will be logged on each checkpoint:
> {code}
> [info] 14:33:13.684 WARN  [Source Data Fetcher for Source: input-kafka-source -> Sink: output-stdout-sink (1/1)#1] o.a.f.c.k.s.reader.KafkaSourceReader - Failed to commit consumer offsets for checkpoint 35
> [info] org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets.
> [info] Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException: The coordinator is not available.
> {code}
> To reproduce this I've attached a small flink job program.  To execute this java8, scala sbt and docker / docker-compose is required.  Also see readme.md for more details.
> The job can be run with `sbt run`, kafka cluster is started by `docker-compose up`. If then the kafka brokers are restarted gracefully by e.g. `docker-compose stop kafka1` and `docker-compose start kafka1` with kafka2 and kafka3 afterwards, this warning will occur and no offsets will be committed into kafka.
> This is not reproducible in flink 1.14.4.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)