You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Alexis Sarda-Espinosa (Jira)" <ji...@apache.org> on 2024/02/06 17:15:00 UTC
[jira] [Updated] (FLINK-34400) Kafka sources with watermark alignment sporadically stop consuming
[ https://issues.apache.org/jira/browse/FLINK-34400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexis Sarda-Espinosa updated FLINK-34400:
------------------------------------------
Description:
I have 2 Kafka sources that read from different topics. I have assigned them to the same watermark alignment group, and I have _not_ enabled idleness explicitly in their watermark strategies. One topic remains pretty much empty most of the time, while the other receives a few events per second all the time. Parallelism of the active source is 2.
This works correctly for some time (10 - 15 minutes in my case) but then 1 of the active sources stops consuming, which causes lag to increase. Weirdly, after another 15 minutes or so, all the backlog is consumed at once, and then everything stops again.
I'm attaching some logs from the Task Manager where the issue appears. You will notice that the Kafka network client reports disconnections (a long time after the deserializer stopped reporting that events were being consumed), I'm not sure if this is related.
was:
I have 2 Kafka sources that read from different topics. I have assigned them to the same watermark alignment group, and I have _not_ enabled idleness explicitly in their watermark strategies. One topic remains pretty much empty most of the time, while the other receives a few events per second all the time. Parallelism of the active source is 2.
This works correctly for some time (10 - 15 minutes in my case) but then 1 of the active sources stops consuming, which causes lag to increase. Weirdly, after another 15 minutes or so, all the backlog is consumed at once, and then everything stops again.
I'm attaching some logs from the Task Manager where the issue appears. You will notice that the Kafka network client reports disconnections - this is because my Kafka cluster was indeed restarted (one broker at a time), I'm not sure if this is related.
> Kafka sources with watermark alignment sporadically stop consuming
> ------------------------------------------------------------------
>
> Key: FLINK-34400
> URL: https://issues.apache.org/jira/browse/FLINK-34400
> Project: Flink
> Issue Type: Bug
> Affects Versions: 1.18.1
> Reporter: Alexis Sarda-Espinosa
> Priority: Major
> Attachments: logs.txt
>
>
> I have 2 Kafka sources that read from different topics. I have assigned them to the same watermark alignment group, and I have _not_ enabled idleness explicitly in their watermark strategies. One topic remains pretty much empty most of the time, while the other receives a few events per second all the time. Parallelism of the active source is 2.
> This works correctly for some time (10 - 15 minutes in my case) but then 1 of the active sources stops consuming, which causes lag to increase. Weirdly, after another 15 minutes or so, all the backlog is consumed at once, and then everything stops again.
> I'm attaching some logs from the Task Manager where the issue appears. You will notice that the Kafka network client reports disconnections (a long time after the deserializer stopped reporting that events were being consumed), I'm not sure if this is related.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)