You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Seth Saperstein via user <us...@flink.apache.org> on 2022/08/29 21:25:03 UTC
Deadlock in Subtask in the FlinkKinesisConsumer
Hi
I wanted to bring awareness to this Jira
<https://issues.apache.org/jira/browse/FLINK-29099> describing a deadlock
state we've experienced for a single subtask in the FlinkKinesisConsumer.
This occurs when we've reached the following conditions in the subtask:
- reached the max lookahead so the RecordEmitter does not emit
- are marked idle and the global watermark does not refresh
More details are in the ticket, however, we intend to resolve this by
updating the global watermark for idle subtasks, as noted here
<https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java#L1268>.
This will allow us to emit records again in the RecordEmitter and bring us
out of the idle state.
I'm curious to hear if anyone has run into similar issues or sees getting
the global watermark when idle as problematic. We plan to test an
implementation of this internally in the coming weeks.
Thanks,
Seth
Re: Deadlock in Subtask in the FlinkKinesisConsumer
Posted by Martijn Visser <ma...@apache.org>.
Hi Seth,
I think the global watermark / mark as idle is one of the reasons why the
FLIP-27 [1] interfaces were created, while FLIP-180 [2] is also relevant
here. Unfortunately, the FlinkKinesisConsumer is still using the old
interface. I'm not sure it can be properly fixed with the old interfaces.
There are tickets for migrating the Kinesis source to new interfaces [3]
[4] but I don't think they are planned yet.
I believe that the only other source that used global watermark / idleness
in Flink was the FlinkKafkaConsumer, which has been deprecated already in
favour of the KafkaSource. You could consider having a look at how the
handling was done for the FlinkKafkaConsumer to see if that can also be
applied for FlinkKinesisConsumer.
Best regards,
Martijn
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-180%3A+Adjust+StreamStatus+and+Idleness+definition
[3] https://issues.apache.org/jira/browse/FLINK-24438
[4] https://issues.apache.org/jira/browse/FLINK-25729
Op ma 29 aug. 2022 om 23:25 schreef Seth Saperstein via user <
user@flink.apache.org>:
> Hi
>
> I wanted to bring awareness to this Jira
> <https://issues.apache.org/jira/browse/FLINK-29099> describing a deadlock
> state we've experienced for a single subtask in the FlinkKinesisConsumer.
>
> This occurs when we've reached the following conditions in the subtask:
> - reached the max lookahead so the RecordEmitter does not emit
> - are marked idle and the global watermark does not refresh
>
> More details are in the ticket, however, we intend to resolve this by
> updating the global watermark for idle subtasks, as noted here
> <https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java#L1268>.
> This will allow us to emit records again in the RecordEmitter and bring us
> out of the idle state.
>
> I'm curious to hear if anyone has run into similar issues or sees getting
> the global watermark when idle as problematic. We plan to test an
> implementation of this internally in the coming weeks.
>
> Thanks,
> Seth
>
Re: Deadlock in Subtask in the FlinkKinesisConsumer
Posted by Martijn Visser <ma...@apache.org>.
Hi Seth,
I think the global watermark / mark as idle is one of the reasons why the
FLIP-27 [1] interfaces were created, while FLIP-180 [2] is also relevant
here. Unfortunately, the FlinkKinesisConsumer is still using the old
interface. I'm not sure it can be properly fixed with the old interfaces.
There are tickets for migrating the Kinesis source to new interfaces [3]
[4] but I don't think they are planned yet.
I believe that the only other source that used global watermark / idleness
in Flink was the FlinkKafkaConsumer, which has been deprecated already in
favour of the KafkaSource. You could consider having a look at how the
handling was done for the FlinkKafkaConsumer to see if that can also be
applied for FlinkKinesisConsumer.
Best regards,
Martijn
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-180%3A+Adjust+StreamStatus+and+Idleness+definition
[3] https://issues.apache.org/jira/browse/FLINK-24438
[4] https://issues.apache.org/jira/browse/FLINK-25729
Op ma 29 aug. 2022 om 23:25 schreef Seth Saperstein via user <
user@flink.apache.org>:
> Hi
>
> I wanted to bring awareness to this Jira
> <https://issues.apache.org/jira/browse/FLINK-29099> describing a deadlock
> state we've experienced for a single subtask in the FlinkKinesisConsumer.
>
> This occurs when we've reached the following conditions in the subtask:
> - reached the max lookahead so the RecordEmitter does not emit
> - are marked idle and the global watermark does not refresh
>
> More details are in the ticket, however, we intend to resolve this by
> updating the global watermark for idle subtasks, as noted here
> <https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java#L1268>.
> This will allow us to emit records again in the RecordEmitter and bring us
> out of the idle state.
>
> I'm curious to hear if anyone has run into similar issues or sees getting
> the global watermark when idle as problematic. We plan to test an
> implementation of this internally in the coming weeks.
>
> Thanks,
> Seth
>