You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Dmytro Dragan <dd...@softserveinc.com> on 2021/03/03 09:55:34 UTC

Source: Impulse switched to FINISHED in Flink streaming mode

Hi guys,

We have a quite simple generator source:

GenerateSequence.from(0)
        .withRate(1000, Duration.standardSeconds(1L))
        .withTimestampFn((Long l) -> Instant.now())

And after starting in Flink streaming mode, we see that Impulse source is switching to FINISHED, while other parts are keep running:

2021-03-03 09:46:40,425 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: Impulse (1/1) (4b2b8c75a7c0d62a2fdf8d0cbab9fd68) switched from RUNNING to FINISHED.

Can someone explain why it happens? (for example, with KafkaIO source, Impulse keeps running as expected).

It is a quite big blocker cause in such way it is not possible to create checkpoints/savepoints: https://issues.apache.org/jira/browse/FLINK-2491

Beam 2.28, flink 1.11

Best regards,

Dmytro Dragan | ddrag@softserveinc.com<ma...@softserveinc.com> | Lead Big Data Engineer| SoftServe<http://www.softserveinc.com/>

Re: Source: Impulse switched to FINISHED in Flink streaming mode

Posted by Dmytro Dragan <dd...@softserveinc.com>.

My bad: the issue was related to skipped checkpointingInterval


Best regards,

Dmytro Dragan | ddrag@softserveinc.com<ma...@softserveinc.com> | Lead Big Data Engineer| SoftServe<http://www.softserveinc.com/>



From: Dmytro Dragan <dd...@softserveinc.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Wednesday, 3 March 2021 at 10:56
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Source: Impulse switched to FINISHED in Flink streaming mode


CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi guys,

We have a quite simple generator source:

GenerateSequence.from(0)
        .withRate(1000, Duration.standardSeconds(1L))
        .withTimestampFn((Long l) -> Instant.now())

And after starting in Flink streaming mode, we see that Impulse source is switching to FINISHED, while other parts are keep running:

2021-03-03 09:46:40,425 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: Impulse (1/1) (4b2b8c75a7c0d62a2fdf8d0cbab9fd68) switched from RUNNING to FINISHED.

Can someone explain why it happens? (for example, with KafkaIO source, Impulse keeps running as expected).

It is a quite big blocker cause in such way it is not possible to create checkpoints/savepoints: https://issues.apache.org/jira/browse/FLINK-2491

Beam 2.28, flink 1.11

Best regards,

Dmytro Dragan | ddrag@softserveinc.com<ma...@softserveinc.com> | Lead Big Data Engineer| SoftServe<http://www.softserveinc.com/>