You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Vijayendra Yadav <co...@gmail.com> on 2021/06/23 03:02:14 UTC

High Flink checkpoint Size

Hi Team,

I have two flink Streaming Jobs
1) Flink streaming from KAFKA and writing to s3
2) Fling Streaming from KINESIS (KDS) and writing to s3

Both Jobs have similar checkpoint duration.

Job #1 (KAFKA) checkpoint size is only 85KB
Job #2 (KINESIS) checkpoint size is 18MB

There are no checkpoint failures. But I want to understand why Kinesis
streaming has such a huge checkpoint size, is there a way to handle it
differently? and reduce the size.

Thanks,
Vijay

Re: High Flink checkpoint Size

Posted by Vijayendra Yadav <co...@gmail.com>.

Thanks Yun. Let me try options provided below.

Thanks,
Vijay

> On Jun 23, 2021, at 4:51 AM, Yun Tang <my...@live.com> wrote:
> 
> 
> Hi Vijay,
> 
> To be honest, an 18MB checkpoint size in total might not be something serious. If you really want to dig what inside, you could use Checkpoints#loadCheckpointMetadata [1] to load the _metadata to see anything unexpected.
> 
> And you could refer to FlinkKafkaConsumerBase#unionOffsetStates [2] and FlinkKinesisConsumer#sequenceNumsToRestore to compare different operator state stored in kafka and kinesis connector.
> 
> [1] https://github.com/apache/flink/blob/10146366bec7feca85acedb23184b99517059bc6/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java#L99
> [2] https://github.com/apache/flink/blob/10146366bec7feca85acedb23184b99517059bc6/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java#L201
> [3] https://github.com/apache/flink/blob/10146366bec7feca85acedb23184b99517059bc6/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L158-L159
> 
> Best,
> Yun Tang
> From: Vijayendra Yadav <co...@gmail.com>
> Sent: Wednesday, June 23, 2021 11:02
> To: user <us...@flink.apache.org>
> Subject: High Flink checkpoint Size
>  
> Hi Team,
> 
> I have two flink Streaming Jobs 
> 1) Flink streaming from KAFKA and writing to s3
> 2) Fling Streaming from KINESIS (KDS) and writing to s3
> 
> Both Jobs have similar checkpoint duration. 
> 
> Job #1 (KAFKA) checkpoint size is only 85KB 
> Job #2 (KINESIS) checkpoint size is 18MB
> 
> There are no checkpoint failures. But I want to understand why Kinesis streaming has such a huge checkpoint size, is there a way to handle it differently? and reduce the size. 
> 
> Thanks,
> Vijay

Re: High Flink checkpoint Size

Posted by Yun Tang <my...@live.com>.

Hi Vijay,

To be honest, an 18MB checkpoint size in total might not be something serious. If you really want to dig what inside, you could use Checkpoints#loadCheckpointMetadata [1] to load the _metadata to see anything unexpected.

And you could refer to FlinkKafkaConsumerBase#unionOffsetStates [2] and FlinkKinesisConsumer#sequenceNumsToRestore to compare different operator state stored in kafka and kinesis connector.

[1] https://github.com/apache/flink/blob/10146366bec7feca85acedb23184b99517059bc6/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java#L99
[2] https://github.com/apache/flink/blob/10146366bec7feca85acedb23184b99517059bc6/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java#L201
[3] https://github.com/apache/flink/blob/10146366bec7feca85acedb23184b99517059bc6/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L158-L159

Best,
Yun Tang
________________________________
From: Vijayendra Yadav <co...@gmail.com>
Sent: Wednesday, June 23, 2021 11:02
To: user <us...@flink.apache.org>
Subject: High Flink checkpoint Size

Hi Team,

I have two flink Streaming Jobs
1) Flink streaming from KAFKA and writing to s3
2) Fling Streaming from KINESIS (KDS) and writing to s3

Both Jobs have similar checkpoint duration.

Job #1 (KAFKA) checkpoint size is only 85KB
Job #2 (KINESIS) checkpoint size is 18MB

There are no checkpoint failures. But I want to understand why Kinesis streaming has such a huge checkpoint size, is there a way to handle it differently? and reduce the size.

Thanks,
Vijay