You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user-zh@flink.apache.org by LakeShen <sh...@gmail.com> on 2020/01/19 10:55:36 UTC

Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

Hi community，
        now I have a flink sql job, and I set the flink sql sate retention
time, there are three dir in flink checkpoint dir :
1. chk -xx dir
2. shared dir
3. taskowned dir

I find the shared dir store the last year checkpoint state,the only reason
I thought is that the latest
checkpoint retain reference of last year checkpoint state file.
Are there any other reason to lead this? Or is it a bug?

Thanks to your replay.

Best wishes,
Lake Shen

Re: Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

Posted by Yun Tang <my...@live.com>.

Hi Lake

A more suitable place for this mail should be in user-mail list.

There are three reasons why this could happen:

  1.  This file is orphan file e.g. file uploaded during one checkpoint but task manager exited unexpectedly leave that checkpoint not completed.
  2.  This file should be removed by checkpoint coordinator but takes too long to complete before job shut down.
  3.  This file is still useful. This is possible in theory because some specific rocksDB sst file might not be selected during compactions for a long time.

Best
Yun Tang
________________________________
From: LakeShen <sh...@gmail.com>
Sent: Sunday, January 19, 2020 18:55
To: user <us...@flink.apache.org>; user-zh@flink.apache.org <us...@flink.apache.org>; dev <de...@flink.apache.org>
Subject: Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

Hi community，
        now I have a flink sql job, and I set the flink sql sate retention time, there are three dir in flink checkpoint dir :
1. chk -xx dir
2. shared dir
3. taskowned dir

I find the shared dir store the last year checkpoint state,the only reason I thought is that the latest
checkpoint retain reference of last year checkpoint state file.
Are there any other reason to lead this? Or is it a bug?

Thanks to your replay.

Best wishes,
Lake Shen

Re: Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

Posted by Yun Tang <my...@live.com>.

Hi Lake

A more suitable place for this mail should be in user-mail list.

There are three reasons why this could happen:

  1.  This file is orphan file e.g. file uploaded during one checkpoint but task manager exited unexpectedly leave that checkpoint not completed.
  2.  This file should be removed by checkpoint coordinator but takes too long to complete before job shut down.
  3.  This file is still useful. This is possible in theory because some specific rocksDB sst file might not be selected during compactions for a long time.

Best
Yun Tang
________________________________
From: LakeShen <sh...@gmail.com>
Sent: Sunday, January 19, 2020 18:55
To: user <us...@flink.apache.org>; user-zh@flink.apache.org <us...@flink.apache.org>; dev <de...@flink.apache.org>
Subject: Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

Hi community，
        now I have a flink sql job, and I set the flink sql sate retention time, there are three dir in flink checkpoint dir :
1. chk -xx dir
2. shared dir
3. taskowned dir

I find the shared dir store the last year checkpoint state,the only reason I thought is that the latest
checkpoint retain reference of last year checkpoint state file.
Are there any other reason to lead this? Or is it a bug?

Thanks to your replay.

Best wishes,
Lake Shen

Re: Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

Posted by Yun Tang <my...@live.com>.

Hi Lake

A more suitable place for this mail should be in user-mail list.

There are three reasons why this could happen:

  1.  This file is orphan file e.g. file uploaded during one checkpoint but task manager exited unexpectedly leave that checkpoint not completed.
  2.  This file should be removed by checkpoint coordinator but takes too long to complete before job shut down.
  3.  This file is still useful. This is possible in theory because some specific rocksDB sst file might not be selected during compactions for a long time.

Best
Yun Tang
________________________________
From: LakeShen <sh...@gmail.com>
Sent: Sunday, January 19, 2020 18:55
To: user <us...@flink.apache.org>; user-zh@flink.apache.org <us...@flink.apache.org>; dev <de...@flink.apache.org>
Subject: Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

Hi community，
        now I have a flink sql job, and I set the flink sql sate retention time, there are three dir in flink checkpoint dir :
1. chk -xx dir
2. shared dir
3. taskowned dir

I find the shared dir store the last year checkpoint state,the only reason I thought is that the latest
checkpoint retain reference of last year checkpoint state file.
Are there any other reason to lead this? Or is it a bug?

Thanks to your replay.

Best wishes,
Lake Shen