You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Zakelly Lan <za...@gmail.com> on 2023/12/25 09:13:57 UTC

[DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Hi devs,

I'd like to start a discussion on FLIP-406: Reorganize State &
Checkpointing & Recovery Configuration[1].

Currently, the configuration options pertaining to checkpointing, recovery,
and state management are primarily grouped under the following prefixes:

   - state.backend.* : configurations related to state accessing and
   checkpointing, as well as specific options for individual state backends
   - execution.checkpointing.* : configurations associated with checkpoint
   execution and recovery
   - execution.savepoint.*: configurations for recovery from savepoint

In addition, there are several individual options such as '
*state.checkpoint-storage*' and '*state.checkpoints.dir*' that fall outside
of these prefixes. The current arrangement of these options, which span
multiple modules, is somewhat haphazard and lacks a systematic structure.
For example, the options under the '*CheckpointingOptions*' and '
*ExecutionCheckpointingOptions*' are related and have no clear boundaries
from the user's perspective, but there is no unified prefix for them. With
the upcoming release of Flink 2.0, we have an excellent opportunity to
overhaul and restructure the configurations related to checkpointing,
recovery, and state management. This FLIP proposes to reorganize these
settings, making it more coherent by module, which would significantly
lower the barriers for understanding and reduce the development costs
moving forward.

Looking forward to hearing from you!

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560

Best,
Zakelly

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi everyone,

It seems too late to make this in 1.19, so I suggest changing it to 1.20.
Another thing I'd like to highlight is that there are some existing option
classes lacking annotations, which are:

   - org.apache.flink.configuration.CheckpointingOptions
   - org.apache.flink.configuration.StateBackendOptions
   - org.apache.flink.contrib.streaming.state.RocksDBOptions

This FLIP will annotate these existing classes with @PublicEvolving in
version 2.0, since almost all of the state-related option classes and APIs
are annotated with @PublicEvolving. Moreover, the migration period of
deprecated options in those classes will last for one minor release (1.20),
which meets the requirement of migration period for @PublicEvolving [1].
And in 2.0, they will be removed.

Based on the discussion so far, I will proceed to start the vote tomorrow.
Thanks!

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process


Best,
Zakelly

On Mon, Jan 22, 2024 at 6:31 PM Zakelly Lan <za...@gmail.com> wrote:

> Hi everyone,
>
> It has been 6 days since the last call for discussion. I'd like to start a
> vote after another 2 days.
>
> Please let me know if you have any concerns. Thanks!
>
>
> Best,
> Zakelly
>
> On Tue, Jan 16, 2024 at 2:54 PM Zakelly Lan <za...@gmail.com> wrote:
>
>> Thanks for the suggestion Rui!  The type is added.
>>
>>
>> Best,
>> Zakelly
>>
>> On Tue, Jan 16, 2024 at 2:33 PM Rui Fan <19...@gmail.com> wrote:
>>
>>> Hi Zakelly,
>>>
>>> Would you mind adding the option type in the FLIP doc?
>>> For example, String, Boolean or Enum, etc. Thank you.
>>>
>>> Best,
>>> Rui
>>>
>>> On Tue, Jan 16, 2024 at 2:29 PM Zakelly Lan <za...@gmail.com>
>>> wrote:
>>>
>>> > Hi everyone,
>>> >
>>> > Thanks all for joining the discussion! I'd like to speed this up since
>>> it
>>> > lasts for nearly a month. I made changes on this FLIP based on
>>> suggestions
>>> > and compromises acceptable to most people. Please feel free to give
>>> your
>>> > opinion. Thanks!
>>> > If there are no more suggestions, I will consider starting a vote
>>> within a
>>> > week.
>>> >
>>> >
>>> > Best,
>>> > Zakelly
>>> >
>>> > On Thu, Jan 11, 2024 at 10:31 AM Xuannan Su <su...@gmail.com>
>>> wrote:
>>> >
>>> > > Hi Zakelly,
>>> > >
>>> > > I am fine with either Option 2 or Option 3. I think the naming in
>>> > > Option 2 makes it clear that it is a boolean configuration. However,
>>> > > most of the currently available boolean configurations do not use
>>> > > "enable" as a suffix. Therefore, Option 3 looks good to me as it
>>> > > follows the current practice.
>>> > >
>>> > > Best regards,
>>> > > Xuannan
>>> > >
>>> > > On Thu, Jan 11, 2024 at 9:50 AM Hangxiang Yu <ma...@gmail.com>
>>> > wrote:
>>> > > >
>>> > > > >
>>> > > > > That's a very good point. I realize that the word 'recovery'
>>> means
>>> > way
>>> > > too
>>> > > > > many things. So I suggest picking a more specific word here, how
>>> > about
>>> > > > > 'execution.state-recovery.*' ? Checkpointing and state recovery
>>> are
>>> > > > > corresponding terms and won't make ambiguity.
>>> > > > >
>>> > > >
>>> > > > This makes the configuration clearer to me. We could focus on the
>>> > > > `state-recovery` at first.
>>> > > >
>>> > > > I think we could create another FLIP for the deprecation of LEGACY
>>> > mode.
>>> > > > >
>>> > > >
>>> > > > LGTM, Let's create a new FLIP to do this.
>>> > > >
>>> > > > IIUC, there is no clear ownership of the local copy files from the
>>> > > previous
>>> > > > > job and it's better to define one. This needs more discussion so
>>> we
>>> > > could
>>> > > > > create another thread for this. WDYT?
>>> > > > >
>>> > > >
>>> > > > Yeah, I have created a new ticket FLINK-34032 to track and discuss
>>> > this.
>>> > > >
>>> > > > On Wed, Jan 10, 2024 at 6:31 PM Zakelly Lan <zakelly.lan@gmail.com
>>> >
>>> > > wrote:
>>> > > >
>>> > > > > Hi everyone,
>>> > > > >
>>> > > > > It seems we still don't have a consensus on the rules for boolean
>>> > type
>>> > > > > options. Let me recap the alternatives we have:
>>> > > > >
>>> > > > > Option 1: Use enumeration options instead if possible. But this
>>> may
>>> > > cause
>>> > > > > some name collisions or confusion as we discussed and we should
>>> unify
>>> > > the
>>> > > > > statement everywhere.
>>> > > > > Option 2: Use boolean options and add 'enabled' as the suffix.
>>> > > > > Option 3: Use boolean options and ONLY add 'enabled' when there
>>> are
>>> > > more
>>> > > > > detailed configurations under the same prefix, to prevent one
>>> name
>>> > from
>>> > > > > serving as a prefix to another.
>>> > > > >
>>> > > > > I am inclined to Option 3, since it is more in line with current
>>> > > practice
>>> > > > > and friendly for existing users. Also It reduces the length of
>>> > > > > configuration names as much as possible.
>>> > > > >
>>> > > > > Looking forward to your opinions! Thanks!
>>> > > > >
>>> > > > >
>>> > > > > Best,
>>> > > > > Zakelly
>>> > > > >
>>> > > > > On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan <
>>> zakelly.lan@gmail.com>
>>> > > wrote:
>>> > > > >
>>> > > > > > Hi Hangxiang,
>>> > > > > >
>>> > > > > > Thanks for your suggestions!
>>> > > > > >
>>> > > > > > 1. Could execution.recovery also contain some other behaviors
>>> about
>>> > > > > >> recovery ? e.g. restart-strategy.
>>> > > > > >
>>> > > > > >
>>> > > > > > That's a very good point. I realize that the word 'recovery'
>>> means
>>> > > way
>>> > > > > too
>>> > > > > > many things. So I suggest picking a more specific word here,
>>> how
>>> > > about
>>> > > > > > 'execution.state-recovery.*' ? Checkpointing and state
>>> recovery are
>>> > > > > > corresponding terms and won't make ambiguity.
>>> > > > > >
>>> > > > > > 2. Could we also remove some legacy configuration value ? e.g.
>>> > LEGACY
>>> > > > > Mode
>>> > > > > >> for
>>> > execution.savepoint-restore-mode/execution.recovery.claim-mode.
>>> > > > > >
>>> > > > > >
>>> > > > > > I think we could create another FLIP for the deprecation of
>>> LEGACY
>>> > > mode.
>>> > > > > >
>>> > > > > >
>>> > > > > >> 3. Could the local checkpoint be cleaned
>>> > > > > >> if execution.checkpointing.local-copy.enabled is true and
>>> > > > > >> execution.recovery.from-local is false ? I found it's also an
>>> > issue
>>> > > if
>>> > > > > >> current local-recovery from enabled to disabled. Maybe another
>>> > > ticket is
>>> > > > > >> needed.
>>> > > > > >
>>> > > > > >
>>> > > > > > IIUC, there is no clear ownership of the local copy files from
>>> the
>>> > > > > > previous job and it's better to define one. This needs more
>>> > > discussion so
>>> > > > > > we could create another thread for this. WDYT?
>>> > > > > >
>>> > > > > >
>>> > > > > > Best,
>>> > > > > > Zakelly
>>> > > > > >
>>> > > > > > On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu <
>>> masteryhx@gmail.com>
>>> > > > > wrote:
>>> > > > > >
>>> > > > > >> Hi, Zakelly.
>>> > > > > >> Thanks for driving this. Overall LGTM as we discussed offline.
>>> > > > > >>
>>> > > > > >> Some comments/suggestions just came to mind:
>>> > > > > >> 1. Could execution.recovery also contain some other behaviors
>>> > about
>>> > > > > >> recovery ? e.g. restart-strategy.
>>> > > > > >> 2. Could we also remove some legacy configuration value ? e.g.
>>> > > LEGACY
>>> > > > > Mode
>>> > > > > >> for
>>> > execution.savepoint-restore-mode/execution.recovery.claim-mode.
>>> > > > > >> 3. Could the local checkpoint be cleaned
>>> > > > > >> if execution.checkpointing.local-copy.enabled is true and
>>> > > > > >> execution.recovery.from-local is false ? I found it's also an
>>> > issue
>>> > > if
>>> > > > > >> current local-recovery from enabled to disabled. Maybe another
>>> > > ticket is
>>> > > > > >> needed.
>>> > > > > >> 4. +1 for enabling execution.checkpointing.incremental by
>>> default
>>> > > which
>>> > > > > is
>>> > > > > >> basically default configuration in our production environment.
>>> > > > > >>
>>> > > > > >>
>>> > > > > >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <
>>> zakelly.lan@gmail.com
>>> > >
>>> > > > > wrote:
>>> > > > > >>
>>> > > > > >> > Hi Yun,
>>> > > > > >> >
>>> > > > > >> > Thanks for your comments!
>>> > > > > >> >
>>> > > > > >> >  1.  We shall not describe the configuration with its
>>> > > implementation
>>> > > > > for
>>> > > > > >> > > 'execution.checkpointing.local-copy.*' options, for
>>> hashmap
>>> > > > > >> > state-backend,
>>> > > > > >> > > it would write two streams and for Rocksdb state-backend,
>>> it
>>> > > would
>>> > > > > use
>>> > > > > >> > > hard-link for backup. Thus, I think
>>> > > > > >> > > 'execution.checkpointing.local-backup.*' looks better.
>>> > > > > >> >
>>> > > > > >> > I agreed that we'd better name the option in user's
>>> perspective
>>> > > > > instead
>>> > > > > >> of
>>> > > > > >> > the implementation, thus I name it as a copy of the
>>> checkpoint
>>> > in
>>> > > the
>>> > > > > >> > local disk, regardless of the way of generating it. The word
>>> > > 'backup'
>>> > > > > is
>>> > > > > >> > also suitable for this case, so I agree to change to
>>> > > > > >> > 'execution.checkpointing.local-backup.*' if no one objects.
>>> > > > > >> >
>>> > > > > >> >  2.  What does the
>>> > 'execution.checkpointing.data-inline-threshold'
>>> > > > > >> mean? It
>>> > > > > >> > > seems not so easy to understand.
>>> > > > > >> >
>>> > > > > >> > The 'execution.checkpointing.data-inline-threshold'
>>> (original
>>> > one
>>> > > as
>>> > > > > >> > 'state.storage.fs.memory-threshold') stands for the size
>>> > threshold
>>> > > > > below
>>> > > > > >> > which state chunks will store inline with the metadata,
>>> thus I
>>> > > call it
>>> > > > > >> > 'data-inline-threshold'.
>>> > > > > >> >
>>> > > > > >> >
>>> > > > > >> > Best,
>>> > > > > >> > Zakelly
>>> > > > > >> >
>>> > > > > >> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com>
>>> > > wrote:
>>> > > > > >> >
>>> > > > > >> > > Hi Zakelly,
>>> > > > > >> > >
>>> > > > > >> > > Thanks for driving this topic. I have two concerns here:
>>> > > > > >> > >
>>> > > > > >> > >   1.  We shall not describe the configuration with its
>>> > > > > implementation
>>> > > > > >> for
>>> > > > > >> > > 'execution.checkpointing.local-copy.*' options, for
>>> hashmap
>>> > > > > >> > state-backend,
>>> > > > > >> > > it would write two streams and for Rocksdb state-backend,
>>> it
>>> > > would
>>> > > > > use
>>> > > > > >> > > hard-link for backup. Thus, I think
>>> > > > > >> > > 'execution.checkpointing.local-backup.*' looks better.
>>> > > > > >> > >   2.  What does the
>>> > > 'execution.checkpointing.data-inline-threshold'
>>> > > > > >> mean?
>>> > > > > >> > > It seems not so easy to understand.
>>> > > > > >> > >
>>> > > > > >> > > Best
>>> > > > > >> > > Yun Tang
>>> > > > > >> > > ________________________________
>>> > > > > >> > > From: Piotr Nowojski <pn...@apache.org>
>>> > > > > >> > > Sent: Thursday, January 4, 2024 22:37
>>> > > > > >> > > To: dev@flink.apache.org <de...@flink.apache.org>
>>> > > > > >> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State &
>>> > > Checkpointing &
>>> > > > > >> > > Recovery Configuration
>>> > > > > >> > >
>>> > > > > >> > > Hi,
>>> > > > > >> > >
>>> > > > > >> > > Thanks for trying to clean this up! I don't have strong
>>> > > opinions on
>>> > > > > >> the
>>> > > > > >> > > topics discussed here, so generally speaking +1 from my
>>> side!
>>> > > > > >> > >
>>> > > > > >> > > Best,
>>> > > > > >> > > Piotrek
>>> > > > > >> > >
>>> > > > > >> > > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com>
>>> > > napisał(a):
>>> > > > > >> > >
>>> > > > > >> > > > Thanks for the feedback!
>>> > > > > >> > > >
>>> > > > > >> > > > Using the `execution.checkpointing.incremental.enabled`,
>>> > > > > >> > > > and enabling it by default sounds good to me.
>>> > > > > >> > > >
>>> > > > > >> > > > Best,
>>> > > > > >> > > > Rui
>>> > > > > >> > > >
>>> > > > > >> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <
>>> > > > > zakelly.lan@gmail.com>
>>> > > > > >> > > wrote:
>>> > > > > >> > > >
>>> > > > > >> > > > > Hi Rui,
>>> > > > > >> > > > >
>>> > > > > >> > > > > Thanks for your comments!
>>> > > > > >> > > > >
>>> > > > > >> > > > > IMO, given that the state backend can be plugably
>>> loaded
>>> > > (as you
>>> > > > > >> can
>>> > > > > >> > > > > specify a state backend factory), I prefer not
>>> providing
>>> > > state
>>> > > > > >> > backend
>>> > > > > >> > > > > specified options in the framework.
>>> > > > > >> > > > >
>>> > > > > >> > > > > Secondly, the incremental checkpoint is actually a
>>> sharing
>>> > > file
>>> > > > > >> > > strategy
>>> > > > > >> > > > > across checkpoints, which means the state backend
>>> *could*
>>> > > reuse
>>> > > > > >> files
>>> > > > > >> > > > from
>>> > > > > >> > > > > previous cp but not *must* do so. When the state
>>> backend
>>> > > could
>>> > > > > not
>>> > > > > >> > > reuse
>>> > > > > >> > > > > the files, it is reasonable to fallback to a full
>>> > > checkpoint.
>>> > > > > >> > > > >
>>> > > > > >> > > > > Thus, I suggest we make it
>>> > > `execution.checkpointing.incremental`
>>> > > > > >> and
>>> > > > > >> > > > enable
>>> > > > > >> > > > > it by default. For those state backends not supporting
>>> > this,
>>> > > > > they
>>> > > > > >> > > perform
>>> > > > > >> > > > > full checkpoints and print a warning to inform users.
>>> > Users
>>> > > do
>>> > > > > not
>>> > > > > >> > need
>>> > > > > >> > > > to
>>> > > > > >> > > > > pay special attention to different options to control
>>> this
>>> > > > > across
>>> > > > > >> > > > different
>>> > > > > >> > > > > state backends. This is more user-friendly in my
>>> opinion.
>>> > > WDYT?
>>> > > > > >> > > > >
>>> > > > > >> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <
>>> > > 1996fanrui@gmail.com>
>>> > > > > >> > wrote:
>>> > > > > >> > > > >
>>> > > > > >> > > > > > Hi Zakelly,
>>> > > > > >> > > > > >
>>> > > > > >> > > > > > I'm not sure whether we could add the state backend
>>> type
>>> > > in
>>> > > > > the
>>> > > > > >> > > > > > new key name of state.backend.incremental. It means
>>> we
>>> > use
>>> > > > > >> > > > > > `execution.checkpointing.rocksdb-incremental` or
>>> > > > > >> > > > > >
>>> `execution.checkpointing.rocksdb-incremental.enabled`.
>>> > > > > >> > > > > >
>>> > > > > >> > > > > > So far, state.backend.incremental only works for
>>> rocksdb
>>> > > state
>>> > > > > >> > > backend.
>>> > > > > >> > > > > > And this feature or optimization is very valuable
>>> and
>>> > > huge for
>>> > > > > >> > large
>>> > > > > >> > > > > > state flink jobs. I believe it's enabled for most
>>> > > production
>>> > > > > >> flink
>>> > > > > >> > > jobs
>>> > > > > >> > > > > > with large rocksdb state.
>>> > > > > >> > > > > >
>>> > > > > >> > > > > > If this option isn't generic for all state backend
>>> > types,
>>> > > I
>>> > > > > >> guess
>>> > > > > >> > we
>>> > > > > >> > > > > > can enable
>>> > > > > `execution.checkpointing.rocksdb-incremental.enabled`
>>> > > > > >> > > > > > by default in Flink 2.0.
>>> > > > > >> > > > > >
>>> > > > > >> > > > > > But if it works for all state backends, it's hard to
>>> > > enable it
>>> > > > > >> by
>>> > > > > >> > > > > default.
>>> > > > > >> > > > > > Enabling great and valuable features or
>>> improvements are
>>> > > > > useful
>>> > > > > >> > > > > > for users, especially a lot of new flink users.
>>> > > Out-of-the-box
>>> > > > > >> > > options
>>> > > > > >> > > > > > are good for users.
>>> > > > > >> > > > > >
>>> > > > > >> > > > > > WDYT?
>>> > > > > >> > > > > >
>>> > > > > >> > > > > > Best,
>>> > > > > >> > > > > > Rui
>>> > > > > >> > > > > >
>>> > > > > >> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <
>>> > > > > >> zakelly.lan@gmail.com
>>> > > > > >> > >
>>> > > > > >> > > > > wrote:
>>> > > > > >> > > > > >
>>> > > > > >> > > > > > > Hi everyone,
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > Thanks all for your comments!
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > As many of you have questions about the names for
>>> > > boolean
>>> > > > > >> > options,
>>> > > > > >> > > I
>>> > > > > >> > > > > > > suggest we make a naming rule for them. For now I
>>> > could
>>> > > > > think
>>> > > > > >> of
>>> > > > > >> > > > three
>>> > > > > >> > > > > > > options:
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > Option 1: Use enumeration options if possible. But
>>> > this
>>> > > may
>>> > > > > >> cause
>>> > > > > >> > > > some
>>> > > > > >> > > > > > name
>>> > > > > >> > > > > > > collisions or confusion as we discussed and we
>>> should
>>> > > unify
>>> > > > > >> the
>>> > > > > >> > > > > statement
>>> > > > > >> > > > > > > everywhere.
>>> > > > > >> > > > > > > Option 2: Use boolean options and add 'enabled'
>>> as the
>>> > > > > suffix.
>>> > > > > >> > > > > > > Option 3: Use boolean options and ONLY add
>>> 'enabled'
>>> > > when
>>> > > > > >> there
>>> > > > > >> > are
>>> > > > > >> > > > > more
>>> > > > > >> > > > > > > detailed configurations under the same prefix, to
>>> > > prevent
>>> > > > > one
>>> > > > > >> > name
>>> > > > > >> > > > from
>>> > > > > >> > > > > > > serving as a prefix to another.
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > I am slightly inclined to Option 3, since it is
>>> more
>>> > in
>>> > > line
>>> > > > > >> with
>>> > > > > >> > > > > current
>>> > > > > >> > > > > > > practice and friendly for existing users. Also It
>>> > > reduces
>>> > > > > the
>>> > > > > >> > > length
>>> > > > > >> > > > of
>>> > > > > >> > > > > > > configuration names as much as possible. I really
>>> want
>>> > > to
>>> > > > > hear
>>> > > > > >> > your
>>> > > > > >> > > > > > > opinions.
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > @Xuannan
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > I agree with your comments 1 and 3.
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > For 2, If we decide to change the name, maybe
>>> > > > > >> > > > > > > `execution.checkpointing.parallel-cleaner` is
>>> better?
>>> > > And as
>>> > > > > >> for
>>> > > > > >> > > > > whether
>>> > > > > >> > > > > > to
>>> > > > > >> > > > > > > add 'enabled' I suggest we discuss the rule above.
>>> > WDYT?
>>> > > > > >> > > > > > > Thanks!
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > Best,
>>> > > > > >> > > > > > > Zakelly
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
>>> > > > > >> > suxuannan95@gmail.com
>>> > > > > >> > > >
>>> > > > > >> > > > > > wrote:
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > > > > Hi Zakelly,
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > > > Thanks for driving this! The organization of the
>>> > > > > >> configuration
>>> > > > > >> > > > option
>>> > > > > >> > > > > > > > in the FLIP looks much cleaner and easier to
>>> > > understand.
>>> > > > > +1
>>> > > > > >> to
>>> > > > > >> > > the
>>> > > > > >> > > > > > > > FLIP.
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > > > Just some questions from me.
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > > > 1. I think the change to the ConfigOptions
>>> should be
>>> > > put
>>> > > > > in
>>> > > > > >> the
>>> > > > > >> > > > > > > > `Public Interface` section, instead of `Proposed
>>> > > Changed`,
>>> > > > > >> as
>>> > > > > >> > > those
>>> > > > > >> > > > > > > > configuration options are public interface.
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > > > 2. The key
>>> `state.checkpoint.cleaner.parallel-mode`
>>> > > seems
>>> > > > > >> > > > confusing.
>>> > > > > >> > > > > > > > It feels like it is used to choose different
>>> modes.
>>> > In
>>> > > > > >> fact, it
>>> > > > > >> > > is
>>> > > > > >> > > > a
>>> > > > > >> > > > > > > > boolean flag to indicate whether to enable
>>> parallel
>>> > > clean.
>>> > > > > >> How
>>> > > > > >> > > > about
>>> > > > > >> > > > > > > > making it
>>> > > > > `state.checkpoint.cleaner.parallel-mode.enabled`?
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > > > 3. The `execution.checkpointing.write-buffer`
>>> may
>>> > > better
>>> > > > > be
>>> > > > > >> > > > > > > > `execution.checkpointing.write-buffer-size` so
>>> that
>>> > we
>>> > > > > know
>>> > > > > >> it
>>> > > > > >> > is
>>> > > > > >> > > > > > > > configuring the size of the buffer.
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > > > Best,
>>> > > > > >> > > > > > > > Xuannan
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
>>> > > > > >> > fredialei@gmail.com>
>>> > > > > >> > > > > > wrote:
>>> > > > > >> > > > > > > > >
>>> > > > > >> > > > > > > > > Hi Zakelly,
>>> > > > > >> > > > > > > > >
>>> > > > > >> > > > > > > > > > Considering the name occupation, how about
>>> > naming
>>> > > it
>>> > > > > as
>>> > > > > >> > > > > > > > `execution.checkpointing.type`?
>>> > > > > >> > > > > > > > >
>>> > > > > >> > > > > > > > > `Checkpoint Type`[1,2] is used to describe
>>> > > > > >> aligned/unaligned
>>> > > > > >> > > > > > > > > checkpoint, I am inclined to make a choice
>>> between
>>> > > > > >> > > > > > > > > `execution.checkpointing.incremental` and
>>> > > > > >> > > > > > > > > `execution.checkpointing.incremental.enabled`.
>>> > > > > >> > > > > > > > >
>>> > > > > >> > > > > > > > >
>>> > > > > >> > > > > > > > > [1]
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > >
>>> > > > > >> > > > >
>>> > > > > >> > > >
>>> > > > > >> > >
>>> > > > > >> >
>>> > > > > >>
>>> > > > >
>>> > >
>>> >
>>> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
>>> > > > > >> > > > > > > > > [2]
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > >
>>> > > > > >> > > > >
>>> > > > > >> > > >
>>> > > > > >> > >
>>> > > > > >> >
>>> > > > > >>
>>> > > > >
>>> > >
>>> >
>>> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
>>> > > > > >> > > > > > > > >
>>> > > > > >> > > > > > > > > --
>>> > > > > >> > > > > > > > > Best,
>>> > > > > >> > > > > > > > > Yanfei
>>> > > > > >> > > > > > > > >
>>> > > > > >> > > > > > > > > Zakelly Lan <za...@gmail.com>
>>> > 于2023年12月27日周三
>>> > > > > >> 14:41写道:
>>> > > > > >> > > > > > > > > >
>>> > > > > >> > > > > > > > > > Hi Lijie,
>>> > > > > >> > > > > > > > > >
>>> > > > > >> > > > > > > > > > Thanks for the reminder! I missed this.
>>> > > > > >> > > > > > > > > >
>>> > > > > >> > > > > > > > > > Considering the name occupation, how about
>>> > naming
>>> > > it
>>> > > > > as
>>> > > > > >> > > > > > > > > > `execution.checkpointing.type`?
>>> > > > > >> > > > > > > > > >
>>> > > > > >> > > > > > > > > > Actually I think the current
>>> > > > > >> `execution.checkpointing.mode`
>>> > > > > >> > > is
>>> > > > > >> > > > > > > > confusing in
>>> > > > > >> > > > > > > > > > some ways, maybe
>>> > > > > >> `execution.checkpointing.data-consistency`
>>> > > > > >> > > is
>>> > > > > >> > > > > > > better.
>>> > > > > >> > > > > > > > > >
>>> > > > > >> > > > > > > > > >
>>> > > > > >> > > > > > > > > > Best,
>>> > > > > >> > > > > > > > > > Zakelly
>>> > > > > >> > > > > > > > > >
>>> > > > > >> > > > > > > > > >
>>> > > > > >> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang
>>> <
>>> > > > > >> > > > > > > wangdachui9501@gmail.com>
>>> > > > > >> > > > > > > > > > wrote:
>>> > > > > >> > > > > > > > > >
>>> > > > > >> > > > > > > > > > > Hi Zakelly,
>>> > > > > >> > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > >> I'm wondering if
>>> > > > > >> > `execution.checkpointing.savepoint-dir`
>>> > > > > >> > > > > would
>>> > > > > >> > > > > > > be
>>> > > > > >> > > > > > > > > > > better.
>>> > > > > >> > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > `execution.checkpointing.dir` and
>>> > > > > >> > > > > > > > `execution.checkpointing.savepoint-dir`
>>> > > > > >> > > > > > > > > > > are also fine for me.
>>> > > > > >> > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > >> So I think an enumeration option
>>> > > > > >> > > > > > `execution.checkpointing.mode`
>>> > > > > >> > > > > > > > which
>>> > > > > >> > > > > > > > > > > can be 'full' (default) or 'incremental'
>>> would
>>> > > be
>>> > > > > >> better
>>> > > > > >> > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > I agree with using an enumeration option.
>>> But
>>> > > > > >> currently
>>> > > > > >> > > there
>>> > > > > >> > > > > is
>>> > > > > >> > > > > > > > already a
>>> > > > > >> > > > > > > > > > > configuration option called
>>> > > > > >> > `execution.checkpointing.mode`,
>>> > > > > >> > > > > which
>>> > > > > >> > > > > > > is
>>> > > > > >> > > > > > > > used
>>> > > > > >> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE.
>>> Maybe
>>> > > we
>>> > > > > >> need to
>>> > > > > >> > > use
>>> > > > > >> > > > > > > > another name
>>> > > > > >> > > > > > > > > > > or merge these two options.
>>> > > > > >> > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > Best,
>>> > > > > >> > > > > > > > > > > Lijie
>>> > > > > >> > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > Zakelly Lan <za...@gmail.com>
>>> > > 于2023年12月27日周三
>>> > > > > >> > > 11:43写道:
>>> > > > > >> > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > Hi everyone,
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > Thanks all for your comments!
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > @Yanfei
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > 1. For some state backends that do not
>>> > > support
>>> > > > > >> > > > incremental
>>> > > > > >> > > > > > > > checkpoint,
>>> > > > > >> > > > > > > > > > > > > how does the
>>> > > > > >> > execution.checkpointing.incrementaloption
>>> > > > > >> > > > take
>>> > > > > >> > > > > > > > effect? Or
>>> > > > > >> > > > > > > > > > > > > is it better to put incremental under
>>> > > > > >> > > > > > > > state.backend.xxx.incremental?
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > I'd rather not put the option for
>>> > incremental
>>> > > > > >> > checkpoint
>>> > > > > >> > > > > under
>>> > > > > >> > > > > > > the
>>> > > > > >> > > > > > > > > > > > 'state.backend', since it is more about
>>> the
>>> > > > > >> > checkpointing
>>> > > > > >> > > > > > instead
>>> > > > > >> > > > > > > > of
>>> > > > > >> > > > > > > > > > > state
>>> > > > > >> > > > > > > > > > > > accessing. Of course, the state backend
>>> may
>>> > > not
>>> > > > > >> > > necessarily
>>> > > > > >> > > > > do
>>> > > > > >> > > > > > > > > > > incremental
>>> > > > > >> > > > > > > > > > > > checkpoint as requested. If the state
>>> > backend
>>> > > is
>>> > > > > not
>>> > > > > >> > > > capable
>>> > > > > >> > > > > of
>>> > > > > >> > > > > > > > taking
>>> > > > > >> > > > > > > > > > > > incremental cp, it is better to
>>> fallback to
>>> > > the
>>> > > > > full
>>> > > > > >> > cp.
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > 2. I'm a little worried that putting all
>>> > > > > >> configurations
>>> > > > > >> > > > into
>>> > > > > >> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
>>> > > introduce
>>> > > > > >> some
>>> > > > > >> > > > > > dependency
>>> > > > > >> > > > > > > > > > > > > problems. Some options would be used
>>> by
>>> > > > > >> flink-runtime
>>> > > > > >> > > > > module,
>>> > > > > >> > > > > > > but
>>> > > > > >> > > > > > > > > > > > > flink-runtime should not depend on
>>> > > > > >> > > flink-streaming-java.
>>> > > > > >> > > > > e.g.
>>> > > > > >> > > > > > > > > > > > > FLINK-28286[1].
>>> > > > > >> > > > > > > > > > > > > So, I prefer to move configurations to
>>> > > > > >> > > > > > `CheckpointingOptions`,
>>> > > > > >> > > > > > > > WDYT?
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > Yes, that's a very good point.  Moving
>>> to
>>> > > > > >> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes
>>> > > sense.
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > @Lijie
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > How about
>>> > > > > >> > > > > > > > > > > > > state.savepoints.dir ->
>>> > > > > >> > > > > execution.checkpointing.savepoint.dir
>>> > > > > >> > > > > > > > > > > > > state.checkpoints.dir ->
>>> > > > > >> > > > > > execution.checkpointing.checkpoint.dir
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > Actually, I think the
>>> > > `checkpointing.checkpoint`
>>> > > > > may
>>> > > > > >> > > cause
>>> > > > > >> > > > > some
>>> > > > > >> > > > > > > > > > > confusion.
>>> > > > > >> > > > > > > > > > > > But I'm ok if others agree.
>>> > > > > >> > > > > > > > > > > > I'm wondering if
>>> > > > > >> > `execution.checkpointing.savepoint-dir`
>>> > > > > >> > > > > would
>>> > > > > >> > > > > > be
>>> > > > > >> > > > > > > > better.
>>> > > > > >> > > > > > > > > > > > WDYT?
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > 2. We changed the
>>> > > > > >> execution.checkpointing.local-copy'
>>> > > > > >> > to
>>> > > > > >> > > > > > > > > > > > >
>>> > > 'execution.checkpointing.local-copy.enabled'.
>>> > > > > >> Should
>>> > > > > >> > we
>>> > > > > >> > > > > also
>>> > > > > >> > > > > > > add
>>> > > > > >> > > > > > > > > > > > "enabled"
>>> > > > > >> > > > > > > > > > > > > suffix for other boolean type
>>> > configuration
>>> > > > > >> options ?
>>> > > > > >> > > For
>>> > > > > >> > > > > > > > example,
>>> > > > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
>>> > > > > >> > > > > > > > > > > > >
>>> > execution.checkpointing.incremental.enabled
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > Actually, the incremental cp is
>>> something
>>> > like
>>> > > > > >> > choosing a
>>> > > > > >> > > > > mode
>>> > > > > >> > > > > > > for
>>> > > > > >> > > > > > > > doing
>>> > > > > >> > > > > > > > > > > > checkpoint instead of enabling a
>>> function.
>>> > So
>>> > > I
>>> > > > > >> think
>>> > > > > >> > an
>>> > > > > >> > > > > > > > enumeration
>>> > > > > >> > > > > > > > > > > option
>>> > > > > >> > > > > > > > > > > > `execution.checkpointing.mode` which
>>> can be
>>> > > 'full'
>>> > > > > >> > > > (default)
>>> > > > > >> > > > > or
>>> > > > > >> > > > > > > > > > > > 'incremental' would be better, WDYT?
>>> > > > > >> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think
>>> about
>>> > > this?
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie
>>> Wang <
>>> > > > > >> > > > > > > > wangdachui9501@gmail.com>
>>> > > > > >> > > > > > > > > > > > wrote:
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > Hi Zakelly,
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > Thanks for driving the discussion.
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > 1.
>>> > > > > >> > > > > > > > > > > > > >> But I'm not so sure since there is
>>> only
>>> > > one
>>> > > > > >> > > > > > > savepoint-related
>>> > > > > >> > > > > > > > > > > option.
>>> > > > > >> > > > > > > > > > > > > Maybe someone else could share some
>>> > thoughts
>>> > > > > here.
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > How about
>>> > > > > >> > > > > > > > > > > > > state.savepoints.dir ->
>>> > > > > >> > > > > execution.checkpointing.savepoint.dir
>>> > > > > >> > > > > > > > > > > > > state.checkpoints.dir ->
>>> > > > > >> > > > > > execution.checkpointing.checkpoint.dir
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > 2. We changed the
>>> > > > > >> execution.checkpointing.local-copy'
>>> > > > > >> > > to
>>> > > > > >> > > > > > > > > > > > >
>>> > > 'execution.checkpointing.local-copy.enabled'.
>>> > > > > >> Should
>>> > > > > >> > we
>>> > > > > >> > > > > also
>>> > > > > >> > > > > > > add
>>> > > > > >> > > > > > > > > > > > "enabled"
>>> > > > > >> > > > > > > > > > > > > suffix for other boolean type
>>> > configuration
>>> > > > > >> options ?
>>> > > > > >> > > For
>>> > > > > >> > > > > > > > example,
>>> > > > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
>>> > > > > >> > > > > > > > > > > > >
>>> > execution.checkpointing.incremental.enabled
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > In this way, the naming style of
>>> > > configuration
>>> > > > > >> > options
>>> > > > > >> > > is
>>> > > > > >> > > > > > > > unified, and
>>> > > > > >> > > > > > > > > > > it
>>> > > > > >> > > > > > > > > > > > > can avoid potential similar problems
>>> (for
>>> > > > > >> example, we
>>> > > > > >> > > may
>>> > > > > >> > > > > > need
>>> > > > > >> > > > > > > > to add
>>> > > > > >> > > > > > > > > > > > more
>>> > > > > >> > > > > > > > > > > > > options for incremental checkpoint in
>>> the
>>> > > > > future).
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > Best,
>>> > > > > >> > > > > > > > > > > > > Lijie
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > Yanfei Lei <fr...@gmail.com>
>>> > > 于2023年12月26日周二
>>> > > > > >> > > 12:05写道:
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > Hi Zakelly,
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > Thank you for creating the FLIP and
>>> > > starting
>>> > > > > the
>>> > > > > >> > > > > > discussion.
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > The current arrangement of these
>>> options
>>> > > is
>>> > > > > >> indeed
>>> > > > > >> > > > > somewhat
>>> > > > > >> > > > > > > > > > > haphazard,
>>> > > > > >> > > > > > > > > > > > > > and the new arrangement looks much
>>> > > better. I
>>> > > > > >> have
>>> > > > > >> > > some
>>> > > > > >> > > > > > > > questions
>>> > > > > >> > > > > > > > > > > about
>>> > > > > >> > > > > > > > > > > > > > the arrangement of some new
>>> > configuration
>>> > > > > >> options:
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > 1. For some state backends that do
>>> not
>>> > > support
>>> > > > > >> > > > > incremental
>>> > > > > >> > > > > > > > > > > checkpoint,
>>> > > > > >> > > > > > > > > > > > > > how does the
>>> > > > > >> > > execution.checkpointing.incrementaloption
>>> > > > > >> > > > > take
>>> > > > > >> > > > > > > > effect?
>>> > > > > >> > > > > > > > > > > Or
>>> > > > > >> > > > > > > > > > > > > > is it better to put incremental
>>> under
>>> > > > > >> > > > > > > > state.backend.xxx.incremental?
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > 2. I'm a little worried that
>>> putting all
>>> > > > > >> > > configurations
>>> > > > > >> > > > > > into
>>> > > > > >> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
>>> > > introduce
>>> > > > > >> some
>>> > > > > >> > > > > > > dependency
>>> > > > > >> > > > > > > > > > > > > > problems. Some options would be
>>> used by
>>> > > > > >> > flink-runtime
>>> > > > > >> > > > > > module,
>>> > > > > >> > > > > > > > but
>>> > > > > >> > > > > > > > > > > > > > flink-runtime should not depend on
>>> > > > > >> > > > flink-streaming-java.
>>> > > > > >> > > > > > e.g.
>>> > > > > >> > > > > > > > > > > > > > FLINK-28286[1].
>>> > > > > >> > > > > > > > > > > > > > So, I prefer to move configurations
>>> to
>>> > > > > >> > > > > > > `CheckpointingOptions`,
>>> > > > > >> > > > > > > > WDYT?
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > [1]
>>> > > > > >> > > https://issues.apache.org/jira/browse/FLINK-28286
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > --
>>> > > > > >> > > > > > > > > > > > > > Best,
>>> > > > > >> > > > > > > > > > > > > > Yanfei
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > Zakelly Lan <za...@gmail.com>
>>> > > > > >> 于2023年12月25日周一
>>> > > > > >> > > > > > 21:14写道:
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > Hi Rui Fan and Junrui,
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > Thanks for the reminder! I agree
>>> to
>>> > > change
>>> > > > > the
>>> > > > > >> > > > > > > > > > > > > > >
>>> 'execution.checkpointing.local-copy'
>>> > to
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > 'execution.checkpointing.local-copy.enabled'.
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > And for other suggestions Rui
>>> > proposed:
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > 1. How about
>>> > > > > >> execution.checkpointing.storage.type
>>> > > > > >> > > > > instead
>>> > > > > >> > > > > > > > > > > > > > > > of
>>> execution.checkpointing.storage?
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > Ah, I missed something here.
>>> Actually
>>> > I
>>> > > > > >> suggest
>>> > > > > >> > we
>>> > > > > >> > > > > could
>>> > > > > >> > > > > > > > merge the
>>> > > > > >> > > > > > > > > > > > > > current
>>> > > > > >> > > > > > > > > > > > > > > 'state.checkpoints.dir' and
>>> > > > > >> > > > 'state.checkpoint-storage'
>>> > > > > >> > > > > > into
>>> > > > > >> > > > > > > > one URI
>>> > > > > >> > > > > > > > > > > > > > > configuration named
>>> > > > > >> > 'execution.checkpointing.dir'.
>>> > > > > >> > > > > WDYT?
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > 3.
>>> > > execution.checkpointing.savepoint.dir is
>>> > > > > a
>>> > > > > >> > > little
>>> > > > > >> > > > > > weird.
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > Yes, I think it is better to make
>>> > > > > 'savepoint'
>>> > > > > >> and
>>> > > > > >> > > > > > > > 'checkpoint' the
>>> > > > > >> > > > > > > > > > > > same
>>> > > > > >> > > > > > > > > > > > > > > level. But I'm not so sure since
>>> there
>>> > > is
>>> > > > > only
>>> > > > > >> > one
>>> > > > > >> > > > > > > > > > > savepoint-related
>>> > > > > >> > > > > > > > > > > > > > > option. Maybe someone else could
>>> share
>>> > > some
>>> > > > > >> > > thoughts
>>> > > > > >> > > > > > here.
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > 4. How about
>>> > > execution.recovery.claim-mode
>>> > > > > >> > instead
>>> > > > > >> > > of
>>> > > > > >> > > > > > > > > > > > > > > > execution.recovery.mode?
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > >  Agreed. That's more accurate.
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > Many thanks for your suggestions!
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > Best,
>>> > > > > >> > > > > > > > > > > > > > > Zakelly
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM
>>> Junrui
>>> > > Lee <
>>> > > > > >> > > > > > > > jrlee.ljr@gmail.com>
>>> > > > > >> > > > > > > > > > > > > wrote:
>>> > > > > >> > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > Hi Zakelly,
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > Thanks for driving this. I agree
>>> > that
>>> > > the
>>> > > > > >> > > proposed
>>> > > > > >> > > > > > > > restructuring
>>> > > > > >> > > > > > > > > > > of
>>> > > > > >> > > > > > > > > > > > > the
>>> > > > > >> > > > > > > > > > > > > > > > configuration options is largely
>>> > > positive.
>>> > > > > >> It
>>> > > > > >> > > will
>>> > > > > >> > > > > make
>>> > > > > >> > > > > > > > > > > > understanding
>>> > > > > >> > > > > > > > > > > > > > and
>>> > > > > >> > > > > > > > > > > > > > > > working with Flink
>>> configurations
>>> > more
>>> > > > > >> > intuitive.
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > Most of the proposed changes
>>> look
>>> > > great.
>>> > > > > >> Just a
>>> > > > > >> > > > > > heads-up,
>>> > > > > >> > > > > > > > as Rui
>>> > > > > >> > > > > > > > > > > > Fan
>>> > > > > >> > > > > > > > > > > > > > > > mentioned, Flink currently
>>> requires
>>> > > that
>>> > > > > no
>>> > > > > >> > > > > > > configOption's
>>> > > > > >> > > > > > > > key be
>>> > > > > >> > > > > > > > > > > > the
>>> > > > > >> > > > > > > > > > > > > > > > prefix of another to avoid
>>> issues
>>> > > when we
>>> > > > > >> > > > eventually
>>> > > > > >> > > > > > > adopt
>>> > > > > >> > > > > > > > a
>>> > > > > >> > > > > > > > > > > > standard
>>> > > > > >> > > > > > > > > > > > > > YAML
>>> > > > > >> > > > > > > > > > > > > > > > parser, as detailed in
>>> FLINK-29372 (
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > https://issues.apache.org/jira/browse/FLINK-29372
>>> > > > > >> > > > ).
>>> > > > > >> > > > > > > > Therefore,
>>> > > > > >> > > > > > > > > > > > it's
>>> > > > > >> > > > > > > > > > > > > > better
>>> > > > > >> > > > > > > > > > > > > > > > to change the key
>>> > > > > >> > > > > 'execution.checkpointing.local-copy'
>>> > > > > >> > > > > > > > because it
>>> > > > > >> > > > > > > > > > > > > > serves as
>>> > > > > >> > > > > > > > > > > > > > > > a prefix to the key
>>> > > > > >> > > > > > > > 'execution.checkpointing.local-copy.dir'.
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > Best regards,
>>> > > > > >> > > > > > > > > > > > > > > > Junrui
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > Rui Fan <19...@gmail.com>
>>> > > > > >> 于2023年12月25日周一
>>> > > > > >> > > > > 19:11写道:
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > Hi Zakelly,
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > Thank you for driving this
>>> > proposal!
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > Overall good for me. I have
>>> some
>>> > > > > questions
>>> > > > > >> > > about
>>> > > > > >> > > > > > these
>>> > > > > >> > > > > > > > names.
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > 1. How about
>>> > > > > >> > > execution.checkpointing.storage.type
>>> > > > > >> > > > > > > > instead of
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> execution.checkpointing.storage?
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > It's similar to
>>> > state.backend.type.
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > 2. How about
>>> > > > > >> > > > > > execution.checkpointing.local-copy.enabled
>>> > > > > >> > > > > > > > instead
>>> > > > > >> > > > > > > > > > > > of
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > execution.checkpointing.local-copy?
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > You added a new option:
>>> > > > > >> > > > > > > > execution.checkpointing.local-copy.dir.
>>> > > > > >> > > > > > > > > > > > > > > > > IIUC, one option name
>>> shouldn't be
>>> > > the
>>> > > > > >> prefix
>>> > > > > >> > > of
>>> > > > > >> > > > > > other
>>> > > > > >> > > > > > > > options.
>>> > > > > >> > > > > > > > > > > > > > > > > If you add a new option
>>> > > > > >> > > > > > > > execution.checkpointing.local-copy,
>>> > > > > >> > > > > > > > > > > > > > > > > flink CI will fail directly.
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > 3.
>>> > > execution.checkpointing.savepoint.dir
>>> > > > > >> is a
>>> > > > > >> > > > > little
>>> > > > > >> > > > > > > > weird.
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > For old options:
>>> > > state.savepoints.dir
>>> > > > > and
>>> > > > > >> > > > > > > > > > > state.checkpoints.dir,
>>> > > > > >> > > > > > > > > > > > > > > > > the savepoint and checkpoint
>>> are
>>> > the
>>> > > > > same
>>> > > > > >> > > level.
>>> > > > > >> > > > It
>>> > > > > >> > > > > > > means
>>> > > > > >> > > > > > > > > > > > > > > > > it's a checkpoint or
>>> savepoint.
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > The new option
>>> > > > > >> execution.checkpointing.dir is
>>> > > > > >> > > > fine
>>> > > > > >> > > > > > for
>>> > > > > >> > > > > > > > me.
>>> > > > > >> > > > > > > > > > > > > > > > > However,
>>> > > > > >> > execution.checkpointing.savepoint.dir
>>> > > > > >> > > > is a
>>> > > > > >> > > > > > > > little
>>> > > > > >> > > > > > > > > > > weird.
>>> > > > > >> > > > > > > > > > > > > > > > > I don't know which name is
>>> better
>>> > > now.
>>> > > > > >> Let us
>>> > > > > >> > > > think
>>> > > > > >> > > > > > > > about it
>>> > > > > >> > > > > > > > > > > > more.
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > 4. How about
>>> > > > > execution.recovery.claim-mode
>>> > > > > >> > > > instead
>>> > > > > >> > > > > of
>>> > > > > >> > > > > > > > > > > > > > > > > execution.recovery.mode?
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > The meaning of mode is too
>>> broad.
>>> > > The
>>> > > > > >> > > claim-mode
>>> > > > > >> > > > > may
>>> > > > > >> > > > > > > > > > > > > > > > > be more accurate for users.
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > WDYT?
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > Best,
>>> > > > > >> > > > > > > > > > > > > > > > > Rui
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at
>>> 5:14 PM
>>> > > Zakelly
>>> > > > > >> Lan <
>>> > > > > >> > > > > > > > > > > > zakelly.lan@gmail.com
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > wrote:
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > > Hi devs,
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > > I'd like to start a
>>> discussion
>>> > on
>>> > > > > >> FLIP-406:
>>> > > > > >> > > > > > > Reorganize
>>> > > > > >> > > > > > > > State
>>> > > > > >> > > > > > > > > > > &
>>> > > > > >> > > > > > > > > > > > > > > > > > Checkpointing & Recovery
>>> > > > > >> Configuration[1].
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > > Currently, the configuration
>>> > > options
>>> > > > > >> > > pertaining
>>> > > > > >> > > > > to
>>> > > > > >> > > > > > > > > > > > checkpointing,
>>> > > > > >> > > > > > > > > > > > > > > > > recovery,
>>> > > > > >> > > > > > > > > > > > > > > > > > and state management are
>>> > primarily
>>> > > > > >> grouped
>>> > > > > >> > > > under
>>> > > > > >> > > > > > the
>>> > > > > >> > > > > > > > > > > following
>>> > > > > >> > > > > > > > > > > > > > > > prefixes:
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > >    - state.backend.* :
>>> > > configurations
>>> > > > > >> > related
>>> > > > > >> > > > to
>>> > > > > >> > > > > > > state
>>> > > > > >> > > > > > > > > > > > accessing
>>> > > > > >> > > > > > > > > > > > > > and
>>> > > > > >> > > > > > > > > > > > > > > > > >    checkpointing, as well as
>>> > > specific
>>> > > > > >> > options
>>> > > > > >> > > > for
>>> > > > > >> > > > > > > > individual
>>> > > > > >> > > > > > > > > > > > > state
>>> > > > > >> > > > > > > > > > > > > > > > > backends
>>> > > > > >> > > > > > > > > > > > > > > > > >    -
>>> execution.checkpointing.* :
>>> > > > > >> > > configurations
>>> > > > > >> > > > > > > > associated
>>> > > > > >> > > > > > > > > > > with
>>> > > > > >> > > > > > > > > > > > > > > > > checkpoint
>>> > > > > >> > > > > > > > > > > > > > > > > >    execution and recovery
>>> > > > > >> > > > > > > > > > > > > > > > > >    - execution.savepoint.*:
>>> > > > > >> configurations
>>> > > > > >> > > for
>>> > > > > >> > > > > > > > recovery from
>>> > > > > >> > > > > > > > > > > > > > savepoint
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > > In addition, there are
>>> several
>>> > > > > >> individual
>>> > > > > >> > > > options
>>> > > > > >> > > > > > > such
>>> > > > > >> > > > > > > > as '
>>> > > > > >> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*'
>>> and
>>> > > > > >> > > > > > > > '*state.checkpoints.dir*'
>>> > > > > >> > > > > > > > > > > that
>>> > > > > >> > > > > > > > > > > > > fall
>>> > > > > >> > > > > > > > > > > > > > > > > outside
>>> > > > > >> > > > > > > > > > > > > > > > > > of these prefixes. The
>>> current
>>> > > > > >> arrangement
>>> > > > > >> > of
>>> > > > > >> > > > > these
>>> > > > > >> > > > > > > > options,
>>> > > > > >> > > > > > > > > > > > > which
>>> > > > > >> > > > > > > > > > > > > > span
>>> > > > > >> > > > > > > > > > > > > > > > > > multiple modules, is
>>> somewhat
>>> > > > > haphazard
>>> > > > > >> and
>>> > > > > >> > > > > lacks a
>>> > > > > >> > > > > > > > > > > systematic
>>> > > > > >> > > > > > > > > > > > > > > > structure.
>>> > > > > >> > > > > > > > > > > > > > > > > > For example, the options
>>> under
>>> > the
>>> > > > > >> > > > > > > > '*CheckpointingOptions*'
>>> > > > > >> > > > > > > > > > > > and '
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> *ExecutionCheckpointingOptions*'
>>> > > are
>>> > > > > >> > related
>>> > > > > >> > > > and
>>> > > > > >> > > > > > have
>>> > > > > >> > > > > > > > no
>>> > > > > >> > > > > > > > > > > clear
>>> > > > > >> > > > > > > > > > > > > > > > boundaries
>>> > > > > >> > > > > > > > > > > > > > > > > > from the user's
>>> perspective, but
>>> > > there
>>> > > > > >> is
>>> > > > > >> > no
>>> > > > > >> > > > > > unified
>>> > > > > >> > > > > > > > prefix
>>> > > > > >> > > > > > > > > > > for
>>> > > > > >> > > > > > > > > > > > > > them.
>>> > > > > >> > > > > > > > > > > > > > > > > With
>>> > > > > >> > > > > > > > > > > > > > > > > > the upcoming release of
>>> Flink
>>> > > 2.0, we
>>> > > > > >> have
>>> > > > > >> > an
>>> > > > > >> > > > > > > excellent
>>> > > > > >> > > > > > > > > > > > > > opportunity to
>>> > > > > >> > > > > > > > > > > > > > > > > > overhaul and restructure the
>>> > > > > >> configurations
>>> > > > > >> > > > > related
>>> > > > > >> > > > > > > to
>>> > > > > >> > > > > > > > > > > > > > checkpointing,
>>> > > > > >> > > > > > > > > > > > > > > > > > recovery, and state
>>> management.
>>> > > This
>>> > > > > >> FLIP
>>> > > > > >> > > > > proposes
>>> > > > > >> > > > > > to
>>> > > > > >> > > > > > > > > > > > reorganize
>>> > > > > >> > > > > > > > > > > > > > these
>>> > > > > >> > > > > > > > > > > > > > > > > > settings, making it more
>>> > coherent
>>> > > by
>>> > > > > >> > module,
>>> > > > > >> > > > > which
>>> > > > > >> > > > > > > > would
>>> > > > > >> > > > > > > > > > > > > > significantly
>>> > > > > >> > > > > > > > > > > > > > > > > > lower the barriers for
>>> > > understanding
>>> > > > > and
>>> > > > > >> > > reduce
>>> > > > > >> > > > > the
>>> > > > > >> > > > > > > > > > > development
>>> > > > > >> > > > > > > > > > > > > > costs
>>> > > > > >> > > > > > > > > > > > > > > > > > moving forward.
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > > Looking forward to hearing
>>> from
>>> > > you!
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > > [1]
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > >
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > >
>>> > > > > >> > > > >
>>> > > > > >> > > >
>>> > > > > >> > >
>>> > > > > >> >
>>> > > > > >>
>>> > > > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > > > Best,
>>> > > > > >> > > > > > > > > > > > > > > > > > Zakelly
>>> > > > > >> > > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > > >
>>> > > > > >> > > > > > > > > > >
>>> > > > > >> > > > > > > >
>>> > > > > >> > > > > > >
>>> > > > > >> > > > > >
>>> > > > > >> > > > >
>>> > > > > >> > > >
>>> > > > > >> > >
>>> > > > > >> >
>>> > > > > >>
>>> > > > > >>
>>> > > > > >> --
>>> > > > > >> Best,
>>> > > > > >> Hangxiang.
>>> > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Best,
>>> > > > Hangxiang.
>>> > >
>>> > >
>>> >
>>>
>>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi everyone,

It has been 6 days since the last call for discussion. I'd like to start a
vote after another 2 days.

Please let me know if you have any concerns. Thanks!


Best,
Zakelly

On Tue, Jan 16, 2024 at 2:54 PM Zakelly Lan <za...@gmail.com> wrote:

> Thanks for the suggestion Rui!  The type is added.
>
>
> Best,
> Zakelly
>
> On Tue, Jan 16, 2024 at 2:33 PM Rui Fan <19...@gmail.com> wrote:
>
>> Hi Zakelly,
>>
>> Would you mind adding the option type in the FLIP doc?
>> For example, String, Boolean or Enum, etc. Thank you.
>>
>> Best,
>> Rui
>>
>> On Tue, Jan 16, 2024 at 2:29 PM Zakelly Lan <za...@gmail.com>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > Thanks all for joining the discussion! I'd like to speed this up since
>> it
>> > lasts for nearly a month. I made changes on this FLIP based on
>> suggestions
>> > and compromises acceptable to most people. Please feel free to give your
>> > opinion. Thanks!
>> > If there are no more suggestions, I will consider starting a vote
>> within a
>> > week.
>> >
>> >
>> > Best,
>> > Zakelly
>> >
>> > On Thu, Jan 11, 2024 at 10:31 AM Xuannan Su <su...@gmail.com>
>> wrote:
>> >
>> > > Hi Zakelly,
>> > >
>> > > I am fine with either Option 2 or Option 3. I think the naming in
>> > > Option 2 makes it clear that it is a boolean configuration. However,
>> > > most of the currently available boolean configurations do not use
>> > > "enable" as a suffix. Therefore, Option 3 looks good to me as it
>> > > follows the current practice.
>> > >
>> > > Best regards,
>> > > Xuannan
>> > >
>> > > On Thu, Jan 11, 2024 at 9:50 AM Hangxiang Yu <ma...@gmail.com>
>> > wrote:
>> > > >
>> > > > >
>> > > > > That's a very good point. I realize that the word 'recovery' means
>> > way
>> > > too
>> > > > > many things. So I suggest picking a more specific word here, how
>> > about
>> > > > > 'execution.state-recovery.*' ? Checkpointing and state recovery
>> are
>> > > > > corresponding terms and won't make ambiguity.
>> > > > >
>> > > >
>> > > > This makes the configuration clearer to me. We could focus on the
>> > > > `state-recovery` at first.
>> > > >
>> > > > I think we could create another FLIP for the deprecation of LEGACY
>> > mode.
>> > > > >
>> > > >
>> > > > LGTM, Let's create a new FLIP to do this.
>> > > >
>> > > > IIUC, there is no clear ownership of the local copy files from the
>> > > previous
>> > > > > job and it's better to define one. This needs more discussion so
>> we
>> > > could
>> > > > > create another thread for this. WDYT?
>> > > > >
>> > > >
>> > > > Yeah, I have created a new ticket FLINK-34032 to track and discuss
>> > this.
>> > > >
>> > > > On Wed, Jan 10, 2024 at 6:31 PM Zakelly Lan <za...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > Hi everyone,
>> > > > >
>> > > > > It seems we still don't have a consensus on the rules for boolean
>> > type
>> > > > > options. Let me recap the alternatives we have:
>> > > > >
>> > > > > Option 1: Use enumeration options instead if possible. But this
>> may
>> > > cause
>> > > > > some name collisions or confusion as we discussed and we should
>> unify
>> > > the
>> > > > > statement everywhere.
>> > > > > Option 2: Use boolean options and add 'enabled' as the suffix.
>> > > > > Option 3: Use boolean options and ONLY add 'enabled' when there
>> are
>> > > more
>> > > > > detailed configurations under the same prefix, to prevent one name
>> > from
>> > > > > serving as a prefix to another.
>> > > > >
>> > > > > I am inclined to Option 3, since it is more in line with current
>> > > practice
>> > > > > and friendly for existing users. Also It reduces the length of
>> > > > > configuration names as much as possible.
>> > > > >
>> > > > > Looking forward to your opinions! Thanks!
>> > > > >
>> > > > >
>> > > > > Best,
>> > > > > Zakelly
>> > > > >
>> > > > > On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan <
>> zakelly.lan@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi Hangxiang,
>> > > > > >
>> > > > > > Thanks for your suggestions!
>> > > > > >
>> > > > > > 1. Could execution.recovery also contain some other behaviors
>> about
>> > > > > >> recovery ? e.g. restart-strategy.
>> > > > > >
>> > > > > >
>> > > > > > That's a very good point. I realize that the word 'recovery'
>> means
>> > > way
>> > > > > too
>> > > > > > many things. So I suggest picking a more specific word here, how
>> > > about
>> > > > > > 'execution.state-recovery.*' ? Checkpointing and state recovery
>> are
>> > > > > > corresponding terms and won't make ambiguity.
>> > > > > >
>> > > > > > 2. Could we also remove some legacy configuration value ? e.g.
>> > LEGACY
>> > > > > Mode
>> > > > > >> for
>> > execution.savepoint-restore-mode/execution.recovery.claim-mode.
>> > > > > >
>> > > > > >
>> > > > > > I think we could create another FLIP for the deprecation of
>> LEGACY
>> > > mode.
>> > > > > >
>> > > > > >
>> > > > > >> 3. Could the local checkpoint be cleaned
>> > > > > >> if execution.checkpointing.local-copy.enabled is true and
>> > > > > >> execution.recovery.from-local is false ? I found it's also an
>> > issue
>> > > if
>> > > > > >> current local-recovery from enabled to disabled. Maybe another
>> > > ticket is
>> > > > > >> needed.
>> > > > > >
>> > > > > >
>> > > > > > IIUC, there is no clear ownership of the local copy files from
>> the
>> > > > > > previous job and it's better to define one. This needs more
>> > > discussion so
>> > > > > > we could create another thread for this. WDYT?
>> > > > > >
>> > > > > >
>> > > > > > Best,
>> > > > > > Zakelly
>> > > > > >
>> > > > > > On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu <
>> masteryhx@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > >> Hi, Zakelly.
>> > > > > >> Thanks for driving this. Overall LGTM as we discussed offline.
>> > > > > >>
>> > > > > >> Some comments/suggestions just came to mind:
>> > > > > >> 1. Could execution.recovery also contain some other behaviors
>> > about
>> > > > > >> recovery ? e.g. restart-strategy.
>> > > > > >> 2. Could we also remove some legacy configuration value ? e.g.
>> > > LEGACY
>> > > > > Mode
>> > > > > >> for
>> > execution.savepoint-restore-mode/execution.recovery.claim-mode.
>> > > > > >> 3. Could the local checkpoint be cleaned
>> > > > > >> if execution.checkpointing.local-copy.enabled is true and
>> > > > > >> execution.recovery.from-local is false ? I found it's also an
>> > issue
>> > > if
>> > > > > >> current local-recovery from enabled to disabled. Maybe another
>> > > ticket is
>> > > > > >> needed.
>> > > > > >> 4. +1 for enabling execution.checkpointing.incremental by
>> default
>> > > which
>> > > > > is
>> > > > > >> basically default configuration in our production environment.
>> > > > > >>
>> > > > > >>
>> > > > > >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <
>> zakelly.lan@gmail.com
>> > >
>> > > > > wrote:
>> > > > > >>
>> > > > > >> > Hi Yun,
>> > > > > >> >
>> > > > > >> > Thanks for your comments!
>> > > > > >> >
>> > > > > >> >  1.  We shall not describe the configuration with its
>> > > implementation
>> > > > > for
>> > > > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
>> > > > > >> > state-backend,
>> > > > > >> > > it would write two streams and for Rocksdb state-backend,
>> it
>> > > would
>> > > > > use
>> > > > > >> > > hard-link for backup. Thus, I think
>> > > > > >> > > 'execution.checkpointing.local-backup.*' looks better.
>> > > > > >> >
>> > > > > >> > I agreed that we'd better name the option in user's
>> perspective
>> > > > > instead
>> > > > > >> of
>> > > > > >> > the implementation, thus I name it as a copy of the
>> checkpoint
>> > in
>> > > the
>> > > > > >> > local disk, regardless of the way of generating it. The word
>> > > 'backup'
>> > > > > is
>> > > > > >> > also suitable for this case, so I agree to change to
>> > > > > >> > 'execution.checkpointing.local-backup.*' if no one objects.
>> > > > > >> >
>> > > > > >> >  2.  What does the
>> > 'execution.checkpointing.data-inline-threshold'
>> > > > > >> mean? It
>> > > > > >> > > seems not so easy to understand.
>> > > > > >> >
>> > > > > >> > The 'execution.checkpointing.data-inline-threshold' (original
>> > one
>> > > as
>> > > > > >> > 'state.storage.fs.memory-threshold') stands for the size
>> > threshold
>> > > > > below
>> > > > > >> > which state chunks will store inline with the metadata, thus
>> I
>> > > call it
>> > > > > >> > 'data-inline-threshold'.
>> > > > > >> >
>> > > > > >> >
>> > > > > >> > Best,
>> > > > > >> > Zakelly
>> > > > > >> >
>> > > > > >> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com>
>> > > wrote:
>> > > > > >> >
>> > > > > >> > > Hi Zakelly,
>> > > > > >> > >
>> > > > > >> > > Thanks for driving this topic. I have two concerns here:
>> > > > > >> > >
>> > > > > >> > >   1.  We shall not describe the configuration with its
>> > > > > implementation
>> > > > > >> for
>> > > > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
>> > > > > >> > state-backend,
>> > > > > >> > > it would write two streams and for Rocksdb state-backend,
>> it
>> > > would
>> > > > > use
>> > > > > >> > > hard-link for backup. Thus, I think
>> > > > > >> > > 'execution.checkpointing.local-backup.*' looks better.
>> > > > > >> > >   2.  What does the
>> > > 'execution.checkpointing.data-inline-threshold'
>> > > > > >> mean?
>> > > > > >> > > It seems not so easy to understand.
>> > > > > >> > >
>> > > > > >> > > Best
>> > > > > >> > > Yun Tang
>> > > > > >> > > ________________________________
>> > > > > >> > > From: Piotr Nowojski <pn...@apache.org>
>> > > > > >> > > Sent: Thursday, January 4, 2024 22:37
>> > > > > >> > > To: dev@flink.apache.org <de...@flink.apache.org>
>> > > > > >> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State &
>> > > Checkpointing &
>> > > > > >> > > Recovery Configuration
>> > > > > >> > >
>> > > > > >> > > Hi,
>> > > > > >> > >
>> > > > > >> > > Thanks for trying to clean this up! I don't have strong
>> > > opinions on
>> > > > > >> the
>> > > > > >> > > topics discussed here, so generally speaking +1 from my
>> side!
>> > > > > >> > >
>> > > > > >> > > Best,
>> > > > > >> > > Piotrek
>> > > > > >> > >
>> > > > > >> > > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com>
>> > > napisał(a):
>> > > > > >> > >
>> > > > > >> > > > Thanks for the feedback!
>> > > > > >> > > >
>> > > > > >> > > > Using the `execution.checkpointing.incremental.enabled`,
>> > > > > >> > > > and enabling it by default sounds good to me.
>> > > > > >> > > >
>> > > > > >> > > > Best,
>> > > > > >> > > > Rui
>> > > > > >> > > >
>> > > > > >> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <
>> > > > > zakelly.lan@gmail.com>
>> > > > > >> > > wrote:
>> > > > > >> > > >
>> > > > > >> > > > > Hi Rui,
>> > > > > >> > > > >
>> > > > > >> > > > > Thanks for your comments!
>> > > > > >> > > > >
>> > > > > >> > > > > IMO, given that the state backend can be plugably
>> loaded
>> > > (as you
>> > > > > >> can
>> > > > > >> > > > > specify a state backend factory), I prefer not
>> providing
>> > > state
>> > > > > >> > backend
>> > > > > >> > > > > specified options in the framework.
>> > > > > >> > > > >
>> > > > > >> > > > > Secondly, the incremental checkpoint is actually a
>> sharing
>> > > file
>> > > > > >> > > strategy
>> > > > > >> > > > > across checkpoints, which means the state backend
>> *could*
>> > > reuse
>> > > > > >> files
>> > > > > >> > > > from
>> > > > > >> > > > > previous cp but not *must* do so. When the state
>> backend
>> > > could
>> > > > > not
>> > > > > >> > > reuse
>> > > > > >> > > > > the files, it is reasonable to fallback to a full
>> > > checkpoint.
>> > > > > >> > > > >
>> > > > > >> > > > > Thus, I suggest we make it
>> > > `execution.checkpointing.incremental`
>> > > > > >> and
>> > > > > >> > > > enable
>> > > > > >> > > > > it by default. For those state backends not supporting
>> > this,
>> > > > > they
>> > > > > >> > > perform
>> > > > > >> > > > > full checkpoints and print a warning to inform users.
>> > Users
>> > > do
>> > > > > not
>> > > > > >> > need
>> > > > > >> > > > to
>> > > > > >> > > > > pay special attention to different options to control
>> this
>> > > > > across
>> > > > > >> > > > different
>> > > > > >> > > > > state backends. This is more user-friendly in my
>> opinion.
>> > > WDYT?
>> > > > > >> > > > >
>> > > > > >> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <
>> > > 1996fanrui@gmail.com>
>> > > > > >> > wrote:
>> > > > > >> > > > >
>> > > > > >> > > > > > Hi Zakelly,
>> > > > > >> > > > > >
>> > > > > >> > > > > > I'm not sure whether we could add the state backend
>> type
>> > > in
>> > > > > the
>> > > > > >> > > > > > new key name of state.backend.incremental. It means
>> we
>> > use
>> > > > > >> > > > > > `execution.checkpointing.rocksdb-incremental` or
>> > > > > >> > > > > >
>> `execution.checkpointing.rocksdb-incremental.enabled`.
>> > > > > >> > > > > >
>> > > > > >> > > > > > So far, state.backend.incremental only works for
>> rocksdb
>> > > state
>> > > > > >> > > backend.
>> > > > > >> > > > > > And this feature or optimization is very valuable and
>> > > huge for
>> > > > > >> > large
>> > > > > >> > > > > > state flink jobs. I believe it's enabled for most
>> > > production
>> > > > > >> flink
>> > > > > >> > > jobs
>> > > > > >> > > > > > with large rocksdb state.
>> > > > > >> > > > > >
>> > > > > >> > > > > > If this option isn't generic for all state backend
>> > types,
>> > > I
>> > > > > >> guess
>> > > > > >> > we
>> > > > > >> > > > > > can enable
>> > > > > `execution.checkpointing.rocksdb-incremental.enabled`
>> > > > > >> > > > > > by default in Flink 2.0.
>> > > > > >> > > > > >
>> > > > > >> > > > > > But if it works for all state backends, it's hard to
>> > > enable it
>> > > > > >> by
>> > > > > >> > > > > default.
>> > > > > >> > > > > > Enabling great and valuable features or improvements
>> are
>> > > > > useful
>> > > > > >> > > > > > for users, especially a lot of new flink users.
>> > > Out-of-the-box
>> > > > > >> > > options
>> > > > > >> > > > > > are good for users.
>> > > > > >> > > > > >
>> > > > > >> > > > > > WDYT?
>> > > > > >> > > > > >
>> > > > > >> > > > > > Best,
>> > > > > >> > > > > > Rui
>> > > > > >> > > > > >
>> > > > > >> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <
>> > > > > >> zakelly.lan@gmail.com
>> > > > > >> > >
>> > > > > >> > > > > wrote:
>> > > > > >> > > > > >
>> > > > > >> > > > > > > Hi everyone,
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > Thanks all for your comments!
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > As many of you have questions about the names for
>> > > boolean
>> > > > > >> > options,
>> > > > > >> > > I
>> > > > > >> > > > > > > suggest we make a naming rule for them. For now I
>> > could
>> > > > > think
>> > > > > >> of
>> > > > > >> > > > three
>> > > > > >> > > > > > > options:
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > Option 1: Use enumeration options if possible. But
>> > this
>> > > may
>> > > > > >> cause
>> > > > > >> > > > some
>> > > > > >> > > > > > name
>> > > > > >> > > > > > > collisions or confusion as we discussed and we
>> should
>> > > unify
>> > > > > >> the
>> > > > > >> > > > > statement
>> > > > > >> > > > > > > everywhere.
>> > > > > >> > > > > > > Option 2: Use boolean options and add 'enabled' as
>> the
>> > > > > suffix.
>> > > > > >> > > > > > > Option 3: Use boolean options and ONLY add
>> 'enabled'
>> > > when
>> > > > > >> there
>> > > > > >> > are
>> > > > > >> > > > > more
>> > > > > >> > > > > > > detailed configurations under the same prefix, to
>> > > prevent
>> > > > > one
>> > > > > >> > name
>> > > > > >> > > > from
>> > > > > >> > > > > > > serving as a prefix to another.
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > I am slightly inclined to Option 3, since it is
>> more
>> > in
>> > > line
>> > > > > >> with
>> > > > > >> > > > > current
>> > > > > >> > > > > > > practice and friendly for existing users. Also It
>> > > reduces
>> > > > > the
>> > > > > >> > > length
>> > > > > >> > > > of
>> > > > > >> > > > > > > configuration names as much as possible. I really
>> want
>> > > to
>> > > > > hear
>> > > > > >> > your
>> > > > > >> > > > > > > opinions.
>> > > > > >> > > > > > >
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > @Xuannan
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > I agree with your comments 1 and 3.
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > For 2, If we decide to change the name, maybe
>> > > > > >> > > > > > > `execution.checkpointing.parallel-cleaner` is
>> better?
>> > > And as
>> > > > > >> for
>> > > > > >> > > > > whether
>> > > > > >> > > > > > to
>> > > > > >> > > > > > > add 'enabled' I suggest we discuss the rule above.
>> > WDYT?
>> > > > > >> > > > > > > Thanks!
>> > > > > >> > > > > > >
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > Best,
>> > > > > >> > > > > > > Zakelly
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
>> > > > > >> > suxuannan95@gmail.com
>> > > > > >> > > >
>> > > > > >> > > > > > wrote:
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > > Hi Zakelly,
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > Thanks for driving this! The organization of the
>> > > > > >> configuration
>> > > > > >> > > > option
>> > > > > >> > > > > > > > in the FLIP looks much cleaner and easier to
>> > > understand.
>> > > > > +1
>> > > > > >> to
>> > > > > >> > > the
>> > > > > >> > > > > > > > FLIP.
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > Just some questions from me.
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > 1. I think the change to the ConfigOptions
>> should be
>> > > put
>> > > > > in
>> > > > > >> the
>> > > > > >> > > > > > > > `Public Interface` section, instead of `Proposed
>> > > Changed`,
>> > > > > >> as
>> > > > > >> > > those
>> > > > > >> > > > > > > > configuration options are public interface.
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > 2. The key
>> `state.checkpoint.cleaner.parallel-mode`
>> > > seems
>> > > > > >> > > > confusing.
>> > > > > >> > > > > > > > It feels like it is used to choose different
>> modes.
>> > In
>> > > > > >> fact, it
>> > > > > >> > > is
>> > > > > >> > > > a
>> > > > > >> > > > > > > > boolean flag to indicate whether to enable
>> parallel
>> > > clean.
>> > > > > >> How
>> > > > > >> > > > about
>> > > > > >> > > > > > > > making it
>> > > > > `state.checkpoint.cleaner.parallel-mode.enabled`?
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > 3. The `execution.checkpointing.write-buffer` may
>> > > better
>> > > > > be
>> > > > > >> > > > > > > > `execution.checkpointing.write-buffer-size` so
>> that
>> > we
>> > > > > know
>> > > > > >> it
>> > > > > >> > is
>> > > > > >> > > > > > > > configuring the size of the buffer.
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > Best,
>> > > > > >> > > > > > > > Xuannan
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
>> > > > > >> > fredialei@gmail.com>
>> > > > > >> > > > > > wrote:
>> > > > > >> > > > > > > > >
>> > > > > >> > > > > > > > > Hi Zakelly,
>> > > > > >> > > > > > > > >
>> > > > > >> > > > > > > > > > Considering the name occupation, how about
>> > naming
>> > > it
>> > > > > as
>> > > > > >> > > > > > > > `execution.checkpointing.type`?
>> > > > > >> > > > > > > > >
>> > > > > >> > > > > > > > > `Checkpoint Type`[1,2] is used to describe
>> > > > > >> aligned/unaligned
>> > > > > >> > > > > > > > > checkpoint, I am inclined to make a choice
>> between
>> > > > > >> > > > > > > > > `execution.checkpointing.incremental` and
>> > > > > >> > > > > > > > > `execution.checkpointing.incremental.enabled`.
>> > > > > >> > > > > > > > >
>> > > > > >> > > > > > > > >
>> > > > > >> > > > > > > > > [1]
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > >
>> > > > > >> > > > > >
>> > > > > >> > > > >
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
>> > > > > >> > > > > > > > > [2]
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > >
>> > > > > >> > > > > >
>> > > > > >> > > > >
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
>> > > > > >> > > > > > > > >
>> > > > > >> > > > > > > > > --
>> > > > > >> > > > > > > > > Best,
>> > > > > >> > > > > > > > > Yanfei
>> > > > > >> > > > > > > > >
>> > > > > >> > > > > > > > > Zakelly Lan <za...@gmail.com>
>> > 于2023年12月27日周三
>> > > > > >> 14:41写道:
>> > > > > >> > > > > > > > > >
>> > > > > >> > > > > > > > > > Hi Lijie,
>> > > > > >> > > > > > > > > >
>> > > > > >> > > > > > > > > > Thanks for the reminder! I missed this.
>> > > > > >> > > > > > > > > >
>> > > > > >> > > > > > > > > > Considering the name occupation, how about
>> > naming
>> > > it
>> > > > > as
>> > > > > >> > > > > > > > > > `execution.checkpointing.type`?
>> > > > > >> > > > > > > > > >
>> > > > > >> > > > > > > > > > Actually I think the current
>> > > > > >> `execution.checkpointing.mode`
>> > > > > >> > > is
>> > > > > >> > > > > > > > confusing in
>> > > > > >> > > > > > > > > > some ways, maybe
>> > > > > >> `execution.checkpointing.data-consistency`
>> > > > > >> > > is
>> > > > > >> > > > > > > better.
>> > > > > >> > > > > > > > > >
>> > > > > >> > > > > > > > > >
>> > > > > >> > > > > > > > > > Best,
>> > > > > >> > > > > > > > > > Zakelly
>> > > > > >> > > > > > > > > >
>> > > > > >> > > > > > > > > >
>> > > > > >> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
>> > > > > >> > > > > > > wangdachui9501@gmail.com>
>> > > > > >> > > > > > > > > > wrote:
>> > > > > >> > > > > > > > > >
>> > > > > >> > > > > > > > > > > Hi Zakelly,
>> > > > > >> > > > > > > > > > >
>> > > > > >> > > > > > > > > > > >> I'm wondering if
>> > > > > >> > `execution.checkpointing.savepoint-dir`
>> > > > > >> > > > > would
>> > > > > >> > > > > > > be
>> > > > > >> > > > > > > > > > > better.
>> > > > > >> > > > > > > > > > >
>> > > > > >> > > > > > > > > > > `execution.checkpointing.dir` and
>> > > > > >> > > > > > > > `execution.checkpointing.savepoint-dir`
>> > > > > >> > > > > > > > > > > are also fine for me.
>> > > > > >> > > > > > > > > > >
>> > > > > >> > > > > > > > > > > >> So I think an enumeration option
>> > > > > >> > > > > > `execution.checkpointing.mode`
>> > > > > >> > > > > > > > which
>> > > > > >> > > > > > > > > > > can be 'full' (default) or 'incremental'
>> would
>> > > be
>> > > > > >> better
>> > > > > >> > > > > > > > > > >
>> > > > > >> > > > > > > > > > > I agree with using an enumeration option.
>> But
>> > > > > >> currently
>> > > > > >> > > there
>> > > > > >> > > > > is
>> > > > > >> > > > > > > > already a
>> > > > > >> > > > > > > > > > > configuration option called
>> > > > > >> > `execution.checkpointing.mode`,
>> > > > > >> > > > > which
>> > > > > >> > > > > > > is
>> > > > > >> > > > > > > > used
>> > > > > >> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE.
>> Maybe
>> > > we
>> > > > > >> need to
>> > > > > >> > > use
>> > > > > >> > > > > > > > another name
>> > > > > >> > > > > > > > > > > or merge these two options.
>> > > > > >> > > > > > > > > > >
>> > > > > >> > > > > > > > > > > Best,
>> > > > > >> > > > > > > > > > > Lijie
>> > > > > >> > > > > > > > > > >
>> > > > > >> > > > > > > > > > > Zakelly Lan <za...@gmail.com>
>> > > 于2023年12月27日周三
>> > > > > >> > > 11:43写道:
>> > > > > >> > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > Hi everyone,
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > Thanks all for your comments!
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > @Yanfei
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > 1. For some state backends that do not
>> > > support
>> > > > > >> > > > incremental
>> > > > > >> > > > > > > > checkpoint,
>> > > > > >> > > > > > > > > > > > > how does the
>> > > > > >> > execution.checkpointing.incrementaloption
>> > > > > >> > > > take
>> > > > > >> > > > > > > > effect? Or
>> > > > > >> > > > > > > > > > > > > is it better to put incremental under
>> > > > > >> > > > > > > > state.backend.xxx.incremental?
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > I'd rather not put the option for
>> > incremental
>> > > > > >> > checkpoint
>> > > > > >> > > > > under
>> > > > > >> > > > > > > the
>> > > > > >> > > > > > > > > > > > 'state.backend', since it is more about
>> the
>> > > > > >> > checkpointing
>> > > > > >> > > > > > instead
>> > > > > >> > > > > > > > of
>> > > > > >> > > > > > > > > > > state
>> > > > > >> > > > > > > > > > > > accessing. Of course, the state backend
>> may
>> > > not
>> > > > > >> > > necessarily
>> > > > > >> > > > > do
>> > > > > >> > > > > > > > > > > incremental
>> > > > > >> > > > > > > > > > > > checkpoint as requested. If the state
>> > backend
>> > > is
>> > > > > not
>> > > > > >> > > > capable
>> > > > > >> > > > > of
>> > > > > >> > > > > > > > taking
>> > > > > >> > > > > > > > > > > > incremental cp, it is better to fallback
>> to
>> > > the
>> > > > > full
>> > > > > >> > cp.
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > 2. I'm a little worried that putting all
>> > > > > >> configurations
>> > > > > >> > > > into
>> > > > > >> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
>> > > introduce
>> > > > > >> some
>> > > > > >> > > > > > dependency
>> > > > > >> > > > > > > > > > > > > problems. Some options would be used by
>> > > > > >> flink-runtime
>> > > > > >> > > > > module,
>> > > > > >> > > > > > > but
>> > > > > >> > > > > > > > > > > > > flink-runtime should not depend on
>> > > > > >> > > flink-streaming-java.
>> > > > > >> > > > > e.g.
>> > > > > >> > > > > > > > > > > > > FLINK-28286[1].
>> > > > > >> > > > > > > > > > > > > So, I prefer to move configurations to
>> > > > > >> > > > > > `CheckpointingOptions`,
>> > > > > >> > > > > > > > WDYT?
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > Yes, that's a very good point.  Moving to
>> > > > > >> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes
>> > > sense.
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > @Lijie
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > How about
>> > > > > >> > > > > > > > > > > > > state.savepoints.dir ->
>> > > > > >> > > > > execution.checkpointing.savepoint.dir
>> > > > > >> > > > > > > > > > > > > state.checkpoints.dir ->
>> > > > > >> > > > > > execution.checkpointing.checkpoint.dir
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > Actually, I think the
>> > > `checkpointing.checkpoint`
>> > > > > may
>> > > > > >> > > cause
>> > > > > >> > > > > some
>> > > > > >> > > > > > > > > > > confusion.
>> > > > > >> > > > > > > > > > > > But I'm ok if others agree.
>> > > > > >> > > > > > > > > > > > I'm wondering if
>> > > > > >> > `execution.checkpointing.savepoint-dir`
>> > > > > >> > > > > would
>> > > > > >> > > > > > be
>> > > > > >> > > > > > > > better.
>> > > > > >> > > > > > > > > > > > WDYT?
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > 2. We changed the
>> > > > > >> execution.checkpointing.local-copy'
>> > > > > >> > to
>> > > > > >> > > > > > > > > > > > >
>> > > 'execution.checkpointing.local-copy.enabled'.
>> > > > > >> Should
>> > > > > >> > we
>> > > > > >> > > > > also
>> > > > > >> > > > > > > add
>> > > > > >> > > > > > > > > > > > "enabled"
>> > > > > >> > > > > > > > > > > > > suffix for other boolean type
>> > configuration
>> > > > > >> options ?
>> > > > > >> > > For
>> > > > > >> > > > > > > > example,
>> > > > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
>> > > > > >> > > > > > > > > > > > >
>> > execution.checkpointing.incremental.enabled
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > Actually, the incremental cp is something
>> > like
>> > > > > >> > choosing a
>> > > > > >> > > > > mode
>> > > > > >> > > > > > > for
>> > > > > >> > > > > > > > doing
>> > > > > >> > > > > > > > > > > > checkpoint instead of enabling a
>> function.
>> > So
>> > > I
>> > > > > >> think
>> > > > > >> > an
>> > > > > >> > > > > > > > enumeration
>> > > > > >> > > > > > > > > > > option
>> > > > > >> > > > > > > > > > > > `execution.checkpointing.mode` which can
>> be
>> > > 'full'
>> > > > > >> > > > (default)
>> > > > > >> > > > > or
>> > > > > >> > > > > > > > > > > > 'incremental' would be better, WDYT?
>> > > > > >> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think
>> about
>> > > this?
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie
>> Wang <
>> > > > > >> > > > > > > > wangdachui9501@gmail.com>
>> > > > > >> > > > > > > > > > > > wrote:
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > Hi Zakelly,
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > Thanks for driving the discussion.
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > 1.
>> > > > > >> > > > > > > > > > > > > >> But I'm not so sure since there is
>> only
>> > > one
>> > > > > >> > > > > > > savepoint-related
>> > > > > >> > > > > > > > > > > option.
>> > > > > >> > > > > > > > > > > > > Maybe someone else could share some
>> > thoughts
>> > > > > here.
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > How about
>> > > > > >> > > > > > > > > > > > > state.savepoints.dir ->
>> > > > > >> > > > > execution.checkpointing.savepoint.dir
>> > > > > >> > > > > > > > > > > > > state.checkpoints.dir ->
>> > > > > >> > > > > > execution.checkpointing.checkpoint.dir
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > 2. We changed the
>> > > > > >> execution.checkpointing.local-copy'
>> > > > > >> > > to
>> > > > > >> > > > > > > > > > > > >
>> > > 'execution.checkpointing.local-copy.enabled'.
>> > > > > >> Should
>> > > > > >> > we
>> > > > > >> > > > > also
>> > > > > >> > > > > > > add
>> > > > > >> > > > > > > > > > > > "enabled"
>> > > > > >> > > > > > > > > > > > > suffix for other boolean type
>> > configuration
>> > > > > >> options ?
>> > > > > >> > > For
>> > > > > >> > > > > > > > example,
>> > > > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
>> > > > > >> > > > > > > > > > > > >
>> > execution.checkpointing.incremental.enabled
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > In this way, the naming style of
>> > > configuration
>> > > > > >> > options
>> > > > > >> > > is
>> > > > > >> > > > > > > > unified, and
>> > > > > >> > > > > > > > > > > it
>> > > > > >> > > > > > > > > > > > > can avoid potential similar problems
>> (for
>> > > > > >> example, we
>> > > > > >> > > may
>> > > > > >> > > > > > need
>> > > > > >> > > > > > > > to add
>> > > > > >> > > > > > > > > > > > more
>> > > > > >> > > > > > > > > > > > > options for incremental checkpoint in
>> the
>> > > > > future).
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > Best,
>> > > > > >> > > > > > > > > > > > > Lijie
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > Yanfei Lei <fr...@gmail.com>
>> > > 于2023年12月26日周二
>> > > > > >> > > 12:05写道:
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > Hi Zakelly,
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > Thank you for creating the FLIP and
>> > > starting
>> > > > > the
>> > > > > >> > > > > > discussion.
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > The current arrangement of these
>> options
>> > > is
>> > > > > >> indeed
>> > > > > >> > > > > somewhat
>> > > > > >> > > > > > > > > > > haphazard,
>> > > > > >> > > > > > > > > > > > > > and the new arrangement looks much
>> > > better. I
>> > > > > >> have
>> > > > > >> > > some
>> > > > > >> > > > > > > > questions
>> > > > > >> > > > > > > > > > > about
>> > > > > >> > > > > > > > > > > > > > the arrangement of some new
>> > configuration
>> > > > > >> options:
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > 1. For some state backends that do
>> not
>> > > support
>> > > > > >> > > > > incremental
>> > > > > >> > > > > > > > > > > checkpoint,
>> > > > > >> > > > > > > > > > > > > > how does the
>> > > > > >> > > execution.checkpointing.incrementaloption
>> > > > > >> > > > > take
>> > > > > >> > > > > > > > effect?
>> > > > > >> > > > > > > > > > > Or
>> > > > > >> > > > > > > > > > > > > > is it better to put incremental under
>> > > > > >> > > > > > > > state.backend.xxx.incremental?
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > 2. I'm a little worried that putting
>> all
>> > > > > >> > > configurations
>> > > > > >> > > > > > into
>> > > > > >> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
>> > > introduce
>> > > > > >> some
>> > > > > >> > > > > > > dependency
>> > > > > >> > > > > > > > > > > > > > problems. Some options would be used
>> by
>> > > > > >> > flink-runtime
>> > > > > >> > > > > > module,
>> > > > > >> > > > > > > > but
>> > > > > >> > > > > > > > > > > > > > flink-runtime should not depend on
>> > > > > >> > > > flink-streaming-java.
>> > > > > >> > > > > > e.g.
>> > > > > >> > > > > > > > > > > > > > FLINK-28286[1].
>> > > > > >> > > > > > > > > > > > > > So, I prefer to move configurations
>> to
>> > > > > >> > > > > > > `CheckpointingOptions`,
>> > > > > >> > > > > > > > WDYT?
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > [1]
>> > > > > >> > > https://issues.apache.org/jira/browse/FLINK-28286
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > --
>> > > > > >> > > > > > > > > > > > > > Best,
>> > > > > >> > > > > > > > > > > > > > Yanfei
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > Zakelly Lan <za...@gmail.com>
>> > > > > >> 于2023年12月25日周一
>> > > > > >> > > > > > 21:14写道:
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > Hi Rui Fan and Junrui,
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > Thanks for the reminder! I agree to
>> > > change
>> > > > > the
>> > > > > >> > > > > > > > > > > > > > >
>> 'execution.checkpointing.local-copy'
>> > to
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > 'execution.checkpointing.local-copy.enabled'.
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > And for other suggestions Rui
>> > proposed:
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > 1. How about
>> > > > > >> execution.checkpointing.storage.type
>> > > > > >> > > > > instead
>> > > > > >> > > > > > > > > > > > > > > > of
>> execution.checkpointing.storage?
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > Ah, I missed something here.
>> Actually
>> > I
>> > > > > >> suggest
>> > > > > >> > we
>> > > > > >> > > > > could
>> > > > > >> > > > > > > > merge the
>> > > > > >> > > > > > > > > > > > > > current
>> > > > > >> > > > > > > > > > > > > > > 'state.checkpoints.dir' and
>> > > > > >> > > > 'state.checkpoint-storage'
>> > > > > >> > > > > > into
>> > > > > >> > > > > > > > one URI
>> > > > > >> > > > > > > > > > > > > > > configuration named
>> > > > > >> > 'execution.checkpointing.dir'.
>> > > > > >> > > > > WDYT?
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > 3.
>> > > execution.checkpointing.savepoint.dir is
>> > > > > a
>> > > > > >> > > little
>> > > > > >> > > > > > weird.
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > Yes, I think it is better to make
>> > > > > 'savepoint'
>> > > > > >> and
>> > > > > >> > > > > > > > 'checkpoint' the
>> > > > > >> > > > > > > > > > > > same
>> > > > > >> > > > > > > > > > > > > > > level. But I'm not so sure since
>> there
>> > > is
>> > > > > only
>> > > > > >> > one
>> > > > > >> > > > > > > > > > > savepoint-related
>> > > > > >> > > > > > > > > > > > > > > option. Maybe someone else could
>> share
>> > > some
>> > > > > >> > > thoughts
>> > > > > >> > > > > > here.
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > 4. How about
>> > > execution.recovery.claim-mode
>> > > > > >> > instead
>> > > > > >> > > of
>> > > > > >> > > > > > > > > > > > > > > > execution.recovery.mode?
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > >  Agreed. That's more accurate.
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > Many thanks for your suggestions!
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > Best,
>> > > > > >> > > > > > > > > > > > > > > Zakelly
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM
>> Junrui
>> > > Lee <
>> > > > > >> > > > > > > > jrlee.ljr@gmail.com>
>> > > > > >> > > > > > > > > > > > > wrote:
>> > > > > >> > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > Hi Zakelly,
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > Thanks for driving this. I agree
>> > that
>> > > the
>> > > > > >> > > proposed
>> > > > > >> > > > > > > > restructuring
>> > > > > >> > > > > > > > > > > of
>> > > > > >> > > > > > > > > > > > > the
>> > > > > >> > > > > > > > > > > > > > > > configuration options is largely
>> > > positive.
>> > > > > >> It
>> > > > > >> > > will
>> > > > > >> > > > > make
>> > > > > >> > > > > > > > > > > > understanding
>> > > > > >> > > > > > > > > > > > > > and
>> > > > > >> > > > > > > > > > > > > > > > working with Flink configurations
>> > more
>> > > > > >> > intuitive.
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > Most of the proposed changes look
>> > > great.
>> > > > > >> Just a
>> > > > > >> > > > > > heads-up,
>> > > > > >> > > > > > > > as Rui
>> > > > > >> > > > > > > > > > > > Fan
>> > > > > >> > > > > > > > > > > > > > > > mentioned, Flink currently
>> requires
>> > > that
>> > > > > no
>> > > > > >> > > > > > > configOption's
>> > > > > >> > > > > > > > key be
>> > > > > >> > > > > > > > > > > > the
>> > > > > >> > > > > > > > > > > > > > > > prefix of another to avoid issues
>> > > when we
>> > > > > >> > > > eventually
>> > > > > >> > > > > > > adopt
>> > > > > >> > > > > > > > a
>> > > > > >> > > > > > > > > > > > standard
>> > > > > >> > > > > > > > > > > > > > YAML
>> > > > > >> > > > > > > > > > > > > > > > parser, as detailed in
>> FLINK-29372 (
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > https://issues.apache.org/jira/browse/FLINK-29372
>> > > > > >> > > > ).
>> > > > > >> > > > > > > > Therefore,
>> > > > > >> > > > > > > > > > > > it's
>> > > > > >> > > > > > > > > > > > > > better
>> > > > > >> > > > > > > > > > > > > > > > to change the key
>> > > > > >> > > > > 'execution.checkpointing.local-copy'
>> > > > > >> > > > > > > > because it
>> > > > > >> > > > > > > > > > > > > > serves as
>> > > > > >> > > > > > > > > > > > > > > > a prefix to the key
>> > > > > >> > > > > > > > 'execution.checkpointing.local-copy.dir'.
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > Best regards,
>> > > > > >> > > > > > > > > > > > > > > > Junrui
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > Rui Fan <19...@gmail.com>
>> > > > > >> 于2023年12月25日周一
>> > > > > >> > > > > 19:11写道:
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > Hi Zakelly,
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > Thank you for driving this
>> > proposal!
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > Overall good for me. I have
>> some
>> > > > > questions
>> > > > > >> > > about
>> > > > > >> > > > > > these
>> > > > > >> > > > > > > > names.
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > 1. How about
>> > > > > >> > > execution.checkpointing.storage.type
>> > > > > >> > > > > > > > instead of
>> > > > > >> > > > > > > > > > > > > > > > >
>> execution.checkpointing.storage?
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > It's similar to
>> > state.backend.type.
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > 2. How about
>> > > > > >> > > > > > execution.checkpointing.local-copy.enabled
>> > > > > >> > > > > > > > instead
>> > > > > >> > > > > > > > > > > > of
>> > > > > >> > > > > > > > > > > > > > > > >
>> > execution.checkpointing.local-copy?
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > You added a new option:
>> > > > > >> > > > > > > > execution.checkpointing.local-copy.dir.
>> > > > > >> > > > > > > > > > > > > > > > > IIUC, one option name
>> shouldn't be
>> > > the
>> > > > > >> prefix
>> > > > > >> > > of
>> > > > > >> > > > > > other
>> > > > > >> > > > > > > > options.
>> > > > > >> > > > > > > > > > > > > > > > > If you add a new option
>> > > > > >> > > > > > > > execution.checkpointing.local-copy,
>> > > > > >> > > > > > > > > > > > > > > > > flink CI will fail directly.
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > 3.
>> > > execution.checkpointing.savepoint.dir
>> > > > > >> is a
>> > > > > >> > > > > little
>> > > > > >> > > > > > > > weird.
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > For old options:
>> > > state.savepoints.dir
>> > > > > and
>> > > > > >> > > > > > > > > > > state.checkpoints.dir,
>> > > > > >> > > > > > > > > > > > > > > > > the savepoint and checkpoint
>> are
>> > the
>> > > > > same
>> > > > > >> > > level.
>> > > > > >> > > > It
>> > > > > >> > > > > > > means
>> > > > > >> > > > > > > > > > > > > > > > > it's a checkpoint or savepoint.
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > The new option
>> > > > > >> execution.checkpointing.dir is
>> > > > > >> > > > fine
>> > > > > >> > > > > > for
>> > > > > >> > > > > > > > me.
>> > > > > >> > > > > > > > > > > > > > > > > However,
>> > > > > >> > execution.checkpointing.savepoint.dir
>> > > > > >> > > > is a
>> > > > > >> > > > > > > > little
>> > > > > >> > > > > > > > > > > weird.
>> > > > > >> > > > > > > > > > > > > > > > > I don't know which name is
>> better
>> > > now.
>> > > > > >> Let us
>> > > > > >> > > > think
>> > > > > >> > > > > > > > about it
>> > > > > >> > > > > > > > > > > > more.
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > 4. How about
>> > > > > execution.recovery.claim-mode
>> > > > > >> > > > instead
>> > > > > >> > > > > of
>> > > > > >> > > > > > > > > > > > > > > > > execution.recovery.mode?
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > The meaning of mode is too
>> broad.
>> > > The
>> > > > > >> > > claim-mode
>> > > > > >> > > > > may
>> > > > > >> > > > > > > > > > > > > > > > > be more accurate for users.
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > WDYT?
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > Best,
>> > > > > >> > > > > > > > > > > > > > > > > Rui
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM
>> > > Zakelly
>> > > > > >> Lan <
>> > > > > >> > > > > > > > > > > > zakelly.lan@gmail.com
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > wrote:
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > > Hi devs,
>> > > > > >> > > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > > I'd like to start a
>> discussion
>> > on
>> > > > > >> FLIP-406:
>> > > > > >> > > > > > > Reorganize
>> > > > > >> > > > > > > > State
>> > > > > >> > > > > > > > > > > &
>> > > > > >> > > > > > > > > > > > > > > > > > Checkpointing & Recovery
>> > > > > >> Configuration[1].
>> > > > > >> > > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > > Currently, the configuration
>> > > options
>> > > > > >> > > pertaining
>> > > > > >> > > > > to
>> > > > > >> > > > > > > > > > > > checkpointing,
>> > > > > >> > > > > > > > > > > > > > > > > recovery,
>> > > > > >> > > > > > > > > > > > > > > > > > and state management are
>> > primarily
>> > > > > >> grouped
>> > > > > >> > > > under
>> > > > > >> > > > > > the
>> > > > > >> > > > > > > > > > > following
>> > > > > >> > > > > > > > > > > > > > > > prefixes:
>> > > > > >> > > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > >    - state.backend.* :
>> > > configurations
>> > > > > >> > related
>> > > > > >> > > > to
>> > > > > >> > > > > > > state
>> > > > > >> > > > > > > > > > > > accessing
>> > > > > >> > > > > > > > > > > > > > and
>> > > > > >> > > > > > > > > > > > > > > > > >    checkpointing, as well as
>> > > specific
>> > > > > >> > options
>> > > > > >> > > > for
>> > > > > >> > > > > > > > individual
>> > > > > >> > > > > > > > > > > > > state
>> > > > > >> > > > > > > > > > > > > > > > > backends
>> > > > > >> > > > > > > > > > > > > > > > > >    -
>> execution.checkpointing.* :
>> > > > > >> > > configurations
>> > > > > >> > > > > > > > associated
>> > > > > >> > > > > > > > > > > with
>> > > > > >> > > > > > > > > > > > > > > > > checkpoint
>> > > > > >> > > > > > > > > > > > > > > > > >    execution and recovery
>> > > > > >> > > > > > > > > > > > > > > > > >    - execution.savepoint.*:
>> > > > > >> configurations
>> > > > > >> > > for
>> > > > > >> > > > > > > > recovery from
>> > > > > >> > > > > > > > > > > > > > savepoint
>> > > > > >> > > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > > In addition, there are
>> several
>> > > > > >> individual
>> > > > > >> > > > options
>> > > > > >> > > > > > > such
>> > > > > >> > > > > > > > as '
>> > > > > >> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*'
>> and
>> > > > > >> > > > > > > > '*state.checkpoints.dir*'
>> > > > > >> > > > > > > > > > > that
>> > > > > >> > > > > > > > > > > > > fall
>> > > > > >> > > > > > > > > > > > > > > > > outside
>> > > > > >> > > > > > > > > > > > > > > > > > of these prefixes. The
>> current
>> > > > > >> arrangement
>> > > > > >> > of
>> > > > > >> > > > > these
>> > > > > >> > > > > > > > options,
>> > > > > >> > > > > > > > > > > > > which
>> > > > > >> > > > > > > > > > > > > > span
>> > > > > >> > > > > > > > > > > > > > > > > > multiple modules, is somewhat
>> > > > > haphazard
>> > > > > >> and
>> > > > > >> > > > > lacks a
>> > > > > >> > > > > > > > > > > systematic
>> > > > > >> > > > > > > > > > > > > > > > structure.
>> > > > > >> > > > > > > > > > > > > > > > > > For example, the options
>> under
>> > the
>> > > > > >> > > > > > > > '*CheckpointingOptions*'
>> > > > > >> > > > > > > > > > > > and '
>> > > > > >> > > > > > > > > > > > > > > > > >
>> *ExecutionCheckpointingOptions*'
>> > > are
>> > > > > >> > related
>> > > > > >> > > > and
>> > > > > >> > > > > > have
>> > > > > >> > > > > > > > no
>> > > > > >> > > > > > > > > > > clear
>> > > > > >> > > > > > > > > > > > > > > > boundaries
>> > > > > >> > > > > > > > > > > > > > > > > > from the user's perspective,
>> but
>> > > there
>> > > > > >> is
>> > > > > >> > no
>> > > > > >> > > > > > unified
>> > > > > >> > > > > > > > prefix
>> > > > > >> > > > > > > > > > > for
>> > > > > >> > > > > > > > > > > > > > them.
>> > > > > >> > > > > > > > > > > > > > > > > With
>> > > > > >> > > > > > > > > > > > > > > > > > the upcoming release of Flink
>> > > 2.0, we
>> > > > > >> have
>> > > > > >> > an
>> > > > > >> > > > > > > excellent
>> > > > > >> > > > > > > > > > > > > > opportunity to
>> > > > > >> > > > > > > > > > > > > > > > > > overhaul and restructure the
>> > > > > >> configurations
>> > > > > >> > > > > related
>> > > > > >> > > > > > > to
>> > > > > >> > > > > > > > > > > > > > checkpointing,
>> > > > > >> > > > > > > > > > > > > > > > > > recovery, and state
>> management.
>> > > This
>> > > > > >> FLIP
>> > > > > >> > > > > proposes
>> > > > > >> > > > > > to
>> > > > > >> > > > > > > > > > > > reorganize
>> > > > > >> > > > > > > > > > > > > > these
>> > > > > >> > > > > > > > > > > > > > > > > > settings, making it more
>> > coherent
>> > > by
>> > > > > >> > module,
>> > > > > >> > > > > which
>> > > > > >> > > > > > > > would
>> > > > > >> > > > > > > > > > > > > > significantly
>> > > > > >> > > > > > > > > > > > > > > > > > lower the barriers for
>> > > understanding
>> > > > > and
>> > > > > >> > > reduce
>> > > > > >> > > > > the
>> > > > > >> > > > > > > > > > > development
>> > > > > >> > > > > > > > > > > > > > costs
>> > > > > >> > > > > > > > > > > > > > > > > > moving forward.
>> > > > > >> > > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > > Looking forward to hearing
>> from
>> > > you!
>> > > > > >> > > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > > [1]
>> > > > > >> > > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > >
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > >
>> > > > > >> > > > > >
>> > > > > >> > > > >
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
>> > > > > >> > > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > > > Best,
>> > > > > >> > > > > > > > > > > > > > > > > > Zakelly
>> > > > > >> > > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > > >
>> > > > > >> > > > > > > > > > > >
>> > > > > >> > > > > > > > > > >
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > >
>> > > > > >> > > > > >
>> > > > > >> > > > >
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > > >>
>> > > > > >> --
>> > > > > >> Best,
>> > > > > >> Hangxiang.
>> > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Best,
>> > > > Hangxiang.
>> > >
>> > >
>> >
>>
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Thanks for the suggestion Rui!  The type is added.


Best,
Zakelly

On Tue, Jan 16, 2024 at 2:33 PM Rui Fan <19...@gmail.com> wrote:

> Hi Zakelly,
>
> Would you mind adding the option type in the FLIP doc?
> For example, String, Boolean or Enum, etc. Thank you.
>
> Best,
> Rui
>
> On Tue, Jan 16, 2024 at 2:29 PM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > Thanks all for joining the discussion! I'd like to speed this up since it
> > lasts for nearly a month. I made changes on this FLIP based on
> suggestions
> > and compromises acceptable to most people. Please feel free to give your
> > opinion. Thanks!
> > If there are no more suggestions, I will consider starting a vote within
> a
> > week.
> >
> >
> > Best,
> > Zakelly
> >
> > On Thu, Jan 11, 2024 at 10:31 AM Xuannan Su <su...@gmail.com>
> wrote:
> >
> > > Hi Zakelly,
> > >
> > > I am fine with either Option 2 or Option 3. I think the naming in
> > > Option 2 makes it clear that it is a boolean configuration. However,
> > > most of the currently available boolean configurations do not use
> > > "enable" as a suffix. Therefore, Option 3 looks good to me as it
> > > follows the current practice.
> > >
> > > Best regards,
> > > Xuannan
> > >
> > > On Thu, Jan 11, 2024 at 9:50 AM Hangxiang Yu <ma...@gmail.com>
> > wrote:
> > > >
> > > > >
> > > > > That's a very good point. I realize that the word 'recovery' means
> > way
> > > too
> > > > > many things. So I suggest picking a more specific word here, how
> > about
> > > > > 'execution.state-recovery.*' ? Checkpointing and state recovery are
> > > > > corresponding terms and won't make ambiguity.
> > > > >
> > > >
> > > > This makes the configuration clearer to me. We could focus on the
> > > > `state-recovery` at first.
> > > >
> > > > I think we could create another FLIP for the deprecation of LEGACY
> > mode.
> > > > >
> > > >
> > > > LGTM, Let's create a new FLIP to do this.
> > > >
> > > > IIUC, there is no clear ownership of the local copy files from the
> > > previous
> > > > > job and it's better to define one. This needs more discussion so we
> > > could
> > > > > create another thread for this. WDYT?
> > > > >
> > > >
> > > > Yeah, I have created a new ticket FLINK-34032 to track and discuss
> > this.
> > > >
> > > > On Wed, Jan 10, 2024 at 6:31 PM Zakelly Lan <za...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > It seems we still don't have a consensus on the rules for boolean
> > type
> > > > > options. Let me recap the alternatives we have:
> > > > >
> > > > > Option 1: Use enumeration options instead if possible. But this may
> > > cause
> > > > > some name collisions or confusion as we discussed and we should
> unify
> > > the
> > > > > statement everywhere.
> > > > > Option 2: Use boolean options and add 'enabled' as the suffix.
> > > > > Option 3: Use boolean options and ONLY add 'enabled' when there are
> > > more
> > > > > detailed configurations under the same prefix, to prevent one name
> > from
> > > > > serving as a prefix to another.
> > > > >
> > > > > I am inclined to Option 3, since it is more in line with current
> > > practice
> > > > > and friendly for existing users. Also It reduces the length of
> > > > > configuration names as much as possible.
> > > > >
> > > > > Looking forward to your opinions! Thanks!
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan <zakelly.lan@gmail.com
> >
> > > wrote:
> > > > >
> > > > > > Hi Hangxiang,
> > > > > >
> > > > > > Thanks for your suggestions!
> > > > > >
> > > > > > 1. Could execution.recovery also contain some other behaviors
> about
> > > > > >> recovery ? e.g. restart-strategy.
> > > > > >
> > > > > >
> > > > > > That's a very good point. I realize that the word 'recovery'
> means
> > > way
> > > > > too
> > > > > > many things. So I suggest picking a more specific word here, how
> > > about
> > > > > > 'execution.state-recovery.*' ? Checkpointing and state recovery
> are
> > > > > > corresponding terms and won't make ambiguity.
> > > > > >
> > > > > > 2. Could we also remove some legacy configuration value ? e.g.
> > LEGACY
> > > > > Mode
> > > > > >> for
> > execution.savepoint-restore-mode/execution.recovery.claim-mode.
> > > > > >
> > > > > >
> > > > > > I think we could create another FLIP for the deprecation of
> LEGACY
> > > mode.
> > > > > >
> > > > > >
> > > > > >> 3. Could the local checkpoint be cleaned
> > > > > >> if execution.checkpointing.local-copy.enabled is true and
> > > > > >> execution.recovery.from-local is false ? I found it's also an
> > issue
> > > if
> > > > > >> current local-recovery from enabled to disabled. Maybe another
> > > ticket is
> > > > > >> needed.
> > > > > >
> > > > > >
> > > > > > IIUC, there is no clear ownership of the local copy files from
> the
> > > > > > previous job and it's better to define one. This needs more
> > > discussion so
> > > > > > we could create another thread for this. WDYT?
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu <
> masteryhx@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Hi, Zakelly.
> > > > > >> Thanks for driving this. Overall LGTM as we discussed offline.
> > > > > >>
> > > > > >> Some comments/suggestions just came to mind:
> > > > > >> 1. Could execution.recovery also contain some other behaviors
> > about
> > > > > >> recovery ? e.g. restart-strategy.
> > > > > >> 2. Could we also remove some legacy configuration value ? e.g.
> > > LEGACY
> > > > > Mode
> > > > > >> for
> > execution.savepoint-restore-mode/execution.recovery.claim-mode.
> > > > > >> 3. Could the local checkpoint be cleaned
> > > > > >> if execution.checkpointing.local-copy.enabled is true and
> > > > > >> execution.recovery.from-local is false ? I found it's also an
> > issue
> > > if
> > > > > >> current local-recovery from enabled to disabled. Maybe another
> > > ticket is
> > > > > >> needed.
> > > > > >> 4. +1 for enabling execution.checkpointing.incremental by
> default
> > > which
> > > > > is
> > > > > >> basically default configuration in our production environment.
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <
> zakelly.lan@gmail.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > Hi Yun,
> > > > > >> >
> > > > > >> > Thanks for your comments!
> > > > > >> >
> > > > > >> >  1.  We shall not describe the configuration with its
> > > implementation
> > > > > for
> > > > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > > > > >> > state-backend,
> > > > > >> > > it would write two streams and for Rocksdb state-backend, it
> > > would
> > > > > use
> > > > > >> > > hard-link for backup. Thus, I think
> > > > > >> > > 'execution.checkpointing.local-backup.*' looks better.
> > > > > >> >
> > > > > >> > I agreed that we'd better name the option in user's
> perspective
> > > > > instead
> > > > > >> of
> > > > > >> > the implementation, thus I name it as a copy of the checkpoint
> > in
> > > the
> > > > > >> > local disk, regardless of the way of generating it. The word
> > > 'backup'
> > > > > is
> > > > > >> > also suitable for this case, so I agree to change to
> > > > > >> > 'execution.checkpointing.local-backup.*' if no one objects.
> > > > > >> >
> > > > > >> >  2.  What does the
> > 'execution.checkpointing.data-inline-threshold'
> > > > > >> mean? It
> > > > > >> > > seems not so easy to understand.
> > > > > >> >
> > > > > >> > The 'execution.checkpointing.data-inline-threshold' (original
> > one
> > > as
> > > > > >> > 'state.storage.fs.memory-threshold') stands for the size
> > threshold
> > > > > below
> > > > > >> > which state chunks will store inline with the metadata, thus I
> > > call it
> > > > > >> > 'data-inline-threshold'.
> > > > > >> >
> > > > > >> >
> > > > > >> > Best,
> > > > > >> > Zakelly
> > > > > >> >
> > > > > >> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com>
> > > wrote:
> > > > > >> >
> > > > > >> > > Hi Zakelly,
> > > > > >> > >
> > > > > >> > > Thanks for driving this topic. I have two concerns here:
> > > > > >> > >
> > > > > >> > >   1.  We shall not describe the configuration with its
> > > > > implementation
> > > > > >> for
> > > > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > > > > >> > state-backend,
> > > > > >> > > it would write two streams and for Rocksdb state-backend, it
> > > would
> > > > > use
> > > > > >> > > hard-link for backup. Thus, I think
> > > > > >> > > 'execution.checkpointing.local-backup.*' looks better.
> > > > > >> > >   2.  What does the
> > > 'execution.checkpointing.data-inline-threshold'
> > > > > >> mean?
> > > > > >> > > It seems not so easy to understand.
> > > > > >> > >
> > > > > >> > > Best
> > > > > >> > > Yun Tang
> > > > > >> > > ________________________________
> > > > > >> > > From: Piotr Nowojski <pn...@apache.org>
> > > > > >> > > Sent: Thursday, January 4, 2024 22:37
> > > > > >> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > >> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State &
> > > Checkpointing &
> > > > > >> > > Recovery Configuration
> > > > > >> > >
> > > > > >> > > Hi,
> > > > > >> > >
> > > > > >> > > Thanks for trying to clean this up! I don't have strong
> > > opinions on
> > > > > >> the
> > > > > >> > > topics discussed here, so generally speaking +1 from my
> side!
> > > > > >> > >
> > > > > >> > > Best,
> > > > > >> > > Piotrek
> > > > > >> > >
> > > > > >> > > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com>
> > > napisał(a):
> > > > > >> > >
> > > > > >> > > > Thanks for the feedback!
> > > > > >> > > >
> > > > > >> > > > Using the `execution.checkpointing.incremental.enabled`,
> > > > > >> > > > and enabling it by default sounds good to me.
> > > > > >> > > >
> > > > > >> > > > Best,
> > > > > >> > > > Rui
> > > > > >> > > >
> > > > > >> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <
> > > > > zakelly.lan@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Rui,
> > > > > >> > > > >
> > > > > >> > > > > Thanks for your comments!
> > > > > >> > > > >
> > > > > >> > > > > IMO, given that the state backend can be plugably loaded
> > > (as you
> > > > > >> can
> > > > > >> > > > > specify a state backend factory), I prefer not providing
> > > state
> > > > > >> > backend
> > > > > >> > > > > specified options in the framework.
> > > > > >> > > > >
> > > > > >> > > > > Secondly, the incremental checkpoint is actually a
> sharing
> > > file
> > > > > >> > > strategy
> > > > > >> > > > > across checkpoints, which means the state backend
> *could*
> > > reuse
> > > > > >> files
> > > > > >> > > > from
> > > > > >> > > > > previous cp but not *must* do so. When the state backend
> > > could
> > > > > not
> > > > > >> > > reuse
> > > > > >> > > > > the files, it is reasonable to fallback to a full
> > > checkpoint.
> > > > > >> > > > >
> > > > > >> > > > > Thus, I suggest we make it
> > > `execution.checkpointing.incremental`
> > > > > >> and
> > > > > >> > > > enable
> > > > > >> > > > > it by default. For those state backends not supporting
> > this,
> > > > > they
> > > > > >> > > perform
> > > > > >> > > > > full checkpoints and print a warning to inform users.
> > Users
> > > do
> > > > > not
> > > > > >> > need
> > > > > >> > > > to
> > > > > >> > > > > pay special attention to different options to control
> this
> > > > > across
> > > > > >> > > > different
> > > > > >> > > > > state backends. This is more user-friendly in my
> opinion.
> > > WDYT?
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <
> > > 1996fanrui@gmail.com>
> > > > > >> > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi Zakelly,
> > > > > >> > > > > >
> > > > > >> > > > > > I'm not sure whether we could add the state backend
> type
> > > in
> > > > > the
> > > > > >> > > > > > new key name of state.backend.incremental. It means we
> > use
> > > > > >> > > > > > `execution.checkpointing.rocksdb-incremental` or
> > > > > >> > > > > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > > > > >> > > > > >
> > > > > >> > > > > > So far, state.backend.incremental only works for
> rocksdb
> > > state
> > > > > >> > > backend.
> > > > > >> > > > > > And this feature or optimization is very valuable and
> > > huge for
> > > > > >> > large
> > > > > >> > > > > > state flink jobs. I believe it's enabled for most
> > > production
> > > > > >> flink
> > > > > >> > > jobs
> > > > > >> > > > > > with large rocksdb state.
> > > > > >> > > > > >
> > > > > >> > > > > > If this option isn't generic for all state backend
> > types,
> > > I
> > > > > >> guess
> > > > > >> > we
> > > > > >> > > > > > can enable
> > > > > `execution.checkpointing.rocksdb-incremental.enabled`
> > > > > >> > > > > > by default in Flink 2.0.
> > > > > >> > > > > >
> > > > > >> > > > > > But if it works for all state backends, it's hard to
> > > enable it
> > > > > >> by
> > > > > >> > > > > default.
> > > > > >> > > > > > Enabling great and valuable features or improvements
> are
> > > > > useful
> > > > > >> > > > > > for users, especially a lot of new flink users.
> > > Out-of-the-box
> > > > > >> > > options
> > > > > >> > > > > > are good for users.
> > > > > >> > > > > >
> > > > > >> > > > > > WDYT?
> > > > > >> > > > > >
> > > > > >> > > > > > Best,
> > > > > >> > > > > > Rui
> > > > > >> > > > > >
> > > > > >> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <
> > > > > >> zakelly.lan@gmail.com
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi everyone,
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks all for your comments!
> > > > > >> > > > > > >
> > > > > >> > > > > > > As many of you have questions about the names for
> > > boolean
> > > > > >> > options,
> > > > > >> > > I
> > > > > >> > > > > > > suggest we make a naming rule for them. For now I
> > could
> > > > > think
> > > > > >> of
> > > > > >> > > > three
> > > > > >> > > > > > > options:
> > > > > >> > > > > > >
> > > > > >> > > > > > > Option 1: Use enumeration options if possible. But
> > this
> > > may
> > > > > >> cause
> > > > > >> > > > some
> > > > > >> > > > > > name
> > > > > >> > > > > > > collisions or confusion as we discussed and we
> should
> > > unify
> > > > > >> the
> > > > > >> > > > > statement
> > > > > >> > > > > > > everywhere.
> > > > > >> > > > > > > Option 2: Use boolean options and add 'enabled' as
> the
> > > > > suffix.
> > > > > >> > > > > > > Option 3: Use boolean options and ONLY add 'enabled'
> > > when
> > > > > >> there
> > > > > >> > are
> > > > > >> > > > > more
> > > > > >> > > > > > > detailed configurations under the same prefix, to
> > > prevent
> > > > > one
> > > > > >> > name
> > > > > >> > > > from
> > > > > >> > > > > > > serving as a prefix to another.
> > > > > >> > > > > > >
> > > > > >> > > > > > > I am slightly inclined to Option 3, since it is more
> > in
> > > line
> > > > > >> with
> > > > > >> > > > > current
> > > > > >> > > > > > > practice and friendly for existing users. Also It
> > > reduces
> > > > > the
> > > > > >> > > length
> > > > > >> > > > of
> > > > > >> > > > > > > configuration names as much as possible. I really
> want
> > > to
> > > > > hear
> > > > > >> > your
> > > > > >> > > > > > > opinions.
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > @Xuannan
> > > > > >> > > > > > >
> > > > > >> > > > > > > I agree with your comments 1 and 3.
> > > > > >> > > > > > >
> > > > > >> > > > > > > For 2, If we decide to change the name, maybe
> > > > > >> > > > > > > `execution.checkpointing.parallel-cleaner` is
> better?
> > > And as
> > > > > >> for
> > > > > >> > > > > whether
> > > > > >> > > > > > to
> > > > > >> > > > > > > add 'enabled' I suggest we discuss the rule above.
> > WDYT?
> > > > > >> > > > > > > Thanks!
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > Best,
> > > > > >> > > > > > > Zakelly
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
> > > > > >> > suxuannan95@gmail.com
> > > > > >> > > >
> > > > > >> > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Zakelly,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Thanks for driving this! The organization of the
> > > > > >> configuration
> > > > > >> > > > option
> > > > > >> > > > > > > > in the FLIP looks much cleaner and easier to
> > > understand.
> > > > > +1
> > > > > >> to
> > > > > >> > > the
> > > > > >> > > > > > > > FLIP.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Just some questions from me.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > 1. I think the change to the ConfigOptions should
> be
> > > put
> > > > > in
> > > > > >> the
> > > > > >> > > > > > > > `Public Interface` section, instead of `Proposed
> > > Changed`,
> > > > > >> as
> > > > > >> > > those
> > > > > >> > > > > > > > configuration options are public interface.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > 2. The key
> `state.checkpoint.cleaner.parallel-mode`
> > > seems
> > > > > >> > > > confusing.
> > > > > >> > > > > > > > It feels like it is used to choose different
> modes.
> > In
> > > > > >> fact, it
> > > > > >> > > is
> > > > > >> > > > a
> > > > > >> > > > > > > > boolean flag to indicate whether to enable
> parallel
> > > clean.
> > > > > >> How
> > > > > >> > > > about
> > > > > >> > > > > > > > making it
> > > > > `state.checkpoint.cleaner.parallel-mode.enabled`?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > 3. The `execution.checkpointing.write-buffer` may
> > > better
> > > > > be
> > > > > >> > > > > > > > `execution.checkpointing.write-buffer-size` so
> that
> > we
> > > > > know
> > > > > >> it
> > > > > >> > is
> > > > > >> > > > > > > > configuring the size of the buffer.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Best,
> > > > > >> > > > > > > > Xuannan
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
> > > > > >> > fredialei@gmail.com>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Hi Zakelly,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Considering the name occupation, how about
> > naming
> > > it
> > > > > as
> > > > > >> > > > > > > > `execution.checkpointing.type`?
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > `Checkpoint Type`[1,2] is used to describe
> > > > > >> aligned/unaligned
> > > > > >> > > > > > > > > checkpoint, I am inclined to make a choice
> between
> > > > > >> > > > > > > > > `execution.checkpointing.incremental` and
> > > > > >> > > > > > > > > `execution.checkpointing.incremental.enabled`.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > [1]
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > > > >> > > > > > > > > [2]
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > --
> > > > > >> > > > > > > > > Best,
> > > > > >> > > > > > > > > Yanfei
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Zakelly Lan <za...@gmail.com>
> > 于2023年12月27日周三
> > > > > >> 14:41写道:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Hi Lijie,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks for the reminder! I missed this.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Considering the name occupation, how about
> > naming
> > > it
> > > > > as
> > > > > >> > > > > > > > > > `execution.checkpointing.type`?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Actually I think the current
> > > > > >> `execution.checkpointing.mode`
> > > > > >> > > is
> > > > > >> > > > > > > > confusing in
> > > > > >> > > > > > > > > > some ways, maybe
> > > > > >> `execution.checkpointing.data-consistency`
> > > > > >> > > is
> > > > > >> > > > > > > better.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Best,
> > > > > >> > > > > > > > > > Zakelly
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > > > > >> > > > > > > wangdachui9501@gmail.com>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > Hi Zakelly,
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > >> I'm wondering if
> > > > > >> > `execution.checkpointing.savepoint-dir`
> > > > > >> > > > > would
> > > > > >> > > > > > > be
> > > > > >> > > > > > > > > > > better.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > `execution.checkpointing.dir` and
> > > > > >> > > > > > > > `execution.checkpointing.savepoint-dir`
> > > > > >> > > > > > > > > > > are also fine for me.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > >> So I think an enumeration option
> > > > > >> > > > > > `execution.checkpointing.mode`
> > > > > >> > > > > > > > which
> > > > > >> > > > > > > > > > > can be 'full' (default) or 'incremental'
> would
> > > be
> > > > > >> better
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > I agree with using an enumeration option.
> But
> > > > > >> currently
> > > > > >> > > there
> > > > > >> > > > > is
> > > > > >> > > > > > > > already a
> > > > > >> > > > > > > > > > > configuration option called
> > > > > >> > `execution.checkpointing.mode`,
> > > > > >> > > > > which
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > used
> > > > > >> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE.
> Maybe
> > > we
> > > > > >> need to
> > > > > >> > > use
> > > > > >> > > > > > > > another name
> > > > > >> > > > > > > > > > > or merge these two options.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > Lijie
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Zakelly Lan <za...@gmail.com>
> > > 于2023年12月27日周三
> > > > > >> > > 11:43写道:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Hi everyone,
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Thanks all for your comments!
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > @Yanfei
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > 1. For some state backends that do not
> > > support
> > > > > >> > > > incremental
> > > > > >> > > > > > > > checkpoint,
> > > > > >> > > > > > > > > > > > > how does the
> > > > > >> > execution.checkpointing.incrementaloption
> > > > > >> > > > take
> > > > > >> > > > > > > > effect? Or
> > > > > >> > > > > > > > > > > > > is it better to put incremental under
> > > > > >> > > > > > > > state.backend.xxx.incremental?
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > I'd rather not put the option for
> > incremental
> > > > > >> > checkpoint
> > > > > >> > > > > under
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > 'state.backend', since it is more about
> the
> > > > > >> > checkpointing
> > > > > >> > > > > > instead
> > > > > >> > > > > > > > of
> > > > > >> > > > > > > > > > > state
> > > > > >> > > > > > > > > > > > accessing. Of course, the state backend
> may
> > > not
> > > > > >> > > necessarily
> > > > > >> > > > > do
> > > > > >> > > > > > > > > > > incremental
> > > > > >> > > > > > > > > > > > checkpoint as requested. If the state
> > backend
> > > is
> > > > > not
> > > > > >> > > > capable
> > > > > >> > > > > of
> > > > > >> > > > > > > > taking
> > > > > >> > > > > > > > > > > > incremental cp, it is better to fallback
> to
> > > the
> > > > > full
> > > > > >> > cp.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > 2. I'm a little worried that putting all
> > > > > >> configurations
> > > > > >> > > > into
> > > > > >> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
> > > introduce
> > > > > >> some
> > > > > >> > > > > > dependency
> > > > > >> > > > > > > > > > > > > problems. Some options would be used by
> > > > > >> flink-runtime
> > > > > >> > > > > module,
> > > > > >> > > > > > > but
> > > > > >> > > > > > > > > > > > > flink-runtime should not depend on
> > > > > >> > > flink-streaming-java.
> > > > > >> > > > > e.g.
> > > > > >> > > > > > > > > > > > > FLINK-28286[1].
> > > > > >> > > > > > > > > > > > > So, I prefer to move configurations to
> > > > > >> > > > > > `CheckpointingOptions`,
> > > > > >> > > > > > > > WDYT?
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Yes, that's a very good point.  Moving to
> > > > > >> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes
> > > sense.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > @Lijie
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > How about
> > > > > >> > > > > > > > > > > > > state.savepoints.dir ->
> > > > > >> > > > > execution.checkpointing.savepoint.dir
> > > > > >> > > > > > > > > > > > > state.checkpoints.dir ->
> > > > > >> > > > > > execution.checkpointing.checkpoint.dir
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Actually, I think the
> > > `checkpointing.checkpoint`
> > > > > may
> > > > > >> > > cause
> > > > > >> > > > > some
> > > > > >> > > > > > > > > > > confusion.
> > > > > >> > > > > > > > > > > > But I'm ok if others agree.
> > > > > >> > > > > > > > > > > > I'm wondering if
> > > > > >> > `execution.checkpointing.savepoint-dir`
> > > > > >> > > > > would
> > > > > >> > > > > > be
> > > > > >> > > > > > > > better.
> > > > > >> > > > > > > > > > > > WDYT?
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > 2. We changed the
> > > > > >> execution.checkpointing.local-copy'
> > > > > >> > to
> > > > > >> > > > > > > > > > > > >
> > > 'execution.checkpointing.local-copy.enabled'.
> > > > > >> Should
> > > > > >> > we
> > > > > >> > > > > also
> > > > > >> > > > > > > add
> > > > > >> > > > > > > > > > > > "enabled"
> > > > > >> > > > > > > > > > > > > suffix for other boolean type
> > configuration
> > > > > >> options ?
> > > > > >> > > For
> > > > > >> > > > > > > > example,
> > > > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > > > > >> > > > > > > > > > > > >
> > execution.checkpointing.incremental.enabled
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Actually, the incremental cp is something
> > like
> > > > > >> > choosing a
> > > > > >> > > > > mode
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > doing
> > > > > >> > > > > > > > > > > > checkpoint instead of enabling a function.
> > So
> > > I
> > > > > >> think
> > > > > >> > an
> > > > > >> > > > > > > > enumeration
> > > > > >> > > > > > > > > > > option
> > > > > >> > > > > > > > > > > > `execution.checkpointing.mode` which can
> be
> > > 'full'
> > > > > >> > > > (default)
> > > > > >> > > > > or
> > > > > >> > > > > > > > > > > > 'incremental' would be better, WDYT?
> > > > > >> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think
> about
> > > this?
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie
> Wang <
> > > > > >> > > > > > > > wangdachui9501@gmail.com>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Hi Zakelly,
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Thanks for driving the discussion.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > 1.
> > > > > >> > > > > > > > > > > > > >> But I'm not so sure since there is
> only
> > > one
> > > > > >> > > > > > > savepoint-related
> > > > > >> > > > > > > > > > > option.
> > > > > >> > > > > > > > > > > > > Maybe someone else could share some
> > thoughts
> > > > > here.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > How about
> > > > > >> > > > > > > > > > > > > state.savepoints.dir ->
> > > > > >> > > > > execution.checkpointing.savepoint.dir
> > > > > >> > > > > > > > > > > > > state.checkpoints.dir ->
> > > > > >> > > > > > execution.checkpointing.checkpoint.dir
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > 2. We changed the
> > > > > >> execution.checkpointing.local-copy'
> > > > > >> > > to
> > > > > >> > > > > > > > > > > > >
> > > 'execution.checkpointing.local-copy.enabled'.
> > > > > >> Should
> > > > > >> > we
> > > > > >> > > > > also
> > > > > >> > > > > > > add
> > > > > >> > > > > > > > > > > > "enabled"
> > > > > >> > > > > > > > > > > > > suffix for other boolean type
> > configuration
> > > > > >> options ?
> > > > > >> > > For
> > > > > >> > > > > > > > example,
> > > > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > > > > >> > > > > > > > > > > > >
> > execution.checkpointing.incremental.enabled
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > In this way, the naming style of
> > > configuration
> > > > > >> > options
> > > > > >> > > is
> > > > > >> > > > > > > > unified, and
> > > > > >> > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > can avoid potential similar problems
> (for
> > > > > >> example, we
> > > > > >> > > may
> > > > > >> > > > > > need
> > > > > >> > > > > > > > to add
> > > > > >> > > > > > > > > > > > more
> > > > > >> > > > > > > > > > > > > options for incremental checkpoint in
> the
> > > > > future).
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > Lijie
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Yanfei Lei <fr...@gmail.com>
> > > 于2023年12月26日周二
> > > > > >> > > 12:05写道:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Hi Zakelly,
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Thank you for creating the FLIP and
> > > starting
> > > > > the
> > > > > >> > > > > > discussion.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > The current arrangement of these
> options
> > > is
> > > > > >> indeed
> > > > > >> > > > > somewhat
> > > > > >> > > > > > > > > > > haphazard,
> > > > > >> > > > > > > > > > > > > > and the new arrangement looks much
> > > better. I
> > > > > >> have
> > > > > >> > > some
> > > > > >> > > > > > > > questions
> > > > > >> > > > > > > > > > > about
> > > > > >> > > > > > > > > > > > > > the arrangement of some new
> > configuration
> > > > > >> options:
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > 1. For some state backends that do not
> > > support
> > > > > >> > > > > incremental
> > > > > >> > > > > > > > > > > checkpoint,
> > > > > >> > > > > > > > > > > > > > how does the
> > > > > >> > > execution.checkpointing.incrementaloption
> > > > > >> > > > > take
> > > > > >> > > > > > > > effect?
> > > > > >> > > > > > > > > > > Or
> > > > > >> > > > > > > > > > > > > > is it better to put incremental under
> > > > > >> > > > > > > > state.backend.xxx.incremental?
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > 2. I'm a little worried that putting
> all
> > > > > >> > > configurations
> > > > > >> > > > > > into
> > > > > >> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
> > > introduce
> > > > > >> some
> > > > > >> > > > > > > dependency
> > > > > >> > > > > > > > > > > > > > problems. Some options would be used
> by
> > > > > >> > flink-runtime
> > > > > >> > > > > > module,
> > > > > >> > > > > > > > but
> > > > > >> > > > > > > > > > > > > > flink-runtime should not depend on
> > > > > >> > > > flink-streaming-java.
> > > > > >> > > > > > e.g.
> > > > > >> > > > > > > > > > > > > > FLINK-28286[1].
> > > > > >> > > > > > > > > > > > > > So, I prefer to move configurations to
> > > > > >> > > > > > > `CheckpointingOptions`,
> > > > > >> > > > > > > > WDYT?
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > [1]
> > > > > >> > > https://issues.apache.org/jira/browse/FLINK-28286
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > --
> > > > > >> > > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > > Yanfei
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Zakelly Lan <za...@gmail.com>
> > > > > >> 于2023年12月25日周一
> > > > > >> > > > > > 21:14写道:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Hi Rui Fan and Junrui,
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Thanks for the reminder! I agree to
> > > change
> > > > > the
> > > > > >> > > > > > > > > > > > > > > 'execution.checkpointing.local-copy'
> > to
> > > > > >> > > > > > > > > > > > > > >
> > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > And for other suggestions Rui
> > proposed:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > 1. How about
> > > > > >> execution.checkpointing.storage.type
> > > > > >> > > > > instead
> > > > > >> > > > > > > > > > > > > > > > of
> execution.checkpointing.storage?
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Ah, I missed something here.
> Actually
> > I
> > > > > >> suggest
> > > > > >> > we
> > > > > >> > > > > could
> > > > > >> > > > > > > > merge the
> > > > > >> > > > > > > > > > > > > > current
> > > > > >> > > > > > > > > > > > > > > 'state.checkpoints.dir' and
> > > > > >> > > > 'state.checkpoint-storage'
> > > > > >> > > > > > into
> > > > > >> > > > > > > > one URI
> > > > > >> > > > > > > > > > > > > > > configuration named
> > > > > >> > 'execution.checkpointing.dir'.
> > > > > >> > > > > WDYT?
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > 3.
> > > execution.checkpointing.savepoint.dir is
> > > > > a
> > > > > >> > > little
> > > > > >> > > > > > weird.
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Yes, I think it is better to make
> > > > > 'savepoint'
> > > > > >> and
> > > > > >> > > > > > > > 'checkpoint' the
> > > > > >> > > > > > > > > > > > same
> > > > > >> > > > > > > > > > > > > > > level. But I'm not so sure since
> there
> > > is
> > > > > only
> > > > > >> > one
> > > > > >> > > > > > > > > > > savepoint-related
> > > > > >> > > > > > > > > > > > > > > option. Maybe someone else could
> share
> > > some
> > > > > >> > > thoughts
> > > > > >> > > > > > here.
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > 4. How about
> > > execution.recovery.claim-mode
> > > > > >> > instead
> > > > > >> > > of
> > > > > >> > > > > > > > > > > > > > > > execution.recovery.mode?
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >  Agreed. That's more accurate.
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Many thanks for your suggestions!
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > > > Zakelly
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM
> Junrui
> > > Lee <
> > > > > >> > > > > > > > jrlee.ljr@gmail.com>
> > > > > >> > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Hi Zakelly,
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Thanks for driving this. I agree
> > that
> > > the
> > > > > >> > > proposed
> > > > > >> > > > > > > > restructuring
> > > > > >> > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > configuration options is largely
> > > positive.
> > > > > >> It
> > > > > >> > > will
> > > > > >> > > > > make
> > > > > >> > > > > > > > > > > > understanding
> > > > > >> > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > working with Flink configurations
> > more
> > > > > >> > intuitive.
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Most of the proposed changes look
> > > great.
> > > > > >> Just a
> > > > > >> > > > > > heads-up,
> > > > > >> > > > > > > > as Rui
> > > > > >> > > > > > > > > > > > Fan
> > > > > >> > > > > > > > > > > > > > > > mentioned, Flink currently
> requires
> > > that
> > > > > no
> > > > > >> > > > > > > configOption's
> > > > > >> > > > > > > > key be
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > prefix of another to avoid issues
> > > when we
> > > > > >> > > > eventually
> > > > > >> > > > > > > adopt
> > > > > >> > > > > > > > a
> > > > > >> > > > > > > > > > > > standard
> > > > > >> > > > > > > > > > > > > > YAML
> > > > > >> > > > > > > > > > > > > > > > parser, as detailed in
> FLINK-29372 (
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > https://issues.apache.org/jira/browse/FLINK-29372
> > > > > >> > > > ).
> > > > > >> > > > > > > > Therefore,
> > > > > >> > > > > > > > > > > > it's
> > > > > >> > > > > > > > > > > > > > better
> > > > > >> > > > > > > > > > > > > > > > to change the key
> > > > > >> > > > > 'execution.checkpointing.local-copy'
> > > > > >> > > > > > > > because it
> > > > > >> > > > > > > > > > > > > > serves as
> > > > > >> > > > > > > > > > > > > > > > a prefix to the key
> > > > > >> > > > > > > > 'execution.checkpointing.local-copy.dir'.
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Best regards,
> > > > > >> > > > > > > > > > > > > > > > Junrui
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Rui Fan <19...@gmail.com>
> > > > > >> 于2023年12月25日周一
> > > > > >> > > > > 19:11写道:
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > Hi Zakelly,
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > Thank you for driving this
> > proposal!
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > Overall good for me. I have some
> > > > > questions
> > > > > >> > > about
> > > > > >> > > > > > these
> > > > > >> > > > > > > > names.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > 1. How about
> > > > > >> > > execution.checkpointing.storage.type
> > > > > >> > > > > > > > instead of
> > > > > >> > > > > > > > > > > > > > > > > execution.checkpointing.storage?
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > It's similar to
> > state.backend.type.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > 2. How about
> > > > > >> > > > > > execution.checkpointing.local-copy.enabled
> > > > > >> > > > > > > > instead
> > > > > >> > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > > > >
> > execution.checkpointing.local-copy?
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > You added a new option:
> > > > > >> > > > > > > > execution.checkpointing.local-copy.dir.
> > > > > >> > > > > > > > > > > > > > > > > IIUC, one option name shouldn't
> be
> > > the
> > > > > >> prefix
> > > > > >> > > of
> > > > > >> > > > > > other
> > > > > >> > > > > > > > options.
> > > > > >> > > > > > > > > > > > > > > > > If you add a new option
> > > > > >> > > > > > > > execution.checkpointing.local-copy,
> > > > > >> > > > > > > > > > > > > > > > > flink CI will fail directly.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > 3.
> > > execution.checkpointing.savepoint.dir
> > > > > >> is a
> > > > > >> > > > > little
> > > > > >> > > > > > > > weird.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > For old options:
> > > state.savepoints.dir
> > > > > and
> > > > > >> > > > > > > > > > > state.checkpoints.dir,
> > > > > >> > > > > > > > > > > > > > > > > the savepoint and checkpoint are
> > the
> > > > > same
> > > > > >> > > level.
> > > > > >> > > > It
> > > > > >> > > > > > > means
> > > > > >> > > > > > > > > > > > > > > > > it's a checkpoint or savepoint.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > The new option
> > > > > >> execution.checkpointing.dir is
> > > > > >> > > > fine
> > > > > >> > > > > > for
> > > > > >> > > > > > > > me.
> > > > > >> > > > > > > > > > > > > > > > > However,
> > > > > >> > execution.checkpointing.savepoint.dir
> > > > > >> > > > is a
> > > > > >> > > > > > > > little
> > > > > >> > > > > > > > > > > weird.
> > > > > >> > > > > > > > > > > > > > > > > I don't know which name is
> better
> > > now.
> > > > > >> Let us
> > > > > >> > > > think
> > > > > >> > > > > > > > about it
> > > > > >> > > > > > > > > > > > more.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > 4. How about
> > > > > execution.recovery.claim-mode
> > > > > >> > > > instead
> > > > > >> > > > > of
> > > > > >> > > > > > > > > > > > > > > > > execution.recovery.mode?
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > The meaning of mode is too
> broad.
> > > The
> > > > > >> > > claim-mode
> > > > > >> > > > > may
> > > > > >> > > > > > > > > > > > > > > > > be more accurate for users.
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > WDYT?
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > > > > > Rui
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM
> > > Zakelly
> > > > > >> Lan <
> > > > > >> > > > > > > > > > > > zakelly.lan@gmail.com
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > Hi devs,
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > I'd like to start a discussion
> > on
> > > > > >> FLIP-406:
> > > > > >> > > > > > > Reorganize
> > > > > >> > > > > > > > State
> > > > > >> > > > > > > > > > > &
> > > > > >> > > > > > > > > > > > > > > > > > Checkpointing & Recovery
> > > > > >> Configuration[1].
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > Currently, the configuration
> > > options
> > > > > >> > > pertaining
> > > > > >> > > > > to
> > > > > >> > > > > > > > > > > > checkpointing,
> > > > > >> > > > > > > > > > > > > > > > > recovery,
> > > > > >> > > > > > > > > > > > > > > > > > and state management are
> > primarily
> > > > > >> grouped
> > > > > >> > > > under
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > > > following
> > > > > >> > > > > > > > > > > > > > > > prefixes:
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > >    - state.backend.* :
> > > configurations
> > > > > >> > related
> > > > > >> > > > to
> > > > > >> > > > > > > state
> > > > > >> > > > > > > > > > > > accessing
> > > > > >> > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > > >    checkpointing, as well as
> > > specific
> > > > > >> > options
> > > > > >> > > > for
> > > > > >> > > > > > > > individual
> > > > > >> > > > > > > > > > > > > state
> > > > > >> > > > > > > > > > > > > > > > > backends
> > > > > >> > > > > > > > > > > > > > > > > >    -
> execution.checkpointing.* :
> > > > > >> > > configurations
> > > > > >> > > > > > > > associated
> > > > > >> > > > > > > > > > > with
> > > > > >> > > > > > > > > > > > > > > > > checkpoint
> > > > > >> > > > > > > > > > > > > > > > > >    execution and recovery
> > > > > >> > > > > > > > > > > > > > > > > >    - execution.savepoint.*:
> > > > > >> configurations
> > > > > >> > > for
> > > > > >> > > > > > > > recovery from
> > > > > >> > > > > > > > > > > > > > savepoint
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > In addition, there are several
> > > > > >> individual
> > > > > >> > > > options
> > > > > >> > > > > > > such
> > > > > >> > > > > > > > as '
> > > > > >> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*'
> and
> > > > > >> > > > > > > > '*state.checkpoints.dir*'
> > > > > >> > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > fall
> > > > > >> > > > > > > > > > > > > > > > > outside
> > > > > >> > > > > > > > > > > > > > > > > > of these prefixes. The current
> > > > > >> arrangement
> > > > > >> > of
> > > > > >> > > > > these
> > > > > >> > > > > > > > options,
> > > > > >> > > > > > > > > > > > > which
> > > > > >> > > > > > > > > > > > > > span
> > > > > >> > > > > > > > > > > > > > > > > > multiple modules, is somewhat
> > > > > haphazard
> > > > > >> and
> > > > > >> > > > > lacks a
> > > > > >> > > > > > > > > > > systematic
> > > > > >> > > > > > > > > > > > > > > > structure.
> > > > > >> > > > > > > > > > > > > > > > > > For example, the options under
> > the
> > > > > >> > > > > > > > '*CheckpointingOptions*'
> > > > > >> > > > > > > > > > > > and '
> > > > > >> > > > > > > > > > > > > > > > > >
> *ExecutionCheckpointingOptions*'
> > > are
> > > > > >> > related
> > > > > >> > > > and
> > > > > >> > > > > > have
> > > > > >> > > > > > > > no
> > > > > >> > > > > > > > > > > clear
> > > > > >> > > > > > > > > > > > > > > > boundaries
> > > > > >> > > > > > > > > > > > > > > > > > from the user's perspective,
> but
> > > there
> > > > > >> is
> > > > > >> > no
> > > > > >> > > > > > unified
> > > > > >> > > > > > > > prefix
> > > > > >> > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > them.
> > > > > >> > > > > > > > > > > > > > > > > With
> > > > > >> > > > > > > > > > > > > > > > > > the upcoming release of Flink
> > > 2.0, we
> > > > > >> have
> > > > > >> > an
> > > > > >> > > > > > > excellent
> > > > > >> > > > > > > > > > > > > > opportunity to
> > > > > >> > > > > > > > > > > > > > > > > > overhaul and restructure the
> > > > > >> configurations
> > > > > >> > > > > related
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > > > > > checkpointing,
> > > > > >> > > > > > > > > > > > > > > > > > recovery, and state
> management.
> > > This
> > > > > >> FLIP
> > > > > >> > > > > proposes
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > > > > reorganize
> > > > > >> > > > > > > > > > > > > > these
> > > > > >> > > > > > > > > > > > > > > > > > settings, making it more
> > coherent
> > > by
> > > > > >> > module,
> > > > > >> > > > > which
> > > > > >> > > > > > > > would
> > > > > >> > > > > > > > > > > > > > significantly
> > > > > >> > > > > > > > > > > > > > > > > > lower the barriers for
> > > understanding
> > > > > and
> > > > > >> > > reduce
> > > > > >> > > > > the
> > > > > >> > > > > > > > > > > development
> > > > > >> > > > > > > > > > > > > > costs
> > > > > >> > > > > > > > > > > > > > > > > > moving forward.
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > Looking forward to hearing
> from
> > > you!
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > [1]
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > > > > > > Zakelly
> > > > > >> > > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Best,
> > > > > >> Hangxiang.
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Hangxiang.
> > >
> > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Rui Fan <19...@gmail.com>.
Hi Zakelly,

Would you mind adding the option type in the FLIP doc?
For example, String, Boolean or Enum, etc. Thank you.

Best,
Rui

On Tue, Jan 16, 2024 at 2:29 PM Zakelly Lan <za...@gmail.com> wrote:

> Hi everyone,
>
> Thanks all for joining the discussion! I'd like to speed this up since it
> lasts for nearly a month. I made changes on this FLIP based on suggestions
> and compromises acceptable to most people. Please feel free to give your
> opinion. Thanks!
> If there are no more suggestions, I will consider starting a vote within a
> week.
>
>
> Best,
> Zakelly
>
> On Thu, Jan 11, 2024 at 10:31 AM Xuannan Su <su...@gmail.com> wrote:
>
> > Hi Zakelly,
> >
> > I am fine with either Option 2 or Option 3. I think the naming in
> > Option 2 makes it clear that it is a boolean configuration. However,
> > most of the currently available boolean configurations do not use
> > "enable" as a suffix. Therefore, Option 3 looks good to me as it
> > follows the current practice.
> >
> > Best regards,
> > Xuannan
> >
> > On Thu, Jan 11, 2024 at 9:50 AM Hangxiang Yu <ma...@gmail.com>
> wrote:
> > >
> > > >
> > > > That's a very good point. I realize that the word 'recovery' means
> way
> > too
> > > > many things. So I suggest picking a more specific word here, how
> about
> > > > 'execution.state-recovery.*' ? Checkpointing and state recovery are
> > > > corresponding terms and won't make ambiguity.
> > > >
> > >
> > > This makes the configuration clearer to me. We could focus on the
> > > `state-recovery` at first.
> > >
> > > I think we could create another FLIP for the deprecation of LEGACY
> mode.
> > > >
> > >
> > > LGTM, Let's create a new FLIP to do this.
> > >
> > > IIUC, there is no clear ownership of the local copy files from the
> > previous
> > > > job and it's better to define one. This needs more discussion so we
> > could
> > > > create another thread for this. WDYT?
> > > >
> > >
> > > Yeah, I have created a new ticket FLINK-34032 to track and discuss
> this.
> > >
> > > On Wed, Jan 10, 2024 at 6:31 PM Zakelly Lan <za...@gmail.com>
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > It seems we still don't have a consensus on the rules for boolean
> type
> > > > options. Let me recap the alternatives we have:
> > > >
> > > > Option 1: Use enumeration options instead if possible. But this may
> > cause
> > > > some name collisions or confusion as we discussed and we should unify
> > the
> > > > statement everywhere.
> > > > Option 2: Use boolean options and add 'enabled' as the suffix.
> > > > Option 3: Use boolean options and ONLY add 'enabled' when there are
> > more
> > > > detailed configurations under the same prefix, to prevent one name
> from
> > > > serving as a prefix to another.
> > > >
> > > > I am inclined to Option 3, since it is more in line with current
> > practice
> > > > and friendly for existing users. Also It reduces the length of
> > > > configuration names as much as possible.
> > > >
> > > > Looking forward to your opinions! Thanks!
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan <za...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Hangxiang,
> > > > >
> > > > > Thanks for your suggestions!
> > > > >
> > > > > 1. Could execution.recovery also contain some other behaviors about
> > > > >> recovery ? e.g. restart-strategy.
> > > > >
> > > > >
> > > > > That's a very good point. I realize that the word 'recovery' means
> > way
> > > > too
> > > > > many things. So I suggest picking a more specific word here, how
> > about
> > > > > 'execution.state-recovery.*' ? Checkpointing and state recovery are
> > > > > corresponding terms and won't make ambiguity.
> > > > >
> > > > > 2. Could we also remove some legacy configuration value ? e.g.
> LEGACY
> > > > Mode
> > > > >> for
> execution.savepoint-restore-mode/execution.recovery.claim-mode.
> > > > >
> > > > >
> > > > > I think we could create another FLIP for the deprecation of LEGACY
> > mode.
> > > > >
> > > > >
> > > > >> 3. Could the local checkpoint be cleaned
> > > > >> if execution.checkpointing.local-copy.enabled is true and
> > > > >> execution.recovery.from-local is false ? I found it's also an
> issue
> > if
> > > > >> current local-recovery from enabled to disabled. Maybe another
> > ticket is
> > > > >> needed.
> > > > >
> > > > >
> > > > > IIUC, there is no clear ownership of the local copy files from the
> > > > > previous job and it's better to define one. This needs more
> > discussion so
> > > > > we could create another thread for this. WDYT?
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu <ma...@gmail.com>
> > > > wrote:
> > > > >
> > > > >> Hi, Zakelly.
> > > > >> Thanks for driving this. Overall LGTM as we discussed offline.
> > > > >>
> > > > >> Some comments/suggestions just came to mind:
> > > > >> 1. Could execution.recovery also contain some other behaviors
> about
> > > > >> recovery ? e.g. restart-strategy.
> > > > >> 2. Could we also remove some legacy configuration value ? e.g.
> > LEGACY
> > > > Mode
> > > > >> for
> execution.savepoint-restore-mode/execution.recovery.claim-mode.
> > > > >> 3. Could the local checkpoint be cleaned
> > > > >> if execution.checkpointing.local-copy.enabled is true and
> > > > >> execution.recovery.from-local is false ? I found it's also an
> issue
> > if
> > > > >> current local-recovery from enabled to disabled. Maybe another
> > ticket is
> > > > >> needed.
> > > > >> 4. +1 for enabling execution.checkpointing.incremental by default
> > which
> > > > is
> > > > >> basically default configuration in our production environment.
> > > > >>
> > > > >>
> > > > >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <zakelly.lan@gmail.com
> >
> > > > wrote:
> > > > >>
> > > > >> > Hi Yun,
> > > > >> >
> > > > >> > Thanks for your comments!
> > > > >> >
> > > > >> >  1.  We shall not describe the configuration with its
> > implementation
> > > > for
> > > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > > > >> > state-backend,
> > > > >> > > it would write two streams and for Rocksdb state-backend, it
> > would
> > > > use
> > > > >> > > hard-link for backup. Thus, I think
> > > > >> > > 'execution.checkpointing.local-backup.*' looks better.
> > > > >> >
> > > > >> > I agreed that we'd better name the option in user's perspective
> > > > instead
> > > > >> of
> > > > >> > the implementation, thus I name it as a copy of the checkpoint
> in
> > the
> > > > >> > local disk, regardless of the way of generating it. The word
> > 'backup'
> > > > is
> > > > >> > also suitable for this case, so I agree to change to
> > > > >> > 'execution.checkpointing.local-backup.*' if no one objects.
> > > > >> >
> > > > >> >  2.  What does the
> 'execution.checkpointing.data-inline-threshold'
> > > > >> mean? It
> > > > >> > > seems not so easy to understand.
> > > > >> >
> > > > >> > The 'execution.checkpointing.data-inline-threshold' (original
> one
> > as
> > > > >> > 'state.storage.fs.memory-threshold') stands for the size
> threshold
> > > > below
> > > > >> > which state chunks will store inline with the metadata, thus I
> > call it
> > > > >> > 'data-inline-threshold'.
> > > > >> >
> > > > >> >
> > > > >> > Best,
> > > > >> > Zakelly
> > > > >> >
> > > > >> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com>
> > wrote:
> > > > >> >
> > > > >> > > Hi Zakelly,
> > > > >> > >
> > > > >> > > Thanks for driving this topic. I have two concerns here:
> > > > >> > >
> > > > >> > >   1.  We shall not describe the configuration with its
> > > > implementation
> > > > >> for
> > > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > > > >> > state-backend,
> > > > >> > > it would write two streams and for Rocksdb state-backend, it
> > would
> > > > use
> > > > >> > > hard-link for backup. Thus, I think
> > > > >> > > 'execution.checkpointing.local-backup.*' looks better.
> > > > >> > >   2.  What does the
> > 'execution.checkpointing.data-inline-threshold'
> > > > >> mean?
> > > > >> > > It seems not so easy to understand.
> > > > >> > >
> > > > >> > > Best
> > > > >> > > Yun Tang
> > > > >> > > ________________________________
> > > > >> > > From: Piotr Nowojski <pn...@apache.org>
> > > > >> > > Sent: Thursday, January 4, 2024 22:37
> > > > >> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > >> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State &
> > Checkpointing &
> > > > >> > > Recovery Configuration
> > > > >> > >
> > > > >> > > Hi,
> > > > >> > >
> > > > >> > > Thanks for trying to clean this up! I don't have strong
> > opinions on
> > > > >> the
> > > > >> > > topics discussed here, so generally speaking +1 from my side!
> > > > >> > >
> > > > >> > > Best,
> > > > >> > > Piotrek
> > > > >> > >
> > > > >> > > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com>
> > napisał(a):
> > > > >> > >
> > > > >> > > > Thanks for the feedback!
> > > > >> > > >
> > > > >> > > > Using the `execution.checkpointing.incremental.enabled`,
> > > > >> > > > and enabling it by default sounds good to me.
> > > > >> > > >
> > > > >> > > > Best,
> > > > >> > > > Rui
> > > > >> > > >
> > > > >> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <
> > > > zakelly.lan@gmail.com>
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Rui,
> > > > >> > > > >
> > > > >> > > > > Thanks for your comments!
> > > > >> > > > >
> > > > >> > > > > IMO, given that the state backend can be plugably loaded
> > (as you
> > > > >> can
> > > > >> > > > > specify a state backend factory), I prefer not providing
> > state
> > > > >> > backend
> > > > >> > > > > specified options in the framework.
> > > > >> > > > >
> > > > >> > > > > Secondly, the incremental checkpoint is actually a sharing
> > file
> > > > >> > > strategy
> > > > >> > > > > across checkpoints, which means the state backend *could*
> > reuse
> > > > >> files
> > > > >> > > > from
> > > > >> > > > > previous cp but not *must* do so. When the state backend
> > could
> > > > not
> > > > >> > > reuse
> > > > >> > > > > the files, it is reasonable to fallback to a full
> > checkpoint.
> > > > >> > > > >
> > > > >> > > > > Thus, I suggest we make it
> > `execution.checkpointing.incremental`
> > > > >> and
> > > > >> > > > enable
> > > > >> > > > > it by default. For those state backends not supporting
> this,
> > > > they
> > > > >> > > perform
> > > > >> > > > > full checkpoints and print a warning to inform users.
> Users
> > do
> > > > not
> > > > >> > need
> > > > >> > > > to
> > > > >> > > > > pay special attention to different options to control this
> > > > across
> > > > >> > > > different
> > > > >> > > > > state backends. This is more user-friendly in my opinion.
> > WDYT?
> > > > >> > > > >
> > > > >> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <
> > 1996fanrui@gmail.com>
> > > > >> > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi Zakelly,
> > > > >> > > > > >
> > > > >> > > > > > I'm not sure whether we could add the state backend type
> > in
> > > > the
> > > > >> > > > > > new key name of state.backend.incremental. It means we
> use
> > > > >> > > > > > `execution.checkpointing.rocksdb-incremental` or
> > > > >> > > > > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > > > >> > > > > >
> > > > >> > > > > > So far, state.backend.incremental only works for rocksdb
> > state
> > > > >> > > backend.
> > > > >> > > > > > And this feature or optimization is very valuable and
> > huge for
> > > > >> > large
> > > > >> > > > > > state flink jobs. I believe it's enabled for most
> > production
> > > > >> flink
> > > > >> > > jobs
> > > > >> > > > > > with large rocksdb state.
> > > > >> > > > > >
> > > > >> > > > > > If this option isn't generic for all state backend
> types,
> > I
> > > > >> guess
> > > > >> > we
> > > > >> > > > > > can enable
> > > > `execution.checkpointing.rocksdb-incremental.enabled`
> > > > >> > > > > > by default in Flink 2.0.
> > > > >> > > > > >
> > > > >> > > > > > But if it works for all state backends, it's hard to
> > enable it
> > > > >> by
> > > > >> > > > > default.
> > > > >> > > > > > Enabling great and valuable features or improvements are
> > > > useful
> > > > >> > > > > > for users, especially a lot of new flink users.
> > Out-of-the-box
> > > > >> > > options
> > > > >> > > > > > are good for users.
> > > > >> > > > > >
> > > > >> > > > > > WDYT?
> > > > >> > > > > >
> > > > >> > > > > > Best,
> > > > >> > > > > > Rui
> > > > >> > > > > >
> > > > >> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <
> > > > >> zakelly.lan@gmail.com
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi everyone,
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks all for your comments!
> > > > >> > > > > > >
> > > > >> > > > > > > As many of you have questions about the names for
> > boolean
> > > > >> > options,
> > > > >> > > I
> > > > >> > > > > > > suggest we make a naming rule for them. For now I
> could
> > > > think
> > > > >> of
> > > > >> > > > three
> > > > >> > > > > > > options:
> > > > >> > > > > > >
> > > > >> > > > > > > Option 1: Use enumeration options if possible. But
> this
> > may
> > > > >> cause
> > > > >> > > > some
> > > > >> > > > > > name
> > > > >> > > > > > > collisions or confusion as we discussed and we should
> > unify
> > > > >> the
> > > > >> > > > > statement
> > > > >> > > > > > > everywhere.
> > > > >> > > > > > > Option 2: Use boolean options and add 'enabled' as the
> > > > suffix.
> > > > >> > > > > > > Option 3: Use boolean options and ONLY add 'enabled'
> > when
> > > > >> there
> > > > >> > are
> > > > >> > > > > more
> > > > >> > > > > > > detailed configurations under the same prefix, to
> > prevent
> > > > one
> > > > >> > name
> > > > >> > > > from
> > > > >> > > > > > > serving as a prefix to another.
> > > > >> > > > > > >
> > > > >> > > > > > > I am slightly inclined to Option 3, since it is more
> in
> > line
> > > > >> with
> > > > >> > > > > current
> > > > >> > > > > > > practice and friendly for existing users. Also It
> > reduces
> > > > the
> > > > >> > > length
> > > > >> > > > of
> > > > >> > > > > > > configuration names as much as possible. I really want
> > to
> > > > hear
> > > > >> > your
> > > > >> > > > > > > opinions.
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > @Xuannan
> > > > >> > > > > > >
> > > > >> > > > > > > I agree with your comments 1 and 3.
> > > > >> > > > > > >
> > > > >> > > > > > > For 2, If we decide to change the name, maybe
> > > > >> > > > > > > `execution.checkpointing.parallel-cleaner` is better?
> > And as
> > > > >> for
> > > > >> > > > > whether
> > > > >> > > > > > to
> > > > >> > > > > > > add 'enabled' I suggest we discuss the rule above.
> WDYT?
> > > > >> > > > > > > Thanks!
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > Best,
> > > > >> > > > > > > Zakelly
> > > > >> > > > > > >
> > > > >> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
> > > > >> > suxuannan95@gmail.com
> > > > >> > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi Zakelly,
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks for driving this! The organization of the
> > > > >> configuration
> > > > >> > > > option
> > > > >> > > > > > > > in the FLIP looks much cleaner and easier to
> > understand.
> > > > +1
> > > > >> to
> > > > >> > > the
> > > > >> > > > > > > > FLIP.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Just some questions from me.
> > > > >> > > > > > > >
> > > > >> > > > > > > > 1. I think the change to the ConfigOptions should be
> > put
> > > > in
> > > > >> the
> > > > >> > > > > > > > `Public Interface` section, instead of `Proposed
> > Changed`,
> > > > >> as
> > > > >> > > those
> > > > >> > > > > > > > configuration options are public interface.
> > > > >> > > > > > > >
> > > > >> > > > > > > > 2. The key `state.checkpoint.cleaner.parallel-mode`
> > seems
> > > > >> > > > confusing.
> > > > >> > > > > > > > It feels like it is used to choose different modes.
> In
> > > > >> fact, it
> > > > >> > > is
> > > > >> > > > a
> > > > >> > > > > > > > boolean flag to indicate whether to enable parallel
> > clean.
> > > > >> How
> > > > >> > > > about
> > > > >> > > > > > > > making it
> > > > `state.checkpoint.cleaner.parallel-mode.enabled`?
> > > > >> > > > > > > >
> > > > >> > > > > > > > 3. The `execution.checkpointing.write-buffer` may
> > better
> > > > be
> > > > >> > > > > > > > `execution.checkpointing.write-buffer-size` so that
> we
> > > > know
> > > > >> it
> > > > >> > is
> > > > >> > > > > > > > configuring the size of the buffer.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Best,
> > > > >> > > > > > > > Xuannan
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
> > > > >> > fredialei@gmail.com>
> > > > >> > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Hi Zakelly,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Considering the name occupation, how about
> naming
> > it
> > > > as
> > > > >> > > > > > > > `execution.checkpointing.type`?
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > `Checkpoint Type`[1,2] is used to describe
> > > > >> aligned/unaligned
> > > > >> > > > > > > > > checkpoint, I am inclined to make a choice between
> > > > >> > > > > > > > > `execution.checkpointing.incremental` and
> > > > >> > > > > > > > > `execution.checkpointing.incremental.enabled`.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > [1]
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > > >> > > > > > > > > [2]
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > --
> > > > >> > > > > > > > > Best,
> > > > >> > > > > > > > > Yanfei
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Zakelly Lan <za...@gmail.com>
> 于2023年12月27日周三
> > > > >> 14:41写道:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Hi Lijie,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks for the reminder! I missed this.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Considering the name occupation, how about
> naming
> > it
> > > > as
> > > > >> > > > > > > > > > `execution.checkpointing.type`?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Actually I think the current
> > > > >> `execution.checkpointing.mode`
> > > > >> > > is
> > > > >> > > > > > > > confusing in
> > > > >> > > > > > > > > > some ways, maybe
> > > > >> `execution.checkpointing.data-consistency`
> > > > >> > > is
> > > > >> > > > > > > better.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Best,
> > > > >> > > > > > > > > > Zakelly
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > > > >> > > > > > > wangdachui9501@gmail.com>
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > Hi Zakelly,
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > >> I'm wondering if
> > > > >> > `execution.checkpointing.savepoint-dir`
> > > > >> > > > > would
> > > > >> > > > > > > be
> > > > >> > > > > > > > > > > better.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > `execution.checkpointing.dir` and
> > > > >> > > > > > > > `execution.checkpointing.savepoint-dir`
> > > > >> > > > > > > > > > > are also fine for me.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > >> So I think an enumeration option
> > > > >> > > > > > `execution.checkpointing.mode`
> > > > >> > > > > > > > which
> > > > >> > > > > > > > > > > can be 'full' (default) or 'incremental' would
> > be
> > > > >> better
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > I agree with using an enumeration option. But
> > > > >> currently
> > > > >> > > there
> > > > >> > > > > is
> > > > >> > > > > > > > already a
> > > > >> > > > > > > > > > > configuration option called
> > > > >> > `execution.checkpointing.mode`,
> > > > >> > > > > which
> > > > >> > > > > > > is
> > > > >> > > > > > > > used
> > > > >> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe
> > we
> > > > >> need to
> > > > >> > > use
> > > > >> > > > > > > > another name
> > > > >> > > > > > > > > > > or merge these two options.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > Lijie
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Zakelly Lan <za...@gmail.com>
> > 于2023年12月27日周三
> > > > >> > > 11:43写道:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > Hi everyone,
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Thanks all for your comments!
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > @Yanfei
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > 1. For some state backends that do not
> > support
> > > > >> > > > incremental
> > > > >> > > > > > > > checkpoint,
> > > > >> > > > > > > > > > > > > how does the
> > > > >> > execution.checkpointing.incrementaloption
> > > > >> > > > take
> > > > >> > > > > > > > effect? Or
> > > > >> > > > > > > > > > > > > is it better to put incremental under
> > > > >> > > > > > > > state.backend.xxx.incremental?
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > I'd rather not put the option for
> incremental
> > > > >> > checkpoint
> > > > >> > > > > under
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > 'state.backend', since it is more about the
> > > > >> > checkpointing
> > > > >> > > > > > instead
> > > > >> > > > > > > > of
> > > > >> > > > > > > > > > > state
> > > > >> > > > > > > > > > > > accessing. Of course, the state backend may
> > not
> > > > >> > > necessarily
> > > > >> > > > > do
> > > > >> > > > > > > > > > > incremental
> > > > >> > > > > > > > > > > > checkpoint as requested. If the state
> backend
> > is
> > > > not
> > > > >> > > > capable
> > > > >> > > > > of
> > > > >> > > > > > > > taking
> > > > >> > > > > > > > > > > > incremental cp, it is better to fallback to
> > the
> > > > full
> > > > >> > cp.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > 2. I'm a little worried that putting all
> > > > >> configurations
> > > > >> > > > into
> > > > >> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
> > introduce
> > > > >> some
> > > > >> > > > > > dependency
> > > > >> > > > > > > > > > > > > problems. Some options would be used by
> > > > >> flink-runtime
> > > > >> > > > > module,
> > > > >> > > > > > > but
> > > > >> > > > > > > > > > > > > flink-runtime should not depend on
> > > > >> > > flink-streaming-java.
> > > > >> > > > > e.g.
> > > > >> > > > > > > > > > > > > FLINK-28286[1].
> > > > >> > > > > > > > > > > > > So, I prefer to move configurations to
> > > > >> > > > > > `CheckpointingOptions`,
> > > > >> > > > > > > > WDYT?
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Yes, that's a very good point.  Moving to
> > > > >> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes
> > sense.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > @Lijie
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > How about
> > > > >> > > > > > > > > > > > > state.savepoints.dir ->
> > > > >> > > > > execution.checkpointing.savepoint.dir
> > > > >> > > > > > > > > > > > > state.checkpoints.dir ->
> > > > >> > > > > > execution.checkpointing.checkpoint.dir
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Actually, I think the
> > `checkpointing.checkpoint`
> > > > may
> > > > >> > > cause
> > > > >> > > > > some
> > > > >> > > > > > > > > > > confusion.
> > > > >> > > > > > > > > > > > But I'm ok if others agree.
> > > > >> > > > > > > > > > > > I'm wondering if
> > > > >> > `execution.checkpointing.savepoint-dir`
> > > > >> > > > > would
> > > > >> > > > > > be
> > > > >> > > > > > > > better.
> > > > >> > > > > > > > > > > > WDYT?
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > 2. We changed the
> > > > >> execution.checkpointing.local-copy'
> > > > >> > to
> > > > >> > > > > > > > > > > > >
> > 'execution.checkpointing.local-copy.enabled'.
> > > > >> Should
> > > > >> > we
> > > > >> > > > > also
> > > > >> > > > > > > add
> > > > >> > > > > > > > > > > > "enabled"
> > > > >> > > > > > > > > > > > > suffix for other boolean type
> configuration
> > > > >> options ?
> > > > >> > > For
> > > > >> > > > > > > > example,
> > > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > > > >> > > > > > > > > > > > >
> execution.checkpointing.incremental.enabled
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Actually, the incremental cp is something
> like
> > > > >> > choosing a
> > > > >> > > > > mode
> > > > >> > > > > > > for
> > > > >> > > > > > > > doing
> > > > >> > > > > > > > > > > > checkpoint instead of enabling a function.
> So
> > I
> > > > >> think
> > > > >> > an
> > > > >> > > > > > > > enumeration
> > > > >> > > > > > > > > > > option
> > > > >> > > > > > > > > > > > `execution.checkpointing.mode` which can be
> > 'full'
> > > > >> > > > (default)
> > > > >> > > > > or
> > > > >> > > > > > > > > > > > 'incremental' would be better, WDYT?
> > > > >> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think about
> > this?
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > > > >> > > > > > > > wangdachui9501@gmail.com>
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Hi Zakelly,
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Thanks for driving the discussion.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > 1.
> > > > >> > > > > > > > > > > > > >> But I'm not so sure since there is only
> > one
> > > > >> > > > > > > savepoint-related
> > > > >> > > > > > > > > > > option.
> > > > >> > > > > > > > > > > > > Maybe someone else could share some
> thoughts
> > > > here.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > How about
> > > > >> > > > > > > > > > > > > state.savepoints.dir ->
> > > > >> > > > > execution.checkpointing.savepoint.dir
> > > > >> > > > > > > > > > > > > state.checkpoints.dir ->
> > > > >> > > > > > execution.checkpointing.checkpoint.dir
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > 2. We changed the
> > > > >> execution.checkpointing.local-copy'
> > > > >> > > to
> > > > >> > > > > > > > > > > > >
> > 'execution.checkpointing.local-copy.enabled'.
> > > > >> Should
> > > > >> > we
> > > > >> > > > > also
> > > > >> > > > > > > add
> > > > >> > > > > > > > > > > > "enabled"
> > > > >> > > > > > > > > > > > > suffix for other boolean type
> configuration
> > > > >> options ?
> > > > >> > > For
> > > > >> > > > > > > > example,
> > > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > > > >> > > > > > > > > > > > >
> execution.checkpointing.incremental.enabled
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > In this way, the naming style of
> > configuration
> > > > >> > options
> > > > >> > > is
> > > > >> > > > > > > > unified, and
> > > > >> > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > can avoid potential similar problems (for
> > > > >> example, we
> > > > >> > > may
> > > > >> > > > > > need
> > > > >> > > > > > > > to add
> > > > >> > > > > > > > > > > > more
> > > > >> > > > > > > > > > > > > options for incremental checkpoint in the
> > > > future).
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > > Lijie
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Yanfei Lei <fr...@gmail.com>
> > 于2023年12月26日周二
> > > > >> > > 12:05写道:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Hi Zakelly,
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Thank you for creating the FLIP and
> > starting
> > > > the
> > > > >> > > > > > discussion.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > The current arrangement of these options
> > is
> > > > >> indeed
> > > > >> > > > > somewhat
> > > > >> > > > > > > > > > > haphazard,
> > > > >> > > > > > > > > > > > > > and the new arrangement looks much
> > better. I
> > > > >> have
> > > > >> > > some
> > > > >> > > > > > > > questions
> > > > >> > > > > > > > > > > about
> > > > >> > > > > > > > > > > > > > the arrangement of some new
> configuration
> > > > >> options:
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > 1. For some state backends that do not
> > support
> > > > >> > > > > incremental
> > > > >> > > > > > > > > > > checkpoint,
> > > > >> > > > > > > > > > > > > > how does the
> > > > >> > > execution.checkpointing.incrementaloption
> > > > >> > > > > take
> > > > >> > > > > > > > effect?
> > > > >> > > > > > > > > > > Or
> > > > >> > > > > > > > > > > > > > is it better to put incremental under
> > > > >> > > > > > > > state.backend.xxx.incremental?
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > 2. I'm a little worried that putting all
> > > > >> > > configurations
> > > > >> > > > > > into
> > > > >> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
> > introduce
> > > > >> some
> > > > >> > > > > > > dependency
> > > > >> > > > > > > > > > > > > > problems. Some options would be used by
> > > > >> > flink-runtime
> > > > >> > > > > > module,
> > > > >> > > > > > > > but
> > > > >> > > > > > > > > > > > > > flink-runtime should not depend on
> > > > >> > > > flink-streaming-java.
> > > > >> > > > > > e.g.
> > > > >> > > > > > > > > > > > > > FLINK-28286[1].
> > > > >> > > > > > > > > > > > > > So, I prefer to move configurations to
> > > > >> > > > > > > `CheckpointingOptions`,
> > > > >> > > > > > > > WDYT?
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > [1]
> > > > >> > > https://issues.apache.org/jira/browse/FLINK-28286
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > --
> > > > >> > > > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > > > Yanfei
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Zakelly Lan <za...@gmail.com>
> > > > >> 于2023年12月25日周一
> > > > >> > > > > > 21:14写道:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Hi Rui Fan and Junrui,
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Thanks for the reminder! I agree to
> > change
> > > > the
> > > > >> > > > > > > > > > > > > > > 'execution.checkpointing.local-copy'
> to
> > > > >> > > > > > > > > > > > > > >
> > > > 'execution.checkpointing.local-copy.enabled'.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > And for other suggestions Rui
> proposed:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > 1. How about
> > > > >> execution.checkpointing.storage.type
> > > > >> > > > > instead
> > > > >> > > > > > > > > > > > > > > > of execution.checkpointing.storage?
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Ah, I missed something here. Actually
> I
> > > > >> suggest
> > > > >> > we
> > > > >> > > > > could
> > > > >> > > > > > > > merge the
> > > > >> > > > > > > > > > > > > > current
> > > > >> > > > > > > > > > > > > > > 'state.checkpoints.dir' and
> > > > >> > > > 'state.checkpoint-storage'
> > > > >> > > > > > into
> > > > >> > > > > > > > one URI
> > > > >> > > > > > > > > > > > > > > configuration named
> > > > >> > 'execution.checkpointing.dir'.
> > > > >> > > > > WDYT?
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > 3.
> > execution.checkpointing.savepoint.dir is
> > > > a
> > > > >> > > little
> > > > >> > > > > > weird.
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Yes, I think it is better to make
> > > > 'savepoint'
> > > > >> and
> > > > >> > > > > > > > 'checkpoint' the
> > > > >> > > > > > > > > > > > same
> > > > >> > > > > > > > > > > > > > > level. But I'm not so sure since there
> > is
> > > > only
> > > > >> > one
> > > > >> > > > > > > > > > > savepoint-related
> > > > >> > > > > > > > > > > > > > > option. Maybe someone else could share
> > some
> > > > >> > > thoughts
> > > > >> > > > > > here.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > 4. How about
> > execution.recovery.claim-mode
> > > > >> > instead
> > > > >> > > of
> > > > >> > > > > > > > > > > > > > > > execution.recovery.mode?
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >  Agreed. That's more accurate.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Many thanks for your suggestions!
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > > > > Zakelly
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui
> > Lee <
> > > > >> > > > > > > > jrlee.ljr@gmail.com>
> > > > >> > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Hi Zakelly,
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Thanks for driving this. I agree
> that
> > the
> > > > >> > > proposed
> > > > >> > > > > > > > restructuring
> > > > >> > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > configuration options is largely
> > positive.
> > > > >> It
> > > > >> > > will
> > > > >> > > > > make
> > > > >> > > > > > > > > > > > understanding
> > > > >> > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > working with Flink configurations
> more
> > > > >> > intuitive.
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Most of the proposed changes look
> > great.
> > > > >> Just a
> > > > >> > > > > > heads-up,
> > > > >> > > > > > > > as Rui
> > > > >> > > > > > > > > > > > Fan
> > > > >> > > > > > > > > > > > > > > > mentioned, Flink currently requires
> > that
> > > > no
> > > > >> > > > > > > configOption's
> > > > >> > > > > > > > key be
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > prefix of another to avoid issues
> > when we
> > > > >> > > > eventually
> > > > >> > > > > > > adopt
> > > > >> > > > > > > > a
> > > > >> > > > > > > > > > > > standard
> > > > >> > > > > > > > > > > > > > YAML
> > > > >> > > > > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > https://issues.apache.org/jira/browse/FLINK-29372
> > > > >> > > > ).
> > > > >> > > > > > > > Therefore,
> > > > >> > > > > > > > > > > > it's
> > > > >> > > > > > > > > > > > > > better
> > > > >> > > > > > > > > > > > > > > > to change the key
> > > > >> > > > > 'execution.checkpointing.local-copy'
> > > > >> > > > > > > > because it
> > > > >> > > > > > > > > > > > > > serves as
> > > > >> > > > > > > > > > > > > > > > a prefix to the key
> > > > >> > > > > > > > 'execution.checkpointing.local-copy.dir'.
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Best regards,
> > > > >> > > > > > > > > > > > > > > > Junrui
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Rui Fan <19...@gmail.com>
> > > > >> 于2023年12月25日周一
> > > > >> > > > > 19:11写道:
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Hi Zakelly,
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Thank you for driving this
> proposal!
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Overall good for me. I have some
> > > > questions
> > > > >> > > about
> > > > >> > > > > > these
> > > > >> > > > > > > > names.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > 1. How about
> > > > >> > > execution.checkpointing.storage.type
> > > > >> > > > > > > > instead of
> > > > >> > > > > > > > > > > > > > > > > execution.checkpointing.storage?
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > It's similar to
> state.backend.type.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > 2. How about
> > > > >> > > > > > execution.checkpointing.local-copy.enabled
> > > > >> > > > > > > > instead
> > > > >> > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > > > >
> execution.checkpointing.local-copy?
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > You added a new option:
> > > > >> > > > > > > > execution.checkpointing.local-copy.dir.
> > > > >> > > > > > > > > > > > > > > > > IIUC, one option name shouldn't be
> > the
> > > > >> prefix
> > > > >> > > of
> > > > >> > > > > > other
> > > > >> > > > > > > > options.
> > > > >> > > > > > > > > > > > > > > > > If you add a new option
> > > > >> > > > > > > > execution.checkpointing.local-copy,
> > > > >> > > > > > > > > > > > > > > > > flink CI will fail directly.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > 3.
> > execution.checkpointing.savepoint.dir
> > > > >> is a
> > > > >> > > > > little
> > > > >> > > > > > > > weird.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > For old options:
> > state.savepoints.dir
> > > > and
> > > > >> > > > > > > > > > > state.checkpoints.dir,
> > > > >> > > > > > > > > > > > > > > > > the savepoint and checkpoint are
> the
> > > > same
> > > > >> > > level.
> > > > >> > > > It
> > > > >> > > > > > > means
> > > > >> > > > > > > > > > > > > > > > > it's a checkpoint or savepoint.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > The new option
> > > > >> execution.checkpointing.dir is
> > > > >> > > > fine
> > > > >> > > > > > for
> > > > >> > > > > > > > me.
> > > > >> > > > > > > > > > > > > > > > > However,
> > > > >> > execution.checkpointing.savepoint.dir
> > > > >> > > > is a
> > > > >> > > > > > > > little
> > > > >> > > > > > > > > > > weird.
> > > > >> > > > > > > > > > > > > > > > > I don't know which name is better
> > now.
> > > > >> Let us
> > > > >> > > > think
> > > > >> > > > > > > > about it
> > > > >> > > > > > > > > > > > more.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > 4. How about
> > > > execution.recovery.claim-mode
> > > > >> > > > instead
> > > > >> > > > > of
> > > > >> > > > > > > > > > > > > > > > > execution.recovery.mode?
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > The meaning of mode is too broad.
> > The
> > > > >> > > claim-mode
> > > > >> > > > > may
> > > > >> > > > > > > > > > > > > > > > > be more accurate for users.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > WDYT?
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > > > > > > Rui
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM
> > Zakelly
> > > > >> Lan <
> > > > >> > > > > > > > > > > > zakelly.lan@gmail.com
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > Hi devs,
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > I'd like to start a discussion
> on
> > > > >> FLIP-406:
> > > > >> > > > > > > Reorganize
> > > > >> > > > > > > > State
> > > > >> > > > > > > > > > > &
> > > > >> > > > > > > > > > > > > > > > > > Checkpointing & Recovery
> > > > >> Configuration[1].
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > Currently, the configuration
> > options
> > > > >> > > pertaining
> > > > >> > > > > to
> > > > >> > > > > > > > > > > > checkpointing,
> > > > >> > > > > > > > > > > > > > > > > recovery,
> > > > >> > > > > > > > > > > > > > > > > > and state management are
> primarily
> > > > >> grouped
> > > > >> > > > under
> > > > >> > > > > > the
> > > > >> > > > > > > > > > > following
> > > > >> > > > > > > > > > > > > > > > prefixes:
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > >    - state.backend.* :
> > configurations
> > > > >> > related
> > > > >> > > > to
> > > > >> > > > > > > state
> > > > >> > > > > > > > > > > > accessing
> > > > >> > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > > >    checkpointing, as well as
> > specific
> > > > >> > options
> > > > >> > > > for
> > > > >> > > > > > > > individual
> > > > >> > > > > > > > > > > > > state
> > > > >> > > > > > > > > > > > > > > > > backends
> > > > >> > > > > > > > > > > > > > > > > >    - execution.checkpointing.* :
> > > > >> > > configurations
> > > > >> > > > > > > > associated
> > > > >> > > > > > > > > > > with
> > > > >> > > > > > > > > > > > > > > > > checkpoint
> > > > >> > > > > > > > > > > > > > > > > >    execution and recovery
> > > > >> > > > > > > > > > > > > > > > > >    - execution.savepoint.*:
> > > > >> configurations
> > > > >> > > for
> > > > >> > > > > > > > recovery from
> > > > >> > > > > > > > > > > > > > savepoint
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > In addition, there are several
> > > > >> individual
> > > > >> > > > options
> > > > >> > > > > > > such
> > > > >> > > > > > > > as '
> > > > >> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*' and
> > > > >> > > > > > > > '*state.checkpoints.dir*'
> > > > >> > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > fall
> > > > >> > > > > > > > > > > > > > > > > outside
> > > > >> > > > > > > > > > > > > > > > > > of these prefixes. The current
> > > > >> arrangement
> > > > >> > of
> > > > >> > > > > these
> > > > >> > > > > > > > options,
> > > > >> > > > > > > > > > > > > which
> > > > >> > > > > > > > > > > > > > span
> > > > >> > > > > > > > > > > > > > > > > > multiple modules, is somewhat
> > > > haphazard
> > > > >> and
> > > > >> > > > > lacks a
> > > > >> > > > > > > > > > > systematic
> > > > >> > > > > > > > > > > > > > > > structure.
> > > > >> > > > > > > > > > > > > > > > > > For example, the options under
> the
> > > > >> > > > > > > > '*CheckpointingOptions*'
> > > > >> > > > > > > > > > > > and '
> > > > >> > > > > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*'
> > are
> > > > >> > related
> > > > >> > > > and
> > > > >> > > > > > have
> > > > >> > > > > > > > no
> > > > >> > > > > > > > > > > clear
> > > > >> > > > > > > > > > > > > > > > boundaries
> > > > >> > > > > > > > > > > > > > > > > > from the user's perspective, but
> > there
> > > > >> is
> > > > >> > no
> > > > >> > > > > > unified
> > > > >> > > > > > > > prefix
> > > > >> > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > them.
> > > > >> > > > > > > > > > > > > > > > > With
> > > > >> > > > > > > > > > > > > > > > > > the upcoming release of Flink
> > 2.0, we
> > > > >> have
> > > > >> > an
> > > > >> > > > > > > excellent
> > > > >> > > > > > > > > > > > > > opportunity to
> > > > >> > > > > > > > > > > > > > > > > > overhaul and restructure the
> > > > >> configurations
> > > > >> > > > > related
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > > > > > checkpointing,
> > > > >> > > > > > > > > > > > > > > > > > recovery, and state management.
> > This
> > > > >> FLIP
> > > > >> > > > > proposes
> > > > >> > > > > > to
> > > > >> > > > > > > > > > > > reorganize
> > > > >> > > > > > > > > > > > > > these
> > > > >> > > > > > > > > > > > > > > > > > settings, making it more
> coherent
> > by
> > > > >> > module,
> > > > >> > > > > which
> > > > >> > > > > > > > would
> > > > >> > > > > > > > > > > > > > significantly
> > > > >> > > > > > > > > > > > > > > > > > lower the barriers for
> > understanding
> > > > and
> > > > >> > > reduce
> > > > >> > > > > the
> > > > >> > > > > > > > > > > development
> > > > >> > > > > > > > > > > > > > costs
> > > > >> > > > > > > > > > > > > > > > > > moving forward.
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > Looking forward to hearing from
> > you!
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > [1]
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > > > > > > > Zakelly
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Best,
> > > > >> Hangxiang.
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Hangxiang.
> >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi everyone,

Thanks all for joining the discussion! I'd like to speed this up since it
lasts for nearly a month. I made changes on this FLIP based on suggestions
and compromises acceptable to most people. Please feel free to give your
opinion. Thanks!
If there are no more suggestions, I will consider starting a vote within a
week.


Best,
Zakelly

On Thu, Jan 11, 2024 at 10:31 AM Xuannan Su <su...@gmail.com> wrote:

> Hi Zakelly,
>
> I am fine with either Option 2 or Option 3. I think the naming in
> Option 2 makes it clear that it is a boolean configuration. However,
> most of the currently available boolean configurations do not use
> "enable" as a suffix. Therefore, Option 3 looks good to me as it
> follows the current practice.
>
> Best regards,
> Xuannan
>
> On Thu, Jan 11, 2024 at 9:50 AM Hangxiang Yu <ma...@gmail.com> wrote:
> >
> > >
> > > That's a very good point. I realize that the word 'recovery' means way
> too
> > > many things. So I suggest picking a more specific word here, how about
> > > 'execution.state-recovery.*' ? Checkpointing and state recovery are
> > > corresponding terms and won't make ambiguity.
> > >
> >
> > This makes the configuration clearer to me. We could focus on the
> > `state-recovery` at first.
> >
> > I think we could create another FLIP for the deprecation of LEGACY mode.
> > >
> >
> > LGTM, Let's create a new FLIP to do this.
> >
> > IIUC, there is no clear ownership of the local copy files from the
> previous
> > > job and it's better to define one. This needs more discussion so we
> could
> > > create another thread for this. WDYT?
> > >
> >
> > Yeah, I have created a new ticket FLINK-34032 to track and discuss this.
> >
> > On Wed, Jan 10, 2024 at 6:31 PM Zakelly Lan <za...@gmail.com>
> wrote:
> >
> > > Hi everyone,
> > >
> > > It seems we still don't have a consensus on the rules for boolean type
> > > options. Let me recap the alternatives we have:
> > >
> > > Option 1: Use enumeration options instead if possible. But this may
> cause
> > > some name collisions or confusion as we discussed and we should unify
> the
> > > statement everywhere.
> > > Option 2: Use boolean options and add 'enabled' as the suffix.
> > > Option 3: Use boolean options and ONLY add 'enabled' when there are
> more
> > > detailed configurations under the same prefix, to prevent one name from
> > > serving as a prefix to another.
> > >
> > > I am inclined to Option 3, since it is more in line with current
> practice
> > > and friendly for existing users. Also It reduces the length of
> > > configuration names as much as possible.
> > >
> > > Looking forward to your opinions! Thanks!
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan <za...@gmail.com>
> wrote:
> > >
> > > > Hi Hangxiang,
> > > >
> > > > Thanks for your suggestions!
> > > >
> > > > 1. Could execution.recovery also contain some other behaviors about
> > > >> recovery ? e.g. restart-strategy.
> > > >
> > > >
> > > > That's a very good point. I realize that the word 'recovery' means
> way
> > > too
> > > > many things. So I suggest picking a more specific word here, how
> about
> > > > 'execution.state-recovery.*' ? Checkpointing and state recovery are
> > > > corresponding terms and won't make ambiguity.
> > > >
> > > > 2. Could we also remove some legacy configuration value ? e.g. LEGACY
> > > Mode
> > > >> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
> > > >
> > > >
> > > > I think we could create another FLIP for the deprecation of LEGACY
> mode.
> > > >
> > > >
> > > >> 3. Could the local checkpoint be cleaned
> > > >> if execution.checkpointing.local-copy.enabled is true and
> > > >> execution.recovery.from-local is false ? I found it's also an issue
> if
> > > >> current local-recovery from enabled to disabled. Maybe another
> ticket is
> > > >> needed.
> > > >
> > > >
> > > > IIUC, there is no clear ownership of the local copy files from the
> > > > previous job and it's better to define one. This needs more
> discussion so
> > > > we could create another thread for this. WDYT?
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu <ma...@gmail.com>
> > > wrote:
> > > >
> > > >> Hi, Zakelly.
> > > >> Thanks for driving this. Overall LGTM as we discussed offline.
> > > >>
> > > >> Some comments/suggestions just came to mind:
> > > >> 1. Could execution.recovery also contain some other behaviors about
> > > >> recovery ? e.g. restart-strategy.
> > > >> 2. Could we also remove some legacy configuration value ? e.g.
> LEGACY
> > > Mode
> > > >> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
> > > >> 3. Could the local checkpoint be cleaned
> > > >> if execution.checkpointing.local-copy.enabled is true and
> > > >> execution.recovery.from-local is false ? I found it's also an issue
> if
> > > >> current local-recovery from enabled to disabled. Maybe another
> ticket is
> > > >> needed.
> > > >> 4. +1 for enabling execution.checkpointing.incremental by default
> which
> > > is
> > > >> basically default configuration in our production environment.
> > > >>
> > > >>
> > > >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <za...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Hi Yun,
> > > >> >
> > > >> > Thanks for your comments!
> > > >> >
> > > >> >  1.  We shall not describe the configuration with its
> implementation
> > > for
> > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > > >> > state-backend,
> > > >> > > it would write two streams and for Rocksdb state-backend, it
> would
> > > use
> > > >> > > hard-link for backup. Thus, I think
> > > >> > > 'execution.checkpointing.local-backup.*' looks better.
> > > >> >
> > > >> > I agreed that we'd better name the option in user's perspective
> > > instead
> > > >> of
> > > >> > the implementation, thus I name it as a copy of the checkpoint in
> the
> > > >> > local disk, regardless of the way of generating it. The word
> 'backup'
> > > is
> > > >> > also suitable for this case, so I agree to change to
> > > >> > 'execution.checkpointing.local-backup.*' if no one objects.
> > > >> >
> > > >> >  2.  What does the 'execution.checkpointing.data-inline-threshold'
> > > >> mean? It
> > > >> > > seems not so easy to understand.
> > > >> >
> > > >> > The 'execution.checkpointing.data-inline-threshold' (original one
> as
> > > >> > 'state.storage.fs.memory-threshold') stands for the size threshold
> > > below
> > > >> > which state chunks will store inline with the metadata, thus I
> call it
> > > >> > 'data-inline-threshold'.
> > > >> >
> > > >> >
> > > >> > Best,
> > > >> > Zakelly
> > > >> >
> > > >> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com>
> wrote:
> > > >> >
> > > >> > > Hi Zakelly,
> > > >> > >
> > > >> > > Thanks for driving this topic. I have two concerns here:
> > > >> > >
> > > >> > >   1.  We shall not describe the configuration with its
> > > implementation
> > > >> for
> > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > > >> > state-backend,
> > > >> > > it would write two streams and for Rocksdb state-backend, it
> would
> > > use
> > > >> > > hard-link for backup. Thus, I think
> > > >> > > 'execution.checkpointing.local-backup.*' looks better.
> > > >> > >   2.  What does the
> 'execution.checkpointing.data-inline-threshold'
> > > >> mean?
> > > >> > > It seems not so easy to understand.
> > > >> > >
> > > >> > > Best
> > > >> > > Yun Tang
> > > >> > > ________________________________
> > > >> > > From: Piotr Nowojski <pn...@apache.org>
> > > >> > > Sent: Thursday, January 4, 2024 22:37
> > > >> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > >> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State &
> Checkpointing &
> > > >> > > Recovery Configuration
> > > >> > >
> > > >> > > Hi,
> > > >> > >
> > > >> > > Thanks for trying to clean this up! I don't have strong
> opinions on
> > > >> the
> > > >> > > topics discussed here, so generally speaking +1 from my side!
> > > >> > >
> > > >> > > Best,
> > > >> > > Piotrek
> > > >> > >
> > > >> > > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com>
> napisał(a):
> > > >> > >
> > > >> > > > Thanks for the feedback!
> > > >> > > >
> > > >> > > > Using the `execution.checkpointing.incremental.enabled`,
> > > >> > > > and enabling it by default sounds good to me.
> > > >> > > >
> > > >> > > > Best,
> > > >> > > > Rui
> > > >> > > >
> > > >> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <
> > > zakelly.lan@gmail.com>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Hi Rui,
> > > >> > > > >
> > > >> > > > > Thanks for your comments!
> > > >> > > > >
> > > >> > > > > IMO, given that the state backend can be plugably loaded
> (as you
> > > >> can
> > > >> > > > > specify a state backend factory), I prefer not providing
> state
> > > >> > backend
> > > >> > > > > specified options in the framework.
> > > >> > > > >
> > > >> > > > > Secondly, the incremental checkpoint is actually a sharing
> file
> > > >> > > strategy
> > > >> > > > > across checkpoints, which means the state backend *could*
> reuse
> > > >> files
> > > >> > > > from
> > > >> > > > > previous cp but not *must* do so. When the state backend
> could
> > > not
> > > >> > > reuse
> > > >> > > > > the files, it is reasonable to fallback to a full
> checkpoint.
> > > >> > > > >
> > > >> > > > > Thus, I suggest we make it
> `execution.checkpointing.incremental`
> > > >> and
> > > >> > > > enable
> > > >> > > > > it by default. For those state backends not supporting this,
> > > they
> > > >> > > perform
> > > >> > > > > full checkpoints and print a warning to inform users. Users
> do
> > > not
> > > >> > need
> > > >> > > > to
> > > >> > > > > pay special attention to different options to control this
> > > across
> > > >> > > > different
> > > >> > > > > state backends. This is more user-friendly in my opinion.
> WDYT?
> > > >> > > > >
> > > >> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <
> 1996fanrui@gmail.com>
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > > Hi Zakelly,
> > > >> > > > > >
> > > >> > > > > > I'm not sure whether we could add the state backend type
> in
> > > the
> > > >> > > > > > new key name of state.backend.incremental. It means we use
> > > >> > > > > > `execution.checkpointing.rocksdb-incremental` or
> > > >> > > > > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > > >> > > > > >
> > > >> > > > > > So far, state.backend.incremental only works for rocksdb
> state
> > > >> > > backend.
> > > >> > > > > > And this feature or optimization is very valuable and
> huge for
> > > >> > large
> > > >> > > > > > state flink jobs. I believe it's enabled for most
> production
> > > >> flink
> > > >> > > jobs
> > > >> > > > > > with large rocksdb state.
> > > >> > > > > >
> > > >> > > > > > If this option isn't generic for all state backend types,
> I
> > > >> guess
> > > >> > we
> > > >> > > > > > can enable
> > > `execution.checkpointing.rocksdb-incremental.enabled`
> > > >> > > > > > by default in Flink 2.0.
> > > >> > > > > >
> > > >> > > > > > But if it works for all state backends, it's hard to
> enable it
> > > >> by
> > > >> > > > > default.
> > > >> > > > > > Enabling great and valuable features or improvements are
> > > useful
> > > >> > > > > > for users, especially a lot of new flink users.
> Out-of-the-box
> > > >> > > options
> > > >> > > > > > are good for users.
> > > >> > > > > >
> > > >> > > > > > WDYT?
> > > >> > > > > >
> > > >> > > > > > Best,
> > > >> > > > > > Rui
> > > >> > > > > >
> > > >> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <
> > > >> zakelly.lan@gmail.com
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi everyone,
> > > >> > > > > > >
> > > >> > > > > > > Thanks all for your comments!
> > > >> > > > > > >
> > > >> > > > > > > As many of you have questions about the names for
> boolean
> > > >> > options,
> > > >> > > I
> > > >> > > > > > > suggest we make a naming rule for them. For now I could
> > > think
> > > >> of
> > > >> > > > three
> > > >> > > > > > > options:
> > > >> > > > > > >
> > > >> > > > > > > Option 1: Use enumeration options if possible. But this
> may
> > > >> cause
> > > >> > > > some
> > > >> > > > > > name
> > > >> > > > > > > collisions or confusion as we discussed and we should
> unify
> > > >> the
> > > >> > > > > statement
> > > >> > > > > > > everywhere.
> > > >> > > > > > > Option 2: Use boolean options and add 'enabled' as the
> > > suffix.
> > > >> > > > > > > Option 3: Use boolean options and ONLY add 'enabled'
> when
> > > >> there
> > > >> > are
> > > >> > > > > more
> > > >> > > > > > > detailed configurations under the same prefix, to
> prevent
> > > one
> > > >> > name
> > > >> > > > from
> > > >> > > > > > > serving as a prefix to another.
> > > >> > > > > > >
> > > >> > > > > > > I am slightly inclined to Option 3, since it is more in
> line
> > > >> with
> > > >> > > > > current
> > > >> > > > > > > practice and friendly for existing users. Also It
> reduces
> > > the
> > > >> > > length
> > > >> > > > of
> > > >> > > > > > > configuration names as much as possible. I really want
> to
> > > hear
> > > >> > your
> > > >> > > > > > > opinions.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > @Xuannan
> > > >> > > > > > >
> > > >> > > > > > > I agree with your comments 1 and 3.
> > > >> > > > > > >
> > > >> > > > > > > For 2, If we decide to change the name, maybe
> > > >> > > > > > > `execution.checkpointing.parallel-cleaner` is better?
> And as
> > > >> for
> > > >> > > > > whether
> > > >> > > > > > to
> > > >> > > > > > > add 'enabled' I suggest we discuss the rule above. WDYT?
> > > >> > > > > > > Thanks!
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > Best,
> > > >> > > > > > > Zakelly
> > > >> > > > > > >
> > > >> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
> > > >> > suxuannan95@gmail.com
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi Zakelly,
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks for driving this! The organization of the
> > > >> configuration
> > > >> > > > option
> > > >> > > > > > > > in the FLIP looks much cleaner and easier to
> understand.
> > > +1
> > > >> to
> > > >> > > the
> > > >> > > > > > > > FLIP.
> > > >> > > > > > > >
> > > >> > > > > > > > Just some questions from me.
> > > >> > > > > > > >
> > > >> > > > > > > > 1. I think the change to the ConfigOptions should be
> put
> > > in
> > > >> the
> > > >> > > > > > > > `Public Interface` section, instead of `Proposed
> Changed`,
> > > >> as
> > > >> > > those
> > > >> > > > > > > > configuration options are public interface.
> > > >> > > > > > > >
> > > >> > > > > > > > 2. The key `state.checkpoint.cleaner.parallel-mode`
> seems
> > > >> > > > confusing.
> > > >> > > > > > > > It feels like it is used to choose different modes. In
> > > >> fact, it
> > > >> > > is
> > > >> > > > a
> > > >> > > > > > > > boolean flag to indicate whether to enable parallel
> clean.
> > > >> How
> > > >> > > > about
> > > >> > > > > > > > making it
> > > `state.checkpoint.cleaner.parallel-mode.enabled`?
> > > >> > > > > > > >
> > > >> > > > > > > > 3. The `execution.checkpointing.write-buffer` may
> better
> > > be
> > > >> > > > > > > > `execution.checkpointing.write-buffer-size` so that we
> > > know
> > > >> it
> > > >> > is
> > > >> > > > > > > > configuring the size of the buffer.
> > > >> > > > > > > >
> > > >> > > > > > > > Best,
> > > >> > > > > > > > Xuannan
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
> > > >> > fredialei@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > Hi Zakelly,
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Considering the name occupation, how about naming
> it
> > > as
> > > >> > > > > > > > `execution.checkpointing.type`?
> > > >> > > > > > > > >
> > > >> > > > > > > > > `Checkpoint Type`[1,2] is used to describe
> > > >> aligned/unaligned
> > > >> > > > > > > > > checkpoint, I am inclined to make a choice between
> > > >> > > > > > > > > `execution.checkpointing.incremental` and
> > > >> > > > > > > > > `execution.checkpointing.incremental.enabled`.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > [1]
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > >> > > > > > > > > [2]
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > >> > > > > > > > >
> > > >> > > > > > > > > --
> > > >> > > > > > > > > Best,
> > > >> > > > > > > > > Yanfei
> > > >> > > > > > > > >
> > > >> > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
> > > >> 14:41写道:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Hi Lijie,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for the reminder! I missed this.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Considering the name occupation, how about naming
> it
> > > as
> > > >> > > > > > > > > > `execution.checkpointing.type`?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Actually I think the current
> > > >> `execution.checkpointing.mode`
> > > >> > > is
> > > >> > > > > > > > confusing in
> > > >> > > > > > > > > > some ways, maybe
> > > >> `execution.checkpointing.data-consistency`
> > > >> > > is
> > > >> > > > > > > better.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Best,
> > > >> > > > > > > > > > Zakelly
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > > >> > > > > > > wangdachui9501@gmail.com>
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi Zakelly,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > >> I'm wondering if
> > > >> > `execution.checkpointing.savepoint-dir`
> > > >> > > > > would
> > > >> > > > > > > be
> > > >> > > > > > > > > > > better.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > `execution.checkpointing.dir` and
> > > >> > > > > > > > `execution.checkpointing.savepoint-dir`
> > > >> > > > > > > > > > > are also fine for me.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > >> So I think an enumeration option
> > > >> > > > > > `execution.checkpointing.mode`
> > > >> > > > > > > > which
> > > >> > > > > > > > > > > can be 'full' (default) or 'incremental' would
> be
> > > >> better
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I agree with using an enumeration option. But
> > > >> currently
> > > >> > > there
> > > >> > > > > is
> > > >> > > > > > > > already a
> > > >> > > > > > > > > > > configuration option called
> > > >> > `execution.checkpointing.mode`,
> > > >> > > > > which
> > > >> > > > > > > is
> > > >> > > > > > > > used
> > > >> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe
> we
> > > >> need to
> > > >> > > use
> > > >> > > > > > > > another name
> > > >> > > > > > > > > > > or merge these two options.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Best,
> > > >> > > > > > > > > > > Lijie
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Zakelly Lan <za...@gmail.com>
> 于2023年12月27日周三
> > > >> > > 11:43写道:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Hi everyone,
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Thanks all for your comments!
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > @Yanfei
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > 1. For some state backends that do not
> support
> > > >> > > > incremental
> > > >> > > > > > > > checkpoint,
> > > >> > > > > > > > > > > > > how does the
> > > >> > execution.checkpointing.incrementaloption
> > > >> > > > take
> > > >> > > > > > > > effect? Or
> > > >> > > > > > > > > > > > > is it better to put incremental under
> > > >> > > > > > > > state.backend.xxx.incremental?
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > I'd rather not put the option for incremental
> > > >> > checkpoint
> > > >> > > > > under
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > 'state.backend', since it is more about the
> > > >> > checkpointing
> > > >> > > > > > instead
> > > >> > > > > > > > of
> > > >> > > > > > > > > > > state
> > > >> > > > > > > > > > > > accessing. Of course, the state backend may
> not
> > > >> > > necessarily
> > > >> > > > > do
> > > >> > > > > > > > > > > incremental
> > > >> > > > > > > > > > > > checkpoint as requested. If the state backend
> is
> > > not
> > > >> > > > capable
> > > >> > > > > of
> > > >> > > > > > > > taking
> > > >> > > > > > > > > > > > incremental cp, it is better to fallback to
> the
> > > full
> > > >> > cp.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > 2. I'm a little worried that putting all
> > > >> configurations
> > > >> > > > into
> > > >> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
> introduce
> > > >> some
> > > >> > > > > > dependency
> > > >> > > > > > > > > > > > > problems. Some options would be used by
> > > >> flink-runtime
> > > >> > > > > module,
> > > >> > > > > > > but
> > > >> > > > > > > > > > > > > flink-runtime should not depend on
> > > >> > > flink-streaming-java.
> > > >> > > > > e.g.
> > > >> > > > > > > > > > > > > FLINK-28286[1].
> > > >> > > > > > > > > > > > > So, I prefer to move configurations to
> > > >> > > > > > `CheckpointingOptions`,
> > > >> > > > > > > > WDYT?
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Yes, that's a very good point.  Moving to
> > > >> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes
> sense.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > @Lijie
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > How about
> > > >> > > > > > > > > > > > > state.savepoints.dir ->
> > > >> > > > > execution.checkpointing.savepoint.dir
> > > >> > > > > > > > > > > > > state.checkpoints.dir ->
> > > >> > > > > > execution.checkpointing.checkpoint.dir
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Actually, I think the
> `checkpointing.checkpoint`
> > > may
> > > >> > > cause
> > > >> > > > > some
> > > >> > > > > > > > > > > confusion.
> > > >> > > > > > > > > > > > But I'm ok if others agree.
> > > >> > > > > > > > > > > > I'm wondering if
> > > >> > `execution.checkpointing.savepoint-dir`
> > > >> > > > > would
> > > >> > > > > > be
> > > >> > > > > > > > better.
> > > >> > > > > > > > > > > > WDYT?
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > 2. We changed the
> > > >> execution.checkpointing.local-copy'
> > > >> > to
> > > >> > > > > > > > > > > > >
> 'execution.checkpointing.local-copy.enabled'.
> > > >> Should
> > > >> > we
> > > >> > > > > also
> > > >> > > > > > > add
> > > >> > > > > > > > > > > > "enabled"
> > > >> > > > > > > > > > > > > suffix for other boolean type configuration
> > > >> options ?
> > > >> > > For
> > > >> > > > > > > > example,
> > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > > >> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Actually, the incremental cp is something like
> > > >> > choosing a
> > > >> > > > > mode
> > > >> > > > > > > for
> > > >> > > > > > > > doing
> > > >> > > > > > > > > > > > checkpoint instead of enabling a function. So
> I
> > > >> think
> > > >> > an
> > > >> > > > > > > > enumeration
> > > >> > > > > > > > > > > option
> > > >> > > > > > > > > > > > `execution.checkpointing.mode` which can be
> 'full'
> > > >> > > > (default)
> > > >> > > > > or
> > > >> > > > > > > > > > > > 'incremental' would be better, WDYT?
> > > >> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think about
> this?
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > > >> > > > > > > > wangdachui9501@gmail.com>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Hi Zakelly,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks for driving the discussion.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > 1.
> > > >> > > > > > > > > > > > > >> But I'm not so sure since there is only
> one
> > > >> > > > > > > savepoint-related
> > > >> > > > > > > > > > > option.
> > > >> > > > > > > > > > > > > Maybe someone else could share some thoughts
> > > here.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > How about
> > > >> > > > > > > > > > > > > state.savepoints.dir ->
> > > >> > > > > execution.checkpointing.savepoint.dir
> > > >> > > > > > > > > > > > > state.checkpoints.dir ->
> > > >> > > > > > execution.checkpointing.checkpoint.dir
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > 2. We changed the
> > > >> execution.checkpointing.local-copy'
> > > >> > > to
> > > >> > > > > > > > > > > > >
> 'execution.checkpointing.local-copy.enabled'.
> > > >> Should
> > > >> > we
> > > >> > > > > also
> > > >> > > > > > > add
> > > >> > > > > > > > > > > > "enabled"
> > > >> > > > > > > > > > > > > suffix for other boolean type configuration
> > > >> options ?
> > > >> > > For
> > > >> > > > > > > > example,
> > > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > > >> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > In this way, the naming style of
> configuration
> > > >> > options
> > > >> > > is
> > > >> > > > > > > > unified, and
> > > >> > > > > > > > > > > it
> > > >> > > > > > > > > > > > > can avoid potential similar problems (for
> > > >> example, we
> > > >> > > may
> > > >> > > > > > need
> > > >> > > > > > > > to add
> > > >> > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > options for incremental checkpoint in the
> > > future).
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > Lijie
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Yanfei Lei <fr...@gmail.com>
> 于2023年12月26日周二
> > > >> > > 12:05写道:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Hi Zakelly,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Thank you for creating the FLIP and
> starting
> > > the
> > > >> > > > > > discussion.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > The current arrangement of these options
> is
> > > >> indeed
> > > >> > > > > somewhat
> > > >> > > > > > > > > > > haphazard,
> > > >> > > > > > > > > > > > > > and the new arrangement looks much
> better. I
> > > >> have
> > > >> > > some
> > > >> > > > > > > > questions
> > > >> > > > > > > > > > > about
> > > >> > > > > > > > > > > > > > the arrangement of some new configuration
> > > >> options:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > 1. For some state backends that do not
> support
> > > >> > > > > incremental
> > > >> > > > > > > > > > > checkpoint,
> > > >> > > > > > > > > > > > > > how does the
> > > >> > > execution.checkpointing.incrementaloption
> > > >> > > > > take
> > > >> > > > > > > > effect?
> > > >> > > > > > > > > > > Or
> > > >> > > > > > > > > > > > > > is it better to put incremental under
> > > >> > > > > > > > state.backend.xxx.incremental?
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > 2. I'm a little worried that putting all
> > > >> > > configurations
> > > >> > > > > > into
> > > >> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will
> introduce
> > > >> some
> > > >> > > > > > > dependency
> > > >> > > > > > > > > > > > > > problems. Some options would be used by
> > > >> > flink-runtime
> > > >> > > > > > module,
> > > >> > > > > > > > but
> > > >> > > > > > > > > > > > > > flink-runtime should not depend on
> > > >> > > > flink-streaming-java.
> > > >> > > > > > e.g.
> > > >> > > > > > > > > > > > > > FLINK-28286[1].
> > > >> > > > > > > > > > > > > > So, I prefer to move configurations to
> > > >> > > > > > > `CheckpointingOptions`,
> > > >> > > > > > > > WDYT?
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > [1]
> > > >> > > https://issues.apache.org/jira/browse/FLINK-28286
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > --
> > > >> > > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > > Yanfei
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Zakelly Lan <za...@gmail.com>
> > > >> 于2023年12月25日周一
> > > >> > > > > > 21:14写道:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Hi Rui Fan and Junrui,
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Thanks for the reminder! I agree to
> change
> > > the
> > > >> > > > > > > > > > > > > > > 'execution.checkpointing.local-copy' to
> > > >> > > > > > > > > > > > > > >
> > > 'execution.checkpointing.local-copy.enabled'.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > And for other suggestions Rui proposed:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > 1. How about
> > > >> execution.checkpointing.storage.type
> > > >> > > > > instead
> > > >> > > > > > > > > > > > > > > > of execution.checkpointing.storage?
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Ah, I missed something here. Actually I
> > > >> suggest
> > > >> > we
> > > >> > > > > could
> > > >> > > > > > > > merge the
> > > >> > > > > > > > > > > > > > current
> > > >> > > > > > > > > > > > > > > 'state.checkpoints.dir' and
> > > >> > > > 'state.checkpoint-storage'
> > > >> > > > > > into
> > > >> > > > > > > > one URI
> > > >> > > > > > > > > > > > > > > configuration named
> > > >> > 'execution.checkpointing.dir'.
> > > >> > > > > WDYT?
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > 3.
> execution.checkpointing.savepoint.dir is
> > > a
> > > >> > > little
> > > >> > > > > > weird.
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Yes, I think it is better to make
> > > 'savepoint'
> > > >> and
> > > >> > > > > > > > 'checkpoint' the
> > > >> > > > > > > > > > > > same
> > > >> > > > > > > > > > > > > > > level. But I'm not so sure since there
> is
> > > only
> > > >> > one
> > > >> > > > > > > > > > > savepoint-related
> > > >> > > > > > > > > > > > > > > option. Maybe someone else could share
> some
> > > >> > > thoughts
> > > >> > > > > > here.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > 4. How about
> execution.recovery.claim-mode
> > > >> > instead
> > > >> > > of
> > > >> > > > > > > > > > > > > > > > execution.recovery.mode?
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >  Agreed. That's more accurate.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Many thanks for your suggestions!
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > > > Zakelly
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui
> Lee <
> > > >> > > > > > > > jrlee.ljr@gmail.com>
> > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Hi Zakelly,
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Thanks for driving this. I agree that
> the
> > > >> > > proposed
> > > >> > > > > > > > restructuring
> > > >> > > > > > > > > > > of
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > configuration options is largely
> positive.
> > > >> It
> > > >> > > will
> > > >> > > > > make
> > > >> > > > > > > > > > > > understanding
> > > >> > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > working with Flink configurations more
> > > >> > intuitive.
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Most of the proposed changes look
> great.
> > > >> Just a
> > > >> > > > > > heads-up,
> > > >> > > > > > > > as Rui
> > > >> > > > > > > > > > > > Fan
> > > >> > > > > > > > > > > > > > > > mentioned, Flink currently requires
> that
> > > no
> > > >> > > > > > > configOption's
> > > >> > > > > > > > key be
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > prefix of another to avoid issues
> when we
> > > >> > > > eventually
> > > >> > > > > > > adopt
> > > >> > > > > > > > a
> > > >> > > > > > > > > > > > standard
> > > >> > > > > > > > > > > > > > YAML
> > > >> > > > > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > >> > > > > > > > > > > > > > > >
> > > >> > > https://issues.apache.org/jira/browse/FLINK-29372
> > > >> > > > ).
> > > >> > > > > > > > Therefore,
> > > >> > > > > > > > > > > > it's
> > > >> > > > > > > > > > > > > > better
> > > >> > > > > > > > > > > > > > > > to change the key
> > > >> > > > > 'execution.checkpointing.local-copy'
> > > >> > > > > > > > because it
> > > >> > > > > > > > > > > > > > serves as
> > > >> > > > > > > > > > > > > > > > a prefix to the key
> > > >> > > > > > > > 'execution.checkpointing.local-copy.dir'.
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Best regards,
> > > >> > > > > > > > > > > > > > > > Junrui
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Rui Fan <19...@gmail.com>
> > > >> 于2023年12月25日周一
> > > >> > > > > 19:11写道:
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Hi Zakelly,
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Thank you for driving this proposal!
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Overall good for me. I have some
> > > questions
> > > >> > > about
> > > >> > > > > > these
> > > >> > > > > > > > names.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > 1. How about
> > > >> > > execution.checkpointing.storage.type
> > > >> > > > > > > > instead of
> > > >> > > > > > > > > > > > > > > > > execution.checkpointing.storage?
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > It's similar to state.backend.type.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > 2. How about
> > > >> > > > > > execution.checkpointing.local-copy.enabled
> > > >> > > > > > > > instead
> > > >> > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > execution.checkpointing.local-copy?
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > You added a new option:
> > > >> > > > > > > > execution.checkpointing.local-copy.dir.
> > > >> > > > > > > > > > > > > > > > > IIUC, one option name shouldn't be
> the
> > > >> prefix
> > > >> > > of
> > > >> > > > > > other
> > > >> > > > > > > > options.
> > > >> > > > > > > > > > > > > > > > > If you add a new option
> > > >> > > > > > > > execution.checkpointing.local-copy,
> > > >> > > > > > > > > > > > > > > > > flink CI will fail directly.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > 3.
> execution.checkpointing.savepoint.dir
> > > >> is a
> > > >> > > > > little
> > > >> > > > > > > > weird.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > For old options:
> state.savepoints.dir
> > > and
> > > >> > > > > > > > > > > state.checkpoints.dir,
> > > >> > > > > > > > > > > > > > > > > the savepoint and checkpoint are the
> > > same
> > > >> > > level.
> > > >> > > > It
> > > >> > > > > > > means
> > > >> > > > > > > > > > > > > > > > > it's a checkpoint or savepoint.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > The new option
> > > >> execution.checkpointing.dir is
> > > >> > > > fine
> > > >> > > > > > for
> > > >> > > > > > > > me.
> > > >> > > > > > > > > > > > > > > > > However,
> > > >> > execution.checkpointing.savepoint.dir
> > > >> > > > is a
> > > >> > > > > > > > little
> > > >> > > > > > > > > > > weird.
> > > >> > > > > > > > > > > > > > > > > I don't know which name is better
> now.
> > > >> Let us
> > > >> > > > think
> > > >> > > > > > > > about it
> > > >> > > > > > > > > > > > more.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > 4. How about
> > > execution.recovery.claim-mode
> > > >> > > > instead
> > > >> > > > > of
> > > >> > > > > > > > > > > > > > > > > execution.recovery.mode?
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > The meaning of mode is too broad.
> The
> > > >> > > claim-mode
> > > >> > > > > may
> > > >> > > > > > > > > > > > > > > > > be more accurate for users.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > WDYT?
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > > > > > Rui
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM
> Zakelly
> > > >> Lan <
> > > >> > > > > > > > > > > > zakelly.lan@gmail.com
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Hi devs,
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > I'd like to start a discussion on
> > > >> FLIP-406:
> > > >> > > > > > > Reorganize
> > > >> > > > > > > > State
> > > >> > > > > > > > > > > &
> > > >> > > > > > > > > > > > > > > > > > Checkpointing & Recovery
> > > >> Configuration[1].
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Currently, the configuration
> options
> > > >> > > pertaining
> > > >> > > > > to
> > > >> > > > > > > > > > > > checkpointing,
> > > >> > > > > > > > > > > > > > > > > recovery,
> > > >> > > > > > > > > > > > > > > > > > and state management are primarily
> > > >> grouped
> > > >> > > > under
> > > >> > > > > > the
> > > >> > > > > > > > > > > following
> > > >> > > > > > > > > > > > > > > > prefixes:
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >    - state.backend.* :
> configurations
> > > >> > related
> > > >> > > > to
> > > >> > > > > > > state
> > > >> > > > > > > > > > > > accessing
> > > >> > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > >    checkpointing, as well as
> specific
> > > >> > options
> > > >> > > > for
> > > >> > > > > > > > individual
> > > >> > > > > > > > > > > > > state
> > > >> > > > > > > > > > > > > > > > > backends
> > > >> > > > > > > > > > > > > > > > > >    - execution.checkpointing.* :
> > > >> > > configurations
> > > >> > > > > > > > associated
> > > >> > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > checkpoint
> > > >> > > > > > > > > > > > > > > > > >    execution and recovery
> > > >> > > > > > > > > > > > > > > > > >    - execution.savepoint.*:
> > > >> configurations
> > > >> > > for
> > > >> > > > > > > > recovery from
> > > >> > > > > > > > > > > > > > savepoint
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > In addition, there are several
> > > >> individual
> > > >> > > > options
> > > >> > > > > > > such
> > > >> > > > > > > > as '
> > > >> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*' and
> > > >> > > > > > > > '*state.checkpoints.dir*'
> > > >> > > > > > > > > > > that
> > > >> > > > > > > > > > > > > fall
> > > >> > > > > > > > > > > > > > > > > outside
> > > >> > > > > > > > > > > > > > > > > > of these prefixes. The current
> > > >> arrangement
> > > >> > of
> > > >> > > > > these
> > > >> > > > > > > > options,
> > > >> > > > > > > > > > > > > which
> > > >> > > > > > > > > > > > > > span
> > > >> > > > > > > > > > > > > > > > > > multiple modules, is somewhat
> > > haphazard
> > > >> and
> > > >> > > > > lacks a
> > > >> > > > > > > > > > > systematic
> > > >> > > > > > > > > > > > > > > > structure.
> > > >> > > > > > > > > > > > > > > > > > For example, the options under the
> > > >> > > > > > > > '*CheckpointingOptions*'
> > > >> > > > > > > > > > > > and '
> > > >> > > > > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*'
> are
> > > >> > related
> > > >> > > > and
> > > >> > > > > > have
> > > >> > > > > > > > no
> > > >> > > > > > > > > > > clear
> > > >> > > > > > > > > > > > > > > > boundaries
> > > >> > > > > > > > > > > > > > > > > > from the user's perspective, but
> there
> > > >> is
> > > >> > no
> > > >> > > > > > unified
> > > >> > > > > > > > prefix
> > > >> > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > them.
> > > >> > > > > > > > > > > > > > > > > With
> > > >> > > > > > > > > > > > > > > > > > the upcoming release of Flink
> 2.0, we
> > > >> have
> > > >> > an
> > > >> > > > > > > excellent
> > > >> > > > > > > > > > > > > > opportunity to
> > > >> > > > > > > > > > > > > > > > > > overhaul and restructure the
> > > >> configurations
> > > >> > > > > related
> > > >> > > > > > > to
> > > >> > > > > > > > > > > > > > checkpointing,
> > > >> > > > > > > > > > > > > > > > > > recovery, and state management.
> This
> > > >> FLIP
> > > >> > > > > proposes
> > > >> > > > > > to
> > > >> > > > > > > > > > > > reorganize
> > > >> > > > > > > > > > > > > > these
> > > >> > > > > > > > > > > > > > > > > > settings, making it more coherent
> by
> > > >> > module,
> > > >> > > > > which
> > > >> > > > > > > > would
> > > >> > > > > > > > > > > > > > significantly
> > > >> > > > > > > > > > > > > > > > > > lower the barriers for
> understanding
> > > and
> > > >> > > reduce
> > > >> > > > > the
> > > >> > > > > > > > > > > development
> > > >> > > > > > > > > > > > > > costs
> > > >> > > > > > > > > > > > > > > > > > moving forward.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Looking forward to hearing from
> you!
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > [1]
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > > > > > > Zakelly
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Best,
> > > >> Hangxiang.
> > > >>
> > > >
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
>
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Xuannan Su <su...@gmail.com>.
Hi Zakelly,

I am fine with either Option 2 or Option 3. I think the naming in
Option 2 makes it clear that it is a boolean configuration. However,
most of the currently available boolean configurations do not use
"enable" as a suffix. Therefore, Option 3 looks good to me as it
follows the current practice.

Best regards,
Xuannan

On Thu, Jan 11, 2024 at 9:50 AM Hangxiang Yu <ma...@gmail.com> wrote:
>
> >
> > That's a very good point. I realize that the word 'recovery' means way too
> > many things. So I suggest picking a more specific word here, how about
> > 'execution.state-recovery.*' ? Checkpointing and state recovery are
> > corresponding terms and won't make ambiguity.
> >
>
> This makes the configuration clearer to me. We could focus on the
> `state-recovery` at first.
>
> I think we could create another FLIP for the deprecation of LEGACY mode.
> >
>
> LGTM, Let's create a new FLIP to do this.
>
> IIUC, there is no clear ownership of the local copy files from the previous
> > job and it's better to define one. This needs more discussion so we could
> > create another thread for this. WDYT?
> >
>
> Yeah, I have created a new ticket FLINK-34032 to track and discuss this.
>
> On Wed, Jan 10, 2024 at 6:31 PM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > It seems we still don't have a consensus on the rules for boolean type
> > options. Let me recap the alternatives we have:
> >
> > Option 1: Use enumeration options instead if possible. But this may cause
> > some name collisions or confusion as we discussed and we should unify the
> > statement everywhere.
> > Option 2: Use boolean options and add 'enabled' as the suffix.
> > Option 3: Use boolean options and ONLY add 'enabled' when there are more
> > detailed configurations under the same prefix, to prevent one name from
> > serving as a prefix to another.
> >
> > I am inclined to Option 3, since it is more in line with current practice
> > and friendly for existing users. Also It reduces the length of
> > configuration names as much as possible.
> >
> > Looking forward to your opinions! Thanks!
> >
> >
> > Best,
> > Zakelly
> >
> > On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan <za...@gmail.com> wrote:
> >
> > > Hi Hangxiang,
> > >
> > > Thanks for your suggestions!
> > >
> > > 1. Could execution.recovery also contain some other behaviors about
> > >> recovery ? e.g. restart-strategy.
> > >
> > >
> > > That's a very good point. I realize that the word 'recovery' means way
> > too
> > > many things. So I suggest picking a more specific word here, how about
> > > 'execution.state-recovery.*' ? Checkpointing and state recovery are
> > > corresponding terms and won't make ambiguity.
> > >
> > > 2. Could we also remove some legacy configuration value ? e.g. LEGACY
> > Mode
> > >> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
> > >
> > >
> > > I think we could create another FLIP for the deprecation of LEGACY mode.
> > >
> > >
> > >> 3. Could the local checkpoint be cleaned
> > >> if execution.checkpointing.local-copy.enabled is true and
> > >> execution.recovery.from-local is false ? I found it's also an issue if
> > >> current local-recovery from enabled to disabled. Maybe another ticket is
> > >> needed.
> > >
> > >
> > > IIUC, there is no clear ownership of the local copy files from the
> > > previous job and it's better to define one. This needs more discussion so
> > > we could create another thread for this. WDYT?
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu <ma...@gmail.com>
> > wrote:
> > >
> > >> Hi, Zakelly.
> > >> Thanks for driving this. Overall LGTM as we discussed offline.
> > >>
> > >> Some comments/suggestions just came to mind:
> > >> 1. Could execution.recovery also contain some other behaviors about
> > >> recovery ? e.g. restart-strategy.
> > >> 2. Could we also remove some legacy configuration value ? e.g. LEGACY
> > Mode
> > >> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
> > >> 3. Could the local checkpoint be cleaned
> > >> if execution.checkpointing.local-copy.enabled is true and
> > >> execution.recovery.from-local is false ? I found it's also an issue if
> > >> current local-recovery from enabled to disabled. Maybe another ticket is
> > >> needed.
> > >> 4. +1 for enabling execution.checkpointing.incremental by default which
> > is
> > >> basically default configuration in our production environment.
> > >>
> > >>
> > >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <za...@gmail.com>
> > wrote:
> > >>
> > >> > Hi Yun,
> > >> >
> > >> > Thanks for your comments!
> > >> >
> > >> >  1.  We shall not describe the configuration with its implementation
> > for
> > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > >> > state-backend,
> > >> > > it would write two streams and for Rocksdb state-backend, it would
> > use
> > >> > > hard-link for backup. Thus, I think
> > >> > > 'execution.checkpointing.local-backup.*' looks better.
> > >> >
> > >> > I agreed that we'd better name the option in user's perspective
> > instead
> > >> of
> > >> > the implementation, thus I name it as a copy of the checkpoint in the
> > >> > local disk, regardless of the way of generating it. The word 'backup'
> > is
> > >> > also suitable for this case, so I agree to change to
> > >> > 'execution.checkpointing.local-backup.*' if no one objects.
> > >> >
> > >> >  2.  What does the 'execution.checkpointing.data-inline-threshold'
> > >> mean? It
> > >> > > seems not so easy to understand.
> > >> >
> > >> > The 'execution.checkpointing.data-inline-threshold' (original one as
> > >> > 'state.storage.fs.memory-threshold') stands for the size threshold
> > below
> > >> > which state chunks will store inline with the metadata, thus I call it
> > >> > 'data-inline-threshold'.
> > >> >
> > >> >
> > >> > Best,
> > >> > Zakelly
> > >> >
> > >> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com> wrote:
> > >> >
> > >> > > Hi Zakelly,
> > >> > >
> > >> > > Thanks for driving this topic. I have two concerns here:
> > >> > >
> > >> > >   1.  We shall not describe the configuration with its
> > implementation
> > >> for
> > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > >> > state-backend,
> > >> > > it would write two streams and for Rocksdb state-backend, it would
> > use
> > >> > > hard-link for backup. Thus, I think
> > >> > > 'execution.checkpointing.local-backup.*' looks better.
> > >> > >   2.  What does the 'execution.checkpointing.data-inline-threshold'
> > >> mean?
> > >> > > It seems not so easy to understand.
> > >> > >
> > >> > > Best
> > >> > > Yun Tang
> > >> > > ________________________________
> > >> > > From: Piotr Nowojski <pn...@apache.org>
> > >> > > Sent: Thursday, January 4, 2024 22:37
> > >> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > >> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing &
> > >> > > Recovery Configuration
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > Thanks for trying to clean this up! I don't have strong opinions on
> > >> the
> > >> > > topics discussed here, so generally speaking +1 from my side!
> > >> > >
> > >> > > Best,
> > >> > > Piotrek
> > >> > >
> > >> > > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com> napisał(a):
> > >> > >
> > >> > > > Thanks for the feedback!
> > >> > > >
> > >> > > > Using the `execution.checkpointing.incremental.enabled`,
> > >> > > > and enabling it by default sounds good to me.
> > >> > > >
> > >> > > > Best,
> > >> > > > Rui
> > >> > > >
> > >> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <
> > zakelly.lan@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > > > Hi Rui,
> > >> > > > >
> > >> > > > > Thanks for your comments!
> > >> > > > >
> > >> > > > > IMO, given that the state backend can be plugably loaded (as you
> > >> can
> > >> > > > > specify a state backend factory), I prefer not providing state
> > >> > backend
> > >> > > > > specified options in the framework.
> > >> > > > >
> > >> > > > > Secondly, the incremental checkpoint is actually a sharing file
> > >> > > strategy
> > >> > > > > across checkpoints, which means the state backend *could* reuse
> > >> files
> > >> > > > from
> > >> > > > > previous cp but not *must* do so. When the state backend could
> > not
> > >> > > reuse
> > >> > > > > the files, it is reasonable to fallback to a full checkpoint.
> > >> > > > >
> > >> > > > > Thus, I suggest we make it `execution.checkpointing.incremental`
> > >> and
> > >> > > > enable
> > >> > > > > it by default. For those state backends not supporting this,
> > they
> > >> > > perform
> > >> > > > > full checkpoints and print a warning to inform users. Users do
> > not
> > >> > need
> > >> > > > to
> > >> > > > > pay special attention to different options to control this
> > across
> > >> > > > different
> > >> > > > > state backends. This is more user-friendly in my opinion. WDYT?
> > >> > > > >
> > >> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com>
> > >> > wrote:
> > >> > > > >
> > >> > > > > > Hi Zakelly,
> > >> > > > > >
> > >> > > > > > I'm not sure whether we could add the state backend type in
> > the
> > >> > > > > > new key name of state.backend.incremental. It means we use
> > >> > > > > > `execution.checkpointing.rocksdb-incremental` or
> > >> > > > > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > >> > > > > >
> > >> > > > > > So far, state.backend.incremental only works for rocksdb state
> > >> > > backend.
> > >> > > > > > And this feature or optimization is very valuable and huge for
> > >> > large
> > >> > > > > > state flink jobs. I believe it's enabled for most production
> > >> flink
> > >> > > jobs
> > >> > > > > > with large rocksdb state.
> > >> > > > > >
> > >> > > > > > If this option isn't generic for all state backend types, I
> > >> guess
> > >> > we
> > >> > > > > > can enable
> > `execution.checkpointing.rocksdb-incremental.enabled`
> > >> > > > > > by default in Flink 2.0.
> > >> > > > > >
> > >> > > > > > But if it works for all state backends, it's hard to enable it
> > >> by
> > >> > > > > default.
> > >> > > > > > Enabling great and valuable features or improvements are
> > useful
> > >> > > > > > for users, especially a lot of new flink users. Out-of-the-box
> > >> > > options
> > >> > > > > > are good for users.
> > >> > > > > >
> > >> > > > > > WDYT?
> > >> > > > > >
> > >> > > > > > Best,
> > >> > > > > > Rui
> > >> > > > > >
> > >> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <
> > >> zakelly.lan@gmail.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi everyone,
> > >> > > > > > >
> > >> > > > > > > Thanks all for your comments!
> > >> > > > > > >
> > >> > > > > > > As many of you have questions about the names for boolean
> > >> > options,
> > >> > > I
> > >> > > > > > > suggest we make a naming rule for them. For now I could
> > think
> > >> of
> > >> > > > three
> > >> > > > > > > options:
> > >> > > > > > >
> > >> > > > > > > Option 1: Use enumeration options if possible. But this may
> > >> cause
> > >> > > > some
> > >> > > > > > name
> > >> > > > > > > collisions or confusion as we discussed and we should unify
> > >> the
> > >> > > > > statement
> > >> > > > > > > everywhere.
> > >> > > > > > > Option 2: Use boolean options and add 'enabled' as the
> > suffix.
> > >> > > > > > > Option 3: Use boolean options and ONLY add 'enabled' when
> > >> there
> > >> > are
> > >> > > > > more
> > >> > > > > > > detailed configurations under the same prefix, to prevent
> > one
> > >> > name
> > >> > > > from
> > >> > > > > > > serving as a prefix to another.
> > >> > > > > > >
> > >> > > > > > > I am slightly inclined to Option 3, since it is more in line
> > >> with
> > >> > > > > current
> > >> > > > > > > practice and friendly for existing users. Also It reduces
> > the
> > >> > > length
> > >> > > > of
> > >> > > > > > > configuration names as much as possible. I really want to
> > hear
> > >> > your
> > >> > > > > > > opinions.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > @Xuannan
> > >> > > > > > >
> > >> > > > > > > I agree with your comments 1 and 3.
> > >> > > > > > >
> > >> > > > > > > For 2, If we decide to change the name, maybe
> > >> > > > > > > `execution.checkpointing.parallel-cleaner` is better? And as
> > >> for
> > >> > > > > whether
> > >> > > > > > to
> > >> > > > > > > add 'enabled' I suggest we discuss the rule above. WDYT?
> > >> > > > > > > Thanks!
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > Best,
> > >> > > > > > > Zakelly
> > >> > > > > > >
> > >> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
> > >> > suxuannan95@gmail.com
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi Zakelly,
> > >> > > > > > > >
> > >> > > > > > > > Thanks for driving this! The organization of the
> > >> configuration
> > >> > > > option
> > >> > > > > > > > in the FLIP looks much cleaner and easier to understand.
> > +1
> > >> to
> > >> > > the
> > >> > > > > > > > FLIP.
> > >> > > > > > > >
> > >> > > > > > > > Just some questions from me.
> > >> > > > > > > >
> > >> > > > > > > > 1. I think the change to the ConfigOptions should be put
> > in
> > >> the
> > >> > > > > > > > `Public Interface` section, instead of `Proposed Changed`,
> > >> as
> > >> > > those
> > >> > > > > > > > configuration options are public interface.
> > >> > > > > > > >
> > >> > > > > > > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems
> > >> > > > confusing.
> > >> > > > > > > > It feels like it is used to choose different modes. In
> > >> fact, it
> > >> > > is
> > >> > > > a
> > >> > > > > > > > boolean flag to indicate whether to enable parallel clean.
> > >> How
> > >> > > > about
> > >> > > > > > > > making it
> > `state.checkpoint.cleaner.parallel-mode.enabled`?
> > >> > > > > > > >
> > >> > > > > > > > 3. The `execution.checkpointing.write-buffer` may better
> > be
> > >> > > > > > > > `execution.checkpointing.write-buffer-size` so that we
> > know
> > >> it
> > >> > is
> > >> > > > > > > > configuring the size of the buffer.
> > >> > > > > > > >
> > >> > > > > > > > Best,
> > >> > > > > > > > Xuannan
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
> > >> > fredialei@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > Hi Zakelly,
> > >> > > > > > > > >
> > >> > > > > > > > > > Considering the name occupation, how about naming it
> > as
> > >> > > > > > > > `execution.checkpointing.type`?
> > >> > > > > > > > >
> > >> > > > > > > > > `Checkpoint Type`[1,2] is used to describe
> > >> aligned/unaligned
> > >> > > > > > > > > checkpoint, I am inclined to make a choice between
> > >> > > > > > > > > `execution.checkpointing.incremental` and
> > >> > > > > > > > > `execution.checkpointing.incremental.enabled`.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > [1]
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > >> > > > > > > > > [2]
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > >> > > > > > > > >
> > >> > > > > > > > > --
> > >> > > > > > > > > Best,
> > >> > > > > > > > > Yanfei
> > >> > > > > > > > >
> > >> > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
> > >> 14:41写道:
> > >> > > > > > > > > >
> > >> > > > > > > > > > Hi Lijie,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for the reminder! I missed this.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Considering the name occupation, how about naming it
> > as
> > >> > > > > > > > > > `execution.checkpointing.type`?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Actually I think the current
> > >> `execution.checkpointing.mode`
> > >> > > is
> > >> > > > > > > > confusing in
> > >> > > > > > > > > > some ways, maybe
> > >> `execution.checkpointing.data-consistency`
> > >> > > is
> > >> > > > > > > better.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Best,
> > >> > > > > > > > > > Zakelly
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > >> > > > > > > wangdachui9501@gmail.com>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Zakelly,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > >> I'm wondering if
> > >> > `execution.checkpointing.savepoint-dir`
> > >> > > > > would
> > >> > > > > > > be
> > >> > > > > > > > > > > better.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > `execution.checkpointing.dir` and
> > >> > > > > > > > `execution.checkpointing.savepoint-dir`
> > >> > > > > > > > > > > are also fine for me.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > >> So I think an enumeration option
> > >> > > > > > `execution.checkpointing.mode`
> > >> > > > > > > > which
> > >> > > > > > > > > > > can be 'full' (default) or 'incremental' would be
> > >> better
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I agree with using an enumeration option. But
> > >> currently
> > >> > > there
> > >> > > > > is
> > >> > > > > > > > already a
> > >> > > > > > > > > > > configuration option called
> > >> > `execution.checkpointing.mode`,
> > >> > > > > which
> > >> > > > > > > is
> > >> > > > > > > > used
> > >> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we
> > >> need to
> > >> > > use
> > >> > > > > > > > another name
> > >> > > > > > > > > > > or merge these two options.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Best,
> > >> > > > > > > > > > > Lijie
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
> > >> > > 11:43写道:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Hi everyone,
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks all for your comments!
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > @Yanfei
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > 1. For some state backends that do not support
> > >> > > > incremental
> > >> > > > > > > > checkpoint,
> > >> > > > > > > > > > > > > how does the
> > >> > execution.checkpointing.incrementaloption
> > >> > > > take
> > >> > > > > > > > effect? Or
> > >> > > > > > > > > > > > > is it better to put incremental under
> > >> > > > > > > > state.backend.xxx.incremental?
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > I'd rather not put the option for incremental
> > >> > checkpoint
> > >> > > > > under
> > >> > > > > > > the
> > >> > > > > > > > > > > > 'state.backend', since it is more about the
> > >> > checkpointing
> > >> > > > > > instead
> > >> > > > > > > > of
> > >> > > > > > > > > > > state
> > >> > > > > > > > > > > > accessing. Of course, the state backend may not
> > >> > > necessarily
> > >> > > > > do
> > >> > > > > > > > > > > incremental
> > >> > > > > > > > > > > > checkpoint as requested. If the state backend is
> > not
> > >> > > > capable
> > >> > > > > of
> > >> > > > > > > > taking
> > >> > > > > > > > > > > > incremental cp, it is better to fallback to the
> > full
> > >> > cp.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > 2. I'm a little worried that putting all
> > >> configurations
> > >> > > > into
> > >> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce
> > >> some
> > >> > > > > > dependency
> > >> > > > > > > > > > > > > problems. Some options would be used by
> > >> flink-runtime
> > >> > > > > module,
> > >> > > > > > > but
> > >> > > > > > > > > > > > > flink-runtime should not depend on
> > >> > > flink-streaming-java.
> > >> > > > > e.g.
> > >> > > > > > > > > > > > > FLINK-28286[1].
> > >> > > > > > > > > > > > > So, I prefer to move configurations to
> > >> > > > > > `CheckpointingOptions`,
> > >> > > > > > > > WDYT?
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Yes, that's a very good point.  Moving to
> > >> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes sense.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > @Lijie
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > How about
> > >> > > > > > > > > > > > > state.savepoints.dir ->
> > >> > > > > execution.checkpointing.savepoint.dir
> > >> > > > > > > > > > > > > state.checkpoints.dir ->
> > >> > > > > > execution.checkpointing.checkpoint.dir
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Actually, I think the `checkpointing.checkpoint`
> > may
> > >> > > cause
> > >> > > > > some
> > >> > > > > > > > > > > confusion.
> > >> > > > > > > > > > > > But I'm ok if others agree.
> > >> > > > > > > > > > > > I'm wondering if
> > >> > `execution.checkpointing.savepoint-dir`
> > >> > > > > would
> > >> > > > > > be
> > >> > > > > > > > better.
> > >> > > > > > > > > > > > WDYT?
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > 2. We changed the
> > >> execution.checkpointing.local-copy'
> > >> > to
> > >> > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > >> Should
> > >> > we
> > >> > > > > also
> > >> > > > > > > add
> > >> > > > > > > > > > > > "enabled"
> > >> > > > > > > > > > > > > suffix for other boolean type configuration
> > >> options ?
> > >> > > For
> > >> > > > > > > > example,
> > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > >> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Actually, the incremental cp is something like
> > >> > choosing a
> > >> > > > > mode
> > >> > > > > > > for
> > >> > > > > > > > doing
> > >> > > > > > > > > > > > checkpoint instead of enabling a function. So I
> > >> think
> > >> > an
> > >> > > > > > > > enumeration
> > >> > > > > > > > > > > option
> > >> > > > > > > > > > > > `execution.checkpointing.mode` which can be 'full'
> > >> > > > (default)
> > >> > > > > or
> > >> > > > > > > > > > > > 'incremental' would be better, WDYT?
> > >> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think about this?
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > >> > > > > > > > wangdachui9501@gmail.com>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hi Zakelly,
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks for driving the discussion.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > 1.
> > >> > > > > > > > > > > > > >> But I'm not so sure since there is only one
> > >> > > > > > > savepoint-related
> > >> > > > > > > > > > > option.
> > >> > > > > > > > > > > > > Maybe someone else could share some thoughts
> > here.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > How about
> > >> > > > > > > > > > > > > state.savepoints.dir ->
> > >> > > > > execution.checkpointing.savepoint.dir
> > >> > > > > > > > > > > > > state.checkpoints.dir ->
> > >> > > > > > execution.checkpointing.checkpoint.dir
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > 2. We changed the
> > >> execution.checkpointing.local-copy'
> > >> > > to
> > >> > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > >> Should
> > >> > we
> > >> > > > > also
> > >> > > > > > > add
> > >> > > > > > > > > > > > "enabled"
> > >> > > > > > > > > > > > > suffix for other boolean type configuration
> > >> options ?
> > >> > > For
> > >> > > > > > > > example,
> > >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > >> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > In this way, the naming style of configuration
> > >> > options
> > >> > > is
> > >> > > > > > > > unified, and
> > >> > > > > > > > > > > it
> > >> > > > > > > > > > > > > can avoid potential similar problems (for
> > >> example, we
> > >> > > may
> > >> > > > > > need
> > >> > > > > > > > to add
> > >> > > > > > > > > > > > more
> > >> > > > > > > > > > > > > options for incremental checkpoint in the
> > future).
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > Lijie
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二
> > >> > > 12:05写道:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Hi Zakelly,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Thank you for creating the FLIP and starting
> > the
> > >> > > > > > discussion.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > The current arrangement of these options is
> > >> indeed
> > >> > > > > somewhat
> > >> > > > > > > > > > > haphazard,
> > >> > > > > > > > > > > > > > and the new arrangement looks much better. I
> > >> have
> > >> > > some
> > >> > > > > > > > questions
> > >> > > > > > > > > > > about
> > >> > > > > > > > > > > > > > the arrangement of some new configuration
> > >> options:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > 1. For some state backends that do not support
> > >> > > > > incremental
> > >> > > > > > > > > > > checkpoint,
> > >> > > > > > > > > > > > > > how does the
> > >> > > execution.checkpointing.incrementaloption
> > >> > > > > take
> > >> > > > > > > > effect?
> > >> > > > > > > > > > > Or
> > >> > > > > > > > > > > > > > is it better to put incremental under
> > >> > > > > > > > state.backend.xxx.incremental?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > 2. I'm a little worried that putting all
> > >> > > configurations
> > >> > > > > > into
> > >> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce
> > >> some
> > >> > > > > > > dependency
> > >> > > > > > > > > > > > > > problems. Some options would be used by
> > >> > flink-runtime
> > >> > > > > > module,
> > >> > > > > > > > but
> > >> > > > > > > > > > > > > > flink-runtime should not depend on
> > >> > > > flink-streaming-java.
> > >> > > > > > e.g.
> > >> > > > > > > > > > > > > > FLINK-28286[1].
> > >> > > > > > > > > > > > > > So, I prefer to move configurations to
> > >> > > > > > > `CheckpointingOptions`,
> > >> > > > > > > > WDYT?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > [1]
> > >> > > https://issues.apache.org/jira/browse/FLINK-28286
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > --
> > >> > > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > > Yanfei
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Zakelly Lan <za...@gmail.com>
> > >> 于2023年12月25日周一
> > >> > > > > > 21:14写道:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Hi Rui Fan and Junrui,
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Thanks for the reminder! I agree to change
> > the
> > >> > > > > > > > > > > > > > > 'execution.checkpointing.local-copy' to
> > >> > > > > > > > > > > > > > >
> > 'execution.checkpointing.local-copy.enabled'.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > And for other suggestions Rui proposed:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > 1. How about
> > >> execution.checkpointing.storage.type
> > >> > > > > instead
> > >> > > > > > > > > > > > > > > > of execution.checkpointing.storage?
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Ah, I missed something here. Actually I
> > >> suggest
> > >> > we
> > >> > > > > could
> > >> > > > > > > > merge the
> > >> > > > > > > > > > > > > > current
> > >> > > > > > > > > > > > > > > 'state.checkpoints.dir' and
> > >> > > > 'state.checkpoint-storage'
> > >> > > > > > into
> > >> > > > > > > > one URI
> > >> > > > > > > > > > > > > > > configuration named
> > >> > 'execution.checkpointing.dir'.
> > >> > > > > WDYT?
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is
> > a
> > >> > > little
> > >> > > > > > weird.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Yes, I think it is better to make
> > 'savepoint'
> > >> and
> > >> > > > > > > > 'checkpoint' the
> > >> > > > > > > > > > > > same
> > >> > > > > > > > > > > > > > > level. But I'm not so sure since there is
> > only
> > >> > one
> > >> > > > > > > > > > > savepoint-related
> > >> > > > > > > > > > > > > > > option. Maybe someone else could share some
> > >> > > thoughts
> > >> > > > > > here.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
> > >> > instead
> > >> > > of
> > >> > > > > > > > > > > > > > > > execution.recovery.mode?
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >  Agreed. That's more accurate.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Many thanks for your suggestions!
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > > > Zakelly
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> > >> > > > > > > > jrlee.ljr@gmail.com>
> > >> > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Hi Zakelly,
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Thanks for driving this. I agree that the
> > >> > > proposed
> > >> > > > > > > > restructuring
> > >> > > > > > > > > > > of
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > configuration options is largely positive.
> > >> It
> > >> > > will
> > >> > > > > make
> > >> > > > > > > > > > > > understanding
> > >> > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > working with Flink configurations more
> > >> > intuitive.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Most of the proposed changes look great.
> > >> Just a
> > >> > > > > > heads-up,
> > >> > > > > > > > as Rui
> > >> > > > > > > > > > > > Fan
> > >> > > > > > > > > > > > > > > > mentioned, Flink currently requires that
> > no
> > >> > > > > > > configOption's
> > >> > > > > > > > key be
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > prefix of another to avoid issues when we
> > >> > > > eventually
> > >> > > > > > > adopt
> > >> > > > > > > > a
> > >> > > > > > > > > > > > standard
> > >> > > > > > > > > > > > > > YAML
> > >> > > > > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > >> > > > > > > > > > > > > > > >
> > >> > > https://issues.apache.org/jira/browse/FLINK-29372
> > >> > > > ).
> > >> > > > > > > > Therefore,
> > >> > > > > > > > > > > > it's
> > >> > > > > > > > > > > > > > better
> > >> > > > > > > > > > > > > > > > to change the key
> > >> > > > > 'execution.checkpointing.local-copy'
> > >> > > > > > > > because it
> > >> > > > > > > > > > > > > > serves as
> > >> > > > > > > > > > > > > > > > a prefix to the key
> > >> > > > > > > > 'execution.checkpointing.local-copy.dir'.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Best regards,
> > >> > > > > > > > > > > > > > > > Junrui
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Rui Fan <19...@gmail.com>
> > >> 于2023年12月25日周一
> > >> > > > > 19:11写道:
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Hi Zakelly,
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Thank you for driving this proposal!
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Overall good for me. I have some
> > questions
> > >> > > about
> > >> > > > > > these
> > >> > > > > > > > names.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > 1. How about
> > >> > > execution.checkpointing.storage.type
> > >> > > > > > > > instead of
> > >> > > > > > > > > > > > > > > > > execution.checkpointing.storage?
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > It's similar to state.backend.type.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > 2. How about
> > >> > > > > > execution.checkpointing.local-copy.enabled
> > >> > > > > > > > instead
> > >> > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > execution.checkpointing.local-copy?
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > You added a new option:
> > >> > > > > > > > execution.checkpointing.local-copy.dir.
> > >> > > > > > > > > > > > > > > > > IIUC, one option name shouldn't be the
> > >> prefix
> > >> > > of
> > >> > > > > > other
> > >> > > > > > > > options.
> > >> > > > > > > > > > > > > > > > > If you add a new option
> > >> > > > > > > > execution.checkpointing.local-copy,
> > >> > > > > > > > > > > > > > > > > flink CI will fail directly.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir
> > >> is a
> > >> > > > > little
> > >> > > > > > > > weird.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > For old options: state.savepoints.dir
> > and
> > >> > > > > > > > > > > state.checkpoints.dir,
> > >> > > > > > > > > > > > > > > > > the savepoint and checkpoint are the
> > same
> > >> > > level.
> > >> > > > It
> > >> > > > > > > means
> > >> > > > > > > > > > > > > > > > > it's a checkpoint or savepoint.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > The new option
> > >> execution.checkpointing.dir is
> > >> > > > fine
> > >> > > > > > for
> > >> > > > > > > > me.
> > >> > > > > > > > > > > > > > > > > However,
> > >> > execution.checkpointing.savepoint.dir
> > >> > > > is a
> > >> > > > > > > > little
> > >> > > > > > > > > > > weird.
> > >> > > > > > > > > > > > > > > > > I don't know which name is better now.
> > >> Let us
> > >> > > > think
> > >> > > > > > > > about it
> > >> > > > > > > > > > > > more.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > 4. How about
> > execution.recovery.claim-mode
> > >> > > > instead
> > >> > > > > of
> > >> > > > > > > > > > > > > > > > > execution.recovery.mode?
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > The meaning of mode is too broad. The
> > >> > > claim-mode
> > >> > > > > may
> > >> > > > > > > > > > > > > > > > > be more accurate for users.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > WDYT?
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > > > > > Rui
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly
> > >> Lan <
> > >> > > > > > > > > > > > zakelly.lan@gmail.com
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Hi devs,
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > I'd like to start a discussion on
> > >> FLIP-406:
> > >> > > > > > > Reorganize
> > >> > > > > > > > State
> > >> > > > > > > > > > > &
> > >> > > > > > > > > > > > > > > > > > Checkpointing & Recovery
> > >> Configuration[1].
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Currently, the configuration options
> > >> > > pertaining
> > >> > > > > to
> > >> > > > > > > > > > > > checkpointing,
> > >> > > > > > > > > > > > > > > > > recovery,
> > >> > > > > > > > > > > > > > > > > > and state management are primarily
> > >> grouped
> > >> > > > under
> > >> > > > > > the
> > >> > > > > > > > > > > following
> > >> > > > > > > > > > > > > > > > prefixes:
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >    - state.backend.* : configurations
> > >> > related
> > >> > > > to
> > >> > > > > > > state
> > >> > > > > > > > > > > > accessing
> > >> > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > >    checkpointing, as well as specific
> > >> > options
> > >> > > > for
> > >> > > > > > > > individual
> > >> > > > > > > > > > > > > state
> > >> > > > > > > > > > > > > > > > > backends
> > >> > > > > > > > > > > > > > > > > >    - execution.checkpointing.* :
> > >> > > configurations
> > >> > > > > > > > associated
> > >> > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > checkpoint
> > >> > > > > > > > > > > > > > > > > >    execution and recovery
> > >> > > > > > > > > > > > > > > > > >    - execution.savepoint.*:
> > >> configurations
> > >> > > for
> > >> > > > > > > > recovery from
> > >> > > > > > > > > > > > > > savepoint
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > In addition, there are several
> > >> individual
> > >> > > > options
> > >> > > > > > > such
> > >> > > > > > > > as '
> > >> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*' and
> > >> > > > > > > > '*state.checkpoints.dir*'
> > >> > > > > > > > > > > that
> > >> > > > > > > > > > > > > fall
> > >> > > > > > > > > > > > > > > > > outside
> > >> > > > > > > > > > > > > > > > > > of these prefixes. The current
> > >> arrangement
> > >> > of
> > >> > > > > these
> > >> > > > > > > > options,
> > >> > > > > > > > > > > > > which
> > >> > > > > > > > > > > > > > span
> > >> > > > > > > > > > > > > > > > > > multiple modules, is somewhat
> > haphazard
> > >> and
> > >> > > > > lacks a
> > >> > > > > > > > > > > systematic
> > >> > > > > > > > > > > > > > > > structure.
> > >> > > > > > > > > > > > > > > > > > For example, the options under the
> > >> > > > > > > > '*CheckpointingOptions*'
> > >> > > > > > > > > > > > and '
> > >> > > > > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are
> > >> > related
> > >> > > > and
> > >> > > > > > have
> > >> > > > > > > > no
> > >> > > > > > > > > > > clear
> > >> > > > > > > > > > > > > > > > boundaries
> > >> > > > > > > > > > > > > > > > > > from the user's perspective, but there
> > >> is
> > >> > no
> > >> > > > > > unified
> > >> > > > > > > > prefix
> > >> > > > > > > > > > > for
> > >> > > > > > > > > > > > > > them.
> > >> > > > > > > > > > > > > > > > > With
> > >> > > > > > > > > > > > > > > > > > the upcoming release of Flink 2.0, we
> > >> have
> > >> > an
> > >> > > > > > > excellent
> > >> > > > > > > > > > > > > > opportunity to
> > >> > > > > > > > > > > > > > > > > > overhaul and restructure the
> > >> configurations
> > >> > > > > related
> > >> > > > > > > to
> > >> > > > > > > > > > > > > > checkpointing,
> > >> > > > > > > > > > > > > > > > > > recovery, and state management. This
> > >> FLIP
> > >> > > > > proposes
> > >> > > > > > to
> > >> > > > > > > > > > > > reorganize
> > >> > > > > > > > > > > > > > these
> > >> > > > > > > > > > > > > > > > > > settings, making it more coherent by
> > >> > module,
> > >> > > > > which
> > >> > > > > > > > would
> > >> > > > > > > > > > > > > > significantly
> > >> > > > > > > > > > > > > > > > > > lower the barriers for understanding
> > and
> > >> > > reduce
> > >> > > > > the
> > >> > > > > > > > > > > development
> > >> > > > > > > > > > > > > > costs
> > >> > > > > > > > > > > > > > > > > > moving forward.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Looking forward to hearing from you!
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > [1]
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > > > > > > Zakelly
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> Best,
> > >> Hangxiang.
> > >>
> > >
> >
>
>
> --
> Best,
> Hangxiang.


Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Hangxiang Yu <ma...@gmail.com>.
>
> That's a very good point. I realize that the word 'recovery' means way too
> many things. So I suggest picking a more specific word here, how about
> 'execution.state-recovery.*' ? Checkpointing and state recovery are
> corresponding terms and won't make ambiguity.
>

This makes the configuration clearer to me. We could focus on the
`state-recovery` at first.

I think we could create another FLIP for the deprecation of LEGACY mode.
>

LGTM, Let's create a new FLIP to do this.

IIUC, there is no clear ownership of the local copy files from the previous
> job and it's better to define one. This needs more discussion so we could
> create another thread for this. WDYT?
>

Yeah, I have created a new ticket FLINK-34032 to track and discuss this.

On Wed, Jan 10, 2024 at 6:31 PM Zakelly Lan <za...@gmail.com> wrote:

> Hi everyone,
>
> It seems we still don't have a consensus on the rules for boolean type
> options. Let me recap the alternatives we have:
>
> Option 1: Use enumeration options instead if possible. But this may cause
> some name collisions or confusion as we discussed and we should unify the
> statement everywhere.
> Option 2: Use boolean options and add 'enabled' as the suffix.
> Option 3: Use boolean options and ONLY add 'enabled' when there are more
> detailed configurations under the same prefix, to prevent one name from
> serving as a prefix to another.
>
> I am inclined to Option 3, since it is more in line with current practice
> and friendly for existing users. Also It reduces the length of
> configuration names as much as possible.
>
> Looking forward to your opinions! Thanks!
>
>
> Best,
> Zakelly
>
> On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi Hangxiang,
> >
> > Thanks for your suggestions!
> >
> > 1. Could execution.recovery also contain some other behaviors about
> >> recovery ? e.g. restart-strategy.
> >
> >
> > That's a very good point. I realize that the word 'recovery' means way
> too
> > many things. So I suggest picking a more specific word here, how about
> > 'execution.state-recovery.*' ? Checkpointing and state recovery are
> > corresponding terms and won't make ambiguity.
> >
> > 2. Could we also remove some legacy configuration value ? e.g. LEGACY
> Mode
> >> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
> >
> >
> > I think we could create another FLIP for the deprecation of LEGACY mode.
> >
> >
> >> 3. Could the local checkpoint be cleaned
> >> if execution.checkpointing.local-copy.enabled is true and
> >> execution.recovery.from-local is false ? I found it's also an issue if
> >> current local-recovery from enabled to disabled. Maybe another ticket is
> >> needed.
> >
> >
> > IIUC, there is no clear ownership of the local copy files from the
> > previous job and it's better to define one. This needs more discussion so
> > we could create another thread for this. WDYT?
> >
> >
> > Best,
> > Zakelly
> >
> > On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu <ma...@gmail.com>
> wrote:
> >
> >> Hi, Zakelly.
> >> Thanks for driving this. Overall LGTM as we discussed offline.
> >>
> >> Some comments/suggestions just came to mind:
> >> 1. Could execution.recovery also contain some other behaviors about
> >> recovery ? e.g. restart-strategy.
> >> 2. Could we also remove some legacy configuration value ? e.g. LEGACY
> Mode
> >> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
> >> 3. Could the local checkpoint be cleaned
> >> if execution.checkpointing.local-copy.enabled is true and
> >> execution.recovery.from-local is false ? I found it's also an issue if
> >> current local-recovery from enabled to disabled. Maybe another ticket is
> >> needed.
> >> 4. +1 for enabling execution.checkpointing.incremental by default which
> is
> >> basically default configuration in our production environment.
> >>
> >>
> >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <za...@gmail.com>
> wrote:
> >>
> >> > Hi Yun,
> >> >
> >> > Thanks for your comments!
> >> >
> >> >  1.  We shall not describe the configuration with its implementation
> for
> >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> >> > state-backend,
> >> > > it would write two streams and for Rocksdb state-backend, it would
> use
> >> > > hard-link for backup. Thus, I think
> >> > > 'execution.checkpointing.local-backup.*' looks better.
> >> >
> >> > I agreed that we'd better name the option in user's perspective
> instead
> >> of
> >> > the implementation, thus I name it as a copy of the checkpoint in the
> >> > local disk, regardless of the way of generating it. The word 'backup'
> is
> >> > also suitable for this case, so I agree to change to
> >> > 'execution.checkpointing.local-backup.*' if no one objects.
> >> >
> >> >  2.  What does the 'execution.checkpointing.data-inline-threshold'
> >> mean? It
> >> > > seems not so easy to understand.
> >> >
> >> > The 'execution.checkpointing.data-inline-threshold' (original one as
> >> > 'state.storage.fs.memory-threshold') stands for the size threshold
> below
> >> > which state chunks will store inline with the metadata, thus I call it
> >> > 'data-inline-threshold'.
> >> >
> >> >
> >> > Best,
> >> > Zakelly
> >> >
> >> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com> wrote:
> >> >
> >> > > Hi Zakelly,
> >> > >
> >> > > Thanks for driving this topic. I have two concerns here:
> >> > >
> >> > >   1.  We shall not describe the configuration with its
> implementation
> >> for
> >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> >> > state-backend,
> >> > > it would write two streams and for Rocksdb state-backend, it would
> use
> >> > > hard-link for backup. Thus, I think
> >> > > 'execution.checkpointing.local-backup.*' looks better.
> >> > >   2.  What does the 'execution.checkpointing.data-inline-threshold'
> >> mean?
> >> > > It seems not so easy to understand.
> >> > >
> >> > > Best
> >> > > Yun Tang
> >> > > ________________________________
> >> > > From: Piotr Nowojski <pn...@apache.org>
> >> > > Sent: Thursday, January 4, 2024 22:37
> >> > > To: dev@flink.apache.org <de...@flink.apache.org>
> >> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing &
> >> > > Recovery Configuration
> >> > >
> >> > > Hi,
> >> > >
> >> > > Thanks for trying to clean this up! I don't have strong opinions on
> >> the
> >> > > topics discussed here, so generally speaking +1 from my side!
> >> > >
> >> > > Best,
> >> > > Piotrek
> >> > >
> >> > > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com> napisał(a):
> >> > >
> >> > > > Thanks for the feedback!
> >> > > >
> >> > > > Using the `execution.checkpointing.incremental.enabled`,
> >> > > > and enabling it by default sounds good to me.
> >> > > >
> >> > > > Best,
> >> > > > Rui
> >> > > >
> >> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <
> zakelly.lan@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Hi Rui,
> >> > > > >
> >> > > > > Thanks for your comments!
> >> > > > >
> >> > > > > IMO, given that the state backend can be plugably loaded (as you
> >> can
> >> > > > > specify a state backend factory), I prefer not providing state
> >> > backend
> >> > > > > specified options in the framework.
> >> > > > >
> >> > > > > Secondly, the incremental checkpoint is actually a sharing file
> >> > > strategy
> >> > > > > across checkpoints, which means the state backend *could* reuse
> >> files
> >> > > > from
> >> > > > > previous cp but not *must* do so. When the state backend could
> not
> >> > > reuse
> >> > > > > the files, it is reasonable to fallback to a full checkpoint.
> >> > > > >
> >> > > > > Thus, I suggest we make it `execution.checkpointing.incremental`
> >> and
> >> > > > enable
> >> > > > > it by default. For those state backends not supporting this,
> they
> >> > > perform
> >> > > > > full checkpoints and print a warning to inform users. Users do
> not
> >> > need
> >> > > > to
> >> > > > > pay special attention to different options to control this
> across
> >> > > > different
> >> > > > > state backends. This is more user-friendly in my opinion. WDYT?
> >> > > > >
> >> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com>
> >> > wrote:
> >> > > > >
> >> > > > > > Hi Zakelly,
> >> > > > > >
> >> > > > > > I'm not sure whether we could add the state backend type in
> the
> >> > > > > > new key name of state.backend.incremental. It means we use
> >> > > > > > `execution.checkpointing.rocksdb-incremental` or
> >> > > > > > `execution.checkpointing.rocksdb-incremental.enabled`.
> >> > > > > >
> >> > > > > > So far, state.backend.incremental only works for rocksdb state
> >> > > backend.
> >> > > > > > And this feature or optimization is very valuable and huge for
> >> > large
> >> > > > > > state flink jobs. I believe it's enabled for most production
> >> flink
> >> > > jobs
> >> > > > > > with large rocksdb state.
> >> > > > > >
> >> > > > > > If this option isn't generic for all state backend types, I
> >> guess
> >> > we
> >> > > > > > can enable
> `execution.checkpointing.rocksdb-incremental.enabled`
> >> > > > > > by default in Flink 2.0.
> >> > > > > >
> >> > > > > > But if it works for all state backends, it's hard to enable it
> >> by
> >> > > > > default.
> >> > > > > > Enabling great and valuable features or improvements are
> useful
> >> > > > > > for users, especially a lot of new flink users. Out-of-the-box
> >> > > options
> >> > > > > > are good for users.
> >> > > > > >
> >> > > > > > WDYT?
> >> > > > > >
> >> > > > > > Best,
> >> > > > > > Rui
> >> > > > > >
> >> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <
> >> zakelly.lan@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi everyone,
> >> > > > > > >
> >> > > > > > > Thanks all for your comments!
> >> > > > > > >
> >> > > > > > > As many of you have questions about the names for boolean
> >> > options,
> >> > > I
> >> > > > > > > suggest we make a naming rule for them. For now I could
> think
> >> of
> >> > > > three
> >> > > > > > > options:
> >> > > > > > >
> >> > > > > > > Option 1: Use enumeration options if possible. But this may
> >> cause
> >> > > > some
> >> > > > > > name
> >> > > > > > > collisions or confusion as we discussed and we should unify
> >> the
> >> > > > > statement
> >> > > > > > > everywhere.
> >> > > > > > > Option 2: Use boolean options and add 'enabled' as the
> suffix.
> >> > > > > > > Option 3: Use boolean options and ONLY add 'enabled' when
> >> there
> >> > are
> >> > > > > more
> >> > > > > > > detailed configurations under the same prefix, to prevent
> one
> >> > name
> >> > > > from
> >> > > > > > > serving as a prefix to another.
> >> > > > > > >
> >> > > > > > > I am slightly inclined to Option 3, since it is more in line
> >> with
> >> > > > > current
> >> > > > > > > practice and friendly for existing users. Also It reduces
> the
> >> > > length
> >> > > > of
> >> > > > > > > configuration names as much as possible. I really want to
> hear
> >> > your
> >> > > > > > > opinions.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > @Xuannan
> >> > > > > > >
> >> > > > > > > I agree with your comments 1 and 3.
> >> > > > > > >
> >> > > > > > > For 2, If we decide to change the name, maybe
> >> > > > > > > `execution.checkpointing.parallel-cleaner` is better? And as
> >> for
> >> > > > > whether
> >> > > > > > to
> >> > > > > > > add 'enabled' I suggest we discuss the rule above. WDYT?
> >> > > > > > > Thanks!
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Best,
> >> > > > > > > Zakelly
> >> > > > > > >
> >> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
> >> > suxuannan95@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi Zakelly,
> >> > > > > > > >
> >> > > > > > > > Thanks for driving this! The organization of the
> >> configuration
> >> > > > option
> >> > > > > > > > in the FLIP looks much cleaner and easier to understand.
> +1
> >> to
> >> > > the
> >> > > > > > > > FLIP.
> >> > > > > > > >
> >> > > > > > > > Just some questions from me.
> >> > > > > > > >
> >> > > > > > > > 1. I think the change to the ConfigOptions should be put
> in
> >> the
> >> > > > > > > > `Public Interface` section, instead of `Proposed Changed`,
> >> as
> >> > > those
> >> > > > > > > > configuration options are public interface.
> >> > > > > > > >
> >> > > > > > > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems
> >> > > > confusing.
> >> > > > > > > > It feels like it is used to choose different modes. In
> >> fact, it
> >> > > is
> >> > > > a
> >> > > > > > > > boolean flag to indicate whether to enable parallel clean.
> >> How
> >> > > > about
> >> > > > > > > > making it
> `state.checkpoint.cleaner.parallel-mode.enabled`?
> >> > > > > > > >
> >> > > > > > > > 3. The `execution.checkpointing.write-buffer` may better
> be
> >> > > > > > > > `execution.checkpointing.write-buffer-size` so that we
> know
> >> it
> >> > is
> >> > > > > > > > configuring the size of the buffer.
> >> > > > > > > >
> >> > > > > > > > Best,
> >> > > > > > > > Xuannan
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
> >> > fredialei@gmail.com>
> >> > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > Hi Zakelly,
> >> > > > > > > > >
> >> > > > > > > > > > Considering the name occupation, how about naming it
> as
> >> > > > > > > > `execution.checkpointing.type`?
> >> > > > > > > > >
> >> > > > > > > > > `Checkpoint Type`[1,2] is used to describe
> >> aligned/unaligned
> >> > > > > > > > > checkpoint, I am inclined to make a choice between
> >> > > > > > > > > `execution.checkpointing.incremental` and
> >> > > > > > > > > `execution.checkpointing.incremental.enabled`.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > [1]
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> >> > > > > > > > > [2]
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > > Best,
> >> > > > > > > > > Yanfei
> >> > > > > > > > >
> >> > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
> >> 14:41写道:
> >> > > > > > > > > >
> >> > > > > > > > > > Hi Lijie,
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for the reminder! I missed this.
> >> > > > > > > > > >
> >> > > > > > > > > > Considering the name occupation, how about naming it
> as
> >> > > > > > > > > > `execution.checkpointing.type`?
> >> > > > > > > > > >
> >> > > > > > > > > > Actually I think the current
> >> `execution.checkpointing.mode`
> >> > > is
> >> > > > > > > > confusing in
> >> > > > > > > > > > some ways, maybe
> >> `execution.checkpointing.data-consistency`
> >> > > is
> >> > > > > > > better.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Best,
> >> > > > > > > > > > Zakelly
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> >> > > > > > > wangdachui9501@gmail.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Zakelly,
> >> > > > > > > > > > >
> >> > > > > > > > > > > >> I'm wondering if
> >> > `execution.checkpointing.savepoint-dir`
> >> > > > > would
> >> > > > > > > be
> >> > > > > > > > > > > better.
> >> > > > > > > > > > >
> >> > > > > > > > > > > `execution.checkpointing.dir` and
> >> > > > > > > > `execution.checkpointing.savepoint-dir`
> >> > > > > > > > > > > are also fine for me.
> >> > > > > > > > > > >
> >> > > > > > > > > > > >> So I think an enumeration option
> >> > > > > > `execution.checkpointing.mode`
> >> > > > > > > > which
> >> > > > > > > > > > > can be 'full' (default) or 'incremental' would be
> >> better
> >> > > > > > > > > > >
> >> > > > > > > > > > > I agree with using an enumeration option. But
> >> currently
> >> > > there
> >> > > > > is
> >> > > > > > > > already a
> >> > > > > > > > > > > configuration option called
> >> > `execution.checkpointing.mode`,
> >> > > > > which
> >> > > > > > > is
> >> > > > > > > > used
> >> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we
> >> need to
> >> > > use
> >> > > > > > > > another name
> >> > > > > > > > > > > or merge these two options.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Best,
> >> > > > > > > > > > > Lijie
> >> > > > > > > > > > >
> >> > > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
> >> > > 11:43写道:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Hi everyone,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks all for your comments!
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > @Yanfei
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > 1. For some state backends that do not support
> >> > > > incremental
> >> > > > > > > > checkpoint,
> >> > > > > > > > > > > > > how does the
> >> > execution.checkpointing.incrementaloption
> >> > > > take
> >> > > > > > > > effect? Or
> >> > > > > > > > > > > > > is it better to put incremental under
> >> > > > > > > > state.backend.xxx.incremental?
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > I'd rather not put the option for incremental
> >> > checkpoint
> >> > > > > under
> >> > > > > > > the
> >> > > > > > > > > > > > 'state.backend', since it is more about the
> >> > checkpointing
> >> > > > > > instead
> >> > > > > > > > of
> >> > > > > > > > > > > state
> >> > > > > > > > > > > > accessing. Of course, the state backend may not
> >> > > necessarily
> >> > > > > do
> >> > > > > > > > > > > incremental
> >> > > > > > > > > > > > checkpoint as requested. If the state backend is
> not
> >> > > > capable
> >> > > > > of
> >> > > > > > > > taking
> >> > > > > > > > > > > > incremental cp, it is better to fallback to the
> full
> >> > cp.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > 2. I'm a little worried that putting all
> >> configurations
> >> > > > into
> >> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce
> >> some
> >> > > > > > dependency
> >> > > > > > > > > > > > > problems. Some options would be used by
> >> flink-runtime
> >> > > > > module,
> >> > > > > > > but
> >> > > > > > > > > > > > > flink-runtime should not depend on
> >> > > flink-streaming-java.
> >> > > > > e.g.
> >> > > > > > > > > > > > > FLINK-28286[1].
> >> > > > > > > > > > > > > So, I prefer to move configurations to
> >> > > > > > `CheckpointingOptions`,
> >> > > > > > > > WDYT?
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Yes, that's a very good point.  Moving to
> >> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes sense.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > @Lijie
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > How about
> >> > > > > > > > > > > > > state.savepoints.dir ->
> >> > > > > execution.checkpointing.savepoint.dir
> >> > > > > > > > > > > > > state.checkpoints.dir ->
> >> > > > > > execution.checkpointing.checkpoint.dir
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Actually, I think the `checkpointing.checkpoint`
> may
> >> > > cause
> >> > > > > some
> >> > > > > > > > > > > confusion.
> >> > > > > > > > > > > > But I'm ok if others agree.
> >> > > > > > > > > > > > I'm wondering if
> >> > `execution.checkpointing.savepoint-dir`
> >> > > > > would
> >> > > > > > be
> >> > > > > > > > better.
> >> > > > > > > > > > > > WDYT?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > 2. We changed the
> >> execution.checkpointing.local-copy'
> >> > to
> >> > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> >> Should
> >> > we
> >> > > > > also
> >> > > > > > > add
> >> > > > > > > > > > > > "enabled"
> >> > > > > > > > > > > > > suffix for other boolean type configuration
> >> options ?
> >> > > For
> >> > > > > > > > example,
> >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> >> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Actually, the incremental cp is something like
> >> > choosing a
> >> > > > > mode
> >> > > > > > > for
> >> > > > > > > > doing
> >> > > > > > > > > > > > checkpoint instead of enabling a function. So I
> >> think
> >> > an
> >> > > > > > > > enumeration
> >> > > > > > > > > > > option
> >> > > > > > > > > > > > `execution.checkpointing.mode` which can be 'full'
> >> > > > (default)
> >> > > > > or
> >> > > > > > > > > > > > 'incremental' would be better, WDYT?
> >> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think about this?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> >> > > > > > > > wangdachui9501@gmail.com>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hi Zakelly,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks for driving the discussion.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > 1.
> >> > > > > > > > > > > > > >> But I'm not so sure since there is only one
> >> > > > > > > savepoint-related
> >> > > > > > > > > > > option.
> >> > > > > > > > > > > > > Maybe someone else could share some thoughts
> here.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > How about
> >> > > > > > > > > > > > > state.savepoints.dir ->
> >> > > > > execution.checkpointing.savepoint.dir
> >> > > > > > > > > > > > > state.checkpoints.dir ->
> >> > > > > > execution.checkpointing.checkpoint.dir
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > 2. We changed the
> >> execution.checkpointing.local-copy'
> >> > > to
> >> > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> >> Should
> >> > we
> >> > > > > also
> >> > > > > > > add
> >> > > > > > > > > > > > "enabled"
> >> > > > > > > > > > > > > suffix for other boolean type configuration
> >> options ?
> >> > > For
> >> > > > > > > > example,
> >> > > > > > > > > > > > > execution.checkpointing.incremental ->
> >> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > In this way, the naming style of configuration
> >> > options
> >> > > is
> >> > > > > > > > unified, and
> >> > > > > > > > > > > it
> >> > > > > > > > > > > > > can avoid potential similar problems (for
> >> example, we
> >> > > may
> >> > > > > > need
> >> > > > > > > > to add
> >> > > > > > > > > > > > more
> >> > > > > > > > > > > > > options for incremental checkpoint in the
> future).
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Best,
> >> > > > > > > > > > > > > Lijie
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二
> >> > > 12:05写道:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Hi Zakelly,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Thank you for creating the FLIP and starting
> the
> >> > > > > > discussion.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > The current arrangement of these options is
> >> indeed
> >> > > > > somewhat
> >> > > > > > > > > > > haphazard,
> >> > > > > > > > > > > > > > and the new arrangement looks much better. I
> >> have
> >> > > some
> >> > > > > > > > questions
> >> > > > > > > > > > > about
> >> > > > > > > > > > > > > > the arrangement of some new configuration
> >> options:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > 1. For some state backends that do not support
> >> > > > > incremental
> >> > > > > > > > > > > checkpoint,
> >> > > > > > > > > > > > > > how does the
> >> > > execution.checkpointing.incrementaloption
> >> > > > > take
> >> > > > > > > > effect?
> >> > > > > > > > > > > Or
> >> > > > > > > > > > > > > > is it better to put incremental under
> >> > > > > > > > state.backend.xxx.incremental?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > 2. I'm a little worried that putting all
> >> > > configurations
> >> > > > > > into
> >> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce
> >> some
> >> > > > > > > dependency
> >> > > > > > > > > > > > > > problems. Some options would be used by
> >> > flink-runtime
> >> > > > > > module,
> >> > > > > > > > but
> >> > > > > > > > > > > > > > flink-runtime should not depend on
> >> > > > flink-streaming-java.
> >> > > > > > e.g.
> >> > > > > > > > > > > > > > FLINK-28286[1].
> >> > > > > > > > > > > > > > So, I prefer to move configurations to
> >> > > > > > > `CheckpointingOptions`,
> >> > > > > > > > WDYT?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > [1]
> >> > > https://issues.apache.org/jira/browse/FLINK-28286
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > --
> >> > > > > > > > > > > > > > Best,
> >> > > > > > > > > > > > > > Yanfei
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Zakelly Lan <za...@gmail.com>
> >> 于2023年12月25日周一
> >> > > > > > 21:14写道:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Hi Rui Fan and Junrui,
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Thanks for the reminder! I agree to change
> the
> >> > > > > > > > > > > > > > > 'execution.checkpointing.local-copy' to
> >> > > > > > > > > > > > > > >
> 'execution.checkpointing.local-copy.enabled'.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > And for other suggestions Rui proposed:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > 1. How about
> >> execution.checkpointing.storage.type
> >> > > > > instead
> >> > > > > > > > > > > > > > > > of execution.checkpointing.storage?
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Ah, I missed something here. Actually I
> >> suggest
> >> > we
> >> > > > > could
> >> > > > > > > > merge the
> >> > > > > > > > > > > > > > current
> >> > > > > > > > > > > > > > > 'state.checkpoints.dir' and
> >> > > > 'state.checkpoint-storage'
> >> > > > > > into
> >> > > > > > > > one URI
> >> > > > > > > > > > > > > > > configuration named
> >> > 'execution.checkpointing.dir'.
> >> > > > > WDYT?
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is
> a
> >> > > little
> >> > > > > > weird.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Yes, I think it is better to make
> 'savepoint'
> >> and
> >> > > > > > > > 'checkpoint' the
> >> > > > > > > > > > > > same
> >> > > > > > > > > > > > > > > level. But I'm not so sure since there is
> only
> >> > one
> >> > > > > > > > > > > savepoint-related
> >> > > > > > > > > > > > > > > option. Maybe someone else could share some
> >> > > thoughts
> >> > > > > > here.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
> >> > instead
> >> > > of
> >> > > > > > > > > > > > > > > > execution.recovery.mode?
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >  Agreed. That's more accurate.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Many thanks for your suggestions!
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Best,
> >> > > > > > > > > > > > > > > Zakelly
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> >> > > > > > > > jrlee.ljr@gmail.com>
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Hi Zakelly,
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Thanks for driving this. I agree that the
> >> > > proposed
> >> > > > > > > > restructuring
> >> > > > > > > > > > > of
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > configuration options is largely positive.
> >> It
> >> > > will
> >> > > > > make
> >> > > > > > > > > > > > understanding
> >> > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > working with Flink configurations more
> >> > intuitive.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Most of the proposed changes look great.
> >> Just a
> >> > > > > > heads-up,
> >> > > > > > > > as Rui
> >> > > > > > > > > > > > Fan
> >> > > > > > > > > > > > > > > > mentioned, Flink currently requires that
> no
> >> > > > > > > configOption's
> >> > > > > > > > key be
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > prefix of another to avoid issues when we
> >> > > > eventually
> >> > > > > > > adopt
> >> > > > > > > > a
> >> > > > > > > > > > > > standard
> >> > > > > > > > > > > > > > YAML
> >> > > > > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> >> > > > > > > > > > > > > > > >
> >> > > https://issues.apache.org/jira/browse/FLINK-29372
> >> > > > ).
> >> > > > > > > > Therefore,
> >> > > > > > > > > > > > it's
> >> > > > > > > > > > > > > > better
> >> > > > > > > > > > > > > > > > to change the key
> >> > > > > 'execution.checkpointing.local-copy'
> >> > > > > > > > because it
> >> > > > > > > > > > > > > > serves as
> >> > > > > > > > > > > > > > > > a prefix to the key
> >> > > > > > > > 'execution.checkpointing.local-copy.dir'.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Best regards,
> >> > > > > > > > > > > > > > > > Junrui
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Rui Fan <19...@gmail.com>
> >> 于2023年12月25日周一
> >> > > > > 19:11写道:
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Hi Zakelly,
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Thank you for driving this proposal!
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Overall good for me. I have some
> questions
> >> > > about
> >> > > > > > these
> >> > > > > > > > names.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > 1. How about
> >> > > execution.checkpointing.storage.type
> >> > > > > > > > instead of
> >> > > > > > > > > > > > > > > > > execution.checkpointing.storage?
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > It's similar to state.backend.type.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > 2. How about
> >> > > > > > execution.checkpointing.local-copy.enabled
> >> > > > > > > > instead
> >> > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > execution.checkpointing.local-copy?
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > You added a new option:
> >> > > > > > > > execution.checkpointing.local-copy.dir.
> >> > > > > > > > > > > > > > > > > IIUC, one option name shouldn't be the
> >> prefix
> >> > > of
> >> > > > > > other
> >> > > > > > > > options.
> >> > > > > > > > > > > > > > > > > If you add a new option
> >> > > > > > > > execution.checkpointing.local-copy,
> >> > > > > > > > > > > > > > > > > flink CI will fail directly.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir
> >> is a
> >> > > > > little
> >> > > > > > > > weird.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > For old options: state.savepoints.dir
> and
> >> > > > > > > > > > > state.checkpoints.dir,
> >> > > > > > > > > > > > > > > > > the savepoint and checkpoint are the
> same
> >> > > level.
> >> > > > It
> >> > > > > > > means
> >> > > > > > > > > > > > > > > > > it's a checkpoint or savepoint.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > The new option
> >> execution.checkpointing.dir is
> >> > > > fine
> >> > > > > > for
> >> > > > > > > > me.
> >> > > > > > > > > > > > > > > > > However,
> >> > execution.checkpointing.savepoint.dir
> >> > > > is a
> >> > > > > > > > little
> >> > > > > > > > > > > weird.
> >> > > > > > > > > > > > > > > > > I don't know which name is better now.
> >> Let us
> >> > > > think
> >> > > > > > > > about it
> >> > > > > > > > > > > > more.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > 4. How about
> execution.recovery.claim-mode
> >> > > > instead
> >> > > > > of
> >> > > > > > > > > > > > > > > > > execution.recovery.mode?
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > The meaning of mode is too broad. The
> >> > > claim-mode
> >> > > > > may
> >> > > > > > > > > > > > > > > > > be more accurate for users.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > WDYT?
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Best,
> >> > > > > > > > > > > > > > > > > Rui
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly
> >> Lan <
> >> > > > > > > > > > > > zakelly.lan@gmail.com
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Hi devs,
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > I'd like to start a discussion on
> >> FLIP-406:
> >> > > > > > > Reorganize
> >> > > > > > > > State
> >> > > > > > > > > > > &
> >> > > > > > > > > > > > > > > > > > Checkpointing & Recovery
> >> Configuration[1].
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Currently, the configuration options
> >> > > pertaining
> >> > > > > to
> >> > > > > > > > > > > > checkpointing,
> >> > > > > > > > > > > > > > > > > recovery,
> >> > > > > > > > > > > > > > > > > > and state management are primarily
> >> grouped
> >> > > > under
> >> > > > > > the
> >> > > > > > > > > > > following
> >> > > > > > > > > > > > > > > > prefixes:
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >    - state.backend.* : configurations
> >> > related
> >> > > > to
> >> > > > > > > state
> >> > > > > > > > > > > > accessing
> >> > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > >    checkpointing, as well as specific
> >> > options
> >> > > > for
> >> > > > > > > > individual
> >> > > > > > > > > > > > > state
> >> > > > > > > > > > > > > > > > > backends
> >> > > > > > > > > > > > > > > > > >    - execution.checkpointing.* :
> >> > > configurations
> >> > > > > > > > associated
> >> > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > checkpoint
> >> > > > > > > > > > > > > > > > > >    execution and recovery
> >> > > > > > > > > > > > > > > > > >    - execution.savepoint.*:
> >> configurations
> >> > > for
> >> > > > > > > > recovery from
> >> > > > > > > > > > > > > > savepoint
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > In addition, there are several
> >> individual
> >> > > > options
> >> > > > > > > such
> >> > > > > > > > as '
> >> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*' and
> >> > > > > > > > '*state.checkpoints.dir*'
> >> > > > > > > > > > > that
> >> > > > > > > > > > > > > fall
> >> > > > > > > > > > > > > > > > > outside
> >> > > > > > > > > > > > > > > > > > of these prefixes. The current
> >> arrangement
> >> > of
> >> > > > > these
> >> > > > > > > > options,
> >> > > > > > > > > > > > > which
> >> > > > > > > > > > > > > > span
> >> > > > > > > > > > > > > > > > > > multiple modules, is somewhat
> haphazard
> >> and
> >> > > > > lacks a
> >> > > > > > > > > > > systematic
> >> > > > > > > > > > > > > > > > structure.
> >> > > > > > > > > > > > > > > > > > For example, the options under the
> >> > > > > > > > '*CheckpointingOptions*'
> >> > > > > > > > > > > > and '
> >> > > > > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are
> >> > related
> >> > > > and
> >> > > > > > have
> >> > > > > > > > no
> >> > > > > > > > > > > clear
> >> > > > > > > > > > > > > > > > boundaries
> >> > > > > > > > > > > > > > > > > > from the user's perspective, but there
> >> is
> >> > no
> >> > > > > > unified
> >> > > > > > > > prefix
> >> > > > > > > > > > > for
> >> > > > > > > > > > > > > > them.
> >> > > > > > > > > > > > > > > > > With
> >> > > > > > > > > > > > > > > > > > the upcoming release of Flink 2.0, we
> >> have
> >> > an
> >> > > > > > > excellent
> >> > > > > > > > > > > > > > opportunity to
> >> > > > > > > > > > > > > > > > > > overhaul and restructure the
> >> configurations
> >> > > > > related
> >> > > > > > > to
> >> > > > > > > > > > > > > > checkpointing,
> >> > > > > > > > > > > > > > > > > > recovery, and state management. This
> >> FLIP
> >> > > > > proposes
> >> > > > > > to
> >> > > > > > > > > > > > reorganize
> >> > > > > > > > > > > > > > these
> >> > > > > > > > > > > > > > > > > > settings, making it more coherent by
> >> > module,
> >> > > > > which
> >> > > > > > > > would
> >> > > > > > > > > > > > > > significantly
> >> > > > > > > > > > > > > > > > > > lower the barriers for understanding
> and
> >> > > reduce
> >> > > > > the
> >> > > > > > > > > > > development
> >> > > > > > > > > > > > > > costs
> >> > > > > > > > > > > > > > > > > > moving forward.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Looking forward to hearing from you!
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > [1]
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Best,
> >> > > > > > > > > > > > > > > > > > Zakelly
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> Best,
> >> Hangxiang.
> >>
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi everyone,

It seems we still don't have a consensus on the rules for boolean type
options. Let me recap the alternatives we have:

Option 1: Use enumeration options instead if possible. But this may cause
some name collisions or confusion as we discussed and we should unify the
statement everywhere.
Option 2: Use boolean options and add 'enabled' as the suffix.
Option 3: Use boolean options and ONLY add 'enabled' when there are more
detailed configurations under the same prefix, to prevent one name from
serving as a prefix to another.

I am inclined to Option 3, since it is more in line with current practice
and friendly for existing users. Also It reduces the length of
configuration names as much as possible.

Looking forward to your opinions! Thanks!


Best,
Zakelly

On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan <za...@gmail.com> wrote:

> Hi Hangxiang,
>
> Thanks for your suggestions!
>
> 1. Could execution.recovery also contain some other behaviors about
>> recovery ? e.g. restart-strategy.
>
>
> That's a very good point. I realize that the word 'recovery' means way too
> many things. So I suggest picking a more specific word here, how about
> 'execution.state-recovery.*' ? Checkpointing and state recovery are
> corresponding terms and won't make ambiguity.
>
> 2. Could we also remove some legacy configuration value ? e.g. LEGACY Mode
>> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
>
>
> I think we could create another FLIP for the deprecation of LEGACY mode.
>
>
>> 3. Could the local checkpoint be cleaned
>> if execution.checkpointing.local-copy.enabled is true and
>> execution.recovery.from-local is false ? I found it's also an issue if
>> current local-recovery from enabled to disabled. Maybe another ticket is
>> needed.
>
>
> IIUC, there is no clear ownership of the local copy files from the
> previous job and it's better to define one. This needs more discussion so
> we could create another thread for this. WDYT?
>
>
> Best,
> Zakelly
>
> On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu <ma...@gmail.com> wrote:
>
>> Hi, Zakelly.
>> Thanks for driving this. Overall LGTM as we discussed offline.
>>
>> Some comments/suggestions just came to mind:
>> 1. Could execution.recovery also contain some other behaviors about
>> recovery ? e.g. restart-strategy.
>> 2. Could we also remove some legacy configuration value ? e.g. LEGACY Mode
>> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
>> 3. Could the local checkpoint be cleaned
>> if execution.checkpointing.local-copy.enabled is true and
>> execution.recovery.from-local is false ? I found it's also an issue if
>> current local-recovery from enabled to disabled. Maybe another ticket is
>> needed.
>> 4. +1 for enabling execution.checkpointing.incremental by default which is
>> basically default configuration in our production environment.
>>
>>
>> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <za...@gmail.com> wrote:
>>
>> > Hi Yun,
>> >
>> > Thanks for your comments!
>> >
>> >  1.  We shall not describe the configuration with its implementation for
>> > > 'execution.checkpointing.local-copy.*' options, for hashmap
>> > state-backend,
>> > > it would write two streams and for Rocksdb state-backend, it would use
>> > > hard-link for backup. Thus, I think
>> > > 'execution.checkpointing.local-backup.*' looks better.
>> >
>> > I agreed that we'd better name the option in user's perspective instead
>> of
>> > the implementation, thus I name it as a copy of the checkpoint in the
>> > local disk, regardless of the way of generating it. The word 'backup' is
>> > also suitable for this case, so I agree to change to
>> > 'execution.checkpointing.local-backup.*' if no one objects.
>> >
>> >  2.  What does the 'execution.checkpointing.data-inline-threshold'
>> mean? It
>> > > seems not so easy to understand.
>> >
>> > The 'execution.checkpointing.data-inline-threshold' (original one as
>> > 'state.storage.fs.memory-threshold') stands for the size threshold below
>> > which state chunks will store inline with the metadata, thus I call it
>> > 'data-inline-threshold'.
>> >
>> >
>> > Best,
>> > Zakelly
>> >
>> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com> wrote:
>> >
>> > > Hi Zakelly,
>> > >
>> > > Thanks for driving this topic. I have two concerns here:
>> > >
>> > >   1.  We shall not describe the configuration with its implementation
>> for
>> > > 'execution.checkpointing.local-copy.*' options, for hashmap
>> > state-backend,
>> > > it would write two streams and for Rocksdb state-backend, it would use
>> > > hard-link for backup. Thus, I think
>> > > 'execution.checkpointing.local-backup.*' looks better.
>> > >   2.  What does the 'execution.checkpointing.data-inline-threshold'
>> mean?
>> > > It seems not so easy to understand.
>> > >
>> > > Best
>> > > Yun Tang
>> > > ________________________________
>> > > From: Piotr Nowojski <pn...@apache.org>
>> > > Sent: Thursday, January 4, 2024 22:37
>> > > To: dev@flink.apache.org <de...@flink.apache.org>
>> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing &
>> > > Recovery Configuration
>> > >
>> > > Hi,
>> > >
>> > > Thanks for trying to clean this up! I don't have strong opinions on
>> the
>> > > topics discussed here, so generally speaking +1 from my side!
>> > >
>> > > Best,
>> > > Piotrek
>> > >
>> > > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com> napisał(a):
>> > >
>> > > > Thanks for the feedback!
>> > > >
>> > > > Using the `execution.checkpointing.incremental.enabled`,
>> > > > and enabling it by default sounds good to me.
>> > > >
>> > > > Best,
>> > > > Rui
>> > > >
>> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <za...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > Hi Rui,
>> > > > >
>> > > > > Thanks for your comments!
>> > > > >
>> > > > > IMO, given that the state backend can be plugably loaded (as you
>> can
>> > > > > specify a state backend factory), I prefer not providing state
>> > backend
>> > > > > specified options in the framework.
>> > > > >
>> > > > > Secondly, the incremental checkpoint is actually a sharing file
>> > > strategy
>> > > > > across checkpoints, which means the state backend *could* reuse
>> files
>> > > > from
>> > > > > previous cp but not *must* do so. When the state backend could not
>> > > reuse
>> > > > > the files, it is reasonable to fallback to a full checkpoint.
>> > > > >
>> > > > > Thus, I suggest we make it `execution.checkpointing.incremental`
>> and
>> > > > enable
>> > > > > it by default. For those state backends not supporting this, they
>> > > perform
>> > > > > full checkpoints and print a warning to inform users. Users do not
>> > need
>> > > > to
>> > > > > pay special attention to different options to control this across
>> > > > different
>> > > > > state backends. This is more user-friendly in my opinion. WDYT?
>> > > > >
>> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > > Hi Zakelly,
>> > > > > >
>> > > > > > I'm not sure whether we could add the state backend type in the
>> > > > > > new key name of state.backend.incremental. It means we use
>> > > > > > `execution.checkpointing.rocksdb-incremental` or
>> > > > > > `execution.checkpointing.rocksdb-incremental.enabled`.
>> > > > > >
>> > > > > > So far, state.backend.incremental only works for rocksdb state
>> > > backend.
>> > > > > > And this feature or optimization is very valuable and huge for
>> > large
>> > > > > > state flink jobs. I believe it's enabled for most production
>> flink
>> > > jobs
>> > > > > > with large rocksdb state.
>> > > > > >
>> > > > > > If this option isn't generic for all state backend types, I
>> guess
>> > we
>> > > > > > can enable `execution.checkpointing.rocksdb-incremental.enabled`
>> > > > > > by default in Flink 2.0.
>> > > > > >
>> > > > > > But if it works for all state backends, it's hard to enable it
>> by
>> > > > > default.
>> > > > > > Enabling great and valuable features or improvements are useful
>> > > > > > for users, especially a lot of new flink users. Out-of-the-box
>> > > options
>> > > > > > are good for users.
>> > > > > >
>> > > > > > WDYT?
>> > > > > >
>> > > > > > Best,
>> > > > > > Rui
>> > > > > >
>> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <
>> zakelly.lan@gmail.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi everyone,
>> > > > > > >
>> > > > > > > Thanks all for your comments!
>> > > > > > >
>> > > > > > > As many of you have questions about the names for boolean
>> > options,
>> > > I
>> > > > > > > suggest we make a naming rule for them. For now I could think
>> of
>> > > > three
>> > > > > > > options:
>> > > > > > >
>> > > > > > > Option 1: Use enumeration options if possible. But this may
>> cause
>> > > > some
>> > > > > > name
>> > > > > > > collisions or confusion as we discussed and we should unify
>> the
>> > > > > statement
>> > > > > > > everywhere.
>> > > > > > > Option 2: Use boolean options and add 'enabled' as the suffix.
>> > > > > > > Option 3: Use boolean options and ONLY add 'enabled' when
>> there
>> > are
>> > > > > more
>> > > > > > > detailed configurations under the same prefix, to prevent one
>> > name
>> > > > from
>> > > > > > > serving as a prefix to another.
>> > > > > > >
>> > > > > > > I am slightly inclined to Option 3, since it is more in line
>> with
>> > > > > current
>> > > > > > > practice and friendly for existing users. Also It reduces the
>> > > length
>> > > > of
>> > > > > > > configuration names as much as possible. I really want to hear
>> > your
>> > > > > > > opinions.
>> > > > > > >
>> > > > > > >
>> > > > > > > @Xuannan
>> > > > > > >
>> > > > > > > I agree with your comments 1 and 3.
>> > > > > > >
>> > > > > > > For 2, If we decide to change the name, maybe
>> > > > > > > `execution.checkpointing.parallel-cleaner` is better? And as
>> for
>> > > > > whether
>> > > > > > to
>> > > > > > > add 'enabled' I suggest we discuss the rule above. WDYT?
>> > > > > > > Thanks!
>> > > > > > >
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Zakelly
>> > > > > > >
>> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
>> > suxuannan95@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Zakelly,
>> > > > > > > >
>> > > > > > > > Thanks for driving this! The organization of the
>> configuration
>> > > > option
>> > > > > > > > in the FLIP looks much cleaner and easier to understand. +1
>> to
>> > > the
>> > > > > > > > FLIP.
>> > > > > > > >
>> > > > > > > > Just some questions from me.
>> > > > > > > >
>> > > > > > > > 1. I think the change to the ConfigOptions should be put in
>> the
>> > > > > > > > `Public Interface` section, instead of `Proposed Changed`,
>> as
>> > > those
>> > > > > > > > configuration options are public interface.
>> > > > > > > >
>> > > > > > > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems
>> > > > confusing.
>> > > > > > > > It feels like it is used to choose different modes. In
>> fact, it
>> > > is
>> > > > a
>> > > > > > > > boolean flag to indicate whether to enable parallel clean.
>> How
>> > > > about
>> > > > > > > > making it `state.checkpoint.cleaner.parallel-mode.enabled`?
>> > > > > > > >
>> > > > > > > > 3. The `execution.checkpointing.write-buffer` may better be
>> > > > > > > > `execution.checkpointing.write-buffer-size` so that we know
>> it
>> > is
>> > > > > > > > configuring the size of the buffer.
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Xuannan
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
>> > fredialei@gmail.com>
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > Hi Zakelly,
>> > > > > > > > >
>> > > > > > > > > > Considering the name occupation, how about naming it as
>> > > > > > > > `execution.checkpointing.type`?
>> > > > > > > > >
>> > > > > > > > > `Checkpoint Type`[1,2] is used to describe
>> aligned/unaligned
>> > > > > > > > > checkpoint, I am inclined to make a choice between
>> > > > > > > > > `execution.checkpointing.incremental` and
>> > > > > > > > > `execution.checkpointing.incremental.enabled`.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > [1]
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
>> > > > > > > > > [2]
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Best,
>> > > > > > > > > Yanfei
>> > > > > > > > >
>> > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
>> 14:41写道:
>> > > > > > > > > >
>> > > > > > > > > > Hi Lijie,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for the reminder! I missed this.
>> > > > > > > > > >
>> > > > > > > > > > Considering the name occupation, how about naming it as
>> > > > > > > > > > `execution.checkpointing.type`?
>> > > > > > > > > >
>> > > > > > > > > > Actually I think the current
>> `execution.checkpointing.mode`
>> > > is
>> > > > > > > > confusing in
>> > > > > > > > > > some ways, maybe
>> `execution.checkpointing.data-consistency`
>> > > is
>> > > > > > > better.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Best,
>> > > > > > > > > > Zakelly
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
>> > > > > > > wangdachui9501@gmail.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Zakelly,
>> > > > > > > > > > >
>> > > > > > > > > > > >> I'm wondering if
>> > `execution.checkpointing.savepoint-dir`
>> > > > > would
>> > > > > > > be
>> > > > > > > > > > > better.
>> > > > > > > > > > >
>> > > > > > > > > > > `execution.checkpointing.dir` and
>> > > > > > > > `execution.checkpointing.savepoint-dir`
>> > > > > > > > > > > are also fine for me.
>> > > > > > > > > > >
>> > > > > > > > > > > >> So I think an enumeration option
>> > > > > > `execution.checkpointing.mode`
>> > > > > > > > which
>> > > > > > > > > > > can be 'full' (default) or 'incremental' would be
>> better
>> > > > > > > > > > >
>> > > > > > > > > > > I agree with using an enumeration option. But
>> currently
>> > > there
>> > > > > is
>> > > > > > > > already a
>> > > > > > > > > > > configuration option called
>> > `execution.checkpointing.mode`,
>> > > > > which
>> > > > > > > is
>> > > > > > > > used
>> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we
>> need to
>> > > use
>> > > > > > > > another name
>> > > > > > > > > > > or merge these two options.
>> > > > > > > > > > >
>> > > > > > > > > > > Best,
>> > > > > > > > > > > Lijie
>> > > > > > > > > > >
>> > > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
>> > > 11:43写道:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi everyone,
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks all for your comments!
>> > > > > > > > > > > >
>> > > > > > > > > > > > @Yanfei
>> > > > > > > > > > > >
>> > > > > > > > > > > > > 1. For some state backends that do not support
>> > > > incremental
>> > > > > > > > checkpoint,
>> > > > > > > > > > > > > how does the
>> > execution.checkpointing.incrementaloption
>> > > > take
>> > > > > > > > effect? Or
>> > > > > > > > > > > > > is it better to put incremental under
>> > > > > > > > state.backend.xxx.incremental?
>> > > > > > > > > > > > >
>> > > > > > > > > > > > I'd rather not put the option for incremental
>> > checkpoint
>> > > > > under
>> > > > > > > the
>> > > > > > > > > > > > 'state.backend', since it is more about the
>> > checkpointing
>> > > > > > instead
>> > > > > > > > of
>> > > > > > > > > > > state
>> > > > > > > > > > > > accessing. Of course, the state backend may not
>> > > necessarily
>> > > > > do
>> > > > > > > > > > > incremental
>> > > > > > > > > > > > checkpoint as requested. If the state backend is not
>> > > > capable
>> > > > > of
>> > > > > > > > taking
>> > > > > > > > > > > > incremental cp, it is better to fallback to the full
>> > cp.
>> > > > > > > > > > > >
>> > > > > > > > > > > > 2. I'm a little worried that putting all
>> configurations
>> > > > into
>> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce
>> some
>> > > > > > dependency
>> > > > > > > > > > > > > problems. Some options would be used by
>> flink-runtime
>> > > > > module,
>> > > > > > > but
>> > > > > > > > > > > > > flink-runtime should not depend on
>> > > flink-streaming-java.
>> > > > > e.g.
>> > > > > > > > > > > > > FLINK-28286[1].
>> > > > > > > > > > > > > So, I prefer to move configurations to
>> > > > > > `CheckpointingOptions`,
>> > > > > > > > WDYT?
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yes, that's a very good point.  Moving to
>> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes sense.
>> > > > > > > > > > > >
>> > > > > > > > > > > > @Lijie
>> > > > > > > > > > > >
>> > > > > > > > > > > > How about
>> > > > > > > > > > > > > state.savepoints.dir ->
>> > > > > execution.checkpointing.savepoint.dir
>> > > > > > > > > > > > > state.checkpoints.dir ->
>> > > > > > execution.checkpointing.checkpoint.dir
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Actually, I think the `checkpointing.checkpoint` may
>> > > cause
>> > > > > some
>> > > > > > > > > > > confusion.
>> > > > > > > > > > > > But I'm ok if others agree.
>> > > > > > > > > > > > I'm wondering if
>> > `execution.checkpointing.savepoint-dir`
>> > > > > would
>> > > > > > be
>> > > > > > > > better.
>> > > > > > > > > > > > WDYT?
>> > > > > > > > > > > >
>> > > > > > > > > > > > 2. We changed the
>> execution.checkpointing.local-copy'
>> > to
>> > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
>> Should
>> > we
>> > > > > also
>> > > > > > > add
>> > > > > > > > > > > > "enabled"
>> > > > > > > > > > > > > suffix for other boolean type configuration
>> options ?
>> > > For
>> > > > > > > > example,
>> > > > > > > > > > > > > execution.checkpointing.incremental ->
>> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Actually, the incremental cp is something like
>> > choosing a
>> > > > > mode
>> > > > > > > for
>> > > > > > > > doing
>> > > > > > > > > > > > checkpoint instead of enabling a function. So I
>> think
>> > an
>> > > > > > > > enumeration
>> > > > > > > > > > > option
>> > > > > > > > > > > > `execution.checkpointing.mode` which can be 'full'
>> > > > (default)
>> > > > > or
>> > > > > > > > > > > > 'incremental' would be better, WDYT?
>> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think about this?
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
>> > > > > > > > wangdachui9501@gmail.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi Zakelly,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for driving the discussion.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 1.
>> > > > > > > > > > > > > >> But I'm not so sure since there is only one
>> > > > > > > savepoint-related
>> > > > > > > > > > > option.
>> > > > > > > > > > > > > Maybe someone else could share some thoughts here.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > How about
>> > > > > > > > > > > > > state.savepoints.dir ->
>> > > > > execution.checkpointing.savepoint.dir
>> > > > > > > > > > > > > state.checkpoints.dir ->
>> > > > > > execution.checkpointing.checkpoint.dir
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 2. We changed the
>> execution.checkpointing.local-copy'
>> > > to
>> > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
>> Should
>> > we
>> > > > > also
>> > > > > > > add
>> > > > > > > > > > > > "enabled"
>> > > > > > > > > > > > > suffix for other boolean type configuration
>> options ?
>> > > For
>> > > > > > > > example,
>> > > > > > > > > > > > > execution.checkpointing.incremental ->
>> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > In this way, the naming style of configuration
>> > options
>> > > is
>> > > > > > > > unified, and
>> > > > > > > > > > > it
>> > > > > > > > > > > > > can avoid potential similar problems (for
>> example, we
>> > > may
>> > > > > > need
>> > > > > > > > to add
>> > > > > > > > > > > > more
>> > > > > > > > > > > > > options for incremental checkpoint in the future).
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > Lijie
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二
>> > > 12:05写道:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi Zakelly,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thank you for creating the FLIP and starting the
>> > > > > > discussion.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > The current arrangement of these options is
>> indeed
>> > > > > somewhat
>> > > > > > > > > > > haphazard,
>> > > > > > > > > > > > > > and the new arrangement looks much better. I
>> have
>> > > some
>> > > > > > > > questions
>> > > > > > > > > > > about
>> > > > > > > > > > > > > > the arrangement of some new configuration
>> options:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > 1. For some state backends that do not support
>> > > > > incremental
>> > > > > > > > > > > checkpoint,
>> > > > > > > > > > > > > > how does the
>> > > execution.checkpointing.incrementaloption
>> > > > > take
>> > > > > > > > effect?
>> > > > > > > > > > > Or
>> > > > > > > > > > > > > > is it better to put incremental under
>> > > > > > > > state.backend.xxx.incremental?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > 2. I'm a little worried that putting all
>> > > configurations
>> > > > > > into
>> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce
>> some
>> > > > > > > dependency
>> > > > > > > > > > > > > > problems. Some options would be used by
>> > flink-runtime
>> > > > > > module,
>> > > > > > > > but
>> > > > > > > > > > > > > > flink-runtime should not depend on
>> > > > flink-streaming-java.
>> > > > > > e.g.
>> > > > > > > > > > > > > > FLINK-28286[1].
>> > > > > > > > > > > > > > So, I prefer to move configurations to
>> > > > > > > `CheckpointingOptions`,
>> > > > > > > > WDYT?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > [1]
>> > > https://issues.apache.org/jira/browse/FLINK-28286
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > Yanfei
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Zakelly Lan <za...@gmail.com>
>> 于2023年12月25日周一
>> > > > > > 21:14写道:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi Rui Fan and Junrui,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Thanks for the reminder! I agree to change the
>> > > > > > > > > > > > > > > 'execution.checkpointing.local-copy' to
>> > > > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > And for other suggestions Rui proposed:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > 1. How about
>> execution.checkpointing.storage.type
>> > > > > instead
>> > > > > > > > > > > > > > > > of execution.checkpointing.storage?
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Ah, I missed something here. Actually I
>> suggest
>> > we
>> > > > > could
>> > > > > > > > merge the
>> > > > > > > > > > > > > > current
>> > > > > > > > > > > > > > > 'state.checkpoints.dir' and
>> > > > 'state.checkpoint-storage'
>> > > > > > into
>> > > > > > > > one URI
>> > > > > > > > > > > > > > > configuration named
>> > 'execution.checkpointing.dir'.
>> > > > > WDYT?
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a
>> > > little
>> > > > > > weird.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Yes, I think it is better to make 'savepoint'
>> and
>> > > > > > > > 'checkpoint' the
>> > > > > > > > > > > > same
>> > > > > > > > > > > > > > > level. But I'm not so sure since there is only
>> > one
>> > > > > > > > > > > savepoint-related
>> > > > > > > > > > > > > > > option. Maybe someone else could share some
>> > > thoughts
>> > > > > > here.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
>> > instead
>> > > of
>> > > > > > > > > > > > > > > > execution.recovery.mode?
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >  Agreed. That's more accurate.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Many thanks for your suggestions!
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > Zakelly
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
>> > > > > > > > jrlee.ljr@gmail.com>
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Hi Zakelly,
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Thanks for driving this. I agree that the
>> > > proposed
>> > > > > > > > restructuring
>> > > > > > > > > > > of
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > configuration options is largely positive.
>> It
>> > > will
>> > > > > make
>> > > > > > > > > > > > understanding
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > working with Flink configurations more
>> > intuitive.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Most of the proposed changes look great.
>> Just a
>> > > > > > heads-up,
>> > > > > > > > as Rui
>> > > > > > > > > > > > Fan
>> > > > > > > > > > > > > > > > mentioned, Flink currently requires that no
>> > > > > > > configOption's
>> > > > > > > > key be
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > prefix of another to avoid issues when we
>> > > > eventually
>> > > > > > > adopt
>> > > > > > > > a
>> > > > > > > > > > > > standard
>> > > > > > > > > > > > > > YAML
>> > > > > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
>> > > > > > > > > > > > > > > >
>> > > https://issues.apache.org/jira/browse/FLINK-29372
>> > > > ).
>> > > > > > > > Therefore,
>> > > > > > > > > > > > it's
>> > > > > > > > > > > > > > better
>> > > > > > > > > > > > > > > > to change the key
>> > > > > 'execution.checkpointing.local-copy'
>> > > > > > > > because it
>> > > > > > > > > > > > > > serves as
>> > > > > > > > > > > > > > > > a prefix to the key
>> > > > > > > > 'execution.checkpointing.local-copy.dir'.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > Junrui
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Rui Fan <19...@gmail.com>
>> 于2023年12月25日周一
>> > > > > 19:11写道:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Hi Zakelly,
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Thank you for driving this proposal!
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Overall good for me. I have some questions
>> > > about
>> > > > > > these
>> > > > > > > > names.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 1. How about
>> > > execution.checkpointing.storage.type
>> > > > > > > > instead of
>> > > > > > > > > > > > > > > > > execution.checkpointing.storage?
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > It's similar to state.backend.type.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 2. How about
>> > > > > > execution.checkpointing.local-copy.enabled
>> > > > > > > > instead
>> > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > execution.checkpointing.local-copy?
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > You added a new option:
>> > > > > > > > execution.checkpointing.local-copy.dir.
>> > > > > > > > > > > > > > > > > IIUC, one option name shouldn't be the
>> prefix
>> > > of
>> > > > > > other
>> > > > > > > > options.
>> > > > > > > > > > > > > > > > > If you add a new option
>> > > > > > > > execution.checkpointing.local-copy,
>> > > > > > > > > > > > > > > > > flink CI will fail directly.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir
>> is a
>> > > > > little
>> > > > > > > > weird.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > For old options: state.savepoints.dir and
>> > > > > > > > > > > state.checkpoints.dir,
>> > > > > > > > > > > > > > > > > the savepoint and checkpoint are the same
>> > > level.
>> > > > It
>> > > > > > > means
>> > > > > > > > > > > > > > > > > it's a checkpoint or savepoint.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > The new option
>> execution.checkpointing.dir is
>> > > > fine
>> > > > > > for
>> > > > > > > > me.
>> > > > > > > > > > > > > > > > > However,
>> > execution.checkpointing.savepoint.dir
>> > > > is a
>> > > > > > > > little
>> > > > > > > > > > > weird.
>> > > > > > > > > > > > > > > > > I don't know which name is better now.
>> Let us
>> > > > think
>> > > > > > > > about it
>> > > > > > > > > > > > more.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
>> > > > instead
>> > > > > of
>> > > > > > > > > > > > > > > > > execution.recovery.mode?
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > The meaning of mode is too broad. The
>> > > claim-mode
>> > > > > may
>> > > > > > > > > > > > > > > > > be more accurate for users.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > WDYT?
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > Rui
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly
>> Lan <
>> > > > > > > > > > > > zakelly.lan@gmail.com
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Hi devs,
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > I'd like to start a discussion on
>> FLIP-406:
>> > > > > > > Reorganize
>> > > > > > > > State
>> > > > > > > > > > > &
>> > > > > > > > > > > > > > > > > > Checkpointing & Recovery
>> Configuration[1].
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Currently, the configuration options
>> > > pertaining
>> > > > > to
>> > > > > > > > > > > > checkpointing,
>> > > > > > > > > > > > > > > > > recovery,
>> > > > > > > > > > > > > > > > > > and state management are primarily
>> grouped
>> > > > under
>> > > > > > the
>> > > > > > > > > > > following
>> > > > > > > > > > > > > > > > prefixes:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >    - state.backend.* : configurations
>> > related
>> > > > to
>> > > > > > > state
>> > > > > > > > > > > > accessing
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > >    checkpointing, as well as specific
>> > options
>> > > > for
>> > > > > > > > individual
>> > > > > > > > > > > > > state
>> > > > > > > > > > > > > > > > > backends
>> > > > > > > > > > > > > > > > > >    - execution.checkpointing.* :
>> > > configurations
>> > > > > > > > associated
>> > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > checkpoint
>> > > > > > > > > > > > > > > > > >    execution and recovery
>> > > > > > > > > > > > > > > > > >    - execution.savepoint.*:
>> configurations
>> > > for
>> > > > > > > > recovery from
>> > > > > > > > > > > > > > savepoint
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > In addition, there are several
>> individual
>> > > > options
>> > > > > > > such
>> > > > > > > > as '
>> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*' and
>> > > > > > > > '*state.checkpoints.dir*'
>> > > > > > > > > > > that
>> > > > > > > > > > > > > fall
>> > > > > > > > > > > > > > > > > outside
>> > > > > > > > > > > > > > > > > > of these prefixes. The current
>> arrangement
>> > of
>> > > > > these
>> > > > > > > > options,
>> > > > > > > > > > > > > which
>> > > > > > > > > > > > > > span
>> > > > > > > > > > > > > > > > > > multiple modules, is somewhat haphazard
>> and
>> > > > > lacks a
>> > > > > > > > > > > systematic
>> > > > > > > > > > > > > > > > structure.
>> > > > > > > > > > > > > > > > > > For example, the options under the
>> > > > > > > > '*CheckpointingOptions*'
>> > > > > > > > > > > > and '
>> > > > > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are
>> > related
>> > > > and
>> > > > > > have
>> > > > > > > > no
>> > > > > > > > > > > clear
>> > > > > > > > > > > > > > > > boundaries
>> > > > > > > > > > > > > > > > > > from the user's perspective, but there
>> is
>> > no
>> > > > > > unified
>> > > > > > > > prefix
>> > > > > > > > > > > for
>> > > > > > > > > > > > > > them.
>> > > > > > > > > > > > > > > > > With
>> > > > > > > > > > > > > > > > > > the upcoming release of Flink 2.0, we
>> have
>> > an
>> > > > > > > excellent
>> > > > > > > > > > > > > > opportunity to
>> > > > > > > > > > > > > > > > > > overhaul and restructure the
>> configurations
>> > > > > related
>> > > > > > > to
>> > > > > > > > > > > > > > checkpointing,
>> > > > > > > > > > > > > > > > > > recovery, and state management. This
>> FLIP
>> > > > > proposes
>> > > > > > to
>> > > > > > > > > > > > reorganize
>> > > > > > > > > > > > > > these
>> > > > > > > > > > > > > > > > > > settings, making it more coherent by
>> > module,
>> > > > > which
>> > > > > > > > would
>> > > > > > > > > > > > > > significantly
>> > > > > > > > > > > > > > > > > > lower the barriers for understanding and
>> > > reduce
>> > > > > the
>> > > > > > > > > > > development
>> > > > > > > > > > > > > > costs
>> > > > > > > > > > > > > > > > > > moving forward.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Looking forward to hearing from you!
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > [1]
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > Zakelly
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> Best,
>> Hangxiang.
>>
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi Hangxiang,

Thanks for your suggestions!

1. Could execution.recovery also contain some other behaviors about
> recovery ? e.g. restart-strategy.


That's a very good point. I realize that the word 'recovery' means way too
many things. So I suggest picking a more specific word here, how about
'execution.state-recovery.*' ? Checkpointing and state recovery are
corresponding terms and won't make ambiguity.

2. Could we also remove some legacy configuration value ? e.g. LEGACY Mode
> for execution.savepoint-restore-mode/execution.recovery.claim-mode.


I think we could create another FLIP for the deprecation of LEGACY mode.


> 3. Could the local checkpoint be cleaned
> if execution.checkpointing.local-copy.enabled is true and
> execution.recovery.from-local is false ? I found it's also an issue if
> current local-recovery from enabled to disabled. Maybe another ticket is
> needed.


IIUC, there is no clear ownership of the local copy files from the previous
job and it's better to define one. This needs more discussion so we could
create another thread for this. WDYT?


Best,
Zakelly

On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu <ma...@gmail.com> wrote:

> Hi, Zakelly.
> Thanks for driving this. Overall LGTM as we discussed offline.
>
> Some comments/suggestions just came to mind:
> 1. Could execution.recovery also contain some other behaviors about
> recovery ? e.g. restart-strategy.
> 2. Could we also remove some legacy configuration value ? e.g. LEGACY Mode
> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
> 3. Could the local checkpoint be cleaned
> if execution.checkpointing.local-copy.enabled is true and
> execution.recovery.from-local is false ? I found it's also an issue if
> current local-recovery from enabled to disabled. Maybe another ticket is
> needed.
> 4. +1 for enabling execution.checkpointing.incremental by default which is
> basically default configuration in our production environment.
>
>
> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi Yun,
> >
> > Thanks for your comments!
> >
> >  1.  We shall not describe the configuration with its implementation for
> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > state-backend,
> > > it would write two streams and for Rocksdb state-backend, it would use
> > > hard-link for backup. Thus, I think
> > > 'execution.checkpointing.local-backup.*' looks better.
> >
> > I agreed that we'd better name the option in user's perspective instead
> of
> > the implementation, thus I name it as a copy of the checkpoint in the
> > local disk, regardless of the way of generating it. The word 'backup' is
> > also suitable for this case, so I agree to change to
> > 'execution.checkpointing.local-backup.*' if no one objects.
> >
> >  2.  What does the 'execution.checkpointing.data-inline-threshold' mean?
> It
> > > seems not so easy to understand.
> >
> > The 'execution.checkpointing.data-inline-threshold' (original one as
> > 'state.storage.fs.memory-threshold') stands for the size threshold below
> > which state chunks will store inline with the metadata, thus I call it
> > 'data-inline-threshold'.
> >
> >
> > Best,
> > Zakelly
> >
> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com> wrote:
> >
> > > Hi Zakelly,
> > >
> > > Thanks for driving this topic. I have two concerns here:
> > >
> > >   1.  We shall not describe the configuration with its implementation
> for
> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > state-backend,
> > > it would write two streams and for Rocksdb state-backend, it would use
> > > hard-link for backup. Thus, I think
> > > 'execution.checkpointing.local-backup.*' looks better.
> > >   2.  What does the 'execution.checkpointing.data-inline-threshold'
> mean?
> > > It seems not so easy to understand.
> > >
> > > Best
> > > Yun Tang
> > > ________________________________
> > > From: Piotr Nowojski <pn...@apache.org>
> > > Sent: Thursday, January 4, 2024 22:37
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing &
> > > Recovery Configuration
> > >
> > > Hi,
> > >
> > > Thanks for trying to clean this up! I don't have strong opinions on the
> > > topics discussed here, so generally speaking +1 from my side!
> > >
> > > Best,
> > > Piotrek
> > >
> > > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com> napisał(a):
> > >
> > > > Thanks for the feedback!
> > > >
> > > > Using the `execution.checkpointing.incremental.enabled`,
> > > > and enabling it by default sounds good to me.
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <za...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Rui,
> > > > >
> > > > > Thanks for your comments!
> > > > >
> > > > > IMO, given that the state backend can be plugably loaded (as you
> can
> > > > > specify a state backend factory), I prefer not providing state
> > backend
> > > > > specified options in the framework.
> > > > >
> > > > > Secondly, the incremental checkpoint is actually a sharing file
> > > strategy
> > > > > across checkpoints, which means the state backend *could* reuse
> files
> > > > from
> > > > > previous cp but not *must* do so. When the state backend could not
> > > reuse
> > > > > the files, it is reasonable to fallback to a full checkpoint.
> > > > >
> > > > > Thus, I suggest we make it `execution.checkpointing.incremental`
> and
> > > > enable
> > > > > it by default. For those state backends not supporting this, they
> > > perform
> > > > > full checkpoints and print a warning to inform users. Users do not
> > need
> > > > to
> > > > > pay special attention to different options to control this across
> > > > different
> > > > > state backends. This is more user-friendly in my opinion. WDYT?
> > > > >
> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > I'm not sure whether we could add the state backend type in the
> > > > > > new key name of state.backend.incremental. It means we use
> > > > > > `execution.checkpointing.rocksdb-incremental` or
> > > > > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > > > > >
> > > > > > So far, state.backend.incremental only works for rocksdb state
> > > backend.
> > > > > > And this feature or optimization is very valuable and huge for
> > large
> > > > > > state flink jobs. I believe it's enabled for most production
> flink
> > > jobs
> > > > > > with large rocksdb state.
> > > > > >
> > > > > > If this option isn't generic for all state backend types, I guess
> > we
> > > > > > can enable `execution.checkpointing.rocksdb-incremental.enabled`
> > > > > > by default in Flink 2.0.
> > > > > >
> > > > > > But if it works for all state backends, it's hard to enable it by
> > > > > default.
> > > > > > Enabling great and valuable features or improvements are useful
> > > > > > for users, especially a lot of new flink users. Out-of-the-box
> > > options
> > > > > > are good for users.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Best,
> > > > > > Rui
> > > > > >
> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <
> zakelly.lan@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thanks all for your comments!
> > > > > > >
> > > > > > > As many of you have questions about the names for boolean
> > options,
> > > I
> > > > > > > suggest we make a naming rule for them. For now I could think
> of
> > > > three
> > > > > > > options:
> > > > > > >
> > > > > > > Option 1: Use enumeration options if possible. But this may
> cause
> > > > some
> > > > > > name
> > > > > > > collisions or confusion as we discussed and we should unify the
> > > > > statement
> > > > > > > everywhere.
> > > > > > > Option 2: Use boolean options and add 'enabled' as the suffix.
> > > > > > > Option 3: Use boolean options and ONLY add 'enabled' when there
> > are
> > > > > more
> > > > > > > detailed configurations under the same prefix, to prevent one
> > name
> > > > from
> > > > > > > serving as a prefix to another.
> > > > > > >
> > > > > > > I am slightly inclined to Option 3, since it is more in line
> with
> > > > > current
> > > > > > > practice and friendly for existing users. Also It reduces the
> > > length
> > > > of
> > > > > > > configuration names as much as possible. I really want to hear
> > your
> > > > > > > opinions.
> > > > > > >
> > > > > > >
> > > > > > > @Xuannan
> > > > > > >
> > > > > > > I agree with your comments 1 and 3.
> > > > > > >
> > > > > > > For 2, If we decide to change the name, maybe
> > > > > > > `execution.checkpointing.parallel-cleaner` is better? And as
> for
> > > > > whether
> > > > > > to
> > > > > > > add 'enabled' I suggest we discuss the rule above. WDYT?
> > > > > > > Thanks!
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
> > suxuannan95@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zakelly,
> > > > > > > >
> > > > > > > > Thanks for driving this! The organization of the
> configuration
> > > > option
> > > > > > > > in the FLIP looks much cleaner and easier to understand. +1
> to
> > > the
> > > > > > > > FLIP.
> > > > > > > >
> > > > > > > > Just some questions from me.
> > > > > > > >
> > > > > > > > 1. I think the change to the ConfigOptions should be put in
> the
> > > > > > > > `Public Interface` section, instead of `Proposed Changed`, as
> > > those
> > > > > > > > configuration options are public interface.
> > > > > > > >
> > > > > > > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems
> > > > confusing.
> > > > > > > > It feels like it is used to choose different modes. In fact,
> it
> > > is
> > > > a
> > > > > > > > boolean flag to indicate whether to enable parallel clean.
> How
> > > > about
> > > > > > > > making it `state.checkpoint.cleaner.parallel-mode.enabled`?
> > > > > > > >
> > > > > > > > 3. The `execution.checkpointing.write-buffer` may better be
> > > > > > > > `execution.checkpointing.write-buffer-size` so that we know
> it
> > is
> > > > > > > > configuring the size of the buffer.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Xuannan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
> > fredialei@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Zakelly,
> > > > > > > > >
> > > > > > > > > > Considering the name occupation, how about naming it as
> > > > > > > > `execution.checkpointing.type`?
> > > > > > > > >
> > > > > > > > > `Checkpoint Type`[1,2] is used to describe
> aligned/unaligned
> > > > > > > > > checkpoint, I am inclined to make a choice between
> > > > > > > > > `execution.checkpointing.incremental` and
> > > > > > > > > `execution.checkpointing.incremental.enabled`.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > > > > > > > [2]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best,
> > > > > > > > > Yanfei
> > > > > > > > >
> > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
> 14:41写道:
> > > > > > > > > >
> > > > > > > > > > Hi Lijie,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reminder! I missed this.
> > > > > > > > > >
> > > > > > > > > > Considering the name occupation, how about naming it as
> > > > > > > > > > `execution.checkpointing.type`?
> > > > > > > > > >
> > > > > > > > > > Actually I think the current
> `execution.checkpointing.mode`
> > > is
> > > > > > > > confusing in
> > > > > > > > > > some ways, maybe
> `execution.checkpointing.data-consistency`
> > > is
> > > > > > > better.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Zakelly
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > > > > > > wangdachui9501@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > >
> > > > > > > > > > > >> I'm wondering if
> > `execution.checkpointing.savepoint-dir`
> > > > > would
> > > > > > > be
> > > > > > > > > > > better.
> > > > > > > > > > >
> > > > > > > > > > > `execution.checkpointing.dir` and
> > > > > > > > `execution.checkpointing.savepoint-dir`
> > > > > > > > > > > are also fine for me.
> > > > > > > > > > >
> > > > > > > > > > > >> So I think an enumeration option
> > > > > > `execution.checkpointing.mode`
> > > > > > > > which
> > > > > > > > > > > can be 'full' (default) or 'incremental' would be
> better
> > > > > > > > > > >
> > > > > > > > > > > I agree with using an enumeration option. But currently
> > > there
> > > > > is
> > > > > > > > already a
> > > > > > > > > > > configuration option called
> > `execution.checkpointing.mode`,
> > > > > which
> > > > > > > is
> > > > > > > > used
> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need
> to
> > > use
> > > > > > > > another name
> > > > > > > > > > > or merge these two options.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Lijie
> > > > > > > > > > >
> > > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
> > > 11:43写道:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks all for your comments!
> > > > > > > > > > > >
> > > > > > > > > > > > @Yanfei
> > > > > > > > > > > >
> > > > > > > > > > > > > 1. For some state backends that do not support
> > > > incremental
> > > > > > > > checkpoint,
> > > > > > > > > > > > > how does the
> > execution.checkpointing.incrementaloption
> > > > take
> > > > > > > > effect? Or
> > > > > > > > > > > > > is it better to put incremental under
> > > > > > > > state.backend.xxx.incremental?
> > > > > > > > > > > > >
> > > > > > > > > > > > I'd rather not put the option for incremental
> > checkpoint
> > > > > under
> > > > > > > the
> > > > > > > > > > > > 'state.backend', since it is more about the
> > checkpointing
> > > > > > instead
> > > > > > > > of
> > > > > > > > > > > state
> > > > > > > > > > > > accessing. Of course, the state backend may not
> > > necessarily
> > > > > do
> > > > > > > > > > > incremental
> > > > > > > > > > > > checkpoint as requested. If the state backend is not
> > > > capable
> > > > > of
> > > > > > > > taking
> > > > > > > > > > > > incremental cp, it is better to fallback to the full
> > cp.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. I'm a little worried that putting all
> configurations
> > > > into
> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > > > > dependency
> > > > > > > > > > > > > problems. Some options would be used by
> flink-runtime
> > > > > module,
> > > > > > > but
> > > > > > > > > > > > > flink-runtime should not depend on
> > > flink-streaming-java.
> > > > > e.g.
> > > > > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > > > > So, I prefer to move configurations to
> > > > > > `CheckpointingOptions`,
> > > > > > > > WDYT?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, that's a very good point.  Moving to
> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes sense.
> > > > > > > > > > > >
> > > > > > > > > > > > @Lijie
> > > > > > > > > > > >
> > > > > > > > > > > > How about
> > > > > > > > > > > > > state.savepoints.dir ->
> > > > > execution.checkpointing.savepoint.dir
> > > > > > > > > > > > > state.checkpoints.dir ->
> > > > > > execution.checkpointing.checkpoint.dir
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Actually, I think the `checkpointing.checkpoint` may
> > > cause
> > > > > some
> > > > > > > > > > > confusion.
> > > > > > > > > > > > But I'm ok if others agree.
> > > > > > > > > > > > I'm wondering if
> > `execution.checkpointing.savepoint-dir`
> > > > > would
> > > > > > be
> > > > > > > > better.
> > > > > > > > > > > > WDYT?
> > > > > > > > > > > >
> > > > > > > > > > > > 2. We changed the execution.checkpointing.local-copy'
> > to
> > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> Should
> > we
> > > > > also
> > > > > > > add
> > > > > > > > > > > > "enabled"
> > > > > > > > > > > > > suffix for other boolean type configuration
> options ?
> > > For
> > > > > > > > example,
> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Actually, the incremental cp is something like
> > choosing a
> > > > > mode
> > > > > > > for
> > > > > > > > doing
> > > > > > > > > > > > checkpoint instead of enabling a function. So I think
> > an
> > > > > > > > enumeration
> > > > > > > > > > > option
> > > > > > > > > > > > `execution.checkpointing.mode` which can be 'full'
> > > > (default)
> > > > > or
> > > > > > > > > > > > 'incremental' would be better, WDYT?
> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think about this?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > > > > > > > wangdachui9501@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for driving the discussion.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1.
> > > > > > > > > > > > > >> But I'm not so sure since there is only one
> > > > > > > savepoint-related
> > > > > > > > > > > option.
> > > > > > > > > > > > > Maybe someone else could share some thoughts here.
> > > > > > > > > > > > >
> > > > > > > > > > > > > How about
> > > > > > > > > > > > > state.savepoints.dir ->
> > > > > execution.checkpointing.savepoint.dir
> > > > > > > > > > > > > state.checkpoints.dir ->
> > > > > > execution.checkpointing.checkpoint.dir
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. We changed the
> execution.checkpointing.local-copy'
> > > to
> > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> Should
> > we
> > > > > also
> > > > > > > add
> > > > > > > > > > > > "enabled"
> > > > > > > > > > > > > suffix for other boolean type configuration
> options ?
> > > For
> > > > > > > > example,
> > > > > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > > > > >
> > > > > > > > > > > > > In this way, the naming style of configuration
> > options
> > > is
> > > > > > > > unified, and
> > > > > > > > > > > it
> > > > > > > > > > > > > can avoid potential similar problems (for example,
> we
> > > may
> > > > > > need
> > > > > > > > to add
> > > > > > > > > > > > more
> > > > > > > > > > > > > options for incremental checkpoint in the future).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Lijie
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二
> > > 12:05写道:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for creating the FLIP and starting the
> > > > > > discussion.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The current arrangement of these options is
> indeed
> > > > > somewhat
> > > > > > > > > > > haphazard,
> > > > > > > > > > > > > > and the new arrangement looks much better. I have
> > > some
> > > > > > > > questions
> > > > > > > > > > > about
> > > > > > > > > > > > > > the arrangement of some new configuration
> options:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. For some state backends that do not support
> > > > > incremental
> > > > > > > > > > > checkpoint,
> > > > > > > > > > > > > > how does the
> > > execution.checkpointing.incrementaloption
> > > > > take
> > > > > > > > effect?
> > > > > > > > > > > Or
> > > > > > > > > > > > > > is it better to put incremental under
> > > > > > > > state.backend.xxx.incremental?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. I'm a little worried that putting all
> > > configurations
> > > > > > into
> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce
> some
> > > > > > > dependency
> > > > > > > > > > > > > > problems. Some options would be used by
> > flink-runtime
> > > > > > module,
> > > > > > > > but
> > > > > > > > > > > > > > flink-runtime should not depend on
> > > > flink-streaming-java.
> > > > > > e.g.
> > > > > > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > > > > > So, I prefer to move configurations to
> > > > > > > `CheckpointingOptions`,
> > > > > > > > WDYT?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > https://issues.apache.org/jira/browse/FLINK-28286
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Yanfei
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Zakelly Lan <za...@gmail.com>
> 于2023年12月25日周一
> > > > > > 21:14写道:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Rui Fan and Junrui,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > > > > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > And for other suggestions Rui proposed:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. How about
> execution.checkpointing.storage.type
> > > > > instead
> > > > > > > > > > > > > > > > of execution.checkpointing.storage?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ah, I missed something here. Actually I suggest
> > we
> > > > > could
> > > > > > > > merge the
> > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > 'state.checkpoints.dir' and
> > > > 'state.checkpoint-storage'
> > > > > > into
> > > > > > > > one URI
> > > > > > > > > > > > > > > configuration named
> > 'execution.checkpointing.dir'.
> > > > > WDYT?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a
> > > little
> > > > > > weird.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, I think it is better to make 'savepoint'
> and
> > > > > > > > 'checkpoint' the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > > level. But I'm not so sure since there is only
> > one
> > > > > > > > > > > savepoint-related
> > > > > > > > > > > > > > > option. Maybe someone else could share some
> > > thoughts
> > > > > > here.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
> > instead
> > > of
> > > > > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  Agreed. That's more accurate.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Many thanks for your suggestions!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> > > > > > > > jrlee.ljr@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for driving this. I agree that the
> > > proposed
> > > > > > > > restructuring
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > configuration options is largely positive. It
> > > will
> > > > > make
> > > > > > > > > > > > understanding
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > working with Flink configurations more
> > intuitive.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Most of the proposed changes look great.
> Just a
> > > > > > heads-up,
> > > > > > > > as Rui
> > > > > > > > > > > > Fan
> > > > > > > > > > > > > > > > mentioned, Flink currently requires that no
> > > > > > > configOption's
> > > > > > > > key be
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > prefix of another to avoid issues when we
> > > > eventually
> > > > > > > adopt
> > > > > > > > a
> > > > > > > > > > > > standard
> > > > > > > > > > > > > > YAML
> > > > > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/FLINK-29372
> > > > ).
> > > > > > > > Therefore,
> > > > > > > > > > > > it's
> > > > > > > > > > > > > > better
> > > > > > > > > > > > > > > > to change the key
> > > > > 'execution.checkpointing.local-copy'
> > > > > > > > because it
> > > > > > > > > > > > > > serves as
> > > > > > > > > > > > > > > > a prefix to the key
> > > > > > > > 'execution.checkpointing.local-copy.dir'.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > Junrui
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Rui Fan <19...@gmail.com>
> 于2023年12月25日周一
> > > > > 19:11写道:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you for driving this proposal!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Overall good for me. I have some questions
> > > about
> > > > > > these
> > > > > > > > names.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. How about
> > > execution.checkpointing.storage.type
> > > > > > > > instead of
> > > > > > > > > > > > > > > > > execution.checkpointing.storage?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It's similar to state.backend.type.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. How about
> > > > > > execution.checkpointing.local-copy.enabled
> > > > > > > > instead
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > You added a new option:
> > > > > > > > execution.checkpointing.local-copy.dir.
> > > > > > > > > > > > > > > > > IIUC, one option name shouldn't be the
> prefix
> > > of
> > > > > > other
> > > > > > > > options.
> > > > > > > > > > > > > > > > > If you add a new option
> > > > > > > > execution.checkpointing.local-copy,
> > > > > > > > > > > > > > > > > flink CI will fail directly.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir
> is a
> > > > > little
> > > > > > > > weird.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For old options: state.savepoints.dir and
> > > > > > > > > > > state.checkpoints.dir,
> > > > > > > > > > > > > > > > > the savepoint and checkpoint are the same
> > > level.
> > > > It
> > > > > > > means
> > > > > > > > > > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The new option execution.checkpointing.dir
> is
> > > > fine
> > > > > > for
> > > > > > > > me.
> > > > > > > > > > > > > > > > > However,
> > execution.checkpointing.savepoint.dir
> > > > is a
> > > > > > > > little
> > > > > > > > > > > weird.
> > > > > > > > > > > > > > > > > I don't know which name is better now. Let
> us
> > > > think
> > > > > > > > about it
> > > > > > > > > > > > more.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
> > > > instead
> > > > > of
> > > > > > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The meaning of mode is too broad. The
> > > claim-mode
> > > > > may
> > > > > > > > > > > > > > > > > be more accurate for users.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Rui
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly
> Lan <
> > > > > > > > > > > > zakelly.lan@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi devs,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'd like to start a discussion on
> FLIP-406:
> > > > > > > Reorganize
> > > > > > > > State
> > > > > > > > > > > &
> > > > > > > > > > > > > > > > > > Checkpointing & Recovery
> Configuration[1].
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Currently, the configuration options
> > > pertaining
> > > > > to
> > > > > > > > > > > > checkpointing,
> > > > > > > > > > > > > > > > > recovery,
> > > > > > > > > > > > > > > > > > and state management are primarily
> grouped
> > > > under
> > > > > > the
> > > > > > > > > > > following
> > > > > > > > > > > > > > > > prefixes:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >    - state.backend.* : configurations
> > related
> > > > to
> > > > > > > state
> > > > > > > > > > > > accessing
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >    checkpointing, as well as specific
> > options
> > > > for
> > > > > > > > individual
> > > > > > > > > > > > > state
> > > > > > > > > > > > > > > > > backends
> > > > > > > > > > > > > > > > > >    - execution.checkpointing.* :
> > > configurations
> > > > > > > > associated
> > > > > > > > > > > with
> > > > > > > > > > > > > > > > > checkpoint
> > > > > > > > > > > > > > > > > >    execution and recovery
> > > > > > > > > > > > > > > > > >    - execution.savepoint.*:
> configurations
> > > for
> > > > > > > > recovery from
> > > > > > > > > > > > > > savepoint
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > In addition, there are several individual
> > > > options
> > > > > > > such
> > > > > > > > as '
> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*' and
> > > > > > > > '*state.checkpoints.dir*'
> > > > > > > > > > > that
> > > > > > > > > > > > > fall
> > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > of these prefixes. The current
> arrangement
> > of
> > > > > these
> > > > > > > > options,
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > span
> > > > > > > > > > > > > > > > > > multiple modules, is somewhat haphazard
> and
> > > > > lacks a
> > > > > > > > > > > systematic
> > > > > > > > > > > > > > > > structure.
> > > > > > > > > > > > > > > > > > For example, the options under the
> > > > > > > > '*CheckpointingOptions*'
> > > > > > > > > > > > and '
> > > > > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are
> > related
> > > > and
> > > > > > have
> > > > > > > > no
> > > > > > > > > > > clear
> > > > > > > > > > > > > > > > boundaries
> > > > > > > > > > > > > > > > > > from the user's perspective, but there is
> > no
> > > > > > unified
> > > > > > > > prefix
> > > > > > > > > > > for
> > > > > > > > > > > > > > them.
> > > > > > > > > > > > > > > > > With
> > > > > > > > > > > > > > > > > > the upcoming release of Flink 2.0, we
> have
> > an
> > > > > > > excellent
> > > > > > > > > > > > > > opportunity to
> > > > > > > > > > > > > > > > > > overhaul and restructure the
> configurations
> > > > > related
> > > > > > > to
> > > > > > > > > > > > > > checkpointing,
> > > > > > > > > > > > > > > > > > recovery, and state management. This FLIP
> > > > > proposes
> > > > > > to
> > > > > > > > > > > > reorganize
> > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > settings, making it more coherent by
> > module,
> > > > > which
> > > > > > > > would
> > > > > > > > > > > > > > significantly
> > > > > > > > > > > > > > > > > > lower the barriers for understanding and
> > > reduce
> > > > > the
> > > > > > > > > > > development
> > > > > > > > > > > > > > costs
> > > > > > > > > > > > > > > > > > moving forward.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best,
> Hangxiang.
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Hangxiang Yu <ma...@gmail.com>.
Hi, Zakelly.
Thanks for driving this. Overall LGTM as we discussed offline.

Some comments/suggestions just came to mind:
1. Could execution.recovery also contain some other behaviors about
recovery ? e.g. restart-strategy.
2. Could we also remove some legacy configuration value ? e.g. LEGACY Mode
for execution.savepoint-restore-mode/execution.recovery.claim-mode.
3. Could the local checkpoint be cleaned
if execution.checkpointing.local-copy.enabled is true and
execution.recovery.from-local is false ? I found it's also an issue if
current local-recovery from enabled to disabled. Maybe another ticket is
needed.
4. +1 for enabling execution.checkpointing.incremental by default which is
basically default configuration in our production environment.


On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <za...@gmail.com> wrote:

> Hi Yun,
>
> Thanks for your comments!
>
>  1.  We shall not describe the configuration with its implementation for
> > 'execution.checkpointing.local-copy.*' options, for hashmap
> state-backend,
> > it would write two streams and for Rocksdb state-backend, it would use
> > hard-link for backup. Thus, I think
> > 'execution.checkpointing.local-backup.*' looks better.
>
> I agreed that we'd better name the option in user's perspective instead of
> the implementation, thus I name it as a copy of the checkpoint in the
> local disk, regardless of the way of generating it. The word 'backup' is
> also suitable for this case, so I agree to change to
> 'execution.checkpointing.local-backup.*' if no one objects.
>
>  2.  What does the 'execution.checkpointing.data-inline-threshold' mean? It
> > seems not so easy to understand.
>
> The 'execution.checkpointing.data-inline-threshold' (original one as
> 'state.storage.fs.memory-threshold') stands for the size threshold below
> which state chunks will store inline with the metadata, thus I call it
> 'data-inline-threshold'.
>
>
> Best,
> Zakelly
>
> On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com> wrote:
>
> > Hi Zakelly,
> >
> > Thanks for driving this topic. I have two concerns here:
> >
> >   1.  We shall not describe the configuration with its implementation for
> > ​'execution.checkpointing.local-copy.*' options, for hashmap
> state-backend,
> > it would write two streams and for Rocksdb state-backend, it would use
> > hard-link for backup​. Thus, I think
> > 'execution.checkpointing.local-backup.*' looks better.
> >   2.  What does the 'execution.checkpointing.data-inline-threshold' mean?
> > It seems not so easy to understand.
> >
> > Best
> > Yun Tang
> > ________________________________
> > From: Piotr Nowojski <pn...@apache.org>
> > Sent: Thursday, January 4, 2024 22:37
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing &
> > Recovery Configuration
> >
> > Hi,
> >
> > Thanks for trying to clean this up! I don't have strong opinions on the
> > topics discussed here, so generally speaking +1 from my side!
> >
> > Best,
> > Piotrek
> >
> > śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com> napisał(a):
> >
> > > Thanks for the feedback!
> > >
> > > Using the `execution.checkpointing.incremental.enabled`,
> > > and enabling it by default sounds good to me.
> > >
> > > Best,
> > > Rui
> > >
> > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <za...@gmail.com>
> > wrote:
> > >
> > > > Hi Rui,
> > > >
> > > > Thanks for your comments!
> > > >
> > > > IMO, given that the state backend can be plugably loaded (as you can
> > > > specify a state backend factory), I prefer not providing state
> backend
> > > > specified options in the framework.
> > > >
> > > > Secondly, the incremental checkpoint is actually a sharing file
> > strategy
> > > > across checkpoints, which means the state backend *could* reuse files
> > > from
> > > > previous cp but not *must* do so. When the state backend could not
> > reuse
> > > > the files, it is reasonable to fallback to a full checkpoint.
> > > >
> > > > Thus, I suggest we make it `execution.checkpointing.incremental` and
> > > enable
> > > > it by default. For those state backends not supporting this, they
> > perform
> > > > full checkpoints and print a warning to inform users. Users do not
> need
> > > to
> > > > pay special attention to different options to control this across
> > > different
> > > > state backends. This is more user-friendly in my opinion. WDYT?
> > > >
> > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com>
> wrote:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > I'm not sure whether we could add the state backend type in the
> > > > > new key name of state.backend.incremental. It means we use
> > > > > `execution.checkpointing.rocksdb-incremental` or
> > > > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > > > >
> > > > > So far, state.backend.incremental only works for rocksdb state
> > backend.
> > > > > And this feature or optimization is very valuable and huge for
> large
> > > > > state flink jobs. I believe it's enabled for most production flink
> > jobs
> > > > > with large rocksdb state.
> > > > >
> > > > > If this option isn't generic for all state backend types, I guess
> we
> > > > > can enable `execution.checkpointing.rocksdb-incremental.enabled`
> > > > > by default in Flink 2.0.
> > > > >
> > > > > But if it works for all state backends, it's hard to enable it by
> > > > default.
> > > > > Enabling great and valuable features or improvements are useful
> > > > > for users, especially a lot of new flink users. Out-of-the-box
> > options
> > > > > are good for users.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Best,
> > > > > Rui
> > > > >
> > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <zakelly.lan@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Thanks all for your comments!
> > > > > >
> > > > > > As many of you have questions about the names for boolean
> options,
> > I
> > > > > > suggest we make a naming rule for them. For now I could think of
> > > three
> > > > > > options:
> > > > > >
> > > > > > Option 1: Use enumeration options if possible. But this may cause
> > > some
> > > > > name
> > > > > > collisions or confusion as we discussed and we should unify the
> > > > statement
> > > > > > everywhere.
> > > > > > Option 2: Use boolean options and add 'enabled' as the suffix.
> > > > > > Option 3: Use boolean options and ONLY add 'enabled' when there
> are
> > > > more
> > > > > > detailed configurations under the same prefix, to prevent one
> name
> > > from
> > > > > > serving as a prefix to another.
> > > > > >
> > > > > > I am slightly inclined to Option 3, since it is more in line with
> > > > current
> > > > > > practice and friendly for existing users. Also It reduces the
> > length
> > > of
> > > > > > configuration names as much as possible. I really want to hear
> your
> > > > > > opinions.
> > > > > >
> > > > > >
> > > > > > @Xuannan
> > > > > >
> > > > > > I agree with your comments 1 and 3.
> > > > > >
> > > > > > For 2, If we decide to change the name, maybe
> > > > > > `execution.checkpointing.parallel-cleaner` is better? And as for
> > > > whether
> > > > > to
> > > > > > add 'enabled' I suggest we discuss the rule above. WDYT?
> > > > > > Thanks!
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <
> suxuannan95@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Zakelly,
> > > > > > >
> > > > > > > Thanks for driving this! The organization of the configuration
> > > option
> > > > > > > in the FLIP looks much cleaner and easier to understand. +1 to
> > the
> > > > > > > FLIP.
> > > > > > >
> > > > > > > Just some questions from me.
> > > > > > >
> > > > > > > 1. I think the change to the ConfigOptions should be put in the
> > > > > > > `Public Interface` section, instead of `Proposed Changed`, as
> > those
> > > > > > > configuration options are public interface.
> > > > > > >
> > > > > > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems
> > > confusing.
> > > > > > > It feels like it is used to choose different modes. In fact, it
> > is
> > > a
> > > > > > > boolean flag to indicate whether to enable parallel clean. How
> > > about
> > > > > > > making it `state.checkpoint.cleaner.parallel-mode.enabled`?
> > > > > > >
> > > > > > > 3. The `execution.checkpointing.write-buffer` may better be
> > > > > > > `execution.checkpointing.write-buffer-size` so that we know it
> is
> > > > > > > configuring the size of the buffer.
> > > > > > >
> > > > > > > Best,
> > > > > > > Xuannan
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <
> fredialei@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Zakelly,
> > > > > > > >
> > > > > > > > > Considering the name occupation, how about naming it as
> > > > > > > `execution.checkpointing.type`?
> > > > > > > >
> > > > > > > > `Checkpoint Type`[1,2] is used to describe aligned/unaligned
> > > > > > > > checkpoint, I am inclined to make a choice between
> > > > > > > > `execution.checkpointing.incremental` and
> > > > > > > > `execution.checkpointing.incremental.enabled`.
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > > > > > > [2]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best,
> > > > > > > > Yanfei
> > > > > > > >
> > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
> > > > > > > > >
> > > > > > > > > Hi Lijie,
> > > > > > > > >
> > > > > > > > > Thanks for the reminder! I missed this.
> > > > > > > > >
> > > > > > > > > Considering the name occupation, how about naming it as
> > > > > > > > > `execution.checkpointing.type`?
> > > > > > > > >
> > > > > > > > > Actually I think the current `execution.checkpointing.mode`
> > is
> > > > > > > confusing in
> > > > > > > > > some ways, maybe `execution.checkpointing.data-consistency`
> > is
> > > > > > better.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Zakelly
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > > > > > wangdachui9501@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Zakelly,
> > > > > > > > > >
> > > > > > > > > > >> I'm wondering if
> `execution.checkpointing.savepoint-dir`
> > > > would
> > > > > > be
> > > > > > > > > > better.
> > > > > > > > > >
> > > > > > > > > > `execution.checkpointing.dir` and
> > > > > > > `execution.checkpointing.savepoint-dir`
> > > > > > > > > > are also fine for me.
> > > > > > > > > >
> > > > > > > > > > >> So I think an enumeration option
> > > > > `execution.checkpointing.mode`
> > > > > > > which
> > > > > > > > > > can be 'full' (default) or 'incremental' would be better
> > > > > > > > > >
> > > > > > > > > > I agree with using an enumeration option. But currently
> > there
> > > > is
> > > > > > > already a
> > > > > > > > > > configuration option called
> `execution.checkpointing.mode`,
> > > > which
> > > > > > is
> > > > > > > used
> > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to
> > use
> > > > > > > another name
> > > > > > > > > > or merge these two options.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Lijie
> > > > > > > > > >
> > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
> > 11:43写道:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > Thanks all for your comments!
> > > > > > > > > > >
> > > > > > > > > > > @Yanfei
> > > > > > > > > > >
> > > > > > > > > > > > 1. For some state backends that do not support
> > > incremental
> > > > > > > checkpoint,
> > > > > > > > > > > > how does the
> execution.checkpointing.incrementaloption
> > > take
> > > > > > > effect? Or
> > > > > > > > > > > > is it better to put incremental under
> > > > > > > state.backend.xxx.incremental?
> > > > > > > > > > > >
> > > > > > > > > > > I'd rather not put the option for incremental
> checkpoint
> > > > under
> > > > > > the
> > > > > > > > > > > 'state.backend', since it is more about the
> checkpointing
> > > > > instead
> > > > > > > of
> > > > > > > > > > state
> > > > > > > > > > > accessing. Of course, the state backend may not
> > necessarily
> > > > do
> > > > > > > > > > incremental
> > > > > > > > > > > checkpoint as requested. If the state backend is not
> > > capable
> > > > of
> > > > > > > taking
> > > > > > > > > > > incremental cp, it is better to fallback to the full
> cp.
> > > > > > > > > > >
> > > > > > > > > > > 2. I'm a little worried that putting all configurations
> > > into
> > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > > > dependency
> > > > > > > > > > > > problems. Some options would be used by flink-runtime
> > > > module,
> > > > > > but
> > > > > > > > > > > > flink-runtime should not depend on
> > flink-streaming-java.
> > > > e.g.
> > > > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > > > So, I prefer to move configurations to
> > > > > `CheckpointingOptions`,
> > > > > > > WDYT?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yes, that's a very good point.  Moving to
> > > > > > > > > > > `CheckpointingOptions`(flink-core) makes sense.
> > > > > > > > > > >
> > > > > > > > > > > @Lijie
> > > > > > > > > > >
> > > > > > > > > > > How about
> > > > > > > > > > > > state.savepoints.dir ->
> > > > execution.checkpointing.savepoint.dir
> > > > > > > > > > > > state.checkpoints.dir ->
> > > > > execution.checkpointing.checkpoint.dir
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Actually, I think the `checkpointing.checkpoint` may
> > cause
> > > > some
> > > > > > > > > > confusion.
> > > > > > > > > > > But I'm ok if others agree.
> > > > > > > > > > > I'm wondering if
> `execution.checkpointing.savepoint-dir`
> > > > would
> > > > > be
> > > > > > > better.
> > > > > > > > > > > WDYT?
> > > > > > > > > > >
> > > > > > > > > > > 2. We changed the execution.checkpointing.local-copy'
> to
> > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should
> we
> > > > also
> > > > > > add
> > > > > > > > > > > "enabled"
> > > > > > > > > > > > suffix for other boolean type configuration options ?
> > For
> > > > > > > example,
> > > > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Actually, the incremental cp is something like
> choosing a
> > > > mode
> > > > > > for
> > > > > > > doing
> > > > > > > > > > > checkpoint instead of enabling a function. So I think
> an
> > > > > > > enumeration
> > > > > > > > > > option
> > > > > > > > > > > `execution.checkpointing.mode` which can be 'full'
> > > (default)
> > > > or
> > > > > > > > > > > 'incremental' would be better, WDYT?
> > > > > > > > > > > And @Rui Fan @Yanfei What do you think about this?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > > > > > > wangdachui9501@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for driving the discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > 1.
> > > > > > > > > > > > >> But I'm not so sure since there is only one
> > > > > > savepoint-related
> > > > > > > > > > option.
> > > > > > > > > > > > Maybe someone else could share some thoughts here.
> > > > > > > > > > > >
> > > > > > > > > > > > How about
> > > > > > > > > > > > state.savepoints.dir ->
> > > > execution.checkpointing.savepoint.dir
> > > > > > > > > > > > state.checkpoints.dir ->
> > > > > execution.checkpointing.checkpoint.dir
> > > > > > > > > > > >
> > > > > > > > > > > > 2. We changed the execution.checkpointing.local-copy'
> > to
> > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should
> we
> > > > also
> > > > > > add
> > > > > > > > > > > "enabled"
> > > > > > > > > > > > suffix for other boolean type configuration options ?
> > For
> > > > > > > example,
> > > > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > > > >
> > > > > > > > > > > > In this way, the naming style of configuration
> options
> > is
> > > > > > > unified, and
> > > > > > > > > > it
> > > > > > > > > > > > can avoid potential similar problems (for example, we
> > may
> > > > > need
> > > > > > > to add
> > > > > > > > > > > more
> > > > > > > > > > > > options for incremental checkpoint in the future).
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Lijie
> > > > > > > > > > > >
> > > > > > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二
> > 12:05写道:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you for creating the FLIP and starting the
> > > > > discussion.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The current arrangement of these options is indeed
> > > > somewhat
> > > > > > > > > > haphazard,
> > > > > > > > > > > > > and the new arrangement looks much better. I have
> > some
> > > > > > > questions
> > > > > > > > > > about
> > > > > > > > > > > > > the arrangement of some new configuration options:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. For some state backends that do not support
> > > > incremental
> > > > > > > > > > checkpoint,
> > > > > > > > > > > > > how does the
> > execution.checkpointing.incrementaloption
> > > > take
> > > > > > > effect?
> > > > > > > > > > Or
> > > > > > > > > > > > > is it better to put incremental under
> > > > > > > state.backend.xxx.incremental?
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. I'm a little worried that putting all
> > configurations
> > > > > into
> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > > > > dependency
> > > > > > > > > > > > > problems. Some options would be used by
> flink-runtime
> > > > > module,
> > > > > > > but
> > > > > > > > > > > > > flink-runtime should not depend on
> > > flink-streaming-java.
> > > > > e.g.
> > > > > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > > > > So, I prefer to move configurations to
> > > > > > `CheckpointingOptions`,
> > > > > > > WDYT?
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > https://issues.apache.org/jira/browse/FLINK-28286
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Yanfei
> > > > > > > > > > > > >
> > > > > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一
> > > > > 21:14写道:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Rui Fan and Junrui,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > > > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > And for other suggestions Rui proposed:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> > > > instead
> > > > > > > > > > > > > > > of execution.checkpointing.storage?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Ah, I missed something here. Actually I suggest
> we
> > > > could
> > > > > > > merge the
> > > > > > > > > > > > > current
> > > > > > > > > > > > > > 'state.checkpoints.dir' and
> > > 'state.checkpoint-storage'
> > > > > into
> > > > > > > one URI
> > > > > > > > > > > > > > configuration named
> 'execution.checkpointing.dir'.
> > > > WDYT?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a
> > little
> > > > > weird.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, I think it is better to make 'savepoint' and
> > > > > > > 'checkpoint' the
> > > > > > > > > > > same
> > > > > > > > > > > > > > level. But I'm not so sure since there is only
> one
> > > > > > > > > > savepoint-related
> > > > > > > > > > > > > > option. Maybe someone else could share some
> > thoughts
> > > > > here.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
> instead
> > of
> > > > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  Agreed. That's more accurate.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Many thanks for your suggestions!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> > > > > > > jrlee.ljr@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for driving this. I agree that the
> > proposed
> > > > > > > restructuring
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > configuration options is largely positive. It
> > will
> > > > make
> > > > > > > > > > > understanding
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > working with Flink configurations more
> intuitive.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Most of the proposed changes look great. Just a
> > > > > heads-up,
> > > > > > > as Rui
> > > > > > > > > > > Fan
> > > > > > > > > > > > > > > mentioned, Flink currently requires that no
> > > > > > configOption's
> > > > > > > key be
> > > > > > > > > > > the
> > > > > > > > > > > > > > > prefix of another to avoid issues when we
> > > eventually
> > > > > > adopt
> > > > > > > a
> > > > > > > > > > > standard
> > > > > > > > > > > > > YAML
> > > > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/FLINK-29372
> > > ).
> > > > > > > Therefore,
> > > > > > > > > > > it's
> > > > > > > > > > > > > better
> > > > > > > > > > > > > > > to change the key
> > > > 'execution.checkpointing.local-copy'
> > > > > > > because it
> > > > > > > > > > > > > serves as
> > > > > > > > > > > > > > > a prefix to the key
> > > > > > > 'execution.checkpointing.local-copy.dir'.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Junrui
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一
> > > > 19:11写道:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you for driving this proposal!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Overall good for me. I have some questions
> > about
> > > > > these
> > > > > > > names.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. How about
> > execution.checkpointing.storage.type
> > > > > > > instead of
> > > > > > > > > > > > > > > > execution.checkpointing.storage?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It's similar to state.backend.type.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. How about
> > > > > execution.checkpointing.local-copy.enabled
> > > > > > > instead
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You added a new option:
> > > > > > > execution.checkpointing.local-copy.dir.
> > > > > > > > > > > > > > > > IIUC, one option name shouldn't be the prefix
> > of
> > > > > other
> > > > > > > options.
> > > > > > > > > > > > > > > > If you add a new option
> > > > > > > execution.checkpointing.local-copy,
> > > > > > > > > > > > > > > > flink CI will fail directly.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a
> > > > little
> > > > > > > weird.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For old options: state.savepoints.dir and
> > > > > > > > > > state.checkpoints.dir,
> > > > > > > > > > > > > > > > the savepoint and checkpoint are the same
> > level.
> > > It
> > > > > > means
> > > > > > > > > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The new option execution.checkpointing.dir is
> > > fine
> > > > > for
> > > > > > > me.
> > > > > > > > > > > > > > > > However,
> execution.checkpointing.savepoint.dir
> > > is a
> > > > > > > little
> > > > > > > > > > weird.
> > > > > > > > > > > > > > > > I don't know which name is better now. Let us
> > > think
> > > > > > > about it
> > > > > > > > > > > more.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
> > > instead
> > > > of
> > > > > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The meaning of mode is too broad. The
> > claim-mode
> > > > may
> > > > > > > > > > > > > > > > be more accurate for users.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Rui
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > > > > > > > > > zakelly.lan@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi devs,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'd like to start a discussion on FLIP-406:
> > > > > > Reorganize
> > > > > > > State
> > > > > > > > > > &
> > > > > > > > > > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Currently, the configuration options
> > pertaining
> > > > to
> > > > > > > > > > > checkpointing,
> > > > > > > > > > > > > > > > recovery,
> > > > > > > > > > > > > > > > > and state management are primarily grouped
> > > under
> > > > > the
> > > > > > > > > > following
> > > > > > > > > > > > > > > prefixes:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >    - state.backend.* : configurations
> related
> > > to
> > > > > > state
> > > > > > > > > > > accessing
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >    checkpointing, as well as specific
> options
> > > for
> > > > > > > individual
> > > > > > > > > > > > state
> > > > > > > > > > > > > > > > backends
> > > > > > > > > > > > > > > > >    - execution.checkpointing.* :
> > configurations
> > > > > > > associated
> > > > > > > > > > with
> > > > > > > > > > > > > > > > checkpoint
> > > > > > > > > > > > > > > > >    execution and recovery
> > > > > > > > > > > > > > > > >    - execution.savepoint.*: configurations
> > for
> > > > > > > recovery from
> > > > > > > > > > > > > savepoint
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > In addition, there are several individual
> > > options
> > > > > > such
> > > > > > > as '
> > > > > > > > > > > > > > > > > *state.checkpoint-storage*' and
> > > > > > > '*state.checkpoints.dir*'
> > > > > > > > > > that
> > > > > > > > > > > > fall
> > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > of these prefixes. The current arrangement
> of
> > > > these
> > > > > > > options,
> > > > > > > > > > > > which
> > > > > > > > > > > > > span
> > > > > > > > > > > > > > > > > multiple modules, is somewhat haphazard and
> > > > lacks a
> > > > > > > > > > systematic
> > > > > > > > > > > > > > > structure.
> > > > > > > > > > > > > > > > > For example, the options under the
> > > > > > > '*CheckpointingOptions*'
> > > > > > > > > > > and '
> > > > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are
> related
> > > and
> > > > > have
> > > > > > > no
> > > > > > > > > > clear
> > > > > > > > > > > > > > > boundaries
> > > > > > > > > > > > > > > > > from the user's perspective, but there is
> no
> > > > > unified
> > > > > > > prefix
> > > > > > > > > > for
> > > > > > > > > > > > > them.
> > > > > > > > > > > > > > > > With
> > > > > > > > > > > > > > > > > the upcoming release of Flink 2.0, we have
> an
> > > > > > excellent
> > > > > > > > > > > > > opportunity to
> > > > > > > > > > > > > > > > > overhaul and restructure the configurations
> > > > related
> > > > > > to
> > > > > > > > > > > > > checkpointing,
> > > > > > > > > > > > > > > > > recovery, and state management. This FLIP
> > > > proposes
> > > > > to
> > > > > > > > > > > reorganize
> > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > settings, making it more coherent by
> module,
> > > > which
> > > > > > > would
> > > > > > > > > > > > > significantly
> > > > > > > > > > > > > > > > > lower the barriers for understanding and
> > reduce
> > > > the
> > > > > > > > > > development
> > > > > > > > > > > > > costs
> > > > > > > > > > > > > > > > > moving forward.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi Yun,

Thanks for your comments!

 1.  We shall not describe the configuration with its implementation for
> 'execution.checkpointing.local-copy.*' options, for hashmap state-backend,
> it would write two streams and for Rocksdb state-backend, it would use
> hard-link for backup. Thus, I think
> 'execution.checkpointing.local-backup.*' looks better.

I agreed that we'd better name the option in user's perspective instead of
the implementation, thus I name it as a copy of the checkpoint in the
local disk, regardless of the way of generating it. The word 'backup' is
also suitable for this case, so I agree to change to
'execution.checkpointing.local-backup.*' if no one objects.

 2.  What does the 'execution.checkpointing.data-inline-threshold' mean? It
> seems not so easy to understand.

The 'execution.checkpointing.data-inline-threshold' (original one as
'state.storage.fs.memory-threshold') stands for the size threshold below
which state chunks will store inline with the metadata, thus I call it
'data-inline-threshold'.


Best,
Zakelly

On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <my...@live.com> wrote:

> Hi Zakelly,
>
> Thanks for driving this topic. I have two concerns here:
>
>   1.  We shall not describe the configuration with its implementation for
> ​'execution.checkpointing.local-copy.*' options, for hashmap state-backend,
> it would write two streams and for Rocksdb state-backend, it would use
> hard-link for backup​. Thus, I think
> 'execution.checkpointing.local-backup.*' looks better.
>   2.  What does the 'execution.checkpointing.data-inline-threshold' mean?
> It seems not so easy to understand.
>
> Best
> Yun Tang
> ________________________________
> From: Piotr Nowojski <pn...@apache.org>
> Sent: Thursday, January 4, 2024 22:37
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing &
> Recovery Configuration
>
> Hi,
>
> Thanks for trying to clean this up! I don't have strong opinions on the
> topics discussed here, so generally speaking +1 from my side!
>
> Best,
> Piotrek
>
> śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com> napisał(a):
>
> > Thanks for the feedback!
> >
> > Using the `execution.checkpointing.incremental.enabled`,
> > and enabling it by default sounds good to me.
> >
> > Best,
> > Rui
> >
> > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <za...@gmail.com>
> wrote:
> >
> > > Hi Rui,
> > >
> > > Thanks for your comments!
> > >
> > > IMO, given that the state backend can be plugably loaded (as you can
> > > specify a state backend factory), I prefer not providing state backend
> > > specified options in the framework.
> > >
> > > Secondly, the incremental checkpoint is actually a sharing file
> strategy
> > > across checkpoints, which means the state backend *could* reuse files
> > from
> > > previous cp but not *must* do so. When the state backend could not
> reuse
> > > the files, it is reasonable to fallback to a full checkpoint.
> > >
> > > Thus, I suggest we make it `execution.checkpointing.incremental` and
> > enable
> > > it by default. For those state backends not supporting this, they
> perform
> > > full checkpoints and print a warning to inform users. Users do not need
> > to
> > > pay special attention to different options to control this across
> > different
> > > state backends. This is more user-friendly in my opinion. WDYT?
> > >
> > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com> wrote:
> > >
> > > > Hi Zakelly,
> > > >
> > > > I'm not sure whether we could add the state backend type in the
> > > > new key name of state.backend.incremental. It means we use
> > > > `execution.checkpointing.rocksdb-incremental` or
> > > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > > >
> > > > So far, state.backend.incremental only works for rocksdb state
> backend.
> > > > And this feature or optimization is very valuable and huge for large
> > > > state flink jobs. I believe it's enabled for most production flink
> jobs
> > > > with large rocksdb state.
> > > >
> > > > If this option isn't generic for all state backend types, I guess we
> > > > can enable `execution.checkpointing.rocksdb-incremental.enabled`
> > > > by default in Flink 2.0.
> > > >
> > > > But if it works for all state backends, it's hard to enable it by
> > > default.
> > > > Enabling great and valuable features or improvements are useful
> > > > for users, especially a lot of new flink users. Out-of-the-box
> options
> > > > are good for users.
> > > >
> > > > WDYT?
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <za...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thanks all for your comments!
> > > > >
> > > > > As many of you have questions about the names for boolean options,
> I
> > > > > suggest we make a naming rule for them. For now I could think of
> > three
> > > > > options:
> > > > >
> > > > > Option 1: Use enumeration options if possible. But this may cause
> > some
> > > > name
> > > > > collisions or confusion as we discussed and we should unify the
> > > statement
> > > > > everywhere.
> > > > > Option 2: Use boolean options and add 'enabled' as the suffix.
> > > > > Option 3: Use boolean options and ONLY add 'enabled' when there are
> > > more
> > > > > detailed configurations under the same prefix, to prevent one name
> > from
> > > > > serving as a prefix to another.
> > > > >
> > > > > I am slightly inclined to Option 3, since it is more in line with
> > > current
> > > > > practice and friendly for existing users. Also It reduces the
> length
> > of
> > > > > configuration names as much as possible. I really want to hear your
> > > > > opinions.
> > > > >
> > > > >
> > > > > @Xuannan
> > > > >
> > > > > I agree with your comments 1 and 3.
> > > > >
> > > > > For 2, If we decide to change the name, maybe
> > > > > `execution.checkpointing.parallel-cleaner` is better? And as for
> > > whether
> > > > to
> > > > > add 'enabled' I suggest we discuss the rule above. WDYT?
> > > > > Thanks!
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <suxuannan95@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > Thanks for driving this! The organization of the configuration
> > option
> > > > > > in the FLIP looks much cleaner and easier to understand. +1 to
> the
> > > > > > FLIP.
> > > > > >
> > > > > > Just some questions from me.
> > > > > >
> > > > > > 1. I think the change to the ConfigOptions should be put in the
> > > > > > `Public Interface` section, instead of `Proposed Changed`, as
> those
> > > > > > configuration options are public interface.
> > > > > >
> > > > > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems
> > confusing.
> > > > > > It feels like it is used to choose different modes. In fact, it
> is
> > a
> > > > > > boolean flag to indicate whether to enable parallel clean. How
> > about
> > > > > > making it `state.checkpoint.cleaner.parallel-mode.enabled`?
> > > > > >
> > > > > > 3. The `execution.checkpointing.write-buffer` may better be
> > > > > > `execution.checkpointing.write-buffer-size` so that we know it is
> > > > > > configuring the size of the buffer.
> > > > > >
> > > > > > Best,
> > > > > > Xuannan
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <fr...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > Hi Zakelly,
> > > > > > >
> > > > > > > > Considering the name occupation, how about naming it as
> > > > > > `execution.checkpointing.type`?
> > > > > > >
> > > > > > > `Checkpoint Type`[1,2] is used to describe aligned/unaligned
> > > > > > > checkpoint, I am inclined to make a choice between
> > > > > > > `execution.checkpointing.incremental` and
> > > > > > > `execution.checkpointing.incremental.enabled`.
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > > > > >
> > > > > > > --
> > > > > > > Best,
> > > > > > > Yanfei
> > > > > > >
> > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
> > > > > > > >
> > > > > > > > Hi Lijie,
> > > > > > > >
> > > > > > > > Thanks for the reminder! I missed this.
> > > > > > > >
> > > > > > > > Considering the name occupation, how about naming it as
> > > > > > > > `execution.checkpointing.type`?
> > > > > > > >
> > > > > > > > Actually I think the current `execution.checkpointing.mode`
> is
> > > > > > confusing in
> > > > > > > > some ways, maybe `execution.checkpointing.data-consistency`
> is
> > > > > better.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zakelly
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > > > > wangdachui9501@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Zakelly,
> > > > > > > > >
> > > > > > > > > >> I'm wondering if `execution.checkpointing.savepoint-dir`
> > > would
> > > > > be
> > > > > > > > > better.
> > > > > > > > >
> > > > > > > > > `execution.checkpointing.dir` and
> > > > > > `execution.checkpointing.savepoint-dir`
> > > > > > > > > are also fine for me.
> > > > > > > > >
> > > > > > > > > >> So I think an enumeration option
> > > > `execution.checkpointing.mode`
> > > > > > which
> > > > > > > > > can be 'full' (default) or 'incremental' would be better
> > > > > > > > >
> > > > > > > > > I agree with using an enumeration option. But currently
> there
> > > is
> > > > > > already a
> > > > > > > > > configuration option called `execution.checkpointing.mode`,
> > > which
> > > > > is
> > > > > > used
> > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to
> use
> > > > > > another name
> > > > > > > > > or merge these two options.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Lijie
> > > > > > > > >
> > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三
> 11:43写道:
> > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > Thanks all for your comments!
> > > > > > > > > >
> > > > > > > > > > @Yanfei
> > > > > > > > > >
> > > > > > > > > > > 1. For some state backends that do not support
> > incremental
> > > > > > checkpoint,
> > > > > > > > > > > how does the execution.checkpointing.incrementaloption
> > take
> > > > > > effect? Or
> > > > > > > > > > > is it better to put incremental under
> > > > > > state.backend.xxx.incremental?
> > > > > > > > > > >
> > > > > > > > > > I'd rather not put the option for incremental checkpoint
> > > under
> > > > > the
> > > > > > > > > > 'state.backend', since it is more about the checkpointing
> > > > instead
> > > > > > of
> > > > > > > > > state
> > > > > > > > > > accessing. Of course, the state backend may not
> necessarily
> > > do
> > > > > > > > > incremental
> > > > > > > > > > checkpoint as requested. If the state backend is not
> > capable
> > > of
> > > > > > taking
> > > > > > > > > > incremental cp, it is better to fallback to the full cp.
> > > > > > > > > >
> > > > > > > > > > 2. I'm a little worried that putting all configurations
> > into
> > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > > dependency
> > > > > > > > > > > problems. Some options would be used by flink-runtime
> > > module,
> > > > > but
> > > > > > > > > > > flink-runtime should not depend on
> flink-streaming-java.
> > > e.g.
> > > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > > So, I prefer to move configurations to
> > > > `CheckpointingOptions`,
> > > > > > WDYT?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yes, that's a very good point.  Moving to
> > > > > > > > > > `CheckpointingOptions`(flink-core) makes sense.
> > > > > > > > > >
> > > > > > > > > > @Lijie
> > > > > > > > > >
> > > > > > > > > > How about
> > > > > > > > > > > state.savepoints.dir ->
> > > execution.checkpointing.savepoint.dir
> > > > > > > > > > > state.checkpoints.dir ->
> > > > execution.checkpointing.checkpoint.dir
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Actually, I think the `checkpointing.checkpoint` may
> cause
> > > some
> > > > > > > > > confusion.
> > > > > > > > > > But I'm ok if others agree.
> > > > > > > > > > I'm wondering if `execution.checkpointing.savepoint-dir`
> > > would
> > > > be
> > > > > > better.
> > > > > > > > > > WDYT?
> > > > > > > > > >
> > > > > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we
> > > also
> > > > > add
> > > > > > > > > > "enabled"
> > > > > > > > > > > suffix for other boolean type configuration options ?
> For
> > > > > > example,
> > > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Actually, the incremental cp is something like choosing a
> > > mode
> > > > > for
> > > > > > doing
> > > > > > > > > > checkpoint instead of enabling a function. So I think an
> > > > > > enumeration
> > > > > > > > > option
> > > > > > > > > > `execution.checkpointing.mode` which can be 'full'
> > (default)
> > > or
> > > > > > > > > > 'incremental' would be better, WDYT?
> > > > > > > > > > And @Rui Fan @Yanfei What do you think about this?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > > > > > wangdachui9501@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for driving the discussion.
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > > >> But I'm not so sure since there is only one
> > > > > savepoint-related
> > > > > > > > > option.
> > > > > > > > > > > Maybe someone else could share some thoughts here.
> > > > > > > > > > >
> > > > > > > > > > > How about
> > > > > > > > > > > state.savepoints.dir ->
> > > execution.checkpointing.savepoint.dir
> > > > > > > > > > > state.checkpoints.dir ->
> > > > execution.checkpointing.checkpoint.dir
> > > > > > > > > > >
> > > > > > > > > > > 2. We changed the execution.checkpointing.local-copy'
> to
> > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we
> > > also
> > > > > add
> > > > > > > > > > "enabled"
> > > > > > > > > > > suffix for other boolean type configuration options ?
> For
> > > > > > example,
> > > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > > >
> > > > > > > > > > > In this way, the naming style of configuration options
> is
> > > > > > unified, and
> > > > > > > > > it
> > > > > > > > > > > can avoid potential similar problems (for example, we
> may
> > > > need
> > > > > > to add
> > > > > > > > > > more
> > > > > > > > > > > options for incremental checkpoint in the future).
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Lijie
> > > > > > > > > > >
> > > > > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二
> 12:05写道:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you for creating the FLIP and starting the
> > > > discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > The current arrangement of these options is indeed
> > > somewhat
> > > > > > > > > haphazard,
> > > > > > > > > > > > and the new arrangement looks much better. I have
> some
> > > > > > questions
> > > > > > > > > about
> > > > > > > > > > > > the arrangement of some new configuration options:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. For some state backends that do not support
> > > incremental
> > > > > > > > > checkpoint,
> > > > > > > > > > > > how does the
> execution.checkpointing.incrementaloption
> > > take
> > > > > > effect?
> > > > > > > > > Or
> > > > > > > > > > > > is it better to put incremental under
> > > > > > state.backend.xxx.incremental?
> > > > > > > > > > > >
> > > > > > > > > > > > 2. I'm a little worried that putting all
> configurations
> > > > into
> > > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > > > dependency
> > > > > > > > > > > > problems. Some options would be used by flink-runtime
> > > > module,
> > > > > > but
> > > > > > > > > > > > flink-runtime should not depend on
> > flink-streaming-java.
> > > > e.g.
> > > > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > > > So, I prefer to move configurations to
> > > > > `CheckpointingOptions`,
> > > > > > WDYT?
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> https://issues.apache.org/jira/browse/FLINK-28286
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Yanfei
> > > > > > > > > > > >
> > > > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一
> > > > 21:14写道:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Rui Fan and Junrui,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And for other suggestions Rui proposed:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> > > instead
> > > > > > > > > > > > > > of execution.checkpointing.storage?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ah, I missed something here. Actually I suggest we
> > > could
> > > > > > merge the
> > > > > > > > > > > > current
> > > > > > > > > > > > > 'state.checkpoints.dir' and
> > 'state.checkpoint-storage'
> > > > into
> > > > > > one URI
> > > > > > > > > > > > > configuration named 'execution.checkpointing.dir'.
> > > WDYT?
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a
> little
> > > > weird.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, I think it is better to make 'savepoint' and
> > > > > > 'checkpoint' the
> > > > > > > > > > same
> > > > > > > > > > > > > level. But I'm not so sure since there is only one
> > > > > > > > > savepoint-related
> > > > > > > > > > > > > option. Maybe someone else could share some
> thoughts
> > > > here.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 4. How about execution.recovery.claim-mode instead
> of
> > > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >  Agreed. That's more accurate.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Many thanks for your suggestions!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> > > > > > jrlee.ljr@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for driving this. I agree that the
> proposed
> > > > > > restructuring
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > configuration options is largely positive. It
> will
> > > make
> > > > > > > > > > understanding
> > > > > > > > > > > > and
> > > > > > > > > > > > > > working with Flink configurations more intuitive.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Most of the proposed changes look great. Just a
> > > > heads-up,
> > > > > > as Rui
> > > > > > > > > > Fan
> > > > > > > > > > > > > > mentioned, Flink currently requires that no
> > > > > configOption's
> > > > > > key be
> > > > > > > > > > the
> > > > > > > > > > > > > > prefix of another to avoid issues when we
> > eventually
> > > > > adopt
> > > > > > a
> > > > > > > > > > standard
> > > > > > > > > > > > YAML
> > > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/FLINK-29372
> > ).
> > > > > > Therefore,
> > > > > > > > > > it's
> > > > > > > > > > > > better
> > > > > > > > > > > > > > to change the key
> > > 'execution.checkpointing.local-copy'
> > > > > > because it
> > > > > > > > > > > > serves as
> > > > > > > > > > > > > > a prefix to the key
> > > > > > 'execution.checkpointing.local-copy.dir'.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Junrui
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一
> > > 19:11写道:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you for driving this proposal!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Overall good for me. I have some questions
> about
> > > > these
> > > > > > names.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. How about
> execution.checkpointing.storage.type
> > > > > > instead of
> > > > > > > > > > > > > > > execution.checkpointing.storage?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It's similar to state.backend.type.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. How about
> > > > execution.checkpointing.local-copy.enabled
> > > > > > instead
> > > > > > > > > > of
> > > > > > > > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You added a new option:
> > > > > > execution.checkpointing.local-copy.dir.
> > > > > > > > > > > > > > > IIUC, one option name shouldn't be the prefix
> of
> > > > other
> > > > > > options.
> > > > > > > > > > > > > > > If you add a new option
> > > > > > execution.checkpointing.local-copy,
> > > > > > > > > > > > > > > flink CI will fail directly.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a
> > > little
> > > > > > weird.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For old options: state.savepoints.dir and
> > > > > > > > > state.checkpoints.dir,
> > > > > > > > > > > > > > > the savepoint and checkpoint are the same
> level.
> > It
> > > > > means
> > > > > > > > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The new option execution.checkpointing.dir is
> > fine
> > > > for
> > > > > > me.
> > > > > > > > > > > > > > > However, execution.checkpointing.savepoint.dir
> > is a
> > > > > > little
> > > > > > > > > weird.
> > > > > > > > > > > > > > > I don't know which name is better now. Let us
> > think
> > > > > > about it
> > > > > > > > > > more.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
> > instead
> > > of
> > > > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The meaning of mode is too broad. The
> claim-mode
> > > may
> > > > > > > > > > > > > > > be more accurate for users.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Rui
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > > > > > > > > zakelly.lan@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi devs,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'd like to start a discussion on FLIP-406:
> > > > > Reorganize
> > > > > > State
> > > > > > > > > &
> > > > > > > > > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Currently, the configuration options
> pertaining
> > > to
> > > > > > > > > > checkpointing,
> > > > > > > > > > > > > > > recovery,
> > > > > > > > > > > > > > > > and state management are primarily grouped
> > under
> > > > the
> > > > > > > > > following
> > > > > > > > > > > > > > prefixes:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >    - state.backend.* : configurations related
> > to
> > > > > state
> > > > > > > > > > accessing
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > >    checkpointing, as well as specific options
> > for
> > > > > > individual
> > > > > > > > > > > state
> > > > > > > > > > > > > > > backends
> > > > > > > > > > > > > > > >    - execution.checkpointing.* :
> configurations
> > > > > > associated
> > > > > > > > > with
> > > > > > > > > > > > > > > checkpoint
> > > > > > > > > > > > > > > >    execution and recovery
> > > > > > > > > > > > > > > >    - execution.savepoint.*: configurations
> for
> > > > > > recovery from
> > > > > > > > > > > > savepoint
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In addition, there are several individual
> > options
> > > > > such
> > > > > > as '
> > > > > > > > > > > > > > > > *state.checkpoint-storage*' and
> > > > > > '*state.checkpoints.dir*'
> > > > > > > > > that
> > > > > > > > > > > fall
> > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > of these prefixes. The current arrangement of
> > > these
> > > > > > options,
> > > > > > > > > > > which
> > > > > > > > > > > > span
> > > > > > > > > > > > > > > > multiple modules, is somewhat haphazard and
> > > lacks a
> > > > > > > > > systematic
> > > > > > > > > > > > > > structure.
> > > > > > > > > > > > > > > > For example, the options under the
> > > > > > '*CheckpointingOptions*'
> > > > > > > > > > and '
> > > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are related
> > and
> > > > have
> > > > > > no
> > > > > > > > > clear
> > > > > > > > > > > > > > boundaries
> > > > > > > > > > > > > > > > from the user's perspective, but there is no
> > > > unified
> > > > > > prefix
> > > > > > > > > for
> > > > > > > > > > > > them.
> > > > > > > > > > > > > > > With
> > > > > > > > > > > > > > > > the upcoming release of Flink 2.0, we have an
> > > > > excellent
> > > > > > > > > > > > opportunity to
> > > > > > > > > > > > > > > > overhaul and restructure the configurations
> > > related
> > > > > to
> > > > > > > > > > > > checkpointing,
> > > > > > > > > > > > > > > > recovery, and state management. This FLIP
> > > proposes
> > > > to
> > > > > > > > > > reorganize
> > > > > > > > > > > > these
> > > > > > > > > > > > > > > > settings, making it more coherent by module,
> > > which
> > > > > > would
> > > > > > > > > > > > significantly
> > > > > > > > > > > > > > > > lower the barriers for understanding and
> reduce
> > > the
> > > > > > > > > development
> > > > > > > > > > > > costs
> > > > > > > > > > > > > > > > moving forward.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Yun Tang <my...@live.com>.
Hi Zakelly,

Thanks for driving this topic. I have two concerns here:

  1.  We shall not describe the configuration with its implementation for ​'execution.checkpointing.local-copy.*' options, for hashmap state-backend, it would write two streams and for Rocksdb state-backend, it would use hard-link for backup​. Thus, I think 'execution.checkpointing.local-backup.*' looks better.
  2.  What does the 'execution.checkpointing.data-inline-threshold' mean? It seems not so easy to understand.

Best
Yun Tang
________________________________
From: Piotr Nowojski <pn...@apache.org>
Sent: Thursday, January 4, 2024 22:37
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Hi,

Thanks for trying to clean this up! I don't have strong opinions on the
topics discussed here, so generally speaking +1 from my side!

Best,
Piotrek

śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com> napisał(a):

> Thanks for the feedback!
>
> Using the `execution.checkpointing.incremental.enabled`,
> and enabling it by default sounds good to me.
>
> Best,
> Rui
>
> On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi Rui,
> >
> > Thanks for your comments!
> >
> > IMO, given that the state backend can be plugably loaded (as you can
> > specify a state backend factory), I prefer not providing state backend
> > specified options in the framework.
> >
> > Secondly, the incremental checkpoint is actually a sharing file strategy
> > across checkpoints, which means the state backend *could* reuse files
> from
> > previous cp but not *must* do so. When the state backend could not reuse
> > the files, it is reasonable to fallback to a full checkpoint.
> >
> > Thus, I suggest we make it `execution.checkpointing.incremental` and
> enable
> > it by default. For those state backends not supporting this, they perform
> > full checkpoints and print a warning to inform users. Users do not need
> to
> > pay special attention to different options to control this across
> different
> > state backends. This is more user-friendly in my opinion. WDYT?
> >
> > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com> wrote:
> >
> > > Hi Zakelly,
> > >
> > > I'm not sure whether we could add the state backend type in the
> > > new key name of state.backend.incremental. It means we use
> > > `execution.checkpointing.rocksdb-incremental` or
> > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > >
> > > So far, state.backend.incremental only works for rocksdb state backend.
> > > And this feature or optimization is very valuable and huge for large
> > > state flink jobs. I believe it's enabled for most production flink jobs
> > > with large rocksdb state.
> > >
> > > If this option isn't generic for all state backend types, I guess we
> > > can enable `execution.checkpointing.rocksdb-incremental.enabled`
> > > by default in Flink 2.0.
> > >
> > > But if it works for all state backends, it's hard to enable it by
> > default.
> > > Enabling great and valuable features or improvements are useful
> > > for users, especially a lot of new flink users. Out-of-the-box options
> > > are good for users.
> > >
> > > WDYT?
> > >
> > > Best,
> > > Rui
> > >
> > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <za...@gmail.com>
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Thanks all for your comments!
> > > >
> > > > As many of you have questions about the names for boolean options, I
> > > > suggest we make a naming rule for them. For now I could think of
> three
> > > > options:
> > > >
> > > > Option 1: Use enumeration options if possible. But this may cause
> some
> > > name
> > > > collisions or confusion as we discussed and we should unify the
> > statement
> > > > everywhere.
> > > > Option 2: Use boolean options and add 'enabled' as the suffix.
> > > > Option 3: Use boolean options and ONLY add 'enabled' when there are
> > more
> > > > detailed configurations under the same prefix, to prevent one name
> from
> > > > serving as a prefix to another.
> > > >
> > > > I am slightly inclined to Option 3, since it is more in line with
> > current
> > > > practice and friendly for existing users. Also It reduces the length
> of
> > > > configuration names as much as possible. I really want to hear your
> > > > opinions.
> > > >
> > > >
> > > > @Xuannan
> > > >
> > > > I agree with your comments 1 and 3.
> > > >
> > > > For 2, If we decide to change the name, maybe
> > > > `execution.checkpointing.parallel-cleaner` is better? And as for
> > whether
> > > to
> > > > add 'enabled' I suggest we discuss the rule above. WDYT?
> > > > Thanks!
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <su...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > Thanks for driving this! The organization of the configuration
> option
> > > > > in the FLIP looks much cleaner and easier to understand. +1 to the
> > > > > FLIP.
> > > > >
> > > > > Just some questions from me.
> > > > >
> > > > > 1. I think the change to the ConfigOptions should be put in the
> > > > > `Public Interface` section, instead of `Proposed Changed`, as those
> > > > > configuration options are public interface.
> > > > >
> > > > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems
> confusing.
> > > > > It feels like it is used to choose different modes. In fact, it is
> a
> > > > > boolean flag to indicate whether to enable parallel clean. How
> about
> > > > > making it `state.checkpoint.cleaner.parallel-mode.enabled`?
> > > > >
> > > > > 3. The `execution.checkpointing.write-buffer` may better be
> > > > > `execution.checkpointing.write-buffer-size` so that we know it is
> > > > > configuring the size of the buffer.
> > > > >
> > > > > Best,
> > > > > Xuannan
> > > > >
> > > > >
> > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <fr...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > > Considering the name occupation, how about naming it as
> > > > > `execution.checkpointing.type`?
> > > > > >
> > > > > > `Checkpoint Type`[1,2] is used to describe aligned/unaligned
> > > > > > checkpoint, I am inclined to make a choice between
> > > > > > `execution.checkpointing.incremental` and
> > > > > > `execution.checkpointing.incremental.enabled`.
> > > > > >
> > > > > >
> > > > > > [1]
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > > > > [2]
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Yanfei
> > > > > >
> > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
> > > > > > >
> > > > > > > Hi Lijie,
> > > > > > >
> > > > > > > Thanks for the reminder! I missed this.
> > > > > > >
> > > > > > > Considering the name occupation, how about naming it as
> > > > > > > `execution.checkpointing.type`?
> > > > > > >
> > > > > > > Actually I think the current `execution.checkpointing.mode` is
> > > > > confusing in
> > > > > > > some ways, maybe `execution.checkpointing.data-consistency` is
> > > > better.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > > > wangdachui9501@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zakelly,
> > > > > > > >
> > > > > > > > >> I'm wondering if `execution.checkpointing.savepoint-dir`
> > would
> > > > be
> > > > > > > > better.
> > > > > > > >
> > > > > > > > `execution.checkpointing.dir` and
> > > > > `execution.checkpointing.savepoint-dir`
> > > > > > > > are also fine for me.
> > > > > > > >
> > > > > > > > >> So I think an enumeration option
> > > `execution.checkpointing.mode`
> > > > > which
> > > > > > > > can be 'full' (default) or 'incremental' would be better
> > > > > > > >
> > > > > > > > I agree with using an enumeration option. But currently there
> > is
> > > > > already a
> > > > > > > > configuration option called `execution.checkpointing.mode`,
> > which
> > > > is
> > > > > used
> > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use
> > > > > another name
> > > > > > > > or merge these two options.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Lijie
> > > > > > > >
> > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > Thanks all for your comments!
> > > > > > > > >
> > > > > > > > > @Yanfei
> > > > > > > > >
> > > > > > > > > > 1. For some state backends that do not support
> incremental
> > > > > checkpoint,
> > > > > > > > > > how does the execution.checkpointing.incrementaloption
> take
> > > > > effect? Or
> > > > > > > > > > is it better to put incremental under
> > > > > state.backend.xxx.incremental?
> > > > > > > > > >
> > > > > > > > > I'd rather not put the option for incremental checkpoint
> > under
> > > > the
> > > > > > > > > 'state.backend', since it is more about the checkpointing
> > > instead
> > > > > of
> > > > > > > > state
> > > > > > > > > accessing. Of course, the state backend may not necessarily
> > do
> > > > > > > > incremental
> > > > > > > > > checkpoint as requested. If the state backend is not
> capable
> > of
> > > > > taking
> > > > > > > > > incremental cp, it is better to fallback to the full cp.
> > > > > > > > >
> > > > > > > > > 2. I'm a little worried that putting all configurations
> into
> > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > dependency
> > > > > > > > > > problems. Some options would be used by flink-runtime
> > module,
> > > > but
> > > > > > > > > > flink-runtime should not depend on flink-streaming-java.
> > e.g.
> > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > So, I prefer to move configurations to
> > > `CheckpointingOptions`,
> > > > > WDYT?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, that's a very good point.  Moving to
> > > > > > > > > `CheckpointingOptions`(flink-core) makes sense.
> > > > > > > > >
> > > > > > > > > @Lijie
> > > > > > > > >
> > > > > > > > > How about
> > > > > > > > > > state.savepoints.dir ->
> > execution.checkpointing.savepoint.dir
> > > > > > > > > > state.checkpoints.dir ->
> > > execution.checkpointing.checkpoint.dir
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Actually, I think the `checkpointing.checkpoint` may cause
> > some
> > > > > > > > confusion.
> > > > > > > > > But I'm ok if others agree.
> > > > > > > > > I'm wondering if `execution.checkpointing.savepoint-dir`
> > would
> > > be
> > > > > better.
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we
> > also
> > > > add
> > > > > > > > > "enabled"
> > > > > > > > > > suffix for other boolean type configuration options ? For
> > > > > example,
> > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Actually, the incremental cp is something like choosing a
> > mode
> > > > for
> > > > > doing
> > > > > > > > > checkpoint instead of enabling a function. So I think an
> > > > > enumeration
> > > > > > > > option
> > > > > > > > > `execution.checkpointing.mode` which can be 'full'
> (default)
> > or
> > > > > > > > > 'incremental' would be better, WDYT?
> > > > > > > > > And @Rui Fan @Yanfei What do you think about this?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > > > > wangdachui9501@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Zakelly,
> > > > > > > > > >
> > > > > > > > > > Thanks for driving the discussion.
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > > >> But I'm not so sure since there is only one
> > > > savepoint-related
> > > > > > > > option.
> > > > > > > > > > Maybe someone else could share some thoughts here.
> > > > > > > > > >
> > > > > > > > > > How about
> > > > > > > > > > state.savepoints.dir ->
> > execution.checkpointing.savepoint.dir
> > > > > > > > > > state.checkpoints.dir ->
> > > execution.checkpointing.checkpoint.dir
> > > > > > > > > >
> > > > > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we
> > also
> > > > add
> > > > > > > > > "enabled"
> > > > > > > > > > suffix for other boolean type configuration options ? For
> > > > > example,
> > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > >
> > > > > > > > > > In this way, the naming style of configuration options is
> > > > > unified, and
> > > > > > > > it
> > > > > > > > > > can avoid potential similar problems (for example, we may
> > > need
> > > > > to add
> > > > > > > > > more
> > > > > > > > > > options for incremental checkpoint in the future).
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Lijie
> > > > > > > > > >
> > > > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> > > > > > > > > >
> > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > >
> > > > > > > > > > > Thank you for creating the FLIP and starting the
> > > discussion.
> > > > > > > > > > >
> > > > > > > > > > > The current arrangement of these options is indeed
> > somewhat
> > > > > > > > haphazard,
> > > > > > > > > > > and the new arrangement looks much better. I have some
> > > > > questions
> > > > > > > > about
> > > > > > > > > > > the arrangement of some new configuration options:
> > > > > > > > > > >
> > > > > > > > > > > 1. For some state backends that do not support
> > incremental
> > > > > > > > checkpoint,
> > > > > > > > > > > how does the execution.checkpointing.incrementaloption
> > take
> > > > > effect?
> > > > > > > > Or
> > > > > > > > > > > is it better to put incremental under
> > > > > state.backend.xxx.incremental?
> > > > > > > > > > >
> > > > > > > > > > > 2. I'm a little worried that putting all configurations
> > > into
> > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > > dependency
> > > > > > > > > > > problems. Some options would be used by flink-runtime
> > > module,
> > > > > but
> > > > > > > > > > > flink-runtime should not depend on
> flink-streaming-java.
> > > e.g.
> > > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > > So, I prefer to move configurations to
> > > > `CheckpointingOptions`,
> > > > > WDYT?
> > > > > > > > > > >
> > > > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Best,
> > > > > > > > > > > Yanfei
> > > > > > > > > > >
> > > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一
> > > 21:14写道:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Rui Fan and Junrui,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > > > > > > >
> > > > > > > > > > > > And for other suggestions Rui proposed:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> > instead
> > > > > > > > > > > > > of execution.checkpointing.storage?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Ah, I missed something here. Actually I suggest we
> > could
> > > > > merge the
> > > > > > > > > > > current
> > > > > > > > > > > > 'state.checkpoints.dir' and
> 'state.checkpoint-storage'
> > > into
> > > > > one URI
> > > > > > > > > > > > configuration named 'execution.checkpointing.dir'.
> > WDYT?
> > > > > > > > > > > >
> > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a little
> > > weird.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, I think it is better to make 'savepoint' and
> > > > > 'checkpoint' the
> > > > > > > > > same
> > > > > > > > > > > > level. But I'm not so sure since there is only one
> > > > > > > > savepoint-related
> > > > > > > > > > > > option. Maybe someone else could share some thoughts
> > > here.
> > > > > > > > > > > >
> > > > > > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >  Agreed. That's more accurate.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Many thanks for your suggestions!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Zakelly
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> > > > > jrlee.ljr@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for driving this. I agree that the proposed
> > > > > restructuring
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > configuration options is largely positive. It will
> > make
> > > > > > > > > understanding
> > > > > > > > > > > and
> > > > > > > > > > > > > working with Flink configurations more intuitive.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Most of the proposed changes look great. Just a
> > > heads-up,
> > > > > as Rui
> > > > > > > > > Fan
> > > > > > > > > > > > > mentioned, Flink currently requires that no
> > > > configOption's
> > > > > key be
> > > > > > > > > the
> > > > > > > > > > > > > prefix of another to avoid issues when we
> eventually
> > > > adopt
> > > > > a
> > > > > > > > > standard
> > > > > > > > > > > YAML
> > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-29372
> ).
> > > > > Therefore,
> > > > > > > > > it's
> > > > > > > > > > > better
> > > > > > > > > > > > > to change the key
> > 'execution.checkpointing.local-copy'
> > > > > because it
> > > > > > > > > > > serves as
> > > > > > > > > > > > > a prefix to the key
> > > > > 'execution.checkpointing.local-copy.dir'.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Junrui
> > > > > > > > > > > > >
> > > > > > > > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一
> > 19:11写道:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for driving this proposal!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Overall good for me. I have some questions about
> > > these
> > > > > names.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> > > > > instead of
> > > > > > > > > > > > > > execution.checkpointing.storage?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It's similar to state.backend.type.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. How about
> > > execution.checkpointing.local-copy.enabled
> > > > > instead
> > > > > > > > > of
> > > > > > > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You added a new option:
> > > > > execution.checkpointing.local-copy.dir.
> > > > > > > > > > > > > > IIUC, one option name shouldn't be the prefix of
> > > other
> > > > > options.
> > > > > > > > > > > > > > If you add a new option
> > > > > execution.checkpointing.local-copy,
> > > > > > > > > > > > > > flink CI will fail directly.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a
> > little
> > > > > weird.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For old options: state.savepoints.dir and
> > > > > > > > state.checkpoints.dir,
> > > > > > > > > > > > > > the savepoint and checkpoint are the same level.
> It
> > > > means
> > > > > > > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The new option execution.checkpointing.dir is
> fine
> > > for
> > > > > me.
> > > > > > > > > > > > > > However, execution.checkpointing.savepoint.dir
> is a
> > > > > little
> > > > > > > > weird.
> > > > > > > > > > > > > > I don't know which name is better now. Let us
> think
> > > > > about it
> > > > > > > > > more.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
> instead
> > of
> > > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The meaning of mode is too broad. The claim-mode
> > may
> > > > > > > > > > > > > > be more accurate for users.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Rui
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > > > > > > > zakelly.lan@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi devs,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd like to start a discussion on FLIP-406:
> > > > Reorganize
> > > > > State
> > > > > > > > &
> > > > > > > > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Currently, the configuration options pertaining
> > to
> > > > > > > > > checkpointing,
> > > > > > > > > > > > > > recovery,
> > > > > > > > > > > > > > > and state management are primarily grouped
> under
> > > the
> > > > > > > > following
> > > > > > > > > > > > > prefixes:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >    - state.backend.* : configurations related
> to
> > > > state
> > > > > > > > > accessing
> > > > > > > > > > > and
> > > > > > > > > > > > > > >    checkpointing, as well as specific options
> for
> > > > > individual
> > > > > > > > > > state
> > > > > > > > > > > > > > backends
> > > > > > > > > > > > > > >    - execution.checkpointing.* : configurations
> > > > > associated
> > > > > > > > with
> > > > > > > > > > > > > > checkpoint
> > > > > > > > > > > > > > >    execution and recovery
> > > > > > > > > > > > > > >    - execution.savepoint.*: configurations for
> > > > > recovery from
> > > > > > > > > > > savepoint
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In addition, there are several individual
> options
> > > > such
> > > > > as '
> > > > > > > > > > > > > > > *state.checkpoint-storage*' and
> > > > > '*state.checkpoints.dir*'
> > > > > > > > that
> > > > > > > > > > fall
> > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > of these prefixes. The current arrangement of
> > these
> > > > > options,
> > > > > > > > > > which
> > > > > > > > > > > span
> > > > > > > > > > > > > > > multiple modules, is somewhat haphazard and
> > lacks a
> > > > > > > > systematic
> > > > > > > > > > > > > structure.
> > > > > > > > > > > > > > > For example, the options under the
> > > > > '*CheckpointingOptions*'
> > > > > > > > > and '
> > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are related
> and
> > > have
> > > > > no
> > > > > > > > clear
> > > > > > > > > > > > > boundaries
> > > > > > > > > > > > > > > from the user's perspective, but there is no
> > > unified
> > > > > prefix
> > > > > > > > for
> > > > > > > > > > > them.
> > > > > > > > > > > > > > With
> > > > > > > > > > > > > > > the upcoming release of Flink 2.0, we have an
> > > > excellent
> > > > > > > > > > > opportunity to
> > > > > > > > > > > > > > > overhaul and restructure the configurations
> > related
> > > > to
> > > > > > > > > > > checkpointing,
> > > > > > > > > > > > > > > recovery, and state management. This FLIP
> > proposes
> > > to
> > > > > > > > > reorganize
> > > > > > > > > > > these
> > > > > > > > > > > > > > > settings, making it more coherent by module,
> > which
> > > > > would
> > > > > > > > > > > significantly
> > > > > > > > > > > > > > > lower the barriers for understanding and reduce
> > the
> > > > > > > > development
> > > > > > > > > > > costs
> > > > > > > > > > > > > > > moving forward.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Piotr Nowojski <pn...@apache.org>.
Hi,

Thanks for trying to clean this up! I don't have strong opinions on the
topics discussed here, so generally speaking +1 from my side!

Best,
Piotrek

śr., 3 sty 2024 o 04:16 Rui Fan <19...@gmail.com> napisał(a):

> Thanks for the feedback!
>
> Using the `execution.checkpointing.incremental.enabled`,
> and enabling it by default sounds good to me.
>
> Best,
> Rui
>
> On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi Rui,
> >
> > Thanks for your comments!
> >
> > IMO, given that the state backend can be plugably loaded (as you can
> > specify a state backend factory), I prefer not providing state backend
> > specified options in the framework.
> >
> > Secondly, the incremental checkpoint is actually a sharing file strategy
> > across checkpoints, which means the state backend *could* reuse files
> from
> > previous cp but not *must* do so. When the state backend could not reuse
> > the files, it is reasonable to fallback to a full checkpoint.
> >
> > Thus, I suggest we make it `execution.checkpointing.incremental` and
> enable
> > it by default. For those state backends not supporting this, they perform
> > full checkpoints and print a warning to inform users. Users do not need
> to
> > pay special attention to different options to control this across
> different
> > state backends. This is more user-friendly in my opinion. WDYT?
> >
> > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com> wrote:
> >
> > > Hi Zakelly,
> > >
> > > I'm not sure whether we could add the state backend type in the
> > > new key name of state.backend.incremental. It means we use
> > > `execution.checkpointing.rocksdb-incremental` or
> > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > >
> > > So far, state.backend.incremental only works for rocksdb state backend.
> > > And this feature or optimization is very valuable and huge for large
> > > state flink jobs. I believe it's enabled for most production flink jobs
> > > with large rocksdb state.
> > >
> > > If this option isn't generic for all state backend types, I guess we
> > > can enable `execution.checkpointing.rocksdb-incremental.enabled`
> > > by default in Flink 2.0.
> > >
> > > But if it works for all state backends, it's hard to enable it by
> > default.
> > > Enabling great and valuable features or improvements are useful
> > > for users, especially a lot of new flink users. Out-of-the-box options
> > > are good for users.
> > >
> > > WDYT?
> > >
> > > Best,
> > > Rui
> > >
> > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <za...@gmail.com>
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Thanks all for your comments!
> > > >
> > > > As many of you have questions about the names for boolean options, I
> > > > suggest we make a naming rule for them. For now I could think of
> three
> > > > options:
> > > >
> > > > Option 1: Use enumeration options if possible. But this may cause
> some
> > > name
> > > > collisions or confusion as we discussed and we should unify the
> > statement
> > > > everywhere.
> > > > Option 2: Use boolean options and add 'enabled' as the suffix.
> > > > Option 3: Use boolean options and ONLY add 'enabled' when there are
> > more
> > > > detailed configurations under the same prefix, to prevent one name
> from
> > > > serving as a prefix to another.
> > > >
> > > > I am slightly inclined to Option 3, since it is more in line with
> > current
> > > > practice and friendly for existing users. Also It reduces the length
> of
> > > > configuration names as much as possible. I really want to hear your
> > > > opinions.
> > > >
> > > >
> > > > @Xuannan
> > > >
> > > > I agree with your comments 1 and 3.
> > > >
> > > > For 2, If we decide to change the name, maybe
> > > > `execution.checkpointing.parallel-cleaner` is better? And as for
> > whether
> > > to
> > > > add 'enabled' I suggest we discuss the rule above. WDYT?
> > > > Thanks!
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <su...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > Thanks for driving this! The organization of the configuration
> option
> > > > > in the FLIP looks much cleaner and easier to understand. +1 to the
> > > > > FLIP.
> > > > >
> > > > > Just some questions from me.
> > > > >
> > > > > 1. I think the change to the ConfigOptions should be put in the
> > > > > `Public Interface` section, instead of `Proposed Changed`, as those
> > > > > configuration options are public interface.
> > > > >
> > > > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems
> confusing.
> > > > > It feels like it is used to choose different modes. In fact, it is
> a
> > > > > boolean flag to indicate whether to enable parallel clean. How
> about
> > > > > making it `state.checkpoint.cleaner.parallel-mode.enabled`?
> > > > >
> > > > > 3. The `execution.checkpointing.write-buffer` may better be
> > > > > `execution.checkpointing.write-buffer-size` so that we know it is
> > > > > configuring the size of the buffer.
> > > > >
> > > > > Best,
> > > > > Xuannan
> > > > >
> > > > >
> > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <fr...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > > Considering the name occupation, how about naming it as
> > > > > `execution.checkpointing.type`?
> > > > > >
> > > > > > `Checkpoint Type`[1,2] is used to describe aligned/unaligned
> > > > > > checkpoint, I am inclined to make a choice between
> > > > > > `execution.checkpointing.incremental` and
> > > > > > `execution.checkpointing.incremental.enabled`.
> > > > > >
> > > > > >
> > > > > > [1]
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > > > > [2]
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Yanfei
> > > > > >
> > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
> > > > > > >
> > > > > > > Hi Lijie,
> > > > > > >
> > > > > > > Thanks for the reminder! I missed this.
> > > > > > >
> > > > > > > Considering the name occupation, how about naming it as
> > > > > > > `execution.checkpointing.type`?
> > > > > > >
> > > > > > > Actually I think the current `execution.checkpointing.mode` is
> > > > > confusing in
> > > > > > > some ways, maybe `execution.checkpointing.data-consistency` is
> > > > better.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > > > wangdachui9501@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zakelly,
> > > > > > > >
> > > > > > > > >> I'm wondering if `execution.checkpointing.savepoint-dir`
> > would
> > > > be
> > > > > > > > better.
> > > > > > > >
> > > > > > > > `execution.checkpointing.dir` and
> > > > > `execution.checkpointing.savepoint-dir`
> > > > > > > > are also fine for me.
> > > > > > > >
> > > > > > > > >> So I think an enumeration option
> > > `execution.checkpointing.mode`
> > > > > which
> > > > > > > > can be 'full' (default) or 'incremental' would be better
> > > > > > > >
> > > > > > > > I agree with using an enumeration option. But currently there
> > is
> > > > > already a
> > > > > > > > configuration option called `execution.checkpointing.mode`,
> > which
> > > > is
> > > > > used
> > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use
> > > > > another name
> > > > > > > > or merge these two options.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Lijie
> > > > > > > >
> > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > Thanks all for your comments!
> > > > > > > > >
> > > > > > > > > @Yanfei
> > > > > > > > >
> > > > > > > > > > 1. For some state backends that do not support
> incremental
> > > > > checkpoint,
> > > > > > > > > > how does the execution.checkpointing.incrementaloption
> take
> > > > > effect? Or
> > > > > > > > > > is it better to put incremental under
> > > > > state.backend.xxx.incremental?
> > > > > > > > > >
> > > > > > > > > I'd rather not put the option for incremental checkpoint
> > under
> > > > the
> > > > > > > > > 'state.backend', since it is more about the checkpointing
> > > instead
> > > > > of
> > > > > > > > state
> > > > > > > > > accessing. Of course, the state backend may not necessarily
> > do
> > > > > > > > incremental
> > > > > > > > > checkpoint as requested. If the state backend is not
> capable
> > of
> > > > > taking
> > > > > > > > > incremental cp, it is better to fallback to the full cp.
> > > > > > > > >
> > > > > > > > > 2. I'm a little worried that putting all configurations
> into
> > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > dependency
> > > > > > > > > > problems. Some options would be used by flink-runtime
> > module,
> > > > but
> > > > > > > > > > flink-runtime should not depend on flink-streaming-java.
> > e.g.
> > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > So, I prefer to move configurations to
> > > `CheckpointingOptions`,
> > > > > WDYT?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, that's a very good point.  Moving to
> > > > > > > > > `CheckpointingOptions`(flink-core) makes sense.
> > > > > > > > >
> > > > > > > > > @Lijie
> > > > > > > > >
> > > > > > > > > How about
> > > > > > > > > > state.savepoints.dir ->
> > execution.checkpointing.savepoint.dir
> > > > > > > > > > state.checkpoints.dir ->
> > > execution.checkpointing.checkpoint.dir
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Actually, I think the `checkpointing.checkpoint` may cause
> > some
> > > > > > > > confusion.
> > > > > > > > > But I'm ok if others agree.
> > > > > > > > > I'm wondering if `execution.checkpointing.savepoint-dir`
> > would
> > > be
> > > > > better.
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we
> > also
> > > > add
> > > > > > > > > "enabled"
> > > > > > > > > > suffix for other boolean type configuration options ? For
> > > > > example,
> > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Actually, the incremental cp is something like choosing a
> > mode
> > > > for
> > > > > doing
> > > > > > > > > checkpoint instead of enabling a function. So I think an
> > > > > enumeration
> > > > > > > > option
> > > > > > > > > `execution.checkpointing.mode` which can be 'full'
> (default)
> > or
> > > > > > > > > 'incremental' would be better, WDYT?
> > > > > > > > > And @Rui Fan @Yanfei What do you think about this?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > > > > wangdachui9501@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Zakelly,
> > > > > > > > > >
> > > > > > > > > > Thanks for driving the discussion.
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > > >> But I'm not so sure since there is only one
> > > > savepoint-related
> > > > > > > > option.
> > > > > > > > > > Maybe someone else could share some thoughts here.
> > > > > > > > > >
> > > > > > > > > > How about
> > > > > > > > > > state.savepoints.dir ->
> > execution.checkpointing.savepoint.dir
> > > > > > > > > > state.checkpoints.dir ->
> > > execution.checkpointing.checkpoint.dir
> > > > > > > > > >
> > > > > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we
> > also
> > > > add
> > > > > > > > > "enabled"
> > > > > > > > > > suffix for other boolean type configuration options ? For
> > > > > example,
> > > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > > >
> > > > > > > > > > In this way, the naming style of configuration options is
> > > > > unified, and
> > > > > > > > it
> > > > > > > > > > can avoid potential similar problems (for example, we may
> > > need
> > > > > to add
> > > > > > > > > more
> > > > > > > > > > options for incremental checkpoint in the future).
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Lijie
> > > > > > > > > >
> > > > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> > > > > > > > > >
> > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > >
> > > > > > > > > > > Thank you for creating the FLIP and starting the
> > > discussion.
> > > > > > > > > > >
> > > > > > > > > > > The current arrangement of these options is indeed
> > somewhat
> > > > > > > > haphazard,
> > > > > > > > > > > and the new arrangement looks much better. I have some
> > > > > questions
> > > > > > > > about
> > > > > > > > > > > the arrangement of some new configuration options:
> > > > > > > > > > >
> > > > > > > > > > > 1. For some state backends that do not support
> > incremental
> > > > > > > > checkpoint,
> > > > > > > > > > > how does the execution.checkpointing.incrementaloption
> > take
> > > > > effect?
> > > > > > > > Or
> > > > > > > > > > > is it better to put incremental under
> > > > > state.backend.xxx.incremental?
> > > > > > > > > > >
> > > > > > > > > > > 2. I'm a little worried that putting all configurations
> > > into
> > > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > > dependency
> > > > > > > > > > > problems. Some options would be used by flink-runtime
> > > module,
> > > > > but
> > > > > > > > > > > flink-runtime should not depend on
> flink-streaming-java.
> > > e.g.
> > > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > > So, I prefer to move configurations to
> > > > `CheckpointingOptions`,
> > > > > WDYT?
> > > > > > > > > > >
> > > > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Best,
> > > > > > > > > > > Yanfei
> > > > > > > > > > >
> > > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一
> > > 21:14写道:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Rui Fan and Junrui,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > > > > > > >
> > > > > > > > > > > > And for other suggestions Rui proposed:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> > instead
> > > > > > > > > > > > > of execution.checkpointing.storage?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Ah, I missed something here. Actually I suggest we
> > could
> > > > > merge the
> > > > > > > > > > > current
> > > > > > > > > > > > 'state.checkpoints.dir' and
> 'state.checkpoint-storage'
> > > into
> > > > > one URI
> > > > > > > > > > > > configuration named 'execution.checkpointing.dir'.
> > WDYT?
> > > > > > > > > > > >
> > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a little
> > > weird.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, I think it is better to make 'savepoint' and
> > > > > 'checkpoint' the
> > > > > > > > > same
> > > > > > > > > > > > level. But I'm not so sure since there is only one
> > > > > > > > savepoint-related
> > > > > > > > > > > > option. Maybe someone else could share some thoughts
> > > here.
> > > > > > > > > > > >
> > > > > > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >  Agreed. That's more accurate.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Many thanks for your suggestions!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Zakelly
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> > > > > jrlee.ljr@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for driving this. I agree that the proposed
> > > > > restructuring
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > configuration options is largely positive. It will
> > make
> > > > > > > > > understanding
> > > > > > > > > > > and
> > > > > > > > > > > > > working with Flink configurations more intuitive.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Most of the proposed changes look great. Just a
> > > heads-up,
> > > > > as Rui
> > > > > > > > > Fan
> > > > > > > > > > > > > mentioned, Flink currently requires that no
> > > > configOption's
> > > > > key be
> > > > > > > > > the
> > > > > > > > > > > > > prefix of another to avoid issues when we
> eventually
> > > > adopt
> > > > > a
> > > > > > > > > standard
> > > > > > > > > > > YAML
> > > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-29372
> ).
> > > > > Therefore,
> > > > > > > > > it's
> > > > > > > > > > > better
> > > > > > > > > > > > > to change the key
> > 'execution.checkpointing.local-copy'
> > > > > because it
> > > > > > > > > > > serves as
> > > > > > > > > > > > > a prefix to the key
> > > > > 'execution.checkpointing.local-copy.dir'.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Junrui
> > > > > > > > > > > > >
> > > > > > > > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一
> > 19:11写道:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for driving this proposal!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Overall good for me. I have some questions about
> > > these
> > > > > names.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> > > > > instead of
> > > > > > > > > > > > > > execution.checkpointing.storage?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It's similar to state.backend.type.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. How about
> > > execution.checkpointing.local-copy.enabled
> > > > > instead
> > > > > > > > > of
> > > > > > > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You added a new option:
> > > > > execution.checkpointing.local-copy.dir.
> > > > > > > > > > > > > > IIUC, one option name shouldn't be the prefix of
> > > other
> > > > > options.
> > > > > > > > > > > > > > If you add a new option
> > > > > execution.checkpointing.local-copy,
> > > > > > > > > > > > > > flink CI will fail directly.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a
> > little
> > > > > weird.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For old options: state.savepoints.dir and
> > > > > > > > state.checkpoints.dir,
> > > > > > > > > > > > > > the savepoint and checkpoint are the same level.
> It
> > > > means
> > > > > > > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The new option execution.checkpointing.dir is
> fine
> > > for
> > > > > me.
> > > > > > > > > > > > > > However, execution.checkpointing.savepoint.dir
> is a
> > > > > little
> > > > > > > > weird.
> > > > > > > > > > > > > > I don't know which name is better now. Let us
> think
> > > > > about it
> > > > > > > > > more.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 4. How about execution.recovery.claim-mode
> instead
> > of
> > > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The meaning of mode is too broad. The claim-mode
> > may
> > > > > > > > > > > > > > be more accurate for users.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Rui
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > > > > > > > zakelly.lan@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi devs,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd like to start a discussion on FLIP-406:
> > > > Reorganize
> > > > > State
> > > > > > > > &
> > > > > > > > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Currently, the configuration options pertaining
> > to
> > > > > > > > > checkpointing,
> > > > > > > > > > > > > > recovery,
> > > > > > > > > > > > > > > and state management are primarily grouped
> under
> > > the
> > > > > > > > following
> > > > > > > > > > > > > prefixes:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >    - state.backend.* : configurations related
> to
> > > > state
> > > > > > > > > accessing
> > > > > > > > > > > and
> > > > > > > > > > > > > > >    checkpointing, as well as specific options
> for
> > > > > individual
> > > > > > > > > > state
> > > > > > > > > > > > > > backends
> > > > > > > > > > > > > > >    - execution.checkpointing.* : configurations
> > > > > associated
> > > > > > > > with
> > > > > > > > > > > > > > checkpoint
> > > > > > > > > > > > > > >    execution and recovery
> > > > > > > > > > > > > > >    - execution.savepoint.*: configurations for
> > > > > recovery from
> > > > > > > > > > > savepoint
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In addition, there are several individual
> options
> > > > such
> > > > > as '
> > > > > > > > > > > > > > > *state.checkpoint-storage*' and
> > > > > '*state.checkpoints.dir*'
> > > > > > > > that
> > > > > > > > > > fall
> > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > of these prefixes. The current arrangement of
> > these
> > > > > options,
> > > > > > > > > > which
> > > > > > > > > > > span
> > > > > > > > > > > > > > > multiple modules, is somewhat haphazard and
> > lacks a
> > > > > > > > systematic
> > > > > > > > > > > > > structure.
> > > > > > > > > > > > > > > For example, the options under the
> > > > > '*CheckpointingOptions*'
> > > > > > > > > and '
> > > > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are related
> and
> > > have
> > > > > no
> > > > > > > > clear
> > > > > > > > > > > > > boundaries
> > > > > > > > > > > > > > > from the user's perspective, but there is no
> > > unified
> > > > > prefix
> > > > > > > > for
> > > > > > > > > > > them.
> > > > > > > > > > > > > > With
> > > > > > > > > > > > > > > the upcoming release of Flink 2.0, we have an
> > > > excellent
> > > > > > > > > > > opportunity to
> > > > > > > > > > > > > > > overhaul and restructure the configurations
> > related
> > > > to
> > > > > > > > > > > checkpointing,
> > > > > > > > > > > > > > > recovery, and state management. This FLIP
> > proposes
> > > to
> > > > > > > > > reorganize
> > > > > > > > > > > these
> > > > > > > > > > > > > > > settings, making it more coherent by module,
> > which
> > > > > would
> > > > > > > > > > > significantly
> > > > > > > > > > > > > > > lower the barriers for understanding and reduce
> > the
> > > > > > > > development
> > > > > > > > > > > costs
> > > > > > > > > > > > > > > moving forward.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Rui Fan <19...@gmail.com>.
Thanks for the feedback!

Using the `execution.checkpointing.incremental.enabled`,
and enabling it by default sounds good to me.

Best,
Rui

On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan <za...@gmail.com> wrote:

> Hi Rui,
>
> Thanks for your comments!
>
> IMO, given that the state backend can be plugably loaded (as you can
> specify a state backend factory), I prefer not providing state backend
> specified options in the framework.
>
> Secondly, the incremental checkpoint is actually a sharing file strategy
> across checkpoints, which means the state backend *could* reuse files from
> previous cp but not *must* do so. When the state backend could not reuse
> the files, it is reasonable to fallback to a full checkpoint.
>
> Thus, I suggest we make it `execution.checkpointing.incremental` and enable
> it by default. For those state backends not supporting this, they perform
> full checkpoints and print a warning to inform users. Users do not need to
> pay special attention to different options to control this across different
> state backends. This is more user-friendly in my opinion. WDYT?
>
> On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com> wrote:
>
> > Hi Zakelly,
> >
> > I'm not sure whether we could add the state backend type in the
> > new key name of state.backend.incremental. It means we use
> > `execution.checkpointing.rocksdb-incremental` or
> > `execution.checkpointing.rocksdb-incremental.enabled`.
> >
> > So far, state.backend.incremental only works for rocksdb state backend.
> > And this feature or optimization is very valuable and huge for large
> > state flink jobs. I believe it's enabled for most production flink jobs
> > with large rocksdb state.
> >
> > If this option isn't generic for all state backend types, I guess we
> > can enable `execution.checkpointing.rocksdb-incremental.enabled`
> > by default in Flink 2.0.
> >
> > But if it works for all state backends, it's hard to enable it by
> default.
> > Enabling great and valuable features or improvements are useful
> > for users, especially a lot of new flink users. Out-of-the-box options
> > are good for users.
> >
> > WDYT?
> >
> > Best,
> > Rui
> >
> > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <za...@gmail.com>
> wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks all for your comments!
> > >
> > > As many of you have questions about the names for boolean options, I
> > > suggest we make a naming rule for them. For now I could think of three
> > > options:
> > >
> > > Option 1: Use enumeration options if possible. But this may cause some
> > name
> > > collisions or confusion as we discussed and we should unify the
> statement
> > > everywhere.
> > > Option 2: Use boolean options and add 'enabled' as the suffix.
> > > Option 3: Use boolean options and ONLY add 'enabled' when there are
> more
> > > detailed configurations under the same prefix, to prevent one name from
> > > serving as a prefix to another.
> > >
> > > I am slightly inclined to Option 3, since it is more in line with
> current
> > > practice and friendly for existing users. Also It reduces the length of
> > > configuration names as much as possible. I really want to hear your
> > > opinions.
> > >
> > >
> > > @Xuannan
> > >
> > > I agree with your comments 1 and 3.
> > >
> > > For 2, If we decide to change the name, maybe
> > > `execution.checkpointing.parallel-cleaner` is better? And as for
> whether
> > to
> > > add 'enabled' I suggest we discuss the rule above. WDYT?
> > > Thanks!
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <su...@gmail.com>
> > wrote:
> > >
> > > > Hi Zakelly,
> > > >
> > > > Thanks for driving this! The organization of the configuration option
> > > > in the FLIP looks much cleaner and easier to understand. +1 to the
> > > > FLIP.
> > > >
> > > > Just some questions from me.
> > > >
> > > > 1. I think the change to the ConfigOptions should be put in the
> > > > `Public Interface` section, instead of `Proposed Changed`, as those
> > > > configuration options are public interface.
> > > >
> > > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems confusing.
> > > > It feels like it is used to choose different modes. In fact, it is a
> > > > boolean flag to indicate whether to enable parallel clean. How about
> > > > making it `state.checkpoint.cleaner.parallel-mode.enabled`?
> > > >
> > > > 3. The `execution.checkpointing.write-buffer` may better be
> > > > `execution.checkpointing.write-buffer-size` so that we know it is
> > > > configuring the size of the buffer.
> > > >
> > > > Best,
> > > > Xuannan
> > > >
> > > >
> > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <fr...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > > Considering the name occupation, how about naming it as
> > > > `execution.checkpointing.type`?
> > > > >
> > > > > `Checkpoint Type`[1,2] is used to describe aligned/unaligned
> > > > > checkpoint, I am inclined to make a choice between
> > > > > `execution.checkpointing.incremental` and
> > > > > `execution.checkpointing.incremental.enabled`.
> > > > >
> > > > >
> > > > > [1]
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > > > [2]
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > > >
> > > > > --
> > > > > Best,
> > > > > Yanfei
> > > > >
> > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
> > > > > >
> > > > > > Hi Lijie,
> > > > > >
> > > > > > Thanks for the reminder! I missed this.
> > > > > >
> > > > > > Considering the name occupation, how about naming it as
> > > > > > `execution.checkpointing.type`?
> > > > > >
> > > > > > Actually I think the current `execution.checkpointing.mode` is
> > > > confusing in
> > > > > > some ways, maybe `execution.checkpointing.data-consistency` is
> > > better.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > > wangdachui9501@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zakelly,
> > > > > > >
> > > > > > > >> I'm wondering if `execution.checkpointing.savepoint-dir`
> would
> > > be
> > > > > > > better.
> > > > > > >
> > > > > > > `execution.checkpointing.dir` and
> > > > `execution.checkpointing.savepoint-dir`
> > > > > > > are also fine for me.
> > > > > > >
> > > > > > > >> So I think an enumeration option
> > `execution.checkpointing.mode`
> > > > which
> > > > > > > can be 'full' (default) or 'incremental' would be better
> > > > > > >
> > > > > > > I agree with using an enumeration option. But currently there
> is
> > > > already a
> > > > > > > configuration option called `execution.checkpointing.mode`,
> which
> > > is
> > > > used
> > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use
> > > > another name
> > > > > > > or merge these two options.
> > > > > > >
> > > > > > > Best,
> > > > > > > Lijie
> > > > > > >
> > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > Thanks all for your comments!
> > > > > > > >
> > > > > > > > @Yanfei
> > > > > > > >
> > > > > > > > > 1. For some state backends that do not support incremental
> > > > checkpoint,
> > > > > > > > > how does the execution.checkpointing.incrementaloption take
> > > > effect? Or
> > > > > > > > > is it better to put incremental under
> > > > state.backend.xxx.incremental?
> > > > > > > > >
> > > > > > > > I'd rather not put the option for incremental checkpoint
> under
> > > the
> > > > > > > > 'state.backend', since it is more about the checkpointing
> > instead
> > > > of
> > > > > > > state
> > > > > > > > accessing. Of course, the state backend may not necessarily
> do
> > > > > > > incremental
> > > > > > > > checkpoint as requested. If the state backend is not capable
> of
> > > > taking
> > > > > > > > incremental cp, it is better to fallback to the full cp.
> > > > > > > >
> > > > > > > > 2. I'm a little worried that putting all configurations into
> > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > dependency
> > > > > > > > > problems. Some options would be used by flink-runtime
> module,
> > > but
> > > > > > > > > flink-runtime should not depend on flink-streaming-java.
> e.g.
> > > > > > > > > FLINK-28286[1].
> > > > > > > > > So, I prefer to move configurations to
> > `CheckpointingOptions`,
> > > > WDYT?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes, that's a very good point.  Moving to
> > > > > > > > `CheckpointingOptions`(flink-core) makes sense.
> > > > > > > >
> > > > > > > > @Lijie
> > > > > > > >
> > > > > > > > How about
> > > > > > > > > state.savepoints.dir ->
> execution.checkpointing.savepoint.dir
> > > > > > > > > state.checkpoints.dir ->
> > execution.checkpointing.checkpoint.dir
> > > > > > > >
> > > > > > > >
> > > > > > > > Actually, I think the `checkpointing.checkpoint` may cause
> some
> > > > > > > confusion.
> > > > > > > > But I'm ok if others agree.
> > > > > > > > I'm wondering if `execution.checkpointing.savepoint-dir`
> would
> > be
> > > > better.
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we
> also
> > > add
> > > > > > > > "enabled"
> > > > > > > > > suffix for other boolean type configuration options ? For
> > > > example,
> > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > >
> > > > > > > >
> > > > > > > > Actually, the incremental cp is something like choosing a
> mode
> > > for
> > > > doing
> > > > > > > > checkpoint instead of enabling a function. So I think an
> > > > enumeration
> > > > > > > option
> > > > > > > > `execution.checkpointing.mode` which can be 'full' (default)
> or
> > > > > > > > 'incremental' would be better, WDYT?
> > > > > > > > And @Rui Fan @Yanfei What do you think about this?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > > > wangdachui9501@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Zakelly,
> > > > > > > > >
> > > > > > > > > Thanks for driving the discussion.
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > > >> But I'm not so sure since there is only one
> > > savepoint-related
> > > > > > > option.
> > > > > > > > > Maybe someone else could share some thoughts here.
> > > > > > > > >
> > > > > > > > > How about
> > > > > > > > > state.savepoints.dir ->
> execution.checkpointing.savepoint.dir
> > > > > > > > > state.checkpoints.dir ->
> > execution.checkpointing.checkpoint.dir
> > > > > > > > >
> > > > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we
> also
> > > add
> > > > > > > > "enabled"
> > > > > > > > > suffix for other boolean type configuration options ? For
> > > > example,
> > > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > > >
> > > > > > > > > In this way, the naming style of configuration options is
> > > > unified, and
> > > > > > > it
> > > > > > > > > can avoid potential similar problems (for example, we may
> > need
> > > > to add
> > > > > > > > more
> > > > > > > > > options for incremental checkpoint in the future).
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Lijie
> > > > > > > > >
> > > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> > > > > > > > >
> > > > > > > > > > Hi Zakelly,
> > > > > > > > > >
> > > > > > > > > > Thank you for creating the FLIP and starting the
> > discussion.
> > > > > > > > > >
> > > > > > > > > > The current arrangement of these options is indeed
> somewhat
> > > > > > > haphazard,
> > > > > > > > > > and the new arrangement looks much better. I have some
> > > > questions
> > > > > > > about
> > > > > > > > > > the arrangement of some new configuration options:
> > > > > > > > > >
> > > > > > > > > > 1. For some state backends that do not support
> incremental
> > > > > > > checkpoint,
> > > > > > > > > > how does the execution.checkpointing.incrementaloption
> take
> > > > effect?
> > > > > > > Or
> > > > > > > > > > is it better to put incremental under
> > > > state.backend.xxx.incremental?
> > > > > > > > > >
> > > > > > > > > > 2. I'm a little worried that putting all configurations
> > into
> > > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > > dependency
> > > > > > > > > > problems. Some options would be used by flink-runtime
> > module,
> > > > but
> > > > > > > > > > flink-runtime should not depend on flink-streaming-java.
> > e.g.
> > > > > > > > > > FLINK-28286[1].
> > > > > > > > > > So, I prefer to move configurations to
> > > `CheckpointingOptions`,
> > > > WDYT?
> > > > > > > > > >
> > > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best,
> > > > > > > > > > Yanfei
> > > > > > > > > >
> > > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一
> > 21:14写道:
> > > > > > > > > > >
> > > > > > > > > > > Hi Rui Fan and Junrui,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > > > > > >
> > > > > > > > > > > And for other suggestions Rui proposed:
> > > > > > > > > > >
> > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> instead
> > > > > > > > > > > > of execution.checkpointing.storage?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Ah, I missed something here. Actually I suggest we
> could
> > > > merge the
> > > > > > > > > > current
> > > > > > > > > > > 'state.checkpoints.dir' and 'state.checkpoint-storage'
> > into
> > > > one URI
> > > > > > > > > > > configuration named 'execution.checkpointing.dir'.
> WDYT?
> > > > > > > > > > >
> > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a little
> > weird.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yes, I think it is better to make 'savepoint' and
> > > > 'checkpoint' the
> > > > > > > > same
> > > > > > > > > > > level. But I'm not so sure since there is only one
> > > > > > > savepoint-related
> > > > > > > > > > > option. Maybe someone else could share some thoughts
> > here.
> > > > > > > > > > >
> > > > > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >  Agreed. That's more accurate.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Many thanks for your suggestions!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Zakelly
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> > > > jrlee.ljr@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for driving this. I agree that the proposed
> > > > restructuring
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > configuration options is largely positive. It will
> make
> > > > > > > > understanding
> > > > > > > > > > and
> > > > > > > > > > > > working with Flink configurations more intuitive.
> > > > > > > > > > > >
> > > > > > > > > > > > Most of the proposed changes look great. Just a
> > heads-up,
> > > > as Rui
> > > > > > > > Fan
> > > > > > > > > > > > mentioned, Flink currently requires that no
> > > configOption's
> > > > key be
> > > > > > > > the
> > > > > > > > > > > > prefix of another to avoid issues when we eventually
> > > adopt
> > > > a
> > > > > > > > standard
> > > > > > > > > > YAML
> > > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-29372).
> > > > Therefore,
> > > > > > > > it's
> > > > > > > > > > better
> > > > > > > > > > > > to change the key
> 'execution.checkpointing.local-copy'
> > > > because it
> > > > > > > > > > serves as
> > > > > > > > > > > > a prefix to the key
> > > > 'execution.checkpointing.local-copy.dir'.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Junrui
> > > > > > > > > > > >
> > > > > > > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一
> 19:11写道:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you for driving this proposal!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Overall good for me. I have some questions about
> > these
> > > > names.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> > > > instead of
> > > > > > > > > > > > > execution.checkpointing.storage?
> > > > > > > > > > > > >
> > > > > > > > > > > > > It's similar to state.backend.type.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. How about
> > execution.checkpointing.local-copy.enabled
> > > > instead
> > > > > > > > of
> > > > > > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > > > > > >
> > > > > > > > > > > > > You added a new option:
> > > > execution.checkpointing.local-copy.dir.
> > > > > > > > > > > > > IIUC, one option name shouldn't be the prefix of
> > other
> > > > options.
> > > > > > > > > > > > > If you add a new option
> > > > execution.checkpointing.local-copy,
> > > > > > > > > > > > > flink CI will fail directly.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a
> little
> > > > weird.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For old options: state.savepoints.dir and
> > > > > > > state.checkpoints.dir,
> > > > > > > > > > > > > the savepoint and checkpoint are the same level. It
> > > means
> > > > > > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The new option execution.checkpointing.dir is fine
> > for
> > > > me.
> > > > > > > > > > > > > However, execution.checkpointing.savepoint.dir is a
> > > > little
> > > > > > > weird.
> > > > > > > > > > > > > I don't know which name is better now. Let us think
> > > > about it
> > > > > > > > more.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 4. How about execution.recovery.claim-mode instead
> of
> > > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > > >
> > > > > > > > > > > > > The meaning of mode is too broad. The claim-mode
> may
> > > > > > > > > > > > > be more accurate for users.
> > > > > > > > > > > > >
> > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Rui
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > > > > > > zakelly.lan@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi devs,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd like to start a discussion on FLIP-406:
> > > Reorganize
> > > > State
> > > > > > > &
> > > > > > > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Currently, the configuration options pertaining
> to
> > > > > > > > checkpointing,
> > > > > > > > > > > > > recovery,
> > > > > > > > > > > > > > and state management are primarily grouped under
> > the
> > > > > > > following
> > > > > > > > > > > > prefixes:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >    - state.backend.* : configurations related to
> > > state
> > > > > > > > accessing
> > > > > > > > > > and
> > > > > > > > > > > > > >    checkpointing, as well as specific options for
> > > > individual
> > > > > > > > > state
> > > > > > > > > > > > > backends
> > > > > > > > > > > > > >    - execution.checkpointing.* : configurations
> > > > associated
> > > > > > > with
> > > > > > > > > > > > > checkpoint
> > > > > > > > > > > > > >    execution and recovery
> > > > > > > > > > > > > >    - execution.savepoint.*: configurations for
> > > > recovery from
> > > > > > > > > > savepoint
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In addition, there are several individual options
> > > such
> > > > as '
> > > > > > > > > > > > > > *state.checkpoint-storage*' and
> > > > '*state.checkpoints.dir*'
> > > > > > > that
> > > > > > > > > fall
> > > > > > > > > > > > > outside
> > > > > > > > > > > > > > of these prefixes. The current arrangement of
> these
> > > > options,
> > > > > > > > > which
> > > > > > > > > > span
> > > > > > > > > > > > > > multiple modules, is somewhat haphazard and
> lacks a
> > > > > > > systematic
> > > > > > > > > > > > structure.
> > > > > > > > > > > > > > For example, the options under the
> > > > '*CheckpointingOptions*'
> > > > > > > > and '
> > > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are related and
> > have
> > > > no
> > > > > > > clear
> > > > > > > > > > > > boundaries
> > > > > > > > > > > > > > from the user's perspective, but there is no
> > unified
> > > > prefix
> > > > > > > for
> > > > > > > > > > them.
> > > > > > > > > > > > > With
> > > > > > > > > > > > > > the upcoming release of Flink 2.0, we have an
> > > excellent
> > > > > > > > > > opportunity to
> > > > > > > > > > > > > > overhaul and restructure the configurations
> related
> > > to
> > > > > > > > > > checkpointing,
> > > > > > > > > > > > > > recovery, and state management. This FLIP
> proposes
> > to
> > > > > > > > reorganize
> > > > > > > > > > these
> > > > > > > > > > > > > > settings, making it more coherent by module,
> which
> > > > would
> > > > > > > > > > significantly
> > > > > > > > > > > > > > lower the barriers for understanding and reduce
> the
> > > > > > > development
> > > > > > > > > > costs
> > > > > > > > > > > > > > moving forward.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi Rui,

Thanks for your comments!

IMO, given that the state backend can be plugably loaded (as you can
specify a state backend factory), I prefer not providing state backend
specified options in the framework.

Secondly, the incremental checkpoint is actually a sharing file strategy
across checkpoints, which means the state backend *could* reuse files from
previous cp but not *must* do so. When the state backend could not reuse
the files, it is reasonable to fallback to a full checkpoint.

Thus, I suggest we make it `execution.checkpointing.incremental` and enable
it by default. For those state backends not supporting this, they perform
full checkpoints and print a warning to inform users. Users do not need to
pay special attention to different options to control this across different
state backends. This is more user-friendly in my opinion. WDYT?

On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <19...@gmail.com> wrote:

> Hi Zakelly,
>
> I'm not sure whether we could add the state backend type in the
> new key name of state.backend.incremental. It means we use
> `execution.checkpointing.rocksdb-incremental` or
> `execution.checkpointing.rocksdb-incremental.enabled`.
>
> So far, state.backend.incremental only works for rocksdb state backend.
> And this feature or optimization is very valuable and huge for large
> state flink jobs. I believe it's enabled for most production flink jobs
> with large rocksdb state.
>
> If this option isn't generic for all state backend types, I guess we
> can enable `execution.checkpointing.rocksdb-incremental.enabled`
> by default in Flink 2.0.
>
> But if it works for all state backends, it's hard to enable it by default.
> Enabling great and valuable features or improvements are useful
> for users, especially a lot of new flink users. Out-of-the-box options
> are good for users.
>
> WDYT?
>
> Best,
> Rui
>
> On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > Thanks all for your comments!
> >
> > As many of you have questions about the names for boolean options, I
> > suggest we make a naming rule for them. For now I could think of three
> > options:
> >
> > Option 1: Use enumeration options if possible. But this may cause some
> name
> > collisions or confusion as we discussed and we should unify the statement
> > everywhere.
> > Option 2: Use boolean options and add 'enabled' as the suffix.
> > Option 3: Use boolean options and ONLY add 'enabled' when there are more
> > detailed configurations under the same prefix, to prevent one name from
> > serving as a prefix to another.
> >
> > I am slightly inclined to Option 3, since it is more in line with current
> > practice and friendly for existing users. Also It reduces the length of
> > configuration names as much as possible. I really want to hear your
> > opinions.
> >
> >
> > @Xuannan
> >
> > I agree with your comments 1 and 3.
> >
> > For 2, If we decide to change the name, maybe
> > `execution.checkpointing.parallel-cleaner` is better? And as for whether
> to
> > add 'enabled' I suggest we discuss the rule above. WDYT?
> > Thanks!
> >
> >
> > Best,
> > Zakelly
> >
> > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <su...@gmail.com>
> wrote:
> >
> > > Hi Zakelly,
> > >
> > > Thanks for driving this! The organization of the configuration option
> > > in the FLIP looks much cleaner and easier to understand. +1 to the
> > > FLIP.
> > >
> > > Just some questions from me.
> > >
> > > 1. I think the change to the ConfigOptions should be put in the
> > > `Public Interface` section, instead of `Proposed Changed`, as those
> > > configuration options are public interface.
> > >
> > > 2. The key `state.checkpoint.cleaner.parallel-mode` seems confusing.
> > > It feels like it is used to choose different modes. In fact, it is a
> > > boolean flag to indicate whether to enable parallel clean. How about
> > > making it `state.checkpoint.cleaner.parallel-mode.enabled`?
> > >
> > > 3. The `execution.checkpointing.write-buffer` may better be
> > > `execution.checkpointing.write-buffer-size` so that we know it is
> > > configuring the size of the buffer.
> > >
> > > Best,
> > > Xuannan
> > >
> > >
> > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <fr...@gmail.com>
> wrote:
> > > >
> > > > Hi Zakelly,
> > > >
> > > > > Considering the name occupation, how about naming it as
> > > `execution.checkpointing.type`?
> > > >
> > > > `Checkpoint Type`[1,2] is used to describe aligned/unaligned
> > > > checkpoint, I am inclined to make a choice between
> > > > `execution.checkpointing.incremental` and
> > > > `execution.checkpointing.incremental.enabled`.
> > > >
> > > >
> > > > [1]
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > > [2]
> > >
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > > >
> > > > --
> > > > Best,
> > > > Yanfei
> > > >
> > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
> > > > >
> > > > > Hi Lijie,
> > > > >
> > > > > Thanks for the reminder! I missed this.
> > > > >
> > > > > Considering the name occupation, how about naming it as
> > > > > `execution.checkpointing.type`?
> > > > >
> > > > > Actually I think the current `execution.checkpointing.mode` is
> > > confusing in
> > > > > some ways, maybe `execution.checkpointing.data-consistency` is
> > better.
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > >
> > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> > wangdachui9501@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > >> I'm wondering if `execution.checkpointing.savepoint-dir` would
> > be
> > > > > > better.
> > > > > >
> > > > > > `execution.checkpointing.dir` and
> > > `execution.checkpointing.savepoint-dir`
> > > > > > are also fine for me.
> > > > > >
> > > > > > >> So I think an enumeration option
> `execution.checkpointing.mode`
> > > which
> > > > > > can be 'full' (default) or 'incremental' would be better
> > > > > >
> > > > > > I agree with using an enumeration option. But currently there is
> > > already a
> > > > > > configuration option called `execution.checkpointing.mode`, which
> > is
> > > used
> > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use
> > > another name
> > > > > > or merge these two options.
> > > > > >
> > > > > > Best,
> > > > > > Lijie
> > > > > >
> > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thanks all for your comments!
> > > > > > >
> > > > > > > @Yanfei
> > > > > > >
> > > > > > > > 1. For some state backends that do not support incremental
> > > checkpoint,
> > > > > > > > how does the execution.checkpointing.incrementaloption take
> > > effect? Or
> > > > > > > > is it better to put incremental under
> > > state.backend.xxx.incremental?
> > > > > > > >
> > > > > > > I'd rather not put the option for incremental checkpoint under
> > the
> > > > > > > 'state.backend', since it is more about the checkpointing
> instead
> > > of
> > > > > > state
> > > > > > > accessing. Of course, the state backend may not necessarily do
> > > > > > incremental
> > > > > > > checkpoint as requested. If the state backend is not capable of
> > > taking
> > > > > > > incremental cp, it is better to fallback to the full cp.
> > > > > > >
> > > > > > > 2. I'm a little worried that putting all configurations into
> > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> dependency
> > > > > > > > problems. Some options would be used by flink-runtime module,
> > but
> > > > > > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > > > > > FLINK-28286[1].
> > > > > > > > So, I prefer to move configurations to
> `CheckpointingOptions`,
> > > WDYT?
> > > > > > > >
> > > > > > >
> > > > > > > Yes, that's a very good point.  Moving to
> > > > > > > `CheckpointingOptions`(flink-core) makes sense.
> > > > > > >
> > > > > > > @Lijie
> > > > > > >
> > > > > > > How about
> > > > > > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > > > > > state.checkpoints.dir ->
> execution.checkpointing.checkpoint.dir
> > > > > > >
> > > > > > >
> > > > > > > Actually, I think the `checkpointing.checkpoint` may cause some
> > > > > > confusion.
> > > > > > > But I'm ok if others agree.
> > > > > > > I'm wondering if `execution.checkpointing.savepoint-dir` would
> be
> > > better.
> > > > > > > WDYT?
> > > > > > >
> > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we also
> > add
> > > > > > > "enabled"
> > > > > > > > suffix for other boolean type configuration options ? For
> > > example,
> > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > >
> > > > > > >
> > > > > > > Actually, the incremental cp is something like choosing a mode
> > for
> > > doing
> > > > > > > checkpoint instead of enabling a function. So I think an
> > > enumeration
> > > > > > option
> > > > > > > `execution.checkpointing.mode` which can be 'full' (default) or
> > > > > > > 'incremental' would be better, WDYT?
> > > > > > > And @Rui Fan @Yanfei What do you think about this?
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > > wangdachui9501@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zakelly,
> > > > > > > >
> > > > > > > > Thanks for driving the discussion.
> > > > > > > >
> > > > > > > > 1.
> > > > > > > > >> But I'm not so sure since there is only one
> > savepoint-related
> > > > > > option.
> > > > > > > > Maybe someone else could share some thoughts here.
> > > > > > > >
> > > > > > > > How about
> > > > > > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > > > > > state.checkpoints.dir ->
> execution.checkpointing.checkpoint.dir
> > > > > > > >
> > > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we also
> > add
> > > > > > > "enabled"
> > > > > > > > suffix for other boolean type configuration options ? For
> > > example,
> > > > > > > > execution.checkpointing.incremental ->
> > > > > > > > execution.checkpointing.incremental.enabled
> > > > > > > >
> > > > > > > > In this way, the naming style of configuration options is
> > > unified, and
> > > > > > it
> > > > > > > > can avoid potential similar problems (for example, we may
> need
> > > to add
> > > > > > > more
> > > > > > > > options for incremental checkpoint in the future).
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Lijie
> > > > > > > >
> > > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> > > > > > > >
> > > > > > > > > Hi Zakelly,
> > > > > > > > >
> > > > > > > > > Thank you for creating the FLIP and starting the
> discussion.
> > > > > > > > >
> > > > > > > > > The current arrangement of these options is indeed somewhat
> > > > > > haphazard,
> > > > > > > > > and the new arrangement looks much better. I have some
> > > questions
> > > > > > about
> > > > > > > > > the arrangement of some new configuration options:
> > > > > > > > >
> > > > > > > > > 1. For some state backends that do not support incremental
> > > > > > checkpoint,
> > > > > > > > > how does the execution.checkpointing.incrementaloption take
> > > effect?
> > > > > > Or
> > > > > > > > > is it better to put incremental under
> > > state.backend.xxx.incremental?
> > > > > > > > >
> > > > > > > > > 2. I'm a little worried that putting all configurations
> into
> > > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> > dependency
> > > > > > > > > problems. Some options would be used by flink-runtime
> module,
> > > but
> > > > > > > > > flink-runtime should not depend on flink-streaming-java.
> e.g.
> > > > > > > > > FLINK-28286[1].
> > > > > > > > > So, I prefer to move configurations to
> > `CheckpointingOptions`,
> > > WDYT?
> > > > > > > > >
> > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best,
> > > > > > > > > Yanfei
> > > > > > > > >
> > > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一
> 21:14写道:
> > > > > > > > > >
> > > > > > > > > > Hi Rui Fan and Junrui,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > > > > >
> > > > > > > > > > And for other suggestions Rui proposed:
> > > > > > > > > >
> > > > > > > > > > 1. How about execution.checkpointing.storage.type instead
> > > > > > > > > > > of execution.checkpointing.storage?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Ah, I missed something here. Actually I suggest we could
> > > merge the
> > > > > > > > > current
> > > > > > > > > > 'state.checkpoints.dir' and 'state.checkpoint-storage'
> into
> > > one URI
> > > > > > > > > > configuration named 'execution.checkpointing.dir'. WDYT?
> > > > > > > > > >
> > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a little
> weird.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yes, I think it is better to make 'savepoint' and
> > > 'checkpoint' the
> > > > > > > same
> > > > > > > > > > level. But I'm not so sure since there is only one
> > > > > > savepoint-related
> > > > > > > > > > option. Maybe someone else could share some thoughts
> here.
> > > > > > > > > >
> > > > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >  Agreed. That's more accurate.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Many thanks for your suggestions!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Zakelly
> > > > > > > > > >
> > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> > > jrlee.ljr@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for driving this. I agree that the proposed
> > > restructuring
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > configuration options is largely positive. It will make
> > > > > > > understanding
> > > > > > > > > and
> > > > > > > > > > > working with Flink configurations more intuitive.
> > > > > > > > > > >
> > > > > > > > > > > Most of the proposed changes look great. Just a
> heads-up,
> > > as Rui
> > > > > > > Fan
> > > > > > > > > > > mentioned, Flink currently requires that no
> > configOption's
> > > key be
> > > > > > > the
> > > > > > > > > > > prefix of another to avoid issues when we eventually
> > adopt
> > > a
> > > > > > > standard
> > > > > > > > > YAML
> > > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-29372).
> > > Therefore,
> > > > > > > it's
> > > > > > > > > better
> > > > > > > > > > > to change the key 'execution.checkpointing.local-copy'
> > > because it
> > > > > > > > > serves as
> > > > > > > > > > > a prefix to the key
> > > 'execution.checkpointing.local-copy.dir'.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Junrui
> > > > > > > > > > >
> > > > > > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you for driving this proposal!
> > > > > > > > > > > >
> > > > > > > > > > > > Overall good for me. I have some questions about
> these
> > > names.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> > > instead of
> > > > > > > > > > > > execution.checkpointing.storage?
> > > > > > > > > > > >
> > > > > > > > > > > > It's similar to state.backend.type.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. How about
> execution.checkpointing.local-copy.enabled
> > > instead
> > > > > > > of
> > > > > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > > > > >
> > > > > > > > > > > > You added a new option:
> > > execution.checkpointing.local-copy.dir.
> > > > > > > > > > > > IIUC, one option name shouldn't be the prefix of
> other
> > > options.
> > > > > > > > > > > > If you add a new option
> > > execution.checkpointing.local-copy,
> > > > > > > > > > > > flink CI will fail directly.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a little
> > > weird.
> > > > > > > > > > > >
> > > > > > > > > > > > For old options: state.savepoints.dir and
> > > > > > state.checkpoints.dir,
> > > > > > > > > > > > the savepoint and checkpoint are the same level. It
> > means
> > > > > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > > > > >
> > > > > > > > > > > > The new option execution.checkpointing.dir is fine
> for
> > > me.
> > > > > > > > > > > > However, execution.checkpointing.savepoint.dir is a
> > > little
> > > > > > weird.
> > > > > > > > > > > > I don't know which name is better now. Let us think
> > > about it
> > > > > > > more.
> > > > > > > > > > > >
> > > > > > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > > >
> > > > > > > > > > > > The meaning of mode is too broad. The claim-mode may
> > > > > > > > > > > > be more accurate for users.
> > > > > > > > > > > >
> > > > > > > > > > > > WDYT?
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Rui
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > > > > > zakelly.lan@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi devs,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd like to start a discussion on FLIP-406:
> > Reorganize
> > > State
> > > > > > &
> > > > > > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > > > > > >
> > > > > > > > > > > > > Currently, the configuration options pertaining to
> > > > > > > checkpointing,
> > > > > > > > > > > > recovery,
> > > > > > > > > > > > > and state management are primarily grouped under
> the
> > > > > > following
> > > > > > > > > > > prefixes:
> > > > > > > > > > > > >
> > > > > > > > > > > > >    - state.backend.* : configurations related to
> > state
> > > > > > > accessing
> > > > > > > > > and
> > > > > > > > > > > > >    checkpointing, as well as specific options for
> > > individual
> > > > > > > > state
> > > > > > > > > > > > backends
> > > > > > > > > > > > >    - execution.checkpointing.* : configurations
> > > associated
> > > > > > with
> > > > > > > > > > > > checkpoint
> > > > > > > > > > > > >    execution and recovery
> > > > > > > > > > > > >    - execution.savepoint.*: configurations for
> > > recovery from
> > > > > > > > > savepoint
> > > > > > > > > > > > >
> > > > > > > > > > > > > In addition, there are several individual options
> > such
> > > as '
> > > > > > > > > > > > > *state.checkpoint-storage*' and
> > > '*state.checkpoints.dir*'
> > > > > > that
> > > > > > > > fall
> > > > > > > > > > > > outside
> > > > > > > > > > > > > of these prefixes. The current arrangement of these
> > > options,
> > > > > > > > which
> > > > > > > > > span
> > > > > > > > > > > > > multiple modules, is somewhat haphazard and lacks a
> > > > > > systematic
> > > > > > > > > > > structure.
> > > > > > > > > > > > > For example, the options under the
> > > '*CheckpointingOptions*'
> > > > > > > and '
> > > > > > > > > > > > > *ExecutionCheckpointingOptions*' are related and
> have
> > > no
> > > > > > clear
> > > > > > > > > > > boundaries
> > > > > > > > > > > > > from the user's perspective, but there is no
> unified
> > > prefix
> > > > > > for
> > > > > > > > > them.
> > > > > > > > > > > > With
> > > > > > > > > > > > > the upcoming release of Flink 2.0, we have an
> > excellent
> > > > > > > > > opportunity to
> > > > > > > > > > > > > overhaul and restructure the configurations related
> > to
> > > > > > > > > checkpointing,
> > > > > > > > > > > > > recovery, and state management. This FLIP proposes
> to
> > > > > > > reorganize
> > > > > > > > > these
> > > > > > > > > > > > > settings, making it more coherent by module, which
> > > would
> > > > > > > > > significantly
> > > > > > > > > > > > > lower the barriers for understanding and reduce the
> > > > > > development
> > > > > > > > > costs
> > > > > > > > > > > > > moving forward.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Rui Fan <19...@gmail.com>.
Hi Zakelly,

I'm not sure whether we could add the state backend type in the
new key name of state.backend.incremental. It means we use
`execution.checkpointing.rocksdb-incremental` or
`execution.checkpointing.rocksdb-incremental.enabled`.

So far, state.backend.incremental only works for rocksdb state backend.
And this feature or optimization is very valuable and huge for large
state flink jobs. I believe it's enabled for most production flink jobs
with large rocksdb state.

If this option isn't generic for all state backend types, I guess we
can enable `execution.checkpointing.rocksdb-incremental.enabled`
by default in Flink 2.0.

But if it works for all state backends, it's hard to enable it by default.
Enabling great and valuable features or improvements are useful
for users, especially a lot of new flink users. Out-of-the-box options
are good for users.

WDYT?

Best,
Rui

On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan <za...@gmail.com> wrote:

> Hi everyone,
>
> Thanks all for your comments!
>
> As many of you have questions about the names for boolean options, I
> suggest we make a naming rule for them. For now I could think of three
> options:
>
> Option 1: Use enumeration options if possible. But this may cause some name
> collisions or confusion as we discussed and we should unify the statement
> everywhere.
> Option 2: Use boolean options and add 'enabled' as the suffix.
> Option 3: Use boolean options and ONLY add 'enabled' when there are more
> detailed configurations under the same prefix, to prevent one name from
> serving as a prefix to another.
>
> I am slightly inclined to Option 3, since it is more in line with current
> practice and friendly for existing users. Also It reduces the length of
> configuration names as much as possible. I really want to hear your
> opinions.
>
>
> @Xuannan
>
> I agree with your comments 1 and 3.
>
> For 2, If we decide to change the name, maybe
> `execution.checkpointing.parallel-cleaner` is better? And as for whether to
> add 'enabled' I suggest we discuss the rule above. WDYT?
> Thanks!
>
>
> Best,
> Zakelly
>
> On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <su...@gmail.com> wrote:
>
> > Hi Zakelly,
> >
> > Thanks for driving this! The organization of the configuration option
> > in the FLIP looks much cleaner and easier to understand. +1 to the
> > FLIP.
> >
> > Just some questions from me.
> >
> > 1. I think the change to the ConfigOptions should be put in the
> > `Public Interface` section, instead of `Proposed Changed`, as those
> > configuration options are public interface.
> >
> > 2. The key `state.checkpoint.cleaner.parallel-mode` seems confusing.
> > It feels like it is used to choose different modes. In fact, it is a
> > boolean flag to indicate whether to enable parallel clean. How about
> > making it `state.checkpoint.cleaner.parallel-mode.enabled`?
> >
> > 3. The `execution.checkpointing.write-buffer` may better be
> > `execution.checkpointing.write-buffer-size` so that we know it is
> > configuring the size of the buffer.
> >
> > Best,
> > Xuannan
> >
> >
> > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <fr...@gmail.com> wrote:
> > >
> > > Hi Zakelly,
> > >
> > > > Considering the name occupation, how about naming it as
> > `execution.checkpointing.type`?
> > >
> > > `Checkpoint Type`[1,2] is used to describe aligned/unaligned
> > > checkpoint, I am inclined to make a choice between
> > > `execution.checkpointing.incremental` and
> > > `execution.checkpointing.incremental.enabled`.
> > >
> > >
> > > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > > [2]
> >
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> > >
> > > --
> > > Best,
> > > Yanfei
> > >
> > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
> > > >
> > > > Hi Lijie,
> > > >
> > > > Thanks for the reminder! I missed this.
> > > >
> > > > Considering the name occupation, how about naming it as
> > > > `execution.checkpointing.type`?
> > > >
> > > > Actually I think the current `execution.checkpointing.mode` is
> > confusing in
> > > > some ways, maybe `execution.checkpointing.data-consistency` is
> better.
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > >
> > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <
> wangdachui9501@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > >> I'm wondering if `execution.checkpointing.savepoint-dir` would
> be
> > > > > better.
> > > > >
> > > > > `execution.checkpointing.dir` and
> > `execution.checkpointing.savepoint-dir`
> > > > > are also fine for me.
> > > > >
> > > > > >> So I think an enumeration option `execution.checkpointing.mode`
> > which
> > > > > can be 'full' (default) or 'incremental' would be better
> > > > >
> > > > > I agree with using an enumeration option. But currently there is
> > already a
> > > > > configuration option called `execution.checkpointing.mode`, which
> is
> > used
> > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use
> > another name
> > > > > or merge these two options.
> > > > >
> > > > > Best,
> > > > > Lijie
> > > > >
> > > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Thanks all for your comments!
> > > > > >
> > > > > > @Yanfei
> > > > > >
> > > > > > > 1. For some state backends that do not support incremental
> > checkpoint,
> > > > > > > how does the execution.checkpointing.incrementaloption take
> > effect? Or
> > > > > > > is it better to put incremental under
> > state.backend.xxx.incremental?
> > > > > > >
> > > > > > I'd rather not put the option for incremental checkpoint under
> the
> > > > > > 'state.backend', since it is more about the checkpointing instead
> > of
> > > > > state
> > > > > > accessing. Of course, the state backend may not necessarily do
> > > > > incremental
> > > > > > checkpoint as requested. If the state backend is not capable of
> > taking
> > > > > > incremental cp, it is better to fallback to the full cp.
> > > > > >
> > > > > > 2. I'm a little worried that putting all configurations into
> > > > > > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > > > > > problems. Some options would be used by flink-runtime module,
> but
> > > > > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > > > > FLINK-28286[1].
> > > > > > > So, I prefer to move configurations to `CheckpointingOptions`,
> > WDYT?
> > > > > > >
> > > > > >
> > > > > > Yes, that's a very good point.  Moving to
> > > > > > `CheckpointingOptions`(flink-core) makes sense.
> > > > > >
> > > > > > @Lijie
> > > > > >
> > > > > > How about
> > > > > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > > > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> > > > > >
> > > > > >
> > > > > > Actually, I think the `checkpointing.checkpoint` may cause some
> > > > > confusion.
> > > > > > But I'm ok if others agree.
> > > > > > I'm wondering if `execution.checkpointing.savepoint-dir` would be
> > better.
> > > > > > WDYT?
> > > > > >
> > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we also
> add
> > > > > > "enabled"
> > > > > > > suffix for other boolean type configuration options ? For
> > example,
> > > > > > > execution.checkpointing.incremental ->
> > > > > > > execution.checkpointing.incremental.enabled
> > > > > > >
> > > > > >
> > > > > > Actually, the incremental cp is something like choosing a mode
> for
> > doing
> > > > > > checkpoint instead of enabling a function. So I think an
> > enumeration
> > > > > option
> > > > > > `execution.checkpointing.mode` which can be 'full' (default) or
> > > > > > 'incremental' would be better, WDYT?
> > > > > > And @Rui Fan @Yanfei What do you think about this?
> > > > > >
> > > > > >
> > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> > wangdachui9501@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zakelly,
> > > > > > >
> > > > > > > Thanks for driving the discussion.
> > > > > > >
> > > > > > > 1.
> > > > > > > >> But I'm not so sure since there is only one
> savepoint-related
> > > > > option.
> > > > > > > Maybe someone else could share some thoughts here.
> > > > > > >
> > > > > > > How about
> > > > > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > > > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> > > > > > >
> > > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > > 'execution.checkpointing.local-copy.enabled'. Should we also
> add
> > > > > > "enabled"
> > > > > > > suffix for other boolean type configuration options ? For
> > example,
> > > > > > > execution.checkpointing.incremental ->
> > > > > > > execution.checkpointing.incremental.enabled
> > > > > > >
> > > > > > > In this way, the naming style of configuration options is
> > unified, and
> > > > > it
> > > > > > > can avoid potential similar problems (for example, we may need
> > to add
> > > > > > more
> > > > > > > options for incremental checkpoint in the future).
> > > > > > >
> > > > > > > Best,
> > > > > > > Lijie
> > > > > > >
> > > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> > > > > > >
> > > > > > > > Hi Zakelly,
> > > > > > > >
> > > > > > > > Thank you for creating the FLIP and starting the discussion.
> > > > > > > >
> > > > > > > > The current arrangement of these options is indeed somewhat
> > > > > haphazard,
> > > > > > > > and the new arrangement looks much better. I have some
> > questions
> > > > > about
> > > > > > > > the arrangement of some new configuration options:
> > > > > > > >
> > > > > > > > 1. For some state backends that do not support incremental
> > > > > checkpoint,
> > > > > > > > how does the execution.checkpointing.incrementaloption take
> > effect?
> > > > > Or
> > > > > > > > is it better to put incremental under
> > state.backend.xxx.incremental?
> > > > > > > >
> > > > > > > > 2. I'm a little worried that putting all configurations into
> > > > > > > > `ExecutionCheckpointingOptions` will introduce some
> dependency
> > > > > > > > problems. Some options would be used by flink-runtime module,
> > but
> > > > > > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > > > > > FLINK-28286[1].
> > > > > > > > So, I prefer to move configurations to
> `CheckpointingOptions`,
> > WDYT?
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best,
> > > > > > > > Yanfei
> > > > > > > >
> > > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一 21:14写道:
> > > > > > > > >
> > > > > > > > > Hi Rui Fan and Junrui,
> > > > > > > > >
> > > > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > > > >
> > > > > > > > > And for other suggestions Rui proposed:
> > > > > > > > >
> > > > > > > > > 1. How about execution.checkpointing.storage.type instead
> > > > > > > > > > of execution.checkpointing.storage?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Ah, I missed something here. Actually I suggest we could
> > merge the
> > > > > > > > current
> > > > > > > > > 'state.checkpoints.dir' and 'state.checkpoint-storage' into
> > one URI
> > > > > > > > > configuration named 'execution.checkpointing.dir'. WDYT?
> > > > > > > > >
> > > > > > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, I think it is better to make 'savepoint' and
> > 'checkpoint' the
> > > > > > same
> > > > > > > > > level. But I'm not so sure since there is only one
> > > > > savepoint-related
> > > > > > > > > option. Maybe someone else could share some thoughts here.
> > > > > > > > >
> > > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > > execution.recovery.mode?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >  Agreed. That's more accurate.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Many thanks for your suggestions!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Zakelly
> > > > > > > > >
> > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> > jrlee.ljr@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Zakelly,
> > > > > > > > > >
> > > > > > > > > > Thanks for driving this. I agree that the proposed
> > restructuring
> > > > > of
> > > > > > > the
> > > > > > > > > > configuration options is largely positive. It will make
> > > > > > understanding
> > > > > > > > and
> > > > > > > > > > working with Flink configurations more intuitive.
> > > > > > > > > >
> > > > > > > > > > Most of the proposed changes look great. Just a heads-up,
> > as Rui
> > > > > > Fan
> > > > > > > > > > mentioned, Flink currently requires that no
> configOption's
> > key be
> > > > > > the
> > > > > > > > > > prefix of another to avoid issues when we eventually
> adopt
> > a
> > > > > > standard
> > > > > > > > YAML
> > > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-29372).
> > Therefore,
> > > > > > it's
> > > > > > > > better
> > > > > > > > > > to change the key 'execution.checkpointing.local-copy'
> > because it
> > > > > > > > serves as
> > > > > > > > > > a prefix to the key
> > 'execution.checkpointing.local-copy.dir'.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Junrui
> > > > > > > > > >
> > > > > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> > > > > > > > > >
> > > > > > > > > > > Hi Zakelly,
> > > > > > > > > > >
> > > > > > > > > > > Thank you for driving this proposal!
> > > > > > > > > > >
> > > > > > > > > > > Overall good for me. I have some questions about these
> > names.
> > > > > > > > > > >
> > > > > > > > > > > 1. How about execution.checkpointing.storage.type
> > instead of
> > > > > > > > > > > execution.checkpointing.storage?
> > > > > > > > > > >
> > > > > > > > > > > It's similar to state.backend.type.
> > > > > > > > > > >
> > > > > > > > > > > 2. How about execution.checkpointing.local-copy.enabled
> > instead
> > > > > > of
> > > > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > > > >
> > > > > > > > > > > You added a new option:
> > execution.checkpointing.local-copy.dir.
> > > > > > > > > > > IIUC, one option name shouldn't be the prefix of other
> > options.
> > > > > > > > > > > If you add a new option
> > execution.checkpointing.local-copy,
> > > > > > > > > > > flink CI will fail directly.
> > > > > > > > > > >
> > > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a little
> > weird.
> > > > > > > > > > >
> > > > > > > > > > > For old options: state.savepoints.dir and
> > > > > state.checkpoints.dir,
> > > > > > > > > > > the savepoint and checkpoint are the same level. It
> means
> > > > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > > > >
> > > > > > > > > > > The new option execution.checkpointing.dir is fine for
> > me.
> > > > > > > > > > > However, execution.checkpointing.savepoint.dir is a
> > little
> > > > > weird.
> > > > > > > > > > > I don't know which name is better now. Let us think
> > about it
> > > > > > more.
> > > > > > > > > > >
> > > > > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > > > execution.recovery.mode?
> > > > > > > > > > >
> > > > > > > > > > > The meaning of mode is too broad. The claim-mode may
> > > > > > > > > > > be more accurate for users.
> > > > > > > > > > >
> > > > > > > > > > > WDYT?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Rui
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > > > > zakelly.lan@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi devs,
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to start a discussion on FLIP-406:
> Reorganize
> > State
> > > > > &
> > > > > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > > > > >
> > > > > > > > > > > > Currently, the configuration options pertaining to
> > > > > > checkpointing,
> > > > > > > > > > > recovery,
> > > > > > > > > > > > and state management are primarily grouped under the
> > > > > following
> > > > > > > > > > prefixes:
> > > > > > > > > > > >
> > > > > > > > > > > >    - state.backend.* : configurations related to
> state
> > > > > > accessing
> > > > > > > > and
> > > > > > > > > > > >    checkpointing, as well as specific options for
> > individual
> > > > > > > state
> > > > > > > > > > > backends
> > > > > > > > > > > >    - execution.checkpointing.* : configurations
> > associated
> > > > > with
> > > > > > > > > > > checkpoint
> > > > > > > > > > > >    execution and recovery
> > > > > > > > > > > >    - execution.savepoint.*: configurations for
> > recovery from
> > > > > > > > savepoint
> > > > > > > > > > > >
> > > > > > > > > > > > In addition, there are several individual options
> such
> > as '
> > > > > > > > > > > > *state.checkpoint-storage*' and
> > '*state.checkpoints.dir*'
> > > > > that
> > > > > > > fall
> > > > > > > > > > > outside
> > > > > > > > > > > > of these prefixes. The current arrangement of these
> > options,
> > > > > > > which
> > > > > > > > span
> > > > > > > > > > > > multiple modules, is somewhat haphazard and lacks a
> > > > > systematic
> > > > > > > > > > structure.
> > > > > > > > > > > > For example, the options under the
> > '*CheckpointingOptions*'
> > > > > > and '
> > > > > > > > > > > > *ExecutionCheckpointingOptions*' are related and have
> > no
> > > > > clear
> > > > > > > > > > boundaries
> > > > > > > > > > > > from the user's perspective, but there is no unified
> > prefix
> > > > > for
> > > > > > > > them.
> > > > > > > > > > > With
> > > > > > > > > > > > the upcoming release of Flink 2.0, we have an
> excellent
> > > > > > > > opportunity to
> > > > > > > > > > > > overhaul and restructure the configurations related
> to
> > > > > > > > checkpointing,
> > > > > > > > > > > > recovery, and state management. This FLIP proposes to
> > > > > > reorganize
> > > > > > > > these
> > > > > > > > > > > > settings, making it more coherent by module, which
> > would
> > > > > > > > significantly
> > > > > > > > > > > > lower the barriers for understanding and reduce the
> > > > > development
> > > > > > > > costs
> > > > > > > > > > > > moving forward.
> > > > > > > > > > > >
> > > > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Zakelly
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi everyone,

Thanks all for your comments!

As many of you have questions about the names for boolean options, I
suggest we make a naming rule for them. For now I could think of three
options:

Option 1: Use enumeration options if possible. But this may cause some name
collisions or confusion as we discussed and we should unify the statement
everywhere.
Option 2: Use boolean options and add 'enabled' as the suffix.
Option 3: Use boolean options and ONLY add 'enabled' when there are more
detailed configurations under the same prefix, to prevent one name from
serving as a prefix to another.

I am slightly inclined to Option 3, since it is more in line with current
practice and friendly for existing users. Also It reduces the length of
configuration names as much as possible. I really want to hear your
opinions.


@Xuannan

I agree with your comments 1 and 3.

For 2, If we decide to change the name, maybe
`execution.checkpointing.parallel-cleaner` is better? And as for whether to
add 'enabled' I suggest we discuss the rule above. WDYT?
Thanks!


Best,
Zakelly

On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su <su...@gmail.com> wrote:

> Hi Zakelly,
>
> Thanks for driving this! The organization of the configuration option
> in the FLIP looks much cleaner and easier to understand. +1 to the
> FLIP.
>
> Just some questions from me.
>
> 1. I think the change to the ConfigOptions should be put in the
> `Public Interface` section, instead of `Proposed Changed`, as those
> configuration options are public interface.
>
> 2. The key `state.checkpoint.cleaner.parallel-mode` seems confusing.
> It feels like it is used to choose different modes. In fact, it is a
> boolean flag to indicate whether to enable parallel clean. How about
> making it `state.checkpoint.cleaner.parallel-mode.enabled`?
>
> 3. The `execution.checkpointing.write-buffer` may better be
> `execution.checkpointing.write-buffer-size` so that we know it is
> configuring the size of the buffer.
>
> Best,
> Xuannan
>
>
> On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <fr...@gmail.com> wrote:
> >
> > Hi Zakelly,
> >
> > > Considering the name occupation, how about naming it as
> `execution.checkpointing.type`?
> >
> > `Checkpoint Type`[1,2] is used to describe aligned/unaligned
> > checkpoint, I am inclined to make a choice between
> > `execution.checkpointing.incremental` and
> > `execution.checkpointing.incremental.enabled`.
> >
> >
> > [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> > [2]
> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
> >
> > --
> > Best,
> > Yanfei
> >
> > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
> > >
> > > Hi Lijie,
> > >
> > > Thanks for the reminder! I missed this.
> > >
> > > Considering the name occupation, how about naming it as
> > > `execution.checkpointing.type`?
> > >
> > > Actually I think the current `execution.checkpointing.mode` is
> confusing in
> > > some ways, maybe `execution.checkpointing.data-consistency` is better.
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > >
> > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <wa...@gmail.com>
> > > wrote:
> > >
> > > > Hi Zakelly,
> > > >
> > > > >> I'm wondering if `execution.checkpointing.savepoint-dir` would be
> > > > better.
> > > >
> > > > `execution.checkpointing.dir` and
> `execution.checkpointing.savepoint-dir`
> > > > are also fine for me.
> > > >
> > > > >> So I think an enumeration option `execution.checkpointing.mode`
> which
> > > > can be 'full' (default) or 'incremental' would be better
> > > >
> > > > I agree with using an enumeration option. But currently there is
> already a
> > > > configuration option called `execution.checkpointing.mode`, which is
> used
> > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use
> another name
> > > > or merge these two options.
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thanks all for your comments!
> > > > >
> > > > > @Yanfei
> > > > >
> > > > > > 1. For some state backends that do not support incremental
> checkpoint,
> > > > > > how does the execution.checkpointing.incrementaloption take
> effect? Or
> > > > > > is it better to put incremental under
> state.backend.xxx.incremental?
> > > > > >
> > > > > I'd rather not put the option for incremental checkpoint under the
> > > > > 'state.backend', since it is more about the checkpointing instead
> of
> > > > state
> > > > > accessing. Of course, the state backend may not necessarily do
> > > > incremental
> > > > > checkpoint as requested. If the state backend is not capable of
> taking
> > > > > incremental cp, it is better to fallback to the full cp.
> > > > >
> > > > > 2. I'm a little worried that putting all configurations into
> > > > > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > > > > problems. Some options would be used by flink-runtime module, but
> > > > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > > > FLINK-28286[1].
> > > > > > So, I prefer to move configurations to `CheckpointingOptions`,
> WDYT?
> > > > > >
> > > > >
> > > > > Yes, that's a very good point.  Moving to
> > > > > `CheckpointingOptions`(flink-core) makes sense.
> > > > >
> > > > > @Lijie
> > > > >
> > > > > How about
> > > > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> > > > >
> > > > >
> > > > > Actually, I think the `checkpointing.checkpoint` may cause some
> > > > confusion.
> > > > > But I'm ok if others agree.
> > > > > I'm wondering if `execution.checkpointing.savepoint-dir` would be
> better.
> > > > > WDYT?
> > > > >
> > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > 'execution.checkpointing.local-copy.enabled'. Should we also add
> > > > > "enabled"
> > > > > > suffix for other boolean type configuration options ? For
> example,
> > > > > > execution.checkpointing.incremental ->
> > > > > > execution.checkpointing.incremental.enabled
> > > > > >
> > > > >
> > > > > Actually, the incremental cp is something like choosing a mode for
> doing
> > > > > checkpoint instead of enabling a function. So I think an
> enumeration
> > > > option
> > > > > `execution.checkpointing.mode` which can be 'full' (default) or
> > > > > 'incremental' would be better, WDYT?
> > > > > And @Rui Fan @Yanfei What do you think about this?
> > > > >
> > > > >
> > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <
> wangdachui9501@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > Thanks for driving the discussion.
> > > > > >
> > > > > > 1.
> > > > > > >> But I'm not so sure since there is only one savepoint-related
> > > > option.
> > > > > > Maybe someone else could share some thoughts here.
> > > > > >
> > > > > > How about
> > > > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> > > > > >
> > > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > > 'execution.checkpointing.local-copy.enabled'. Should we also add
> > > > > "enabled"
> > > > > > suffix for other boolean type configuration options ? For
> example,
> > > > > > execution.checkpointing.incremental ->
> > > > > > execution.checkpointing.incremental.enabled
> > > > > >
> > > > > > In this way, the naming style of configuration options is
> unified, and
> > > > it
> > > > > > can avoid potential similar problems (for example, we may need
> to add
> > > > > more
> > > > > > options for incremental checkpoint in the future).
> > > > > >
> > > > > > Best,
> > > > > > Lijie
> > > > > >
> > > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> > > > > >
> > > > > > > Hi Zakelly,
> > > > > > >
> > > > > > > Thank you for creating the FLIP and starting the discussion.
> > > > > > >
> > > > > > > The current arrangement of these options is indeed somewhat
> > > > haphazard,
> > > > > > > and the new arrangement looks much better. I have some
> questions
> > > > about
> > > > > > > the arrangement of some new configuration options:
> > > > > > >
> > > > > > > 1. For some state backends that do not support incremental
> > > > checkpoint,
> > > > > > > how does the execution.checkpointing.incrementaloption take
> effect?
> > > > Or
> > > > > > > is it better to put incremental under
> state.backend.xxx.incremental?
> > > > > > >
> > > > > > > 2. I'm a little worried that putting all configurations into
> > > > > > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > > > > > problems. Some options would be used by flink-runtime module,
> but
> > > > > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > > > > FLINK-28286[1].
> > > > > > > So, I prefer to move configurations to `CheckpointingOptions`,
> WDYT?
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > > > > > >
> > > > > > > --
> > > > > > > Best,
> > > > > > > Yanfei
> > > > > > >
> > > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一 21:14写道:
> > > > > > > >
> > > > > > > > Hi Rui Fan and Junrui,
> > > > > > > >
> > > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > > >
> > > > > > > > And for other suggestions Rui proposed:
> > > > > > > >
> > > > > > > > 1. How about execution.checkpointing.storage.type instead
> > > > > > > > > of execution.checkpointing.storage?
> > > > > > > >
> > > > > > > >
> > > > > > > > Ah, I missed something here. Actually I suggest we could
> merge the
> > > > > > > current
> > > > > > > > 'state.checkpoints.dir' and 'state.checkpoint-storage' into
> one URI
> > > > > > > > configuration named 'execution.checkpointing.dir'. WDYT?
> > > > > > > >
> > > > > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes, I think it is better to make 'savepoint' and
> 'checkpoint' the
> > > > > same
> > > > > > > > level. But I'm not so sure since there is only one
> > > > savepoint-related
> > > > > > > > option. Maybe someone else could share some thoughts here.
> > > > > > > >
> > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > execution.recovery.mode?
> > > > > > > > >
> > > > > > > >
> > > > > > > >  Agreed. That's more accurate.
> > > > > > > >
> > > > > > > >
> > > > > > > > Many thanks for your suggestions!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zakelly
> > > > > > > >
> > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <
> jrlee.ljr@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Zakelly,
> > > > > > > > >
> > > > > > > > > Thanks for driving this. I agree that the proposed
> restructuring
> > > > of
> > > > > > the
> > > > > > > > > configuration options is largely positive. It will make
> > > > > understanding
> > > > > > > and
> > > > > > > > > working with Flink configurations more intuitive.
> > > > > > > > >
> > > > > > > > > Most of the proposed changes look great. Just a heads-up,
> as Rui
> > > > > Fan
> > > > > > > > > mentioned, Flink currently requires that no configOption's
> key be
> > > > > the
> > > > > > > > > prefix of another to avoid issues when we eventually adopt
> a
> > > > > standard
> > > > > > > YAML
> > > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-29372).
> Therefore,
> > > > > it's
> > > > > > > better
> > > > > > > > > to change the key 'execution.checkpointing.local-copy'
> because it
> > > > > > > serves as
> > > > > > > > > a prefix to the key
> 'execution.checkpointing.local-copy.dir'.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Junrui
> > > > > > > > >
> > > > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> > > > > > > > >
> > > > > > > > > > Hi Zakelly,
> > > > > > > > > >
> > > > > > > > > > Thank you for driving this proposal!
> > > > > > > > > >
> > > > > > > > > > Overall good for me. I have some questions about these
> names.
> > > > > > > > > >
> > > > > > > > > > 1. How about execution.checkpointing.storage.type
> instead of
> > > > > > > > > > execution.checkpointing.storage?
> > > > > > > > > >
> > > > > > > > > > It's similar to state.backend.type.
> > > > > > > > > >
> > > > > > > > > > 2. How about execution.checkpointing.local-copy.enabled
> instead
> > > > > of
> > > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > > >
> > > > > > > > > > You added a new option:
> execution.checkpointing.local-copy.dir.
> > > > > > > > > > IIUC, one option name shouldn't be the prefix of other
> options.
> > > > > > > > > > If you add a new option
> execution.checkpointing.local-copy,
> > > > > > > > > > flink CI will fail directly.
> > > > > > > > > >
> > > > > > > > > > 3. execution.checkpointing.savepoint.dir is a little
> weird.
> > > > > > > > > >
> > > > > > > > > > For old options: state.savepoints.dir and
> > > > state.checkpoints.dir,
> > > > > > > > > > the savepoint and checkpoint are the same level. It means
> > > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > > >
> > > > > > > > > > The new option execution.checkpointing.dir is fine for
> me.
> > > > > > > > > > However, execution.checkpointing.savepoint.dir is a
> little
> > > > weird.
> > > > > > > > > > I don't know which name is better now. Let us think
> about it
> > > > > more.
> > > > > > > > > >
> > > > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > > execution.recovery.mode?
> > > > > > > > > >
> > > > > > > > > > The meaning of mode is too broad. The claim-mode may
> > > > > > > > > > be more accurate for users.
> > > > > > > > > >
> > > > > > > > > > WDYT?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Rui
> > > > > > > > > >
> > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > > > zakelly.lan@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi devs,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to start a discussion on FLIP-406: Reorganize
> State
> > > > &
> > > > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > > > >
> > > > > > > > > > > Currently, the configuration options pertaining to
> > > > > checkpointing,
> > > > > > > > > > recovery,
> > > > > > > > > > > and state management are primarily grouped under the
> > > > following
> > > > > > > > > prefixes:
> > > > > > > > > > >
> > > > > > > > > > >    - state.backend.* : configurations related to state
> > > > > accessing
> > > > > > > and
> > > > > > > > > > >    checkpointing, as well as specific options for
> individual
> > > > > > state
> > > > > > > > > > backends
> > > > > > > > > > >    - execution.checkpointing.* : configurations
> associated
> > > > with
> > > > > > > > > > checkpoint
> > > > > > > > > > >    execution and recovery
> > > > > > > > > > >    - execution.savepoint.*: configurations for
> recovery from
> > > > > > > savepoint
> > > > > > > > > > >
> > > > > > > > > > > In addition, there are several individual options such
> as '
> > > > > > > > > > > *state.checkpoint-storage*' and
> '*state.checkpoints.dir*'
> > > > that
> > > > > > fall
> > > > > > > > > > outside
> > > > > > > > > > > of these prefixes. The current arrangement of these
> options,
> > > > > > which
> > > > > > > span
> > > > > > > > > > > multiple modules, is somewhat haphazard and lacks a
> > > > systematic
> > > > > > > > > structure.
> > > > > > > > > > > For example, the options under the
> '*CheckpointingOptions*'
> > > > > and '
> > > > > > > > > > > *ExecutionCheckpointingOptions*' are related and have
> no
> > > > clear
> > > > > > > > > boundaries
> > > > > > > > > > > from the user's perspective, but there is no unified
> prefix
> > > > for
> > > > > > > them.
> > > > > > > > > > With
> > > > > > > > > > > the upcoming release of Flink 2.0, we have an excellent
> > > > > > > opportunity to
> > > > > > > > > > > overhaul and restructure the configurations related to
> > > > > > > checkpointing,
> > > > > > > > > > > recovery, and state management. This FLIP proposes to
> > > > > reorganize
> > > > > > > these
> > > > > > > > > > > settings, making it more coherent by module, which
> would
> > > > > > > significantly
> > > > > > > > > > > lower the barriers for understanding and reduce the
> > > > development
> > > > > > > costs
> > > > > > > > > > > moving forward.
> > > > > > > > > > >
> > > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Zakelly
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Xuannan Su <su...@gmail.com>.
Hi Zakelly,

Thanks for driving this! The organization of the configuration option
in the FLIP looks much cleaner and easier to understand. +1 to the
FLIP.

Just some questions from me.

1. I think the change to the ConfigOptions should be put in the
`Public Interface` section, instead of `Proposed Changed`, as those
configuration options are public interface.

2. The key `state.checkpoint.cleaner.parallel-mode` seems confusing.
It feels like it is used to choose different modes. In fact, it is a
boolean flag to indicate whether to enable parallel clean. How about
making it `state.checkpoint.cleaner.parallel-mode.enabled`?

3. The `execution.checkpointing.write-buffer` may better be
`execution.checkpointing.write-buffer-size` so that we know it is
configuring the size of the buffer.

Best,
Xuannan


On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei <fr...@gmail.com> wrote:
>
> Hi Zakelly,
>
> > Considering the name occupation, how about naming it as `execution.checkpointing.type`?
>
> `Checkpoint Type`[1,2] is used to describe aligned/unaligned
> checkpoint, I am inclined to make a choice between
> `execution.checkpointing.incremental` and
> `execution.checkpointing.incremental.enabled`.
>
>
> [1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
> [2] https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27
>
> --
> Best,
> Yanfei
>
> Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
> >
> > Hi Lijie,
> >
> > Thanks for the reminder! I missed this.
> >
> > Considering the name occupation, how about naming it as
> > `execution.checkpointing.type`?
> >
> > Actually I think the current `execution.checkpointing.mode` is confusing in
> > some ways, maybe `execution.checkpointing.data-consistency` is better.
> >
> >
> > Best,
> > Zakelly
> >
> >
> > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <wa...@gmail.com>
> > wrote:
> >
> > > Hi Zakelly,
> > >
> > > >> I'm wondering if `execution.checkpointing.savepoint-dir` would be
> > > better.
> > >
> > > `execution.checkpointing.dir` and `execution.checkpointing.savepoint-dir`
> > > are also fine for me.
> > >
> > > >> So I think an enumeration option `execution.checkpointing.mode` which
> > > can be 'full' (default) or 'incremental' would be better
> > >
> > > I agree with using an enumeration option. But currently there is already a
> > > configuration option called `execution.checkpointing.mode`, which is used
> > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use another name
> > > or merge these two options.
> > >
> > > Best,
> > > Lijie
> > >
> > > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:
> > >
> > > > Hi everyone,
> > > >
> > > > Thanks all for your comments!
> > > >
> > > > @Yanfei
> > > >
> > > > > 1. For some state backends that do not support incremental checkpoint,
> > > > > how does the execution.checkpointing.incrementaloption take effect? Or
> > > > > is it better to put incremental under state.backend.xxx.incremental?
> > > > >
> > > > I'd rather not put the option for incremental checkpoint under the
> > > > 'state.backend', since it is more about the checkpointing instead of
> > > state
> > > > accessing. Of course, the state backend may not necessarily do
> > > incremental
> > > > checkpoint as requested. If the state backend is not capable of taking
> > > > incremental cp, it is better to fallback to the full cp.
> > > >
> > > > 2. I'm a little worried that putting all configurations into
> > > > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > > > problems. Some options would be used by flink-runtime module, but
> > > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > > FLINK-28286[1].
> > > > > So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
> > > > >
> > > >
> > > > Yes, that's a very good point.  Moving to
> > > > `CheckpointingOptions`(flink-core) makes sense.
> > > >
> > > > @Lijie
> > > >
> > > > How about
> > > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> > > >
> > > >
> > > > Actually, I think the `checkpointing.checkpoint` may cause some
> > > confusion.
> > > > But I'm ok if others agree.
> > > > I'm wondering if `execution.checkpointing.savepoint-dir` would be better.
> > > > WDYT?
> > > >
> > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > 'execution.checkpointing.local-copy.enabled'. Should we also add
> > > > "enabled"
> > > > > suffix for other boolean type configuration options ? For example,
> > > > > execution.checkpointing.incremental ->
> > > > > execution.checkpointing.incremental.enabled
> > > > >
> > > >
> > > > Actually, the incremental cp is something like choosing a mode for doing
> > > > checkpoint instead of enabling a function. So I think an enumeration
> > > option
> > > > `execution.checkpointing.mode` which can be 'full' (default) or
> > > > 'incremental' would be better, WDYT?
> > > > And @Rui Fan @Yanfei What do you think about this?
> > > >
> > > >
> > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <wa...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > Thanks for driving the discussion.
> > > > >
> > > > > 1.
> > > > > >> But I'm not so sure since there is only one savepoint-related
> > > option.
> > > > > Maybe someone else could share some thoughts here.
> > > > >
> > > > > How about
> > > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> > > > >
> > > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > > 'execution.checkpointing.local-copy.enabled'. Should we also add
> > > > "enabled"
> > > > > suffix for other boolean type configuration options ? For example,
> > > > > execution.checkpointing.incremental ->
> > > > > execution.checkpointing.incremental.enabled
> > > > >
> > > > > In this way, the naming style of configuration options is unified, and
> > > it
> > > > > can avoid potential similar problems (for example, we may need to add
> > > > more
> > > > > options for incremental checkpoint in the future).
> > > > >
> > > > > Best,
> > > > > Lijie
> > > > >
> > > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > Thank you for creating the FLIP and starting the discussion.
> > > > > >
> > > > > > The current arrangement of these options is indeed somewhat
> > > haphazard,
> > > > > > and the new arrangement looks much better. I have some questions
> > > about
> > > > > > the arrangement of some new configuration options:
> > > > > >
> > > > > > 1. For some state backends that do not support incremental
> > > checkpoint,
> > > > > > how does the execution.checkpointing.incrementaloption take effect?
> > > Or
> > > > > > is it better to put incremental under state.backend.xxx.incremental?
> > > > > >
> > > > > > 2. I'm a little worried that putting all configurations into
> > > > > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > > > > problems. Some options would be used by flink-runtime module, but
> > > > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > > > FLINK-28286[1].
> > > > > > So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Yanfei
> > > > > >
> > > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一 21:14写道:
> > > > > > >
> > > > > > > Hi Rui Fan and Junrui,
> > > > > > >
> > > > > > > Thanks for the reminder! I agree to change the
> > > > > > > 'execution.checkpointing.local-copy' to
> > > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > > >
> > > > > > > And for other suggestions Rui proposed:
> > > > > > >
> > > > > > > 1. How about execution.checkpointing.storage.type instead
> > > > > > > > of execution.checkpointing.storage?
> > > > > > >
> > > > > > >
> > > > > > > Ah, I missed something here. Actually I suggest we could merge the
> > > > > > current
> > > > > > > 'state.checkpoints.dir' and 'state.checkpoint-storage' into one URI
> > > > > > > configuration named 'execution.checkpointing.dir'. WDYT?
> > > > > > >
> > > > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > > > > >
> > > > > > >
> > > > > > > Yes, I think it is better to make 'savepoint' and 'checkpoint' the
> > > > same
> > > > > > > level. But I'm not so sure since there is only one
> > > savepoint-related
> > > > > > > option. Maybe someone else could share some thoughts here.
> > > > > > >
> > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > execution.recovery.mode?
> > > > > > > >
> > > > > > >
> > > > > > >  Agreed. That's more accurate.
> > > > > > >
> > > > > > >
> > > > > > > Many thanks for your suggestions!
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <jr...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zakelly,
> > > > > > > >
> > > > > > > > Thanks for driving this. I agree that the proposed restructuring
> > > of
> > > > > the
> > > > > > > > configuration options is largely positive. It will make
> > > > understanding
> > > > > > and
> > > > > > > > working with Flink configurations more intuitive.
> > > > > > > >
> > > > > > > > Most of the proposed changes look great. Just a heads-up, as Rui
> > > > Fan
> > > > > > > > mentioned, Flink currently requires that no configOption's key be
> > > > the
> > > > > > > > prefix of another to avoid issues when we eventually adopt a
> > > > standard
> > > > > > YAML
> > > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-29372). Therefore,
> > > > it's
> > > > > > better
> > > > > > > > to change the key 'execution.checkpointing.local-copy' because it
> > > > > > serves as
> > > > > > > > a prefix to the key 'execution.checkpointing.local-copy.dir'.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Junrui
> > > > > > > >
> > > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> > > > > > > >
> > > > > > > > > Hi Zakelly,
> > > > > > > > >
> > > > > > > > > Thank you for driving this proposal!
> > > > > > > > >
> > > > > > > > > Overall good for me. I have some questions about these names.
> > > > > > > > >
> > > > > > > > > 1. How about execution.checkpointing.storage.type instead of
> > > > > > > > > execution.checkpointing.storage?
> > > > > > > > >
> > > > > > > > > It's similar to state.backend.type.
> > > > > > > > >
> > > > > > > > > 2. How about execution.checkpointing.local-copy.enabled instead
> > > > of
> > > > > > > > > execution.checkpointing.local-copy?
> > > > > > > > >
> > > > > > > > > You added a new option: execution.checkpointing.local-copy.dir.
> > > > > > > > > IIUC, one option name shouldn't be the prefix of other options.
> > > > > > > > > If you add a new option execution.checkpointing.local-copy,
> > > > > > > > > flink CI will fail directly.
> > > > > > > > >
> > > > > > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > > > > > >
> > > > > > > > > For old options: state.savepoints.dir and
> > > state.checkpoints.dir,
> > > > > > > > > the savepoint and checkpoint are the same level. It means
> > > > > > > > > it's a checkpoint or savepoint.
> > > > > > > > >
> > > > > > > > > The new option execution.checkpointing.dir is fine for me.
> > > > > > > > > However, execution.checkpointing.savepoint.dir is a little
> > > weird.
> > > > > > > > > I don't know which name is better now. Let us think about it
> > > > more.
> > > > > > > > >
> > > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > > execution.recovery.mode?
> > > > > > > > >
> > > > > > > > > The meaning of mode is too broad. The claim-mode may
> > > > > > > > > be more accurate for users.
> > > > > > > > >
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Rui
> > > > > > > > >
> > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > > zakelly.lan@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi devs,
> > > > > > > > > >
> > > > > > > > > > I'd like to start a discussion on FLIP-406: Reorganize State
> > > &
> > > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > > >
> > > > > > > > > > Currently, the configuration options pertaining to
> > > > checkpointing,
> > > > > > > > > recovery,
> > > > > > > > > > and state management are primarily grouped under the
> > > following
> > > > > > > > prefixes:
> > > > > > > > > >
> > > > > > > > > >    - state.backend.* : configurations related to state
> > > > accessing
> > > > > > and
> > > > > > > > > >    checkpointing, as well as specific options for individual
> > > > > state
> > > > > > > > > backends
> > > > > > > > > >    - execution.checkpointing.* : configurations associated
> > > with
> > > > > > > > > checkpoint
> > > > > > > > > >    execution and recovery
> > > > > > > > > >    - execution.savepoint.*: configurations for recovery from
> > > > > > savepoint
> > > > > > > > > >
> > > > > > > > > > In addition, there are several individual options such as '
> > > > > > > > > > *state.checkpoint-storage*' and '*state.checkpoints.dir*'
> > > that
> > > > > fall
> > > > > > > > > outside
> > > > > > > > > > of these prefixes. The current arrangement of these options,
> > > > > which
> > > > > > span
> > > > > > > > > > multiple modules, is somewhat haphazard and lacks a
> > > systematic
> > > > > > > > structure.
> > > > > > > > > > For example, the options under the '*CheckpointingOptions*'
> > > > and '
> > > > > > > > > > *ExecutionCheckpointingOptions*' are related and have no
> > > clear
> > > > > > > > boundaries
> > > > > > > > > > from the user's perspective, but there is no unified prefix
> > > for
> > > > > > them.
> > > > > > > > > With
> > > > > > > > > > the upcoming release of Flink 2.0, we have an excellent
> > > > > > opportunity to
> > > > > > > > > > overhaul and restructure the configurations related to
> > > > > > checkpointing,
> > > > > > > > > > recovery, and state management. This FLIP proposes to
> > > > reorganize
> > > > > > these
> > > > > > > > > > settings, making it more coherent by module, which would
> > > > > > significantly
> > > > > > > > > > lower the barriers for understanding and reduce the
> > > development
> > > > > > costs
> > > > > > > > > > moving forward.
> > > > > > > > > >
> > > > > > > > > > Looking forward to hearing from you!
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Zakelly
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Yanfei Lei <fr...@gmail.com>.
Hi Zakelly,

> Considering the name occupation, how about naming it as `execution.checkpointing.type`?

`Checkpoint Type`[1,2] is used to describe aligned/unaligned
checkpoint, I am inclined to make a choice between
`execution.checkpointing.incremental` and
`execution.checkpointing.incremental.enabled`.


[1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/
[2] https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27

--
Best,
Yanfei

Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 14:41写道:
>
> Hi Lijie,
>
> Thanks for the reminder! I missed this.
>
> Considering the name occupation, how about naming it as
> `execution.checkpointing.type`?
>
> Actually I think the current `execution.checkpointing.mode` is confusing in
> some ways, maybe `execution.checkpointing.data-consistency` is better.
>
>
> Best,
> Zakelly
>
>
> On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <wa...@gmail.com>
> wrote:
>
> > Hi Zakelly,
> >
> > >> I'm wondering if `execution.checkpointing.savepoint-dir` would be
> > better.
> >
> > `execution.checkpointing.dir` and `execution.checkpointing.savepoint-dir`
> > are also fine for me.
> >
> > >> So I think an enumeration option `execution.checkpointing.mode` which
> > can be 'full' (default) or 'incremental' would be better
> >
> > I agree with using an enumeration option. But currently there is already a
> > configuration option called `execution.checkpointing.mode`, which is used
> > to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use another name
> > or merge these two options.
> >
> > Best,
> > Lijie
> >
> > Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:
> >
> > > Hi everyone,
> > >
> > > Thanks all for your comments!
> > >
> > > @Yanfei
> > >
> > > > 1. For some state backends that do not support incremental checkpoint,
> > > > how does the execution.checkpointing.incrementaloption take effect? Or
> > > > is it better to put incremental under state.backend.xxx.incremental?
> > > >
> > > I'd rather not put the option for incremental checkpoint under the
> > > 'state.backend', since it is more about the checkpointing instead of
> > state
> > > accessing. Of course, the state backend may not necessarily do
> > incremental
> > > checkpoint as requested. If the state backend is not capable of taking
> > > incremental cp, it is better to fallback to the full cp.
> > >
> > > 2. I'm a little worried that putting all configurations into
> > > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > > problems. Some options would be used by flink-runtime module, but
> > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > FLINK-28286[1].
> > > > So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
> > > >
> > >
> > > Yes, that's a very good point.  Moving to
> > > `CheckpointingOptions`(flink-core) makes sense.
> > >
> > > @Lijie
> > >
> > > How about
> > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> > >
> > >
> > > Actually, I think the `checkpointing.checkpoint` may cause some
> > confusion.
> > > But I'm ok if others agree.
> > > I'm wondering if `execution.checkpointing.savepoint-dir` would be better.
> > > WDYT?
> > >
> > > 2. We changed the execution.checkpointing.local-copy' to
> > > > 'execution.checkpointing.local-copy.enabled'. Should we also add
> > > "enabled"
> > > > suffix for other boolean type configuration options ? For example,
> > > > execution.checkpointing.incremental ->
> > > > execution.checkpointing.incremental.enabled
> > > >
> > >
> > > Actually, the incremental cp is something like choosing a mode for doing
> > > checkpoint instead of enabling a function. So I think an enumeration
> > option
> > > `execution.checkpointing.mode` which can be 'full' (default) or
> > > 'incremental' would be better, WDYT?
> > > And @Rui Fan @Yanfei What do you think about this?
> > >
> > >
> > > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <wa...@gmail.com>
> > > wrote:
> > >
> > > > Hi Zakelly,
> > > >
> > > > Thanks for driving the discussion.
> > > >
> > > > 1.
> > > > >> But I'm not so sure since there is only one savepoint-related
> > option.
> > > > Maybe someone else could share some thoughts here.
> > > >
> > > > How about
> > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> > > >
> > > > 2. We changed the execution.checkpointing.local-copy' to
> > > > 'execution.checkpointing.local-copy.enabled'. Should we also add
> > > "enabled"
> > > > suffix for other boolean type configuration options ? For example,
> > > > execution.checkpointing.incremental ->
> > > > execution.checkpointing.incremental.enabled
> > > >
> > > > In this way, the naming style of configuration options is unified, and
> > it
> > > > can avoid potential similar problems (for example, we may need to add
> > > more
> > > > options for incremental checkpoint in the future).
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > Thank you for creating the FLIP and starting the discussion.
> > > > >
> > > > > The current arrangement of these options is indeed somewhat
> > haphazard,
> > > > > and the new arrangement looks much better. I have some questions
> > about
> > > > > the arrangement of some new configuration options:
> > > > >
> > > > > 1. For some state backends that do not support incremental
> > checkpoint,
> > > > > how does the execution.checkpointing.incrementaloption take effect?
> > Or
> > > > > is it better to put incremental under state.backend.xxx.incremental?
> > > > >
> > > > > 2. I'm a little worried that putting all configurations into
> > > > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > > > problems. Some options would be used by flink-runtime module, but
> > > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > > FLINK-28286[1].
> > > > > So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > > > >
> > > > > --
> > > > > Best,
> > > > > Yanfei
> > > > >
> > > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一 21:14写道:
> > > > > >
> > > > > > Hi Rui Fan and Junrui,
> > > > > >
> > > > > > Thanks for the reminder! I agree to change the
> > > > > > 'execution.checkpointing.local-copy' to
> > > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > > >
> > > > > > And for other suggestions Rui proposed:
> > > > > >
> > > > > > 1. How about execution.checkpointing.storage.type instead
> > > > > > > of execution.checkpointing.storage?
> > > > > >
> > > > > >
> > > > > > Ah, I missed something here. Actually I suggest we could merge the
> > > > > current
> > > > > > 'state.checkpoints.dir' and 'state.checkpoint-storage' into one URI
> > > > > > configuration named 'execution.checkpointing.dir'. WDYT?
> > > > > >
> > > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > > > >
> > > > > >
> > > > > > Yes, I think it is better to make 'savepoint' and 'checkpoint' the
> > > same
> > > > > > level. But I'm not so sure since there is only one
> > savepoint-related
> > > > > > option. Maybe someone else could share some thoughts here.
> > > > > >
> > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > execution.recovery.mode?
> > > > > > >
> > > > > >
> > > > > >  Agreed. That's more accurate.
> > > > > >
> > > > > >
> > > > > > Many thanks for your suggestions!
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <jr...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Zakelly,
> > > > > > >
> > > > > > > Thanks for driving this. I agree that the proposed restructuring
> > of
> > > > the
> > > > > > > configuration options is largely positive. It will make
> > > understanding
> > > > > and
> > > > > > > working with Flink configurations more intuitive.
> > > > > > >
> > > > > > > Most of the proposed changes look great. Just a heads-up, as Rui
> > > Fan
> > > > > > > mentioned, Flink currently requires that no configOption's key be
> > > the
> > > > > > > prefix of another to avoid issues when we eventually adopt a
> > > standard
> > > > > YAML
> > > > > > > parser, as detailed in FLINK-29372 (
> > > > > > > https://issues.apache.org/jira/browse/FLINK-29372). Therefore,
> > > it's
> > > > > better
> > > > > > > to change the key 'execution.checkpointing.local-copy' because it
> > > > > serves as
> > > > > > > a prefix to the key 'execution.checkpointing.local-copy.dir'.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Junrui
> > > > > > >
> > > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> > > > > > >
> > > > > > > > Hi Zakelly,
> > > > > > > >
> > > > > > > > Thank you for driving this proposal!
> > > > > > > >
> > > > > > > > Overall good for me. I have some questions about these names.
> > > > > > > >
> > > > > > > > 1. How about execution.checkpointing.storage.type instead of
> > > > > > > > execution.checkpointing.storage?
> > > > > > > >
> > > > > > > > It's similar to state.backend.type.
> > > > > > > >
> > > > > > > > 2. How about execution.checkpointing.local-copy.enabled instead
> > > of
> > > > > > > > execution.checkpointing.local-copy?
> > > > > > > >
> > > > > > > > You added a new option: execution.checkpointing.local-copy.dir.
> > > > > > > > IIUC, one option name shouldn't be the prefix of other options.
> > > > > > > > If you add a new option execution.checkpointing.local-copy,
> > > > > > > > flink CI will fail directly.
> > > > > > > >
> > > > > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > > > > >
> > > > > > > > For old options: state.savepoints.dir and
> > state.checkpoints.dir,
> > > > > > > > the savepoint and checkpoint are the same level. It means
> > > > > > > > it's a checkpoint or savepoint.
> > > > > > > >
> > > > > > > > The new option execution.checkpointing.dir is fine for me.
> > > > > > > > However, execution.checkpointing.savepoint.dir is a little
> > weird.
> > > > > > > > I don't know which name is better now. Let us think about it
> > > more.
> > > > > > > >
> > > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > > execution.recovery.mode?
> > > > > > > >
> > > > > > > > The meaning of mode is too broad. The claim-mode may
> > > > > > > > be more accurate for users.
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Rui
> > > > > > > >
> > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > > zakelly.lan@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi devs,
> > > > > > > > >
> > > > > > > > > I'd like to start a discussion on FLIP-406: Reorganize State
> > &
> > > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > > >
> > > > > > > > > Currently, the configuration options pertaining to
> > > checkpointing,
> > > > > > > > recovery,
> > > > > > > > > and state management are primarily grouped under the
> > following
> > > > > > > prefixes:
> > > > > > > > >
> > > > > > > > >    - state.backend.* : configurations related to state
> > > accessing
> > > > > and
> > > > > > > > >    checkpointing, as well as specific options for individual
> > > > state
> > > > > > > > backends
> > > > > > > > >    - execution.checkpointing.* : configurations associated
> > with
> > > > > > > > checkpoint
> > > > > > > > >    execution and recovery
> > > > > > > > >    - execution.savepoint.*: configurations for recovery from
> > > > > savepoint
> > > > > > > > >
> > > > > > > > > In addition, there are several individual options such as '
> > > > > > > > > *state.checkpoint-storage*' and '*state.checkpoints.dir*'
> > that
> > > > fall
> > > > > > > > outside
> > > > > > > > > of these prefixes. The current arrangement of these options,
> > > > which
> > > > > span
> > > > > > > > > multiple modules, is somewhat haphazard and lacks a
> > systematic
> > > > > > > structure.
> > > > > > > > > For example, the options under the '*CheckpointingOptions*'
> > > and '
> > > > > > > > > *ExecutionCheckpointingOptions*' are related and have no
> > clear
> > > > > > > boundaries
> > > > > > > > > from the user's perspective, but there is no unified prefix
> > for
> > > > > them.
> > > > > > > > With
> > > > > > > > > the upcoming release of Flink 2.0, we have an excellent
> > > > > opportunity to
> > > > > > > > > overhaul and restructure the configurations related to
> > > > > checkpointing,
> > > > > > > > > recovery, and state management. This FLIP proposes to
> > > reorganize
> > > > > these
> > > > > > > > > settings, making it more coherent by module, which would
> > > > > significantly
> > > > > > > > > lower the barriers for understanding and reduce the
> > development
> > > > > costs
> > > > > > > > > moving forward.
> > > > > > > > >
> > > > > > > > > Looking forward to hearing from you!
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Zakelly
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi Lijie,

Thanks for the reminder! I missed this.

Considering the name occupation, how about naming it as
`execution.checkpointing.type`?

Actually I think the current `execution.checkpointing.mode` is confusing in
some ways, maybe `execution.checkpointing.data-consistency` is better.


Best,
Zakelly


On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang <wa...@gmail.com>
wrote:

> Hi Zakelly,
>
> >> I'm wondering if `execution.checkpointing.savepoint-dir` would be
> better.
>
> `execution.checkpointing.dir` and `execution.checkpointing.savepoint-dir`
> are also fine for me.
>
> >> So I think an enumeration option `execution.checkpointing.mode` which
> can be 'full' (default) or 'incremental' would be better
>
> I agree with using an enumeration option. But currently there is already a
> configuration option called `execution.checkpointing.mode`, which is used
> to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use another name
> or merge these two options.
>
> Best,
> Lijie
>
> Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:
>
> > Hi everyone,
> >
> > Thanks all for your comments!
> >
> > @Yanfei
> >
> > > 1. For some state backends that do not support incremental checkpoint,
> > > how does the execution.checkpointing.incrementaloption take effect? Or
> > > is it better to put incremental under state.backend.xxx.incremental?
> > >
> > I'd rather not put the option for incremental checkpoint under the
> > 'state.backend', since it is more about the checkpointing instead of
> state
> > accessing. Of course, the state backend may not necessarily do
> incremental
> > checkpoint as requested. If the state backend is not capable of taking
> > incremental cp, it is better to fallback to the full cp.
> >
> > 2. I'm a little worried that putting all configurations into
> > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > problems. Some options would be used by flink-runtime module, but
> > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > FLINK-28286[1].
> > > So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
> > >
> >
> > Yes, that's a very good point.  Moving to
> > `CheckpointingOptions`(flink-core) makes sense.
> >
> > @Lijie
> >
> > How about
> > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> >
> >
> > Actually, I think the `checkpointing.checkpoint` may cause some
> confusion.
> > But I'm ok if others agree.
> > I'm wondering if `execution.checkpointing.savepoint-dir` would be better.
> > WDYT?
> >
> > 2. We changed the execution.checkpointing.local-copy' to
> > > 'execution.checkpointing.local-copy.enabled'. Should we also add
> > "enabled"
> > > suffix for other boolean type configuration options ? For example,
> > > execution.checkpointing.incremental ->
> > > execution.checkpointing.incremental.enabled
> > >
> >
> > Actually, the incremental cp is something like choosing a mode for doing
> > checkpoint instead of enabling a function. So I think an enumeration
> option
> > `execution.checkpointing.mode` which can be 'full' (default) or
> > 'incremental' would be better, WDYT?
> > And @Rui Fan @Yanfei What do you think about this?
> >
> >
> > On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <wa...@gmail.com>
> > wrote:
> >
> > > Hi Zakelly,
> > >
> > > Thanks for driving the discussion.
> > >
> > > 1.
> > > >> But I'm not so sure since there is only one savepoint-related
> option.
> > > Maybe someone else could share some thoughts here.
> > >
> > > How about
> > > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> > >
> > > 2. We changed the execution.checkpointing.local-copy' to
> > > 'execution.checkpointing.local-copy.enabled'. Should we also add
> > "enabled"
> > > suffix for other boolean type configuration options ? For example,
> > > execution.checkpointing.incremental ->
> > > execution.checkpointing.incremental.enabled
> > >
> > > In this way, the naming style of configuration options is unified, and
> it
> > > can avoid potential similar problems (for example, we may need to add
> > more
> > > options for incremental checkpoint in the future).
> > >
> > > Best,
> > > Lijie
> > >
> > > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> > >
> > > > Hi Zakelly,
> > > >
> > > > Thank you for creating the FLIP and starting the discussion.
> > > >
> > > > The current arrangement of these options is indeed somewhat
> haphazard,
> > > > and the new arrangement looks much better. I have some questions
> about
> > > > the arrangement of some new configuration options:
> > > >
> > > > 1. For some state backends that do not support incremental
> checkpoint,
> > > > how does the execution.checkpointing.incrementaloption take effect?
> Or
> > > > is it better to put incremental under state.backend.xxx.incremental?
> > > >
> > > > 2. I'm a little worried that putting all configurations into
> > > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > > problems. Some options would be used by flink-runtime module, but
> > > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > > FLINK-28286[1].
> > > > So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > > >
> > > > --
> > > > Best,
> > > > Yanfei
> > > >
> > > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一 21:14写道:
> > > > >
> > > > > Hi Rui Fan and Junrui,
> > > > >
> > > > > Thanks for the reminder! I agree to change the
> > > > > 'execution.checkpointing.local-copy' to
> > > > > 'execution.checkpointing.local-copy.enabled'.
> > > > >
> > > > > And for other suggestions Rui proposed:
> > > > >
> > > > > 1. How about execution.checkpointing.storage.type instead
> > > > > > of execution.checkpointing.storage?
> > > > >
> > > > >
> > > > > Ah, I missed something here. Actually I suggest we could merge the
> > > > current
> > > > > 'state.checkpoints.dir' and 'state.checkpoint-storage' into one URI
> > > > > configuration named 'execution.checkpointing.dir'. WDYT?
> > > > >
> > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > > >
> > > > >
> > > > > Yes, I think it is better to make 'savepoint' and 'checkpoint' the
> > same
> > > > > level. But I'm not so sure since there is only one
> savepoint-related
> > > > > option. Maybe someone else could share some thoughts here.
> > > > >
> > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > execution.recovery.mode?
> > > > > >
> > > > >
> > > > >  Agreed. That's more accurate.
> > > > >
> > > > >
> > > > > Many thanks for your suggestions!
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <jr...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > Thanks for driving this. I agree that the proposed restructuring
> of
> > > the
> > > > > > configuration options is largely positive. It will make
> > understanding
> > > > and
> > > > > > working with Flink configurations more intuitive.
> > > > > >
> > > > > > Most of the proposed changes look great. Just a heads-up, as Rui
> > Fan
> > > > > > mentioned, Flink currently requires that no configOption's key be
> > the
> > > > > > prefix of another to avoid issues when we eventually adopt a
> > standard
> > > > YAML
> > > > > > parser, as detailed in FLINK-29372 (
> > > > > > https://issues.apache.org/jira/browse/FLINK-29372). Therefore,
> > it's
> > > > better
> > > > > > to change the key 'execution.checkpointing.local-copy' because it
> > > > serves as
> > > > > > a prefix to the key 'execution.checkpointing.local-copy.dir'.
> > > > > >
> > > > > > Best regards,
> > > > > > Junrui
> > > > > >
> > > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> > > > > >
> > > > > > > Hi Zakelly,
> > > > > > >
> > > > > > > Thank you for driving this proposal!
> > > > > > >
> > > > > > > Overall good for me. I have some questions about these names.
> > > > > > >
> > > > > > > 1. How about execution.checkpointing.storage.type instead of
> > > > > > > execution.checkpointing.storage?
> > > > > > >
> > > > > > > It's similar to state.backend.type.
> > > > > > >
> > > > > > > 2. How about execution.checkpointing.local-copy.enabled instead
> > of
> > > > > > > execution.checkpointing.local-copy?
> > > > > > >
> > > > > > > You added a new option: execution.checkpointing.local-copy.dir.
> > > > > > > IIUC, one option name shouldn't be the prefix of other options.
> > > > > > > If you add a new option execution.checkpointing.local-copy,
> > > > > > > flink CI will fail directly.
> > > > > > >
> > > > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > > > >
> > > > > > > For old options: state.savepoints.dir and
> state.checkpoints.dir,
> > > > > > > the savepoint and checkpoint are the same level. It means
> > > > > > > it's a checkpoint or savepoint.
> > > > > > >
> > > > > > > The new option execution.checkpointing.dir is fine for me.
> > > > > > > However, execution.checkpointing.savepoint.dir is a little
> weird.
> > > > > > > I don't know which name is better now. Let us think about it
> > more.
> > > > > > >
> > > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > > execution.recovery.mode?
> > > > > > >
> > > > > > > The meaning of mode is too broad. The claim-mode may
> > > > > > > be more accurate for users.
> > > > > > >
> > > > > > > WDYT?
> > > > > > >
> > > > > > > Best,
> > > > > > > Rui
> > > > > > >
> > > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> > zakelly.lan@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi devs,
> > > > > > > >
> > > > > > > > I'd like to start a discussion on FLIP-406: Reorganize State
> &
> > > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > > >
> > > > > > > > Currently, the configuration options pertaining to
> > checkpointing,
> > > > > > > recovery,
> > > > > > > > and state management are primarily grouped under the
> following
> > > > > > prefixes:
> > > > > > > >
> > > > > > > >    - state.backend.* : configurations related to state
> > accessing
> > > > and
> > > > > > > >    checkpointing, as well as specific options for individual
> > > state
> > > > > > > backends
> > > > > > > >    - execution.checkpointing.* : configurations associated
> with
> > > > > > > checkpoint
> > > > > > > >    execution and recovery
> > > > > > > >    - execution.savepoint.*: configurations for recovery from
> > > > savepoint
> > > > > > > >
> > > > > > > > In addition, there are several individual options such as '
> > > > > > > > *state.checkpoint-storage*' and '*state.checkpoints.dir*'
> that
> > > fall
> > > > > > > outside
> > > > > > > > of these prefixes. The current arrangement of these options,
> > > which
> > > > span
> > > > > > > > multiple modules, is somewhat haphazard and lacks a
> systematic
> > > > > > structure.
> > > > > > > > For example, the options under the '*CheckpointingOptions*'
> > and '
> > > > > > > > *ExecutionCheckpointingOptions*' are related and have no
> clear
> > > > > > boundaries
> > > > > > > > from the user's perspective, but there is no unified prefix
> for
> > > > them.
> > > > > > > With
> > > > > > > > the upcoming release of Flink 2.0, we have an excellent
> > > > opportunity to
> > > > > > > > overhaul and restructure the configurations related to
> > > > checkpointing,
> > > > > > > > recovery, and state management. This FLIP proposes to
> > reorganize
> > > > these
> > > > > > > > settings, making it more coherent by module, which would
> > > > significantly
> > > > > > > > lower the barriers for understanding and reduce the
> development
> > > > costs
> > > > > > > > moving forward.
> > > > > > > >
> > > > > > > > Looking forward to hearing from you!
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zakelly
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Lijie Wang <wa...@gmail.com>.
Hi Zakelly,

>> I'm wondering if `execution.checkpointing.savepoint-dir` would be better.

`execution.checkpointing.dir` and `execution.checkpointing.savepoint-dir`
are also fine for me.

>> So I think an enumeration option `execution.checkpointing.mode` which
can be 'full' (default) or 'incremental' would be better

I agree with using an enumeration option. But currently there is already a
configuration option called `execution.checkpointing.mode`, which is used
to choose EXACTLY_ONCE or AT_LEAST_ONCE. Maybe we need to use another name
or merge these two options.

Best,
Lijie

Zakelly Lan <za...@gmail.com> 于2023年12月27日周三 11:43写道:

> Hi everyone,
>
> Thanks all for your comments!
>
> @Yanfei
>
> > 1. For some state backends that do not support incremental checkpoint,
> > how does the execution.checkpointing.incrementaloption take effect? Or
> > is it better to put incremental under state.backend.xxx.incremental?
> >
> I'd rather not put the option for incremental checkpoint under the
> 'state.backend', since it is more about the checkpointing instead of state
> accessing. Of course, the state backend may not necessarily do incremental
> checkpoint as requested. If the state backend is not capable of taking
> incremental cp, it is better to fallback to the full cp.
>
> 2. I'm a little worried that putting all configurations into
> > `ExecutionCheckpointingOptions` will introduce some dependency
> > problems. Some options would be used by flink-runtime module, but
> > flink-runtime should not depend on flink-streaming-java. e.g.
> > FLINK-28286[1].
> > So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
> >
>
> Yes, that's a very good point.  Moving to
> `CheckpointingOptions`(flink-core) makes sense.
>
> @Lijie
>
> How about
> > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
>
>
> Actually, I think the `checkpointing.checkpoint` may cause some confusion.
> But I'm ok if others agree.
> I'm wondering if `execution.checkpointing.savepoint-dir` would be better.
> WDYT?
>
> 2. We changed the execution.checkpointing.local-copy' to
> > 'execution.checkpointing.local-copy.enabled'. Should we also add
> "enabled"
> > suffix for other boolean type configuration options ? For example,
> > execution.checkpointing.incremental ->
> > execution.checkpointing.incremental.enabled
> >
>
> Actually, the incremental cp is something like choosing a mode for doing
> checkpoint instead of enabling a function. So I think an enumeration option
> `execution.checkpointing.mode` which can be 'full' (default) or
> 'incremental' would be better, WDYT?
> And @Rui Fan @Yanfei What do you think about this?
>
>
> On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <wa...@gmail.com>
> wrote:
>
> > Hi Zakelly,
> >
> > Thanks for driving the discussion.
> >
> > 1.
> > >> But I'm not so sure since there is only one savepoint-related option.
> > Maybe someone else could share some thoughts here.
> >
> > How about
> > state.savepoints.dir -> execution.checkpointing.savepoint.dir
> > state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
> >
> > 2. We changed the execution.checkpointing.local-copy' to
> > 'execution.checkpointing.local-copy.enabled'. Should we also add
> "enabled"
> > suffix for other boolean type configuration options ? For example,
> > execution.checkpointing.incremental ->
> > execution.checkpointing.incremental.enabled
> >
> > In this way, the naming style of configuration options is unified, and it
> > can avoid potential similar problems (for example, we may need to add
> more
> > options for incremental checkpoint in the future).
> >
> > Best,
> > Lijie
> >
> > Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
> >
> > > Hi Zakelly,
> > >
> > > Thank you for creating the FLIP and starting the discussion.
> > >
> > > The current arrangement of these options is indeed somewhat haphazard,
> > > and the new arrangement looks much better. I have some questions about
> > > the arrangement of some new configuration options:
> > >
> > > 1. For some state backends that do not support incremental checkpoint,
> > > how does the execution.checkpointing.incrementaloption take effect? Or
> > > is it better to put incremental under state.backend.xxx.incremental?
> > >
> > > 2. I'm a little worried that putting all configurations into
> > > `ExecutionCheckpointingOptions` will introduce some dependency
> > > problems. Some options would be used by flink-runtime module, but
> > > flink-runtime should not depend on flink-streaming-java. e.g.
> > > FLINK-28286[1].
> > > So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-28286
> > >
> > > --
> > > Best,
> > > Yanfei
> > >
> > > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一 21:14写道:
> > > >
> > > > Hi Rui Fan and Junrui,
> > > >
> > > > Thanks for the reminder! I agree to change the
> > > > 'execution.checkpointing.local-copy' to
> > > > 'execution.checkpointing.local-copy.enabled'.
> > > >
> > > > And for other suggestions Rui proposed:
> > > >
> > > > 1. How about execution.checkpointing.storage.type instead
> > > > > of execution.checkpointing.storage?
> > > >
> > > >
> > > > Ah, I missed something here. Actually I suggest we could merge the
> > > current
> > > > 'state.checkpoints.dir' and 'state.checkpoint-storage' into one URI
> > > > configuration named 'execution.checkpointing.dir'. WDYT?
> > > >
> > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > >
> > > >
> > > > Yes, I think it is better to make 'savepoint' and 'checkpoint' the
> same
> > > > level. But I'm not so sure since there is only one savepoint-related
> > > > option. Maybe someone else could share some thoughts here.
> > > >
> > > > 4. How about execution.recovery.claim-mode instead of
> > > > > execution.recovery.mode?
> > > > >
> > > >
> > > >  Agreed. That's more accurate.
> > > >
> > > >
> > > > Many thanks for your suggestions!
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <jr...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > Thanks for driving this. I agree that the proposed restructuring of
> > the
> > > > > configuration options is largely positive. It will make
> understanding
> > > and
> > > > > working with Flink configurations more intuitive.
> > > > >
> > > > > Most of the proposed changes look great. Just a heads-up, as Rui
> Fan
> > > > > mentioned, Flink currently requires that no configOption's key be
> the
> > > > > prefix of another to avoid issues when we eventually adopt a
> standard
> > > YAML
> > > > > parser, as detailed in FLINK-29372 (
> > > > > https://issues.apache.org/jira/browse/FLINK-29372). Therefore,
> it's
> > > better
> > > > > to change the key 'execution.checkpointing.local-copy' because it
> > > serves as
> > > > > a prefix to the key 'execution.checkpointing.local-copy.dir'.
> > > > >
> > > > > Best regards,
> > > > > Junrui
> > > > >
> > > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > Thank you for driving this proposal!
> > > > > >
> > > > > > Overall good for me. I have some questions about these names.
> > > > > >
> > > > > > 1. How about execution.checkpointing.storage.type instead of
> > > > > > execution.checkpointing.storage?
> > > > > >
> > > > > > It's similar to state.backend.type.
> > > > > >
> > > > > > 2. How about execution.checkpointing.local-copy.enabled instead
> of
> > > > > > execution.checkpointing.local-copy?
> > > > > >
> > > > > > You added a new option: execution.checkpointing.local-copy.dir.
> > > > > > IIUC, one option name shouldn't be the prefix of other options.
> > > > > > If you add a new option execution.checkpointing.local-copy,
> > > > > > flink CI will fail directly.
> > > > > >
> > > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > > >
> > > > > > For old options: state.savepoints.dir and state.checkpoints.dir,
> > > > > > the savepoint and checkpoint are the same level. It means
> > > > > > it's a checkpoint or savepoint.
> > > > > >
> > > > > > The new option execution.checkpointing.dir is fine for me.
> > > > > > However, execution.checkpointing.savepoint.dir is a little weird.
> > > > > > I don't know which name is better now. Let us think about it
> more.
> > > > > >
> > > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > > execution.recovery.mode?
> > > > > >
> > > > > > The meaning of mode is too broad. The claim-mode may
> > > > > > be more accurate for users.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Best,
> > > > > > Rui
> > > > > >
> > > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <
> zakelly.lan@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi devs,
> > > > > > >
> > > > > > > I'd like to start a discussion on FLIP-406: Reorganize State &
> > > > > > > Checkpointing & Recovery Configuration[1].
> > > > > > >
> > > > > > > Currently, the configuration options pertaining to
> checkpointing,
> > > > > > recovery,
> > > > > > > and state management are primarily grouped under the following
> > > > > prefixes:
> > > > > > >
> > > > > > >    - state.backend.* : configurations related to state
> accessing
> > > and
> > > > > > >    checkpointing, as well as specific options for individual
> > state
> > > > > > backends
> > > > > > >    - execution.checkpointing.* : configurations associated with
> > > > > > checkpoint
> > > > > > >    execution and recovery
> > > > > > >    - execution.savepoint.*: configurations for recovery from
> > > savepoint
> > > > > > >
> > > > > > > In addition, there are several individual options such as '
> > > > > > > *state.checkpoint-storage*' and '*state.checkpoints.dir*' that
> > fall
> > > > > > outside
> > > > > > > of these prefixes. The current arrangement of these options,
> > which
> > > span
> > > > > > > multiple modules, is somewhat haphazard and lacks a systematic
> > > > > structure.
> > > > > > > For example, the options under the '*CheckpointingOptions*'
> and '
> > > > > > > *ExecutionCheckpointingOptions*' are related and have no clear
> > > > > boundaries
> > > > > > > from the user's perspective, but there is no unified prefix for
> > > them.
> > > > > > With
> > > > > > > the upcoming release of Flink 2.0, we have an excellent
> > > opportunity to
> > > > > > > overhaul and restructure the configurations related to
> > > checkpointing,
> > > > > > > recovery, and state management. This FLIP proposes to
> reorganize
> > > these
> > > > > > > settings, making it more coherent by module, which would
> > > significantly
> > > > > > > lower the barriers for understanding and reduce the development
> > > costs
> > > > > > > moving forward.
> > > > > > >
> > > > > > > Looking forward to hearing from you!
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi everyone,

Thanks all for your comments!

@Yanfei

> 1. For some state backends that do not support incremental checkpoint,
> how does the execution.checkpointing.incrementaloption take effect? Or
> is it better to put incremental under state.backend.xxx.incremental?
>
I'd rather not put the option for incremental checkpoint under the
'state.backend', since it is more about the checkpointing instead of state
accessing. Of course, the state backend may not necessarily do incremental
checkpoint as requested. If the state backend is not capable of taking
incremental cp, it is better to fallback to the full cp.

2. I'm a little worried that putting all configurations into
> `ExecutionCheckpointingOptions` will introduce some dependency
> problems. Some options would be used by flink-runtime module, but
> flink-runtime should not depend on flink-streaming-java. e.g.
> FLINK-28286[1].
> So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
>

Yes, that's a very good point.  Moving to
`CheckpointingOptions`(flink-core) makes sense.

@Lijie

How about
> state.savepoints.dir -> execution.checkpointing.savepoint.dir
> state.checkpoints.dir -> execution.checkpointing.checkpoint.dir


Actually, I think the `checkpointing.checkpoint` may cause some confusion.
But I'm ok if others agree.
I'm wondering if `execution.checkpointing.savepoint-dir` would be better.
WDYT?

2. We changed the execution.checkpointing.local-copy' to
> 'execution.checkpointing.local-copy.enabled'. Should we also add "enabled"
> suffix for other boolean type configuration options ? For example,
> execution.checkpointing.incremental ->
> execution.checkpointing.incremental.enabled
>

Actually, the incremental cp is something like choosing a mode for doing
checkpoint instead of enabling a function. So I think an enumeration option
`execution.checkpointing.mode` which can be 'full' (default) or
'incremental' would be better, WDYT?
And @Rui Fan @Yanfei What do you think about this?


On Tue, Dec 26, 2023 at 5:15 PM Lijie Wang <wa...@gmail.com> wrote:

> Hi Zakelly,
>
> Thanks for driving the discussion.
>
> 1.
> >> But I'm not so sure since there is only one savepoint-related option.
> Maybe someone else could share some thoughts here.
>
> How about
> state.savepoints.dir -> execution.checkpointing.savepoint.dir
> state.checkpoints.dir -> execution.checkpointing.checkpoint.dir
>
> 2. We changed the execution.checkpointing.local-copy' to
> 'execution.checkpointing.local-copy.enabled'. Should we also add "enabled"
> suffix for other boolean type configuration options ? For example,
> execution.checkpointing.incremental ->
> execution.checkpointing.incremental.enabled
>
> In this way, the naming style of configuration options is unified, and it
> can avoid potential similar problems (for example, we may need to add more
> options for incremental checkpoint in the future).
>
> Best,
> Lijie
>
> Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:
>
> > Hi Zakelly,
> >
> > Thank you for creating the FLIP and starting the discussion.
> >
> > The current arrangement of these options is indeed somewhat haphazard,
> > and the new arrangement looks much better. I have some questions about
> > the arrangement of some new configuration options:
> >
> > 1. For some state backends that do not support incremental checkpoint,
> > how does the execution.checkpointing.incrementaloption take effect? Or
> > is it better to put incremental under state.backend.xxx.incremental?
> >
> > 2. I'm a little worried that putting all configurations into
> > `ExecutionCheckpointingOptions` will introduce some dependency
> > problems. Some options would be used by flink-runtime module, but
> > flink-runtime should not depend on flink-streaming-java. e.g.
> > FLINK-28286[1].
> > So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-28286
> >
> > --
> > Best,
> > Yanfei
> >
> > Zakelly Lan <za...@gmail.com> 于2023年12月25日周一 21:14写道:
> > >
> > > Hi Rui Fan and Junrui,
> > >
> > > Thanks for the reminder! I agree to change the
> > > 'execution.checkpointing.local-copy' to
> > > 'execution.checkpointing.local-copy.enabled'.
> > >
> > > And for other suggestions Rui proposed:
> > >
> > > 1. How about execution.checkpointing.storage.type instead
> > > > of execution.checkpointing.storage?
> > >
> > >
> > > Ah, I missed something here. Actually I suggest we could merge the
> > current
> > > 'state.checkpoints.dir' and 'state.checkpoint-storage' into one URI
> > > configuration named 'execution.checkpointing.dir'. WDYT?
> > >
> > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > >
> > >
> > > Yes, I think it is better to make 'savepoint' and 'checkpoint' the same
> > > level. But I'm not so sure since there is only one savepoint-related
> > > option. Maybe someone else could share some thoughts here.
> > >
> > > 4. How about execution.recovery.claim-mode instead of
> > > > execution.recovery.mode?
> > > >
> > >
> > >  Agreed. That's more accurate.
> > >
> > >
> > > Many thanks for your suggestions!
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <jr...@gmail.com>
> wrote:
> > >
> > > > Hi Zakelly,
> > > >
> > > > Thanks for driving this. I agree that the proposed restructuring of
> the
> > > > configuration options is largely positive. It will make understanding
> > and
> > > > working with Flink configurations more intuitive.
> > > >
> > > > Most of the proposed changes look great. Just a heads-up, as Rui Fan
> > > > mentioned, Flink currently requires that no configOption's key be the
> > > > prefix of another to avoid issues when we eventually adopt a standard
> > YAML
> > > > parser, as detailed in FLINK-29372 (
> > > > https://issues.apache.org/jira/browse/FLINK-29372). Therefore, it's
> > better
> > > > to change the key 'execution.checkpointing.local-copy' because it
> > serves as
> > > > a prefix to the key 'execution.checkpointing.local-copy.dir'.
> > > >
> > > > Best regards,
> > > > Junrui
> > > >
> > > > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > Thank you for driving this proposal!
> > > > >
> > > > > Overall good for me. I have some questions about these names.
> > > > >
> > > > > 1. How about execution.checkpointing.storage.type instead of
> > > > > execution.checkpointing.storage?
> > > > >
> > > > > It's similar to state.backend.type.
> > > > >
> > > > > 2. How about execution.checkpointing.local-copy.enabled instead of
> > > > > execution.checkpointing.local-copy?
> > > > >
> > > > > You added a new option: execution.checkpointing.local-copy.dir.
> > > > > IIUC, one option name shouldn't be the prefix of other options.
> > > > > If you add a new option execution.checkpointing.local-copy,
> > > > > flink CI will fail directly.
> > > > >
> > > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > > >
> > > > > For old options: state.savepoints.dir and state.checkpoints.dir,
> > > > > the savepoint and checkpoint are the same level. It means
> > > > > it's a checkpoint or savepoint.
> > > > >
> > > > > The new option execution.checkpointing.dir is fine for me.
> > > > > However, execution.checkpointing.savepoint.dir is a little weird.
> > > > > I don't know which name is better now. Let us think about it more.
> > > > >
> > > > > 4. How about execution.recovery.claim-mode instead of
> > > > > execution.recovery.mode?
> > > > >
> > > > > The meaning of mode is too broad. The claim-mode may
> > > > > be more accurate for users.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Best,
> > > > > Rui
> > > > >
> > > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <zakelly.lan@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > I'd like to start a discussion on FLIP-406: Reorganize State &
> > > > > > Checkpointing & Recovery Configuration[1].
> > > > > >
> > > > > > Currently, the configuration options pertaining to checkpointing,
> > > > > recovery,
> > > > > > and state management are primarily grouped under the following
> > > > prefixes:
> > > > > >
> > > > > >    - state.backend.* : configurations related to state accessing
> > and
> > > > > >    checkpointing, as well as specific options for individual
> state
> > > > > backends
> > > > > >    - execution.checkpointing.* : configurations associated with
> > > > > checkpoint
> > > > > >    execution and recovery
> > > > > >    - execution.savepoint.*: configurations for recovery from
> > savepoint
> > > > > >
> > > > > > In addition, there are several individual options such as '
> > > > > > *state.checkpoint-storage*' and '*state.checkpoints.dir*' that
> fall
> > > > > outside
> > > > > > of these prefixes. The current arrangement of these options,
> which
> > span
> > > > > > multiple modules, is somewhat haphazard and lacks a systematic
> > > > structure.
> > > > > > For example, the options under the '*CheckpointingOptions*' and '
> > > > > > *ExecutionCheckpointingOptions*' are related and have no clear
> > > > boundaries
> > > > > > from the user's perspective, but there is no unified prefix for
> > them.
> > > > > With
> > > > > > the upcoming release of Flink 2.0, we have an excellent
> > opportunity to
> > > > > > overhaul and restructure the configurations related to
> > checkpointing,
> > > > > > recovery, and state management. This FLIP proposes to reorganize
> > these
> > > > > > settings, making it more coherent by module, which would
> > significantly
> > > > > > lower the barriers for understanding and reduce the development
> > costs
> > > > > > moving forward.
> > > > > >
> > > > > > Looking forward to hearing from you!
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > >
> > > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Lijie Wang <wa...@gmail.com>.
Hi Zakelly,

Thanks for driving the discussion.

1.
>> But I'm not so sure since there is only one savepoint-related option.
Maybe someone else could share some thoughts here.

How about
state.savepoints.dir -> execution.checkpointing.savepoint.dir
state.checkpoints.dir -> execution.checkpointing.checkpoint.dir

2. We changed the execution.checkpointing.local-copy' to
'execution.checkpointing.local-copy.enabled'. Should we also add "enabled"
suffix for other boolean type configuration options ? For example,
execution.checkpointing.incremental ->
execution.checkpointing.incremental.enabled

In this way, the naming style of configuration options is unified, and it
can avoid potential similar problems (for example, we may need to add more
options for incremental checkpoint in the future).

Best,
Lijie

Yanfei Lei <fr...@gmail.com> 于2023年12月26日周二 12:05写道:

> Hi Zakelly,
>
> Thank you for creating the FLIP and starting the discussion.
>
> The current arrangement of these options is indeed somewhat haphazard,
> and the new arrangement looks much better. I have some questions about
> the arrangement of some new configuration options:
>
> 1. For some state backends that do not support incremental checkpoint,
> how does the execution.checkpointing.incrementaloption take effect? Or
> is it better to put incremental under state.backend.xxx.incremental?
>
> 2. I'm a little worried that putting all configurations into
> `ExecutionCheckpointingOptions` will introduce some dependency
> problems. Some options would be used by flink-runtime module, but
> flink-runtime should not depend on flink-streaming-java. e.g.
> FLINK-28286[1].
> So, I prefer to move configurations to `CheckpointingOptions`, WDYT?
>
> [1] https://issues.apache.org/jira/browse/FLINK-28286
>
> --
> Best,
> Yanfei
>
> Zakelly Lan <za...@gmail.com> 于2023年12月25日周一 21:14写道:
> >
> > Hi Rui Fan and Junrui,
> >
> > Thanks for the reminder! I agree to change the
> > 'execution.checkpointing.local-copy' to
> > 'execution.checkpointing.local-copy.enabled'.
> >
> > And for other suggestions Rui proposed:
> >
> > 1. How about execution.checkpointing.storage.type instead
> > > of execution.checkpointing.storage?
> >
> >
> > Ah, I missed something here. Actually I suggest we could merge the
> current
> > 'state.checkpoints.dir' and 'state.checkpoint-storage' into one URI
> > configuration named 'execution.checkpointing.dir'. WDYT?
> >
> > 3. execution.checkpointing.savepoint.dir is a little weird.
> > >
> >
> > Yes, I think it is better to make 'savepoint' and 'checkpoint' the same
> > level. But I'm not so sure since there is only one savepoint-related
> > option. Maybe someone else could share some thoughts here.
> >
> > 4. How about execution.recovery.claim-mode instead of
> > > execution.recovery.mode?
> > >
> >
> >  Agreed. That's more accurate.
> >
> >
> > Many thanks for your suggestions!
> >
> > Best,
> > Zakelly
> >
> > On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <jr...@gmail.com> wrote:
> >
> > > Hi Zakelly,
> > >
> > > Thanks for driving this. I agree that the proposed restructuring of the
> > > configuration options is largely positive. It will make understanding
> and
> > > working with Flink configurations more intuitive.
> > >
> > > Most of the proposed changes look great. Just a heads-up, as Rui Fan
> > > mentioned, Flink currently requires that no configOption's key be the
> > > prefix of another to avoid issues when we eventually adopt a standard
> YAML
> > > parser, as detailed in FLINK-29372 (
> > > https://issues.apache.org/jira/browse/FLINK-29372). Therefore, it's
> better
> > > to change the key 'execution.checkpointing.local-copy' because it
> serves as
> > > a prefix to the key 'execution.checkpointing.local-copy.dir'.
> > >
> > > Best regards,
> > > Junrui
> > >
> > > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> > >
> > > > Hi Zakelly,
> > > >
> > > > Thank you for driving this proposal!
> > > >
> > > > Overall good for me. I have some questions about these names.
> > > >
> > > > 1. How about execution.checkpointing.storage.type instead of
> > > > execution.checkpointing.storage?
> > > >
> > > > It's similar to state.backend.type.
> > > >
> > > > 2. How about execution.checkpointing.local-copy.enabled instead of
> > > > execution.checkpointing.local-copy?
> > > >
> > > > You added a new option: execution.checkpointing.local-copy.dir.
> > > > IIUC, one option name shouldn't be the prefix of other options.
> > > > If you add a new option execution.checkpointing.local-copy,
> > > > flink CI will fail directly.
> > > >
> > > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > > >
> > > > For old options: state.savepoints.dir and state.checkpoints.dir,
> > > > the savepoint and checkpoint are the same level. It means
> > > > it's a checkpoint or savepoint.
> > > >
> > > > The new option execution.checkpointing.dir is fine for me.
> > > > However, execution.checkpointing.savepoint.dir is a little weird.
> > > > I don't know which name is better now. Let us think about it more.
> > > >
> > > > 4. How about execution.recovery.claim-mode instead of
> > > > execution.recovery.mode?
> > > >
> > > > The meaning of mode is too broad. The claim-mode may
> > > > be more accurate for users.
> > > >
> > > > WDYT?
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <za...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I'd like to start a discussion on FLIP-406: Reorganize State &
> > > > > Checkpointing & Recovery Configuration[1].
> > > > >
> > > > > Currently, the configuration options pertaining to checkpointing,
> > > > recovery,
> > > > > and state management are primarily grouped under the following
> > > prefixes:
> > > > >
> > > > >    - state.backend.* : configurations related to state accessing
> and
> > > > >    checkpointing, as well as specific options for individual state
> > > > backends
> > > > >    - execution.checkpointing.* : configurations associated with
> > > > checkpoint
> > > > >    execution and recovery
> > > > >    - execution.savepoint.*: configurations for recovery from
> savepoint
> > > > >
> > > > > In addition, there are several individual options such as '
> > > > > *state.checkpoint-storage*' and '*state.checkpoints.dir*' that fall
> > > > outside
> > > > > of these prefixes. The current arrangement of these options, which
> span
> > > > > multiple modules, is somewhat haphazard and lacks a systematic
> > > structure.
> > > > > For example, the options under the '*CheckpointingOptions*' and '
> > > > > *ExecutionCheckpointingOptions*' are related and have no clear
> > > boundaries
> > > > > from the user's perspective, but there is no unified prefix for
> them.
> > > > With
> > > > > the upcoming release of Flink 2.0, we have an excellent
> opportunity to
> > > > > overhaul and restructure the configurations related to
> checkpointing,
> > > > > recovery, and state management. This FLIP proposes to reorganize
> these
> > > > > settings, making it more coherent by module, which would
> significantly
> > > > > lower the barriers for understanding and reduce the development
> costs
> > > > > moving forward.
> > > > >
> > > > > Looking forward to hearing from you!
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > >
> > >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Yanfei Lei <fr...@gmail.com>.
Hi Zakelly,

Thank you for creating the FLIP and starting the discussion.

The current arrangement of these options is indeed somewhat haphazard,
and the new arrangement looks much better. I have some questions about
the arrangement of some new configuration options:

1. For some state backends that do not support incremental checkpoint,
how does the execution.checkpointing.incrementaloption take effect? Or
is it better to put incremental under state.backend.xxx.incremental?

2. I'm a little worried that putting all configurations into
`ExecutionCheckpointingOptions` will introduce some dependency
problems. Some options would be used by flink-runtime module, but
flink-runtime should not depend on flink-streaming-java. e.g.
FLINK-28286[1].
So, I prefer to move configurations to `CheckpointingOptions`, WDYT?

[1] https://issues.apache.org/jira/browse/FLINK-28286

--
Best,
Yanfei

Zakelly Lan <za...@gmail.com> 于2023年12月25日周一 21:14写道:
>
> Hi Rui Fan and Junrui,
>
> Thanks for the reminder! I agree to change the
> 'execution.checkpointing.local-copy' to
> 'execution.checkpointing.local-copy.enabled'.
>
> And for other suggestions Rui proposed:
>
> 1. How about execution.checkpointing.storage.type instead
> > of execution.checkpointing.storage?
>
>
> Ah, I missed something here. Actually I suggest we could merge the current
> 'state.checkpoints.dir' and 'state.checkpoint-storage' into one URI
> configuration named 'execution.checkpointing.dir'. WDYT?
>
> 3. execution.checkpointing.savepoint.dir is a little weird.
> >
>
> Yes, I think it is better to make 'savepoint' and 'checkpoint' the same
> level. But I'm not so sure since there is only one savepoint-related
> option. Maybe someone else could share some thoughts here.
>
> 4. How about execution.recovery.claim-mode instead of
> > execution.recovery.mode?
> >
>
>  Agreed. That's more accurate.
>
>
> Many thanks for your suggestions!
>
> Best,
> Zakelly
>
> On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <jr...@gmail.com> wrote:
>
> > Hi Zakelly,
> >
> > Thanks for driving this. I agree that the proposed restructuring of the
> > configuration options is largely positive. It will make understanding and
> > working with Flink configurations more intuitive.
> >
> > Most of the proposed changes look great. Just a heads-up, as Rui Fan
> > mentioned, Flink currently requires that no configOption's key be the
> > prefix of another to avoid issues when we eventually adopt a standard YAML
> > parser, as detailed in FLINK-29372 (
> > https://issues.apache.org/jira/browse/FLINK-29372). Therefore, it's better
> > to change the key 'execution.checkpointing.local-copy' because it serves as
> > a prefix to the key 'execution.checkpointing.local-copy.dir'.
> >
> > Best regards,
> > Junrui
> >
> > Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
> >
> > > Hi Zakelly,
> > >
> > > Thank you for driving this proposal!
> > >
> > > Overall good for me. I have some questions about these names.
> > >
> > > 1. How about execution.checkpointing.storage.type instead of
> > > execution.checkpointing.storage?
> > >
> > > It's similar to state.backend.type.
> > >
> > > 2. How about execution.checkpointing.local-copy.enabled instead of
> > > execution.checkpointing.local-copy?
> > >
> > > You added a new option: execution.checkpointing.local-copy.dir.
> > > IIUC, one option name shouldn't be the prefix of other options.
> > > If you add a new option execution.checkpointing.local-copy,
> > > flink CI will fail directly.
> > >
> > > 3. execution.checkpointing.savepoint.dir is a little weird.
> > >
> > > For old options: state.savepoints.dir and state.checkpoints.dir,
> > > the savepoint and checkpoint are the same level. It means
> > > it's a checkpoint or savepoint.
> > >
> > > The new option execution.checkpointing.dir is fine for me.
> > > However, execution.checkpointing.savepoint.dir is a little weird.
> > > I don't know which name is better now. Let us think about it more.
> > >
> > > 4. How about execution.recovery.claim-mode instead of
> > > execution.recovery.mode?
> > >
> > > The meaning of mode is too broad. The claim-mode may
> > > be more accurate for users.
> > >
> > > WDYT?
> > >
> > > Best,
> > > Rui
> > >
> > > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <za...@gmail.com>
> > wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I'd like to start a discussion on FLIP-406: Reorganize State &
> > > > Checkpointing & Recovery Configuration[1].
> > > >
> > > > Currently, the configuration options pertaining to checkpointing,
> > > recovery,
> > > > and state management are primarily grouped under the following
> > prefixes:
> > > >
> > > >    - state.backend.* : configurations related to state accessing and
> > > >    checkpointing, as well as specific options for individual state
> > > backends
> > > >    - execution.checkpointing.* : configurations associated with
> > > checkpoint
> > > >    execution and recovery
> > > >    - execution.savepoint.*: configurations for recovery from savepoint
> > > >
> > > > In addition, there are several individual options such as '
> > > > *state.checkpoint-storage*' and '*state.checkpoints.dir*' that fall
> > > outside
> > > > of these prefixes. The current arrangement of these options, which span
> > > > multiple modules, is somewhat haphazard and lacks a systematic
> > structure.
> > > > For example, the options under the '*CheckpointingOptions*' and '
> > > > *ExecutionCheckpointingOptions*' are related and have no clear
> > boundaries
> > > > from the user's perspective, but there is no unified prefix for them.
> > > With
> > > > the upcoming release of Flink 2.0, we have an excellent opportunity to
> > > > overhaul and restructure the configurations related to checkpointing,
> > > > recovery, and state management. This FLIP proposes to reorganize these
> > > > settings, making it more coherent by module, which would significantly
> > > > lower the barriers for understanding and reduce the development costs
> > > > moving forward.
> > > >
> > > > Looking forward to hearing from you!
> > > >
> > > > [1]
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > >
> >

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Zakelly Lan <za...@gmail.com>.
Hi Rui Fan and Junrui,

Thanks for the reminder! I agree to change the
'execution.checkpointing.local-copy' to
'execution.checkpointing.local-copy.enabled'.

And for other suggestions Rui proposed:

1. How about execution.checkpointing.storage.type instead
> of execution.checkpointing.storage?


Ah, I missed something here. Actually I suggest we could merge the current
'state.checkpoints.dir' and 'state.checkpoint-storage' into one URI
configuration named 'execution.checkpointing.dir'. WDYT?

3. execution.checkpointing.savepoint.dir is a little weird.
>

Yes, I think it is better to make 'savepoint' and 'checkpoint' the same
level. But I'm not so sure since there is only one savepoint-related
option. Maybe someone else could share some thoughts here.

4. How about execution.recovery.claim-mode instead of
> execution.recovery.mode?
>

 Agreed. That's more accurate.


Many thanks for your suggestions!

Best,
Zakelly

On Mon, Dec 25, 2023 at 8:18 PM Junrui Lee <jr...@gmail.com> wrote:

> Hi Zakelly,
>
> Thanks for driving this. I agree that the proposed restructuring of the
> configuration options is largely positive. It will make understanding and
> working with Flink configurations more intuitive.
>
> Most of the proposed changes look great. Just a heads-up, as Rui Fan
> mentioned, Flink currently requires that no configOption's key be the
> prefix of another to avoid issues when we eventually adopt a standard YAML
> parser, as detailed in FLINK-29372 (
> https://issues.apache.org/jira/browse/FLINK-29372). Therefore, it's better
> to change the key 'execution.checkpointing.local-copy' because it serves as
> a prefix to the key 'execution.checkpointing.local-copy.dir'.
>
> Best regards,
> Junrui
>
> Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:
>
> > Hi Zakelly,
> >
> > Thank you for driving this proposal!
> >
> > Overall good for me. I have some questions about these names.
> >
> > 1. How about execution.checkpointing.storage.type instead of
> > execution.checkpointing.storage?
> >
> > It's similar to state.backend.type.
> >
> > 2. How about execution.checkpointing.local-copy.enabled instead of
> > execution.checkpointing.local-copy?
> >
> > You added a new option: execution.checkpointing.local-copy.dir.
> > IIUC, one option name shouldn't be the prefix of other options.
> > If you add a new option execution.checkpointing.local-copy,
> > flink CI will fail directly.
> >
> > 3. execution.checkpointing.savepoint.dir is a little weird.
> >
> > For old options: state.savepoints.dir and state.checkpoints.dir,
> > the savepoint and checkpoint are the same level. It means
> > it's a checkpoint or savepoint.
> >
> > The new option execution.checkpointing.dir is fine for me.
> > However, execution.checkpointing.savepoint.dir is a little weird.
> > I don't know which name is better now. Let us think about it more.
> >
> > 4. How about execution.recovery.claim-mode instead of
> > execution.recovery.mode?
> >
> > The meaning of mode is too broad. The claim-mode may
> > be more accurate for users.
> >
> > WDYT?
> >
> > Best,
> > Rui
> >
> > On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <za...@gmail.com>
> wrote:
> >
> > > Hi devs,
> > >
> > > I'd like to start a discussion on FLIP-406: Reorganize State &
> > > Checkpointing & Recovery Configuration[1].
> > >
> > > Currently, the configuration options pertaining to checkpointing,
> > recovery,
> > > and state management are primarily grouped under the following
> prefixes:
> > >
> > >    - state.backend.* : configurations related to state accessing and
> > >    checkpointing, as well as specific options for individual state
> > backends
> > >    - execution.checkpointing.* : configurations associated with
> > checkpoint
> > >    execution and recovery
> > >    - execution.savepoint.*: configurations for recovery from savepoint
> > >
> > > In addition, there are several individual options such as '
> > > *state.checkpoint-storage*' and '*state.checkpoints.dir*' that fall
> > outside
> > > of these prefixes. The current arrangement of these options, which span
> > > multiple modules, is somewhat haphazard and lacks a systematic
> structure.
> > > For example, the options under the '*CheckpointingOptions*' and '
> > > *ExecutionCheckpointingOptions*' are related and have no clear
> boundaries
> > > from the user's perspective, but there is no unified prefix for them.
> > With
> > > the upcoming release of Flink 2.0, we have an excellent opportunity to
> > > overhaul and restructure the configurations related to checkpointing,
> > > recovery, and state management. This FLIP proposes to reorganize these
> > > settings, making it more coherent by module, which would significantly
> > > lower the barriers for understanding and reduce the development costs
> > > moving forward.
> > >
> > > Looking forward to hearing from you!
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > >
> > > Best,
> > > Zakelly
> > >
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Junrui Lee <jr...@gmail.com>.
Hi Zakelly,

Thanks for driving this. I agree that the proposed restructuring of the
configuration options is largely positive. It will make understanding and
working with Flink configurations more intuitive.

Most of the proposed changes look great. Just a heads-up, as Rui Fan
mentioned, Flink currently requires that no configOption's key be the
prefix of another to avoid issues when we eventually adopt a standard YAML
parser, as detailed in FLINK-29372 (
https://issues.apache.org/jira/browse/FLINK-29372). Therefore, it's better
to change the key 'execution.checkpointing.local-copy' because it serves as
a prefix to the key 'execution.checkpointing.local-copy.dir'.

Best regards,
Junrui

Rui Fan <19...@gmail.com> 于2023年12月25日周一 19:11写道:

> Hi Zakelly,
>
> Thank you for driving this proposal!
>
> Overall good for me. I have some questions about these names.
>
> 1. How about execution.checkpointing.storage.type instead of
> execution.checkpointing.storage?
>
> It's similar to state.backend.type.
>
> 2. How about execution.checkpointing.local-copy.enabled instead of
> execution.checkpointing.local-copy?
>
> You added a new option: execution.checkpointing.local-copy.dir.
> IIUC, one option name shouldn't be the prefix of other options.
> If you add a new option execution.checkpointing.local-copy,
> flink CI will fail directly.
>
> 3. execution.checkpointing.savepoint.dir is a little weird.
>
> For old options: state.savepoints.dir and state.checkpoints.dir,
> the savepoint and checkpoint are the same level. It means
> it's a checkpoint or savepoint.
>
> The new option execution.checkpointing.dir is fine for me.
> However, execution.checkpointing.savepoint.dir is a little weird.
> I don't know which name is better now. Let us think about it more.
>
> 4. How about execution.recovery.claim-mode instead of
> execution.recovery.mode?
>
> The meaning of mode is too broad. The claim-mode may
> be more accurate for users.
>
> WDYT?
>
> Best,
> Rui
>
> On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi devs,
> >
> > I'd like to start a discussion on FLIP-406: Reorganize State &
> > Checkpointing & Recovery Configuration[1].
> >
> > Currently, the configuration options pertaining to checkpointing,
> recovery,
> > and state management are primarily grouped under the following prefixes:
> >
> >    - state.backend.* : configurations related to state accessing and
> >    checkpointing, as well as specific options for individual state
> backends
> >    - execution.checkpointing.* : configurations associated with
> checkpoint
> >    execution and recovery
> >    - execution.savepoint.*: configurations for recovery from savepoint
> >
> > In addition, there are several individual options such as '
> > *state.checkpoint-storage*' and '*state.checkpoints.dir*' that fall
> outside
> > of these prefixes. The current arrangement of these options, which span
> > multiple modules, is somewhat haphazard and lacks a systematic structure.
> > For example, the options under the '*CheckpointingOptions*' and '
> > *ExecutionCheckpointingOptions*' are related and have no clear boundaries
> > from the user's perspective, but there is no unified prefix for them.
> With
> > the upcoming release of Flink 2.0, we have an excellent opportunity to
> > overhaul and restructure the configurations related to checkpointing,
> > recovery, and state management. This FLIP proposes to reorganize these
> > settings, making it more coherent by module, which would significantly
> > lower the barriers for understanding and reduce the development costs
> > moving forward.
> >
> > Looking forward to hearing from you!
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> >
> > Best,
> > Zakelly
> >
>

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Posted by Rui Fan <19...@gmail.com>.
Hi Zakelly,

Thank you for driving this proposal!

Overall good for me. I have some questions about these names.

1. How about execution.checkpointing.storage.type instead of
execution.checkpointing.storage?

It's similar to state.backend.type.

2. How about execution.checkpointing.local-copy.enabled instead of
execution.checkpointing.local-copy?

You added a new option: execution.checkpointing.local-copy.dir.
IIUC, one option name shouldn't be the prefix of other options.
If you add a new option execution.checkpointing.local-copy,
flink CI will fail directly.

3. execution.checkpointing.savepoint.dir is a little weird.

For old options: state.savepoints.dir and state.checkpoints.dir,
the savepoint and checkpoint are the same level. It means
it's a checkpoint or savepoint.

The new option execution.checkpointing.dir is fine for me.
However, execution.checkpointing.savepoint.dir is a little weird.
I don't know which name is better now. Let us think about it more.

4. How about execution.recovery.claim-mode instead of
execution.recovery.mode?

The meaning of mode is too broad. The claim-mode may
be more accurate for users.

WDYT?

Best,
Rui

On Mon, Dec 25, 2023 at 5:14 PM Zakelly Lan <za...@gmail.com> wrote:

> Hi devs,
>
> I'd like to start a discussion on FLIP-406: Reorganize State &
> Checkpointing & Recovery Configuration[1].
>
> Currently, the configuration options pertaining to checkpointing, recovery,
> and state management are primarily grouped under the following prefixes:
>
>    - state.backend.* : configurations related to state accessing and
>    checkpointing, as well as specific options for individual state backends
>    - execution.checkpointing.* : configurations associated with checkpoint
>    execution and recovery
>    - execution.savepoint.*: configurations for recovery from savepoint
>
> In addition, there are several individual options such as '
> *state.checkpoint-storage*' and '*state.checkpoints.dir*' that fall outside
> of these prefixes. The current arrangement of these options, which span
> multiple modules, is somewhat haphazard and lacks a systematic structure.
> For example, the options under the '*CheckpointingOptions*' and '
> *ExecutionCheckpointingOptions*' are related and have no clear boundaries
> from the user's perspective, but there is no unified prefix for them. With
> the upcoming release of Flink 2.0, we have an excellent opportunity to
> overhaul and restructure the configurations related to checkpointing,
> recovery, and state management. This FLIP proposes to reorganize these
> settings, making it more coherent by module, which would significantly
> lower the barriers for understanding and reduce the development costs
> moving forward.
>
> Looking forward to hearing from you!
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
>
> Best,
> Zakelly
>