You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Zakelly Lan <za...@gmail.com> on 2026/09/03 15:26:36 UTC
[DISCUSS] Promote Unified File Merging (FLIP-306) to PublicEvolving and Deprecate FLINK-26803
Hi everyone,
The unified file merging mechanism introduced by FLIP-306 [1] has been
available since Flink 1.20.0, which was released on August 2, 2024 [2].
Over the past two years, we have received much feedback from real-world use
cases and fixed a number of issues. The feature has now reached a good
level of stability.
In the FLIP-600 discussion [3], Rui Fan suggested using FLIP-306 to handle
channel-state file merging as well, allowing us to deprecate the existing
mechanism introduced under FLINK-26803 [4]. With FLINK-39977 [5], the final
piece required to support merged channel-state recovery, now merged, I
believe we are ready to take the next step.
I would therefore like to propose:
1. Promoting the configurations introduced by FLIP-306 from
@Experimental to @PublicEvolving.
2. Deprecating the file-merging mechanism introduced under FLINK-26803
in favor of the unified mechanism.
FLIP-306 has now been available for more than two years and across multiple
release cycles, so this promotion would be consistent with Flink’s API
stability graduation process [6]. Consolidating on the unified mechanism
would also reduce duplication and provide a single file-merging solution
for both checkpoint data and channel state.
Looking forward to your thoughts.
Best,
Zakelly
[1] https://cwiki.apache.org/confluence/x/DwsNDw
[2]
https://flink.apache.org/2024/08/02/announcing-the-release-of-apache-flink-1.20/
[3] https://lists.apache.org/thread/qpztk0jdpcmhomszjx63l53xv26xnmwf
[4] https://issues.apache.org/jira/browse/FLINK-26803
[5] https://issues.apache.org/jira/browse/FLINK-39977
[6] https://cwiki.apache.org/confluence/x/J5eqCw
Re: [DISCUSS] Promote Unified File Merging (FLIP-306) to PublicEvolving and Deprecate FLINK-26803
Posted by Zhe Wang <ja...@gmail.com>.
Hi Zakelly,
For the comparison tests, would it be useful to distinguish three cases:
unified merging disabled with the current channel-state limit of 5, unified
merging enabled with that limit still at 5, and unified merging enabled with
the old limit set to 1? The current write path can use both mechanisms
together,
so the second case alone would not isolate the replacement of the old
mechanism.
I would initially keep across-checkpoint-boundary merging disabled in all
applicable cases to avoid changing another dimension of the comparison.
It may also help to keep two migration checks separate: how an existing
configuration containing the old key is handled, and whether checkpoints
written by the old path still restore correctly, including after rescaling.
Accepting an old key as a no-op would address the first but would not by
itself
establish the second. I have checked the current configuration and writer
paths, but have not run a performance or cross-version restore comparison.
Best regards,
Zhe Wang
Zakelly Lan <za...@gmail.com> 于2026年9月8日周二 00:17写道：
> Hi Rui,
>
> Agreed, I think we can have the deprecation and
> 'execution.checkpointing.file-merging.enabled' activated by default
> together. Let me do more testing to ensure an experience that is no worse
> than the previous approach.
>
>
> Best,
> Zakelly
>
> On Sat, Sep 5, 2026 at 8:31 PM Rui Fan <19...@gmail.com> wrote:
>
> > Hey Zakelly,
> >
> > Thank you for driving it!
> >
> > > I would therefore like to propose:
> > >
> > > 1. Promoting the configurations introduced by FLIP-306
> > from @Experimental to @PublicEvolving.
> > > 2. Deprecating the file-merging mechanism introduced under
> FLINK-26803
> > in favor of the unified mechanism.
> >
> > Sounds make sense to me. Only one minor comment here:
> > is it possible to enable execution.checkpointing.file-merging.enabled by
> > default
> > while removing
> > execution.checkpointing.unaligned.max-subtasks-per-channel-state-file?
> > Since the default value is 5 right now.
> >
> > If we are confident in the unified file-merging feature, I think enabling
> > it by default
> > would be a good choice. It provides users with a production configuration
> > that
> > works right out of the box.
> >
> > Best,
> > Rui
> >
> >
> > On Thu, Sep 3, 2026 at 5:26 PM Zakelly Lan <za...@gmail.com>
> wrote:
> >
> > > Hi everyone,
> > >
> > > The unified file merging mechanism introduced by FLIP-306 [1] has been
> > > available since Flink 1.20.0, which was released on August 2, 2024 [2].
> > > Over the past two years, we have received much feedback from real-world
> > use
> > > cases and fixed a number of issues. The feature has now reached a good
> > > level of stability.
> > >
> > > In the FLIP-600 discussion [3], Rui Fan suggested using FLIP-306 to
> > handle
> > > channel-state file merging as well, allowing us to deprecate the
> existing
> > > mechanism introduced under FLINK-26803 [4]. With FLINK-39977 [5], the
> > final
> > > piece required to support merged channel-state recovery, now merged, I
> > > believe we are ready to take the next step.
> > >
> > > I would therefore like to propose:
> > >
> > > 1. Promoting the configurations introduced by FLIP-306 from
> > > @Experimental to @PublicEvolving.
> > > 2. Deprecating the file-merging mechanism introduced under
> FLINK-26803
> > > in favor of the unified mechanism.
> > >
> > > FLIP-306 has now been available for more than two years and across
> > multiple
> > > release cycles, so this promotion would be consistent with Flink’s API
> > > stability graduation process [6]. Consolidating on the unified
> mechanism
> > > would also reduce duplication and provide a single file-merging
> solution
> > > for both checkpoint data and channel state.
> > >
> > > Looking forward to your thoughts.
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > [1] https://cwiki.apache.org/confluence/x/DwsNDw
> > > [2]
> > >
> > >
> >
> https://flink.apache.org/2024/08/02/announcing-the-release-of-apache-flink-1.20/
> > > [3] https://lists.apache.org/thread/qpztk0jdpcmhomszjx63l53xv26xnmwf
> > > [4] https://issues.apache.org/jira/browse/FLINK-26803
> > > [5] https://issues.apache.org/jira/browse/FLINK-39977
> > > [6] https://cwiki.apache.org/confluence/x/J5eqCw
> > >
> >
>
Re: [DISCUSS] Promote Unified File Merging (FLIP-306) to PublicEvolving and Deprecate FLINK-26803
Posted by Zakelly Lan <za...@gmail.com>.
Hi Rui,
Agreed, I think we can have the deprecation and
'execution.checkpointing.file-merging.enabled' activated by default
together. Let me do more testing to ensure an experience that is no worse
than the previous approach.
Best,
Zakelly
On Sat, Sep 5, 2026 at 8:31 PM Rui Fan <19...@gmail.com> wrote:
> Hey Zakelly,
>
> Thank you for driving it!
>
> > I would therefore like to propose:
> >
> > 1. Promoting the configurations introduced by FLIP-306
> from @Experimental to @PublicEvolving.
> > 2. Deprecating the file-merging mechanism introduced under FLINK-26803
> in favor of the unified mechanism.
>
> Sounds make sense to me. Only one minor comment here:
> is it possible to enable execution.checkpointing.file-merging.enabled by
> default
> while removing
> execution.checkpointing.unaligned.max-subtasks-per-channel-state-file?
> Since the default value is 5 right now.
>
> If we are confident in the unified file-merging feature, I think enabling
> it by default
> would be a good choice. It provides users with a production configuration
> that
> works right out of the box.
>
> Best,
> Rui
>
>
> On Thu, Sep 3, 2026 at 5:26 PM Zakelly Lan <za...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > The unified file merging mechanism introduced by FLIP-306 [1] has been
> > available since Flink 1.20.0, which was released on August 2, 2024 [2].
> > Over the past two years, we have received much feedback from real-world
> use
> > cases and fixed a number of issues. The feature has now reached a good
> > level of stability.
> >
> > In the FLIP-600 discussion [3], Rui Fan suggested using FLIP-306 to
> handle
> > channel-state file merging as well, allowing us to deprecate the existing
> > mechanism introduced under FLINK-26803 [4]. With FLINK-39977 [5], the
> final
> > piece required to support merged channel-state recovery, now merged, I
> > believe we are ready to take the next step.
> >
> > I would therefore like to propose:
> >
> > 1. Promoting the configurations introduced by FLIP-306 from
> > @Experimental to @PublicEvolving.
> > 2. Deprecating the file-merging mechanism introduced under FLINK-26803
> > in favor of the unified mechanism.
> >
> > FLIP-306 has now been available for more than two years and across
> multiple
> > release cycles, so this promotion would be consistent with Flink’s API
> > stability graduation process [6]. Consolidating on the unified mechanism
> > would also reduce duplication and provide a single file-merging solution
> > for both checkpoint data and channel state.
> >
> > Looking forward to your thoughts.
> >
> >
> > Best,
> > Zakelly
> >
> > [1] https://cwiki.apache.org/confluence/x/DwsNDw
> > [2]
> >
> >
> https://flink.apache.org/2024/08/02/announcing-the-release-of-apache-flink-1.20/
> > [3] https://lists.apache.org/thread/qpztk0jdpcmhomszjx63l53xv26xnmwf
> > [4] https://issues.apache.org/jira/browse/FLINK-26803
> > [5] https://issues.apache.org/jira/browse/FLINK-39977
> > [6] https://cwiki.apache.org/confluence/x/J5eqCw
> >
>
Re: [DISCUSS] Promote Unified File Merging (FLIP-306) to PublicEvolving and Deprecate FLINK-26803
Posted by Rui Fan <19...@gmail.com>.
Hey Zakelly,
Thank you for driving it!
> I would therefore like to propose:
>
> 1. Promoting the configurations introduced by FLIP-306
from @Experimental to @PublicEvolving.
> 2. Deprecating the file-merging mechanism introduced under FLINK-26803
in favor of the unified mechanism.
Sounds make sense to me. Only one minor comment here:
is it possible to enable execution.checkpointing.file-merging.enabled by
default
while removing
execution.checkpointing.unaligned.max-subtasks-per-channel-state-file?
Since the default value is 5 right now.
If we are confident in the unified file-merging feature, I think enabling
it by default
would be a good choice. It provides users with a production configuration
that
works right out of the box.
Best,
Rui
On Thu, Sep 3, 2026 at 5:26 PM Zakelly Lan <za...@gmail.com> wrote:
> Hi everyone,
>
> The unified file merging mechanism introduced by FLIP-306 [1] has been
> available since Flink 1.20.0, which was released on August 2, 2024 [2].
> Over the past two years, we have received much feedback from real-world use
> cases and fixed a number of issues. The feature has now reached a good
> level of stability.
>
> In the FLIP-600 discussion [3], Rui Fan suggested using FLIP-306 to handle
> channel-state file merging as well, allowing us to deprecate the existing
> mechanism introduced under FLINK-26803 [4]. With FLINK-39977 [5], the final
> piece required to support merged channel-state recovery, now merged, I
> believe we are ready to take the next step.
>
> I would therefore like to propose:
>
> 1. Promoting the configurations introduced by FLIP-306 from
> @Experimental to @PublicEvolving.
> 2. Deprecating the file-merging mechanism introduced under FLINK-26803
> in favor of the unified mechanism.
>
> FLIP-306 has now been available for more than two years and across multiple
> release cycles, so this promotion would be consistent with Flink’s API
> stability graduation process [6]. Consolidating on the unified mechanism
> would also reduce duplication and provide a single file-merging solution
> for both checkpoint data and channel state.
>
> Looking forward to your thoughts.
>
>
> Best,
> Zakelly
>
> [1] https://cwiki.apache.org/confluence/x/DwsNDw
> [2]
>
> https://flink.apache.org/2024/08/02/announcing-the-release-of-apache-flink-1.20/
> [3] https://lists.apache.org/thread/qpztk0jdpcmhomszjx63l53xv26xnmwf
> [4] https://issues.apache.org/jira/browse/FLINK-26803
> [5] https://issues.apache.org/jira/browse/FLINK-39977
> [6] https://cwiki.apache.org/confluence/x/J5eqCw
>