You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Fabian Paul (Jira)" <ji...@apache.org> on 2022/11/29 11:02:00 UTC
[jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint
Fabian Paul created FLINK-30238:
-----------------------------------
Summary: Unified Sink committer does not clean up state on final savepoint
Key: FLINK-30238
URL: https://issues.apache.org/jira/browse/FLINK-30238
Project: Flink
Issue Type: Bug
Components: Connectors / Common
Affects Versions: 1.15.3, 1.17.0, 1.16.1
Reporter: Fabian Paul
During stop-with-savepoint the committer only commits the pending committables on notifyCheckpointComplete.
This has several downsides.
* Last committableSummary has checkpoint id LONG.MAX and is never cleared from the state leading to that stop-with-savepoint does not work when the pipeline recovers from a savepoint
* While the committables are committed during stop-with-savepoint they are not forwarded to post-commit topology, potentially losing data and preventing to close open transactions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint
Posted by Fabian Paul <fp...@apache.org>.
Yes, the StreamingFileSink is not affected.
Best,
Fabian
Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint
Posted by Galen Warren <ga...@cvillewarrens.com>.
Thanks Fabian.
Do you know if the legacy StreamingFileSink has the same issues? Not asking
you to do research but just wondering if you happen to know.
On Tue, Nov 29, 2022 at 8:46 AM Fabian Paul <fp...@apache.org> wrote:
> Hi folks,
>
> I did some initial investigation, and the problem seems twofold.
>
> If no post-commit topology is used, we do not run into a problem where
> we could lose data but since we do not clean up the state correctly,
> we will hit this [1] when trying to stop the pipeline with a savepoint
> after we have started it from a savepoint.
> AFAICT all two-phase commit sinks are affected Kafka, File etc.
>
> For sinks using the post-commit topology, the same applies.
> Additionally, we might never do the commit from the post-commit
> topology resulting in lost data.
>
> Best,
> Fabian
>
> [1]
> https://github.com/apache/flink/blob/ed46cb2fd64f1cb306ae5b7654d2b4d64ab69f22/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java#L83
>
Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint
Posted by Fabian Paul <fp...@apache.org>.
Hi folks,
I did some initial investigation, and the problem seems twofold.
If no post-commit topology is used, we do not run into a problem where
we could lose data but since we do not clean up the state correctly,
we will hit this [1] when trying to stop the pipeline with a savepoint
after we have started it from a savepoint.
AFAICT all two-phase commit sinks are affected Kafka, File etc.
For sinks using the post-commit topology, the same applies.
Additionally, we might never do the commit from the post-commit
topology resulting in lost data.
Best,
Fabian
[1] https://github.com/apache/flink/blob/ed46cb2fd64f1cb306ae5b7654d2b4d64ab69f22/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java#L83
Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint
Posted by Martijn Visser <ma...@apache.org>.
Let's loop in Fabian to clarify. I'm not sure if this only occurs when
using a post-commit topology (like compaction), but he can definitely
clarify :)
On Tue, Nov 29, 2022 at 2:19 PM Galen Warren <ga...@cvillewarrens.com>
wrote:
> This seems scary -- am I interpreting it correctly to mean that unified
> FileSink doesn't work properly with jobs that need to be
> stopped-with-savepoints and restarted?
>
> Should one use the deprecated StreamingFileSink until this is resolved?
>
> On Tue, Nov 29, 2022 at 6:02 AM Fabian Paul (Jira) <ji...@apache.org>
> wrote:
>
> > Fabian Paul created FLINK-30238:
> > -----------------------------------
> >
> > Summary: Unified Sink committer does not clean up state on
> > final savepoint
> > Key: FLINK-30238
> > URL: https://issues.apache.org/jira/browse/FLINK-30238
> > Project: Flink
> > Issue Type: Bug
> > Components: Connectors / Common
> > Affects Versions: 1.15.3, 1.17.0, 1.16.1
> > Reporter: Fabian Paul
> >
> >
> > During stop-with-savepoint the committer only commits the pending
> > committables on notifyCheckpointComplete.
> >
> > This has several downsides.
> > * Last committableSummary has checkpoint id LONG.MAX and is never
> cleared
> > from the state leading to that stop-with-savepoint does not work when the
> > pipeline recovers from a savepoint
> > * While the committables are committed during stop-with-savepoint they
> > are not forwarded to post-commit topology, potentially losing data and
> > preventing to close open transactions.
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.20.10#820010)
> >
>
Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint
Posted by Galen Warren <ga...@cvillewarrens.com>.
This seems scary -- am I interpreting it correctly to mean that unified
FileSink doesn't work properly with jobs that need to be
stopped-with-savepoints and restarted?
Should one use the deprecated StreamingFileSink until this is resolved?
On Tue, Nov 29, 2022 at 6:02 AM Fabian Paul (Jira) <ji...@apache.org> wrote:
> Fabian Paul created FLINK-30238:
> -----------------------------------
>
> Summary: Unified Sink committer does not clean up state on
> final savepoint
> Key: FLINK-30238
> URL: https://issues.apache.org/jira/browse/FLINK-30238
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
> Affects Versions: 1.15.3, 1.17.0, 1.16.1
> Reporter: Fabian Paul
>
>
> During stop-with-savepoint the committer only commits the pending
> committables on notifyCheckpointComplete.
>
> This has several downsides.
> * Last committableSummary has checkpoint id LONG.MAX and is never cleared
> from the state leading to that stop-with-savepoint does not work when the
> pipeline recovers from a savepoint
> * While the committables are committed during stop-with-savepoint they
> are not forwarded to post-commit topology, potentially losing data and
> preventing to close open transactions.
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.20.10#820010)
>