You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@paimon.apache.org by Yong Fang <zj...@gmail.com> on 2025/01/23 09:44:20 UTC
[DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Hi devs,
I would like to start a discussion about PIP-30: Improvement For Paimon
Committer In Flink [1].
Currently Flink writes data to Paimon based on Two-Phase Commit which will
generate a global committer node and connect all tasks in one region. If
any task fails, it will lead to a global failover in Flink job.
To solve this issue, we would like to introduce a Paimon Writer Coordinator
to perform table commit operation, enabling Flink paimon jobs to support
region failover and improving stability.
Looking forward to hearing from you, thanks!
[1]
https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
Best,
Fang Yong
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Xintong Song <to...@gmail.com>.
Thanks for the clarification, Yong. Sounds good to me.
Best,
Xintong
On Wed, Feb 12, 2025 at 10:15 AM wj wang <ho...@gmail.com> wrote:
> To @Yanquan Lv
> The concept of 'Region' is in Blink or old Flink version.
> Now the 'Region' is means `Restart Pipelined Region Failover Strategy` in
> Flink.
>
> On Wed, Feb 12, 2025 at 9:35 AM Yanquan Lv <de...@gmail.com> wrote:
> >
> > Hi Fang Yong. Thanks for driving it.
> > Adding an alternative way to complete the global commit is good for me.
> >
> > I still have a confusion similar with xiangyu feng,
> > > This committer node will connect all the tasks in Flink job together,
> and all the tasks are within one region. As a result, when any task fails,
> it will trigger a global failover of the Flink job.
> > Why would the failure of a subtask have impact on other subtasks? The
> explanation on how to form a region in the Flink document is not
> particularly detailed. Can you explain more on it.
> >
> >
> > > 2025年2月11日 16:55,Yong Fang <zj...@gmail.com> 写道:
> > >
> > > Thanks for all the feedback.
> > >
> > > To @xiangyu feng:
> > > 1. Regarding region failover, I'm referring to the `Restart Pipelined
> > > Region Failover Strategy` [1] in Flink.
> > > 2. This feature mainly enables support for region failover, enhancing
> > > stability. Of course, in practical use, users can configure the JM to
> > > provide more suitable resources for the committer node.
> > >
> > > To @Xintong Song:
> > > It doesn't matter if the notifyComplete of one checkpoint fails to be
> > > called. Since the data will be stored in the HDFS path of the
> coordinator,
> > > when the notifyComplete of the next CP is called, the data files
> generated
> > > by the two CPs will be merged to create a Paimon snapshot.
> > >
> > > To @Yunfeng Zhou:
> > > The pressure on the JM (JobManager) depends on two aspects:
> > > Message communication pressure: It is determined by the checkpoint (CP)
> > > interval and the number of sinks. Each sink generates one message in
> each
> > > CP. If there are a total of 5000 sinks, the JM will receive 5000
> messages
> > > in one CP, which may take about a few seconds.
> > > Pressure from executing table commits: It is necessary to create a
> Paimon
> > > snapshot, and compaction may even be triggered. Therefore, asynchronous
> > > threads should be considered for these operations to avoid blocking the
> > > main thread of the JM.
> > >
> > > To @Jingsong and @wj wang
> > > When using Flink SinkV2 to write to Paimon, a global committer node
> will
> > > also be generated in the physical execution plan to create and manage
> > > Paimon snapshots. Therefore, the same problem arises
> > >
> > > [1]
> > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/#restart-pipelined-region-failover-strategy
> > >
> > > Best,
> > > Fang Yong
> > >
> > > On Tue, Feb 11, 2025 at 11:26 AM wj wang <ho...@gmail.com> wrote:
> > >
> > >> Hi Yong, thanks for driving this PIP.
> > >> I have a small question:
> > >> Why not use Flink SinkV2 instead of moving the committer logic into
> > >> JM's OperatorCoordinator?
> > >>
> > >> On Tue, Feb 11, 2025 at 10:34 AM Jingsong Li <ji...@gmail.com>
> > >> wrote:
> > >>>
> > >>> Thanks Yong for driving this PIP!
> > >>>
> > >>>>> Currently, there will be a global committer node in Flink Paimon
> job
> > >> which is used to ensure the consistency of written data in Paimon.
> This
> > >> committer node will connect all the tasks in Flink job together, and
> all
> > >> the tasks are within one region. As a result, when any task fails, it
> will
> > >> trigger a global failover of the Flink job. We use HDFS as the remote
> > >> storage, and we often encounter situations where the global failover
> of
> > >> jobs is triggered due to write timeouts or errors when writing to
> HDFS,
> > >> which are quite a few stability issues.
> > >>>
> > >>> I know that Flink SinkV2 is also committed through a regular node,
> > >>> does this mean that SinkV2 also has this drawback?
> > >>>
> > >>> Best,
> > >>> Jingsong
> > >>>
> > >>> On Mon, Feb 10, 2025 at 4:06 PM Yunfeng Zhou
> > >>> <fl...@gmail.com> wrote:
> > >>>>
> > >>>> Hi Yong,
> > >>>>
> > >>>> The general idea looks good to me. Is there any statistics on the
> > >> number of operator events that need to be transmitted between the
> > >> coordinator and the writer operators? This information could help
> provide
> > >> estimations on the additional workload to the JM, preventing the JM
> from
> > >> being a single bottleneck to the throughput of Paimon sinks.
> > >>>>
> > >>>> Best,
> > >>>> Yunfeng
> > >>>>
> > >>>>> 2025年1月23日 17:44,Yong Fang <zj...@gmail.com> 写道:
> > >>>>>
> > >>>>> Hi devs,
> > >>>>>
> > >>>>> I would like to start a discussion about PIP-30: Improvement For
> > >> Paimon
> > >>>>> Committer In Flink [1].
> > >>>>>
> > >>>>> Currently Flink writes data to Paimon based on Two-Phase Commit
> > >> which will
> > >>>>> generate a global committer node and connect all tasks in one
> > >> region. If
> > >>>>> any task fails, it will lead to a global failover in Flink job.
> > >>>>>
> > >>>>> To solve this issue, we would like to introduce a Paimon Writer
> > >> Coordinator
> > >>>>> to perform table commit operation, enabling Flink paimon jobs to
> > >> support
> > >>>>> region failover and improving stability.
> > >>>>>
> > >>>>> Looking forward to hearing from you, thanks!
> > >>>>>
> > >>>>> [1]
> > >>>>>
> > >>
> https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
> > >>>>>
> > >>>>>
> > >>>>> Best,
> > >>>>> Fang Yong
> > >>>>
> > >>
> >
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by wj wang <ho...@gmail.com>.
To @Yanquan Lv
The concept of 'Region' is in Blink or old Flink version.
Now the 'Region' is means `Restart Pipelined Region Failover Strategy` in Flink.
On Wed, Feb 12, 2025 at 9:35 AM Yanquan Lv <de...@gmail.com> wrote:
>
> Hi Fang Yong. Thanks for driving it.
> Adding an alternative way to complete the global commit is good for me.
>
> I still have a confusion similar with xiangyu feng,
> > This committer node will connect all the tasks in Flink job together, and all the tasks are within one region. As a result, when any task fails, it will trigger a global failover of the Flink job.
> Why would the failure of a subtask have impact on other subtasks? The explanation on how to form a region in the Flink document is not particularly detailed. Can you explain more on it.
>
>
> > 2025年2月11日 16:55,Yong Fang <zj...@gmail.com> 写道:
> >
> > Thanks for all the feedback.
> >
> > To @xiangyu feng:
> > 1. Regarding region failover, I'm referring to the `Restart Pipelined
> > Region Failover Strategy` [1] in Flink.
> > 2. This feature mainly enables support for region failover, enhancing
> > stability. Of course, in practical use, users can configure the JM to
> > provide more suitable resources for the committer node.
> >
> > To @Xintong Song:
> > It doesn't matter if the notifyComplete of one checkpoint fails to be
> > called. Since the data will be stored in the HDFS path of the coordinator,
> > when the notifyComplete of the next CP is called, the data files generated
> > by the two CPs will be merged to create a Paimon snapshot.
> >
> > To @Yunfeng Zhou:
> > The pressure on the JM (JobManager) depends on two aspects:
> > Message communication pressure: It is determined by the checkpoint (CP)
> > interval and the number of sinks. Each sink generates one message in each
> > CP. If there are a total of 5000 sinks, the JM will receive 5000 messages
> > in one CP, which may take about a few seconds.
> > Pressure from executing table commits: It is necessary to create a Paimon
> > snapshot, and compaction may even be triggered. Therefore, asynchronous
> > threads should be considered for these operations to avoid blocking the
> > main thread of the JM.
> >
> > To @Jingsong and @wj wang
> > When using Flink SinkV2 to write to Paimon, a global committer node will
> > also be generated in the physical execution plan to create and manage
> > Paimon snapshots. Therefore, the same problem arises
> >
> > [1]
> > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/#restart-pipelined-region-failover-strategy
> >
> > Best,
> > Fang Yong
> >
> > On Tue, Feb 11, 2025 at 11:26 AM wj wang <ho...@gmail.com> wrote:
> >
> >> Hi Yong, thanks for driving this PIP.
> >> I have a small question:
> >> Why not use Flink SinkV2 instead of moving the committer logic into
> >> JM's OperatorCoordinator?
> >>
> >> On Tue, Feb 11, 2025 at 10:34 AM Jingsong Li <ji...@gmail.com>
> >> wrote:
> >>>
> >>> Thanks Yong for driving this PIP!
> >>>
> >>>>> Currently, there will be a global committer node in Flink Paimon job
> >> which is used to ensure the consistency of written data in Paimon. This
> >> committer node will connect all the tasks in Flink job together, and all
> >> the tasks are within one region. As a result, when any task fails, it will
> >> trigger a global failover of the Flink job. We use HDFS as the remote
> >> storage, and we often encounter situations where the global failover of
> >> jobs is triggered due to write timeouts or errors when writing to HDFS,
> >> which are quite a few stability issues.
> >>>
> >>> I know that Flink SinkV2 is also committed through a regular node,
> >>> does this mean that SinkV2 also has this drawback?
> >>>
> >>> Best,
> >>> Jingsong
> >>>
> >>> On Mon, Feb 10, 2025 at 4:06 PM Yunfeng Zhou
> >>> <fl...@gmail.com> wrote:
> >>>>
> >>>> Hi Yong,
> >>>>
> >>>> The general idea looks good to me. Is there any statistics on the
> >> number of operator events that need to be transmitted between the
> >> coordinator and the writer operators? This information could help provide
> >> estimations on the additional workload to the JM, preventing the JM from
> >> being a single bottleneck to the throughput of Paimon sinks.
> >>>>
> >>>> Best,
> >>>> Yunfeng
> >>>>
> >>>>> 2025年1月23日 17:44,Yong Fang <zj...@gmail.com> 写道:
> >>>>>
> >>>>> Hi devs,
> >>>>>
> >>>>> I would like to start a discussion about PIP-30: Improvement For
> >> Paimon
> >>>>> Committer In Flink [1].
> >>>>>
> >>>>> Currently Flink writes data to Paimon based on Two-Phase Commit
> >> which will
> >>>>> generate a global committer node and connect all tasks in one
> >> region. If
> >>>>> any task fails, it will lead to a global failover in Flink job.
> >>>>>
> >>>>> To solve this issue, we would like to introduce a Paimon Writer
> >> Coordinator
> >>>>> to perform table commit operation, enabling Flink paimon jobs to
> >> support
> >>>>> region failover and improving stability.
> >>>>>
> >>>>> Looking forward to hearing from you, thanks!
> >>>>>
> >>>>> [1]
> >>>>>
> >> https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
> >>>>>
> >>>>>
> >>>>> Best,
> >>>>> Fang Yong
> >>>>
> >>
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Yanquan Lv <de...@gmail.com>.
Hi Fang Yong. Thanks for driving it.
Adding an alternative way to complete the global commit is good for me.
I still have a confusion similar with xiangyu feng,
> This committer node will connect all the tasks in Flink job together, and all the tasks are within one region. As a result, when any task fails, it will trigger a global failover of the Flink job.
Why would the failure of a subtask have impact on other subtasks? The explanation on how to form a region in the Flink document is not particularly detailed. Can you explain more on it.
> 2025年2月11日 16:55,Yong Fang <zj...@gmail.com> 写道:
>
> Thanks for all the feedback.
>
> To @xiangyu feng:
> 1. Regarding region failover, I'm referring to the `Restart Pipelined
> Region Failover Strategy` [1] in Flink.
> 2. This feature mainly enables support for region failover, enhancing
> stability. Of course, in practical use, users can configure the JM to
> provide more suitable resources for the committer node.
>
> To @Xintong Song:
> It doesn't matter if the notifyComplete of one checkpoint fails to be
> called. Since the data will be stored in the HDFS path of the coordinator,
> when the notifyComplete of the next CP is called, the data files generated
> by the two CPs will be merged to create a Paimon snapshot.
>
> To @Yunfeng Zhou:
> The pressure on the JM (JobManager) depends on two aspects:
> Message communication pressure: It is determined by the checkpoint (CP)
> interval and the number of sinks. Each sink generates one message in each
> CP. If there are a total of 5000 sinks, the JM will receive 5000 messages
> in one CP, which may take about a few seconds.
> Pressure from executing table commits: It is necessary to create a Paimon
> snapshot, and compaction may even be triggered. Therefore, asynchronous
> threads should be considered for these operations to avoid blocking the
> main thread of the JM.
>
> To @Jingsong and @wj wang
> When using Flink SinkV2 to write to Paimon, a global committer node will
> also be generated in the physical execution plan to create and manage
> Paimon snapshots. Therefore, the same problem arises
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/#restart-pipelined-region-failover-strategy
>
> Best,
> Fang Yong
>
> On Tue, Feb 11, 2025 at 11:26 AM wj wang <ho...@gmail.com> wrote:
>
>> Hi Yong, thanks for driving this PIP.
>> I have a small question:
>> Why not use Flink SinkV2 instead of moving the committer logic into
>> JM's OperatorCoordinator?
>>
>> On Tue, Feb 11, 2025 at 10:34 AM Jingsong Li <ji...@gmail.com>
>> wrote:
>>>
>>> Thanks Yong for driving this PIP!
>>>
>>>>> Currently, there will be a global committer node in Flink Paimon job
>> which is used to ensure the consistency of written data in Paimon. This
>> committer node will connect all the tasks in Flink job together, and all
>> the tasks are within one region. As a result, when any task fails, it will
>> trigger a global failover of the Flink job. We use HDFS as the remote
>> storage, and we often encounter situations where the global failover of
>> jobs is triggered due to write timeouts or errors when writing to HDFS,
>> which are quite a few stability issues.
>>>
>>> I know that Flink SinkV2 is also committed through a regular node,
>>> does this mean that SinkV2 also has this drawback?
>>>
>>> Best,
>>> Jingsong
>>>
>>> On Mon, Feb 10, 2025 at 4:06 PM Yunfeng Zhou
>>> <fl...@gmail.com> wrote:
>>>>
>>>> Hi Yong,
>>>>
>>>> The general idea looks good to me. Is there any statistics on the
>> number of operator events that need to be transmitted between the
>> coordinator and the writer operators? This information could help provide
>> estimations on the additional workload to the JM, preventing the JM from
>> being a single bottleneck to the throughput of Paimon sinks.
>>>>
>>>> Best,
>>>> Yunfeng
>>>>
>>>>> 2025年1月23日 17:44,Yong Fang <zj...@gmail.com> 写道:
>>>>>
>>>>> Hi devs,
>>>>>
>>>>> I would like to start a discussion about PIP-30: Improvement For
>> Paimon
>>>>> Committer In Flink [1].
>>>>>
>>>>> Currently Flink writes data to Paimon based on Two-Phase Commit
>> which will
>>>>> generate a global committer node and connect all tasks in one
>> region. If
>>>>> any task fails, it will lead to a global failover in Flink job.
>>>>>
>>>>> To solve this issue, we would like to introduce a Paimon Writer
>> Coordinator
>>>>> to perform table commit operation, enabling Flink paimon jobs to
>> support
>>>>> region failover and improving stability.
>>>>>
>>>>> Looking forward to hearing from you, thanks!
>>>>>
>>>>> [1]
>>>>>
>> https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
>>>>>
>>>>>
>>>>> Best,
>>>>> Fang Yong
>>>>
>>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Jingsong Li <ji...@gmail.com>.
+1 to Biao, let’s move forward
Biao Liu <mm...@gmail.com>于2026年4月22日 周三23:33写道:
> Hi devs,
>
> I would like to share the follow-up details of this proposal. We have
> implemented the solution proposed by Yong Fang in PIP-30 on our internal
> version of Paimon. As we encountered several issues during implementation,
> the final implementation differs from the original PIP-30 document.
> I have prepared an updated version of the document, with all changes
> highlighted using a yellow background. The corresponding figures below the
> relevant text have also been revised accordingly.
>
>
> https://docs.google.com/document/d/1asWKzoytfeB1D8bS_yRIAHnpR40frLf0fnPn2-WSL74/edit?usp=sharing
>
> We look forward to your feedback and comments.
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Wed, 19 Feb 2025 at 10:23, Yong Fang <zj...@gmail.com> wrote:
>
> > Thanks for all the feedback, if there are no more questions, I will
> > initiate a vote later.
> >
> > Best,
> > Fang Yong
> >
> > On Thu, Feb 13, 2025 at 12:52 PM xiangyu feng <xi...@gmail.com>
> > wrote:
> >
> > > Hi Yong,
> > >
> > > I see your point now. The overall design LGTM.
> > >
> > > Besides the support for region failover, we can also consider how to
> > > support regional committer in the future. Each region can advance its
> > data
> > > commitment mutually independent. This will also bring great value.
> > >
> > > Best,
> > > Xiangyu
> > >
> > > Jingsong Li <ji...@gmail.com> 于2025年2月12日周三 22:25写道:
> > >
> > > > Thanks Yong for the explanation.
> > > >
> > > > Sounds good to me.
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Wed, Feb 12, 2025 at 6:06 PM Yunfeng Zhou
> > > > <fl...@gmail.com> wrote:
> > > > >
> > > > > Hi Yong,
> > > > >
> > > > > > It is necessary to create a Paimon
> > > > > snapshot, and compaction may even be triggered. Therefore,
> > asynchronous
> > > > > threads should be considered for these operations to avoid blocking
> > the
> > > > > main thread of the JM.
> > > > >
> > > > > This statement sounds like it would be JM to perform the
> compaction,
> > > > regardless of sync or async. AFAIK, it should be writer operators to
> do
> > > the
> > > > compact, so JM need not consider this kind of overhead.
> > > > >
> > > > > Just a correction on a possible misunderstanding. The design and
> > > > response itself sounds good to me.
> > > > >
> > > > > Best,
> > > > > Yunfeng
> > > > >
> > > > > > 2025年2月11日 16:55,Yong Fang <zj...@gmail.com> 写道:
> > > > > >
> > > > > > It is necessary to create a Paimon
> > > > > > snapshot, and compaction may even be triggered. Therefore,
> > > asynchronous
> > > > > > threads should be considered for these operations to avoid
> blocking
> > > the
> > > > > > main thread of the JM.
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Biao Liu <mm...@gmail.com>.
Hi devs,
I would like to share the follow-up details of this proposal. We have
implemented the solution proposed by Yong Fang in PIP-30 on our internal
version of Paimon. As we encountered several issues during implementation,
the final implementation differs from the original PIP-30 document.
I have prepared an updated version of the document, with all changes
highlighted using a yellow background. The corresponding figures below the
relevant text have also been revised accordingly.
https://docs.google.com/document/d/1asWKzoytfeB1D8bS_yRIAHnpR40frLf0fnPn2-WSL74/edit?usp=sharing
We look forward to your feedback and comments.
Thanks,
Biao /'bɪ.aʊ/
On Wed, 19 Feb 2025 at 10:23, Yong Fang <zj...@gmail.com> wrote:
> Thanks for all the feedback, if there are no more questions, I will
> initiate a vote later.
>
> Best,
> Fang Yong
>
> On Thu, Feb 13, 2025 at 12:52 PM xiangyu feng <xi...@gmail.com>
> wrote:
>
> > Hi Yong,
> >
> > I see your point now. The overall design LGTM.
> >
> > Besides the support for region failover, we can also consider how to
> > support regional committer in the future. Each region can advance its
> data
> > commitment mutually independent. This will also bring great value.
> >
> > Best,
> > Xiangyu
> >
> > Jingsong Li <ji...@gmail.com> 于2025年2月12日周三 22:25写道:
> >
> > > Thanks Yong for the explanation.
> > >
> > > Sounds good to me.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Wed, Feb 12, 2025 at 6:06 PM Yunfeng Zhou
> > > <fl...@gmail.com> wrote:
> > > >
> > > > Hi Yong,
> > > >
> > > > > It is necessary to create a Paimon
> > > > snapshot, and compaction may even be triggered. Therefore,
> asynchronous
> > > > threads should be considered for these operations to avoid blocking
> the
> > > > main thread of the JM.
> > > >
> > > > This statement sounds like it would be JM to perform the compaction,
> > > regardless of sync or async. AFAIK, it should be writer operators to do
> > the
> > > compact, so JM need not consider this kind of overhead.
> > > >
> > > > Just a correction on a possible misunderstanding. The design and
> > > response itself sounds good to me.
> > > >
> > > > Best,
> > > > Yunfeng
> > > >
> > > > > 2025年2月11日 16:55,Yong Fang <zj...@gmail.com> 写道:
> > > > >
> > > > > It is necessary to create a Paimon
> > > > > snapshot, and compaction may even be triggered. Therefore,
> > asynchronous
> > > > > threads should be considered for these operations to avoid blocking
> > the
> > > > > main thread of the JM.
> > > >
> > >
> >
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Yong Fang <zj...@gmail.com>.
Thanks for all the feedback, if there are no more questions, I will
initiate a vote later.
Best,
Fang Yong
On Thu, Feb 13, 2025 at 12:52 PM xiangyu feng <xi...@gmail.com> wrote:
> Hi Yong,
>
> I see your point now. The overall design LGTM.
>
> Besides the support for region failover, we can also consider how to
> support regional committer in the future. Each region can advance its data
> commitment mutually independent. This will also bring great value.
>
> Best,
> Xiangyu
>
> Jingsong Li <ji...@gmail.com> 于2025年2月12日周三 22:25写道:
>
> > Thanks Yong for the explanation.
> >
> > Sounds good to me.
> >
> > Best,
> > Jingsong
> >
> > On Wed, Feb 12, 2025 at 6:06 PM Yunfeng Zhou
> > <fl...@gmail.com> wrote:
> > >
> > > Hi Yong,
> > >
> > > > It is necessary to create a Paimon
> > > snapshot, and compaction may even be triggered. Therefore, asynchronous
> > > threads should be considered for these operations to avoid blocking the
> > > main thread of the JM.
> > >
> > > This statement sounds like it would be JM to perform the compaction,
> > regardless of sync or async. AFAIK, it should be writer operators to do
> the
> > compact, so JM need not consider this kind of overhead.
> > >
> > > Just a correction on a possible misunderstanding. The design and
> > response itself sounds good to me.
> > >
> > > Best,
> > > Yunfeng
> > >
> > > > 2025年2月11日 16:55,Yong Fang <zj...@gmail.com> 写道:
> > > >
> > > > It is necessary to create a Paimon
> > > > snapshot, and compaction may even be triggered. Therefore,
> asynchronous
> > > > threads should be considered for these operations to avoid blocking
> the
> > > > main thread of the JM.
> > >
> >
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by xiangyu feng <xi...@gmail.com>.
Hi Yong,
I see your point now. The overall design LGTM.
Besides the support for region failover, we can also consider how to
support regional committer in the future. Each region can advance its data
commitment mutually independent. This will also bring great value.
Best,
Xiangyu
Jingsong Li <ji...@gmail.com> 于2025年2月12日周三 22:25写道:
> Thanks Yong for the explanation.
>
> Sounds good to me.
>
> Best,
> Jingsong
>
> On Wed, Feb 12, 2025 at 6:06 PM Yunfeng Zhou
> <fl...@gmail.com> wrote:
> >
> > Hi Yong,
> >
> > > It is necessary to create a Paimon
> > snapshot, and compaction may even be triggered. Therefore, asynchronous
> > threads should be considered for these operations to avoid blocking the
> > main thread of the JM.
> >
> > This statement sounds like it would be JM to perform the compaction,
> regardless of sync or async. AFAIK, it should be writer operators to do the
> compact, so JM need not consider this kind of overhead.
> >
> > Just a correction on a possible misunderstanding. The design and
> response itself sounds good to me.
> >
> > Best,
> > Yunfeng
> >
> > > 2025年2月11日 16:55,Yong Fang <zj...@gmail.com> 写道:
> > >
> > > It is necessary to create a Paimon
> > > snapshot, and compaction may even be triggered. Therefore, asynchronous
> > > threads should be considered for these operations to avoid blocking the
> > > main thread of the JM.
> >
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Yong for the explanation.
Sounds good to me.
Best,
Jingsong
On Wed, Feb 12, 2025 at 6:06 PM Yunfeng Zhou
<fl...@gmail.com> wrote:
>
> Hi Yong,
>
> > It is necessary to create a Paimon
> snapshot, and compaction may even be triggered. Therefore, asynchronous
> threads should be considered for these operations to avoid blocking the
> main thread of the JM.
>
> This statement sounds like it would be JM to perform the compaction, regardless of sync or async. AFAIK, it should be writer operators to do the compact, so JM need not consider this kind of overhead.
>
> Just a correction on a possible misunderstanding. The design and response itself sounds good to me.
>
> Best,
> Yunfeng
>
> > 2025年2月11日 16:55,Yong Fang <zj...@gmail.com> 写道:
> >
> > It is necessary to create a Paimon
> > snapshot, and compaction may even be triggered. Therefore, asynchronous
> > threads should be considered for these operations to avoid blocking the
> > main thread of the JM.
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Yunfeng Zhou <fl...@gmail.com>.
Hi Yong,
> It is necessary to create a Paimon
snapshot, and compaction may even be triggered. Therefore, asynchronous
threads should be considered for these operations to avoid blocking the
main thread of the JM.
This statement sounds like it would be JM to perform the compaction, regardless of sync or async. AFAIK, it should be writer operators to do the compact, so JM need not consider this kind of overhead.
Just a correction on a possible misunderstanding. The design and response itself sounds good to me.
Best,
Yunfeng
> 2025年2月11日 16:55,Yong Fang <zj...@gmail.com> 写道:
>
> It is necessary to create a Paimon
> snapshot, and compaction may even be triggered. Therefore, asynchronous
> threads should be considered for these operations to avoid blocking the
> main thread of the JM.
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Yong Fang <zj...@gmail.com>.
Thanks for all the feedback.
To @xiangyu feng:
1. Regarding region failover, I'm referring to the `Restart Pipelined
Region Failover Strategy` [1] in Flink.
2. This feature mainly enables support for region failover, enhancing
stability. Of course, in practical use, users can configure the JM to
provide more suitable resources for the committer node.
To @Xintong Song:
It doesn't matter if the notifyComplete of one checkpoint fails to be
called. Since the data will be stored in the HDFS path of the coordinator,
when the notifyComplete of the next CP is called, the data files generated
by the two CPs will be merged to create a Paimon snapshot.
To @Yunfeng Zhou:
The pressure on the JM (JobManager) depends on two aspects:
Message communication pressure: It is determined by the checkpoint (CP)
interval and the number of sinks. Each sink generates one message in each
CP. If there are a total of 5000 sinks, the JM will receive 5000 messages
in one CP, which may take about a few seconds.
Pressure from executing table commits: It is necessary to create a Paimon
snapshot, and compaction may even be triggered. Therefore, asynchronous
threads should be considered for these operations to avoid blocking the
main thread of the JM.
To @Jingsong and @wj wang
When using Flink SinkV2 to write to Paimon, a global committer node will
also be generated in the physical execution plan to create and manage
Paimon snapshots. Therefore, the same problem arises
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/#restart-pipelined-region-failover-strategy
Best,
Fang Yong
On Tue, Feb 11, 2025 at 11:26 AM wj wang <ho...@gmail.com> wrote:
> Hi Yong, thanks for driving this PIP.
> I have a small question:
> Why not use Flink SinkV2 instead of moving the committer logic into
> JM's OperatorCoordinator?
>
> On Tue, Feb 11, 2025 at 10:34 AM Jingsong Li <ji...@gmail.com>
> wrote:
> >
> > Thanks Yong for driving this PIP!
> >
> > >> Currently, there will be a global committer node in Flink Paimon job
> which is used to ensure the consistency of written data in Paimon. This
> committer node will connect all the tasks in Flink job together, and all
> the tasks are within one region. As a result, when any task fails, it will
> trigger a global failover of the Flink job. We use HDFS as the remote
> storage, and we often encounter situations where the global failover of
> jobs is triggered due to write timeouts or errors when writing to HDFS,
> which are quite a few stability issues.
> >
> > I know that Flink SinkV2 is also committed through a regular node,
> > does this mean that SinkV2 also has this drawback?
> >
> > Best,
> > Jingsong
> >
> > On Mon, Feb 10, 2025 at 4:06 PM Yunfeng Zhou
> > <fl...@gmail.com> wrote:
> > >
> > > Hi Yong,
> > >
> > > The general idea looks good to me. Is there any statistics on the
> number of operator events that need to be transmitted between the
> coordinator and the writer operators? This information could help provide
> estimations on the additional workload to the JM, preventing the JM from
> being a single bottleneck to the throughput of Paimon sinks.
> > >
> > > Best,
> > > Yunfeng
> > >
> > > > 2025年1月23日 17:44,Yong Fang <zj...@gmail.com> 写道:
> > > >
> > > > Hi devs,
> > > >
> > > > I would like to start a discussion about PIP-30: Improvement For
> Paimon
> > > > Committer In Flink [1].
> > > >
> > > > Currently Flink writes data to Paimon based on Two-Phase Commit
> which will
> > > > generate a global committer node and connect all tasks in one
> region. If
> > > > any task fails, it will lead to a global failover in Flink job.
> > > >
> > > > To solve this issue, we would like to introduce a Paimon Writer
> Coordinator
> > > > to perform table commit operation, enabling Flink paimon jobs to
> support
> > > > region failover and improving stability.
> > > >
> > > > Looking forward to hearing from you, thanks!
> > > >
> > > > [1]
> > > >
> https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
> > > >
> > > >
> > > > Best,
> > > > Fang Yong
> > >
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by wj wang <ho...@gmail.com>.
Hi Yong, thanks for driving this PIP.
I have a small question:
Why not use Flink SinkV2 instead of moving the committer logic into
JM's OperatorCoordinator?
On Tue, Feb 11, 2025 at 10:34 AM Jingsong Li <ji...@gmail.com> wrote:
>
> Thanks Yong for driving this PIP!
>
> >> Currently, there will be a global committer node in Flink Paimon job which is used to ensure the consistency of written data in Paimon. This committer node will connect all the tasks in Flink job together, and all the tasks are within one region. As a result, when any task fails, it will trigger a global failover of the Flink job. We use HDFS as the remote storage, and we often encounter situations where the global failover of jobs is triggered due to write timeouts or errors when writing to HDFS, which are quite a few stability issues.
>
> I know that Flink SinkV2 is also committed through a regular node,
> does this mean that SinkV2 also has this drawback?
>
> Best,
> Jingsong
>
> On Mon, Feb 10, 2025 at 4:06 PM Yunfeng Zhou
> <fl...@gmail.com> wrote:
> >
> > Hi Yong,
> >
> > The general idea looks good to me. Is there any statistics on the number of operator events that need to be transmitted between the coordinator and the writer operators? This information could help provide estimations on the additional workload to the JM, preventing the JM from being a single bottleneck to the throughput of Paimon sinks.
> >
> > Best,
> > Yunfeng
> >
> > > 2025年1月23日 17:44,Yong Fang <zj...@gmail.com> 写道:
> > >
> > > Hi devs,
> > >
> > > I would like to start a discussion about PIP-30: Improvement For Paimon
> > > Committer In Flink [1].
> > >
> > > Currently Flink writes data to Paimon based on Two-Phase Commit which will
> > > generate a global committer node and connect all tasks in one region. If
> > > any task fails, it will lead to a global failover in Flink job.
> > >
> > > To solve this issue, we would like to introduce a Paimon Writer Coordinator
> > > to perform table commit operation, enabling Flink paimon jobs to support
> > > region failover and improving stability.
> > >
> > > Looking forward to hearing from you, thanks!
> > >
> > > [1]
> > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
> > >
> > >
> > > Best,
> > > Fang Yong
> >
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Yong for driving this PIP!
>> Currently, there will be a global committer node in Flink Paimon job which is used to ensure the consistency of written data in Paimon. This committer node will connect all the tasks in Flink job together, and all the tasks are within one region. As a result, when any task fails, it will trigger a global failover of the Flink job. We use HDFS as the remote storage, and we often encounter situations where the global failover of jobs is triggered due to write timeouts or errors when writing to HDFS, which are quite a few stability issues.
I know that Flink SinkV2 is also committed through a regular node,
does this mean that SinkV2 also has this drawback?
Best,
Jingsong
On Mon, Feb 10, 2025 at 4:06 PM Yunfeng Zhou
<fl...@gmail.com> wrote:
>
> Hi Yong,
>
> The general idea looks good to me. Is there any statistics on the number of operator events that need to be transmitted between the coordinator and the writer operators? This information could help provide estimations on the additional workload to the JM, preventing the JM from being a single bottleneck to the throughput of Paimon sinks.
>
> Best,
> Yunfeng
>
> > 2025年1月23日 17:44,Yong Fang <zj...@gmail.com> 写道:
> >
> > Hi devs,
> >
> > I would like to start a discussion about PIP-30: Improvement For Paimon
> > Committer In Flink [1].
> >
> > Currently Flink writes data to Paimon based on Two-Phase Commit which will
> > generate a global committer node and connect all tasks in one region. If
> > any task fails, it will lead to a global failover in Flink job.
> >
> > To solve this issue, we would like to introduce a Paimon Writer Coordinator
> > to perform table commit operation, enabling Flink paimon jobs to support
> > region failover and improving stability.
> >
> > Looking forward to hearing from you, thanks!
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
> >
> >
> > Best,
> > Fang Yong
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Yunfeng Zhou <fl...@gmail.com>.
Hi Yong,
The general idea looks good to me. Is there any statistics on the number of operator events that need to be transmitted between the coordinator and the writer operators? This information could help provide estimations on the additional workload to the JM, preventing the JM from being a single bottleneck to the throughput of Paimon sinks.
Best,
Yunfeng
> 2025年1月23日 17:44,Yong Fang <zj...@gmail.com> 写道:
>
> Hi devs,
>
> I would like to start a discussion about PIP-30: Improvement For Paimon
> Committer In Flink [1].
>
> Currently Flink writes data to Paimon based on Two-Phase Commit which will
> generate a global committer node and connect all tasks in one region. If
> any task fails, it will lead to a global failover in Flink job.
>
> To solve this issue, we would like to introduce a Paimon Writer Coordinator
> to perform table commit operation, enabling Flink paimon jobs to support
> region failover and improving stability.
>
> Looking forward to hearing from you, thanks!
>
> [1]
> https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
>
>
> Best,
> Fang Yong
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Xintong Song <to...@gmail.com>.
Hi Yong,
Big +1 for moving the committer logics into operator coordinator.
Regarding the design, I have only one question: What happens if JM fails
after a checkpoint is completed but before the
`OperatorCoordinator#notifyCheckpointComplete()` is called?
The JavaDoc of CheckpointListener states that, `notifyCheckpointComplete`
and `notifyCheckpointAborted` are best efforts. If a failure occurs
directly after the checkpoint is completed, the coordinator may not get the
notification. While this PIP already described the scenario that an
uncompleted checkpoint will be retried after recovery, IIUC, a completed
but not notified checkpoint will not be re-notified. Does this mean
`OperatorCoordinator#notifyCheckpointComplete()` may never be called for
that checkpoint, thus the table meta of the corresponding snapshot will be
left incomplete?
Best,
Xintong
On Wed, Feb 5, 2025 at 4:00 PM xiangyu feng <xi...@gmail.com> wrote:
> Hi Yong,
>
> Thx for initiating this discussion. I like the idea to remove the global
> committer operator in DAG to improve the overall streaming write stability.
> I have following questions:
>
> 1. The benefit of this improvement is that when Flink writes to append
> table without bucket it can support region failover. However, "region
> failover" seems like an internal concept to community users. AFAIK,
> there
> is only one concept as "pipelined region" in Flink scheduling and there
> are
> no concepts named "region" in Flink checkpoint. I'm not sure these two
> concepts are equivalent. Maybe we should add more descriptions here.
> 2. Besides supporting region failover, are there any other benefits to
> this feature?
>
>
> Best,
> Xiangyu Feng
>
> Yong Fang <zj...@gmail.com> 于2025年1月24日周五 15:51写道:
>
> > Thanks zhanghao, we can do this in flink too
> >
> > Best,
> > Fang Yong
> >
> > Zhanghao Chen <zh...@outlook.com> 写道:
> >
> > > Hi Yong,
> > >
> > > Thanks for raising it! It is a common problem shared by all sinks using
> > > the global committer pattern. Would it be better to initiate a
> discussion
> > > in the Flink community as well?
> > >
> > > Best,
> > > Zhanghao Chen
> > > ________________________________
> > > From: Jingsong Li <ji...@gmail.com>
> > > Sent: Thursday, January 23, 2025 20:38
> > > To: dev@paimon.apache.org <de...@paimon.apache.org>
> > > Subject: Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In
> Flink
> > >
> > > Thanks Yong!
> > >
> > > Looks fantastic! We can discuss this in detail after the Spring
> Festival.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Thu, Jan 23, 2025 at 5:44 PM Yong Fang <zj...@gmail.com> wrote:
> > > >
> > > > Hi devs,
> > > >
> > > > I would like to start a discussion about PIP-30: Improvement For
> Paimon
> > > > Committer In Flink [1].
> > > >
> > > > Currently Flink writes data to Paimon based on Two-Phase Commit which
> > > will
> > > > generate a global committer node and connect all tasks in one region.
> > If
> > > > any task fails, it will lead to a global failover in Flink job.
> > > >
> > > > To solve this issue, we would like to introduce a Paimon Writer
> > > Coordinator
> > > > to perform table commit operation, enabling Flink paimon jobs to
> > support
> > > > region failover and improving stability.
> > > >
> > > > Looking forward to hearing from you, thanks!
> > > >
> > > > [1]
> > > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-
> > > 30%3A+Improvement+For+Paimon+Committer+In+Flink
> > > >
> > > >
> > > > Best,
> > > > Fang Yong
> > >
> >
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by xiangyu feng <xi...@gmail.com>.
Hi Yong,
Thx for initiating this discussion. I like the idea to remove the global
committer operator in DAG to improve the overall streaming write stability.
I have following questions:
1. The benefit of this improvement is that when Flink writes to append
table without bucket it can support region failover. However, "region
failover" seems like an internal concept to community users. AFAIK, there
is only one concept as "pipelined region" in Flink scheduling and there are
no concepts named "region" in Flink checkpoint. I'm not sure these two
concepts are equivalent. Maybe we should add more descriptions here.
2. Besides supporting region failover, are there any other benefits to
this feature?
Best,
Xiangyu Feng
Yong Fang <zj...@gmail.com> 于2025年1月24日周五 15:51写道:
> Thanks zhanghao, we can do this in flink too
>
> Best,
> Fang Yong
>
> Zhanghao Chen <zh...@outlook.com> 写道:
>
> > Hi Yong,
> >
> > Thanks for raising it! It is a common problem shared by all sinks using
> > the global committer pattern. Would it be better to initiate a discussion
> > in the Flink community as well?
> >
> > Best,
> > Zhanghao Chen
> > ________________________________
> > From: Jingsong Li <ji...@gmail.com>
> > Sent: Thursday, January 23, 2025 20:38
> > To: dev@paimon.apache.org <de...@paimon.apache.org>
> > Subject: Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
> >
> > Thanks Yong!
> >
> > Looks fantastic! We can discuss this in detail after the Spring Festival.
> >
> > Best,
> > Jingsong
> >
> > On Thu, Jan 23, 2025 at 5:44 PM Yong Fang <zj...@gmail.com> wrote:
> > >
> > > Hi devs,
> > >
> > > I would like to start a discussion about PIP-30: Improvement For Paimon
> > > Committer In Flink [1].
> > >
> > > Currently Flink writes data to Paimon based on Two-Phase Commit which
> > will
> > > generate a global committer node and connect all tasks in one region.
> If
> > > any task fails, it will lead to a global failover in Flink job.
> > >
> > > To solve this issue, we would like to introduce a Paimon Writer
> > Coordinator
> > > to perform table commit operation, enabling Flink paimon jobs to
> support
> > > region failover and improving stability.
> > >
> > > Looking forward to hearing from you, thanks!
> > >
> > > [1]
> > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-
> > 30%3A+Improvement+For+Paimon+Committer+In+Flink
> > >
> > >
> > > Best,
> > > Fang Yong
> >
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Yong Fang <zj...@gmail.com>.
Thanks zhanghao, we can do this in flink too
Best,
Fang Yong
Zhanghao Chen <zh...@outlook.com> 写道:
> Hi Yong,
>
> Thanks for raising it! It is a common problem shared by all sinks using
> the global committer pattern. Would it be better to initiate a discussion
> in the Flink community as well?
>
> Best,
> Zhanghao Chen
> ________________________________
> From: Jingsong Li <ji...@gmail.com>
> Sent: Thursday, January 23, 2025 20:38
> To: dev@paimon.apache.org <de...@paimon.apache.org>
> Subject: Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
>
> Thanks Yong!
>
> Looks fantastic! We can discuss this in detail after the Spring Festival.
>
> Best,
> Jingsong
>
> On Thu, Jan 23, 2025 at 5:44 PM Yong Fang <zj...@gmail.com> wrote:
> >
> > Hi devs,
> >
> > I would like to start a discussion about PIP-30: Improvement For Paimon
> > Committer In Flink [1].
> >
> > Currently Flink writes data to Paimon based on Two-Phase Commit which
> will
> > generate a global committer node and connect all tasks in one region. If
> > any task fails, it will lead to a global failover in Flink job.
> >
> > To solve this issue, we would like to introduce a Paimon Writer
> Coordinator
> > to perform table commit operation, enabling Flink paimon jobs to support
> > region failover and improving stability.
> >
> > Looking forward to hearing from you, thanks!
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/PAIMON/PIP-
> 30%3A+Improvement+For+Paimon+Committer+In+Flink
> >
> >
> > Best,
> > Fang Yong
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Zhanghao Chen <zh...@outlook.com>.
Hi Yong,
Thanks for raising it! It is a common problem shared by all sinks using the global committer pattern. Would it be better to initiate a discussion in the Flink community as well?
Best,
Zhanghao Chen
________________________________
From: Jingsong Li <ji...@gmail.com>
Sent: Thursday, January 23, 2025 20:38
To: dev@paimon.apache.org <de...@paimon.apache.org>
Subject: Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Thanks Yong!
Looks fantastic! We can discuss this in detail after the Spring Festival.
Best,
Jingsong
On Thu, Jan 23, 2025 at 5:44 PM Yong Fang <zj...@gmail.com> wrote:
>
> Hi devs,
>
> I would like to start a discussion about PIP-30: Improvement For Paimon
> Committer In Flink [1].
>
> Currently Flink writes data to Paimon based on Two-Phase Commit which will
> generate a global committer node and connect all tasks in one region. If
> any task fails, it will lead to a global failover in Flink job.
>
> To solve this issue, we would like to introduce a Paimon Writer Coordinator
> to perform table commit operation, enabling Flink paimon jobs to support
> region failover and improving stability.
>
> Looking forward to hearing from you, thanks!
>
> [1]
> https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
>
>
> Best,
> Fang Yong
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Yong Fang <zj...@gmail.com>.
Thanks Jingsong, Happy Spring Festival !
Best,
Fang Yong
On Thu, Jan 23, 2025 at 8:39 PM Jingsong Li <ji...@gmail.com> wrote:
> Thanks Yong!
>
> Looks fantastic! We can discuss this in detail after the Spring Festival.
>
> Best,
> Jingsong
>
> On Thu, Jan 23, 2025 at 5:44 PM Yong Fang <zj...@gmail.com> wrote:
> >
> > Hi devs,
> >
> > I would like to start a discussion about PIP-30: Improvement For Paimon
> > Committer In Flink [1].
> >
> > Currently Flink writes data to Paimon based on Two-Phase Commit which
> will
> > generate a global committer node and connect all tasks in one region. If
> > any task fails, it will lead to a global failover in Flink job.
> >
> > To solve this issue, we would like to introduce a Paimon Writer
> Coordinator
> > to perform table commit operation, enabling Flink paimon jobs to support
> > region failover and improving stability.
> >
> > Looking forward to hearing from you, thanks!
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
> >
> >
> > Best,
> > Fang Yong
>
Re: [DISCUSS] PIP-30: Improvement For Paimon Committer In Flink
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Yong!
Looks fantastic! We can discuss this in detail after the Spring Festival.
Best,
Jingsong
On Thu, Jan 23, 2025 at 5:44 PM Yong Fang <zj...@gmail.com> wrote:
>
> Hi devs,
>
> I would like to start a discussion about PIP-30: Improvement For Paimon
> Committer In Flink [1].
>
> Currently Flink writes data to Paimon based on Two-Phase Commit which will
> generate a global committer node and connect all tasks in one region. If
> any task fails, it will lead to a global failover in Flink job.
>
> To solve this issue, we would like to introduce a Paimon Writer Coordinator
> to perform table commit operation, enabling Flink paimon jobs to support
> region failover and improving stability.
>
> Looking forward to hearing from you, thanks!
>
> [1]
> https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink
>
>
> Best,
> Fang Yong