You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Yuxin Tan <ta...@gmail.com> on 2024/05/28 12:34:21 UTC

[DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Hi all,

I would like to start a discussion on FLIP-459 Support Flink hybrid shuffle
integration with
Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
memory, disk, and
remote storage to improve performance and job stability. Concurrently,
Apache Celeborn
provides a stable, performant, scalable remote shuffle service. This
integration proposal is to
harness the benefits from both hybrid shuffle and Celeborn simultaneously.

Note that this proposal has two parts.
1. The Flink-side modifications are in FLIP-459[1].
2. The Celeborn-side changes are in CIP-6[2].

Looking forward to everyone's feedback and suggestions. Thank you!

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
[2]
https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn

Best,
Yuxin

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Yuxin Tan <ta...@gmail.com>.

Hi, Venkatakrishnan

The current proposal is designed to initialize a single remote
tier, such as Celeborn, if the remote tier factory is configured.
However, this is a temporary approach because the tier interfaces
are not stable. Our aim is to eventually support multiple storage
tiers: memory tier, disk tier, and remote tier. The remote tier
could be cloud storage or a remote shuffle service like Celeborn.

Initially, we don't expect to need both a cloud storage and a
remote shuffle service as two remote tiers concurrently due to
their large capacities. However, we're open and receptive to
adding both tiers concurrently if future discussions suggest it's
necessary. The architecture is designed for flexibility, allowing
each tier to function independently and be easily added to the
tiered storage with minimal modifications.

Best,
Yuxin


Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年6月17日周一 15:30写道：

> Yuxin,
>
> One question, in the current proposal is it limited to only one tiered
> storage implementation for eg: celeborn? Is it possible to have multiple
> tiered storages like a separate cloud storage and Celeborn or some such?
>
>
> On Thu, Jun 6, 2024, 9:46 AM Jeyhun Karimov <je...@gmail.com> wrote:
>
> > Hi Yuxin,
> >
> > +1 for this proposal.
> > This change will greatly alleviate the pressure on local storage
> resources
> > (especially when there is limited local storage)
> > particularly in the context of cloud-native environments.
> >
> > Regards,
> > Jeyhun
> >
> > On Thu, Jun 6, 2024 at 1:20 PM Yuxin Tan <ta...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Thanks for all the feedback and suggestions so far.
> > >
> > > If there is no further comment, we will open the voting thread
> tomorrow.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 15:40写道：
> > >
> > > > Thanks Zhu for the suggestion.
> > > > I have updated the description of the option.
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Zhu Zhu <re...@gmail.com> 于2024年6月6日周四 14:59写道：
> > > >
> > > >> +1
> > > >>
> > > >> Maybe explain in the description of
> > > >>
> > `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
> > > >> that it only accepts Celeborn as the remote shuffle tier at this
> > moment?
> > > >>
> > > >> Thanks,
> > > >> Zhu
> > > >>
> > > >> Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 13:49写道：
> > > >>
> > > >> > Thanks Yuxin for your answer. +1 for this proposal.
> > > >> >
> > > >> > Best,
> > > >> > Junrui.
> > > >> >
> > > >> > Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 13:42写道：
> > > >> >
> > > >> > > Thanks Junrui for your question.
> > > >> > >
> > > >> > > > I wonder if the current interface design support the
> > > >> > > future adaptation for batch job recovery
> > > >> > >
> > > >> > > I noticed that FLIP-383 supports batch job recovery by
> introducing
> > > >> > > some new APIs. These APIs can also be added to the Tier-related
> > > >> > > interfaces to facilitate the feature. Since these modifications
> > are
> > > >> not
> > > >> > > directly related to the current integration tasks and the
> > > integration
> > > >> > > does not conflict with the batch job recovery, I propose that
> this
> > > >> FLIP
> > > >> > > doesn't involve these particular changes. Moreover, considering
> > that
> > > >> > > the Tier interfaces are not public currently, it is also
> feasible
> > to
> > > >> add
> > > >> > > the interfaces directly if necessary.
> > > >> > > WDYT?
> > > >> > >
> > > >> > > Best,
> > > >> > > Yuxin
> > > >> > >
> > > >> > >
> > > >> > > Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 11:02写道：
> > > >> > >
> > > >> > > > Thanks Yuxin for driving this proposal!
> > > >> > > >
> > > >> > > > I have a question about the public interface compatibility in
> > the
> > > >> > context
> > > >> > > > of FLIP-459. As we've supported batch job recovery from
> > jobMaster
> > > >> > > failures
> > > >> > > > in FLIP-383 which will be released in Flink 1.20. I wonder if
> > the
> > > >> > current
> > > >> > > > interface design support the future adaptation for batch job
> > > >> recovery?
> > > >> > > >
> > > >> > > > Looking forward to your feedback.
> > > >> > > >
> > > >> > > > Best,
> > > >> > > > Junrui.
> > > >> > > >
> > > >> > > > weijie guo <gu...@gmail.com> 于2024年6月5日周三 10:13写道：
> > > >> > > >
> > > >> > > > > Thanks Yuxin for the proposal!
> > > >> > > > >
> > > >> > > > > When we first proposed Hybrid Shuffle, I wanted to support
> > > >> pluggable
> > > >> > > > > storage tier in the future. However, limited by the
> > architecture
> > > >> of
> > > >> > the
> > > >> > > > > legacy Hybrid Shuffle at that time, this idea has not been
> > > >> realized.
> > > >> > > The
> > > >> > > > > new architecture abstracts the tier nicely, and now it's
> time
> > to
> > > >> > > > introduce
> > > >> > > > > support for external storage.
> > > >> > > > >
> > > >> > > > > Big +1 for this one!
> > > >> > > > >
> > > >> > > > > Best regards,
> > > >> > > > >
> > > >> > > > > Weijie
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：
> > > >> > > > >
> > > >> > > > > > Thanks Yuxin for the proposal. +1,  as a member of the
> > Apache
> > > >> > > Celeborn
> > > >> > > > > > community, I am very excited about the integration of
> > Flink's
> > > >> > Hybrid
> > > >> > > > > > Shuffle with Apache Celeborn. The whole design of CIP-6
> > looks
> > > >> good
> > > >> > to
> > > >> > > > > me. I
> > > >> > > > > > am looking forward to this integration.
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > Jiashu Xiong
> > > >> > > > > >
> > > >> > > > > > Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
> > > >> > > > > >
> > > >> > > > > > > +1 for this proposal.
> > > >> > > > > > >
> > > >> > > > > > > After internally reviewing the prototype of CIP-6, this
> > > would
> > > >> > > improve
> > > >> > > > > > > performance and stability for Flink users using
> Celeborn.
> > > >> > > > > > >
> > > >> > > > > > > Expect to see this feature come out to the community.
> > > >> > > > > > >
> > > >> > > > > > > As I come from the Celeborn community, I hope more users
> > can
> > > >> try
> > > >> > to
> > > >> > > > > > > use Celeborn when there are Flink batch jobs.
> > > >> > > > > > >
> > > >> > > > > > > Thanks,
> > > >> > > > > > > Ethan Feng
> > > >> > > > > > >
> > > >> > > > > > > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二
> 16:34写道：
> > > >> > > > > > > >
> > > >> > > > > > > > Hi, Venkatakrishnan,
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks for joining the discussion. We appreciate your
> > > >> interest
> > > >> > > > > > > > in contributing to the work. Once the FLIP and CIP
> > > proposals
> > > >> > > > > > > > have been approved, we will create some JIRA tickets
> in
> > > >> Flink
> > > >> > > > > > > > and Celeborn projects. Please feel free to take a look
> > at
> > > >> the
> > > >> > > > > > > > tickets and select any that resonate with your
> > interests.
> > > >> > > > > > > >
> > > >> > > > > > > > Best,
> > > >> > > > > > > > Yuxin
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > Venkatakrishnan Sowrirajan <vs...@asu.edu>
> > > 于2024年5月31日周五
> > > >> > > > 23:11写道：
> > > >> > > > > > > >
> > > >> > > > > > > > > Thanks for this FLIP. We are also interested in
> > > >> > > > > learning/contributing
> > > >> > > > > > > to
> > > >> > > > > > > > > the hybrid shuffle integration with celeborn for
> batch
> > > >> > > > executions.
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <
> > > >> > > tanyuxinwork@gmail.com>
> > > >> > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Hi, Xintong,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > >  I think we can also publish the prototype codes
> > so
> > > >> the
> > > >> > > > > > > > > > community can better understand and help with it.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Ok, I agree on the point. I will prepare and
> publish
> > > the
> > > >> > code
> > > >> > > > > > > > > > recently.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Rui,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
> > > >> loaded.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for the reminder. I've updated the images.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Best,
> > > >> > > > > > > > > > Yuxin
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三
> > 09:33写道：
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Thanks Yuxin for driving this proposal!
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
> > > >> loaded.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > [1]
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Best,
> > > >> > > > > > > > > > > Rui
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > > >> > > > > > > tonysong820@gmail.com>
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > +1 for this proposal.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > We have been prototyping this feature
> internally
> > > at
> > > >> > > Alibaba
> > > >> > > > > > for a
> > > >> > > > > > > > > > couple
> > > >> > > > > > > > > > > of
> > > >> > > > > > > > > > > > months. Yuxin, I think we can also publish the
> > > >> > prototype
> > > >> > > > > codes
> > > >> > > > > > > so the
> > > >> > > > > > > > > > > > community can better understand and help with
> > it.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Xintong
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> > > >> > > > > > > tanyuxinwork@gmail.com>
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Hi all,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > I would like to start a discussion on
> FLIP-459
> > > >> > Support
> > > >> > > > > Flink
> > > >> > > > > > > hybrid
> > > >> > > > > > > > > > > > shuffle
> > > >> > > > > > > > > > > > > integration with
> > > >> > > > > > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle
> > > supports
> > > >> > > > > transitions
> > > >> > > > > > > > > between
> > > >> > > > > > > > > > > > > memory, disk, and
> > > >> > > > > > > > > > > > > remote storage to improve performance and
> job
> > > >> > > stability.
> > > >> > > > > > > > > > Concurrently,
> > > >> > > > > > > > > > > > > Apache Celeborn
> > > >> > > > > > > > > > > > > provides a stable, performant, scalable
> remote
> > > >> > shuffle
> > > >> > > > > > service.
> > > >> > > > > > > > > This
> > > >> > > > > > > > > > > > > integration proposal is to
> > > >> > > > > > > > > > > > > harness the benefits from both hybrid
> shuffle
> > > and
> > > >> > > > Celeborn
> > > >> > > > > > > > > > > > simultaneously.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Note that this proposal has two parts.
> > > >> > > > > > > > > > > > > 1. The Flink-side modifications are in
> > > >> FLIP-459[1].
> > > >> > > > > > > > > > > > > 2. The Celeborn-side changes are in
> CIP-6[2].
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Looking forward to everyone's feedback and
> > > >> > suggestions.
> > > >> > > > > Thank
> > > >> > > > > > > you!
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > [1]
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > >> > > > > > > > > > > > > [2]
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > Yuxin
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Venkatakrishnan Sowrirajan <vs...@asu.edu>.

Yuxin,

One question, in the current proposal is it limited to only one tiered
storage implementation for eg: celeborn? Is it possible to have multiple
tiered storages like a separate cloud storage and Celeborn or some such?


On Thu, Jun 6, 2024, 9:46 AM Jeyhun Karimov <je...@gmail.com> wrote:

> Hi Yuxin,
>
> +1 for this proposal.
> This change will greatly alleviate the pressure on local storage resources
> (especially when there is limited local storage)
> particularly in the context of cloud-native environments.
>
> Regards,
> Jeyhun
>
> On Thu, Jun 6, 2024 at 1:20 PM Yuxin Tan <ta...@gmail.com> wrote:
>
> > Hi all,
> >
> > Thanks for all the feedback and suggestions so far.
> >
> > If there is no further comment, we will open the voting thread tomorrow.
> >
> > Best,
> > Yuxin
> >
> >
> > Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 15:40写道：
> >
> > > Thanks Zhu for the suggestion.
> > > I have updated the description of the option.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Zhu Zhu <re...@gmail.com> 于2024年6月6日周四 14:59写道：
> > >
> > >> +1
> > >>
> > >> Maybe explain in the description of
> > >>
> `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
> > >> that it only accepts Celeborn as the remote shuffle tier at this
> moment?
> > >>
> > >> Thanks,
> > >> Zhu
> > >>
> > >> Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 13:49写道：
> > >>
> > >> > Thanks Yuxin for your answer. +1 for this proposal.
> > >> >
> > >> > Best,
> > >> > Junrui.
> > >> >
> > >> > Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 13:42写道：
> > >> >
> > >> > > Thanks Junrui for your question.
> > >> > >
> > >> > > > I wonder if the current interface design support the
> > >> > > future adaptation for batch job recovery
> > >> > >
> > >> > > I noticed that FLIP-383 supports batch job recovery by introducing
> > >> > > some new APIs. These APIs can also be added to the Tier-related
> > >> > > interfaces to facilitate the feature. Since these modifications
> are
> > >> not
> > >> > > directly related to the current integration tasks and the
> > integration
> > >> > > does not conflict with the batch job recovery, I propose that this
> > >> FLIP
> > >> > > doesn't involve these particular changes. Moreover, considering
> that
> > >> > > the Tier interfaces are not public currently, it is also feasible
> to
> > >> add
> > >> > > the interfaces directly if necessary.
> > >> > > WDYT?
> > >> > >
> > >> > > Best,
> > >> > > Yuxin
> > >> > >
> > >> > >
> > >> > > Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 11:02写道：
> > >> > >
> > >> > > > Thanks Yuxin for driving this proposal!
> > >> > > >
> > >> > > > I have a question about the public interface compatibility in
> the
> > >> > context
> > >> > > > of FLIP-459. As we've supported batch job recovery from
> jobMaster
> > >> > > failures
> > >> > > > in FLIP-383 which will be released in Flink 1.20. I wonder if
> the
> > >> > current
> > >> > > > interface design support the future adaptation for batch job
> > >> recovery?
> > >> > > >
> > >> > > > Looking forward to your feedback.
> > >> > > >
> > >> > > > Best,
> > >> > > > Junrui.
> > >> > > >
> > >> > > > weijie guo <gu...@gmail.com> 于2024年6月5日周三 10:13写道：
> > >> > > >
> > >> > > > > Thanks Yuxin for the proposal!
> > >> > > > >
> > >> > > > > When we first proposed Hybrid Shuffle, I wanted to support
> > >> pluggable
> > >> > > > > storage tier in the future. However, limited by the
> architecture
> > >> of
> > >> > the
> > >> > > > > legacy Hybrid Shuffle at that time, this idea has not been
> > >> realized.
> > >> > > The
> > >> > > > > new architecture abstracts the tier nicely, and now it's time
> to
> > >> > > > introduce
> > >> > > > > support for external storage.
> > >> > > > >
> > >> > > > > Big +1 for this one!
> > >> > > > >
> > >> > > > > Best regards,
> > >> > > > >
> > >> > > > > Weijie
> > >> > > > >
> > >> > > > >
> > >> > > > > rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：
> > >> > > > >
> > >> > > > > > Thanks Yuxin for the proposal. +1,  as a member of the
> Apache
> > >> > > Celeborn
> > >> > > > > > community, I am very excited about the integration of
> Flink's
> > >> > Hybrid
> > >> > > > > > Shuffle with Apache Celeborn. The whole design of CIP-6
> looks
> > >> good
> > >> > to
> > >> > > > > me. I
> > >> > > > > > am looking forward to this integration.
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Jiashu Xiong
> > >> > > > > >
> > >> > > > > > Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
> > >> > > > > >
> > >> > > > > > > +1 for this proposal.
> > >> > > > > > >
> > >> > > > > > > After internally reviewing the prototype of CIP-6, this
> > would
> > >> > > improve
> > >> > > > > > > performance and stability for Flink users using Celeborn.
> > >> > > > > > >
> > >> > > > > > > Expect to see this feature come out to the community.
> > >> > > > > > >
> > >> > > > > > > As I come from the Celeborn community, I hope more users
> can
> > >> try
> > >> > to
> > >> > > > > > > use Celeborn when there are Flink batch jobs.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Ethan Feng
> > >> > > > > > >
> > >> > > > > > > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
> > >> > > > > > > >
> > >> > > > > > > > Hi, Venkatakrishnan,
> > >> > > > > > > >
> > >> > > > > > > > Thanks for joining the discussion. We appreciate your
> > >> interest
> > >> > > > > > > > in contributing to the work. Once the FLIP and CIP
> > proposals
> > >> > > > > > > > have been approved, we will create some JIRA tickets in
> > >> Flink
> > >> > > > > > > > and Celeborn projects. Please feel free to take a look
> at
> > >> the
> > >> > > > > > > > tickets and select any that resonate with your
> interests.
> > >> > > > > > > >
> > >> > > > > > > > Best,
> > >> > > > > > > > Yuxin
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Venkatakrishnan Sowrirajan <vs...@asu.edu>
> > 于2024年5月31日周五
> > >> > > > 23:11写道：
> > >> > > > > > > >
> > >> > > > > > > > > Thanks for this FLIP. We are also interested in
> > >> > > > > learning/contributing
> > >> > > > > > > to
> > >> > > > > > > > > the hybrid shuffle integration with celeborn for batch
> > >> > > > executions.
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <
> > >> > > tanyuxinwork@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi, Xintong,
> > >> > > > > > > > > >
> > >> > > > > > > > > > >  I think we can also publish the prototype codes
> so
> > >> the
> > >> > > > > > > > > > community can better understand and help with it.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Ok, I agree on the point. I will prepare and publish
> > the
> > >> > code
> > >> > > > > > > > > > recently.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Rui,
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
> > >> loaded.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for the reminder. I've updated the images.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Best,
> > >> > > > > > > > > > Yuxin
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三
> 09:33写道：
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Thanks Yuxin for driving this proposal!
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
> > >> loaded.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > [1]
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Best,
> > >> > > > > > > > > > > Rui
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > >> > > > > > > tonysong820@gmail.com>
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > +1 for this proposal.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > We have been prototyping this feature internally
> > at
> > >> > > Alibaba
> > >> > > > > > for a
> > >> > > > > > > > > > couple
> > >> > > > > > > > > > > of
> > >> > > > > > > > > > > > months. Yuxin, I think we can also publish the
> > >> > prototype
> > >> > > > > codes
> > >> > > > > > > so the
> > >> > > > > > > > > > > > community can better understand and help with
> it.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Best,
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Xintong
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> > >> > > > > > > tanyuxinwork@gmail.com>
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hi all,
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > I would like to start a discussion on FLIP-459
> > >> > Support
> > >> > > > > Flink
> > >> > > > > > > hybrid
> > >> > > > > > > > > > > > shuffle
> > >> > > > > > > > > > > > > integration with
> > >> > > > > > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle
> > supports
> > >> > > > > transitions
> > >> > > > > > > > > between
> > >> > > > > > > > > > > > > memory, disk, and
> > >> > > > > > > > > > > > > remote storage to improve performance and job
> > >> > > stability.
> > >> > > > > > > > > > Concurrently,
> > >> > > > > > > > > > > > > Apache Celeborn
> > >> > > > > > > > > > > > > provides a stable, performant, scalable remote
> > >> > shuffle
> > >> > > > > > service.
> > >> > > > > > > > > This
> > >> > > > > > > > > > > > > integration proposal is to
> > >> > > > > > > > > > > > > harness the benefits from both hybrid shuffle
> > and
> > >> > > > Celeborn
> > >> > > > > > > > > > > > simultaneously.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Note that this proposal has two parts.
> > >> > > > > > > > > > > > > 1. The Flink-side modifications are in
> > >> FLIP-459[1].
> > >> > > > > > > > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Looking forward to everyone's feedback and
> > >> > suggestions.
> > >> > > > > Thank
> > >> > > > > > > you!
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > [1]
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > >> > > > > > > > > > > > > [2]
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > Yuxin
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Jeyhun Karimov <je...@gmail.com>.

Hi Yuxin,

+1 for this proposal.
This change will greatly alleviate the pressure on local storage resources
(especially when there is limited local storage)
particularly in the context of cloud-native environments.

Regards,
Jeyhun

On Thu, Jun 6, 2024 at 1:20 PM Yuxin Tan <ta...@gmail.com> wrote:

> Hi all,
>
> Thanks for all the feedback and suggestions so far.
>
> If there is no further comment, we will open the voting thread tomorrow.
>
> Best,
> Yuxin
>
>
> Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 15:40写道：
>
> > Thanks Zhu for the suggestion.
> > I have updated the description of the option.
> >
> > Best,
> > Yuxin
> >
> >
> > Zhu Zhu <re...@gmail.com> 于2024年6月6日周四 14:59写道：
> >
> >> +1
> >>
> >> Maybe explain in the description of
> >> `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
> >> that it only accepts Celeborn as the remote shuffle tier at this moment?
> >>
> >> Thanks,
> >> Zhu
> >>
> >> Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 13:49写道：
> >>
> >> > Thanks Yuxin for your answer. +1 for this proposal.
> >> >
> >> > Best,
> >> > Junrui.
> >> >
> >> > Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 13:42写道：
> >> >
> >> > > Thanks Junrui for your question.
> >> > >
> >> > > > I wonder if the current interface design support the
> >> > > future adaptation for batch job recovery
> >> > >
> >> > > I noticed that FLIP-383 supports batch job recovery by introducing
> >> > > some new APIs. These APIs can also be added to the Tier-related
> >> > > interfaces to facilitate the feature. Since these modifications are
> >> not
> >> > > directly related to the current integration tasks and the
> integration
> >> > > does not conflict with the batch job recovery, I propose that this
> >> FLIP
> >> > > doesn't involve these particular changes. Moreover, considering that
> >> > > the Tier interfaces are not public currently, it is also feasible to
> >> add
> >> > > the interfaces directly if necessary.
> >> > > WDYT?
> >> > >
> >> > > Best,
> >> > > Yuxin
> >> > >
> >> > >
> >> > > Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 11:02写道：
> >> > >
> >> > > > Thanks Yuxin for driving this proposal!
> >> > > >
> >> > > > I have a question about the public interface compatibility in the
> >> > context
> >> > > > of FLIP-459. As we've supported batch job recovery from jobMaster
> >> > > failures
> >> > > > in FLIP-383 which will be released in Flink 1.20. I wonder if the
> >> > current
> >> > > > interface design support the future adaptation for batch job
> >> recovery?
> >> > > >
> >> > > > Looking forward to your feedback.
> >> > > >
> >> > > > Best,
> >> > > > Junrui.
> >> > > >
> >> > > > weijie guo <gu...@gmail.com> 于2024年6月5日周三 10:13写道：
> >> > > >
> >> > > > > Thanks Yuxin for the proposal!
> >> > > > >
> >> > > > > When we first proposed Hybrid Shuffle, I wanted to support
> >> pluggable
> >> > > > > storage tier in the future. However, limited by the architecture
> >> of
> >> > the
> >> > > > > legacy Hybrid Shuffle at that time, this idea has not been
> >> realized.
> >> > > The
> >> > > > > new architecture abstracts the tier nicely, and now it's time to
> >> > > > introduce
> >> > > > > support for external storage.
> >> > > > >
> >> > > > > Big +1 for this one!
> >> > > > >
> >> > > > > Best regards,
> >> > > > >
> >> > > > > Weijie
> >> > > > >
> >> > > > >
> >> > > > > rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：
> >> > > > >
> >> > > > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
> >> > > Celeborn
> >> > > > > > community, I am very excited about the integration of Flink's
> >> > Hybrid
> >> > > > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks
> >> good
> >> > to
> >> > > > > me. I
> >> > > > > > am looking forward to this integration.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Jiashu Xiong
> >> > > > > >
> >> > > > > > Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
> >> > > > > >
> >> > > > > > > +1 for this proposal.
> >> > > > > > >
> >> > > > > > > After internally reviewing the prototype of CIP-6, this
> would
> >> > > improve
> >> > > > > > > performance and stability for Flink users using Celeborn.
> >> > > > > > >
> >> > > > > > > Expect to see this feature come out to the community.
> >> > > > > > >
> >> > > > > > > As I come from the Celeborn community, I hope more users can
> >> try
> >> > to
> >> > > > > > > use Celeborn when there are Flink batch jobs.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Ethan Feng
> >> > > > > > >
> >> > > > > > > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
> >> > > > > > > >
> >> > > > > > > > Hi, Venkatakrishnan,
> >> > > > > > > >
> >> > > > > > > > Thanks for joining the discussion. We appreciate your
> >> interest
> >> > > > > > > > in contributing to the work. Once the FLIP and CIP
> proposals
> >> > > > > > > > have been approved, we will create some JIRA tickets in
> >> Flink
> >> > > > > > > > and Celeborn projects. Please feel free to take a look at
> >> the
> >> > > > > > > > tickets and select any that resonate with your interests.
> >> > > > > > > >
> >> > > > > > > > Best,
> >> > > > > > > > Yuxin
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Venkatakrishnan Sowrirajan <vs...@asu.edu>
> 于2024年5月31日周五
> >> > > > 23:11写道：
> >> > > > > > > >
> >> > > > > > > > > Thanks for this FLIP. We are also interested in
> >> > > > > learning/contributing
> >> > > > > > > to
> >> > > > > > > > > the hybrid shuffle integration with celeborn for batch
> >> > > > executions.
> >> > > > > > > > >
> >> > > > > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <
> >> > > tanyuxinwork@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi, Xintong,
> >> > > > > > > > > >
> >> > > > > > > > > > >  I think we can also publish the prototype codes so
> >> the
> >> > > > > > > > > > community can better understand and help with it.
> >> > > > > > > > > >
> >> > > > > > > > > > Ok, I agree on the point. I will prepare and publish
> the
> >> > code
> >> > > > > > > > > > recently.
> >> > > > > > > > > >
> >> > > > > > > > > > Rui,
> >> > > > > > > > > >
> >> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
> >> loaded.
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for the reminder. I've updated the images.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Best,
> >> > > > > > > > > > Yuxin
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> >> > > > > > > > > >
> >> > > > > > > > > > > Thanks Yuxin for driving this proposal!
> >> > > > > > > > > > >
> >> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
> >> loaded.
> >> > > > > > > > > > >
> >> > > > > > > > > > > [1]
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> >> > > > > > > > > > >
> >> > > > > > > > > > > Best,
> >> > > > > > > > > > > Rui
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> >> > > > > > > tonysong820@gmail.com>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > +1 for this proposal.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > We have been prototyping this feature internally
> at
> >> > > Alibaba
> >> > > > > > for a
> >> > > > > > > > > > couple
> >> > > > > > > > > > > of
> >> > > > > > > > > > > > months. Yuxin, I think we can also publish the
> >> > prototype
> >> > > > > codes
> >> > > > > > > so the
> >> > > > > > > > > > > > community can better understand and help with it.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Best,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Xintong
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> >> > > > > > > tanyuxinwork@gmail.com>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hi all,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > I would like to start a discussion on FLIP-459
> >> > Support
> >> > > > > Flink
> >> > > > > > > hybrid
> >> > > > > > > > > > > > shuffle
> >> > > > > > > > > > > > > integration with
> >> > > > > > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle
> supports
> >> > > > > transitions
> >> > > > > > > > > between
> >> > > > > > > > > > > > > memory, disk, and
> >> > > > > > > > > > > > > remote storage to improve performance and job
> >> > > stability.
> >> > > > > > > > > > Concurrently,
> >> > > > > > > > > > > > > Apache Celeborn
> >> > > > > > > > > > > > > provides a stable, performant, scalable remote
> >> > shuffle
> >> > > > > > service.
> >> > > > > > > > > This
> >> > > > > > > > > > > > > integration proposal is to
> >> > > > > > > > > > > > > harness the benefits from both hybrid shuffle
> and
> >> > > > Celeborn
> >> > > > > > > > > > > > simultaneously.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Note that this proposal has two parts.
> >> > > > > > > > > > > > > 1. The Flink-side modifications are in
> >> FLIP-459[1].
> >> > > > > > > > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Looking forward to everyone's feedback and
> >> > suggestions.
> >> > > > > Thank
> >> > > > > > > you!
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > [1]
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> >> > > > > > > > > > > > > [2]
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Best,
> >> > > > > > > > > > > > > Yuxin
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Yuxin Tan <ta...@gmail.com>.

Hi all,

Thanks for all the feedback and suggestions so far.

If there is no further comment, we will open the voting thread tomorrow.

Best,
Yuxin


Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 15:40写道：

> Thanks Zhu for the suggestion.
> I have updated the description of the option.
>
> Best,
> Yuxin
>
>
> Zhu Zhu <re...@gmail.com> 于2024年6月6日周四 14:59写道：
>
>> +1
>>
>> Maybe explain in the description of
>> `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
>> that it only accepts Celeborn as the remote shuffle tier at this moment?
>>
>> Thanks,
>> Zhu
>>
>> Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 13:49写道：
>>
>> > Thanks Yuxin for your answer. +1 for this proposal.
>> >
>> > Best,
>> > Junrui.
>> >
>> > Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 13:42写道：
>> >
>> > > Thanks Junrui for your question.
>> > >
>> > > > I wonder if the current interface design support the
>> > > future adaptation for batch job recovery
>> > >
>> > > I noticed that FLIP-383 supports batch job recovery by introducing
>> > > some new APIs. These APIs can also be added to the Tier-related
>> > > interfaces to facilitate the feature. Since these modifications are
>> not
>> > > directly related to the current integration tasks and the integration
>> > > does not conflict with the batch job recovery, I propose that this
>> FLIP
>> > > doesn't involve these particular changes. Moreover, considering that
>> > > the Tier interfaces are not public currently, it is also feasible to
>> add
>> > > the interfaces directly if necessary.
>> > > WDYT?
>> > >
>> > > Best,
>> > > Yuxin
>> > >
>> > >
>> > > Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 11:02写道：
>> > >
>> > > > Thanks Yuxin for driving this proposal!
>> > > >
>> > > > I have a question about the public interface compatibility in the
>> > context
>> > > > of FLIP-459. As we've supported batch job recovery from jobMaster
>> > > failures
>> > > > in FLIP-383 which will be released in Flink 1.20. I wonder if the
>> > current
>> > > > interface design support the future adaptation for batch job
>> recovery?
>> > > >
>> > > > Looking forward to your feedback.
>> > > >
>> > > > Best,
>> > > > Junrui.
>> > > >
>> > > > weijie guo <gu...@gmail.com> 于2024年6月5日周三 10:13写道：
>> > > >
>> > > > > Thanks Yuxin for the proposal!
>> > > > >
>> > > > > When we first proposed Hybrid Shuffle, I wanted to support
>> pluggable
>> > > > > storage tier in the future. However, limited by the architecture
>> of
>> > the
>> > > > > legacy Hybrid Shuffle at that time, this idea has not been
>> realized.
>> > > The
>> > > > > new architecture abstracts the tier nicely, and now it's time to
>> > > > introduce
>> > > > > support for external storage.
>> > > > >
>> > > > > Big +1 for this one!
>> > > > >
>> > > > > Best regards,
>> > > > >
>> > > > > Weijie
>> > > > >
>> > > > >
>> > > > > rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：
>> > > > >
>> > > > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
>> > > Celeborn
>> > > > > > community, I am very excited about the integration of Flink's
>> > Hybrid
>> > > > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks
>> good
>> > to
>> > > > > me. I
>> > > > > > am looking forward to this integration.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Jiashu Xiong
>> > > > > >
>> > > > > > Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
>> > > > > >
>> > > > > > > +1 for this proposal.
>> > > > > > >
>> > > > > > > After internally reviewing the prototype of CIP-6, this would
>> > > improve
>> > > > > > > performance and stability for Flink users using Celeborn.
>> > > > > > >
>> > > > > > > Expect to see this feature come out to the community.
>> > > > > > >
>> > > > > > > As I come from the Celeborn community, I hope more users can
>> try
>> > to
>> > > > > > > use Celeborn when there are Flink batch jobs.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Ethan Feng
>> > > > > > >
>> > > > > > > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
>> > > > > > > >
>> > > > > > > > Hi, Venkatakrishnan,
>> > > > > > > >
>> > > > > > > > Thanks for joining the discussion. We appreciate your
>> interest
>> > > > > > > > in contributing to the work. Once the FLIP and CIP proposals
>> > > > > > > > have been approved, we will create some JIRA tickets in
>> Flink
>> > > > > > > > and Celeborn projects. Please feel free to take a look at
>> the
>> > > > > > > > tickets and select any that resonate with your interests.
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Yuxin
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五
>> > > > 23:11写道：
>> > > > > > > >
>> > > > > > > > > Thanks for this FLIP. We are also interested in
>> > > > > learning/contributing
>> > > > > > > to
>> > > > > > > > > the hybrid shuffle integration with celeborn for batch
>> > > > executions.
>> > > > > > > > >
>> > > > > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <
>> > > tanyuxinwork@gmail.com>
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi, Xintong,
>> > > > > > > > > >
>> > > > > > > > > > >  I think we can also publish the prototype codes so
>> the
>> > > > > > > > > > community can better understand and help with it.
>> > > > > > > > > >
>> > > > > > > > > > Ok, I agree on the point. I will prepare and publish the
>> > code
>> > > > > > > > > > recently.
>> > > > > > > > > >
>> > > > > > > > > > Rui,
>> > > > > > > > > >
>> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
>> loaded.
>> > > > > > > > > >
>> > > > > > > > > > Thanks for the reminder. I've updated the images.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Best,
>> > > > > > > > > > Yuxin
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
>> > > > > > > > > >
>> > > > > > > > > > > Thanks Yuxin for driving this proposal!
>> > > > > > > > > > >
>> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
>> loaded.
>> > > > > > > > > > >
>> > > > > > > > > > > [1]
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
>> > > > > > > > > > >
>> > > > > > > > > > > Best,
>> > > > > > > > > > > Rui
>> > > > > > > > > > >
>> > > > > > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
>> > > > > > > tonysong820@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > +1 for this proposal.
>> > > > > > > > > > > >
>> > > > > > > > > > > > We have been prototyping this feature internally at
>> > > Alibaba
>> > > > > > for a
>> > > > > > > > > > couple
>> > > > > > > > > > > of
>> > > > > > > > > > > > months. Yuxin, I think we can also publish the
>> > prototype
>> > > > > codes
>> > > > > > > so the
>> > > > > > > > > > > > community can better understand and help with it.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Best,
>> > > > > > > > > > > >
>> > > > > > > > > > > > Xintong
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
>> > > > > > > tanyuxinwork@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi all,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I would like to start a discussion on FLIP-459
>> > Support
>> > > > > Flink
>> > > > > > > hybrid
>> > > > > > > > > > > > shuffle
>> > > > > > > > > > > > > integration with
>> > > > > > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports
>> > > > > transitions
>> > > > > > > > > between
>> > > > > > > > > > > > > memory, disk, and
>> > > > > > > > > > > > > remote storage to improve performance and job
>> > > stability.
>> > > > > > > > > > Concurrently,
>> > > > > > > > > > > > > Apache Celeborn
>> > > > > > > > > > > > > provides a stable, performant, scalable remote
>> > shuffle
>> > > > > > service.
>> > > > > > > > > This
>> > > > > > > > > > > > > integration proposal is to
>> > > > > > > > > > > > > harness the benefits from both hybrid shuffle and
>> > > > Celeborn
>> > > > > > > > > > > > simultaneously.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Note that this proposal has two parts.
>> > > > > > > > > > > > > 1. The Flink-side modifications are in
>> FLIP-459[1].
>> > > > > > > > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Looking forward to everyone's feedback and
>> > suggestions.
>> > > > > Thank
>> > > > > > > you!
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > [1]
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
>> > > > > > > > > > > > > [2]
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > Yuxin
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Yuxin Tan <ta...@gmail.com>.

Thanks Zhu for the suggestion.
I have updated the description of the option.

Best,
Yuxin


Zhu Zhu <re...@gmail.com> 于2024年6月6日周四 14:59写道：

> +1
>
> Maybe explain in the description of
> `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
> that it only accepts Celeborn as the remote shuffle tier at this moment?
>
> Thanks,
> Zhu
>
> Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 13:49写道：
>
> > Thanks Yuxin for your answer. +1 for this proposal.
> >
> > Best,
> > Junrui.
> >
> > Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 13:42写道：
> >
> > > Thanks Junrui for your question.
> > >
> > > > I wonder if the current interface design support the
> > > future adaptation for batch job recovery
> > >
> > > I noticed that FLIP-383 supports batch job recovery by introducing
> > > some new APIs. These APIs can also be added to the Tier-related
> > > interfaces to facilitate the feature. Since these modifications are not
> > > directly related to the current integration tasks and the integration
> > > does not conflict with the batch job recovery, I propose that this FLIP
> > > doesn't involve these particular changes. Moreover, considering that
> > > the Tier interfaces are not public currently, it is also feasible to
> add
> > > the interfaces directly if necessary.
> > > WDYT?
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 11:02写道：
> > >
> > > > Thanks Yuxin for driving this proposal!
> > > >
> > > > I have a question about the public interface compatibility in the
> > context
> > > > of FLIP-459. As we've supported batch job recovery from jobMaster
> > > failures
> > > > in FLIP-383 which will be released in Flink 1.20. I wonder if the
> > current
> > > > interface design support the future adaptation for batch job
> recovery?
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > > Best,
> > > > Junrui.
> > > >
> > > > weijie guo <gu...@gmail.com> 于2024年6月5日周三 10:13写道：
> > > >
> > > > > Thanks Yuxin for the proposal!
> > > > >
> > > > > When we first proposed Hybrid Shuffle, I wanted to support
> pluggable
> > > > > storage tier in the future. However, limited by the architecture of
> > the
> > > > > legacy Hybrid Shuffle at that time, this idea has not been
> realized.
> > > The
> > > > > new architecture abstracts the tier nicely, and now it's time to
> > > > introduce
> > > > > support for external storage.
> > > > >
> > > > > Big +1 for this one!
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Weijie
> > > > >
> > > > >
> > > > > rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：
> > > > >
> > > > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
> > > Celeborn
> > > > > > community, I am very excited about the integration of Flink's
> > Hybrid
> > > > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks
> good
> > to
> > > > > me. I
> > > > > > am looking forward to this integration.
> > > > > >
> > > > > > Thanks,
> > > > > > Jiashu Xiong
> > > > > >
> > > > > > Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
> > > > > >
> > > > > > > +1 for this proposal.
> > > > > > >
> > > > > > > After internally reviewing the prototype of CIP-6, this would
> > > improve
> > > > > > > performance and stability for Flink users using Celeborn.
> > > > > > >
> > > > > > > Expect to see this feature come out to the community.
> > > > > > >
> > > > > > > As I come from the Celeborn community, I hope more users can
> try
> > to
> > > > > > > use Celeborn when there are Flink batch jobs.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ethan Feng
> > > > > > >
> > > > > > > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
> > > > > > > >
> > > > > > > > Hi, Venkatakrishnan,
> > > > > > > >
> > > > > > > > Thanks for joining the discussion. We appreciate your
> interest
> > > > > > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > > > > > have been approved, we will create some JIRA tickets in Flink
> > > > > > > > and Celeborn projects. Please feel free to take a look at the
> > > > > > > > tickets and select any that resonate with your interests.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yuxin
> > > > > > > >
> > > > > > > >
> > > > > > > > Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五
> > > > 23:11写道：
> > > > > > > >
> > > > > > > > > Thanks for this FLIP. We are also interested in
> > > > > learning/contributing
> > > > > > > to
> > > > > > > > > the hybrid shuffle integration with celeborn for batch
> > > > executions.
> > > > > > > > >
> > > > > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <
> > > tanyuxinwork@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Xintong,
> > > > > > > > > >
> > > > > > > > > > >  I think we can also publish the prototype codes so the
> > > > > > > > > > community can better understand and help with it.
> > > > > > > > > >
> > > > > > > > > > Ok, I agree on the point. I will prepare and publish the
> > code
> > > > > > > > > > recently.
> > > > > > > > > >
> > > > > > > > > > Rui,
> > > > > > > > > >
> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
> loaded.
> > > > > > > > > >
> > > > > > > > > > Thanks for the reminder. I've updated the images.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Yuxin
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > > > > > > > >
> > > > > > > > > > > Thanks Yuxin for driving this proposal!
> > > > > > > > > > >
> > > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be
> loaded.
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Rui
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > > > > > > tonysong820@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1 for this proposal.
> > > > > > > > > > > >
> > > > > > > > > > > > We have been prototyping this feature internally at
> > > Alibaba
> > > > > > for a
> > > > > > > > > > couple
> > > > > > > > > > > of
> > > > > > > > > > > > months. Yuxin, I think we can also publish the
> > prototype
> > > > > codes
> > > > > > > so the
> > > > > > > > > > > > community can better understand and help with it.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > >
> > > > > > > > > > > > Xintong
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> > > > > > > tanyuxinwork@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I would like to start a discussion on FLIP-459
> > Support
> > > > > Flink
> > > > > > > hybrid
> > > > > > > > > > > > shuffle
> > > > > > > > > > > > > integration with
> > > > > > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports
> > > > > transitions
> > > > > > > > > between
> > > > > > > > > > > > > memory, disk, and
> > > > > > > > > > > > > remote storage to improve performance and job
> > > stability.
> > > > > > > > > > Concurrently,
> > > > > > > > > > > > > Apache Celeborn
> > > > > > > > > > > > > provides a stable, performant, scalable remote
> > shuffle
> > > > > > service.
> > > > > > > > > This
> > > > > > > > > > > > > integration proposal is to
> > > > > > > > > > > > > harness the benefits from both hybrid shuffle and
> > > > Celeborn
> > > > > > > > > > > > simultaneously.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Note that this proposal has two parts.
> > > > > > > > > > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > > > > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > > > > > > > > > >
> > > > > > > > > > > > > Looking forward to everyone's feedback and
> > suggestions.
> > > > > Thank
> > > > > > > you!
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > > > > > > > > > [2]
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Yuxin
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Zhu Zhu <re...@gmail.com>.

+1

Maybe explain in the description of
`taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
that it only accepts Celeborn as the remote shuffle tier at this moment?

Thanks,
Zhu

Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 13:49写道：

> Thanks Yuxin for your answer. +1 for this proposal.
>
> Best,
> Junrui.
>
> Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 13:42写道：
>
> > Thanks Junrui for your question.
> >
> > > I wonder if the current interface design support the
> > future adaptation for batch job recovery
> >
> > I noticed that FLIP-383 supports batch job recovery by introducing
> > some new APIs. These APIs can also be added to the Tier-related
> > interfaces to facilitate the feature. Since these modifications are not
> > directly related to the current integration tasks and the integration
> > does not conflict with the batch job recovery, I propose that this FLIP
> > doesn't involve these particular changes. Moreover, considering that
> > the Tier interfaces are not public currently, it is also feasible to add
> > the interfaces directly if necessary.
> > WDYT?
> >
> > Best,
> > Yuxin
> >
> >
> > Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 11:02写道：
> >
> > > Thanks Yuxin for driving this proposal!
> > >
> > > I have a question about the public interface compatibility in the
> context
> > > of FLIP-459. As we've supported batch job recovery from jobMaster
> > failures
> > > in FLIP-383 which will be released in Flink 1.20. I wonder if the
> current
> > > interface design support the future adaptation for batch job recovery?
> > >
> > > Looking forward to your feedback.
> > >
> > > Best,
> > > Junrui.
> > >
> > > weijie guo <gu...@gmail.com> 于2024年6月5日周三 10:13写道：
> > >
> > > > Thanks Yuxin for the proposal!
> > > >
> > > > When we first proposed Hybrid Shuffle, I wanted to support pluggable
> > > > storage tier in the future. However, limited by the architecture of
> the
> > > > legacy Hybrid Shuffle at that time, this idea has not been realized.
> > The
> > > > new architecture abstracts the tier nicely, and now it's time to
> > > introduce
> > > > support for external storage.
> > > >
> > > > Big +1 for this one!
> > > >
> > > > Best regards,
> > > >
> > > > Weijie
> > > >
> > > >
> > > > rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：
> > > >
> > > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
> > Celeborn
> > > > > community, I am very excited about the integration of Flink's
> Hybrid
> > > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks good
> to
> > > > me. I
> > > > > am looking forward to this integration.
> > > > >
> > > > > Thanks,
> > > > > Jiashu Xiong
> > > > >
> > > > > Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
> > > > >
> > > > > > +1 for this proposal.
> > > > > >
> > > > > > After internally reviewing the prototype of CIP-6, this would
> > improve
> > > > > > performance and stability for Flink users using Celeborn.
> > > > > >
> > > > > > Expect to see this feature come out to the community.
> > > > > >
> > > > > > As I come from the Celeborn community, I hope more users can try
> to
> > > > > > use Celeborn when there are Flink batch jobs.
> > > > > >
> > > > > > Thanks,
> > > > > > Ethan Feng
> > > > > >
> > > > > > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
> > > > > > >
> > > > > > > Hi, Venkatakrishnan,
> > > > > > >
> > > > > > > Thanks for joining the discussion. We appreciate your interest
> > > > > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > > > > have been approved, we will create some JIRA tickets in Flink
> > > > > > > and Celeborn projects. Please feel free to take a look at the
> > > > > > > tickets and select any that resonate with your interests.
> > > > > > >
> > > > > > > Best,
> > > > > > > Yuxin
> > > > > > >
> > > > > > >
> > > > > > > Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五
> > > 23:11写道：
> > > > > > >
> > > > > > > > Thanks for this FLIP. We are also interested in
> > > > learning/contributing
> > > > > > to
> > > > > > > > the hybrid shuffle integration with celeborn for batch
> > > executions.
> > > > > > > >
> > > > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <
> > tanyuxinwork@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Xintong,
> > > > > > > > >
> > > > > > > > > >  I think we can also publish the prototype codes so the
> > > > > > > > > community can better understand and help with it.
> > > > > > > > >
> > > > > > > > > Ok, I agree on the point. I will prepare and publish the
> code
> > > > > > > > > recently.
> > > > > > > > >
> > > > > > > > > Rui,
> > > > > > > > >
> > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > > > >
> > > > > > > > > Thanks for the reminder. I've updated the images.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Yuxin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > > > > > > >
> > > > > > > > > > Thanks Yuxin for driving this proposal!
> > > > > > > > > >
> > > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Rui
> > > > > > > > > >
> > > > > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > > > > > tonysong820@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 for this proposal.
> > > > > > > > > > >
> > > > > > > > > > > We have been prototyping this feature internally at
> > Alibaba
> > > > > for a
> > > > > > > > > couple
> > > > > > > > > > of
> > > > > > > > > > > months. Yuxin, I think we can also publish the
> prototype
> > > > codes
> > > > > > so the
> > > > > > > > > > > community can better understand and help with it.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > Xintong
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> > > > > > tanyuxinwork@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I would like to start a discussion on FLIP-459
> Support
> > > > Flink
> > > > > > hybrid
> > > > > > > > > > > shuffle
> > > > > > > > > > > > integration with
> > > > > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports
> > > > transitions
> > > > > > > > between
> > > > > > > > > > > > memory, disk, and
> > > > > > > > > > > > remote storage to improve performance and job
> > stability.
> > > > > > > > > Concurrently,
> > > > > > > > > > > > Apache Celeborn
> > > > > > > > > > > > provides a stable, performant, scalable remote
> shuffle
> > > > > service.
> > > > > > > > This
> > > > > > > > > > > > integration proposal is to
> > > > > > > > > > > > harness the benefits from both hybrid shuffle and
> > > Celeborn
> > > > > > > > > > > simultaneously.
> > > > > > > > > > > >
> > > > > > > > > > > > Note that this proposal has two parts.
> > > > > > > > > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > > > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > > > > > > > > >
> > > > > > > > > > > > Looking forward to everyone's feedback and
> suggestions.
> > > > Thank
> > > > > > you!
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > > > > > > > > [2]
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Yuxin
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Junrui Lee <jr...@gmail.com>.

Thanks Yuxin for your answer. +1 for this proposal.

Best,
Junrui.

Yuxin Tan <ta...@gmail.com> 于2024年6月6日周四 13:42写道：

> Thanks Junrui for your question.
>
> > I wonder if the current interface design support the
> future adaptation for batch job recovery
>
> I noticed that FLIP-383 supports batch job recovery by introducing
> some new APIs. These APIs can also be added to the Tier-related
> interfaces to facilitate the feature. Since these modifications are not
> directly related to the current integration tasks and the integration
> does not conflict with the batch job recovery, I propose that this FLIP
> doesn't involve these particular changes. Moreover, considering that
> the Tier interfaces are not public currently, it is also feasible to add
> the interfaces directly if necessary.
> WDYT?
>
> Best,
> Yuxin
>
>
> Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 11:02写道：
>
> > Thanks Yuxin for driving this proposal!
> >
> > I have a question about the public interface compatibility in the context
> > of FLIP-459. As we've supported batch job recovery from jobMaster
> failures
> > in FLIP-383 which will be released in Flink 1.20. I wonder if the current
> > interface design support the future adaptation for batch job recovery?
> >
> > Looking forward to your feedback.
> >
> > Best,
> > Junrui.
> >
> > weijie guo <gu...@gmail.com> 于2024年6月5日周三 10:13写道：
> >
> > > Thanks Yuxin for the proposal!
> > >
> > > When we first proposed Hybrid Shuffle, I wanted to support pluggable
> > > storage tier in the future. However, limited by the architecture of the
> > > legacy Hybrid Shuffle at that time, this idea has not been realized.
> The
> > > new architecture abstracts the tier nicely, and now it's time to
> > introduce
> > > support for external storage.
> > >
> > > Big +1 for this one!
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：
> > >
> > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
> Celeborn
> > > > community, I am very excited about the integration of Flink's Hybrid
> > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to
> > > me. I
> > > > am looking forward to this integration.
> > > >
> > > > Thanks,
> > > > Jiashu Xiong
> > > >
> > > > Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
> > > >
> > > > > +1 for this proposal.
> > > > >
> > > > > After internally reviewing the prototype of CIP-6, this would
> improve
> > > > > performance and stability for Flink users using Celeborn.
> > > > >
> > > > > Expect to see this feature come out to the community.
> > > > >
> > > > > As I come from the Celeborn community, I hope more users can try to
> > > > > use Celeborn when there are Flink batch jobs.
> > > > >
> > > > > Thanks,
> > > > > Ethan Feng
> > > > >
> > > > > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
> > > > > >
> > > > > > Hi, Venkatakrishnan,
> > > > > >
> > > > > > Thanks for joining the discussion. We appreciate your interest
> > > > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > > > have been approved, we will create some JIRA tickets in Flink
> > > > > > and Celeborn projects. Please feel free to take a look at the
> > > > > > tickets and select any that resonate with your interests.
> > > > > >
> > > > > > Best,
> > > > > > Yuxin
> > > > > >
> > > > > >
> > > > > > Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五
> > 23:11写道：
> > > > > >
> > > > > > > Thanks for this FLIP. We are also interested in
> > > learning/contributing
> > > > > to
> > > > > > > the hybrid shuffle integration with celeborn for batch
> > executions.
> > > > > > >
> > > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <
> tanyuxinwork@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Xintong,
> > > > > > > >
> > > > > > > > >  I think we can also publish the prototype codes so the
> > > > > > > > community can better understand and help with it.
> > > > > > > >
> > > > > > > > Ok, I agree on the point. I will prepare and publish the code
> > > > > > > > recently.
> > > > > > > >
> > > > > > > > Rui,
> > > > > > > >
> > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > > >
> > > > > > > > Thanks for the reminder. I've updated the images.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yuxin
> > > > > > > >
> > > > > > > >
> > > > > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > > > > > >
> > > > > > > > > Thanks Yuxin for driving this proposal!
> > > > > > > > >
> > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Rui
> > > > > > > > >
> > > > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > > > > tonysong820@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 for this proposal.
> > > > > > > > > >
> > > > > > > > > > We have been prototyping this feature internally at
> Alibaba
> > > > for a
> > > > > > > > couple
> > > > > > > > > of
> > > > > > > > > > months. Yuxin, I think we can also publish the prototype
> > > codes
> > > > > so the
> > > > > > > > > > community can better understand and help with it.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > Xintong
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> > > > > tanyuxinwork@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I would like to start a discussion on FLIP-459 Support
> > > Flink
> > > > > hybrid
> > > > > > > > > > shuffle
> > > > > > > > > > > integration with
> > > > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports
> > > transitions
> > > > > > > between
> > > > > > > > > > > memory, disk, and
> > > > > > > > > > > remote storage to improve performance and job
> stability.
> > > > > > > > Concurrently,
> > > > > > > > > > > Apache Celeborn
> > > > > > > > > > > provides a stable, performant, scalable remote shuffle
> > > > service.
> > > > > > > This
> > > > > > > > > > > integration proposal is to
> > > > > > > > > > > harness the benefits from both hybrid shuffle and
> > Celeborn
> > > > > > > > > > simultaneously.
> > > > > > > > > > >
> > > > > > > > > > > Note that this proposal has two parts.
> > > > > > > > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > > > > > > > >
> > > > > > > > > > > Looking forward to everyone's feedback and suggestions.
> > > Thank
> > > > > you!
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > > > > > > > [2]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Yuxin
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Yuxin Tan <ta...@gmail.com>.

Thanks Junrui for your question.

> I wonder if the current interface design support the
future adaptation for batch job recovery

I noticed that FLIP-383 supports batch job recovery by introducing
some new APIs. These APIs can also be added to the Tier-related
interfaces to facilitate the feature. Since these modifications are not
directly related to the current integration tasks and the integration
does not conflict with the batch job recovery, I propose that this FLIP
doesn't involve these particular changes. Moreover, considering that
the Tier interfaces are not public currently, it is also feasible to add
the interfaces directly if necessary.
WDYT?

Best,
Yuxin


Junrui Lee <jr...@gmail.com> 于2024年6月6日周四 11:02写道：

> Thanks Yuxin for driving this proposal!
>
> I have a question about the public interface compatibility in the context
> of FLIP-459. As we've supported batch job recovery from jobMaster failures
> in FLIP-383 which will be released in Flink 1.20. I wonder if the current
> interface design support the future adaptation for batch job recovery?
>
> Looking forward to your feedback.
>
> Best,
> Junrui.
>
> weijie guo <gu...@gmail.com> 于2024年6月5日周三 10:13写道：
>
> > Thanks Yuxin for the proposal!
> >
> > When we first proposed Hybrid Shuffle, I wanted to support pluggable
> > storage tier in the future. However, limited by the architecture of the
> > legacy Hybrid Shuffle at that time, this idea has not been realized. The
> > new architecture abstracts the tier nicely, and now it's time to
> introduce
> > support for external storage.
> >
> > Big +1 for this one!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：
> >
> > > Thanks Yuxin for the proposal. +1,  as a member of the Apache Celeborn
> > > community, I am very excited about the integration of Flink's Hybrid
> > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to
> > me. I
> > > am looking forward to this integration.
> > >
> > > Thanks,
> > > Jiashu Xiong
> > >
> > > Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
> > >
> > > > +1 for this proposal.
> > > >
> > > > After internally reviewing the prototype of CIP-6, this would improve
> > > > performance and stability for Flink users using Celeborn.
> > > >
> > > > Expect to see this feature come out to the community.
> > > >
> > > > As I come from the Celeborn community, I hope more users can try to
> > > > use Celeborn when there are Flink batch jobs.
> > > >
> > > > Thanks,
> > > > Ethan Feng
> > > >
> > > > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
> > > > >
> > > > > Hi, Venkatakrishnan,
> > > > >
> > > > > Thanks for joining the discussion. We appreciate your interest
> > > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > > have been approved, we will create some JIRA tickets in Flink
> > > > > and Celeborn projects. Please feel free to take a look at the
> > > > > tickets and select any that resonate with your interests.
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > > >
> > > > > Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五
> 23:11写道：
> > > > >
> > > > > > Thanks for this FLIP. We are also interested in
> > learning/contributing
> > > > to
> > > > > > the hybrid shuffle integration with celeborn for batch
> executions.
> > > > > >
> > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <ta...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi, Xintong,
> > > > > > >
> > > > > > > >  I think we can also publish the prototype codes so the
> > > > > > > community can better understand and help with it.
> > > > > > >
> > > > > > > Ok, I agree on the point. I will prepare and publish the code
> > > > > > > recently.
> > > > > > >
> > > > > > > Rui,
> > > > > > >
> > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > >
> > > > > > > Thanks for the reminder. I've updated the images.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Yuxin
> > > > > > >
> > > > > > >
> > > > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > > > > >
> > > > > > > > Thanks Yuxin for driving this proposal!
> > > > > > > >
> > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Rui
> > > > > > > >
> > > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > > > tonysong820@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 for this proposal.
> > > > > > > > >
> > > > > > > > > We have been prototyping this feature internally at Alibaba
> > > for a
> > > > > > > couple
> > > > > > > > of
> > > > > > > > > months. Yuxin, I think we can also publish the prototype
> > codes
> > > > so the
> > > > > > > > > community can better understand and help with it.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Xintong
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> > > > tanyuxinwork@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I would like to start a discussion on FLIP-459 Support
> > Flink
> > > > hybrid
> > > > > > > > > shuffle
> > > > > > > > > > integration with
> > > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports
> > transitions
> > > > > > between
> > > > > > > > > > memory, disk, and
> > > > > > > > > > remote storage to improve performance and job stability.
> > > > > > > Concurrently,
> > > > > > > > > > Apache Celeborn
> > > > > > > > > > provides a stable, performant, scalable remote shuffle
> > > service.
> > > > > > This
> > > > > > > > > > integration proposal is to
> > > > > > > > > > harness the benefits from both hybrid shuffle and
> Celeborn
> > > > > > > > > simultaneously.
> > > > > > > > > >
> > > > > > > > > > Note that this proposal has two parts.
> > > > > > > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > > > > > > >
> > > > > > > > > > Looking forward to everyone's feedback and suggestions.
> > Thank
> > > > you!
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > > > > > > [2]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Yuxin
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Junrui Lee <jr...@gmail.com>.

Thanks Yuxin for driving this proposal!

I have a question about the public interface compatibility in the context
of FLIP-459. As we've supported batch job recovery from jobMaster failures
in FLIP-383 which will be released in Flink 1.20. I wonder if the current
interface design support the future adaptation for batch job recovery?

Looking forward to your feedback.

Best,
Junrui.

weijie guo <gu...@gmail.com> 于2024年6月5日周三 10:13写道：

> Thanks Yuxin for the proposal!
>
> When we first proposed Hybrid Shuffle, I wanted to support pluggable
> storage tier in the future. However, limited by the architecture of the
> legacy Hybrid Shuffle at that time, this idea has not been realized. The
> new architecture abstracts the tier nicely, and now it's time to introduce
> support for external storage.
>
> Big +1 for this one!
>
> Best regards,
>
> Weijie
>
>
> rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：
>
> > Thanks Yuxin for the proposal. +1,  as a member of the Apache Celeborn
> > community, I am very excited about the integration of Flink's Hybrid
> > Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to
> me. I
> > am looking forward to this integration.
> >
> > Thanks,
> > Jiashu Xiong
> >
> > Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
> >
> > > +1 for this proposal.
> > >
> > > After internally reviewing the prototype of CIP-6, this would improve
> > > performance and stability for Flink users using Celeborn.
> > >
> > > Expect to see this feature come out to the community.
> > >
> > > As I come from the Celeborn community, I hope more users can try to
> > > use Celeborn when there are Flink batch jobs.
> > >
> > > Thanks,
> > > Ethan Feng
> > >
> > > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
> > > >
> > > > Hi, Venkatakrishnan,
> > > >
> > > > Thanks for joining the discussion. We appreciate your interest
> > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > have been approved, we will create some JIRA tickets in Flink
> > > > and Celeborn projects. Please feel free to take a look at the
> > > > tickets and select any that resonate with your interests.
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五 23:11写道：
> > > >
> > > > > Thanks for this FLIP. We are also interested in
> learning/contributing
> > > to
> > > > > the hybrid shuffle integration with celeborn for batch executions.
> > > > >
> > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <ta...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi, Xintong,
> > > > > >
> > > > > > >  I think we can also publish the prototype codes so the
> > > > > > community can better understand and help with it.
> > > > > >
> > > > > > Ok, I agree on the point. I will prepare and publish the code
> > > > > > recently.
> > > > > >
> > > > > > Rui,
> > > > > >
> > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > >
> > > > > > Thanks for the reminder. I've updated the images.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Yuxin
> > > > > >
> > > > > >
> > > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > > > >
> > > > > > > Thanks Yuxin for driving this proposal!
> > > > > > >
> > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > >
> > > > > > > Best,
> > > > > > > Rui
> > > > > > >
> > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > > tonysong820@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 for this proposal.
> > > > > > > >
> > > > > > > > We have been prototyping this feature internally at Alibaba
> > for a
> > > > > > couple
> > > > > > > of
> > > > > > > > months. Yuxin, I think we can also publish the prototype
> codes
> > > so the
> > > > > > > > community can better understand and help with it.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Xintong
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> > > tanyuxinwork@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I would like to start a discussion on FLIP-459 Support
> Flink
> > > hybrid
> > > > > > > > shuffle
> > > > > > > > > integration with
> > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports
> transitions
> > > > > between
> > > > > > > > > memory, disk, and
> > > > > > > > > remote storage to improve performance and job stability.
> > > > > > Concurrently,
> > > > > > > > > Apache Celeborn
> > > > > > > > > provides a stable, performant, scalable remote shuffle
> > service.
> > > > > This
> > > > > > > > > integration proposal is to
> > > > > > > > > harness the benefits from both hybrid shuffle and Celeborn
> > > > > > > > simultaneously.
> > > > > > > > >
> > > > > > > > > Note that this proposal has two parts.
> > > > > > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > > > > > >
> > > > > > > > > Looking forward to everyone's feedback and suggestions.
> Thank
> > > you!
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > > > > > [2]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Yuxin
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by weijie guo <gu...@gmail.com>.

Thanks Yuxin for the proposal!

When we first proposed Hybrid Shuffle, I wanted to support pluggable
storage tier in the future. However, limited by the architecture of the
legacy Hybrid Shuffle at that time, this idea has not been realized. The
new architecture abstracts the tier nicely, and now it's time to introduce
support for external storage.

Big +1 for this one!

Best regards,

Weijie


rexxiong <re...@apache.org> 于2024年6月5日周三 00:08写道：

> Thanks Yuxin for the proposal. +1,  as a member of the Apache Celeborn
> community, I am very excited about the integration of Flink's Hybrid
> Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to me. I
> am looking forward to this integration.
>
> Thanks,
> Jiashu Xiong
>
> Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：
>
> > +1 for this proposal.
> >
> > After internally reviewing the prototype of CIP-6, this would improve
> > performance and stability for Flink users using Celeborn.
> >
> > Expect to see this feature come out to the community.
> >
> > As I come from the Celeborn community, I hope more users can try to
> > use Celeborn when there are Flink batch jobs.
> >
> > Thanks,
> > Ethan Feng
> >
> > Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
> > >
> > > Hi, Venkatakrishnan,
> > >
> > > Thanks for joining the discussion. We appreciate your interest
> > > in contributing to the work. Once the FLIP and CIP proposals
> > > have been approved, we will create some JIRA tickets in Flink
> > > and Celeborn projects. Please feel free to take a look at the
> > > tickets and select any that resonate with your interests.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五 23:11写道：
> > >
> > > > Thanks for this FLIP. We are also interested in learning/contributing
> > to
> > > > the hybrid shuffle integration with celeborn for batch executions.
> > > >
> > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <ta...@gmail.com>
> > wrote:
> > > >
> > > > > Hi, Xintong,
> > > > >
> > > > > >  I think we can also publish the prototype codes so the
> > > > > community can better understand and help with it.
> > > > >
> > > > > Ok, I agree on the point. I will prepare and publish the code
> > > > > recently.
> > > > >
> > > > > Rui,
> > > > >
> > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > >
> > > > > Thanks for the reminder. I've updated the images.
> > > > >
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > > >
> > > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > > >
> > > > > > Thanks Yuxin for driving this proposal!
> > > > > >
> > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > >
> > > > > > Best,
> > > > > > Rui
> > > > > >
> > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > tonysong820@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 for this proposal.
> > > > > > >
> > > > > > > We have been prototyping this feature internally at Alibaba
> for a
> > > > > couple
> > > > > > of
> > > > > > > months. Yuxin, I think we can also publish the prototype codes
> > so the
> > > > > > > community can better understand and help with it.
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Xintong
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> > tanyuxinwork@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I would like to start a discussion on FLIP-459 Support Flink
> > hybrid
> > > > > > > shuffle
> > > > > > > > integration with
> > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions
> > > > between
> > > > > > > > memory, disk, and
> > > > > > > > remote storage to improve performance and job stability.
> > > > > Concurrently,
> > > > > > > > Apache Celeborn
> > > > > > > > provides a stable, performant, scalable remote shuffle
> service.
> > > > This
> > > > > > > > integration proposal is to
> > > > > > > > harness the benefits from both hybrid shuffle and Celeborn
> > > > > > > simultaneously.
> > > > > > > >
> > > > > > > > Note that this proposal has two parts.
> > > > > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > > > > >
> > > > > > > > Looking forward to everyone's feedback and suggestions. Thank
> > you!
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > > > > [2]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yuxin
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by rexxiong <re...@apache.org>.

Thanks Yuxin for the proposal. +1,  as a member of the Apache Celeborn
community, I am very excited about the integration of Flink's Hybrid
Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to me. I
am looking forward to this integration.

Thanks,
Jiashu Xiong

Ethan Feng <et...@apache.org> 于2024年6月4日周二 16:47写道：

> +1 for this proposal.
>
> After internally reviewing the prototype of CIP-6, this would improve
> performance and stability for Flink users using Celeborn.
>
> Expect to see this feature come out to the community.
>
> As I come from the Celeborn community, I hope more users can try to
> use Celeborn when there are Flink batch jobs.
>
> Thanks,
> Ethan Feng
>
> Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
> >
> > Hi, Venkatakrishnan,
> >
> > Thanks for joining the discussion. We appreciate your interest
> > in contributing to the work. Once the FLIP and CIP proposals
> > have been approved, we will create some JIRA tickets in Flink
> > and Celeborn projects. Please feel free to take a look at the
> > tickets and select any that resonate with your interests.
> >
> > Best,
> > Yuxin
> >
> >
> > Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五 23:11写道：
> >
> > > Thanks for this FLIP. We are also interested in learning/contributing
> to
> > > the hybrid shuffle integration with celeborn for batch executions.
> > >
> > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <ta...@gmail.com>
> wrote:
> > >
> > > > Hi, Xintong,
> > > >
> > > > >  I think we can also publish the prototype codes so the
> > > > community can better understand and help with it.
> > > >
> > > > Ok, I agree on the point. I will prepare and publish the code
> > > > recently.
> > > >
> > > > Rui,
> > > >
> > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > >
> > > > Thanks for the reminder. I've updated the images.
> > > >
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > >
> > > > > Thanks Yuxin for driving this proposal!
> > > > >
> > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > >
> > > > > Best,
> > > > > Rui
> > > > >
> > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> tonysong820@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1 for this proposal.
> > > > > >
> > > > > > We have been prototyping this feature internally at Alibaba for a
> > > > couple
> > > > > of
> > > > > > months. Yuxin, I think we can also publish the prototype codes
> so the
> > > > > > community can better understand and help with it.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> tanyuxinwork@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I would like to start a discussion on FLIP-459 Support Flink
> hybrid
> > > > > > shuffle
> > > > > > > integration with
> > > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions
> > > between
> > > > > > > memory, disk, and
> > > > > > > remote storage to improve performance and job stability.
> > > > Concurrently,
> > > > > > > Apache Celeborn
> > > > > > > provides a stable, performant, scalable remote shuffle service.
> > > This
> > > > > > > integration proposal is to
> > > > > > > harness the benefits from both hybrid shuffle and Celeborn
> > > > > > simultaneously.
> > > > > > >
> > > > > > > Note that this proposal has two parts.
> > > > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > > > >
> > > > > > > Looking forward to everyone's feedback and suggestions. Thank
> you!
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > > > [2]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > >
> > > > > > > Best,
> > > > > > > Yuxin
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Ethan Feng <et...@apache.org>.

+1 for this proposal.

After internally reviewing the prototype of CIP-6, this would improve
performance and stability for Flink users using Celeborn.

Expect to see this feature come out to the community.

As I come from the Celeborn community, I hope more users can try to
use Celeborn when there are Flink batch jobs.

Thanks,
Ethan Feng

Yuxin Tan <ta...@gmail.com> 于2024年6月4日周二 16:34写道：
>
> Hi, Venkatakrishnan,
>
> Thanks for joining the discussion. We appreciate your interest
> in contributing to the work. Once the FLIP and CIP proposals
> have been approved, we will create some JIRA tickets in Flink
> and Celeborn projects. Please feel free to take a look at the
> tickets and select any that resonate with your interests.
>
> Best,
> Yuxin
>
>
> Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五 23:11写道：
>
> > Thanks for this FLIP. We are also interested in learning/contributing to
> > the hybrid shuffle integration with celeborn for batch executions.
> >
> > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <ta...@gmail.com> wrote:
> >
> > > Hi, Xintong,
> > >
> > > >  I think we can also publish the prototype codes so the
> > > community can better understand and help with it.
> > >
> > > Ok, I agree on the point. I will prepare and publish the code
> > > recently.
> > >
> > > Rui,
> > >
> > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > >
> > > Thanks for the reminder. I've updated the images.
> > >
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> > >
> > > > Thanks Yuxin for driving this proposal!
> > > >
> > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <to...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 for this proposal.
> > > > >
> > > > > We have been prototyping this feature internally at Alibaba for a
> > > couple
> > > > of
> > > > > months. Yuxin, I think we can also publish the prototype codes so the
> > > > > community can better understand and help with it.
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <ta...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I would like to start a discussion on FLIP-459 Support Flink hybrid
> > > > > shuffle
> > > > > > integration with
> > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions
> > between
> > > > > > memory, disk, and
> > > > > > remote storage to improve performance and job stability.
> > > Concurrently,
> > > > > > Apache Celeborn
> > > > > > provides a stable, performant, scalable remote shuffle service.
> > This
> > > > > > integration proposal is to
> > > > > > harness the benefits from both hybrid shuffle and Celeborn
> > > > > simultaneously.
> > > > > >
> > > > > > Note that this proposal has two parts.
> > > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > > >
> > > > > > Looking forward to everyone's feedback and suggestions. Thank you!
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > >
> > > > > > Best,
> > > > > > Yuxin
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Yuxin Tan <ta...@gmail.com>.

Hi, Venkatakrishnan,

Thanks for joining the discussion. We appreciate your interest
in contributing to the work. Once the FLIP and CIP proposals
have been approved, we will create some JIRA tickets in Flink
and Celeborn projects. Please feel free to take a look at the
tickets and select any that resonate with your interests.

Best,
Yuxin


Venkatakrishnan Sowrirajan <vs...@asu.edu> 于2024年5月31日周五 23:11写道：

> Thanks for this FLIP. We are also interested in learning/contributing to
> the hybrid shuffle integration with celeborn for batch executions.
>
> On Tue, May 28, 2024, 7:07 PM Yuxin Tan <ta...@gmail.com> wrote:
>
> > Hi, Xintong,
> >
> > >  I think we can also publish the prototype codes so the
> > community can better understand and help with it.
> >
> > Ok, I agree on the point. I will prepare and publish the code
> > recently.
> >
> > Rui,
> >
> > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> >
> > Thanks for the reminder. I've updated the images.
> >
> >
> > Best,
> > Yuxin
> >
> >
> > Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
> >
> > > Thanks Yuxin for driving this proposal!
> > >
> > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > >
> > > [1]
> > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > >
> > > Best,
> > > Rui
> > >
> > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <to...@gmail.com>
> > > wrote:
> > >
> > > > +1 for this proposal.
> > > >
> > > > We have been prototyping this feature internally at Alibaba for a
> > couple
> > > of
> > > > months. Yuxin, I think we can also publish the prototype codes so the
> > > > community can better understand and help with it.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <ta...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start a discussion on FLIP-459 Support Flink hybrid
> > > > shuffle
> > > > > integration with
> > > > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions
> between
> > > > > memory, disk, and
> > > > > remote storage to improve performance and job stability.
> > Concurrently,
> > > > > Apache Celeborn
> > > > > provides a stable, performant, scalable remote shuffle service.
> This
> > > > > integration proposal is to
> > > > > harness the benefits from both hybrid shuffle and Celeborn
> > > > simultaneously.
> > > > >
> > > > > Note that this proposal has two parts.
> > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > >
> > > > > Looking forward to everyone's feedback and suggestions. Thank you!
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Venkatakrishnan Sowrirajan <vs...@asu.edu>.

Thanks for this FLIP. We are also interested in learning/contributing to
the hybrid shuffle integration with celeborn for batch executions.

On Tue, May 28, 2024, 7:07 PM Yuxin Tan <ta...@gmail.com> wrote:

> Hi, Xintong,
>
> >  I think we can also publish the prototype codes so the
> community can better understand and help with it.
>
> Ok, I agree on the point. I will prepare and publish the code
> recently.
>
> Rui,
>
> > Kindly reminder: the image of CIP-6[1] cannot be loaded.
>
> Thanks for the reminder. I've updated the images.
>
>
> Best,
> Yuxin
>
>
> Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：
>
> > Thanks Yuxin for driving this proposal!
> >
> > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> >
> > [1]
> >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> >
> > Best,
> > Rui
> >
> > On Wed, May 29, 2024 at 9:03 AM Xintong Song <to...@gmail.com>
> > wrote:
> >
> > > +1 for this proposal.
> > >
> > > We have been prototyping this feature internally at Alibaba for a
> couple
> > of
> > > months. Yuxin, I think we can also publish the prototype codes so the
> > > community can better understand and help with it.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <ta...@gmail.com>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would like to start a discussion on FLIP-459 Support Flink hybrid
> > > shuffle
> > > > integration with
> > > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> > > > memory, disk, and
> > > > remote storage to improve performance and job stability.
> Concurrently,
> > > > Apache Celeborn
> > > > provides a stable, performant, scalable remote shuffle service. This
> > > > integration proposal is to
> > > > harness the benefits from both hybrid shuffle and Celeborn
> > > simultaneously.
> > > >
> > > > Note that this proposal has two parts.
> > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > >
> > > > Looking forward to everyone's feedback and suggestions. Thank you!
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > [2]
> > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Yuxin Tan <ta...@gmail.com>.

Hi, Xintong,

>  I think we can also publish the prototype codes so the
community can better understand and help with it.

Ok, I agree on the point. I will prepare and publish the code
recently.

Rui,

> Kindly reminder: the image of CIP-6[1] cannot be loaded.

Thanks for the reminder. I've updated the images.


Best,
Yuxin


Rui Fan <19...@gmail.com> 于2024年5月29日周三 09:33写道：

> Thanks Yuxin for driving this proposal!
>
> Kindly reminder: the image of CIP-6[1] cannot be loaded.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
>
> Best,
> Rui
>
> On Wed, May 29, 2024 at 9:03 AM Xintong Song <to...@gmail.com>
> wrote:
>
> > +1 for this proposal.
> >
> > We have been prototyping this feature internally at Alibaba for a couple
> of
> > months. Yuxin, I think we can also publish the prototype codes so the
> > community can better understand and help with it.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <ta...@gmail.com>
> wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a discussion on FLIP-459 Support Flink hybrid
> > shuffle
> > > integration with
> > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> > > memory, disk, and
> > > remote storage to improve performance and job stability. Concurrently,
> > > Apache Celeborn
> > > provides a stable, performant, scalable remote shuffle service. This
> > > integration proposal is to
> > > harness the benefits from both hybrid shuffle and Celeborn
> > simultaneously.
> > >
> > > Note that this proposal has two parts.
> > > 1. The Flink-side modifications are in FLIP-459[1].
> > > 2. The Celeborn-side changes are in CIP-6[2].
> > >
> > > Looking forward to everyone's feedback and suggestions. Thank you!
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > >
> > > Best,
> > > Yuxin
> > >
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Rui Fan <19...@gmail.com>.

Thanks Yuxin for driving this proposal!

Kindly reminder: the image of CIP-6[1] cannot be loaded.

[1]
https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn

Best,
Rui

On Wed, May 29, 2024 at 9:03 AM Xintong Song <to...@gmail.com> wrote:

> +1 for this proposal.
>
> We have been prototyping this feature internally at Alibaba for a couple of
> months. Yuxin, I think we can also publish the prototype codes so the
> community can better understand and help with it.
>
> Best,
>
> Xintong
>
>
>
> On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <ta...@gmail.com> wrote:
>
> > Hi all,
> >
> > I would like to start a discussion on FLIP-459 Support Flink hybrid
> shuffle
> > integration with
> > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> > memory, disk, and
> > remote storage to improve performance and job stability. Concurrently,
> > Apache Celeborn
> > provides a stable, performant, scalable remote shuffle service. This
> > integration proposal is to
> > harness the benefits from both hybrid shuffle and Celeborn
> simultaneously.
> >
> > Note that this proposal has two parts.
> > 1. The Flink-side modifications are in FLIP-459[1].
> > 2. The Celeborn-side changes are in CIP-6[2].
> >
> > Looking forward to everyone's feedback and suggestions. Thank you!
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> >
> > Best,
> > Yuxin
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Posted by Xintong Song <to...@gmail.com>.

+1 for this proposal.

We have been prototyping this feature internally at Alibaba for a couple of
months. Yuxin, I think we can also publish the prototype codes so the
community can better understand and help with it.

Best,

Xintong



On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <ta...@gmail.com> wrote:

> Hi all,
>
> I would like to start a discussion on FLIP-459 Support Flink hybrid shuffle
> integration with
> Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> memory, disk, and
> remote storage to improve performance and job stability. Concurrently,
> Apache Celeborn
> provides a stable, performant, scalable remote shuffle service. This
> integration proposal is to
> harness the benefits from both hybrid shuffle and Celeborn simultaneously.
>
> Note that this proposal has two parts.
> 1. The Flink-side modifications are in FLIP-459[1].
> 2. The Celeborn-side changes are in CIP-6[2].
>
> Looking forward to everyone's feedback and suggestions. Thank you!
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> [2]
>
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
>
> Best,
> Yuxin
>