You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@celeborn.apache.org by Aravind Patnam <ak...@gmail.com> on 2024/06/05 23:33:02 UTC

[DISCUSS] Celeborn CLI Proposal

Hi all,

I have written up a proposal about introducing a CLI for Celeborn. You can
find the proposal
<https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing>
here.
Please let me know if you have any comments or questions.

TLDR by introducing a CLI, it would complement the existing dashboard and
would benefit us internally. We rely on CLI tools internally a lot for
automation and other operations.

FYI, I was not able to access the cwiki page to put this proposal there,
there seems to be some permissions issue. Hope it is okay to just share as
a google doc here for now.

-- 
Aravind K. Patnam

 Apache Celeborn CLI Proposal
<https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web>

Re: [DISCUSS] Celeborn CLI Proposal

Posted by Aravind Patnam <ak...@gmail.com>.
Sounds good!! I will start a vote thread.

Thanks,
Aravind

Aravind K. Patnam


On Tue, Jun 11, 2024 at 1:01 AM rexxiong <re...@apache.org> wrote:

> Hi Aravind,
>
> Thank you for the proposal. It looks good to me, agree with Yu that there
> is no blocker. Please go ahead and initiate a formal vote for this CIP, See
> the process section in the CIP guidelines
> <
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> >
> .
>
>
> Thanks,
> Jiashu Xiong
>
> Yu Li <ca...@gmail.com> 于2024年6月11日周二 15:26写道:
>
> > IIUIC, we need a formal vote on this CIP and make sure there are (at
> > least one) committer(s) to help review the coming up PRs.
> >
> > From the existing discussions, I believe there's no blocker but just a
> > normal process for this CIP to move on (smile).
> >
> > And thanks for the proposal, Aravind!
> >
> > Best Regards,
> > Yu
> >
> > On Tue, 11 Jun 2024 at 13:10, Aravind Patnam <ak...@gmail.com>
> wrote:
> > >
> > > Hi all,
> > >
> > > Thanks for the reviews everyone!!
> > >
> > > Is there any other process required (such as a vote thread), or can I
> > start
> > > contributing the CLI now?
> > >
> > > Thanks,
> > > Aravind
> > >
> > > Aravind K. Patnam
> > >
> > >
> > > On Mon, Jun 10, 2024 at 11:29 AM Mridul Muralidharan <mridul@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >   Looks good to me as well, I had reviewed this proposal internally
> > already
> > > > :-)
> > > >
> > > > Regards,
> > > > Mridul
> > > >
> > > >
> > > > On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou <zh...@apache.org>
> wrote:
> > > >
> > > > > Hi Aravind,
> > > > >
> > > > > Thanks for the proposal! The proposal LGTM, I think it's very
> > valuable.
> > > > >
> > > > > Regards,
> > > > > Keyong Zhou
> > > > >
> > > > > Aravind Patnam <ak...@gmail.com> 于2024年6月7日周五 12:47写道:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Thanks Nicholas for the comments!
> > > > > >
> > > > > > I now got access to put the proposal in Confluence in the form of
> > CIP,
> > > > > here
> > > > > > <
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > > > > > >
> > > > > > it is.
> > > > > >
> > > > > > Regarding your questions:
> > > > > >
> > > > > > > 1. From a user's perspective, the CLI is more used for some
> > > > maintenance
> > > > > > operations such as online and offline of server, rescaling of
> > cluster
> > > > > etc,
> > > > > > not only based on the REST API. What CLI interfaces are there
> that
> > the
> > > > > REST
> > > > > > API doesn’t have for maintenance?
> > > > > > This is highly dependent on what the user is leveraging to manage
> > their
> > > > > > cluster. For example, in k8s, you would be using k8s APIs to
> > achieve
> > > > > this.
> > > > > > We can probably add a generic interface API for it that provides
> > basic
> > > > > > operations that users can implement themselves for their cluster
> > > > > management
> > > > > > logic based on what cluster managers they are using. Although, I
> > think
> > > > > this
> > > > > > will likely be a later evolution of the CLI, once basic REST API
> > > > > operations
> > > > > > are implemented in the CLI. WDYT?
> > > > > >
> > > > > > > 2. There are same sub-commands between MASTER and WORKER. Why
> not
> > > > these
> > > > > > sub-commands belong to BOTH?
> > > > > > Agreed - this was a formatting mistake. I fixed it now, thanks
> for
> > > > > pointing
> > > > > > that out.
> > > > > >
> > > > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the
> > CLI
> > > > > works
> > > > > > well no matter the server is alive.
> > > > > > Yes, I agree. I think for this we would have to talk to the
> cluster
> > > > > > manager, similar to my response to #1. We would have to query the
> > > > > specific
> > > > > > cluster manager to get details if the Celeborn servers are dead,
> > since
> > > > > the
> > > > > > Celeborn REST API would not work then. We can add a generic API
> > that
> > > > > users
> > > > > > can implement based on their own environment.
> > > > > >
> > > > > > Thanks,
> > > > > > Aravind
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <
> > > > nicholasjiang@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Aravind,
> > > > > > >
> > > > > > > Thanks for driving this CIP about Celeborn CLI. I have some
> > comments
> > > > on
> > > > > > > this CIP:
> > > > > > >
> > > > > > > 1. From a user's perspective, the CLI is more used for some
> > > > maintenance
> > > > > > > operations such as online and offline of server, rescaling of
> > cluster
> > > > > > etc,
> > > > > > > not only based on the REST API. What CLI interfaces are there
> > that
> > > > the
> > > > > > REST
> > > > > > > API doesn’t have for maintenance?
> > > > > > >
> > > > > > > 2. There are same sub-commands between MASTER and WORKER. Why
> not
> > > > these
> > > > > > > sub-commands belong to BOTH?
> > > > > > >
> > > > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the
> > CLI
> > > > > works
> > > > > > > well no matter the server is alive.
> > > > > > >
> > > > > > > BTW, could this design doc of proposal follow the template of
> > CIP[1]?
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> > > > > > >
> > > > > > > Regards,
> > > > > > > Nicholas Jiang
> > > > > > >
> > > > > > > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I have written up a proposal about introducing a CLI for
> > Celeborn.
> > > > > You
> > > > > > > can
> > > > > > > > find the proposal
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > > > > > > >
> > > > > > > > here.
> > > > > > > > Please let me know if you have any comments or questions.
> > > > > > > >
> > > > > > > > TLDR by introducing a CLI, it would complement the existing
> > > > dashboard
> > > > > > and
> > > > > > > > would benefit us internally. We rely on CLI tools internally
> a
> > lot
> > > > > for
> > > > > > > > automation and other operations.
> > > > > > > >
> > > > > > > > FYI, I was not able to access the cwiki page to put this
> > proposal
> > > > > > there,
> > > > > > > > there seems to be some permissions issue. Hope it is okay to
> > just
> > > > > share
> > > > > > > as
> > > > > > > > a google doc here for now.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Aravind K. Patnam
> > > > > > > >
> > > > > > > >  Apache Celeborn CLI Proposal
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Aravind K. Patnam
> > > > > >
> > > > >
> > > >
> >
>

Re: [DISCUSS] Celeborn CLI Proposal

Posted by rexxiong <re...@apache.org>.
Hi Aravind,

Thank you for the proposal. It looks good to me, agree with Yu that there
is no blocker. Please go ahead and initiate a formal vote for this CIP, See
the process section in the CIP guidelines
<https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals>
.


Thanks,
Jiashu Xiong

Yu Li <ca...@gmail.com> 于2024年6月11日周二 15:26写道:

> IIUIC, we need a formal vote on this CIP and make sure there are (at
> least one) committer(s) to help review the coming up PRs.
>
> From the existing discussions, I believe there's no blocker but just a
> normal process for this CIP to move on (smile).
>
> And thanks for the proposal, Aravind!
>
> Best Regards,
> Yu
>
> On Tue, 11 Jun 2024 at 13:10, Aravind Patnam <ak...@gmail.com> wrote:
> >
> > Hi all,
> >
> > Thanks for the reviews everyone!!
> >
> > Is there any other process required (such as a vote thread), or can I
> start
> > contributing the CLI now?
> >
> > Thanks,
> > Aravind
> >
> > Aravind K. Patnam
> >
> >
> > On Mon, Jun 10, 2024 at 11:29 AM Mridul Muralidharan <mr...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > >   Looks good to me as well, I had reviewed this proposal internally
> already
> > > :-)
> > >
> > > Regards,
> > > Mridul
> > >
> > >
> > > On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou <zh...@apache.org> wrote:
> > >
> > > > Hi Aravind,
> > > >
> > > > Thanks for the proposal! The proposal LGTM, I think it's very
> valuable.
> > > >
> > > > Regards,
> > > > Keyong Zhou
> > > >
> > > > Aravind Patnam <ak...@gmail.com> 于2024年6月7日周五 12:47写道:
> > > >
> > > > > Hi,
> > > > >
> > > > > Thanks Nicholas for the comments!
> > > > >
> > > > > I now got access to put the proposal in Confluence in the form of
> CIP,
> > > > here
> > > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > > > > >
> > > > > it is.
> > > > >
> > > > > Regarding your questions:
> > > > >
> > > > > > 1. From a user's perspective, the CLI is more used for some
> > > maintenance
> > > > > operations such as online and offline of server, rescaling of
> cluster
> > > > etc,
> > > > > not only based on the REST API. What CLI interfaces are there that
> the
> > > > REST
> > > > > API doesn’t have for maintenance?
> > > > > This is highly dependent on what the user is leveraging to manage
> their
> > > > > cluster. For example, in k8s, you would be using k8s APIs to
> achieve
> > > > this.
> > > > > We can probably add a generic interface API for it that provides
> basic
> > > > > operations that users can implement themselves for their cluster
> > > > management
> > > > > logic based on what cluster managers they are using. Although, I
> think
> > > > this
> > > > > will likely be a later evolution of the CLI, once basic REST API
> > > > operations
> > > > > are implemented in the CLI. WDYT?
> > > > >
> > > > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> > > these
> > > > > sub-commands belong to BOTH?
> > > > > Agreed - this was a formatting mistake. I fixed it now, thanks for
> > > > pointing
> > > > > that out.
> > > > >
> > > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the
> CLI
> > > > works
> > > > > well no matter the server is alive.
> > > > > Yes, I agree. I think for this we would have to talk to the cluster
> > > > > manager, similar to my response to #1. We would have to query the
> > > > specific
> > > > > cluster manager to get details if the Celeborn servers are dead,
> since
> > > > the
> > > > > Celeborn REST API would not work then. We can add a generic API
> that
> > > > users
> > > > > can implement based on their own environment.
> > > > >
> > > > > Thanks,
> > > > > Aravind
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <
> > > nicholasjiang@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Aravind,
> > > > > >
> > > > > > Thanks for driving this CIP about Celeborn CLI. I have some
> comments
> > > on
> > > > > > this CIP:
> > > > > >
> > > > > > 1. From a user's perspective, the CLI is more used for some
> > > maintenance
> > > > > > operations such as online and offline of server, rescaling of
> cluster
> > > > > etc,
> > > > > > not only based on the REST API. What CLI interfaces are there
> that
> > > the
> > > > > REST
> > > > > > API doesn’t have for maintenance?
> > > > > >
> > > > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> > > these
> > > > > > sub-commands belong to BOTH?
> > > > > >
> > > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the
> CLI
> > > > works
> > > > > > well no matter the server is alive.
> > > > > >
> > > > > > BTW, could this design doc of proposal follow the template of
> CIP[1]?
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> > > > > >
> > > > > > Regards,
> > > > > > Nicholas Jiang
> > > > > >
> > > > > > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I have written up a proposal about introducing a CLI for
> Celeborn.
> > > > You
> > > > > > can
> > > > > > > find the proposal
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > > > > > >
> > > > > > > here.
> > > > > > > Please let me know if you have any comments or questions.
> > > > > > >
> > > > > > > TLDR by introducing a CLI, it would complement the existing
> > > dashboard
> > > > > and
> > > > > > > would benefit us internally. We rely on CLI tools internally a
> lot
> > > > for
> > > > > > > automation and other operations.
> > > > > > >
> > > > > > > FYI, I was not able to access the cwiki page to put this
> proposal
> > > > > there,
> > > > > > > there seems to be some permissions issue. Hope it is okay to
> just
> > > > share
> > > > > > as
> > > > > > > a google doc here for now.
> > > > > > >
> > > > > > > --
> > > > > > > Aravind K. Patnam
> > > > > > >
> > > > > > >  Apache Celeborn CLI Proposal
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Aravind K. Patnam
> > > > >
> > > >
> > >
>

Re: [DISCUSS] Celeborn CLI Proposal

Posted by Yu Li <ca...@gmail.com>.
IIUIC, we need a formal vote on this CIP and make sure there are (at
least one) committer(s) to help review the coming up PRs.

From the existing discussions, I believe there's no blocker but just a
normal process for this CIP to move on (smile).

And thanks for the proposal, Aravind!

Best Regards,
Yu

On Tue, 11 Jun 2024 at 13:10, Aravind Patnam <ak...@gmail.com> wrote:
>
> Hi all,
>
> Thanks for the reviews everyone!!
>
> Is there any other process required (such as a vote thread), or can I start
> contributing the CLI now?
>
> Thanks,
> Aravind
>
> Aravind K. Patnam
>
>
> On Mon, Jun 10, 2024 at 11:29 AM Mridul Muralidharan <mr...@gmail.com>
> wrote:
>
> > Hi,
> >
> >   Looks good to me as well, I had reviewed this proposal internally already
> > :-)
> >
> > Regards,
> > Mridul
> >
> >
> > On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou <zh...@apache.org> wrote:
> >
> > > Hi Aravind,
> > >
> > > Thanks for the proposal! The proposal LGTM, I think it's very valuable.
> > >
> > > Regards,
> > > Keyong Zhou
> > >
> > > Aravind Patnam <ak...@gmail.com> 于2024年6月7日周五 12:47写道:
> > >
> > > > Hi,
> > > >
> > > > Thanks Nicholas for the comments!
> > > >
> > > > I now got access to put the proposal in Confluence in the form of CIP,
> > > here
> > > > <
> > >
> > https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > > > >
> > > > it is.
> > > >
> > > > Regarding your questions:
> > > >
> > > > > 1. From a user's perspective, the CLI is more used for some
> > maintenance
> > > > operations such as online and offline of server, rescaling of cluster
> > > etc,
> > > > not only based on the REST API. What CLI interfaces are there that the
> > > REST
> > > > API doesn’t have for maintenance?
> > > > This is highly dependent on what the user is leveraging to manage their
> > > > cluster. For example, in k8s, you would be using k8s APIs to achieve
> > > this.
> > > > We can probably add a generic interface API for it that provides basic
> > > > operations that users can implement themselves for their cluster
> > > management
> > > > logic based on what cluster managers they are using. Although, I think
> > > this
> > > > will likely be a later evolution of the CLI, once basic REST API
> > > operations
> > > > are implemented in the CLI. WDYT?
> > > >
> > > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> > these
> > > > sub-commands belong to BOTH?
> > > > Agreed - this was a formatting mistake. I fixed it now, thanks for
> > > pointing
> > > > that out.
> > > >
> > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> > > works
> > > > well no matter the server is alive.
> > > > Yes, I agree. I think for this we would have to talk to the cluster
> > > > manager, similar to my response to #1. We would have to query the
> > > specific
> > > > cluster manager to get details if the Celeborn servers are dead, since
> > > the
> > > > Celeborn REST API would not work then. We can add a generic API that
> > > users
> > > > can implement based on their own environment.
> > > >
> > > > Thanks,
> > > > Aravind
> > > >
> > > >
> > > >
> > > > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <
> > nicholasjiang@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi Aravind,
> > > > >
> > > > > Thanks for driving this CIP about Celeborn CLI. I have some comments
> > on
> > > > > this CIP:
> > > > >
> > > > > 1. From a user's perspective, the CLI is more used for some
> > maintenance
> > > > > operations such as online and offline of server, rescaling of cluster
> > > > etc,
> > > > > not only based on the REST API. What CLI interfaces are there that
> > the
> > > > REST
> > > > > API doesn’t have for maintenance?
> > > > >
> > > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> > these
> > > > > sub-commands belong to BOTH?
> > > > >
> > > > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> > > works
> > > > > well no matter the server is alive.
> > > > >
> > > > > BTW, could this design doc of proposal follow the template of CIP[1]?
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> > > > >
> > > > > Regards,
> > > > > Nicholas Jiang
> > > > >
> > > > > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I have written up a proposal about introducing a CLI for Celeborn.
> > > You
> > > > > can
> > > > > > find the proposal
> > > > > > <
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > > > > >
> > > > > > here.
> > > > > > Please let me know if you have any comments or questions.
> > > > > >
> > > > > > TLDR by introducing a CLI, it would complement the existing
> > dashboard
> > > > and
> > > > > > would benefit us internally. We rely on CLI tools internally a lot
> > > for
> > > > > > automation and other operations.
> > > > > >
> > > > > > FYI, I was not able to access the cwiki page to put this proposal
> > > > there,
> > > > > > there seems to be some permissions issue. Hope it is okay to just
> > > share
> > > > > as
> > > > > > a google doc here for now.
> > > > > >
> > > > > > --
> > > > > > Aravind K. Patnam
> > > > > >
> > > > > >  Apache Celeborn CLI Proposal
> > > > > > <
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Aravind K. Patnam
> > > >
> > >
> >

Re: [DISCUSS] Celeborn CLI Proposal

Posted by Aravind Patnam <ak...@gmail.com>.
Hi all,

Thanks for the reviews everyone!!

Is there any other process required (such as a vote thread), or can I start
contributing the CLI now?

Thanks,
Aravind

Aravind K. Patnam


On Mon, Jun 10, 2024 at 11:29 AM Mridul Muralidharan <mr...@gmail.com>
wrote:

> Hi,
>
>   Looks good to me as well, I had reviewed this proposal internally already
> :-)
>
> Regards,
> Mridul
>
>
> On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou <zh...@apache.org> wrote:
>
> > Hi Aravind,
> >
> > Thanks for the proposal! The proposal LGTM, I think it's very valuable.
> >
> > Regards,
> > Keyong Zhou
> >
> > Aravind Patnam <ak...@gmail.com> 于2024年6月7日周五 12:47写道:
> >
> > > Hi,
> > >
> > > Thanks Nicholas for the comments!
> > >
> > > I now got access to put the proposal in Confluence in the form of CIP,
> > here
> > > <
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > > >
> > > it is.
> > >
> > > Regarding your questions:
> > >
> > > > 1. From a user's perspective, the CLI is more used for some
> maintenance
> > > operations such as online and offline of server, rescaling of cluster
> > etc,
> > > not only based on the REST API. What CLI interfaces are there that the
> > REST
> > > API doesn’t have for maintenance?
> > > This is highly dependent on what the user is leveraging to manage their
> > > cluster. For example, in k8s, you would be using k8s APIs to achieve
> > this.
> > > We can probably add a generic interface API for it that provides basic
> > > operations that users can implement themselves for their cluster
> > management
> > > logic based on what cluster managers they are using. Although, I think
> > this
> > > will likely be a later evolution of the CLI, once basic REST API
> > operations
> > > are implemented in the CLI. WDYT?
> > >
> > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> these
> > > sub-commands belong to BOTH?
> > > Agreed - this was a formatting mistake. I fixed it now, thanks for
> > pointing
> > > that out.
> > >
> > > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> > works
> > > well no matter the server is alive.
> > > Yes, I agree. I think for this we would have to talk to the cluster
> > > manager, similar to my response to #1. We would have to query the
> > specific
> > > cluster manager to get details if the Celeborn servers are dead, since
> > the
> > > Celeborn REST API would not work then. We can add a generic API that
> > users
> > > can implement based on their own environment.
> > >
> > > Thanks,
> > > Aravind
> > >
> > >
> > >
> > > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <
> nicholasjiang@apache.org
> > >
> > > wrote:
> > >
> > > > Hi Aravind,
> > > >
> > > > Thanks for driving this CIP about Celeborn CLI. I have some comments
> on
> > > > this CIP:
> > > >
> > > > 1. From a user's perspective, the CLI is more used for some
> maintenance
> > > > operations such as online and offline of server, rescaling of cluster
> > > etc,
> > > > not only based on the REST API. What CLI interfaces are there that
> the
> > > REST
> > > > API doesn’t have for maintenance?
> > > >
> > > > 2. There are same sub-commands between MASTER and WORKER. Why not
> these
> > > > sub-commands belong to BOTH?
> > > >
> > > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> > works
> > > > well no matter the server is alive.
> > > >
> > > > BTW, could this design doc of proposal follow the template of CIP[1]?
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> > > >
> > > > Regards,
> > > > Nicholas Jiang
> > > >
> > > > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > > > Hi all,
> > > > >
> > > > > I have written up a proposal about introducing a CLI for Celeborn.
> > You
> > > > can
> > > > > find the proposal
> > > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > > > >
> > > > > here.
> > > > > Please let me know if you have any comments or questions.
> > > > >
> > > > > TLDR by introducing a CLI, it would complement the existing
> dashboard
> > > and
> > > > > would benefit us internally. We rely on CLI tools internally a lot
> > for
> > > > > automation and other operations.
> > > > >
> > > > > FYI, I was not able to access the cwiki page to put this proposal
> > > there,
> > > > > there seems to be some permissions issue. Hope it is okay to just
> > share
> > > > as
> > > > > a google doc here for now.
> > > > >
> > > > > --
> > > > > Aravind K. Patnam
> > > > >
> > > > >  Apache Celeborn CLI Proposal
> > > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Aravind K. Patnam
> > >
> >
>

Re: [DISCUSS] Celeborn CLI Proposal

Posted by Mridul Muralidharan <mr...@gmail.com>.
Hi,

  Looks good to me as well, I had reviewed this proposal internally already
:-)

Regards,
Mridul


On Fri, Jun 7, 2024 at 11:32 PM Keyong Zhou <zh...@apache.org> wrote:

> Hi Aravind,
>
> Thanks for the proposal! The proposal LGTM, I think it's very valuable.
>
> Regards,
> Keyong Zhou
>
> Aravind Patnam <ak...@gmail.com> 于2024年6月7日周五 12:47写道:
>
> > Hi,
> >
> > Thanks Nicholas for the comments!
> >
> > I now got access to put the proposal in Confluence in the form of CIP,
> here
> > <
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> > >
> > it is.
> >
> > Regarding your questions:
> >
> > > 1. From a user's perspective, the CLI is more used for some maintenance
> > operations such as online and offline of server, rescaling of cluster
> etc,
> > not only based on the REST API. What CLI interfaces are there that the
> REST
> > API doesn’t have for maintenance?
> > This is highly dependent on what the user is leveraging to manage their
> > cluster. For example, in k8s, you would be using k8s APIs to achieve
> this.
> > We can probably add a generic interface API for it that provides basic
> > operations that users can implement themselves for their cluster
> management
> > logic based on what cluster managers they are using. Although, I think
> this
> > will likely be a later evolution of the CLI, once basic REST API
> operations
> > are implemented in the CLI. WDYT?
> >
> > > 2. There are same sub-commands between MASTER and WORKER. Why not these
> > sub-commands belong to BOTH?
> > Agreed - this was a formatting mistake. I fixed it now, thanks for
> pointing
> > that out.
> >
> > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> works
> > well no matter the server is alive.
> > Yes, I agree. I think for this we would have to talk to the cluster
> > manager, similar to my response to #1. We would have to query the
> specific
> > cluster manager to get details if the Celeborn servers are dead, since
> the
> > Celeborn REST API would not work then. We can add a generic API that
> users
> > can implement based on their own environment.
> >
> > Thanks,
> > Aravind
> >
> >
> >
> > On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <nicholasjiang@apache.org
> >
> > wrote:
> >
> > > Hi Aravind,
> > >
> > > Thanks for driving this CIP about Celeborn CLI. I have some comments on
> > > this CIP:
> > >
> > > 1. From a user's perspective, the CLI is more used for some maintenance
> > > operations such as online and offline of server, rescaling of cluster
> > etc,
> > > not only based on the REST API. What CLI interfaces are there that the
> > REST
> > > API doesn’t have for maintenance?
> > >
> > > 2. There are same sub-commands between MASTER and WORKER. Why not these
> > > sub-commands belong to BOTH?
> > >
> > > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI
> works
> > > well no matter the server is alive.
> > >
> > > BTW, could this design doc of proposal follow the template of CIP[1]?
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> > >
> > > Regards,
> > > Nicholas Jiang
> > >
> > > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > > Hi all,
> > > >
> > > > I have written up a proposal about introducing a CLI for Celeborn.
> You
> > > can
> > > > find the proposal
> > > > <
> > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > > >
> > > > here.
> > > > Please let me know if you have any comments or questions.
> > > >
> > > > TLDR by introducing a CLI, it would complement the existing dashboard
> > and
> > > > would benefit us internally. We rely on CLI tools internally a lot
> for
> > > > automation and other operations.
> > > >
> > > > FYI, I was not able to access the cwiki page to put this proposal
> > there,
> > > > there seems to be some permissions issue. Hope it is okay to just
> share
> > > as
> > > > a google doc here for now.
> > > >
> > > > --
> > > > Aravind K. Patnam
> > > >
> > > >  Apache Celeborn CLI Proposal
> > > > <
> > >
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> > > >
> > > >
> > >
> >
> >
> > --
> > Aravind K. Patnam
> >
>

Re: [DISCUSS] Celeborn CLI Proposal

Posted by Keyong Zhou <zh...@apache.org>.
Hi Aravind,

Thanks for the proposal! The proposal LGTM, I think it's very valuable.

Regards,
Keyong Zhou

Aravind Patnam <ak...@gmail.com> 于2024年6月7日周五 12:47写道:

> Hi,
>
> Thanks Nicholas for the comments!
>
> I now got access to put the proposal in Confluence in the form of CIP, here
> <https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI
> >
> it is.
>
> Regarding your questions:
>
> > 1. From a user's perspective, the CLI is more used for some maintenance
> operations such as online and offline of server, rescaling of cluster etc,
> not only based on the REST API. What CLI interfaces are there that the REST
> API doesn’t have for maintenance?
> This is highly dependent on what the user is leveraging to manage their
> cluster. For example, in k8s, you would be using k8s APIs to achieve this.
> We can probably add a generic interface API for it that provides basic
> operations that users can implement themselves for their cluster management
> logic based on what cluster managers they are using. Although, I think this
> will likely be a later evolution of the CLI, once basic REST API operations
> are implemented in the CLI. WDYT?
>
> > 2. There are same sub-commands between MASTER and WORKER. Why not these
> sub-commands belong to BOTH?
> Agreed - this was a formatting mistake. I fixed it now, thanks for pointing
> that out.
>
> > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI works
> well no matter the server is alive.
> Yes, I agree. I think for this we would have to talk to the cluster
> manager, similar to my response to #1. We would have to query the specific
> cluster manager to get details if the Celeborn servers are dead, since the
> Celeborn REST API would not work then. We can add a generic API that users
> can implement based on their own environment.
>
> Thanks,
> Aravind
>
>
>
> On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <ni...@apache.org>
> wrote:
>
> > Hi Aravind,
> >
> > Thanks for driving this CIP about Celeborn CLI. I have some comments on
> > this CIP:
> >
> > 1. From a user's perspective, the CLI is more used for some maintenance
> > operations such as online and offline of server, rescaling of cluster
> etc,
> > not only based on the REST API. What CLI interfaces are there that the
> REST
> > API doesn’t have for maintenance?
> >
> > 2. There are same sub-commands between MASTER and WORKER. Why not these
> > sub-commands belong to BOTH?
> >
> > 3. Does the implementation of CLI invoke the REST API? IMO, the CLI works
> > well no matter the server is alive.
> >
> > BTW, could this design doc of proposal follow the template of CIP[1]?
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
> >
> > Regards,
> > Nicholas Jiang
> >
> > On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > > Hi all,
> > >
> > > I have written up a proposal about introducing a CLI for Celeborn. You
> > can
> > > find the proposal
> > > <
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> > >
> > > here.
> > > Please let me know if you have any comments or questions.
> > >
> > > TLDR by introducing a CLI, it would complement the existing dashboard
> and
> > > would benefit us internally. We rely on CLI tools internally a lot for
> > > automation and other operations.
> > >
> > > FYI, I was not able to access the cwiki page to put this proposal
> there,
> > > there seems to be some permissions issue. Hope it is okay to just share
> > as
> > > a google doc here for now.
> > >
> > > --
> > > Aravind K. Patnam
> > >
> > >  Apache Celeborn CLI Proposal
> > > <
> >
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> > >
> > >
> >
>
>
> --
> Aravind K. Patnam
>

Re: [DISCUSS] Celeborn CLI Proposal

Posted by Aravind Patnam <ak...@gmail.com>.
Hi,

Thanks Nicholas for the comments!

I now got access to put the proposal in Confluence in the form of CIP, here
<https://cwiki.apache.org/confluence/display/CELEBORN/CIP+7+-+Celeborn+CLI>
it is.

Regarding your questions:

> 1. From a user's perspective, the CLI is more used for some maintenance
operations such as online and offline of server, rescaling of cluster etc,
not only based on the REST API. What CLI interfaces are there that the REST
API doesn’t have for maintenance?
This is highly dependent on what the user is leveraging to manage their
cluster. For example, in k8s, you would be using k8s APIs to achieve this.
We can probably add a generic interface API for it that provides basic
operations that users can implement themselves for their cluster management
logic based on what cluster managers they are using. Although, I think this
will likely be a later evolution of the CLI, once basic REST API operations
are implemented in the CLI. WDYT?

> 2. There are same sub-commands between MASTER and WORKER. Why not these
sub-commands belong to BOTH?
Agreed - this was a formatting mistake. I fixed it now, thanks for pointing
that out.

> 3. Does the implementation of CLI invoke the REST API? IMO, the CLI works
well no matter the server is alive.
Yes, I agree. I think for this we would have to talk to the cluster
manager, similar to my response to #1. We would have to query the specific
cluster manager to get details if the Celeborn servers are dead, since the
Celeborn REST API would not work then. We can add a generic API that users
can implement based on their own environment.

Thanks,
Aravind



On Wed, Jun 5, 2024 at 10:43 PM Nicholas Jiang <ni...@apache.org>
wrote:

> Hi Aravind,
>
> Thanks for driving this CIP about Celeborn CLI. I have some comments on
> this CIP:
>
> 1. From a user's perspective, the CLI is more used for some maintenance
> operations such as online and offline of server, rescaling of cluster etc,
> not only based on the REST API. What CLI interfaces are there that the REST
> API doesn’t have for maintenance?
>
> 2. There are same sub-commands between MASTER and WORKER. Why not these
> sub-commands belong to BOTH?
>
> 3. Does the implementation of CLI invoke the REST API? IMO, the CLI works
> well no matter the server is alive.
>
> BTW, could this design doc of proposal follow the template of CIP[1]?
>
> [1]
> https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals
>
> Regards,
> Nicholas Jiang
>
> On 2024/06/05 23:33:02 Aravind Patnam wrote:
> > Hi all,
> >
> > I have written up a proposal about introducing a CLI for Celeborn. You
> can
> > find the proposal
> > <
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing
> >
> > here.
> > Please let me know if you have any comments or questions.
> >
> > TLDR by introducing a CLI, it would complement the existing dashboard and
> > would benefit us internally. We rely on CLI tools internally a lot for
> > automation and other operations.
> >
> > FYI, I was not able to access the cwiki page to put this proposal there,
> > there seems to be some permissions issue. Hope it is okay to just share
> as
> > a google doc here for now.
> >
> > --
> > Aravind K. Patnam
> >
> >  Apache Celeborn CLI Proposal
> > <
> https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web
> >
> >
>


-- 
Aravind K. Patnam

Re: [DISCUSS] Celeborn CLI Proposal

Posted by Nicholas Jiang <ni...@apache.org>.
Hi Aravind,

Thanks for driving this CIP about Celeborn CLI. I have some comments on this CIP:

1. From a user's perspective, the CLI is more used for some maintenance operations such as online and offline of server, rescaling of cluster etc, not only based on the REST API. What CLI interfaces are there that the REST API doesn’t have for maintenance?

2. There are same sub-commands between MASTER and WORKER. Why not these sub-commands belong to BOTH?

3. Does the implementation of CLI invoke the REST API? IMO, the CLI works well no matter the server is alive.

BTW, could this design doc of proposal follow the template of CIP[1]?

[1] https://cwiki.apache.org/confluence/display/CELEBORN/Celeborn+Improvement+Proposals

Regards,
Nicholas Jiang

On 2024/06/05 23:33:02 Aravind Patnam wrote:
> Hi all,
> 
> I have written up a proposal about introducing a CLI for Celeborn. You can
> find the proposal
> <https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=sharing>
> here.
> Please let me know if you have any comments or questions.
> 
> TLDR by introducing a CLI, it would complement the existing dashboard and
> would benefit us internally. We rely on CLI tools internally a lot for
> automation and other operations.
> 
> FYI, I was not able to access the cwiki page to put this proposal there,
> there seems to be some permissions issue. Hope it is okay to just share as
> a google doc here for now.
> 
> -- 
> Aravind K. Patnam
> 
>  Apache Celeborn CLI Proposal
> <https://docs.google.com/document/d/1j9wKFSR_ychYDF0NU5YN67WCCtNAgYTbN5CN8V3SOnk/edit?usp=drive_web>
>