You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by Fei Wang <fe...@apache.org> on 2023/01/08 13:06:52 UTC

Separate kyuubi client modules into kyuubi-client repo

Hi Kyuubi community,

I start this thread to discuss moving kyuubi client modules into a
dedicated repo.
- kyuubi-hive-jdbc
- kyuubi-hive-jdbc-shaded
- kyuubi-hive-beeline
- kyuubi-rest-client
- kyuubi-ctl

So that we can
- release the kyuubi server and the kyuubi client separately.
- release light-weight kyuubi client binary.

Here is the ticket: https://github.com/apache/kyuubi/issues/4124
Welcome to join and discuss, any suggestion is appreciated.

Best wishes,
Wang, Fei

Re: Separate kyuubi client modules into kyuubi-client repo

Posted by Wanqiang Ji <ji...@apache.org>.
More repos raise the cost for project management and sync, also bring a
higher threshold for newbies. How do we resolve this problems?


On Tue, Jan 10, 2023 at 11:33 Fei Wang <fe...@apache.org> wrote:

> Hi pan,
>
>
> Thanks for the discussion.
>
>
> > what’s the benefit to separate them now?
>
> I think the benefit is that, we can release kyuubi client more flexible
> and more light weight and decouple the kyuubi server and client modules.
>
> > how to process the shared code, e.g. kyuubi-ctl depends on kyuubi-common.
>
> now kyuubi-hive-jdbc, kyuubi-hive-jdbc-shaded, kyuubi-rest-client and
> kyuubi-hive-beeline does not share any code.
>
> Now kyuubi-ctl shares kyuubi-common and kyuubi-ha, kyuubi-zookeeper. Here
> is the work around, I plan to split kyuubi-ctl to two modules.
> One is kyuubi-server-ctl(used for kyuubi administrator), it dependens
> kyuubi-ha and kyuubi-zookeeper, the functions include
> - operate engine/server zookeeper/etcd nodes
> - kyuubi-admin refresh hadoopConfig
> - list session
>
>
> The another one is kyuubi-client-ctl(used for user side), it just dependes
> little code of kyuubi-common and should be easy to maintain, the functions
> include
> - batch commands
> - list/delete engine with restful api
>
>
> > if we want to introduce the feature which must touch both client and
> server, how to process it?
>
> The developer can also debug with installing the kyuubi-client modules
> locally.
> They need raise two pathces to kyuubi parent repo and kyuubi client repo.
> The kyuubi-client snapshot dependencies will be releaed nightly or more
> frequently.
>
> Thanks,
> Fei
>
> On 2023/01/09 03:19:50 Cheng Pan wrote:
> > Thanks for re-raising this discussion.
> >
> > The last time I raised a similar discussion[1] is to address bad
> developer
> > experience, especially for IDEA debug cases, but now, we have addressed
> it
> > after [2], so what’s the benefit to separate them now?
> >
> > Besides, after separating, we need to consider how to process the shared
> > code, e.g. kyuubi-ctl depends on kyuubi-common, the separating will break
> > it. And if we want to introduce the feature which must touch both client
> > and server, how to process it?
> >
> > [1] https://www.mail-archive.com/dev@kyuubi.apache.org/msg02459.html
> > [2] https://github.com/apache/kyuubi/pull/3847
> >
> > Thanks,
> > Cheng Pan
> >
> >
> > On Jan 9, 2023 at 11:08:54, Kaifei Yi <yi...@gmail.com> wrote:
> >
> > > Looks good, +1, Thanks for working on this.
> > >
> > > 廖天琳 <ti...@163.com> 于2023年1月9日周一 11:01写道：
> > >
> > > +1
> > >
> > > ---- Replied Message ----
> > >
> > > | From | Kent Yao<ya...@apache.org> |
> > >
> > > | Date | 01/09/2023 09:38 |
> > >
> > > | To | dev<de...@kyuubi.apache.org> |
> > >
> > > | Subject | Re: Separate kyuubi client modules into kyuubi-client repo
> |
> > >
> > > +1
> > >
> > >
> > > On 2023/01/08 13:06:52 Fei Wang wrote:
> > >
> > > > Hi Kyuubi community,
> > >
> > > >
> > >
> > > > I start this thread to discuss moving kyuubi client modules into a
> > >
> > > > dedicated repo.
> > >
> > > > - kyuubi-hive-jdbc
> > >
> > > > - kyuubi-hive-jdbc-shaded
> > >
> > > > - kyuubi-hive-beeline
> > >
> > > > - kyuubi-rest-client
> > >
> > > > - kyuubi-ctl
> > >
> > > >
> > >
> > > > So that we can
> > >
> > > > - release the kyuubi server and the kyuubi client separately.
> > >
> > > > - release light-weight kyuubi client binary.
> > >
> > > >
> > >
> > > > Here is the ticket: https://github.com/apache/kyuubi/issues/4124
> > >
> > > > Welcome to join and discuss, any suggestion is appreciated.
> > >
> > > >
> > >
> > > > Best wishes,
> > >
> > > > Wang, Fei
> > >
> > > >
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Best wishes,
> > >
> > > Y.K.F
> > >
> >
>

Re: Separate kyuubi client modules into kyuubi-client repo

Posted by Fei Wang <fe...@apache.org>.
Hi pan,


Thanks for the discussion.


> what’s the benefit to separate them now?

I think the benefit is that, we can release kyuubi client more flexible and more light weight and decouple the kyuubi server and client modules.

> how to process the shared code, e.g. kyuubi-ctl depends on kyuubi-common.

now kyuubi-hive-jdbc, kyuubi-hive-jdbc-shaded, kyuubi-rest-client and kyuubi-hive-beeline does not share any code.

Now kyuubi-ctl shares kyuubi-common and kyuubi-ha, kyuubi-zookeeper. Here is the work around, I plan to split kyuubi-ctl to two modules.
One is kyuubi-server-ctl(used for kyuubi administrator), it dependens kyuubi-ha and kyuubi-zookeeper, the functions include
- operate engine/server zookeeper/etcd nodes
- kyuubi-admin refresh hadoopConfig
- list session


The another one is kyuubi-client-ctl(used for user side), it just dependes little code of kyuubi-common and should be easy to maintain, the functions include
- batch commands
- list/delete engine with restful api


> if we want to introduce the feature which must touch both client and server, how to process it?

The developer can also debug with installing the kyuubi-client modules locally.
They need raise two pathces to kyuubi parent repo and kyuubi client repo.
The kyuubi-client snapshot dependencies will be releaed nightly or more frequently.

Thanks,
Fei

On 2023/01/09 03:19:50 Cheng Pan wrote:
> Thanks for re-raising this discussion.
> 
> The last time I raised a similar discussion[1] is to address bad developer
> experience, especially for IDEA debug cases, but now, we have addressed it
> after [2], so what’s the benefit to separate them now?
> 
> Besides, after separating, we need to consider how to process the shared
> code, e.g. kyuubi-ctl depends on kyuubi-common, the separating will break
> it. And if we want to introduce the feature which must touch both client
> and server, how to process it?
> 
> [1] https://www.mail-archive.com/dev@kyuubi.apache.org/msg02459.html
> [2] https://github.com/apache/kyuubi/pull/3847
> 
> Thanks,
> Cheng Pan
> 
> 
> On Jan 9, 2023 at 11:08:54, Kaifei Yi <yi...@gmail.com> wrote:
> 
> > Looks good, +1, Thanks for working on this.
> >
> > 廖天琳 <ti...@163.com> 于2023年1月9日周一 11:01写道：
> >
> > +1
> >
> > ---- Replied Message ----
> >
> > | From | Kent Yao<ya...@apache.org> |
> >
> > | Date | 01/09/2023 09:38 |
> >
> > | To | dev<de...@kyuubi.apache.org> |
> >
> > | Subject | Re: Separate kyuubi client modules into kyuubi-client repo |
> >
> > +1
> >
> >
> > On 2023/01/08 13:06:52 Fei Wang wrote:
> >
> > > Hi Kyuubi community,
> >
> > >
> >
> > > I start this thread to discuss moving kyuubi client modules into a
> >
> > > dedicated repo.
> >
> > > - kyuubi-hive-jdbc
> >
> > > - kyuubi-hive-jdbc-shaded
> >
> > > - kyuubi-hive-beeline
> >
> > > - kyuubi-rest-client
> >
> > > - kyuubi-ctl
> >
> > >
> >
> > > So that we can
> >
> > > - release the kyuubi server and the kyuubi client separately.
> >
> > > - release light-weight kyuubi client binary.
> >
> > >
> >
> > > Here is the ticket: https://github.com/apache/kyuubi/issues/4124
> >
> > > Welcome to join and discuss, any suggestion is appreciated.
> >
> > >
> >
> > > Best wishes,
> >
> > > Wang, Fei
> >
> > >
> >
> >
> >
> >
> > --
> >
> > Best wishes,
> >
> > Y.K.F
> >
>

Re: Separate kyuubi client modules into kyuubi-client repo

Posted by Cheng Pan <ch...@apache.org>.
Thanks for re-raising this discussion.

The last time I raised a similar discussion[1] is to address bad developer
experience, especially for IDEA debug cases, but now, we have addressed it
after [2], so what’s the benefit to separate them now?

Besides, after separating, we need to consider how to process the shared
code, e.g. kyuubi-ctl depends on kyuubi-common, the separating will break
it. And if we want to introduce the feature which must touch both client
and server, how to process it?

[1] https://www.mail-archive.com/dev@kyuubi.apache.org/msg02459.html
[2] https://github.com/apache/kyuubi/pull/3847

Thanks,
Cheng Pan


On Jan 9, 2023 at 11:08:54, Kaifei Yi <yi...@gmail.com> wrote:

> Looks good, +1, Thanks for working on this.
>
> 廖天琳 <ti...@163.com> 于2023年1月9日周一 11:01写道：
>
> +1
>
> ---- Replied Message ----
>
> | From | Kent Yao<ya...@apache.org> |
>
> | Date | 01/09/2023 09:38 |
>
> | To | dev<de...@kyuubi.apache.org> |
>
> | Subject | Re: Separate kyuubi client modules into kyuubi-client repo |
>
> +1
>
>
> On 2023/01/08 13:06:52 Fei Wang wrote:
>
> > Hi Kyuubi community,
>
> >
>
> > I start this thread to discuss moving kyuubi client modules into a
>
> > dedicated repo.
>
> > - kyuubi-hive-jdbc
>
> > - kyuubi-hive-jdbc-shaded
>
> > - kyuubi-hive-beeline
>
> > - kyuubi-rest-client
>
> > - kyuubi-ctl
>
> >
>
> > So that we can
>
> > - release the kyuubi server and the kyuubi client separately.
>
> > - release light-weight kyuubi client binary.
>
> >
>
> > Here is the ticket: https://github.com/apache/kyuubi/issues/4124
>
> > Welcome to join and discuss, any suggestion is appreciated.
>
> >
>
> > Best wishes,
>
> > Wang, Fei
>
> >
>
>
>
>
> --
>
> Best wishes,
>
> Y.K.F
>

Re: Separate kyuubi client modules into kyuubi-client repo

Posted by Kaifei Yi <yi...@gmail.com>.
Looks good, +1, Thanks for working on this.

廖天琳 <ti...@163.com> 于2023年1月9日周一 11:01写道：

> +1
> ---- Replied Message ----
> | From | Kent Yao<ya...@apache.org> |
> | Date | 01/09/2023 09:38 |
> | To | dev<de...@kyuubi.apache.org> |
> | Subject | Re: Separate kyuubi client modules into kyuubi-client repo |
> +1
>
> On 2023/01/08 13:06:52 Fei Wang wrote:
> > Hi Kyuubi community,
> >
> > I start this thread to discuss moving kyuubi client modules into a
> > dedicated repo.
> > - kyuubi-hive-jdbc
> > - kyuubi-hive-jdbc-shaded
> > - kyuubi-hive-beeline
> > - kyuubi-rest-client
> > - kyuubi-ctl
> >
> > So that we can
> > - release the kyuubi server and the kyuubi client separately.
> > - release light-weight kyuubi client binary.
> >
> > Here is the ticket: https://github.com/apache/kyuubi/issues/4124
> > Welcome to join and discuss, any suggestion is appreciated.
> >
> > Best wishes,
> > Wang, Fei
> >
>


-- 

Best wishes,

Y.K.F

Re: Separate kyuubi client modules into kyuubi-client repo

Posted by 廖天琳 <ti...@163.com>.
+1
---- Replied Message ----
| From | Kent Yao<ya...@apache.org> |
| Date | 01/09/2023 09:38 |
| To | dev<de...@kyuubi.apache.org> |
| Subject | Re: Separate kyuubi client modules into kyuubi-client repo |
+1

On 2023/01/08 13:06:52 Fei Wang wrote:
> Hi Kyuubi community,
>
> I start this thread to discuss moving kyuubi client modules into a
> dedicated repo.
> - kyuubi-hive-jdbc
> - kyuubi-hive-jdbc-shaded
> - kyuubi-hive-beeline
> - kyuubi-rest-client
> - kyuubi-ctl
>
> So that we can
> - release the kyuubi server and the kyuubi client separately.
> - release light-weight kyuubi client binary.
>
> Here is the ticket: https://github.com/apache/kyuubi/issues/4124
> Welcome to join and discuss, any suggestion is appreciated.
>
> Best wishes,
> Wang, Fei
>

Re: Separate kyuubi client modules into kyuubi-client repo

Posted by Fu Chen <cf...@gmail.com>.
+1

Kent Yao <ya...@apache.org> 于2023年1月9日周一 09:38写道：

> +1
>
> On 2023/01/08 13:06:52 Fei Wang wrote:
> > Hi Kyuubi community,
> >
> > I start this thread to discuss moving kyuubi client modules into a
> > dedicated repo.
> > - kyuubi-hive-jdbc
> > - kyuubi-hive-jdbc-shaded
> > - kyuubi-hive-beeline
> > - kyuubi-rest-client
> > - kyuubi-ctl
> >
> > So that we can
> > - release the kyuubi server and the kyuubi client separately.
> > - release light-weight kyuubi client binary.
> >
> > Here is the ticket: https://github.com/apache/kyuubi/issues/4124
> > Welcome to join and discuss, any suggestion is appreciated.
> >
> > Best wishes,
> > Wang, Fei
> >
>

Re: Separate kyuubi client modules into kyuubi-client repo

Posted by Kent Yao <ya...@apache.org>.
+1

On 2023/01/08 13:06:52 Fei Wang wrote:
> Hi Kyuubi community,
> 
> I start this thread to discuss moving kyuubi client modules into a
> dedicated repo.
> - kyuubi-hive-jdbc
> - kyuubi-hive-jdbc-shaded
> - kyuubi-hive-beeline
> - kyuubi-rest-client
> - kyuubi-ctl
> 
> So that we can
> - release the kyuubi server and the kyuubi client separately.
> - release light-weight kyuubi client binary.
> 
> Here is the ticket: https://github.com/apache/kyuubi/issues/4124
> Welcome to join and discuss, any suggestion is appreciated.
> 
> Best wishes,
> Wang, Fei
>