You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by Kent Yao <ya...@apache.org> on 2021/05/31 06:21:14 UTC
[DISCUSS] Incubating Proposal of Kyuubi
Dear all,
We want to propose Kyuubi as a new Apache Incubator project.
Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
large-scale data management, processing, and analytics, built on top
of Apache Spark and designed to support more engines (i.e., Apache
Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
warehouses and data lakes. Kyuubi concentrates on improving the
experience for non-Spark users, making it easy for them to use Spark
as a service and focus on business data of their own, not where it
stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
https://github.com/NetEase/kyuubi.
Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
Ajisaka for being the mentors of the Kyuubi project and helping us set
up this proposal. The proposal can be found at
https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
Looking forward to your feedback and thank you very much.
Best regards,
Kent Yao
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by "Wang, Fei" <fw...@ebay.com.INVALID>.
+1 (non-binding).
Good luck.
Regards,
Fei
On 2021/5/31, 3:45 PM, "Kent Yao" <ya...@apache.org> wrote:
External Email
Hi Jean-Baptiste Onofre,
thanks for your reply.
You are correct. And we are focusing on the part processing with Spark
not storage with Hive. Since Spark provides Datasource V2, the data
access and connectivity need not bind to Hive in the Spark layer, it
becomes more and more promising.
Bests
Kent Yao
Jean-Baptiste Onofre <jb...@nanthrax.net> 于2021年5月31日周一 下午2:36写道：
>
> Hi Kent Yao,
>
> It looks interesting. Correct me if I’m wrong, but it seems to be more a SQL gateway, more than a "server", meaning that the storage is delegated to Hive and processing to Spark, right ?
>
> Regards
> JB
>
> > Le 31 mai 2021 à 08:21, Kent Yao <ya...@apache.org> a écrit :
> >
> > Dear all,
> >
> > We want to propose Kyuubi as a new Apache Incubator project.
> >
> > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > large-scale data management, processing, and analytics, built on top
> > of Apache Spark and designed to support more engines (i.e., Apache
> > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> > warehouses and data lakes. Kyuubi concentrates on improving the
> > experience for non-Spark users, making it easy for them to use Spark
> > as a service and focus on business data of their own, not where it
> > stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNetEase%2Fkyuubi&data=04%7C01%7Cfwang12%40ebay.com%7Cf48249cda8644d70595d08d9240818b8%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637580439456537832%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pT8oxxS9UlyadADqOCK9pu0V26IG1rpvPuj4Z9uyQUw%3D&reserved=0.
> >
> > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> > Ajisaka for being the mentors of the Kyuubi project and helping us set
> > up this proposal. The proposal can be found at
> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FINCUBATOR%2FKyuubiProposal&data=04%7C01%7Cfwang12%40ebay.com%7Cf48249cda8644d70595d08d9240818b8%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637580439456547790%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=gTGTk2xS8ZsaSPwqVKIgkpVM0CoiqkrSnFXRJLk6ovU%3D&reserved=0.
> >
> > Looking forward to your feedback and thank you very much.
> >
> > Best regards,
> > Kent Yao
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
--
Kent Yao
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubiis a unified multi-tenant JDBC interface for large-scale data
processing and analytics, built on top of Apache Spark.
spark-authorizerA Spark SQL extension which provides SQL Standard
Authorization for Apache Spark.
spark-postgres A library for reading data from and transferring data
to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
itatchiA library that brings useful functions from various modern
database management systems to Apache Spark.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Hi Jean-Baptiste Onofre,
thanks for your reply.
You are correct. And we are focusing on the part processing with Spark
not storage with Hive. Since Spark provides Datasource V2, the data
access and connectivity need not bind to Hive in the Spark layer, it
becomes more and more promising.
Bests
Kent Yao
Jean-Baptiste Onofre <jb...@nanthrax.net> 于2021年5月31日周一 下午2:36写道：
>
> Hi Kent Yao,
>
> It looks interesting. Correct me if I’m wrong, but it seems to be more a SQL gateway, more than a "server", meaning that the storage is delegated to Hive and processing to Spark, right ?
>
> Regards
> JB
>
> > Le 31 mai 2021 à 08:21, Kent Yao <ya...@apache.org> a écrit :
> >
> > Dear all,
> >
> > We want to propose Kyuubi as a new Apache Incubator project.
> >
> > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > large-scale data management, processing, and analytics, built on top
> > of Apache Spark and designed to support more engines (i.e., Apache
> > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> > warehouses and data lakes. Kyuubi concentrates on improving the
> > experience for non-Spark users, making it easy for them to use Spark
> > as a service and focus on business data of their own, not where it
> > stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> > https://github.com/NetEase/kyuubi.
> >
> > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> > Ajisaka for being the mentors of the Kyuubi project and helping us set
> > up this proposal. The proposal can be found at
> > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> >
> > Looking forward to your feedback and thank you very much.
> >
> > Best regards,
> > Kent Yao
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
--
Kent Yao
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubiis a unified multi-tenant JDBC interface for large-scale data
processing and analytics, built on top of Apache Spark.
spark-authorizerA Spark SQL extension which provides SQL Standard
Authorization for Apache Spark.
spark-postgres A library for reading data from and transferring data
to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
itatchiA library that brings useful functions from various modern
database management systems to Apache Spark.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi Kent Yao,
It looks interesting. Correct me if I’m wrong, but it seems to be more a SQL gateway, more than a "server", meaning that the storage is delegated to Hive and processing to Spark, right ?
Regards
JB
> Le 31 mai 2021 à 08:21, Kent Yao <ya...@apache.org> a écrit :
>
> Dear all,
>
> We want to propose Kyuubi as a new Apache Incubator project.
>
> Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> large-scale data management, processing, and analytics, built on top
> of Apache Spark and designed to support more engines (i.e., Apache
> Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> warehouses and data lakes. Kyuubi concentrates on improving the
> experience for non-Spark users, making it easy for them to use Spark
> as a service and focus on business data of their own, not where it
> stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> https://github.com/NetEase/kyuubi.
>
> Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> Ajisaka for being the mentors of the Kyuubi project and helping us set
> up this proposal. The proposal can be found at
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
>
> Looking forward to your feedback and thank you very much.
>
> Best regards,
> Kent Yao
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by hang zheng <22...@qq.com>.
looks interesting
On 2021/05/31 06:21:14, Kent Yao <ya...@apache.org> wrote:
> Dear all,
>
> We want to propose Kyuubi as a new Apache Incubator project.
>
> Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> large-scale data management, processing, and analytics, built on top
> of Apache Spark and designed to support more engines (i.e., Apache
> Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> warehouses and data lakes. Kyuubi concentrates on improving the
> experience for non-Spark users, making it easy for them to use Spark
> as a service and focus on business data of their own, not where it
> stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> https://github.com/NetEase/kyuubi.
>
> Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> Ajisaka for being the mentors of the Kyuubi project and helping us set
> up this proposal. The proposal can be found at
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
>
> Looking forward to your feedback and thank you very much.
>
> Best regards,
> Kent Yao
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by hang zheng <22...@qq.com>.
looks useful
On 2021/05/31 06:21:14, Kent Yao <ya...@apache.org> wrote:
> Dear all,
>
> We want to propose Kyuubi as a new Apache Incubator project.
>
> Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> large-scale data management, processing, and analytics, built on top
> of Apache Spark and designed to support more engines (i.e., Apache
> Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> warehouses and data lakes. Kyuubi concentrates on improving the
> experience for non-Spark users, making it easy for them to use Spark
> as a service and focus on business data of their own, not where it
> stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> https://github.com/NetEase/kyuubi.
>
> Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> Ajisaka for being the mentors of the Kyuubi project and helping us set
> up this proposal. The proposal can be found at
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
>
> Looking forward to your feedback and thank you very much.
>
> Best regards,
> Kent Yao
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by ������ <le...@gmail.com>.
looks niubility
On 2021/05/31 06:21:14, Kent Yao <ya...@apache.org> wrote:
> Dear all,
>
> We want to propose Kyuubi as a new Apache Incubator project.
>
> Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> large-scale data management, processing, and analytics, built on top
> of Apache Spark and designed to support more engines (i.e., Apache
> Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> warehouses and data lakes. Kyuubi concentrates on improving the
> experience for non-Spark users, making it easy for them to use Spark
> as a service and focus on business data of their own, not where it
> stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> https://github.com/NetEase/kyuubi.
>
> Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> Ajisaka for being the mentors of the Kyuubi project and helping us set
> up this proposal. The proposal can be found at
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
>
> Looking forward to your feedback and thank you very much.
>
> Best regards,
> Kent Yao
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by chuan jin <ji...@gmail.com>.
Nice job, i am looking foward to more interesting feature.
On 2021/05/31 06:21:14, Kent Yao <ya...@apache.org> wrote:
> Dear all,
>
> We want to propose Kyuubi as a new Apache Incubator project.
>
> Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> large-scale data management, processing, and analytics, built on top
> of Apache Spark and designed to support more engines (i.e., Apache
> Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> warehouses and data lakes. Kyuubi concentrates on improving the
> experience for non-Spark users, making it easy for them to use Spark
> as a service and focus on business data of their own, not where it
> stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> https://github.com/NetEase/kyuubi.
>
> Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> Ajisaka for being the mentors of the Kyuubi project and helping us set
> up this proposal. The proposal can be found at
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
>
> Looking forward to your feedback and thank you very much.
>
> Best regards,
> Kent Yao
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by bei peng <mr...@gmail.com>.
looks like a very practical project
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by hddong <ho...@gmail.com>.
+1
Kyuubi is useful for us. In chinamobile soft's big data platform, we use kyuubi to solve the problem of multi tenant access for a long time. It's helpful and easy to use.
I just joined Kyuubi community for a short time, but I really enjoy the process of cooperation with the community. I hope I can make a long-term contribution in the future.
Looking forword to kyuubi to be a ASF project.
Best regard
hongdd
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by ��������� <we...@gmail.com>.
+1 (non-binding) for incubation.
On 2021/05/31 07:29:31, Kent Yao <ya...@apache.org> wrote:
> Hi Alexander Alten,
>
> Thanks for your interest and +1 for Kyuubi.
>
> Not being very familiar with Dremel, I just took a quick look at the
> profile of a very similar and exciting project, Apache Drill. If I
> understand correctly, Kyuubi is built on top of the compute layer
> (Apache Spark), while Apache Drill is built on top of the storage
> layer.
>
> Apache Spark has a large user base and usage scenarios all over the
> world. When users have some basic Spark programming and tuning skills,
> things are easy for them. But many times, most users do not have all
> of these abilities when they use Spark directly or indirectly. We want
> to build a more easy-to-use platform on top of it to help these users.
> On the one hand, we will work on the interface layer to make it easy
> to access (of course, this effort will make our project look very
> similar to other projects, such as HiveServer2). On the other hand, we
> will work under Kyuubi's multi-tenant architecture to extend our
> on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more stable
> and efficient in our scenario.
>
> What's more, we are going to add more on-prem Kyuubi engine types, e.g. Flink.
>
> Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
>
> >
> > Hey,
> >
> > Looks like an interesting project, competition to Starburst. How does it
> > compare to Apache Dremel?
> >
> > +1 (non-binding) for incubation.
> >
> > Thanks and stay safe,
> > --alex
> >
> > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> >
> > > Dear all,
> > >
> > > We want to propose Kyuubi as a new Apache Incubator project.
> > >
> > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > > large-scale data management, processing, and analytics, built on top
> > > of Apache Spark and designed to support more engines (i.e., Apache
> > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> > > warehouses and data lakes. Kyuubi concentrates on improving the
> > > experience for non-Spark users, making it easy for them to use Spark
> > > as a service and focus on business data of their own, not where it
> > > stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> > > https://github.com/NetEase/kyuubi.
> > >
> > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> > > Ajisaka for being the mentors of the Kyuubi project and helping us set
> > > up this proposal. The proposal can be found at
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> > >
> > > Looking forward to your feedback and thank you very much.
> > >
> > > Best regards,
> > > Kent Yao
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
>
>
>
> --
>
> Kent Yao
> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> a spark enthusiast
> kyuubiis a unified multi-tenant JDBC interface for large-scale data
> processing and analytics, built on top of Apache Spark.
>
> spark-authorizerA Spark SQL extension which provides SQL Standard
> Authorization for Apache Spark.
> spark-postgres A library for reading data from and transferring data
> to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
> itatchiA library that brings useful functions from various modern
> database management systems to Apache Spark.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
RE: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Cheng Pan <37...@qq.com>.
Hi all,
Thanks for the review and questions.
I'm one of the Kyuubi committers, and have participated in the recent 6 months, include code changes[1], PR reviews,
documents improvements, discussions, etc.
I'd like to say, both Kent and other committers are nice and patient persons, and most of the discussions and
code reviews happen in GitHub, I got many benefits from the contribution of Kyuubi.
Kyuubi community is still young, but growing.
From GitHub statistics[2], in the last 6 months, there are over 20 people participated in Kyuubi code development, include
6 people over hundred lines change, 4 people over thousand lines change.
As a new committer, besides on code contribution, I'm also trying my best to extend the community. I recommend those engineers
who are finding the way to migrate ETL infrastructure from Hive to Spark SQL to have a try on Kyuubi, and collect the bug
reports[3][4] and some feature requests from them. I'm also looking for some potential contributors from the open-source
community. For example, there is a project[5] build on top of Kyuubi to simplify to the deployment of Spark SQL on Kubernetes.
And there is a blog[6](in Chinese) written in recent days that analyzed the architecture design and code implementation of Kyuubi.
And from the mail lists in Apache Incubator discussions, I also see some people are interested in this project.
I know there is still a long way to go to building the community, and hope to get helped by Apache Incubator.
[1] https://github.com/NetEase/kyuubi/commits?author=pan3793
[2] https://github.com/NetEase/kyuubi/graphs/contributors?from=2020-12-01&to=2021-06-02&type=c
[3] https://github.com/NetEase/kyuubi/issues/280
[4] https://github.com/NetEase/kyuubi/issues/647
[5] https://github.com/yilong2001/spark-sql-on-k8s
[6] https://blog.51cto.com/xpleaf/2780248
Best,
Cheng Pan
------------------ 原始邮件 ------------------
发件人: "general" <atri@apache.org>;
发送时间: 2021年6月2日(星期三) 晚上11:26
收件人: "general"<general@incubator.apache.org>;
主题: Re: [DISCUSS] Incubating Proposal of Kyuubi
I am genuinely worried about one contributor being the dominant of all.
This puts the stability of the project in jeopardy and unless this is
remedied, I feel the project should focus on developing a more diverse
community first.
On Wed, 2 Jun 2021, 20:23 Sheng Wu, <wu.sheng.841108@gmail.com> wrote:
> Hi Kent
>
> Thanks for replying, but to be honest, your mail format is chaos.
> Take a look at bellowing, which is what I saw in the gmail.
>
> This may be an issue of your mail client or some else.
>
> > The current #.2 - 5 top
> > contributors will stay as committers, and they will join the PPMC during
> > incubation.
>
> PPMC usually is the default for all initial committers, considering your
> community is still small and not very active.
> You should invite all people you trusted to join, rather than pushing them
> away.
> This is not just a too-high bar issue. Excluding 4 out of 5 is strange.
> I can't guess what is the root cause, why the other 4 PPMC members don't
> touch codes but are trusted completely. Meanwhile, people with easily
> measurable code contributions are just as committers?
> We(ASF) always remind the community to recognize no-code contribution, but
> in this case, it seems the opposite? Don't recognize code contributors
> enough?
> Please reach your mentors, this seems not a typical ASF style. If you have
> other people already involved in the ASF, asking them too.
>
> And from your description of the release process, I think you should talk
> more with your mentors.
> Such as what is a qualified ASF release, what should be checked.
> This is not a block, the point I mentioned this is, you will need more
> people.
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Kent Yao <yao@apache.org> 于2021年6月2日周三 下午10:37写道：
>
> > Hi Sheng,
> >
> > Thanks for your detailed check and valuable comments. They are
> > accommodating. I have addressed some of them directly in the proposal.
> > Please also check the table below for some Q&As.
> > QA
> > You listed only 5 core maintainers, but clearly only one has dominated
> > contributions. The distortion and skewness of the contributions are
> mainly
> > caused by the way how we merge PRs changed(mentioned in another email on
> > the list).
> > I agree with Atri, orphaned is measurable. It is better to adjust this
> > statement a little. Address the comment to notify the risk in the Kyuubi
> > proposal.
> > Also, with this fact, the core maintainers should notice whether this
> small
> > group of people to make sure your features iteration, bug fix, and
> release
> > process working by following ASF's requirements.This would not be an easy
> > job. Currently, we learn the release process from some Apache projects,
> > using major.minor.patch for versioning. As we can see, since Kyuubi 1.0,
> it
> > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes
> until
> > EOL. And it takes us about two months to finish a minor release with 3 ~
> 5
> > big improvements and various small features/bug fixes. We also will start
> > with some release candidates for preview and test before the final. This
> is
> > how we develop and release Kyuubi for now, but we do have a long way to
> go
> > and learn to match the ASF requirement.
> > And your committer list is very strange... 1. Why some of them are PPMC
> and
> > others are committers? Where this hierarchy comes from? The current PPMC
> > candidates help us a lot build the Kyuubi project and community, not
> > directly to the codebase but as mentors(before Apache). Most of them are
> > from Apache mature TLPs, which have close relationships with Kyuubi. They
> > have sufficient Apache Way experience to guide us to become a more
> > Apache-like project before submitting this proposal in many ways. We will
> > continue to promote more contributors to the PPMC/committer based on
> their
> > code or non-code contributions during incubation.
> > 2. Your core maintainers have 5, which only 4 PPMC? The current #.2 - 5
> top
> > contributors will stay as committers, and they will join the PPMC during
> > incubation.
> > 3. I think only the top 4 have over 400 LOC, why you declared you have 5
> > core developers? I didn't notice that 400 LOC is a bar for core
> developer,
> > will change 5 to 4 core developers accordingly
> > I think you don't have enough no-salaried developers, am I missing
> > anything? Yes.
> > Changed in the proposal, most of the contributors are salaried, and some
> of
> > them are willing to help in the spare time. Thanks
> > This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA in
> > Proposal. All work here is approved by NetEase and can be forward.
> > At last, you are asking about github.com/apache/incubator-kyuubi-shaded
> > repo,
> > what does this represent? I can only see you have only one source repo
> and
> > a doc repo. Removed this requirement in the proposal, and we will apply
> it
> > from IPMC if we do need this in the future.
> > ------------------------------
> >
> > Bests
> >
> > Kent Yao
> >
> > Sheng Wu <wu.sheng.841108@gmail.com> 于2021年6月2日周三 下午5:45写道：
> >
> > > Hi
> > >
> > > I read the contributor list and the proposal
> > > > The risk of the Kyuubi project being abandoned is minimal. Many
> > > organizations are using Kyuubi to build critical big data pipelines and
> > > willing to help develop Kyuubi's community if it becomes an ASF
> project.
> > >
> > > I think the team should evaluate the status again. You listed only 5
> core
> > > maintainers, but clearly only one has dominated contributions. I agree
> > with
> > > Atri, orphaned is measurable.
> > > It is better to adjust this statement a little. Also, with this fact,
> the
> > > core maintainers should notice whether this small group of people to
> make
> > > sure your features iteration, bug fix, and release process working by
> > > following ASF's requirements.
> > > This would not be an easy job.
> > >
> > > And your committer list is very strange
> > > ```
> > > Kent Yao (yao@apache.org) (PPMC)
> > > Wenchen Fan (wenchen@apache.org) (PPMC)
> > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > > Xun Liu (liuxun@apache.org) (PPMC)
> > > Fei Wang (cn.feiwang@gmail.com)
> > > Ulysses You (ulyssesyou18@gmail.com)
> > > Cheng Pan (pan3793@gmail.com)
> > > Hongxiang Jiang (jhx1008@gmail.com)
> > > Zen Jin(xinjingziranchan@gmail.com)
> > > ```
> > > 1. Why some of them are PPMC and others are committers? Where this
> > > hierarchy comes from?
> > > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy
> makes
> > > sense, most likely not)
> > > 3. I think only the top 4 have over 400 LOC, why you declared you have
> 5
> > > core developers?
> > >
> > > Another re-evaluation request is about `Reliance on Salaried
> Developers:`
> > > > They are all passionate about the project, and we are confident that
> > the
> > > project will continue even if no salaried developers contribute to the
> > > project.
> > >
> > > I think you don't have enough no-salaried developers, am I missing
> > > anything?
> > >
> > > About IP clearance, `Initial Source and Intellectual Property
> Submission
> > > Plan`
> > > > As soon as Kyuubi is approved to join Apache Incubator, our initial
> > > committers will submit iCLA(s) and CCLA(s). The codebase is already
> > > licensed under the Apache License 2.0.
> > >
> > > This project belongs to NetEase Inc. clearly, why NO SGA?
> > >
> > > At last, you are asking about `
> > github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> > <http://github.com/apache/incubator-kyuubi-shaded>
> > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > repo, what does this represent? I can only see you have only one source
> > > repo and a doc repo.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Atri Sharma <atri@apache.org> 于2021年6月2日周三 下午4:49写道：
> > >
> > > > Thanks.
> > > >
> > > > I looked at the link you advised on and if I am reading it correctly,
> > top
> > > > committer has 630ish commits and the next committer has 73 commits?
> > > >
> > > > IMO this is an indication of the community not being active enough
> and
> > > > hence the risk of project getting orphaned is measurable.
> > > >
> > > > Atri
> > > >
> > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <willem.jiang@gmail.com>
> > wrote:
> > > >
> > > > > Hi Atri Sharma,
> > > > >
> > > > > It's easier to check out the recent contribution of kyuubi from
> this
> > > > > link[1].
> > > > > We already identified the one person contribution issue of kyuubi
> > when
> > > > > Kent came to us and we kept reminding him to put more effort into
> > > > > building the community.
> > > > > Last week I had a long F2F conversation with Kent, I think he
> already
> > > > > knows about it and there have been some improvements in recent
> months
> > > > > but there is still a long way to go if Kyuubi is accepted as a
> > podling
> > > > > in the Apache Incubator.
> > > > >
> > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > > >
> > > > > Willem Jiang
> > > > >
> > > > > Twitter: willemjiang
> > > > > Weibo: 姜宁willem
> > > > >
> > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <atri@apache.org>
> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I took a look at commit log and seems like all the commits for
> > last 3
> > > > > > months have been done by a single person.
> > > > > >
> > > > > > How active are the remaining committers?
> > > > > >
> > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <yao@apache.org> wrote:
> > > > > >
> > > > > > > Hi Alexander Alten,
> > > > > > >
> > > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > > >
> > > > > > > Not being very familiar with Dremel, I just took a quick look
> at
> > > the
> > > > > > > profile of a very similar and exciting project, Apache Drill.
> If
> > I
> > > > > > > understand correctly, Kyuubi is built on top of the compute
> layer
> > > > > > > (Apache Spark), while Apache Drill is built on top of the
> storage
> > > > > > > layer.
> > > > > > >
> > > > > > > Apache Spark has a large user base and usage scenarios all over
> > the
> > > > > > > world. When users have some basic Spark programming and tuning
> > > > skills,
> > > > > > > things are easy for them. But many times, most users do not
> have
> > > all
> > > > > > > of these abilities when they use Spark directly or indirectly.
> We
> > > > want
> > > > > > > to build a more easy-to-use platform on top of it to help these
> > > > users.
> > > > > > > On the one hand, we will work on the interface layer to make it
> > > easy
> > > > > > > to access (of course, this effort will make our project look
> very
> > > > > > > similar to other projects, such as HiveServer2). On the other
> > hand,
> > > > we
> > > > > > > will work under Kyuubi's multi-tenant architecture to extend
> our
> > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more
> > > stable
> > > > > > > and efficient in our scenario.
> > > > > > >
> > > > > > > What's more, we are going to add more on-prem Kyuubi engine
> > types,
> > > > e.g.
> > > > > > > Flink.
> > > > > > >
> > > > > > > Alexander Alten <alo@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > > > >
> > > > > > > >
> > > > > > > > Hey,
> > > > > > > >
> > > > > > > > Looks like an interesting project, competition to Starburst.
> > How
> > > > > does it
> > > > > > > > compare to Apache Dremel?
> > > > > > > >
> > > > > > > > +1 (non-binding) for incubation.
> > > > > > > >
> > > > > > > > Thanks and stay safe,
> > > > > > > > --alex
> > > > > > > >
> > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <yao@apache.org> wrote:
> > > > > > > >
> > > > > > > > > Dear all,
> > > > > > > > >
> > > > > > > > > We want to propose Kyuubi as a new Apache Incubator
> project.
> > > > > > > > >
> > > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC
> server
> > > for
> > > > > > > > > large-scale data management, processing, and analytics,
> built
> > > on
> > > > > top
> > > > > > > > > of Apache Spark and designed to support more engines (i.e.,
> > > > Apache
> > > > > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box"
> tool
> > > for
> > > > > data
> > > > > > > > > warehouses and data lakes. Kyuubi concentrates on improving
> > the
> > > > > > > > > experience for non-Spark users, making it easy for them to
> > use
> > > > > Spark
> > > > > > > > > as a service and focus on business data of their own, not
> > where
> > > > it
> > > > > > > > > stores, how it computes. NetEase has open-sourced Kyuubi
> > since
> > > > > 2018,
> > > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > > >
> > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo
> > Zhang,
> > > > > Akira
> > > > > > > > > Ajisaka for being the mentors of the Kyuubi project and
> > helping
> > > > us
> > > > > set
> > > > > > > > > up this proposal. The proposal can be found at
> > > > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> > .
> > > > > > > > >
> > > > > > > > > Looking forward to your feedback and thank you very much.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Kent Yao
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail:
> > > general-unsubscribe@incubator.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Kent Yao
> > > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
> Corp.
> > > > > > > a spark enthusiast
> > > > > > > kyuubiis a unified multi-tenant JDBC interface for large-scale
> > data
> > > > > > > processing and analytics, built on top of Apache Spark.
> > > > > > >
> > > > > > > spark-authorizerA Spark SQL extension which provides SQL
> Standard
> > > > > > > Authorization for Apache Spark.
> > > > > > > spark-postgres A library for reading data from and transferring
> > > data
> > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x
> > > > faster.
> > > > > > > itatchiA library that brings useful functions from various
> modern
> > > > > > > database management systems to Apache Spark.
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > *Kent Yao *
> > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > a spark enthusiast
> > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> JDBC
> > interface for large-scale data processing and analytics, built on top of
> > Apache
> > Spark <http://spark.apache.org/>.
> > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> SQL
> > extension which provides SQL Standard Authorization for Apache Spark
> > <http://spark.apache.org/>.
> > spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
> for
> > reading data from and transferring data to Postgres / Greenplum with
> Spark
> > SQL and DataFrames, 10~100x faster.
> > itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> > brings useful functions from various modern database management
> > systems to Apache
> > Spark <http://spark.apache.org/>.
> >
>
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Atri Sharma <at...@apache.org>.
I am genuinely worried about one contributor being the dominant of all.
This puts the stability of the project in jeopardy and unless this is
remedied, I feel the project should focus on developing a more diverse
community first.
On Wed, 2 Jun 2021, 20:23 Sheng Wu, <wu...@gmail.com> wrote:
> Hi Kent
>
> Thanks for replying, but to be honest, your mail format is chaos.
> Take a look at bellowing, which is what I saw in the gmail.
>
> This may be an issue of your mail client or some else.
>
> > The current #.2 - 5 top
> > contributors will stay as committers, and they will join the PPMC during
> > incubation.
>
> PPMC usually is the default for all initial committers, considering your
> community is still small and not very active.
> You should invite all people you trusted to join, rather than pushing them
> away.
> This is not just a too-high bar issue. Excluding 4 out of 5 is strange.
> I can't guess what is the root cause, why the other 4 PPMC members don't
> touch codes but are trusted completely. Meanwhile, people with easily
> measurable code contributions are just as committers?
> We(ASF) always remind the community to recognize no-code contribution, but
> in this case, it seems the opposite? Don't recognize code contributors
> enough?
> Please reach your mentors, this seems not a typical ASF style. If you have
> other people already involved in the ASF, asking them too.
>
> And from your description of the release process, I think you should talk
> more with your mentors.
> Such as what is a qualified ASF release, what should be checked.
> This is not a block, the point I mentioned this is, you will need more
> people.
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
>
> > Hi Sheng,
> >
> > Thanks for your detailed check and valuable comments. They are
> > accommodating. I have addressed some of them directly in the proposal.
> > Please also check the table below for some Q&As.
> > QA
> > You listed only 5 core maintainers, but clearly only one has dominated
> > contributions. The distortion and skewness of the contributions are
> mainly
> > caused by the way how we merge PRs changed(mentioned in another email on
> > the list).
> > I agree with Atri, orphaned is measurable. It is better to adjust this
> > statement a little. Address the comment to notify the risk in the Kyuubi
> > proposal.
> > Also, with this fact, the core maintainers should notice whether this
> small
> > group of people to make sure your features iteration, bug fix, and
> release
> > process working by following ASF's requirements.This would not be an easy
> > job. Currently, we learn the release process from some Apache projects,
> > using major.minor.patch for versioning. As we can see, since Kyuubi 1.0,
> it
> > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes
> until
> > EOL. And it takes us about two months to finish a minor release with 3 ~
> 5
> > big improvements and various small features/bug fixes. We also will start
> > with some release candidates for preview and test before the final. This
> is
> > how we develop and release Kyuubi for now, but we do have a long way to
> go
> > and learn to match the ASF requirement.
> > And your committer list is very strange... 1. Why some of them are PPMC
> and
> > others are committers? Where this hierarchy comes from? The current PPMC
> > candidates help us a lot build the Kyuubi project and community, not
> > directly to the codebase but as mentors(before Apache). Most of them are
> > from Apache mature TLPs, which have close relationships with Kyuubi. They
> > have sufficient Apache Way experience to guide us to become a more
> > Apache-like project before submitting this proposal in many ways. We will
> > continue to promote more contributors to the PPMC/committer based on
> their
> > code or non-code contributions during incubation.
> > 2. Your core maintainers have 5, which only 4 PPMC? The current #.2 - 5
> top
> > contributors will stay as committers, and they will join the PPMC during
> > incubation.
> > 3. I think only the top 4 have over 400 LOC, why you declared you have 5
> > core developers? I didn't notice that 400 LOC is a bar for core
> developer,
> > will change 5 to 4 core developers accordingly
> > I think you don't have enough no-salaried developers, am I missing
> > anything? Yes.
> > Changed in the proposal, most of the contributors are salaried, and some
> of
> > them are willing to help in the spare time. Thanks
> > This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA in
> > Proposal. All work here is approved by NetEase and can be forward.
> > At last, you are asking about github.com/apache/incubator-kyuubi-shaded
> > repo,
> > what does this represent? I can only see you have only one source repo
> and
> > a doc repo. Removed this requirement in the proposal, and we will apply
> it
> > from IPMC if we do need this in the future.
> > ------------------------------
> >
> > Bests
> >
> > Kent Yao
> >
> > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> >
> > > Hi
> > >
> > > I read the contributor list and the proposal
> > > > The risk of the Kyuubi project being abandoned is minimal. Many
> > > organizations are using Kyuubi to build critical big data pipelines and
> > > willing to help develop Kyuubi's community if it becomes an ASF
> project.
> > >
> > > I think the team should evaluate the status again. You listed only 5
> core
> > > maintainers, but clearly only one has dominated contributions. I agree
> > with
> > > Atri, orphaned is measurable.
> > > It is better to adjust this statement a little. Also, with this fact,
> the
> > > core maintainers should notice whether this small group of people to
> make
> > > sure your features iteration, bug fix, and release process working by
> > > following ASF's requirements.
> > > This would not be an easy job.
> > >
> > > And your committer list is very strange
> > > ```
> > > Kent Yao (yao@apache.org) (PPMC)
> > > Wenchen Fan (wenchen@apache.org) (PPMC)
> > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > > Xun Liu (liuxun@apache.org) (PPMC)
> > > Fei Wang (cn.feiwang@gmail.com)
> > > Ulysses You (ulyssesyou18@gmail.com)
> > > Cheng Pan (pan3793@gmail.com)
> > > Hongxiang Jiang (jhx1008@gmail.com)
> > > Zen Jin(xinjingziranchan@gmail.com)
> > > ```
> > > 1. Why some of them are PPMC and others are committers? Where this
> > > hierarchy comes from?
> > > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy
> makes
> > > sense, most likely not)
> > > 3. I think only the top 4 have over 400 LOC, why you declared you have
> 5
> > > core developers?
> > >
> > > Another re-evaluation request is about `Reliance on Salaried
> Developers:`
> > > > They are all passionate about the project, and we are confident that
> > the
> > > project will continue even if no salaried developers contribute to the
> > > project.
> > >
> > > I think you don't have enough no-salaried developers, am I missing
> > > anything?
> > >
> > > About IP clearance, `Initial Source and Intellectual Property
> Submission
> > > Plan`
> > > > As soon as Kyuubi is approved to join Apache Incubator, our initial
> > > committers will submit iCLA(s) and CCLA(s). The codebase is already
> > > licensed under the Apache License 2.0.
> > >
> > > This project belongs to NetEase Inc. clearly, why NO SGA?
> > >
> > > At last, you are asking about `
> > github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> > <http://github.com/apache/incubator-kyuubi-shaded>
> > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > repo, what does this represent? I can only see you have only one source
> > > repo and a doc repo.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> > >
> > > > Thanks.
> > > >
> > > > I looked at the link you advised on and if I am reading it correctly,
> > top
> > > > committer has 630ish commits and the next committer has 73 commits?
> > > >
> > > > IMO this is an indication of the community not being active enough
> and
> > > > hence the risk of project getting orphaned is measurable.
> > > >
> > > > Atri
> > > >
> > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Atri Sharma,
> > > > >
> > > > > It's easier to check out the recent contribution of kyuubi from
> this
> > > > > link[1].
> > > > > We already identified the one person contribution issue of kyuubi
> > when
> > > > > Kent came to us and we kept reminding him to put more effort into
> > > > > building the community.
> > > > > Last week I had a long F2F conversation with Kent, I think he
> already
> > > > > knows about it and there have been some improvements in recent
> months
> > > > > but there is still a long way to go if Kyuubi is accepted as a
> > podling
> > > > > in the Apache Incubator.
> > > > >
> > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > > >
> > > > > Willem Jiang
> > > > >
> > > > > Twitter: willemjiang
> > > > > Weibo: 姜宁willem
> > > > >
> > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org>
> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I took a look at commit log and seems like all the commits for
> > last 3
> > > > > > months have been done by a single person.
> > > > > >
> > > > > > How active are the remaining committers?
> > > > > >
> > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > > > > >
> > > > > > > Hi Alexander Alten,
> > > > > > >
> > > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > > >
> > > > > > > Not being very familiar with Dremel, I just took a quick look
> at
> > > the
> > > > > > > profile of a very similar and exciting project, Apache Drill.
> If
> > I
> > > > > > > understand correctly, Kyuubi is built on top of the compute
> layer
> > > > > > > (Apache Spark), while Apache Drill is built on top of the
> storage
> > > > > > > layer.
> > > > > > >
> > > > > > > Apache Spark has a large user base and usage scenarios all over
> > the
> > > > > > > world. When users have some basic Spark programming and tuning
> > > > skills,
> > > > > > > things are easy for them. But many times, most users do not
> have
> > > all
> > > > > > > of these abilities when they use Spark directly or indirectly.
> We
> > > > want
> > > > > > > to build a more easy-to-use platform on top of it to help these
> > > > users.
> > > > > > > On the one hand, we will work on the interface layer to make it
> > > easy
> > > > > > > to access (of course, this effort will make our project look
> very
> > > > > > > similar to other projects, such as HiveServer2). On the other
> > hand,
> > > > we
> > > > > > > will work under Kyuubi's multi-tenant architecture to extend
> our
> > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more
> > > stable
> > > > > > > and efficient in our scenario.
> > > > > > >
> > > > > > > What's more, we are going to add more on-prem Kyuubi engine
> > types,
> > > > e.g.
> > > > > > > Flink.
> > > > > > >
> > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > > > >
> > > > > > > >
> > > > > > > > Hey,
> > > > > > > >
> > > > > > > > Looks like an interesting project, competition to Starburst.
> > How
> > > > > does it
> > > > > > > > compare to Apache Dremel?
> > > > > > > >
> > > > > > > > +1 (non-binding) for incubation.
> > > > > > > >
> > > > > > > > Thanks and stay safe,
> > > > > > > > --alex
> > > > > > > >
> > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > > > > > > >
> > > > > > > > > Dear all,
> > > > > > > > >
> > > > > > > > > We want to propose Kyuubi as a new Apache Incubator
> project.
> > > > > > > > >
> > > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC
> server
> > > for
> > > > > > > > > large-scale data management, processing, and analytics,
> built
> > > on
> > > > > top
> > > > > > > > > of Apache Spark and designed to support more engines (i.e.,
> > > > Apache
> > > > > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box"
> tool
> > > for
> > > > > data
> > > > > > > > > warehouses and data lakes. Kyuubi concentrates on improving
> > the
> > > > > > > > > experience for non-Spark users, making it easy for them to
> > use
> > > > > Spark
> > > > > > > > > as a service and focus on business data of their own, not
> > where
> > > > it
> > > > > > > > > stores, how it computes. NetEase has open-sourced Kyuubi
> > since
> > > > > 2018,
> > > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > > >
> > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo
> > Zhang,
> > > > > Akira
> > > > > > > > > Ajisaka for being the mentors of the Kyuubi project and
> > helping
> > > > us
> > > > > set
> > > > > > > > > up this proposal. The proposal can be found at
> > > > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> > .
> > > > > > > > >
> > > > > > > > > Looking forward to your feedback and thank you very much.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Kent Yao
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail:
> > > general-unsubscribe@incubator.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Kent Yao
> > > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
> Corp.
> > > > > > > a spark enthusiast
> > > > > > > kyuubiis a unified multi-tenant JDBC interface for large-scale
> > data
> > > > > > > processing and analytics, built on top of Apache Spark.
> > > > > > >
> > > > > > > spark-authorizerA Spark SQL extension which provides SQL
> Standard
> > > > > > > Authorization for Apache Spark.
> > > > > > > spark-postgres A library for reading data from and transferring
> > > data
> > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x
> > > > faster.
> > > > > > > itatchiA library that brings useful functions from various
> modern
> > > > > > > database management systems to Apache Spark.
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > *Kent Yao *
> > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > a spark enthusiast
> > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> JDBC
> > interface for large-scale data processing and analytics, built on top of
> > Apache
> > Spark <http://spark.apache.org/>.
> > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> SQL
> > extension which provides SQL Standard Authorization for Apache Spark
> > <http://spark.apache.org/>.
> > spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
> for
> > reading data from and transferring data to Postgres / Greenplum with
> Spark
> > SQL and DataFrames, 10~100x faster.
> > itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> > brings useful functions from various modern database management
> > systems to Apache
> > Spark <http://spark.apache.org/>.
> >
>
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Akira Ajisaka <aa...@apache.org>.
Hi Atri,
I've checked the contribution this year:
https://github.com/NetEase/kyuubi/graphs/contributors?from=2021-01-01&to=2021-05-31&type=c
According to this page, I don't think one contributor/company is
dominating. The number of active developers is 4, which is currently
low, but I think it is reasonable. During incubation, we can grow the
community and increase the number.
Thanks,
Akira
On Thu, Jun 3, 2021 at 12:52 PM Atri Sharma <at...@apache.org> wrote:
>
> I would like to hear from Kyuubi's mentors on the topic before reaching a
> conclusion.
>
> On Thu, 3 Jun 2021, 05:40 Sheng Wu, <wu...@gmail.com> wrote:
>
> > Kent
> >
> > Thanks.
> > The proposal is generally better this time.
> >
> > If your initial committers and your mentors agree, you are ready to embrace
> > the ASF culture, I am good with that.
> > Just from my experience, your community looks a little too young, like
> > Atri's concern, which is reasonable.
> > But I would like to trust you and your mentors' judgment and determination.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Kent Yao <ya...@apache.org> 于2021年6月3日周四 上午12:54写道：
> >
> > > Hi Sheng,
> > >
> > > Truly sorry for the format issue.
> > >
> > > Please trust me I did not mean to hurt anybody and push any contributor
> > > away. We drafted this proposal together based on the New Podling
> > Proposal.
> > > We never tried to use PPMC and committer to measure whose contribution is
> > > higher. This mistake has been fixed in the proposal.
> > >
> > > About releases, we will follow
> > > https://www.apache.org/legal/release-policy.html and the guide of our
> > > mentors. We will also take the initiative to learn other documents.
> > >
> > > > This is not a block, the point I mentioned this is, you will need more
> > > people.
> > >
> > > We will keep working on this.
> > >
> > > Bests,
> > >
> > > Kent Yao
> > >
> > >
> > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
> > >
> > > > Hi Kent
> > > >
> > > > Thanks for replying, but to be honest, your mail format is chaos.
> > > > Take a look at bellowing, which is what I saw in the gmail.
> > > >
> > > > This may be an issue of your mail client or some else.
> > > >
> > > > > The current #.2 - 5 top
> > > > > contributors will stay as committers, and they will join the PPMC
> > > during
> > > > > incubation.
> > > >
> > > > PPMC usually is the default for all initial committers, considering
> > your
> > > > community is still small and not very active.
> > > > You should invite all people you trusted to join, rather than pushing
> > > them
> > > > away.
> > > > This is not just a too-high bar issue. Excluding 4 out of 5 is strange.
> > > > I can't guess what is the root cause, why the other 4 PPMC members
> > don't
> > > > touch codes but are trusted completely. Meanwhile, people with easily
> > > > measurable code contributions are just as committers?
> > > > We(ASF) always remind the community to recognize no-code contribution,
> > > but
> > > > in this case, it seems the opposite? Don't recognize code contributors
> > > > enough?
> > > > Please reach your mentors, this seems not a typical ASF style. If you
> > > have
> > > > other people already involved in the ASF, asking them too.
> > > >
> > > > And from your description of the release process, I think you should
> > talk
> > > > more with your mentors.
> > > > Such as what is a qualified ASF release, what should be checked.
> > > > This is not a block, the point I mentioned this is, you will need more
> > > > people.
> > > >
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
> > > >
> > > > > Hi Sheng,
> > > > >
> > > > > Thanks for your detailed check and valuable comments. They are
> > > > > accommodating. I have addressed some of them directly in the
> > proposal.
> > > > > Please also check the table below for some Q&As.
> > > > > QA
> > > > > You listed only 5 core maintainers, but clearly only one has
> > dominated
> > > > > contributions. The distortion and skewness of the contributions are
> > > > mainly
> > > > > caused by the way how we merge PRs changed(mentioned in another email
> > > on
> > > > > the list).
> > > > > I agree with Atri, orphaned is measurable. It is better to adjust
> > this
> > > > > statement a little. Address the comment to notify the risk in the
> > > Kyuubi
> > > > > proposal.
> > > > > Also, with this fact, the core maintainers should notice whether this
> > > > small
> > > > > group of people to make sure your features iteration, bug fix, and
> > > > release
> > > > > process working by following ASF's requirements.This would not be an
> > > easy
> > > > > job. Currently, we learn the release process from some Apache
> > projects,
> > > > > using major.minor.patch for versioning. As we can see, since Kyuubi
> > > 1.0,
> > > > it
> > > > > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes
> > > > until
> > > > > EOL. And it takes us about two months to finish a minor release with
> > 3
> > > ~
> > > > 5
> > > > > big improvements and various small features/bug fixes. We also will
> > > start
> > > > > with some release candidates for preview and test before the final.
> > > This
> > > > is
> > > > > how we develop and release Kyuubi for now, but we do have a long way
> > to
> > > > go
> > > > > and learn to match the ASF requirement.
> > > > > And your committer list is very strange... 1. Why some of them are
> > PPMC
> > > > and
> > > > > others are committers? Where this hierarchy comes from? The current
> > > PPMC
> > > > > candidates help us a lot build the Kyuubi project and community, not
> > > > > directly to the codebase but as mentors(before Apache). Most of them
> > > are
> > > > > from Apache mature TLPs, which have close relationships with Kyuubi.
> > > They
> > > > > have sufficient Apache Way experience to guide us to become a more
> > > > > Apache-like project before submitting this proposal in many ways. We
> > > will
> > > > > continue to promote more contributors to the PPMC/committer based on
> > > > their
> > > > > code or non-code contributions during incubation.
> > > > > 2. Your core maintainers have 5, which only 4 PPMC? The current #.2
> > - 5
> > > > top
> > > > > contributors will stay as committers, and they will join the PPMC
> > > during
> > > > > incubation.
> > > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> > have
> > > 5
> > > > > core developers? I didn't notice that 400 LOC is a bar for core
> > > > developer,
> > > > > will change 5 to 4 core developers accordingly
> > > > > I think you don't have enough no-salaried developers, am I missing
> > > > > anything? Yes.
> > > > > Changed in the proposal, most of the contributors are salaried, and
> > > some
> > > > of
> > > > > them are willing to help in the spare time. Thanks
> > > > > This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA
> > in
> > > > > Proposal. All work here is approved by NetEase and can be forward.
> > > > > At last, you are asking about
> > > github.com/apache/incubator-kyuubi-shaded
> > > > > repo,
> > > > > what does this represent? I can only see you have only one source
> > repo
> > > > and
> > > > > a doc repo. Removed this requirement in the proposal, and we will
> > apply
> > > > it
> > > > > from IPMC if we do need this in the future.
> > > > > ------------------------------
> > > > >
> > > > > Bests
> > > > >
> > > > > Kent Yao
> > > > >
> > > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > I read the contributor list and the proposal
> > > > > > > The risk of the Kyuubi project being abandoned is minimal. Many
> > > > > > organizations are using Kyuubi to build critical big data pipelines
> > > and
> > > > > > willing to help develop Kyuubi's community if it becomes an ASF
> > > > project.
> > > > > >
> > > > > > I think the team should evaluate the status again. You listed only
> > 5
> > > > core
> > > > > > maintainers, but clearly only one has dominated contributions. I
> > > agree
> > > > > with
> > > > > > Atri, orphaned is measurable.
> > > > > > It is better to adjust this statement a little. Also, with this
> > fact,
> > > > the
> > > > > > core maintainers should notice whether this small group of people
> > to
> > > > make
> > > > > > sure your features iteration, bug fix, and release process working
> > by
> > > > > > following ASF's requirements.
> > > > > > This would not be an easy job.
> > > > > >
> > > > > > And your committer list is very strange
> > > > > > ```
> > > > > > Kent Yao (yao@apache.org) (PPMC)
> > > > > > Wenchen Fan (wenchen@apache.org) (PPMC)
> > > > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > > > > > Xun Liu (liuxun@apache.org) (PPMC)
> > > > > > Fei Wang (cn.feiwang@gmail.com)
> > > > > > Ulysses You (ulyssesyou18@gmail.com)
> > > > > > Cheng Pan (pan3793@gmail.com)
> > > > > > Hongxiang Jiang (jhx1008@gmail.com)
> > > > > > Zen Jin(xinjingziranchan@gmail.com)
> > > > > > ```
> > > > > > 1. Why some of them are PPMC and others are committers? Where this
> > > > > > hierarchy comes from?
> > > > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy
> > > > makes
> > > > > > sense, most likely not)
> > > > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> > > have
> > > > 5
> > > > > > core developers?
> > > > > >
> > > > > > Another re-evaluation request is about `Reliance on Salaried
> > > > Developers:`
> > > > > > > They are all passionate about the project, and we are confident
> > > that
> > > > > the
> > > > > > project will continue even if no salaried developers contribute to
> > > the
> > > > > > project.
> > > > > >
> > > > > > I think you don't have enough no-salaried developers, am I missing
> > > > > > anything?
> > > > > >
> > > > > > About IP clearance, `Initial Source and Intellectual Property
> > > > Submission
> > > > > > Plan`
> > > > > > > As soon as Kyuubi is approved to join Apache Incubator, our
> > initial
> > > > > > committers will submit iCLA(s) and CCLA(s). The codebase is already
> > > > > > licensed under the Apache License 2.0.
> > > > > >
> > > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
> > > > > >
> > > > > > At last, you are asking about `
> > > > > github.com/apache/incubator-kyuubi-shaded`
> > <http://github.com/apache/incubator-kyuubi-shaded>
> > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > > repo, what does this represent? I can only see you have only one
> > > source
> > > > > > repo and a doc repo.
> > > > > >
> > > > > > Sheng Wu 吴晟
> > > > > > Twitter, wusheng1108
> > > > > >
> > > > > >
> > > > > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > I looked at the link you advised on and if I am reading it
> > > correctly,
> > > > > top
> > > > > > > committer has 630ish commits and the next committer has 73
> > commits?
> > > > > > >
> > > > > > > IMO this is an indication of the community not being active
> > enough
> > > > and
> > > > > > > hence the risk of project getting orphaned is measurable.
> > > > > > >
> > > > > > > Atri
> > > > > > >
> > > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Atri Sharma,
> > > > > > > >
> > > > > > > > It's easier to check out the recent contribution of kyuubi from
> > > > this
> > > > > > > > link[1].
> > > > > > > > We already identified the one person contribution issue of
> > kyuubi
> > > > > when
> > > > > > > > Kent came to us and we kept reminding him to put more effort
> > into
> > > > > > > > building the community.
> > > > > > > > Last week I had a long F2F conversation with Kent, I think he
> > > > already
> > > > > > > > knows about it and there have been some improvements in recent
> > > > months
> > > > > > > > but there is still a long way to go if Kyuubi is accepted as a
> > > > > podling
> > > > > > > > in the Apache Incubator.
> > > > > > > >
> > > > > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > > > > > >
> > > > > > > > Willem Jiang
> > > > > > > >
> > > > > > > > Twitter: willemjiang
> > > > > > > > Weibo: 姜宁willem
> > > > > > > >
> > > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I took a look at commit log and seems like all the commits
> > for
> > > > > last 3
> > > > > > > > > months have been done by a single person.
> > > > > > > > >
> > > > > > > > > How active are the remaining committers?
> > > > > > > > >
> > > > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Alexander Alten,
> > > > > > > > > >
> > > > > > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > > > > > >
> > > > > > > > > > Not being very familiar with Dremel, I just took a quick
> > look
> > > > at
> > > > > > the
> > > > > > > > > > profile of a very similar and exciting project, Apache
> > Drill.
> > > > If
> > > > > I
> > > > > > > > > > understand correctly, Kyuubi is built on top of the compute
> > > > layer
> > > > > > > > > > (Apache Spark), while Apache Drill is built on top of the
> > > > storage
> > > > > > > > > > layer.
> > > > > > > > > >
> > > > > > > > > > Apache Spark has a large user base and usage scenarios all
> > > over
> > > > > the
> > > > > > > > > > world. When users have some basic Spark programming and
> > > tuning
> > > > > > > skills,
> > > > > > > > > > things are easy for them. But many times, most users do not
> > > > have
> > > > > > all
> > > > > > > > > > of these abilities when they use Spark directly or
> > > indirectly.
> > > > We
> > > > > > > want
> > > > > > > > > > to build a more easy-to-use platform on top of it to help
> > > these
> > > > > > > users.
> > > > > > > > > > On the one hand, we will work on the interface layer to
> > make
> > > it
> > > > > > easy
> > > > > > > > > > to access (of course, this effort will make our project
> > look
> > > > very
> > > > > > > > > > similar to other projects, such as HiveServer2). On the
> > other
> > > > > hand,
> > > > > > > we
> > > > > > > > > > will work under Kyuubi's multi-tenant architecture to
> > extend
> > > > our
> > > > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it
> > more
> > > > > > stable
> > > > > > > > > > and efficient in our scenario.
> > > > > > > > > >
> > > > > > > > > > What's more, we are going to add more on-prem Kyuubi engine
> > > > > types,
> > > > > > > e.g.
> > > > > > > > > > Flink.
> > > > > > > > > >
> > > > > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hey,
> > > > > > > > > > >
> > > > > > > > > > > Looks like an interesting project, competition to
> > > Starburst.
> > > > > How
> > > > > > > > does it
> > > > > > > > > > > compare to Apache Dremel?
> > > > > > > > > > >
> > > > > > > > > > > +1 (non-binding) for incubation.
> > > > > > > > > > >
> > > > > > > > > > > Thanks and stay safe,
> > > > > > > > > > > --alex
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org>
> > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Dear all,
> > > > > > > > > > > >
> > > > > > > > > > > > We want to propose Kyuubi as a new Apache Incubator
> > > > project.
> > > > > > > > > > > >
> > > > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC
> > > > server
> > > > > > for
> > > > > > > > > > > > large-scale data management, processing, and analytics,
> > > > built
> > > > > > on
> > > > > > > > top
> > > > > > > > > > > > of Apache Spark and designed to support more engines
> > > (i.e.,
> > > > > > > Apache
> > > > > > > > > > > > Flink). We are aiming to make Kyuubi an
> > "out-of-the-box"
> > > > tool
> > > > > > for
> > > > > > > > data
> > > > > > > > > > > > warehouses and data lakes. Kyuubi concentrates on
> > > improving
> > > > > the
> > > > > > > > > > > > experience for non-Spark users, making it easy for them
> > > to
> > > > > use
> > > > > > > > Spark
> > > > > > > > > > > > as a service and focus on business data of their own,
> > not
> > > > > where
> > > > > > > it
> > > > > > > > > > > > stores, how it computes. NetEase has open-sourced
> > Kyuubi
> > > > > since
> > > > > > > > 2018,
> > > > > > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > > > > > >
> > > > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo
> > > > > Zhang,
> > > > > > > > Akira
> > > > > > > > > > > > Ajisaka for being the mentors of the Kyuubi project and
> > > > > helping
> > > > > > > us
> > > > > > > > set
> > > > > > > > > > > > up this proposal. The proposal can be found at
> > > > > > > > > > > >
> > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> > > > > .
> > > > > > > > > > > >
> > > > > > > > > > > > Looking forward to your feedback and thank you very
> > much.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Kent Yao
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > general-unsubscribe@incubator.apache.org
> > > > > > > > > > > > For additional commands, e-mail:
> > > > > > > general-help@incubator.apache.org
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Kent Yao
> > > > > > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
> > > > Corp.
> > > > > > > > > > a spark enthusiast
> > > > > > > > > > kyuubiis a unified multi-tenant JDBC interface for
> > > large-scale
> > > > > data
> > > > > > > > > > processing and analytics, built on top of Apache Spark.
> > > > > > > > > >
> > > > > > > > > > spark-authorizerA Spark SQL extension which provides SQL
> > > > Standard
> > > > > > > > > > Authorization for Apache Spark.
> > > > > > > > > > spark-postgres A library for reading data from and
> > > transferring
> > > > > > data
> > > > > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames,
> > > 10~100x
> > > > > > > faster.
> > > > > > > > > > itatchiA library that brings useful functions from various
> > > > modern
> > > > > > > > > > database management systems to Apache Spark.
> > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail:
> > > > general-unsubscribe@incubator.apache.org
> > > > > > > > > > For additional commands, e-mail:
> > > > > general-help@incubator.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail:
> > general-unsubscribe@incubator.apache.org
> > > > > > > > For additional commands, e-mail:
> > > general-help@incubator.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Kent Yao *
> > > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > > a spark enthusiast
> > > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> > > > JDBC
> > > > > interface for large-scale data processing and analytics, built on top
> > > of
> > > > > Apache
> > > > > Spark <http://spark.apache.org/>.
> > > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
> > Spark
> > > > SQL
> > > > > extension which provides SQL Standard Authorization for Apache Spark
> > > > > <http://spark.apache.org/>.
> > > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
> > library
> > > > for
> > > > > reading data from and transferring data to Postgres / Greenplum with
> > > > Spark
> > > > > SQL and DataFrames, 10~100x faster.
> > > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library
> > that
> > > > > brings useful functions from various modern database management
> > > > > systems to Apache
> > > > > Spark <http://spark.apache.org/>.
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > *Kent Yao *
> > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > a spark enthusiast
> > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> > JDBC
> > > interface for large-scale data processing and analytics, built on top of
> > > Apache
> > > Spark <http://spark.apache.org/>.
> > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> > SQL
> > > extension which provides SQL Standard Authorization for Apache Spark
> > > <http://spark.apache.org/>.
> > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
> > for
> > > reading data from and transferring data to Postgres / Greenplum with
> > Spark
> > > SQL and DataFrames, 10~100x faster.
> > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> > > brings useful functions from various modern database management
> > > systems to Apache
> > > Spark <http://spark.apache.org/>.
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Hello everyone,
It's been 12 days since this discussion started on May 31, 2021. After June
6, there are no more new questions arising. So I'd like to summarize the
results of this discussion so we can move forward.
Thank you very much to everyone at the Apache IPMC for suggesting changes
to our proposal and everyone else who participated in the discussion. We
have corrected some sections and content of the proposal during this
discussion, such as Orphaned products, Core Developers, Initial Committers,
SGA, etc., based on your recommendations.
In addition, we had a discussion with our mentors and all former candidates
about the composition of Initial Committers. Based on the Apache Way,
considering the current status of the Kyuubi community, the contribution of
each contributor, the current Initial Committers were adjusted as follows
with the consensus of everyone.
Kent Yao (yao@apache.org)
Fei Wang (cn.feiwang@gmail.com)
Ulysses You (ulyssesyou18@gmail.com)
Cheng Pan (pan3793@gmail.com)
Hongxiang Jiang (jhx1008@gmail.com)
Thank you all again.
Bests
Kent Yao
Kent Yao <ya...@apache.org> 于2021年6月6日周日 下午10:44写道：
> Hi Ming,
>
> Thanks for your question.
>
> As mentioned in the proposal, Kyuubi was first developed and applied in
> NetEase in 2018. The early scenario we wanted to address was
> straightforward - to provide data analysts with a Spark-based analytics
> platform. This platform needed to support multi-tenancy [1] and
> authentication and authorization [2]. From the project's perspective, it
> was just an alternative implementation of Apache Spark Thrift Server. And
> for the workloads and users that it serves are not very big in NetEase. So
> at that time, I was the only one who was maintaining the project.
>
> Another reason for this problem should be the limited usage scenario
> caused by its architecture. It was implemented based on
> *spark.driver.allowMultipleContexts*, which was not good judging from the
> dismal feedback after we open-sourced it. Many people have found Kyuubi or
> me through GitHub, hoping that Kyuubi could solve a problem they were
> having in their scenario. From all of these things, I remember most and
> feel sorry for is when they said they were using Kyuubi to help medical
> staff fight COVID-19 [9]. Limited by design, there was very little that
> could be done at the time.
>
> Starting with Spark 3.0.0, the Spark community also removed this feature
> [3] for Kyuubi's old architecture. This allowed us to rethink and redesign
> Kyuubi architecture. A prototype was quickly designed, but we did not start
> it in a hurry. As Kyuubi relies heavily on Spark, we decided to actively
> and deeply participate in the Spark community beginning around September
> 2019 [4][5][6], both to help the Spark community and to verify that the
> design of the new Kyuubi architecture [7] was future-proofing enough. In
> the process of implementing the new architecture, Zen Jin became a
> contributor to the project and contributed several great ideas, which made
> Kyuubi's architecture and the scenarios it can cover more general and
> comprehensive. This has made Kyuubi's architecture and the scenarios it can
> cover more general. This advancement has made Kyuubi more attractive than
> older architecture projects.
>
> As a result, many like-minded people [6] started to gather around the
> project and the new architecture, and some of them have already become our
> contributors. There are also many people who have encouraged us to
> contribute to this project to Apache, which will be very beneficial for
> them to be more directly involved in the project.
>
> In addition, we've learned a lot about community operations while
> contributing to other Apache projects like Spark, Submarine, and Hudi. Some
> of these communities are new to Apache, and some are ten years out of
> school. We also actively participated in some online and offline
> conferences or meetups, published technical articles in different media to
> promote Kyuubi.
>
> From our practical experience, we have learned that promoting a project is
> not an easy task. We want to donate Kyuubi to the community and bring in
> potentially willing people to help the Kyuubi community grow.
>
> Bests
> Kent Yao
>
> [1] https://issues.apache.org/jira/browse/SPARK-2243
> [2] https://issues.apache.org/jira/browse/SPARK-5159
> [3] https://issues.apache.org/jira/browse/SPARK-26362
> [4] https://github.com/apache/spark/commits?author=yaooqinn
> [5] https://github.com/apache/spark/commits?author=turboFei
> [6] https://github.com/apache/spark/commits?author=ulysses-you
> [7] https://github.com/NetEase/kyuubi/releases/tag/v1.0.0
> [8]
> https://github.com/NetEase/kyuubi/graphs/contributors?from=2021-01-01&to=2021-06-06&type=c
> [9] https://github.com/NetEase/kyuubi/issues/228
>
> Ming Wen <we...@apache.org> 于2021年6月6日周日 上午7:37写道：
>
>> You can see the trend chart from
>>
>> https://www.apiseven.com/zh/contributor-graph?chart=contributorMonthlyActivity&repo=NetEase/kyuubi
>>
>>
>> Ming Wen <we...@apache.org>于2021年6月6日 周日上午7:28写道：
>>
>> > hi, Kent,
>> >
>> > From 2018 to 2020, kuyybi is not an active project, most of the time
>> there
>> > is only one contributor per month. But in 2021, there will be 8 active
>> > contributors every month. What is the reason for this?
>> >
>> > Willem Jiang <wi...@gmail.com>于2021年6月3日 周四下午12:53写道：
>> >
>> >> I think we can revisit the PPMC part and have more discussion with the
>> >> initial committers.
>> >> It's a good chance to let them know better about how to run an open
>> >> source project in Apache Way.
>> >>
>> >>
>> >> Willem Jiang
>> >>
>> >> Twitter: willemjiang
>> >> Weibo: 姜宁willem
>> >>
>> >> On Thu, Jun 3, 2021 at 11:52 AM Atri Sharma <at...@apache.org> wrote:
>> >> >
>> >> > I would like to hear from Kyuubi's mentors on the topic before
>> reaching
>> >> a
>> >> > conclusion.
>> >> >
>> >> > On Thu, 3 Jun 2021, 05:40 Sheng Wu, <wu...@gmail.com>
>> wrote:
>> >> >
>> >> > > Kent
>> >> > >
>> >> > > Thanks.
>> >> > > The proposal is generally better this time.
>> >> > >
>> >> > > If your initial committers and your mentors agree, you are ready to
>> >> embrace
>> >> > > the ASF culture, I am good with that.
>> >> > > Just from my experience, your community looks a little too young,
>> like
>> >> > > Atri's concern, which is reasonable.
>> >> > > But I would like to trust you and your mentors' judgment and
>> >> determination.
>> >> > >
>> >> > > Sheng Wu 吴晟
>> >> > > Twitter, wusheng1108
>> >> > >
>> >> > >
>> >> > > Kent Yao <ya...@apache.org> 于2021年6月3日周四 上午12:54写道：
>> >> > >
>> >> > > > Hi Sheng,
>> >> > > >
>> >> > > > Truly sorry for the format issue.
>> >> > > >
>> >> > > > Please trust me I did not mean to hurt anybody and push any
>> >> contributor
>> >> > > > away. We drafted this proposal together based on the New Podling
>> >> > > Proposal.
>> >> > > > We never tried to use PPMC and committer to measure whose
>> >> contribution is
>> >> > > > higher. This mistake has been fixed in the proposal.
>> >> > > >
>> >> > > > About releases, we will follow
>> >> > > > https://www.apache.org/legal/release-policy.html and the guide
>> of
>> >> our
>> >> > > > mentors. We will also take the initiative to learn other
>> documents.
>> >> > > >
>> >> > > > > This is not a block, the point I mentioned this is, you will
>> need
>> >> more
>> >> > > > people.
>> >> > > >
>> >> > > > We will keep working on this.
>> >> > > >
>> >> > > > Bests,
>> >> > > >
>> >> > > > Kent Yao
>> >> > > >
>> >> > > >
>> >> > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
>> >> > > >
>> >> > > > > Hi Kent
>> >> > > > >
>> >> > > > > Thanks for replying, but to be honest, your mail format is
>> chaos.
>> >> > > > > Take a look at bellowing, which is what I saw in the gmail.
>> >> > > > >
>> >> > > > > This may be an issue of your mail client or some else.
>> >> > > > >
>> >> > > > > > The current #.2 - 5 top
>> >> > > > > > contributors will stay as committers, and they will join the
>> >> PPMC
>> >> > > > during
>> >> > > > > > incubation.
>> >> > > > >
>> >> > > > > PPMC usually is the default for all initial committers,
>> >> considering
>> >> > > your
>> >> > > > > community is still small and not very active.
>> >> > > > > You should invite all people you trusted to join, rather than
>> >> pushing
>> >> > > > them
>> >> > > > > away.
>> >> > > > > This is not just a too-high bar issue. Excluding 4 out of 5 is
>> >> strange.
>> >> > > > > I can't guess what is the root cause, why the other 4 PPMC
>> members
>> >> > > don't
>> >> > > > > touch codes but are trusted completely. Meanwhile, people with
>> >> easily
>> >> > > > > measurable code contributions are just as committers?
>> >> > > > > We(ASF) always remind the community to recognize no-code
>> >> contribution,
>> >> > > > but
>> >> > > > > in this case, it seems the opposite? Don't recognize code
>> >> contributors
>> >> > > > > enough?
>> >> > > > > Please reach your mentors, this seems not a typical ASF style.
>> If
>> >> you
>> >> > > > have
>> >> > > > > other people already involved in the ASF, asking them too.
>> >> > > > >
>> >> > > > > And from your description of the release process, I think you
>> >> should
>> >> > > talk
>> >> > > > > more with your mentors.
>> >> > > > > Such as what is a qualified ASF release, what should be
>> checked.
>> >> > > > > This is not a block, the point I mentioned this is, you will
>> need
>> >> more
>> >> > > > > people.
>> >> > > > >
>> >> > > > >
>> >> > > > > Sheng Wu 吴晟
>> >> > > > > Twitter, wusheng1108
>> >> > > > >
>> >> > > > >
>> >> > > > > Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
>> >> > > > >
>> >> > > > > > Hi Sheng,
>> >> > > > > >
>> >> > > > > > Thanks for your detailed check and valuable comments. They
>> are
>> >> > > > > > accommodating. I have addressed some of them directly in the
>> >> > > proposal.
>> >> > > > > > Please also check the table below for some Q&As.
>> >> > > > > > QA
>> >> > > > > > You listed only 5 core maintainers, but clearly only one has
>> >> > > dominated
>> >> > > > > > contributions. The distortion and skewness of the
>> contributions
>> >> are
>> >> > > > > mainly
>> >> > > > > > caused by the way how we merge PRs changed(mentioned in
>> another
>> >> email
>> >> > > > on
>> >> > > > > > the list).
>> >> > > > > > I agree with Atri, orphaned is measurable. It is better to
>> >> adjust
>> >> > > this
>> >> > > > > > statement a little. Address the comment to notify the risk in
>> >> the
>> >> > > > Kyuubi
>> >> > > > > > proposal.
>> >> > > > > > Also, with this fact, the core maintainers should notice
>> >> whether this
>> >> > > > > small
>> >> > > > > > group of people to make sure your features iteration, bug
>> fix,
>> >> and
>> >> > > > > release
>> >> > > > > > process working by following ASF's requirements.This would
>> not
>> >> be an
>> >> > > > easy
>> >> > > > > > job. Currently, we learn the release process from some Apache
>> >> > > projects,
>> >> > > > > > using major.minor.patch for versioning. As we can see, since
>> >> Kyuubi
>> >> > > > 1.0,
>> >> > > > > it
>> >> > > > > > takes us 2 ~ 3 weeks for a patch release that contains 5-10
>> bug
>> >> fixes
>> >> > > > > until
>> >> > > > > > EOL. And it takes us about two months to finish a minor
>> release
>> >> with
>> >> > > 3
>> >> > > > ~
>> >> > > > > 5
>> >> > > > > > big improvements and various small features/bug fixes. We
>> also
>> >> will
>> >> > > > start
>> >> > > > > > with some release candidates for preview and test before the
>> >> final.
>> >> > > > This
>> >> > > > > is
>> >> > > > > > how we develop and release Kyuubi for now, but we do have a
>> >> long way
>> >> > > to
>> >> > > > > go
>> >> > > > > > and learn to match the ASF requirement.
>> >> > > > > > And your committer list is very strange... 1. Why some of
>> them
>> >> are
>> >> > > PPMC
>> >> > > > > and
>> >> > > > > > others are committers? Where this hierarchy comes from? The
>> >> current
>> >> > > > PPMC
>> >> > > > > > candidates help us a lot build the Kyuubi project and
>> >> community, not
>> >> > > > > > directly to the codebase but as mentors(before Apache). Most
>> of
>> >> them
>> >> > > > are
>> >> > > > > > from Apache mature TLPs, which have close relationships with
>> >> Kyuubi.
>> >> > > > They
>> >> > > > > > have sufficient Apache Way experience to guide us to become a
>> >> more
>> >> > > > > > Apache-like project before submitting this proposal in many
>> >> ways. We
>> >> > > > will
>> >> > > > > > continue to promote more contributors to the PPMC/committer
>> >> based on
>> >> > > > > their
>> >> > > > > > code or non-code contributions during incubation.
>> >> > > > > > 2. Your core maintainers have 5, which only 4 PPMC? The
>> current
>> >> #.2
>> >> > > - 5
>> >> > > > > top
>> >> > > > > > contributors will stay as committers, and they will join the
>> >> PPMC
>> >> > > > during
>> >> > > > > > incubation.
>> >> > > > > > 3. I think only the top 4 have over 400 LOC, why you declared
>> >> you
>> >> > > have
>> >> > > > 5
>> >> > > > > > core developers? I didn't notice that 400 LOC is a bar for
>> core
>> >> > > > > developer,
>> >> > > > > > will change 5 to 4 core developers accordingly
>> >> > > > > > I think you don't have enough no-salaried developers, am I
>> >> missing
>> >> > > > > > anything? Yes.
>> >> > > > > > Changed in the proposal, most of the contributors are
>> salaried,
>> >> and
>> >> > > > some
>> >> > > > > of
>> >> > > > > > them are willing to help in the spare time. Thanks
>> >> > > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
>> Added
>> >> SGA
>> >> > > in
>> >> > > > > > Proposal. All work here is approved by NetEase and can be
>> >> forward.
>> >> > > > > > At last, you are asking about
>> >> > > > github.com/apache/incubator-kyuubi-shaded
>> >> > > > > > repo,
>> >> > > > > > what does this represent? I can only see you have only one
>> >> source
>> >> > > repo
>> >> > > > > and
>> >> > > > > > a doc repo. Removed this requirement in the proposal, and we
>> >> will
>> >> > > apply
>> >> > > > > it
>> >> > > > > > from IPMC if we do need this in the future.
>> >> > > > > > ------------------------------
>> >> > > > > >
>> >> > > > > > Bests
>> >> > > > > >
>> >> > > > > > Kent Yao
>> >> > > > > >
>> >> > > > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
>> >> > > > > >
>> >> > > > > > > Hi
>> >> > > > > > >
>> >> > > > > > > I read the contributor list and the proposal
>> >> > > > > > > > The risk of the Kyuubi project being abandoned is
>> minimal.
>> >> Many
>> >> > > > > > > organizations are using Kyuubi to build critical big data
>> >> pipelines
>> >> > > > and
>> >> > > > > > > willing to help develop Kyuubi's community if it becomes an
>> >> ASF
>> >> > > > > project.
>> >> > > > > > >
>> >> > > > > > > I think the team should evaluate the status again. You
>> listed
>> >> only
>> >> > > 5
>> >> > > > > core
>> >> > > > > > > maintainers, but clearly only one has dominated
>> >> contributions. I
>> >> > > > agree
>> >> > > > > > with
>> >> > > > > > > Atri, orphaned is measurable.
>> >> > > > > > > It is better to adjust this statement a little. Also, with
>> >> this
>> >> > > fact,
>> >> > > > > the
>> >> > > > > > > core maintainers should notice whether this small group of
>> >> people
>> >> > > to
>> >> > > > > make
>> >> > > > > > > sure your features iteration, bug fix, and release process
>> >> working
>> >> > > by
>> >> > > > > > > following ASF's requirements.
>> >> > > > > > > This would not be an easy job.
>> >> > > > > > >
>> >> > > > > > > And your committer list is very strange
>> >> > > > > > > ```
>> >> > > > > > > Kent Yao (yao@apache.org) (PPMC)
>> >> > > > > > > Wenchen Fan (wenchen@apache.org) (PPMC)
>> >> > > > > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
>> >> > > > > > > Xun Liu (liuxun@apache.org) (PPMC)
>> >> > > > > > > Fei Wang (cn.feiwang@gmail.com)
>> >> > > > > > > Ulysses You (ulyssesyou18@gmail.com)
>> >> > > > > > > Cheng Pan (pan3793@gmail.com)
>> >> > > > > > > Hongxiang Jiang (jhx1008@gmail.com)
>> >> > > > > > > Zen Jin(xinjingziranchan@gmail.com)
>> >> > > > > > > ```
>> >> > > > > > > 1. Why some of them are PPMC and others are committers?
>> Where
>> >> this
>> >> > > > > > > hierarchy comes from?
>> >> > > > > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even
>> >> hierarchy
>> >> > > > > makes
>> >> > > > > > > sense, most likely not)
>> >> > > > > > > 3. I think only the top 4 have over 400 LOC, why you
>> declared
>> >> you
>> >> > > > have
>> >> > > > > 5
>> >> > > > > > > core developers?
>> >> > > > > > >
>> >> > > > > > > Another re-evaluation request is about `Reliance on
>> Salaried
>> >> > > > > Developers:`
>> >> > > > > > > > They are all passionate about the project, and we are
>> >> confident
>> >> > > > that
>> >> > > > > > the
>> >> > > > > > > project will continue even if no salaried developers
>> >> contribute to
>> >> > > > the
>> >> > > > > > > project.
>> >> > > > > > >
>> >> > > > > > > I think you don't have enough no-salaried developers, am I
>> >> missing
>> >> > > > > > > anything?
>> >> > > > > > >
>> >> > > > > > > About IP clearance, `Initial Source and Intellectual
>> Property
>> >> > > > > Submission
>> >> > > > > > > Plan`
>> >> > > > > > > > As soon as Kyuubi is approved to join Apache Incubator,
>> our
>> >> > > initial
>> >> > > > > > > committers will submit iCLA(s) and CCLA(s). The codebase is
>> >> already
>> >> > > > > > > licensed under the Apache License 2.0.
>> >> > > > > > >
>> >> > > > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
>> >> > > > > > >
>> >> > > > > > > At last, you are asking about `
>> >> > > > > > github.com/apache/incubator-kyuubi-shaded`
>> <http://github.com/apache/incubator-kyuubi-shaded>
>> >> <http://github.com/apache/incubator-kyuubi-shaded>
>> >> > > <http://github.com/apache/incubator-kyuubi-shaded>
>> >> > > > <http://github.com/apache/incubator-kyuubi-shaded>
>> >> > > > > <http://github.com/apache/incubator-kyuubi-shaded>
>> >> > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
>> >> > > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
>> >> > > > > > > repo, what does this represent? I can only see you have
>> only
>> >> one
>> >> > > > source
>> >> > > > > > > repo and a doc repo.
>> >> > > > > > >
>> >> > > > > > > Sheng Wu 吴晟
>> >> > > > > > > Twitter, wusheng1108
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
>> >> > > > > > >
>> >> > > > > > > > Thanks.
>> >> > > > > > > >
>> >> > > > > > > > I looked at the link you advised on and if I am reading
>> it
>> >> > > > correctly,
>> >> > > > > > top
>> >> > > > > > > > committer has 630ish commits and the next committer has
>> 73
>> >> > > commits?
>> >> > > > > > > >
>> >> > > > > > > > IMO this is an indication of the community not being
>> active
>> >> > > enough
>> >> > > > > and
>> >> > > > > > > > hence the risk of project getting orphaned is measurable.
>> >> > > > > > > >
>> >> > > > > > > > Atri
>> >> > > > > > > >
>> >> > > > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <
>> >> willem.jiang@gmail.com>
>> >> > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > Hi Atri Sharma,
>> >> > > > > > > > >
>> >> > > > > > > > > It's easier to check out the recent contribution of
>> >> kyuubi from
>> >> > > > > this
>> >> > > > > > > > > link[1].
>> >> > > > > > > > > We already identified the one person contribution
>> issue of
>> >> > > kyuubi
>> >> > > > > > when
>> >> > > > > > > > > Kent came to us and we kept reminding him to put more
>> >> effort
>> >> > > into
>> >> > > > > > > > > building the community.
>> >> > > > > > > > > Last week I had a long F2F conversation with Kent, I
>> >> think he
>> >> > > > > already
>> >> > > > > > > > > knows about it and there have been some improvements in
>> >> recent
>> >> > > > > months
>> >> > > > > > > > > but there is still a long way to go if Kyuubi is
>> accepted
>> >> as a
>> >> > > > > > podling
>> >> > > > > > > > > in the Apache Incubator.
>> >> > > > > > > > >
>> >> > > > > > > > > [1]
>> https://github.com/NetEase/kyuubi/graphs/contributors
>> >> > > > > > > > >
>> >> > > > > > > > > Willem Jiang
>> >> > > > > > > > >
>> >> > > > > > > > > Twitter: willemjiang
>> >> > > > > > > > > Weibo: 姜宁willem
>> >> > > > > > > > >
>> >> > > > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <
>> >> atri@apache.org>
>> >> > > > > wrote:
>> >> > > > > > > > > >
>> >> > > > > > > > > > Hi,
>> >> > > > > > > > > >
>> >> > > > > > > > > > I took a look at commit log and seems like all the
>> >> commits
>> >> > > for
>> >> > > > > > last 3
>> >> > > > > > > > > > months have been done by a single person.
>> >> > > > > > > > > >
>> >> > > > > > > > > > How active are the remaining committers?
>> >> > > > > > > > > >
>> >> > > > > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <yao@apache.org
>> >
>> >> wrote:
>> >> > > > > > > > > >
>> >> > > > > > > > > > > Hi Alexander Alten,
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > Thanks for your interest and +1 for Kyuubi.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > Not being very familiar with Dremel, I just took a
>> >> quick
>> >> > > look
>> >> > > > > at
>> >> > > > > > > the
>> >> > > > > > > > > > > profile of a very similar and exciting project,
>> Apache
>> >> > > Drill.
>> >> > > > > If
>> >> > > > > > I
>> >> > > > > > > > > > > understand correctly, Kyuubi is built on top of the
>> >> compute
>> >> > > > > layer
>> >> > > > > > > > > > > (Apache Spark), while Apache Drill is built on top
>> of
>> >> the
>> >> > > > > storage
>> >> > > > > > > > > > > layer.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > Apache Spark has a large user base and usage
>> >> scenarios all
>> >> > > > over
>> >> > > > > > the
>> >> > > > > > > > > > > world. When users have some basic Spark programming
>> >> and
>> >> > > > tuning
>> >> > > > > > > > skills,
>> >> > > > > > > > > > > things are easy for them. But many times, most
>> users
>> >> do not
>> >> > > > > have
>> >> > > > > > > all
>> >> > > > > > > > > > > of these abilities when they use Spark directly or
>> >> > > > indirectly.
>> >> > > > > We
>> >> > > > > > > > want
>> >> > > > > > > > > > > to build a more easy-to-use platform on top of it
>> to
>> >> help
>> >> > > > these
>> >> > > > > > > > users.
>> >> > > > > > > > > > > On the one hand, we will work on the interface
>> layer
>> >> to
>> >> > > make
>> >> > > > it
>> >> > > > > > > easy
>> >> > > > > > > > > > > to access (of course, this effort will make our
>> >> project
>> >> > > look
>> >> > > > > very
>> >> > > > > > > > > > > similar to other projects, such as HiveServer2). On
>> >> the
>> >> > > other
>> >> > > > > > hand,
>> >> > > > > > > > we
>> >> > > > > > > > > > > will work under Kyuubi's multi-tenant architecture
>> to
>> >> > > extend
>> >> > > > > our
>> >> > > > > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to
>> make
>> >> it
>> >> > > more
>> >> > > > > > > stable
>> >> > > > > > > > > > > and efficient in our scenario.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > What's more, we are going to add more on-prem
>> Kyuubi
>> >> engine
>> >> > > > > > types,
>> >> > > > > > > > e.g.
>> >> > > > > > > > > > > Flink.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一
>> >> 下午2:35写道：
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > > > Hey,
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > > > Looks like an interesting project, competition to
>> >> > > > Starburst.
>> >> > > > > > How
>> >> > > > > > > > > does it
>> >> > > > > > > > > > > > compare to Apache Dremel?
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > > > +1 (non-binding) for incubation.
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > > > Thanks and stay safe,
>> >> > > > > > > > > > > > --alex
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <
>> >> yao@apache.org>
>> >> > > > wrote:
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > > > > Dear all,
>> >> > > > > > > > > > > > >
>> >> > > > > > > > > > > > > We want to propose Kyuubi as a new Apache
>> >> Incubator
>> >> > > > > project.
>> >> > > > > > > > > > > > >
>> >> > > > > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift
>> >> JDBC/ODBC
>> >> > > > > server
>> >> > > > > > > for
>> >> > > > > > > > > > > > > large-scale data management, processing, and
>> >> analytics,
>> >> > > > > built
>> >> > > > > > > on
>> >> > > > > > > > > top
>> >> > > > > > > > > > > > > of Apache Spark and designed to support more
>> >> engines
>> >> > > > (i.e.,
>> >> > > > > > > > Apache
>> >> > > > > > > > > > > > > Flink). We are aiming to make Kyuubi an
>> >> > > "out-of-the-box"
>> >> > > > > tool
>> >> > > > > > > for
>> >> > > > > > > > > data
>> >> > > > > > > > > > > > > warehouses and data lakes. Kyuubi concentrates
>> on
>> >> > > > improving
>> >> > > > > > the
>> >> > > > > > > > > > > > > experience for non-Spark users, making it easy
>> >> for them
>> >> > > > to
>> >> > > > > > use
>> >> > > > > > > > > Spark
>> >> > > > > > > > > > > > > as a service and focus on business data of
>> their
>> >> own,
>> >> > > not
>> >> > > > > > where
>> >> > > > > > > > it
>> >> > > > > > > > > > > > > stores, how it computes. NetEase has
>> open-sourced
>> >> > > Kyuubi
>> >> > > > > > since
>> >> > > > > > > > > 2018,
>> >> > > > > > > > > > > > > https://github.com/NetEase/kyuubi.
>> >> > > > > > > > > > > > >
>> >> > > > > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff
>> >> Zhang, Duo
>> >> > > > > > Zhang,
>> >> > > > > > > > > Akira
>> >> > > > > > > > > > > > > Ajisaka for being the mentors of the Kyuubi
>> >> project and
>> >> > > > > > helping
>> >> > > > > > > > us
>> >> > > > > > > > > set
>> >> > > > > > > > > > > > > up this proposal. The proposal can be found at
>> >> > > > > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > >
>> >> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
>> >> > > > > > .
>> >> > > > > > > > > > > > >
>> >> > > > > > > > > > > > > Looking forward to your feedback and thank you
>> >> very
>> >> > > much.
>> >> > > > > > > > > > > > >
>> >> > > > > > > > > > > > > Best regards,
>> >> > > > > > > > > > > > > Kent Yao
>> >> > > > > > > > > > > > >
>> >> > > > > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > >
>> >> ---------------------------------------------------------------------
>> >> > > > > > > > > > > > > To unsubscribe, e-mail:
>> >> > > > > > > general-unsubscribe@incubator.apache.org
>> >> > > > > > > > > > > > > For additional commands, e-mail:
>> >> > > > > > > > general-help@incubator.apache.org
>> >> > > > > > > > > > > > >
>> >> > > > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > --
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > Kent Yao
>> >> > > > > > > > > > > @ Data Science Center, Hangzhou Research Institute,
>> >> NetEase
>> >> > > > > Corp.
>> >> > > > > > > > > > > a spark enthusiast
>> >> > > > > > > > > > > kyuubiis a unified multi-tenant JDBC interface for
>> >> > > > large-scale
>> >> > > > > > data
>> >> > > > > > > > > > > processing and analytics, built on top of Apache
>> >> Spark.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > spark-authorizerA Spark SQL extension which
>> provides
>> >> SQL
>> >> > > > > Standard
>> >> > > > > > > > > > > Authorization for Apache Spark.
>> >> > > > > > > > > > > spark-postgres A library for reading data from and
>> >> > > > transferring
>> >> > > > > > > data
>> >> > > > > > > > > > > to Postgres / Greenplum with Spark SQL and
>> DataFrames,
>> >> > > > 10~100x
>> >> > > > > > > > faster.
>> >> > > > > > > > > > > itatchiA library that brings useful functions from
>> >> various
>> >> > > > > modern
>> >> > > > > > > > > > > database management systems to Apache Spark.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > >
>> >> > >
>> ---------------------------------------------------------------------
>> >> > > > > > > > > > > To unsubscribe, e-mail:
>> >> > > > > general-unsubscribe@incubator.apache.org
>> >> > > > > > > > > > > For additional commands, e-mail:
>> >> > > > > > general-help@incubator.apache.org
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > >
>> >> ---------------------------------------------------------------------
>> >> > > > > > > > > To unsubscribe, e-mail:
>> >> > > general-unsubscribe@incubator.apache.org
>> >> > > > > > > > > For additional commands, e-mail:
>> >> > > > general-help@incubator.apache.org
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > --
>> >> > > > > >
>> >> > > > > > *Kent Yao *
>> >> > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
>> >> Corp.
>> >> > > > > > a spark enthusiast
>> >> > > > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified
>> >> multi-tenant
>> >> > > > > JDBC
>> >> > > > > > interface for large-scale data processing and analytics,
>> built
>> >> on top
>> >> > > > of
>> >> > > > > > Apache
>> >> > > > > > Spark <http://spark.apache.org/>.
>> >> > > > > > spark-authorizer <
>> https://github.com/yaooqinn/spark-authorizer
>> >> >A
>> >> > > Spark
>> >> > > > > SQL
>> >> > > > > > extension which provides SQL Standard Authorization for
>> Apache
>> >> Spark
>> >> > > > > > <http://spark.apache.org/>.
>> >> > > > > > spark-postgres <https://github.com/yaooqinn/spark-postgres>
>> A
>> >> > > library
>> >> > > > > for
>> >> > > > > > reading data from and transferring data to Postgres /
>> Greenplum
>> >> with
>> >> > > > > Spark
>> >> > > > > > SQL and DataFrames, 10~100x faster.
>> >> > > > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A
>> >> library
>> >> > > that
>> >> > > > > > brings useful functions from various modern database
>> management
>> >> > > > > > systems to Apache
>> >> > > > > > Spark <http://spark.apache.org/>.
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > >
>> >> > > > *Kent Yao *
>> >> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
>> >> > > > a spark enthusiast
>> >> > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified
>> >> multi-tenant
>> >> > > JDBC
>> >> > > > interface for large-scale data processing and analytics, built on
>> >> top of
>> >> > > > Apache
>> >> > > > Spark <http://spark.apache.org/>.
>> >> > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
>> >> Spark
>> >> > > SQL
>> >> > > > extension which provides SQL Standard Authorization for Apache
>> Spark
>> >> > > > <http://spark.apache.org/>.
>> >> > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
>> >> library
>> >> > > for
>> >> > > > reading data from and transferring data to Postgres / Greenplum
>> with
>> >> > > Spark
>> >> > > > SQL and DataFrames, 10~100x faster.
>> >> > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library
>> >> that
>> >> > > > brings useful functions from various modern database management
>> >> > > > systems to Apache
>> >> > > > Spark <http://spark.apache.org/>.
>> >> > > >
>> >> > >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> >> For additional commands, e-mail: general-help@incubator.apache.org
>> >>
>> >> --
>> > Thanks,
>> > Ming Wen, Apache APISIX PMC Chair
>> > Twitter: _WenMing
>> >
>> --
>> Thanks,
>> Ming Wen, Apache APISIX PMC Chair
>> Twitter: _WenMing
>>
>
>
> --
>
> *Kent Yao *
> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> a spark enthusiast
> kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
> interface for large-scale data processing and analytics, built on top of Apache
> Spark <http://spark.apache.org/>.
> spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> SQL extension which provides SQL Standard Authorization for Apache Spark
> <http://spark.apache.org/>.
> spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
> reading data from and transferring data to Postgres / Greenplum with Spark
> SQL and DataFrames, 10~100x faster.
> itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> brings useful functions from various modern database management systems to Apache
> Spark <http://spark.apache.org/>.
>
>
>
--
*Kent Yao *
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of Apache
Spark <http://spark.apache.org/>.
spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
extension which provides SQL Standard Authorization for Apache Spark
<http://spark.apache.org/>.
spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.
itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
brings useful functions from various modern database management
systems to Apache
Spark <http://spark.apache.org/>.
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Hi Ming,
Thanks for your question.
As mentioned in the proposal, Kyuubi was first developed and applied in
NetEase in 2018. The early scenario we wanted to address was
straightforward - to provide data analysts with a Spark-based analytics
platform. This platform needed to support multi-tenancy [1] and
authentication and authorization [2]. From the project's perspective, it
was just an alternative implementation of Apache Spark Thrift Server. And
for the workloads and users that it serves are not very big in NetEase. So
at that time, I was the only one who was maintaining the project.
Another reason for this problem should be the limited usage scenario caused
by its architecture. It was implemented based on
*spark.driver.allowMultipleContexts*, which was not good judging from the
dismal feedback after we open-sourced it. Many people have found Kyuubi or
me through GitHub, hoping that Kyuubi could solve a problem they were
having in their scenario. From all of these things, I remember most and
feel sorry for is when they said they were using Kyuubi to help medical
staff fight COVID-19 [9]. Limited by design, there was very little that
could be done at the time.
Starting with Spark 3.0.0, the Spark community also removed this feature
[3] for Kyuubi's old architecture. This allowed us to rethink and redesign
Kyuubi architecture. A prototype was quickly designed, but we did not start
it in a hurry. As Kyuubi relies heavily on Spark, we decided to actively
and deeply participate in the Spark community beginning around September
2019 [4][5][6], both to help the Spark community and to verify that the
design of the new Kyuubi architecture [7] was future-proofing enough. In
the process of implementing the new architecture, Zen Jin became a
contributor to the project and contributed several great ideas, which made
Kyuubi's architecture and the scenarios it can cover more general and
comprehensive. This has made Kyuubi's architecture and the scenarios it can
cover more general. This advancement has made Kyuubi more attractive than
older architecture projects.
As a result, many like-minded people [6] started to gather around the
project and the new architecture, and some of them have already become our
contributors. There are also many people who have encouraged us to
contribute to this project to Apache, which will be very beneficial for
them to be more directly involved in the project.
In addition, we've learned a lot about community operations while
contributing to other Apache projects like Spark, Submarine, and Hudi. Some
of these communities are new to Apache, and some are ten years out of
school. We also actively participated in some online and offline
conferences or meetups, published technical articles in different media to
promote Kyuubi.
From our practical experience, we have learned that promoting a project is
not an easy task. We want to donate Kyuubi to the community and bring in
potentially willing people to help the Kyuubi community grow.
Bests
Kent Yao
[1] https://issues.apache.org/jira/browse/SPARK-2243
[2] https://issues.apache.org/jira/browse/SPARK-5159
[3] https://issues.apache.org/jira/browse/SPARK-26362
[4] https://github.com/apache/spark/commits?author=yaooqinn
[5] https://github.com/apache/spark/commits?author=turboFei
[6] https://github.com/apache/spark/commits?author=ulysses-you
[7] https://github.com/NetEase/kyuubi/releases/tag/v1.0.0
[8]
https://github.com/NetEase/kyuubi/graphs/contributors?from=2021-01-01&to=2021-06-06&type=c
[9] https://github.com/NetEase/kyuubi/issues/228
Ming Wen <we...@apache.org> 于2021年6月6日周日 上午7:37写道：
> You can see the trend chart from
>
> https://www.apiseven.com/zh/contributor-graph?chart=contributorMonthlyActivity&repo=NetEase/kyuubi
>
>
> Ming Wen <we...@apache.org>于2021年6月6日 周日上午7:28写道：
>
> > hi, Kent,
> >
> > From 2018 to 2020, kuyybi is not an active project, most of the time
> there
> > is only one contributor per month. But in 2021, there will be 8 active
> > contributors every month. What is the reason for this?
> >
> > Willem Jiang <wi...@gmail.com>于2021年6月3日 周四下午12:53写道：
> >
> >> I think we can revisit the PPMC part and have more discussion with the
> >> initial committers.
> >> It's a good chance to let them know better about how to run an open
> >> source project in Apache Way.
> >>
> >>
> >> Willem Jiang
> >>
> >> Twitter: willemjiang
> >> Weibo: 姜宁willem
> >>
> >> On Thu, Jun 3, 2021 at 11:52 AM Atri Sharma <at...@apache.org> wrote:
> >> >
> >> > I would like to hear from Kyuubi's mentors on the topic before
> reaching
> >> a
> >> > conclusion.
> >> >
> >> > On Thu, 3 Jun 2021, 05:40 Sheng Wu, <wu...@gmail.com>
> wrote:
> >> >
> >> > > Kent
> >> > >
> >> > > Thanks.
> >> > > The proposal is generally better this time.
> >> > >
> >> > > If your initial committers and your mentors agree, you are ready to
> >> embrace
> >> > > the ASF culture, I am good with that.
> >> > > Just from my experience, your community looks a little too young,
> like
> >> > > Atri's concern, which is reasonable.
> >> > > But I would like to trust you and your mentors' judgment and
> >> determination.
> >> > >
> >> > > Sheng Wu 吴晟
> >> > > Twitter, wusheng1108
> >> > >
> >> > >
> >> > > Kent Yao <ya...@apache.org> 于2021年6月3日周四 上午12:54写道：
> >> > >
> >> > > > Hi Sheng,
> >> > > >
> >> > > > Truly sorry for the format issue.
> >> > > >
> >> > > > Please trust me I did not mean to hurt anybody and push any
> >> contributor
> >> > > > away. We drafted this proposal together based on the New Podling
> >> > > Proposal.
> >> > > > We never tried to use PPMC and committer to measure whose
> >> contribution is
> >> > > > higher. This mistake has been fixed in the proposal.
> >> > > >
> >> > > > About releases, we will follow
> >> > > > https://www.apache.org/legal/release-policy.html and the guide of
> >> our
> >> > > > mentors. We will also take the initiative to learn other
> documents.
> >> > > >
> >> > > > > This is not a block, the point I mentioned this is, you will
> need
> >> more
> >> > > > people.
> >> > > >
> >> > > > We will keep working on this.
> >> > > >
> >> > > > Bests,
> >> > > >
> >> > > > Kent Yao
> >> > > >
> >> > > >
> >> > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
> >> > > >
> >> > > > > Hi Kent
> >> > > > >
> >> > > > > Thanks for replying, but to be honest, your mail format is
> chaos.
> >> > > > > Take a look at bellowing, which is what I saw in the gmail.
> >> > > > >
> >> > > > > This may be an issue of your mail client or some else.
> >> > > > >
> >> > > > > > The current #.2 - 5 top
> >> > > > > > contributors will stay as committers, and they will join the
> >> PPMC
> >> > > > during
> >> > > > > > incubation.
> >> > > > >
> >> > > > > PPMC usually is the default for all initial committers,
> >> considering
> >> > > your
> >> > > > > community is still small and not very active.
> >> > > > > You should invite all people you trusted to join, rather than
> >> pushing
> >> > > > them
> >> > > > > away.
> >> > > > > This is not just a too-high bar issue. Excluding 4 out of 5 is
> >> strange.
> >> > > > > I can't guess what is the root cause, why the other 4 PPMC
> members
> >> > > don't
> >> > > > > touch codes but are trusted completely. Meanwhile, people with
> >> easily
> >> > > > > measurable code contributions are just as committers?
> >> > > > > We(ASF) always remind the community to recognize no-code
> >> contribution,
> >> > > > but
> >> > > > > in this case, it seems the opposite? Don't recognize code
> >> contributors
> >> > > > > enough?
> >> > > > > Please reach your mentors, this seems not a typical ASF style.
> If
> >> you
> >> > > > have
> >> > > > > other people already involved in the ASF, asking them too.
> >> > > > >
> >> > > > > And from your description of the release process, I think you
> >> should
> >> > > talk
> >> > > > > more with your mentors.
> >> > > > > Such as what is a qualified ASF release, what should be checked.
> >> > > > > This is not a block, the point I mentioned this is, you will
> need
> >> more
> >> > > > > people.
> >> > > > >
> >> > > > >
> >> > > > > Sheng Wu 吴晟
> >> > > > > Twitter, wusheng1108
> >> > > > >
> >> > > > >
> >> > > > > Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
> >> > > > >
> >> > > > > > Hi Sheng,
> >> > > > > >
> >> > > > > > Thanks for your detailed check and valuable comments. They are
> >> > > > > > accommodating. I have addressed some of them directly in the
> >> > > proposal.
> >> > > > > > Please also check the table below for some Q&As.
> >> > > > > > QA
> >> > > > > > You listed only 5 core maintainers, but clearly only one has
> >> > > dominated
> >> > > > > > contributions. The distortion and skewness of the
> contributions
> >> are
> >> > > > > mainly
> >> > > > > > caused by the way how we merge PRs changed(mentioned in
> another
> >> email
> >> > > > on
> >> > > > > > the list).
> >> > > > > > I agree with Atri, orphaned is measurable. It is better to
> >> adjust
> >> > > this
> >> > > > > > statement a little. Address the comment to notify the risk in
> >> the
> >> > > > Kyuubi
> >> > > > > > proposal.
> >> > > > > > Also, with this fact, the core maintainers should notice
> >> whether this
> >> > > > > small
> >> > > > > > group of people to make sure your features iteration, bug fix,
> >> and
> >> > > > > release
> >> > > > > > process working by following ASF's requirements.This would not
> >> be an
> >> > > > easy
> >> > > > > > job. Currently, we learn the release process from some Apache
> >> > > projects,
> >> > > > > > using major.minor.patch for versioning. As we can see, since
> >> Kyuubi
> >> > > > 1.0,
> >> > > > > it
> >> > > > > > takes us 2 ~ 3 weeks for a patch release that contains 5-10
> bug
> >> fixes
> >> > > > > until
> >> > > > > > EOL. And it takes us about two months to finish a minor
> release
> >> with
> >> > > 3
> >> > > > ~
> >> > > > > 5
> >> > > > > > big improvements and various small features/bug fixes. We also
> >> will
> >> > > > start
> >> > > > > > with some release candidates for preview and test before the
> >> final.
> >> > > > This
> >> > > > > is
> >> > > > > > how we develop and release Kyuubi for now, but we do have a
> >> long way
> >> > > to
> >> > > > > go
> >> > > > > > and learn to match the ASF requirement.
> >> > > > > > And your committer list is very strange... 1. Why some of them
> >> are
> >> > > PPMC
> >> > > > > and
> >> > > > > > others are committers? Where this hierarchy comes from? The
> >> current
> >> > > > PPMC
> >> > > > > > candidates help us a lot build the Kyuubi project and
> >> community, not
> >> > > > > > directly to the codebase but as mentors(before Apache). Most
> of
> >> them
> >> > > > are
> >> > > > > > from Apache mature TLPs, which have close relationships with
> >> Kyuubi.
> >> > > > They
> >> > > > > > have sufficient Apache Way experience to guide us to become a
> >> more
> >> > > > > > Apache-like project before submitting this proposal in many
> >> ways. We
> >> > > > will
> >> > > > > > continue to promote more contributors to the PPMC/committer
> >> based on
> >> > > > > their
> >> > > > > > code or non-code contributions during incubation.
> >> > > > > > 2. Your core maintainers have 5, which only 4 PPMC? The
> current
> >> #.2
> >> > > - 5
> >> > > > > top
> >> > > > > > contributors will stay as committers, and they will join the
> >> PPMC
> >> > > > during
> >> > > > > > incubation.
> >> > > > > > 3. I think only the top 4 have over 400 LOC, why you declared
> >> you
> >> > > have
> >> > > > 5
> >> > > > > > core developers? I didn't notice that 400 LOC is a bar for
> core
> >> > > > > developer,
> >> > > > > > will change 5 to 4 core developers accordingly
> >> > > > > > I think you don't have enough no-salaried developers, am I
> >> missing
> >> > > > > > anything? Yes.
> >> > > > > > Changed in the proposal, most of the contributors are
> salaried,
> >> and
> >> > > > some
> >> > > > > of
> >> > > > > > them are willing to help in the spare time. Thanks
> >> > > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
> Added
> >> SGA
> >> > > in
> >> > > > > > Proposal. All work here is approved by NetEase and can be
> >> forward.
> >> > > > > > At last, you are asking about
> >> > > > github.com/apache/incubator-kyuubi-shaded
> >> > > > > > repo,
> >> > > > > > what does this represent? I can only see you have only one
> >> source
> >> > > repo
> >> > > > > and
> >> > > > > > a doc repo. Removed this requirement in the proposal, and we
> >> will
> >> > > apply
> >> > > > > it
> >> > > > > > from IPMC if we do need this in the future.
> >> > > > > > ------------------------------
> >> > > > > >
> >> > > > > > Bests
> >> > > > > >
> >> > > > > > Kent Yao
> >> > > > > >
> >> > > > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> >> > > > > >
> >> > > > > > > Hi
> >> > > > > > >
> >> > > > > > > I read the contributor list and the proposal
> >> > > > > > > > The risk of the Kyuubi project being abandoned is minimal.
> >> Many
> >> > > > > > > organizations are using Kyuubi to build critical big data
> >> pipelines
> >> > > > and
> >> > > > > > > willing to help develop Kyuubi's community if it becomes an
> >> ASF
> >> > > > > project.
> >> > > > > > >
> >> > > > > > > I think the team should evaluate the status again. You
> listed
> >> only
> >> > > 5
> >> > > > > core
> >> > > > > > > maintainers, but clearly only one has dominated
> >> contributions. I
> >> > > > agree
> >> > > > > > with
> >> > > > > > > Atri, orphaned is measurable.
> >> > > > > > > It is better to adjust this statement a little. Also, with
> >> this
> >> > > fact,
> >> > > > > the
> >> > > > > > > core maintainers should notice whether this small group of
> >> people
> >> > > to
> >> > > > > make
> >> > > > > > > sure your features iteration, bug fix, and release process
> >> working
> >> > > by
> >> > > > > > > following ASF's requirements.
> >> > > > > > > This would not be an easy job.
> >> > > > > > >
> >> > > > > > > And your committer list is very strange
> >> > > > > > > ```
> >> > > > > > > Kent Yao (yao@apache.org) (PPMC)
> >> > > > > > > Wenchen Fan (wenchen@apache.org) (PPMC)
> >> > > > > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> >> > > > > > > Xun Liu (liuxun@apache.org) (PPMC)
> >> > > > > > > Fei Wang (cn.feiwang@gmail.com)
> >> > > > > > > Ulysses You (ulyssesyou18@gmail.com)
> >> > > > > > > Cheng Pan (pan3793@gmail.com)
> >> > > > > > > Hongxiang Jiang (jhx1008@gmail.com)
> >> > > > > > > Zen Jin(xinjingziranchan@gmail.com)
> >> > > > > > > ```
> >> > > > > > > 1. Why some of them are PPMC and others are committers?
> Where
> >> this
> >> > > > > > > hierarchy comes from?
> >> > > > > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even
> >> hierarchy
> >> > > > > makes
> >> > > > > > > sense, most likely not)
> >> > > > > > > 3. I think only the top 4 have over 400 LOC, why you
> declared
> >> you
> >> > > > have
> >> > > > > 5
> >> > > > > > > core developers?
> >> > > > > > >
> >> > > > > > > Another re-evaluation request is about `Reliance on Salaried
> >> > > > > Developers:`
> >> > > > > > > > They are all passionate about the project, and we are
> >> confident
> >> > > > that
> >> > > > > > the
> >> > > > > > > project will continue even if no salaried developers
> >> contribute to
> >> > > > the
> >> > > > > > > project.
> >> > > > > > >
> >> > > > > > > I think you don't have enough no-salaried developers, am I
> >> missing
> >> > > > > > > anything?
> >> > > > > > >
> >> > > > > > > About IP clearance, `Initial Source and Intellectual
> Property
> >> > > > > Submission
> >> > > > > > > Plan`
> >> > > > > > > > As soon as Kyuubi is approved to join Apache Incubator,
> our
> >> > > initial
> >> > > > > > > committers will submit iCLA(s) and CCLA(s). The codebase is
> >> already
> >> > > > > > > licensed under the Apache License 2.0.
> >> > > > > > >
> >> > > > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
> >> > > > > > >
> >> > > > > > > At last, you are asking about `
> >> > > > > > github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> >> <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > > > > repo, what does this represent? I can only see you have only
> >> one
> >> > > > source
> >> > > > > > > repo and a doc repo.
> >> > > > > > >
> >> > > > > > > Sheng Wu 吴晟
> >> > > > > > > Twitter, wusheng1108
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> >> > > > > > >
> >> > > > > > > > Thanks.
> >> > > > > > > >
> >> > > > > > > > I looked at the link you advised on and if I am reading it
> >> > > > correctly,
> >> > > > > > top
> >> > > > > > > > committer has 630ish commits and the next committer has 73
> >> > > commits?
> >> > > > > > > >
> >> > > > > > > > IMO this is an indication of the community not being
> active
> >> > > enough
> >> > > > > and
> >> > > > > > > > hence the risk of project getting orphaned is measurable.
> >> > > > > > > >
> >> > > > > > > > Atri
> >> > > > > > > >
> >> > > > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <
> >> willem.jiang@gmail.com>
> >> > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Atri Sharma,
> >> > > > > > > > >
> >> > > > > > > > > It's easier to check out the recent contribution of
> >> kyuubi from
> >> > > > > this
> >> > > > > > > > > link[1].
> >> > > > > > > > > We already identified the one person contribution issue
> of
> >> > > kyuubi
> >> > > > > > when
> >> > > > > > > > > Kent came to us and we kept reminding him to put more
> >> effort
> >> > > into
> >> > > > > > > > > building the community.
> >> > > > > > > > > Last week I had a long F2F conversation with Kent, I
> >> think he
> >> > > > > already
> >> > > > > > > > > knows about it and there have been some improvements in
> >> recent
> >> > > > > months
> >> > > > > > > > > but there is still a long way to go if Kyuubi is
> accepted
> >> as a
> >> > > > > > podling
> >> > > > > > > > > in the Apache Incubator.
> >> > > > > > > > >
> >> > > > > > > > > [1]
> https://github.com/NetEase/kyuubi/graphs/contributors
> >> > > > > > > > >
> >> > > > > > > > > Willem Jiang
> >> > > > > > > > >
> >> > > > > > > > > Twitter: willemjiang
> >> > > > > > > > > Weibo: 姜宁willem
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <
> >> atri@apache.org>
> >> > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > > I took a look at commit log and seems like all the
> >> commits
> >> > > for
> >> > > > > > last 3
> >> > > > > > > > > > months have been done by a single person.
> >> > > > > > > > > >
> >> > > > > > > > > > How active are the remaining committers?
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org>
> >> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Alexander Alten,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks for your interest and +1 for Kyuubi.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Not being very familiar with Dremel, I just took a
> >> quick
> >> > > look
> >> > > > > at
> >> > > > > > > the
> >> > > > > > > > > > > profile of a very similar and exciting project,
> Apache
> >> > > Drill.
> >> > > > > If
> >> > > > > > I
> >> > > > > > > > > > > understand correctly, Kyuubi is built on top of the
> >> compute
> >> > > > > layer
> >> > > > > > > > > > > (Apache Spark), while Apache Drill is built on top
> of
> >> the
> >> > > > > storage
> >> > > > > > > > > > > layer.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Apache Spark has a large user base and usage
> >> scenarios all
> >> > > > over
> >> > > > > > the
> >> > > > > > > > > > > world. When users have some basic Spark programming
> >> and
> >> > > > tuning
> >> > > > > > > > skills,
> >> > > > > > > > > > > things are easy for them. But many times, most users
> >> do not
> >> > > > > have
> >> > > > > > > all
> >> > > > > > > > > > > of these abilities when they use Spark directly or
> >> > > > indirectly.
> >> > > > > We
> >> > > > > > > > want
> >> > > > > > > > > > > to build a more easy-to-use platform on top of it to
> >> help
> >> > > > these
> >> > > > > > > > users.
> >> > > > > > > > > > > On the one hand, we will work on the interface layer
> >> to
> >> > > make
> >> > > > it
> >> > > > > > > easy
> >> > > > > > > > > > > to access (of course, this effort will make our
> >> project
> >> > > look
> >> > > > > very
> >> > > > > > > > > > > similar to other projects, such as HiveServer2). On
> >> the
> >> > > other
> >> > > > > > hand,
> >> > > > > > > > we
> >> > > > > > > > > > > will work under Kyuubi's multi-tenant architecture
> to
> >> > > extend
> >> > > > > our
> >> > > > > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to
> make
> >> it
> >> > > more
> >> > > > > > > stable
> >> > > > > > > > > > > and efficient in our scenario.
> >> > > > > > > > > > >
> >> > > > > > > > > > > What's more, we are going to add more on-prem Kyuubi
> >> engine
> >> > > > > > types,
> >> > > > > > > > e.g.
> >> > > > > > > > > > > Flink.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一
> >> 下午2:35写道：
> >> > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Hey,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Looks like an interesting project, competition to
> >> > > > Starburst.
> >> > > > > > How
> >> > > > > > > > > does it
> >> > > > > > > > > > > > compare to Apache Dremel?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > +1 (non-binding) for incubation.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks and stay safe,
> >> > > > > > > > > > > > --alex
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <
> >> yao@apache.org>
> >> > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Dear all,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > We want to propose Kyuubi as a new Apache
> >> Incubator
> >> > > > > project.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift
> >> JDBC/ODBC
> >> > > > > server
> >> > > > > > > for
> >> > > > > > > > > > > > > large-scale data management, processing, and
> >> analytics,
> >> > > > > built
> >> > > > > > > on
> >> > > > > > > > > top
> >> > > > > > > > > > > > > of Apache Spark and designed to support more
> >> engines
> >> > > > (i.e.,
> >> > > > > > > > Apache
> >> > > > > > > > > > > > > Flink). We are aiming to make Kyuubi an
> >> > > "out-of-the-box"
> >> > > > > tool
> >> > > > > > > for
> >> > > > > > > > > data
> >> > > > > > > > > > > > > warehouses and data lakes. Kyuubi concentrates
> on
> >> > > > improving
> >> > > > > > the
> >> > > > > > > > > > > > > experience for non-Spark users, making it easy
> >> for them
> >> > > > to
> >> > > > > > use
> >> > > > > > > > > Spark
> >> > > > > > > > > > > > > as a service and focus on business data of their
> >> own,
> >> > > not
> >> > > > > > where
> >> > > > > > > > it
> >> > > > > > > > > > > > > stores, how it computes. NetEase has
> open-sourced
> >> > > Kyuubi
> >> > > > > > since
> >> > > > > > > > > 2018,
> >> > > > > > > > > > > > > https://github.com/NetEase/kyuubi.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff
> >> Zhang, Duo
> >> > > > > > Zhang,
> >> > > > > > > > > Akira
> >> > > > > > > > > > > > > Ajisaka for being the mentors of the Kyuubi
> >> project and
> >> > > > > > helping
> >> > > > > > > > us
> >> > > > > > > > > set
> >> > > > > > > > > > > > > up this proposal. The proposal can be found at
> >> > > > > > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> >> > > > > > .
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Looking forward to your feedback and thank you
> >> very
> >> > > much.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Best regards,
> >> > > > > > > > > > > > > Kent Yao
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> ---------------------------------------------------------------------
> >> > > > > > > > > > > > > To unsubscribe, e-mail:
> >> > > > > > > general-unsubscribe@incubator.apache.org
> >> > > > > > > > > > > > > For additional commands, e-mail:
> >> > > > > > > > general-help@incubator.apache.org
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > >
> >> > > > > > > > > > > Kent Yao
> >> > > > > > > > > > > @ Data Science Center, Hangzhou Research Institute,
> >> NetEase
> >> > > > > Corp.
> >> > > > > > > > > > > a spark enthusiast
> >> > > > > > > > > > > kyuubiis a unified multi-tenant JDBC interface for
> >> > > > large-scale
> >> > > > > > data
> >> > > > > > > > > > > processing and analytics, built on top of Apache
> >> Spark.
> >> > > > > > > > > > >
> >> > > > > > > > > > > spark-authorizerA Spark SQL extension which provides
> >> SQL
> >> > > > > Standard
> >> > > > > > > > > > > Authorization for Apache Spark.
> >> > > > > > > > > > > spark-postgres A library for reading data from and
> >> > > > transferring
> >> > > > > > > data
> >> > > > > > > > > > > to Postgres / Greenplum with Spark SQL and
> DataFrames,
> >> > > > 10~100x
> >> > > > > > > > faster.
> >> > > > > > > > > > > itatchiA library that brings useful functions from
> >> various
> >> > > > > modern
> >> > > > > > > > > > > database management systems to Apache Spark.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > >
> >> > >
> ---------------------------------------------------------------------
> >> > > > > > > > > > > To unsubscribe, e-mail:
> >> > > > > general-unsubscribe@incubator.apache.org
> >> > > > > > > > > > > For additional commands, e-mail:
> >> > > > > > general-help@incubator.apache.org
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> ---------------------------------------------------------------------
> >> > > > > > > > > To unsubscribe, e-mail:
> >> > > general-unsubscribe@incubator.apache.org
> >> > > > > > > > > For additional commands, e-mail:
> >> > > > general-help@incubator.apache.org
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > >
> >> > > > > > *Kent Yao *
> >> > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
> >> Corp.
> >> > > > > > a spark enthusiast
> >> > > > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified
> >> multi-tenant
> >> > > > > JDBC
> >> > > > > > interface for large-scale data processing and analytics, built
> >> on top
> >> > > > of
> >> > > > > > Apache
> >> > > > > > Spark <http://spark.apache.org/>.
> >> > > > > > spark-authorizer <
> https://github.com/yaooqinn/spark-authorizer
> >> >A
> >> > > Spark
> >> > > > > SQL
> >> > > > > > extension which provides SQL Standard Authorization for Apache
> >> Spark
> >> > > > > > <http://spark.apache.org/>.
> >> > > > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
> >> > > library
> >> > > > > for
> >> > > > > > reading data from and transferring data to Postgres /
> Greenplum
> >> with
> >> > > > > Spark
> >> > > > > > SQL and DataFrames, 10~100x faster.
> >> > > > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A
> >> library
> >> > > that
> >> > > > > > brings useful functions from various modern database
> management
> >> > > > > > systems to Apache
> >> > > > > > Spark <http://spark.apache.org/>.
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > > *Kent Yao *
> >> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> >> > > > a spark enthusiast
> >> > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified
> >> multi-tenant
> >> > > JDBC
> >> > > > interface for large-scale data processing and analytics, built on
> >> top of
> >> > > > Apache
> >> > > > Spark <http://spark.apache.org/>.
> >> > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
> >> Spark
> >> > > SQL
> >> > > > extension which provides SQL Standard Authorization for Apache
> Spark
> >> > > > <http://spark.apache.org/>.
> >> > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
> >> library
> >> > > for
> >> > > > reading data from and transferring data to Postgres / Greenplum
> with
> >> > > Spark
> >> > > > SQL and DataFrames, 10~100x faster.
> >> > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library
> >> that
> >> > > > brings useful functions from various modern database management
> >> > > > systems to Apache
> >> > > > Spark <http://spark.apache.org/>.
> >> > > >
> >> > >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >> --
> > Thanks,
> > Ming Wen, Apache APISIX PMC Chair
> > Twitter: _WenMing
> >
> --
> Thanks,
> Ming Wen, Apache APISIX PMC Chair
> Twitter: _WenMing
>
--
*Kent Yao *
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of Apache
Spark <http://spark.apache.org/>.
spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
extension which provides SQL Standard Authorization for Apache Spark
<http://spark.apache.org/>.
spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.
itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
brings useful functions from various modern database management
systems to Apache
Spark <http://spark.apache.org/>.
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Ming Wen <we...@apache.org>.
You can see the trend chart from
https://www.apiseven.com/zh/contributor-graph?chart=contributorMonthlyActivity&repo=NetEase/kyuubi
Ming Wen <we...@apache.org>于2021年6月6日 周日上午7:28写道：
> hi, Kent,
>
> From 2018 to 2020, kuyybi is not an active project, most of the time there
> is only one contributor per month. But in 2021, there will be 8 active
> contributors every month. What is the reason for this?
>
> Willem Jiang <wi...@gmail.com>于2021年6月3日 周四下午12:53写道：
>
>> I think we can revisit the PPMC part and have more discussion with the
>> initial committers.
>> It's a good chance to let them know better about how to run an open
>> source project in Apache Way.
>>
>>
>> Willem Jiang
>>
>> Twitter: willemjiang
>> Weibo: 姜宁willem
>>
>> On Thu, Jun 3, 2021 at 11:52 AM Atri Sharma <at...@apache.org> wrote:
>> >
>> > I would like to hear from Kyuubi's mentors on the topic before reaching
>> a
>> > conclusion.
>> >
>> > On Thu, 3 Jun 2021, 05:40 Sheng Wu, <wu...@gmail.com> wrote:
>> >
>> > > Kent
>> > >
>> > > Thanks.
>> > > The proposal is generally better this time.
>> > >
>> > > If your initial committers and your mentors agree, you are ready to
>> embrace
>> > > the ASF culture, I am good with that.
>> > > Just from my experience, your community looks a little too young, like
>> > > Atri's concern, which is reasonable.
>> > > But I would like to trust you and your mentors' judgment and
>> determination.
>> > >
>> > > Sheng Wu 吴晟
>> > > Twitter, wusheng1108
>> > >
>> > >
>> > > Kent Yao <ya...@apache.org> 于2021年6月3日周四 上午12:54写道：
>> > >
>> > > > Hi Sheng,
>> > > >
>> > > > Truly sorry for the format issue.
>> > > >
>> > > > Please trust me I did not mean to hurt anybody and push any
>> contributor
>> > > > away. We drafted this proposal together based on the New Podling
>> > > Proposal.
>> > > > We never tried to use PPMC and committer to measure whose
>> contribution is
>> > > > higher. This mistake has been fixed in the proposal.
>> > > >
>> > > > About releases, we will follow
>> > > > https://www.apache.org/legal/release-policy.html and the guide of
>> our
>> > > > mentors. We will also take the initiative to learn other documents.
>> > > >
>> > > > > This is not a block, the point I mentioned this is, you will need
>> more
>> > > > people.
>> > > >
>> > > > We will keep working on this.
>> > > >
>> > > > Bests,
>> > > >
>> > > > Kent Yao
>> > > >
>> > > >
>> > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
>> > > >
>> > > > > Hi Kent
>> > > > >
>> > > > > Thanks for replying, but to be honest, your mail format is chaos.
>> > > > > Take a look at bellowing, which is what I saw in the gmail.
>> > > > >
>> > > > > This may be an issue of your mail client or some else.
>> > > > >
>> > > > > > The current #.2 - 5 top
>> > > > > > contributors will stay as committers, and they will join the
>> PPMC
>> > > > during
>> > > > > > incubation.
>> > > > >
>> > > > > PPMC usually is the default for all initial committers,
>> considering
>> > > your
>> > > > > community is still small and not very active.
>> > > > > You should invite all people you trusted to join, rather than
>> pushing
>> > > > them
>> > > > > away.
>> > > > > This is not just a too-high bar issue. Excluding 4 out of 5 is
>> strange.
>> > > > > I can't guess what is the root cause, why the other 4 PPMC members
>> > > don't
>> > > > > touch codes but are trusted completely. Meanwhile, people with
>> easily
>> > > > > measurable code contributions are just as committers?
>> > > > > We(ASF) always remind the community to recognize no-code
>> contribution,
>> > > > but
>> > > > > in this case, it seems the opposite? Don't recognize code
>> contributors
>> > > > > enough?
>> > > > > Please reach your mentors, this seems not a typical ASF style. If
>> you
>> > > > have
>> > > > > other people already involved in the ASF, asking them too.
>> > > > >
>> > > > > And from your description of the release process, I think you
>> should
>> > > talk
>> > > > > more with your mentors.
>> > > > > Such as what is a qualified ASF release, what should be checked.
>> > > > > This is not a block, the point I mentioned this is, you will need
>> more
>> > > > > people.
>> > > > >
>> > > > >
>> > > > > Sheng Wu 吴晟
>> > > > > Twitter, wusheng1108
>> > > > >
>> > > > >
>> > > > > Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
>> > > > >
>> > > > > > Hi Sheng,
>> > > > > >
>> > > > > > Thanks for your detailed check and valuable comments. They are
>> > > > > > accommodating. I have addressed some of them directly in the
>> > > proposal.
>> > > > > > Please also check the table below for some Q&As.
>> > > > > > QA
>> > > > > > You listed only 5 core maintainers, but clearly only one has
>> > > dominated
>> > > > > > contributions. The distortion and skewness of the contributions
>> are
>> > > > > mainly
>> > > > > > caused by the way how we merge PRs changed(mentioned in another
>> email
>> > > > on
>> > > > > > the list).
>> > > > > > I agree with Atri, orphaned is measurable. It is better to
>> adjust
>> > > this
>> > > > > > statement a little. Address the comment to notify the risk in
>> the
>> > > > Kyuubi
>> > > > > > proposal.
>> > > > > > Also, with this fact, the core maintainers should notice
>> whether this
>> > > > > small
>> > > > > > group of people to make sure your features iteration, bug fix,
>> and
>> > > > > release
>> > > > > > process working by following ASF's requirements.This would not
>> be an
>> > > > easy
>> > > > > > job. Currently, we learn the release process from some Apache
>> > > projects,
>> > > > > > using major.minor.patch for versioning. As we can see, since
>> Kyuubi
>> > > > 1.0,
>> > > > > it
>> > > > > > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug
>> fixes
>> > > > > until
>> > > > > > EOL. And it takes us about two months to finish a minor release
>> with
>> > > 3
>> > > > ~
>> > > > > 5
>> > > > > > big improvements and various small features/bug fixes. We also
>> will
>> > > > start
>> > > > > > with some release candidates for preview and test before the
>> final.
>> > > > This
>> > > > > is
>> > > > > > how we develop and release Kyuubi for now, but we do have a
>> long way
>> > > to
>> > > > > go
>> > > > > > and learn to match the ASF requirement.
>> > > > > > And your committer list is very strange... 1. Why some of them
>> are
>> > > PPMC
>> > > > > and
>> > > > > > others are committers? Where this hierarchy comes from? The
>> current
>> > > > PPMC
>> > > > > > candidates help us a lot build the Kyuubi project and
>> community, not
>> > > > > > directly to the codebase but as mentors(before Apache). Most of
>> them
>> > > > are
>> > > > > > from Apache mature TLPs, which have close relationships with
>> Kyuubi.
>> > > > They
>> > > > > > have sufficient Apache Way experience to guide us to become a
>> more
>> > > > > > Apache-like project before submitting this proposal in many
>> ways. We
>> > > > will
>> > > > > > continue to promote more contributors to the PPMC/committer
>> based on
>> > > > > their
>> > > > > > code or non-code contributions during incubation.
>> > > > > > 2. Your core maintainers have 5, which only 4 PPMC? The current
>> #.2
>> > > - 5
>> > > > > top
>> > > > > > contributors will stay as committers, and they will join the
>> PPMC
>> > > > during
>> > > > > > incubation.
>> > > > > > 3. I think only the top 4 have over 400 LOC, why you declared
>> you
>> > > have
>> > > > 5
>> > > > > > core developers? I didn't notice that 400 LOC is a bar for core
>> > > > > developer,
>> > > > > > will change 5 to 4 core developers accordingly
>> > > > > > I think you don't have enough no-salaried developers, am I
>> missing
>> > > > > > anything? Yes.
>> > > > > > Changed in the proposal, most of the contributors are salaried,
>> and
>> > > > some
>> > > > > of
>> > > > > > them are willing to help in the spare time. Thanks
>> > > > > > This project belongs to NetEase Inc. clearly, why NO SGA? Added
>> SGA
>> > > in
>> > > > > > Proposal. All work here is approved by NetEase and can be
>> forward.
>> > > > > > At last, you are asking about
>> > > > github.com/apache/incubator-kyuubi-shaded
>> > > > > > repo,
>> > > > > > what does this represent? I can only see you have only one
>> source
>> > > repo
>> > > > > and
>> > > > > > a doc repo. Removed this requirement in the proposal, and we
>> will
>> > > apply
>> > > > > it
>> > > > > > from IPMC if we do need this in the future.
>> > > > > > ------------------------------
>> > > > > >
>> > > > > > Bests
>> > > > > >
>> > > > > > Kent Yao
>> > > > > >
>> > > > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
>> > > > > >
>> > > > > > > Hi
>> > > > > > >
>> > > > > > > I read the contributor list and the proposal
>> > > > > > > > The risk of the Kyuubi project being abandoned is minimal.
>> Many
>> > > > > > > organizations are using Kyuubi to build critical big data
>> pipelines
>> > > > and
>> > > > > > > willing to help develop Kyuubi's community if it becomes an
>> ASF
>> > > > > project.
>> > > > > > >
>> > > > > > > I think the team should evaluate the status again. You listed
>> only
>> > > 5
>> > > > > core
>> > > > > > > maintainers, but clearly only one has dominated
>> contributions. I
>> > > > agree
>> > > > > > with
>> > > > > > > Atri, orphaned is measurable.
>> > > > > > > It is better to adjust this statement a little. Also, with
>> this
>> > > fact,
>> > > > > the
>> > > > > > > core maintainers should notice whether this small group of
>> people
>> > > to
>> > > > > make
>> > > > > > > sure your features iteration, bug fix, and release process
>> working
>> > > by
>> > > > > > > following ASF's requirements.
>> > > > > > > This would not be an easy job.
>> > > > > > >
>> > > > > > > And your committer list is very strange
>> > > > > > > ```
>> > > > > > > Kent Yao (yao@apache.org) (PPMC)
>> > > > > > > Wenchen Fan (wenchen@apache.org) (PPMC)
>> > > > > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
>> > > > > > > Xun Liu (liuxun@apache.org) (PPMC)
>> > > > > > > Fei Wang (cn.feiwang@gmail.com)
>> > > > > > > Ulysses You (ulyssesyou18@gmail.com)
>> > > > > > > Cheng Pan (pan3793@gmail.com)
>> > > > > > > Hongxiang Jiang (jhx1008@gmail.com)
>> > > > > > > Zen Jin(xinjingziranchan@gmail.com)
>> > > > > > > ```
>> > > > > > > 1. Why some of them are PPMC and others are committers? Where
>> this
>> > > > > > > hierarchy comes from?
>> > > > > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even
>> hierarchy
>> > > > > makes
>> > > > > > > sense, most likely not)
>> > > > > > > 3. I think only the top 4 have over 400 LOC, why you declared
>> you
>> > > > have
>> > > > > 5
>> > > > > > > core developers?
>> > > > > > >
>> > > > > > > Another re-evaluation request is about `Reliance on Salaried
>> > > > > Developers:`
>> > > > > > > > They are all passionate about the project, and we are
>> confident
>> > > > that
>> > > > > > the
>> > > > > > > project will continue even if no salaried developers
>> contribute to
>> > > > the
>> > > > > > > project.
>> > > > > > >
>> > > > > > > I think you don't have enough no-salaried developers, am I
>> missing
>> > > > > > > anything?
>> > > > > > >
>> > > > > > > About IP clearance, `Initial Source and Intellectual Property
>> > > > > Submission
>> > > > > > > Plan`
>> > > > > > > > As soon as Kyuubi is approved to join Apache Incubator, our
>> > > initial
>> > > > > > > committers will submit iCLA(s) and CCLA(s). The codebase is
>> already
>> > > > > > > licensed under the Apache License 2.0.
>> > > > > > >
>> > > > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
>> > > > > > >
>> > > > > > > At last, you are asking about `
>> > > > > > github.com/apache/incubator-kyuubi-shaded`
>> <http://github.com/apache/incubator-kyuubi-shaded>
>> > > <http://github.com/apache/incubator-kyuubi-shaded>
>> > > > <http://github.com/apache/incubator-kyuubi-shaded>
>> > > > > <http://github.com/apache/incubator-kyuubi-shaded>
>> > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
>> > > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
>> > > > > > > repo, what does this represent? I can only see you have only
>> one
>> > > > source
>> > > > > > > repo and a doc repo.
>> > > > > > >
>> > > > > > > Sheng Wu 吴晟
>> > > > > > > Twitter, wusheng1108
>> > > > > > >
>> > > > > > >
>> > > > > > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
>> > > > > > >
>> > > > > > > > Thanks.
>> > > > > > > >
>> > > > > > > > I looked at the link you advised on and if I am reading it
>> > > > correctly,
>> > > > > > top
>> > > > > > > > committer has 630ish commits and the next committer has 73
>> > > commits?
>> > > > > > > >
>> > > > > > > > IMO this is an indication of the community not being active
>> > > enough
>> > > > > and
>> > > > > > > > hence the risk of project getting orphaned is measurable.
>> > > > > > > >
>> > > > > > > > Atri
>> > > > > > > >
>> > > > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <
>> willem.jiang@gmail.com>
>> > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Atri Sharma,
>> > > > > > > > >
>> > > > > > > > > It's easier to check out the recent contribution of
>> kyuubi from
>> > > > > this
>> > > > > > > > > link[1].
>> > > > > > > > > We already identified the one person contribution issue of
>> > > kyuubi
>> > > > > > when
>> > > > > > > > > Kent came to us and we kept reminding him to put more
>> effort
>> > > into
>> > > > > > > > > building the community.
>> > > > > > > > > Last week I had a long F2F conversation with Kent, I
>> think he
>> > > > > already
>> > > > > > > > > knows about it and there have been some improvements in
>> recent
>> > > > > months
>> > > > > > > > > but there is still a long way to go if Kyuubi is accepted
>> as a
>> > > > > > podling
>> > > > > > > > > in the Apache Incubator.
>> > > > > > > > >
>> > > > > > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
>> > > > > > > > >
>> > > > > > > > > Willem Jiang
>> > > > > > > > >
>> > > > > > > > > Twitter: willemjiang
>> > > > > > > > > Weibo: 姜宁willem
>> > > > > > > > >
>> > > > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <
>> atri@apache.org>
>> > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > Hi,
>> > > > > > > > > >
>> > > > > > > > > > I took a look at commit log and seems like all the
>> commits
>> > > for
>> > > > > > last 3
>> > > > > > > > > > months have been done by a single person.
>> > > > > > > > > >
>> > > > > > > > > > How active are the remaining committers?
>> > > > > > > > > >
>> > > > > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org>
>> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Alexander Alten,
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks for your interest and +1 for Kyuubi.
>> > > > > > > > > > >
>> > > > > > > > > > > Not being very familiar with Dremel, I just took a
>> quick
>> > > look
>> > > > > at
>> > > > > > > the
>> > > > > > > > > > > profile of a very similar and exciting project, Apache
>> > > Drill.
>> > > > > If
>> > > > > > I
>> > > > > > > > > > > understand correctly, Kyuubi is built on top of the
>> compute
>> > > > > layer
>> > > > > > > > > > > (Apache Spark), while Apache Drill is built on top of
>> the
>> > > > > storage
>> > > > > > > > > > > layer.
>> > > > > > > > > > >
>> > > > > > > > > > > Apache Spark has a large user base and usage
>> scenarios all
>> > > > over
>> > > > > > the
>> > > > > > > > > > > world. When users have some basic Spark programming
>> and
>> > > > tuning
>> > > > > > > > skills,
>> > > > > > > > > > > things are easy for them. But many times, most users
>> do not
>> > > > > have
>> > > > > > > all
>> > > > > > > > > > > of these abilities when they use Spark directly or
>> > > > indirectly.
>> > > > > We
>> > > > > > > > want
>> > > > > > > > > > > to build a more easy-to-use platform on top of it to
>> help
>> > > > these
>> > > > > > > > users.
>> > > > > > > > > > > On the one hand, we will work on the interface layer
>> to
>> > > make
>> > > > it
>> > > > > > > easy
>> > > > > > > > > > > to access (of course, this effort will make our
>> project
>> > > look
>> > > > > very
>> > > > > > > > > > > similar to other projects, such as HiveServer2). On
>> the
>> > > other
>> > > > > > hand,
>> > > > > > > > we
>> > > > > > > > > > > will work under Kyuubi's multi-tenant architecture to
>> > > extend
>> > > > > our
>> > > > > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make
>> it
>> > > more
>> > > > > > > stable
>> > > > > > > > > > > and efficient in our scenario.
>> > > > > > > > > > >
>> > > > > > > > > > > What's more, we are going to add more on-prem Kyuubi
>> engine
>> > > > > > types,
>> > > > > > > > e.g.
>> > > > > > > > > > > Flink.
>> > > > > > > > > > >
>> > > > > > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一
>> 下午2:35写道：
>> > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Hey,
>> > > > > > > > > > > >
>> > > > > > > > > > > > Looks like an interesting project, competition to
>> > > > Starburst.
>> > > > > > How
>> > > > > > > > > does it
>> > > > > > > > > > > > compare to Apache Dremel?
>> > > > > > > > > > > >
>> > > > > > > > > > > > +1 (non-binding) for incubation.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks and stay safe,
>> > > > > > > > > > > > --alex
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <
>> yao@apache.org>
>> > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Dear all,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > We want to propose Kyuubi as a new Apache
>> Incubator
>> > > > > project.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift
>> JDBC/ODBC
>> > > > > server
>> > > > > > > for
>> > > > > > > > > > > > > large-scale data management, processing, and
>> analytics,
>> > > > > built
>> > > > > > > on
>> > > > > > > > > top
>> > > > > > > > > > > > > of Apache Spark and designed to support more
>> engines
>> > > > (i.e.,
>> > > > > > > > Apache
>> > > > > > > > > > > > > Flink). We are aiming to make Kyuubi an
>> > > "out-of-the-box"
>> > > > > tool
>> > > > > > > for
>> > > > > > > > > data
>> > > > > > > > > > > > > warehouses and data lakes. Kyuubi concentrates on
>> > > > improving
>> > > > > > the
>> > > > > > > > > > > > > experience for non-Spark users, making it easy
>> for them
>> > > > to
>> > > > > > use
>> > > > > > > > > Spark
>> > > > > > > > > > > > > as a service and focus on business data of their
>> own,
>> > > not
>> > > > > > where
>> > > > > > > > it
>> > > > > > > > > > > > > stores, how it computes. NetEase has open-sourced
>> > > Kyuubi
>> > > > > > since
>> > > > > > > > > 2018,
>> > > > > > > > > > > > > https://github.com/NetEase/kyuubi.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff
>> Zhang, Duo
>> > > > > > Zhang,
>> > > > > > > > > Akira
>> > > > > > > > > > > > > Ajisaka for being the mentors of the Kyuubi
>> project and
>> > > > > > helping
>> > > > > > > > us
>> > > > > > > > > set
>> > > > > > > > > > > > > up this proposal. The proposal can be found at
>> > > > > > > > > > > > >
>> > > > > > > > >
>> > > > >
>> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
>> > > > > > .
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Looking forward to your feedback and thank you
>> very
>> > > much.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > Kent Yao
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > >
>> > > > >
>> ---------------------------------------------------------------------
>> > > > > > > > > > > > > To unsubscribe, e-mail:
>> > > > > > > general-unsubscribe@incubator.apache.org
>> > > > > > > > > > > > > For additional commands, e-mail:
>> > > > > > > > general-help@incubator.apache.org
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > >
>> > > > > > > > > > > Kent Yao
>> > > > > > > > > > > @ Data Science Center, Hangzhou Research Institute,
>> NetEase
>> > > > > Corp.
>> > > > > > > > > > > a spark enthusiast
>> > > > > > > > > > > kyuubiis a unified multi-tenant JDBC interface for
>> > > > large-scale
>> > > > > > data
>> > > > > > > > > > > processing and analytics, built on top of Apache
>> Spark.
>> > > > > > > > > > >
>> > > > > > > > > > > spark-authorizerA Spark SQL extension which provides
>> SQL
>> > > > > Standard
>> > > > > > > > > > > Authorization for Apache Spark.
>> > > > > > > > > > > spark-postgres A library for reading data from and
>> > > > transferring
>> > > > > > > data
>> > > > > > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames,
>> > > > 10~100x
>> > > > > > > > faster.
>> > > > > > > > > > > itatchiA library that brings useful functions from
>> various
>> > > > > modern
>> > > > > > > > > > > database management systems to Apache Spark.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > >
>> > > ---------------------------------------------------------------------
>> > > > > > > > > > > To unsubscribe, e-mail:
>> > > > > general-unsubscribe@incubator.apache.org
>> > > > > > > > > > > For additional commands, e-mail:
>> > > > > > general-help@incubator.apache.org
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > >
>> ---------------------------------------------------------------------
>> > > > > > > > > To unsubscribe, e-mail:
>> > > general-unsubscribe@incubator.apache.org
>> > > > > > > > > For additional commands, e-mail:
>> > > > general-help@incubator.apache.org
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > >
>> > > > > > *Kent Yao *
>> > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
>> Corp.
>> > > > > > a spark enthusiast
>> > > > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified
>> multi-tenant
>> > > > > JDBC
>> > > > > > interface for large-scale data processing and analytics, built
>> on top
>> > > > of
>> > > > > > Apache
>> > > > > > Spark <http://spark.apache.org/>.
>> > > > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer
>> >A
>> > > Spark
>> > > > > SQL
>> > > > > > extension which provides SQL Standard Authorization for Apache
>> Spark
>> > > > > > <http://spark.apache.org/>.
>> > > > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
>> > > library
>> > > > > for
>> > > > > > reading data from and transferring data to Postgres / Greenplum
>> with
>> > > > > Spark
>> > > > > > SQL and DataFrames, 10~100x faster.
>> > > > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A
>> library
>> > > that
>> > > > > > brings useful functions from various modern database management
>> > > > > > systems to Apache
>> > > > > > Spark <http://spark.apache.org/>.
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > *Kent Yao *
>> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
>> > > > a spark enthusiast
>> > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified
>> multi-tenant
>> > > JDBC
>> > > > interface for large-scale data processing and analytics, built on
>> top of
>> > > > Apache
>> > > > Spark <http://spark.apache.org/>.
>> > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
>> Spark
>> > > SQL
>> > > > extension which provides SQL Standard Authorization for Apache Spark
>> > > > <http://spark.apache.org/>.
>> > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
>> library
>> > > for
>> > > > reading data from and transferring data to Postgres / Greenplum with
>> > > Spark
>> > > > SQL and DataFrames, 10~100x faster.
>> > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library
>> that
>> > > > brings useful functions from various modern database management
>> > > > systems to Apache
>> > > > Spark <http://spark.apache.org/>.
>> > > >
>> > >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>> --
> Thanks,
> Ming Wen, Apache APISIX PMC Chair
> Twitter: _WenMing
>
--
Thanks,
Ming Wen, Apache APISIX PMC Chair
Twitter: _WenMing
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Ming Wen <we...@apache.org>.
hi, Kent,
From 2018 to 2020, kuyybi is not an active project, most of the time there
is only one contributor per month. But in 2021, there will be 8 active
contributors every month. What is the reason for this?
Willem Jiang <wi...@gmail.com>于2021年6月3日 周四下午12:53写道：
> I think we can revisit the PPMC part and have more discussion with the
> initial committers.
> It's a good chance to let them know better about how to run an open
> source project in Apache Way.
>
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Thu, Jun 3, 2021 at 11:52 AM Atri Sharma <at...@apache.org> wrote:
> >
> > I would like to hear from Kyuubi's mentors on the topic before reaching a
> > conclusion.
> >
> > On Thu, 3 Jun 2021, 05:40 Sheng Wu, <wu...@gmail.com> wrote:
> >
> > > Kent
> > >
> > > Thanks.
> > > The proposal is generally better this time.
> > >
> > > If your initial committers and your mentors agree, you are ready to
> embrace
> > > the ASF culture, I am good with that.
> > > Just from my experience, your community looks a little too young, like
> > > Atri's concern, which is reasonable.
> > > But I would like to trust you and your mentors' judgment and
> determination.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Kent Yao <ya...@apache.org> 于2021年6月3日周四 上午12:54写道：
> > >
> > > > Hi Sheng,
> > > >
> > > > Truly sorry for the format issue.
> > > >
> > > > Please trust me I did not mean to hurt anybody and push any
> contributor
> > > > away. We drafted this proposal together based on the New Podling
> > > Proposal.
> > > > We never tried to use PPMC and committer to measure whose
> contribution is
> > > > higher. This mistake has been fixed in the proposal.
> > > >
> > > > About releases, we will follow
> > > > https://www.apache.org/legal/release-policy.html and the guide of
> our
> > > > mentors. We will also take the initiative to learn other documents.
> > > >
> > > > > This is not a block, the point I mentioned this is, you will need
> more
> > > > people.
> > > >
> > > > We will keep working on this.
> > > >
> > > > Bests,
> > > >
> > > > Kent Yao
> > > >
> > > >
> > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
> > > >
> > > > > Hi Kent
> > > > >
> > > > > Thanks for replying, but to be honest, your mail format is chaos.
> > > > > Take a look at bellowing, which is what I saw in the gmail.
> > > > >
> > > > > This may be an issue of your mail client or some else.
> > > > >
> > > > > > The current #.2 - 5 top
> > > > > > contributors will stay as committers, and they will join the PPMC
> > > > during
> > > > > > incubation.
> > > > >
> > > > > PPMC usually is the default for all initial committers, considering
> > > your
> > > > > community is still small and not very active.
> > > > > You should invite all people you trusted to join, rather than
> pushing
> > > > them
> > > > > away.
> > > > > This is not just a too-high bar issue. Excluding 4 out of 5 is
> strange.
> > > > > I can't guess what is the root cause, why the other 4 PPMC members
> > > don't
> > > > > touch codes but are trusted completely. Meanwhile, people with
> easily
> > > > > measurable code contributions are just as committers?
> > > > > We(ASF) always remind the community to recognize no-code
> contribution,
> > > > but
> > > > > in this case, it seems the opposite? Don't recognize code
> contributors
> > > > > enough?
> > > > > Please reach your mentors, this seems not a typical ASF style. If
> you
> > > > have
> > > > > other people already involved in the ASF, asking them too.
> > > > >
> > > > > And from your description of the release process, I think you
> should
> > > talk
> > > > > more with your mentors.
> > > > > Such as what is a qualified ASF release, what should be checked.
> > > > > This is not a block, the point I mentioned this is, you will need
> more
> > > > > people.
> > > > >
> > > > >
> > > > > Sheng Wu 吴晟
> > > > > Twitter, wusheng1108
> > > > >
> > > > >
> > > > > Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
> > > > >
> > > > > > Hi Sheng,
> > > > > >
> > > > > > Thanks for your detailed check and valuable comments. They are
> > > > > > accommodating. I have addressed some of them directly in the
> > > proposal.
> > > > > > Please also check the table below for some Q&As.
> > > > > > QA
> > > > > > You listed only 5 core maintainers, but clearly only one has
> > > dominated
> > > > > > contributions. The distortion and skewness of the contributions
> are
> > > > > mainly
> > > > > > caused by the way how we merge PRs changed(mentioned in another
> email
> > > > on
> > > > > > the list).
> > > > > > I agree with Atri, orphaned is measurable. It is better to adjust
> > > this
> > > > > > statement a little. Address the comment to notify the risk in the
> > > > Kyuubi
> > > > > > proposal.
> > > > > > Also, with this fact, the core maintainers should notice whether
> this
> > > > > small
> > > > > > group of people to make sure your features iteration, bug fix,
> and
> > > > > release
> > > > > > process working by following ASF's requirements.This would not
> be an
> > > > easy
> > > > > > job. Currently, we learn the release process from some Apache
> > > projects,
> > > > > > using major.minor.patch for versioning. As we can see, since
> Kyuubi
> > > > 1.0,
> > > > > it
> > > > > > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug
> fixes
> > > > > until
> > > > > > EOL. And it takes us about two months to finish a minor release
> with
> > > 3
> > > > ~
> > > > > 5
> > > > > > big improvements and various small features/bug fixes. We also
> will
> > > > start
> > > > > > with some release candidates for preview and test before the
> final.
> > > > This
> > > > > is
> > > > > > how we develop and release Kyuubi for now, but we do have a long
> way
> > > to
> > > > > go
> > > > > > and learn to match the ASF requirement.
> > > > > > And your committer list is very strange... 1. Why some of them
> are
> > > PPMC
> > > > > and
> > > > > > others are committers? Where this hierarchy comes from? The
> current
> > > > PPMC
> > > > > > candidates help us a lot build the Kyuubi project and community,
> not
> > > > > > directly to the codebase but as mentors(before Apache). Most of
> them
> > > > are
> > > > > > from Apache mature TLPs, which have close relationships with
> Kyuubi.
> > > > They
> > > > > > have sufficient Apache Way experience to guide us to become a
> more
> > > > > > Apache-like project before submitting this proposal in many
> ways. We
> > > > will
> > > > > > continue to promote more contributors to the PPMC/committer
> based on
> > > > > their
> > > > > > code or non-code contributions during incubation.
> > > > > > 2. Your core maintainers have 5, which only 4 PPMC? The current
> #.2
> > > - 5
> > > > > top
> > > > > > contributors will stay as committers, and they will join the PPMC
> > > > during
> > > > > > incubation.
> > > > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> > > have
> > > > 5
> > > > > > core developers? I didn't notice that 400 LOC is a bar for core
> > > > > developer,
> > > > > > will change 5 to 4 core developers accordingly
> > > > > > I think you don't have enough no-salaried developers, am I
> missing
> > > > > > anything? Yes.
> > > > > > Changed in the proposal, most of the contributors are salaried,
> and
> > > > some
> > > > > of
> > > > > > them are willing to help in the spare time. Thanks
> > > > > > This project belongs to NetEase Inc. clearly, why NO SGA? Added
> SGA
> > > in
> > > > > > Proposal. All work here is approved by NetEase and can be
> forward.
> > > > > > At last, you are asking about
> > > > github.com/apache/incubator-kyuubi-shaded
> > > > > > repo,
> > > > > > what does this represent? I can only see you have only one source
> > > repo
> > > > > and
> > > > > > a doc repo. Removed this requirement in the proposal, and we will
> > > apply
> > > > > it
> > > > > > from IPMC if we do need this in the future.
> > > > > > ------------------------------
> > > > > >
> > > > > > Bests
> > > > > >
> > > > > > Kent Yao
> > > > > >
> > > > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > I read the contributor list and the proposal
> > > > > > > > The risk of the Kyuubi project being abandoned is minimal.
> Many
> > > > > > > organizations are using Kyuubi to build critical big data
> pipelines
> > > > and
> > > > > > > willing to help develop Kyuubi's community if it becomes an ASF
> > > > > project.
> > > > > > >
> > > > > > > I think the team should evaluate the status again. You listed
> only
> > > 5
> > > > > core
> > > > > > > maintainers, but clearly only one has dominated contributions.
> I
> > > > agree
> > > > > > with
> > > > > > > Atri, orphaned is measurable.
> > > > > > > It is better to adjust this statement a little. Also, with this
> > > fact,
> > > > > the
> > > > > > > core maintainers should notice whether this small group of
> people
> > > to
> > > > > make
> > > > > > > sure your features iteration, bug fix, and release process
> working
> > > by
> > > > > > > following ASF's requirements.
> > > > > > > This would not be an easy job.
> > > > > > >
> > > > > > > And your committer list is very strange
> > > > > > > ```
> > > > > > > Kent Yao (yao@apache.org) (PPMC)
> > > > > > > Wenchen Fan (wenchen@apache.org) (PPMC)
> > > > > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > > > > > > Xun Liu (liuxun@apache.org) (PPMC)
> > > > > > > Fei Wang (cn.feiwang@gmail.com)
> > > > > > > Ulysses You (ulyssesyou18@gmail.com)
> > > > > > > Cheng Pan (pan3793@gmail.com)
> > > > > > > Hongxiang Jiang (jhx1008@gmail.com)
> > > > > > > Zen Jin(xinjingziranchan@gmail.com)
> > > > > > > ```
> > > > > > > 1. Why some of them are PPMC and others are committers? Where
> this
> > > > > > > hierarchy comes from?
> > > > > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even
> hierarchy
> > > > > makes
> > > > > > > sense, most likely not)
> > > > > > > 3. I think only the top 4 have over 400 LOC, why you declared
> you
> > > > have
> > > > > 5
> > > > > > > core developers?
> > > > > > >
> > > > > > > Another re-evaluation request is about `Reliance on Salaried
> > > > > Developers:`
> > > > > > > > They are all passionate about the project, and we are
> confident
> > > > that
> > > > > > the
> > > > > > > project will continue even if no salaried developers
> contribute to
> > > > the
> > > > > > > project.
> > > > > > >
> > > > > > > I think you don't have enough no-salaried developers, am I
> missing
> > > > > > > anything?
> > > > > > >
> > > > > > > About IP clearance, `Initial Source and Intellectual Property
> > > > > Submission
> > > > > > > Plan`
> > > > > > > > As soon as Kyuubi is approved to join Apache Incubator, our
> > > initial
> > > > > > > committers will submit iCLA(s) and CCLA(s). The codebase is
> already
> > > > > > > licensed under the Apache License 2.0.
> > > > > > >
> > > > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
> > > > > > >
> > > > > > > At last, you are asking about `
> > > > > > github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > > > repo, what does this represent? I can only see you have only
> one
> > > > source
> > > > > > > repo and a doc repo.
> > > > > > >
> > > > > > > Sheng Wu 吴晟
> > > > > > > Twitter, wusheng1108
> > > > > > >
> > > > > > >
> > > > > > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > > I looked at the link you advised on and if I am reading it
> > > > correctly,
> > > > > > top
> > > > > > > > committer has 630ish commits and the next committer has 73
> > > commits?
> > > > > > > >
> > > > > > > > IMO this is an indication of the community not being active
> > > enough
> > > > > and
> > > > > > > > hence the risk of project getting orphaned is measurable.
> > > > > > > >
> > > > > > > > Atri
> > > > > > > >
> > > > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <
> willem.jiang@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Atri Sharma,
> > > > > > > > >
> > > > > > > > > It's easier to check out the recent contribution of kyuubi
> from
> > > > > this
> > > > > > > > > link[1].
> > > > > > > > > We already identified the one person contribution issue of
> > > kyuubi
> > > > > > when
> > > > > > > > > Kent came to us and we kept reminding him to put more
> effort
> > > into
> > > > > > > > > building the community.
> > > > > > > > > Last week I had a long F2F conversation with Kent, I think
> he
> > > > > already
> > > > > > > > > knows about it and there have been some improvements in
> recent
> > > > > months
> > > > > > > > > but there is still a long way to go if Kyuubi is accepted
> as a
> > > > > > podling
> > > > > > > > > in the Apache Incubator.
> > > > > > > > >
> > > > > > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > > > > > > >
> > > > > > > > > Willem Jiang
> > > > > > > > >
> > > > > > > > > Twitter: willemjiang
> > > > > > > > > Weibo: 姜宁willem
> > > > > > > > >
> > > > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <
> atri@apache.org>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I took a look at commit log and seems like all the
> commits
> > > for
> > > > > > last 3
> > > > > > > > > > months have been done by a single person.
> > > > > > > > > >
> > > > > > > > > > How active are the remaining committers?
> > > > > > > > > >
> > > > > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org>
> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Alexander Alten,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > > > > > > >
> > > > > > > > > > > Not being very familiar with Dremel, I just took a
> quick
> > > look
> > > > > at
> > > > > > > the
> > > > > > > > > > > profile of a very similar and exciting project, Apache
> > > Drill.
> > > > > If
> > > > > > I
> > > > > > > > > > > understand correctly, Kyuubi is built on top of the
> compute
> > > > > layer
> > > > > > > > > > > (Apache Spark), while Apache Drill is built on top of
> the
> > > > > storage
> > > > > > > > > > > layer.
> > > > > > > > > > >
> > > > > > > > > > > Apache Spark has a large user base and usage scenarios
> all
> > > > over
> > > > > > the
> > > > > > > > > > > world. When users have some basic Spark programming and
> > > > tuning
> > > > > > > > skills,
> > > > > > > > > > > things are easy for them. But many times, most users
> do not
> > > > > have
> > > > > > > all
> > > > > > > > > > > of these abilities when they use Spark directly or
> > > > indirectly.
> > > > > We
> > > > > > > > want
> > > > > > > > > > > to build a more easy-to-use platform on top of it to
> help
> > > > these
> > > > > > > > users.
> > > > > > > > > > > On the one hand, we will work on the interface layer to
> > > make
> > > > it
> > > > > > > easy
> > > > > > > > > > > to access (of course, this effort will make our project
> > > look
> > > > > very
> > > > > > > > > > > similar to other projects, such as HiveServer2). On the
> > > other
> > > > > > hand,
> > > > > > > > we
> > > > > > > > > > > will work under Kyuubi's multi-tenant architecture to
> > > extend
> > > > > our
> > > > > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make
> it
> > > more
> > > > > > > stable
> > > > > > > > > > > and efficient in our scenario.
> > > > > > > > > > >
> > > > > > > > > > > What's more, we are going to add more on-prem Kyuubi
> engine
> > > > > > types,
> > > > > > > > e.g.
> > > > > > > > > > > Flink.
> > > > > > > > > > >
> > > > > > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一
> 下午2:35写道：
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hey,
> > > > > > > > > > > >
> > > > > > > > > > > > Looks like an interesting project, competition to
> > > > Starburst.
> > > > > > How
> > > > > > > > > does it
> > > > > > > > > > > > compare to Apache Dremel?
> > > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding) for incubation.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks and stay safe,
> > > > > > > > > > > > --alex
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <yao@apache.org
> >
> > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Dear all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > We want to propose Kyuubi as a new Apache Incubator
> > > > > project.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift
> JDBC/ODBC
> > > > > server
> > > > > > > for
> > > > > > > > > > > > > large-scale data management, processing, and
> analytics,
> > > > > built
> > > > > > > on
> > > > > > > > > top
> > > > > > > > > > > > > of Apache Spark and designed to support more
> engines
> > > > (i.e.,
> > > > > > > > Apache
> > > > > > > > > > > > > Flink). We are aiming to make Kyuubi an
> > > "out-of-the-box"
> > > > > tool
> > > > > > > for
> > > > > > > > > data
> > > > > > > > > > > > > warehouses and data lakes. Kyuubi concentrates on
> > > > improving
> > > > > > the
> > > > > > > > > > > > > experience for non-Spark users, making it easy for
> them
> > > > to
> > > > > > use
> > > > > > > > > Spark
> > > > > > > > > > > > > as a service and focus on business data of their
> own,
> > > not
> > > > > > where
> > > > > > > > it
> > > > > > > > > > > > > stores, how it computes. NetEase has open-sourced
> > > Kyuubi
> > > > > > since
> > > > > > > > > 2018,
> > > > > > > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff
> Zhang, Duo
> > > > > > Zhang,
> > > > > > > > > Akira
> > > > > > > > > > > > > Ajisaka for being the mentors of the Kyuubi
> project and
> > > > > > helping
> > > > > > > > us
> > > > > > > > > set
> > > > > > > > > > > > > up this proposal. The proposal can be found at
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> > > > > > .
> > > > > > > > > > > > >
> > > > > > > > > > > > > Looking forward to your feedback and thank you very
> > > much.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Kent Yao
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > > general-unsubscribe@incubator.apache.org
> > > > > > > > > > > > > For additional commands, e-mail:
> > > > > > > > general-help@incubator.apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > > Kent Yao
> > > > > > > > > > > @ Data Science Center, Hangzhou Research Institute,
> NetEase
> > > > > Corp.
> > > > > > > > > > > a spark enthusiast
> > > > > > > > > > > kyuubiis a unified multi-tenant JDBC interface for
> > > > large-scale
> > > > > > data
> > > > > > > > > > > processing and analytics, built on top of Apache Spark.
> > > > > > > > > > >
> > > > > > > > > > > spark-authorizerA Spark SQL extension which provides
> SQL
> > > > > Standard
> > > > > > > > > > > Authorization for Apache Spark.
> > > > > > > > > > > spark-postgres A library for reading data from and
> > > > transferring
> > > > > > > data
> > > > > > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames,
> > > > 10~100x
> > > > > > > > faster.
> > > > > > > > > > > itatchiA library that brings useful functions from
> various
> > > > > modern
> > > > > > > > > > > database management systems to Apache Spark.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail:
> > > > > general-unsubscribe@incubator.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> > > > > > general-help@incubator.apache.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail:
> > > general-unsubscribe@incubator.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > *Kent Yao *
> > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > > > a spark enthusiast
> > > > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified
> multi-tenant
> > > > > JDBC
> > > > > > interface for large-scale data processing and analytics, built
> on top
> > > > of
> > > > > > Apache
> > > > > > Spark <http://spark.apache.org/>.
> > > > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
> > > Spark
> > > > > SQL
> > > > > > extension which provides SQL Standard Authorization for Apache
> Spark
> > > > > > <http://spark.apache.org/>.
> > > > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
> > > library
> > > > > for
> > > > > > reading data from and transferring data to Postgres / Greenplum
> with
> > > > > Spark
> > > > > > SQL and DataFrames, 10~100x faster.
> > > > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library
> > > that
> > > > > > brings useful functions from various modern database management
> > > > > > systems to Apache
> > > > > > Spark <http://spark.apache.org/>.
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Kent Yao *
> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > a spark enthusiast
> > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> > > JDBC
> > > > interface for large-scale data processing and analytics, built on
> top of
> > > > Apache
> > > > Spark <http://spark.apache.org/>.
> > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
> Spark
> > > SQL
> > > > extension which provides SQL Standard Authorization for Apache Spark
> > > > <http://spark.apache.org/>.
> > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
> library
> > > for
> > > > reading data from and transferring data to Postgres / Greenplum with
> > > Spark
> > > > SQL and DataFrames, 10~100x faster.
> > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library
> that
> > > > brings useful functions from various modern database management
> > > > systems to Apache
> > > > Spark <http://spark.apache.org/>.
> > > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
> --
Thanks,
Ming Wen, Apache APISIX PMC Chair
Twitter: _WenMing
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Willem Jiang <wi...@gmail.com>.
I think we can revisit the PPMC part and have more discussion with the
initial committers.
It's a good chance to let them know better about how to run an open
source project in Apache Way.
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Thu, Jun 3, 2021 at 11:52 AM Atri Sharma <at...@apache.org> wrote:
>
> I would like to hear from Kyuubi's mentors on the topic before reaching a
> conclusion.
>
> On Thu, 3 Jun 2021, 05:40 Sheng Wu, <wu...@gmail.com> wrote:
>
> > Kent
> >
> > Thanks.
> > The proposal is generally better this time.
> >
> > If your initial committers and your mentors agree, you are ready to embrace
> > the ASF culture, I am good with that.
> > Just from my experience, your community looks a little too young, like
> > Atri's concern, which is reasonable.
> > But I would like to trust you and your mentors' judgment and determination.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Kent Yao <ya...@apache.org> 于2021年6月3日周四 上午12:54写道：
> >
> > > Hi Sheng,
> > >
> > > Truly sorry for the format issue.
> > >
> > > Please trust me I did not mean to hurt anybody and push any contributor
> > > away. We drafted this proposal together based on the New Podling
> > Proposal.
> > > We never tried to use PPMC and committer to measure whose contribution is
> > > higher. This mistake has been fixed in the proposal.
> > >
> > > About releases, we will follow
> > > https://www.apache.org/legal/release-policy.html and the guide of our
> > > mentors. We will also take the initiative to learn other documents.
> > >
> > > > This is not a block, the point I mentioned this is, you will need more
> > > people.
> > >
> > > We will keep working on this.
> > >
> > > Bests,
> > >
> > > Kent Yao
> > >
> > >
> > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
> > >
> > > > Hi Kent
> > > >
> > > > Thanks for replying, but to be honest, your mail format is chaos.
> > > > Take a look at bellowing, which is what I saw in the gmail.
> > > >
> > > > This may be an issue of your mail client or some else.
> > > >
> > > > > The current #.2 - 5 top
> > > > > contributors will stay as committers, and they will join the PPMC
> > > during
> > > > > incubation.
> > > >
> > > > PPMC usually is the default for all initial committers, considering
> > your
> > > > community is still small and not very active.
> > > > You should invite all people you trusted to join, rather than pushing
> > > them
> > > > away.
> > > > This is not just a too-high bar issue. Excluding 4 out of 5 is strange.
> > > > I can't guess what is the root cause, why the other 4 PPMC members
> > don't
> > > > touch codes but are trusted completely. Meanwhile, people with easily
> > > > measurable code contributions are just as committers?
> > > > We(ASF) always remind the community to recognize no-code contribution,
> > > but
> > > > in this case, it seems the opposite? Don't recognize code contributors
> > > > enough?
> > > > Please reach your mentors, this seems not a typical ASF style. If you
> > > have
> > > > other people already involved in the ASF, asking them too.
> > > >
> > > > And from your description of the release process, I think you should
> > talk
> > > > more with your mentors.
> > > > Such as what is a qualified ASF release, what should be checked.
> > > > This is not a block, the point I mentioned this is, you will need more
> > > > people.
> > > >
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
> > > >
> > > > > Hi Sheng,
> > > > >
> > > > > Thanks for your detailed check and valuable comments. They are
> > > > > accommodating. I have addressed some of them directly in the
> > proposal.
> > > > > Please also check the table below for some Q&As.
> > > > > QA
> > > > > You listed only 5 core maintainers, but clearly only one has
> > dominated
> > > > > contributions. The distortion and skewness of the contributions are
> > > > mainly
> > > > > caused by the way how we merge PRs changed(mentioned in another email
> > > on
> > > > > the list).
> > > > > I agree with Atri, orphaned is measurable. It is better to adjust
> > this
> > > > > statement a little. Address the comment to notify the risk in the
> > > Kyuubi
> > > > > proposal.
> > > > > Also, with this fact, the core maintainers should notice whether this
> > > > small
> > > > > group of people to make sure your features iteration, bug fix, and
> > > > release
> > > > > process working by following ASF's requirements.This would not be an
> > > easy
> > > > > job. Currently, we learn the release process from some Apache
> > projects,
> > > > > using major.minor.patch for versioning. As we can see, since Kyuubi
> > > 1.0,
> > > > it
> > > > > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes
> > > > until
> > > > > EOL. And it takes us about two months to finish a minor release with
> > 3
> > > ~
> > > > 5
> > > > > big improvements and various small features/bug fixes. We also will
> > > start
> > > > > with some release candidates for preview and test before the final.
> > > This
> > > > is
> > > > > how we develop and release Kyuubi for now, but we do have a long way
> > to
> > > > go
> > > > > and learn to match the ASF requirement.
> > > > > And your committer list is very strange... 1. Why some of them are
> > PPMC
> > > > and
> > > > > others are committers? Where this hierarchy comes from? The current
> > > PPMC
> > > > > candidates help us a lot build the Kyuubi project and community, not
> > > > > directly to the codebase but as mentors(before Apache). Most of them
> > > are
> > > > > from Apache mature TLPs, which have close relationships with Kyuubi.
> > > They
> > > > > have sufficient Apache Way experience to guide us to become a more
> > > > > Apache-like project before submitting this proposal in many ways. We
> > > will
> > > > > continue to promote more contributors to the PPMC/committer based on
> > > > their
> > > > > code or non-code contributions during incubation.
> > > > > 2. Your core maintainers have 5, which only 4 PPMC? The current #.2
> > - 5
> > > > top
> > > > > contributors will stay as committers, and they will join the PPMC
> > > during
> > > > > incubation.
> > > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> > have
> > > 5
> > > > > core developers? I didn't notice that 400 LOC is a bar for core
> > > > developer,
> > > > > will change 5 to 4 core developers accordingly
> > > > > I think you don't have enough no-salaried developers, am I missing
> > > > > anything? Yes.
> > > > > Changed in the proposal, most of the contributors are salaried, and
> > > some
> > > > of
> > > > > them are willing to help in the spare time. Thanks
> > > > > This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA
> > in
> > > > > Proposal. All work here is approved by NetEase and can be forward.
> > > > > At last, you are asking about
> > > github.com/apache/incubator-kyuubi-shaded
> > > > > repo,
> > > > > what does this represent? I can only see you have only one source
> > repo
> > > > and
> > > > > a doc repo. Removed this requirement in the proposal, and we will
> > apply
> > > > it
> > > > > from IPMC if we do need this in the future.
> > > > > ------------------------------
> > > > >
> > > > > Bests
> > > > >
> > > > > Kent Yao
> > > > >
> > > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > I read the contributor list and the proposal
> > > > > > > The risk of the Kyuubi project being abandoned is minimal. Many
> > > > > > organizations are using Kyuubi to build critical big data pipelines
> > > and
> > > > > > willing to help develop Kyuubi's community if it becomes an ASF
> > > > project.
> > > > > >
> > > > > > I think the team should evaluate the status again. You listed only
> > 5
> > > > core
> > > > > > maintainers, but clearly only one has dominated contributions. I
> > > agree
> > > > > with
> > > > > > Atri, orphaned is measurable.
> > > > > > It is better to adjust this statement a little. Also, with this
> > fact,
> > > > the
> > > > > > core maintainers should notice whether this small group of people
> > to
> > > > make
> > > > > > sure your features iteration, bug fix, and release process working
> > by
> > > > > > following ASF's requirements.
> > > > > > This would not be an easy job.
> > > > > >
> > > > > > And your committer list is very strange
> > > > > > ```
> > > > > > Kent Yao (yao@apache.org) (PPMC)
> > > > > > Wenchen Fan (wenchen@apache.org) (PPMC)
> > > > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > > > > > Xun Liu (liuxun@apache.org) (PPMC)
> > > > > > Fei Wang (cn.feiwang@gmail.com)
> > > > > > Ulysses You (ulyssesyou18@gmail.com)
> > > > > > Cheng Pan (pan3793@gmail.com)
> > > > > > Hongxiang Jiang (jhx1008@gmail.com)
> > > > > > Zen Jin(xinjingziranchan@gmail.com)
> > > > > > ```
> > > > > > 1. Why some of them are PPMC and others are committers? Where this
> > > > > > hierarchy comes from?
> > > > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy
> > > > makes
> > > > > > sense, most likely not)
> > > > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> > > have
> > > > 5
> > > > > > core developers?
> > > > > >
> > > > > > Another re-evaluation request is about `Reliance on Salaried
> > > > Developers:`
> > > > > > > They are all passionate about the project, and we are confident
> > > that
> > > > > the
> > > > > > project will continue even if no salaried developers contribute to
> > > the
> > > > > > project.
> > > > > >
> > > > > > I think you don't have enough no-salaried developers, am I missing
> > > > > > anything?
> > > > > >
> > > > > > About IP clearance, `Initial Source and Intellectual Property
> > > > Submission
> > > > > > Plan`
> > > > > > > As soon as Kyuubi is approved to join Apache Incubator, our
> > initial
> > > > > > committers will submit iCLA(s) and CCLA(s). The codebase is already
> > > > > > licensed under the Apache License 2.0.
> > > > > >
> > > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
> > > > > >
> > > > > > At last, you are asking about `
> > > > > github.com/apache/incubator-kyuubi-shaded`
> > <http://github.com/apache/incubator-kyuubi-shaded>
> > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > > repo, what does this represent? I can only see you have only one
> > > source
> > > > > > repo and a doc repo.
> > > > > >
> > > > > > Sheng Wu 吴晟
> > > > > > Twitter, wusheng1108
> > > > > >
> > > > > >
> > > > > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > I looked at the link you advised on and if I am reading it
> > > correctly,
> > > > > top
> > > > > > > committer has 630ish commits and the next committer has 73
> > commits?
> > > > > > >
> > > > > > > IMO this is an indication of the community not being active
> > enough
> > > > and
> > > > > > > hence the risk of project getting orphaned is measurable.
> > > > > > >
> > > > > > > Atri
> > > > > > >
> > > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Atri Sharma,
> > > > > > > >
> > > > > > > > It's easier to check out the recent contribution of kyuubi from
> > > > this
> > > > > > > > link[1].
> > > > > > > > We already identified the one person contribution issue of
> > kyuubi
> > > > > when
> > > > > > > > Kent came to us and we kept reminding him to put more effort
> > into
> > > > > > > > building the community.
> > > > > > > > Last week I had a long F2F conversation with Kent, I think he
> > > > already
> > > > > > > > knows about it and there have been some improvements in recent
> > > > months
> > > > > > > > but there is still a long way to go if Kyuubi is accepted as a
> > > > > podling
> > > > > > > > in the Apache Incubator.
> > > > > > > >
> > > > > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > > > > > >
> > > > > > > > Willem Jiang
> > > > > > > >
> > > > > > > > Twitter: willemjiang
> > > > > > > > Weibo: 姜宁willem
> > > > > > > >
> > > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I took a look at commit log and seems like all the commits
> > for
> > > > > last 3
> > > > > > > > > months have been done by a single person.
> > > > > > > > >
> > > > > > > > > How active are the remaining committers?
> > > > > > > > >
> > > > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Alexander Alten,
> > > > > > > > > >
> > > > > > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > > > > > >
> > > > > > > > > > Not being very familiar with Dremel, I just took a quick
> > look
> > > > at
> > > > > > the
> > > > > > > > > > profile of a very similar and exciting project, Apache
> > Drill.
> > > > If
> > > > > I
> > > > > > > > > > understand correctly, Kyuubi is built on top of the compute
> > > > layer
> > > > > > > > > > (Apache Spark), while Apache Drill is built on top of the
> > > > storage
> > > > > > > > > > layer.
> > > > > > > > > >
> > > > > > > > > > Apache Spark has a large user base and usage scenarios all
> > > over
> > > > > the
> > > > > > > > > > world. When users have some basic Spark programming and
> > > tuning
> > > > > > > skills,
> > > > > > > > > > things are easy for them. But many times, most users do not
> > > > have
> > > > > > all
> > > > > > > > > > of these abilities when they use Spark directly or
> > > indirectly.
> > > > We
> > > > > > > want
> > > > > > > > > > to build a more easy-to-use platform on top of it to help
> > > these
> > > > > > > users.
> > > > > > > > > > On the one hand, we will work on the interface layer to
> > make
> > > it
> > > > > > easy
> > > > > > > > > > to access (of course, this effort will make our project
> > look
> > > > very
> > > > > > > > > > similar to other projects, such as HiveServer2). On the
> > other
> > > > > hand,
> > > > > > > we
> > > > > > > > > > will work under Kyuubi's multi-tenant architecture to
> > extend
> > > > our
> > > > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it
> > more
> > > > > > stable
> > > > > > > > > > and efficient in our scenario.
> > > > > > > > > >
> > > > > > > > > > What's more, we are going to add more on-prem Kyuubi engine
> > > > > types,
> > > > > > > e.g.
> > > > > > > > > > Flink.
> > > > > > > > > >
> > > > > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hey,
> > > > > > > > > > >
> > > > > > > > > > > Looks like an interesting project, competition to
> > > Starburst.
> > > > > How
> > > > > > > > does it
> > > > > > > > > > > compare to Apache Dremel?
> > > > > > > > > > >
> > > > > > > > > > > +1 (non-binding) for incubation.
> > > > > > > > > > >
> > > > > > > > > > > Thanks and stay safe,
> > > > > > > > > > > --alex
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org>
> > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Dear all,
> > > > > > > > > > > >
> > > > > > > > > > > > We want to propose Kyuubi as a new Apache Incubator
> > > > project.
> > > > > > > > > > > >
> > > > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC
> > > > server
> > > > > > for
> > > > > > > > > > > > large-scale data management, processing, and analytics,
> > > > built
> > > > > > on
> > > > > > > > top
> > > > > > > > > > > > of Apache Spark and designed to support more engines
> > > (i.e.,
> > > > > > > Apache
> > > > > > > > > > > > Flink). We are aiming to make Kyuubi an
> > "out-of-the-box"
> > > > tool
> > > > > > for
> > > > > > > > data
> > > > > > > > > > > > warehouses and data lakes. Kyuubi concentrates on
> > > improving
> > > > > the
> > > > > > > > > > > > experience for non-Spark users, making it easy for them
> > > to
> > > > > use
> > > > > > > > Spark
> > > > > > > > > > > > as a service and focus on business data of their own,
> > not
> > > > > where
> > > > > > > it
> > > > > > > > > > > > stores, how it computes. NetEase has open-sourced
> > Kyuubi
> > > > > since
> > > > > > > > 2018,
> > > > > > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > > > > > >
> > > > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo
> > > > > Zhang,
> > > > > > > > Akira
> > > > > > > > > > > > Ajisaka for being the mentors of the Kyuubi project and
> > > > > helping
> > > > > > > us
> > > > > > > > set
> > > > > > > > > > > > up this proposal. The proposal can be found at
> > > > > > > > > > > >
> > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> > > > > .
> > > > > > > > > > > >
> > > > > > > > > > > > Looking forward to your feedback and thank you very
> > much.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Kent Yao
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > general-unsubscribe@incubator.apache.org
> > > > > > > > > > > > For additional commands, e-mail:
> > > > > > > general-help@incubator.apache.org
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Kent Yao
> > > > > > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
> > > > Corp.
> > > > > > > > > > a spark enthusiast
> > > > > > > > > > kyuubiis a unified multi-tenant JDBC interface for
> > > large-scale
> > > > > data
> > > > > > > > > > processing and analytics, built on top of Apache Spark.
> > > > > > > > > >
> > > > > > > > > > spark-authorizerA Spark SQL extension which provides SQL
> > > > Standard
> > > > > > > > > > Authorization for Apache Spark.
> > > > > > > > > > spark-postgres A library for reading data from and
> > > transferring
> > > > > > data
> > > > > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames,
> > > 10~100x
> > > > > > > faster.
> > > > > > > > > > itatchiA library that brings useful functions from various
> > > > modern
> > > > > > > > > > database management systems to Apache Spark.
> > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail:
> > > > general-unsubscribe@incubator.apache.org
> > > > > > > > > > For additional commands, e-mail:
> > > > > general-help@incubator.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail:
> > general-unsubscribe@incubator.apache.org
> > > > > > > > For additional commands, e-mail:
> > > general-help@incubator.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Kent Yao *
> > > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > > a spark enthusiast
> > > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> > > > JDBC
> > > > > interface for large-scale data processing and analytics, built on top
> > > of
> > > > > Apache
> > > > > Spark <http://spark.apache.org/>.
> > > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
> > Spark
> > > > SQL
> > > > > extension which provides SQL Standard Authorization for Apache Spark
> > > > > <http://spark.apache.org/>.
> > > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
> > library
> > > > for
> > > > > reading data from and transferring data to Postgres / Greenplum with
> > > > Spark
> > > > > SQL and DataFrames, 10~100x faster.
> > > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library
> > that
> > > > > brings useful functions from various modern database management
> > > > > systems to Apache
> > > > > Spark <http://spark.apache.org/>.
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > *Kent Yao *
> > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > a spark enthusiast
> > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> > JDBC
> > > interface for large-scale data processing and analytics, built on top of
> > > Apache
> > > Spark <http://spark.apache.org/>.
> > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> > SQL
> > > extension which provides SQL Standard Authorization for Apache Spark
> > > <http://spark.apache.org/>.
> > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
> > for
> > > reading data from and transferring data to Postgres / Greenplum with
> > Spark
> > > SQL and DataFrames, 10~100x faster.
> > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> > > brings useful functions from various modern database management
> > > systems to Apache
> > > Spark <http://spark.apache.org/>.
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Atri Sharma <at...@apache.org>.
I would like to hear from Kyuubi's mentors on the topic before reaching a
conclusion.
On Thu, 3 Jun 2021, 05:40 Sheng Wu, <wu...@gmail.com> wrote:
> Kent
>
> Thanks.
> The proposal is generally better this time.
>
> If your initial committers and your mentors agree, you are ready to embrace
> the ASF culture, I am good with that.
> Just from my experience, your community looks a little too young, like
> Atri's concern, which is reasonable.
> But I would like to trust you and your mentors' judgment and determination.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Kent Yao <ya...@apache.org> 于2021年6月3日周四 上午12:54写道：
>
> > Hi Sheng,
> >
> > Truly sorry for the format issue.
> >
> > Please trust me I did not mean to hurt anybody and push any contributor
> > away. We drafted this proposal together based on the New Podling
> Proposal.
> > We never tried to use PPMC and committer to measure whose contribution is
> > higher. This mistake has been fixed in the proposal.
> >
> > About releases, we will follow
> > https://www.apache.org/legal/release-policy.html and the guide of our
> > mentors. We will also take the initiative to learn other documents.
> >
> > > This is not a block, the point I mentioned this is, you will need more
> > people.
> >
> > We will keep working on this.
> >
> > Bests,
> >
> > Kent Yao
> >
> >
> > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
> >
> > > Hi Kent
> > >
> > > Thanks for replying, but to be honest, your mail format is chaos.
> > > Take a look at bellowing, which is what I saw in the gmail.
> > >
> > > This may be an issue of your mail client or some else.
> > >
> > > > The current #.2 - 5 top
> > > > contributors will stay as committers, and they will join the PPMC
> > during
> > > > incubation.
> > >
> > > PPMC usually is the default for all initial committers, considering
> your
> > > community is still small and not very active.
> > > You should invite all people you trusted to join, rather than pushing
> > them
> > > away.
> > > This is not just a too-high bar issue. Excluding 4 out of 5 is strange.
> > > I can't guess what is the root cause, why the other 4 PPMC members
> don't
> > > touch codes but are trusted completely. Meanwhile, people with easily
> > > measurable code contributions are just as committers?
> > > We(ASF) always remind the community to recognize no-code contribution,
> > but
> > > in this case, it seems the opposite? Don't recognize code contributors
> > > enough?
> > > Please reach your mentors, this seems not a typical ASF style. If you
> > have
> > > other people already involved in the ASF, asking them too.
> > >
> > > And from your description of the release process, I think you should
> talk
> > > more with your mentors.
> > > Such as what is a qualified ASF release, what should be checked.
> > > This is not a block, the point I mentioned this is, you will need more
> > > people.
> > >
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
> > >
> > > > Hi Sheng,
> > > >
> > > > Thanks for your detailed check and valuable comments. They are
> > > > accommodating. I have addressed some of them directly in the
> proposal.
> > > > Please also check the table below for some Q&As.
> > > > QA
> > > > You listed only 5 core maintainers, but clearly only one has
> dominated
> > > > contributions. The distortion and skewness of the contributions are
> > > mainly
> > > > caused by the way how we merge PRs changed(mentioned in another email
> > on
> > > > the list).
> > > > I agree with Atri, orphaned is measurable. It is better to adjust
> this
> > > > statement a little. Address the comment to notify the risk in the
> > Kyuubi
> > > > proposal.
> > > > Also, with this fact, the core maintainers should notice whether this
> > > small
> > > > group of people to make sure your features iteration, bug fix, and
> > > release
> > > > process working by following ASF's requirements.This would not be an
> > easy
> > > > job. Currently, we learn the release process from some Apache
> projects,
> > > > using major.minor.patch for versioning. As we can see, since Kyuubi
> > 1.0,
> > > it
> > > > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes
> > > until
> > > > EOL. And it takes us about two months to finish a minor release with
> 3
> > ~
> > > 5
> > > > big improvements and various small features/bug fixes. We also will
> > start
> > > > with some release candidates for preview and test before the final.
> > This
> > > is
> > > > how we develop and release Kyuubi for now, but we do have a long way
> to
> > > go
> > > > and learn to match the ASF requirement.
> > > > And your committer list is very strange... 1. Why some of them are
> PPMC
> > > and
> > > > others are committers? Where this hierarchy comes from? The current
> > PPMC
> > > > candidates help us a lot build the Kyuubi project and community, not
> > > > directly to the codebase but as mentors(before Apache). Most of them
> > are
> > > > from Apache mature TLPs, which have close relationships with Kyuubi.
> > They
> > > > have sufficient Apache Way experience to guide us to become a more
> > > > Apache-like project before submitting this proposal in many ways. We
> > will
> > > > continue to promote more contributors to the PPMC/committer based on
> > > their
> > > > code or non-code contributions during incubation.
> > > > 2. Your core maintainers have 5, which only 4 PPMC? The current #.2
> - 5
> > > top
> > > > contributors will stay as committers, and they will join the PPMC
> > during
> > > > incubation.
> > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> have
> > 5
> > > > core developers? I didn't notice that 400 LOC is a bar for core
> > > developer,
> > > > will change 5 to 4 core developers accordingly
> > > > I think you don't have enough no-salaried developers, am I missing
> > > > anything? Yes.
> > > > Changed in the proposal, most of the contributors are salaried, and
> > some
> > > of
> > > > them are willing to help in the spare time. Thanks
> > > > This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA
> in
> > > > Proposal. All work here is approved by NetEase and can be forward.
> > > > At last, you are asking about
> > github.com/apache/incubator-kyuubi-shaded
> > > > repo,
> > > > what does this represent? I can only see you have only one source
> repo
> > > and
> > > > a doc repo. Removed this requirement in the proposal, and we will
> apply
> > > it
> > > > from IPMC if we do need this in the future.
> > > > ------------------------------
> > > >
> > > > Bests
> > > >
> > > > Kent Yao
> > > >
> > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> > > >
> > > > > Hi
> > > > >
> > > > > I read the contributor list and the proposal
> > > > > > The risk of the Kyuubi project being abandoned is minimal. Many
> > > > > organizations are using Kyuubi to build critical big data pipelines
> > and
> > > > > willing to help develop Kyuubi's community if it becomes an ASF
> > > project.
> > > > >
> > > > > I think the team should evaluate the status again. You listed only
> 5
> > > core
> > > > > maintainers, but clearly only one has dominated contributions. I
> > agree
> > > > with
> > > > > Atri, orphaned is measurable.
> > > > > It is better to adjust this statement a little. Also, with this
> fact,
> > > the
> > > > > core maintainers should notice whether this small group of people
> to
> > > make
> > > > > sure your features iteration, bug fix, and release process working
> by
> > > > > following ASF's requirements.
> > > > > This would not be an easy job.
> > > > >
> > > > > And your committer list is very strange
> > > > > ```
> > > > > Kent Yao (yao@apache.org) (PPMC)
> > > > > Wenchen Fan (wenchen@apache.org) (PPMC)
> > > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > > > > Xun Liu (liuxun@apache.org) (PPMC)
> > > > > Fei Wang (cn.feiwang@gmail.com)
> > > > > Ulysses You (ulyssesyou18@gmail.com)
> > > > > Cheng Pan (pan3793@gmail.com)
> > > > > Hongxiang Jiang (jhx1008@gmail.com)
> > > > > Zen Jin(xinjingziranchan@gmail.com)
> > > > > ```
> > > > > 1. Why some of them are PPMC and others are committers? Where this
> > > > > hierarchy comes from?
> > > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy
> > > makes
> > > > > sense, most likely not)
> > > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> > have
> > > 5
> > > > > core developers?
> > > > >
> > > > > Another re-evaluation request is about `Reliance on Salaried
> > > Developers:`
> > > > > > They are all passionate about the project, and we are confident
> > that
> > > > the
> > > > > project will continue even if no salaried developers contribute to
> > the
> > > > > project.
> > > > >
> > > > > I think you don't have enough no-salaried developers, am I missing
> > > > > anything?
> > > > >
> > > > > About IP clearance, `Initial Source and Intellectual Property
> > > Submission
> > > > > Plan`
> > > > > > As soon as Kyuubi is approved to join Apache Incubator, our
> initial
> > > > > committers will submit iCLA(s) and CCLA(s). The codebase is already
> > > > > licensed under the Apache License 2.0.
> > > > >
> > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
> > > > >
> > > > > At last, you are asking about `
> > > > github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> > <http://github.com/apache/incubator-kyuubi-shaded>
> > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > repo, what does this represent? I can only see you have only one
> > source
> > > > > repo and a doc repo.
> > > > >
> > > > > Sheng Wu 吴晟
> > > > > Twitter, wusheng1108
> > > > >
> > > > >
> > > > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > I looked at the link you advised on and if I am reading it
> > correctly,
> > > > top
> > > > > > committer has 630ish commits and the next committer has 73
> commits?
> > > > > >
> > > > > > IMO this is an indication of the community not being active
> enough
> > > and
> > > > > > hence the risk of project getting orphaned is measurable.
> > > > > >
> > > > > > Atri
> > > > > >
> > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Atri Sharma,
> > > > > > >
> > > > > > > It's easier to check out the recent contribution of kyuubi from
> > > this
> > > > > > > link[1].
> > > > > > > We already identified the one person contribution issue of
> kyuubi
> > > > when
> > > > > > > Kent came to us and we kept reminding him to put more effort
> into
> > > > > > > building the community.
> > > > > > > Last week I had a long F2F conversation with Kent, I think he
> > > already
> > > > > > > knows about it and there have been some improvements in recent
> > > months
> > > > > > > but there is still a long way to go if Kyuubi is accepted as a
> > > > podling
> > > > > > > in the Apache Incubator.
> > > > > > >
> > > > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > > > > >
> > > > > > > Willem Jiang
> > > > > > >
> > > > > > > Twitter: willemjiang
> > > > > > > Weibo: 姜宁willem
> > > > > > >
> > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I took a look at commit log and seems like all the commits
> for
> > > > last 3
> > > > > > > > months have been done by a single person.
> > > > > > > >
> > > > > > > > How active are the remaining committers?
> > > > > > > >
> > > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > > > > > > >
> > > > > > > > > Hi Alexander Alten,
> > > > > > > > >
> > > > > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > > > > >
> > > > > > > > > Not being very familiar with Dremel, I just took a quick
> look
> > > at
> > > > > the
> > > > > > > > > profile of a very similar and exciting project, Apache
> Drill.
> > > If
> > > > I
> > > > > > > > > understand correctly, Kyuubi is built on top of the compute
> > > layer
> > > > > > > > > (Apache Spark), while Apache Drill is built on top of the
> > > storage
> > > > > > > > > layer.
> > > > > > > > >
> > > > > > > > > Apache Spark has a large user base and usage scenarios all
> > over
> > > > the
> > > > > > > > > world. When users have some basic Spark programming and
> > tuning
> > > > > > skills,
> > > > > > > > > things are easy for them. But many times, most users do not
> > > have
> > > > > all
> > > > > > > > > of these abilities when they use Spark directly or
> > indirectly.
> > > We
> > > > > > want
> > > > > > > > > to build a more easy-to-use platform on top of it to help
> > these
> > > > > > users.
> > > > > > > > > On the one hand, we will work on the interface layer to
> make
> > it
> > > > > easy
> > > > > > > > > to access (of course, this effort will make our project
> look
> > > very
> > > > > > > > > similar to other projects, such as HiveServer2). On the
> other
> > > > hand,
> > > > > > we
> > > > > > > > > will work under Kyuubi's multi-tenant architecture to
> extend
> > > our
> > > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it
> more
> > > > > stable
> > > > > > > > > and efficient in our scenario.
> > > > > > > > >
> > > > > > > > > What's more, we are going to add more on-prem Kyuubi engine
> > > > types,
> > > > > > e.g.
> > > > > > > > > Flink.
> > > > > > > > >
> > > > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hey,
> > > > > > > > > >
> > > > > > > > > > Looks like an interesting project, competition to
> > Starburst.
> > > > How
> > > > > > > does it
> > > > > > > > > > compare to Apache Dremel?
> > > > > > > > > >
> > > > > > > > > > +1 (non-binding) for incubation.
> > > > > > > > > >
> > > > > > > > > > Thanks and stay safe,
> > > > > > > > > > --alex
> > > > > > > > > >
> > > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org>
> > wrote:
> > > > > > > > > >
> > > > > > > > > > > Dear all,
> > > > > > > > > > >
> > > > > > > > > > > We want to propose Kyuubi as a new Apache Incubator
> > > project.
> > > > > > > > > > >
> > > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC
> > > server
> > > > > for
> > > > > > > > > > > large-scale data management, processing, and analytics,
> > > built
> > > > > on
> > > > > > > top
> > > > > > > > > > > of Apache Spark and designed to support more engines
> > (i.e.,
> > > > > > Apache
> > > > > > > > > > > Flink). We are aiming to make Kyuubi an
> "out-of-the-box"
> > > tool
> > > > > for
> > > > > > > data
> > > > > > > > > > > warehouses and data lakes. Kyuubi concentrates on
> > improving
> > > > the
> > > > > > > > > > > experience for non-Spark users, making it easy for them
> > to
> > > > use
> > > > > > > Spark
> > > > > > > > > > > as a service and focus on business data of their own,
> not
> > > > where
> > > > > > it
> > > > > > > > > > > stores, how it computes. NetEase has open-sourced
> Kyuubi
> > > > since
> > > > > > > 2018,
> > > > > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > > > > >
> > > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo
> > > > Zhang,
> > > > > > > Akira
> > > > > > > > > > > Ajisaka for being the mentors of the Kyuubi project and
> > > > helping
> > > > > > us
> > > > > > > set
> > > > > > > > > > > up this proposal. The proposal can be found at
> > > > > > > > > > >
> > > > > > >
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> > > > .
> > > > > > > > > > >
> > > > > > > > > > > Looking forward to your feedback and thank you very
> much.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Kent Yao
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail:
> > > > > general-unsubscribe@incubator.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> > > > > > general-help@incubator.apache.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Kent Yao
> > > > > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
> > > Corp.
> > > > > > > > > a spark enthusiast
> > > > > > > > > kyuubiis a unified multi-tenant JDBC interface for
> > large-scale
> > > > data
> > > > > > > > > processing and analytics, built on top of Apache Spark.
> > > > > > > > >
> > > > > > > > > spark-authorizerA Spark SQL extension which provides SQL
> > > Standard
> > > > > > > > > Authorization for Apache Spark.
> > > > > > > > > spark-postgres A library for reading data from and
> > transferring
> > > > > data
> > > > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames,
> > 10~100x
> > > > > > faster.
> > > > > > > > > itatchiA library that brings useful functions from various
> > > modern
> > > > > > > > > database management systems to Apache Spark.
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail:
> > > general-unsubscribe@incubator.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Kent Yao *
> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > a spark enthusiast
> > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> > > JDBC
> > > > interface for large-scale data processing and analytics, built on top
> > of
> > > > Apache
> > > > Spark <http://spark.apache.org/>.
> > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
> Spark
> > > SQL
> > > > extension which provides SQL Standard Authorization for Apache Spark
> > > > <http://spark.apache.org/>.
> > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
> library
> > > for
> > > > reading data from and transferring data to Postgres / Greenplum with
> > > Spark
> > > > SQL and DataFrames, 10~100x faster.
> > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library
> that
> > > > brings useful functions from various modern database management
> > > > systems to Apache
> > > > Spark <http://spark.apache.org/>.
> > > >
> > >
> >
> >
> > --
> >
> > *Kent Yao *
> > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > a spark enthusiast
> > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> JDBC
> > interface for large-scale data processing and analytics, built on top of
> > Apache
> > Spark <http://spark.apache.org/>.
> > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> SQL
> > extension which provides SQL Standard Authorization for Apache Spark
> > <http://spark.apache.org/>.
> > spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
> for
> > reading data from and transferring data to Postgres / Greenplum with
> Spark
> > SQL and DataFrames, 10~100x faster.
> > itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> > brings useful functions from various modern database management
> > systems to Apache
> > Spark <http://spark.apache.org/>.
> >
>
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Thanks very much, Sheng :)
Sheng Wu <wu...@gmail.com> 于2021年6月3日周四 上午8:10写道：
> Kent
>
> Thanks.
> The proposal is generally better this time.
>
> If your initial committers and your mentors agree, you are ready to embrace
> the ASF culture, I am good with that.
> Just from my experience, your community looks a little too young, like
> Atri's concern, which is reasonable.
> But I would like to trust you and your mentors' judgment and determination.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Kent Yao <ya...@apache.org> 于2021年6月3日周四 上午12:54写道：
>
> > Hi Sheng,
> >
> > Truly sorry for the format issue.
> >
> > Please trust me I did not mean to hurt anybody and push any contributor
> > away. We drafted this proposal together based on the New Podling
> Proposal.
> > We never tried to use PPMC and committer to measure whose contribution is
> > higher. This mistake has been fixed in the proposal.
> >
> > About releases, we will follow
> > https://www.apache.org/legal/release-policy.html and the guide of our
> > mentors. We will also take the initiative to learn other documents.
> >
> > > This is not a block, the point I mentioned this is, you will need more
> > people.
> >
> > We will keep working on this.
> >
> > Bests,
> >
> > Kent Yao
> >
> >
> > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
> >
> > > Hi Kent
> > >
> > > Thanks for replying, but to be honest, your mail format is chaos.
> > > Take a look at bellowing, which is what I saw in the gmail.
> > >
> > > This may be an issue of your mail client or some else.
> > >
> > > > The current #.2 - 5 top
> > > > contributors will stay as committers, and they will join the PPMC
> > during
> > > > incubation.
> > >
> > > PPMC usually is the default for all initial committers, considering
> your
> > > community is still small and not very active.
> > > You should invite all people you trusted to join, rather than pushing
> > them
> > > away.
> > > This is not just a too-high bar issue. Excluding 4 out of 5 is strange.
> > > I can't guess what is the root cause, why the other 4 PPMC members
> don't
> > > touch codes but are trusted completely. Meanwhile, people with easily
> > > measurable code contributions are just as committers?
> > > We(ASF) always remind the community to recognize no-code contribution,
> > but
> > > in this case, it seems the opposite? Don't recognize code contributors
> > > enough?
> > > Please reach your mentors, this seems not a typical ASF style. If you
> > have
> > > other people already involved in the ASF, asking them too.
> > >
> > > And from your description of the release process, I think you should
> talk
> > > more with your mentors.
> > > Such as what is a qualified ASF release, what should be checked.
> > > This is not a block, the point I mentioned this is, you will need more
> > > people.
> > >
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
> > >
> > > > Hi Sheng,
> > > >
> > > > Thanks for your detailed check and valuable comments. They are
> > > > accommodating. I have addressed some of them directly in the
> proposal.
> > > > Please also check the table below for some Q&As.
> > > > QA
> > > > You listed only 5 core maintainers, but clearly only one has
> dominated
> > > > contributions. The distortion and skewness of the contributions are
> > > mainly
> > > > caused by the way how we merge PRs changed(mentioned in another email
> > on
> > > > the list).
> > > > I agree with Atri, orphaned is measurable. It is better to adjust
> this
> > > > statement a little. Address the comment to notify the risk in the
> > Kyuubi
> > > > proposal.
> > > > Also, with this fact, the core maintainers should notice whether this
> > > small
> > > > group of people to make sure your features iteration, bug fix, and
> > > release
> > > > process working by following ASF's requirements.This would not be an
> > easy
> > > > job. Currently, we learn the release process from some Apache
> projects,
> > > > using major.minor.patch for versioning. As we can see, since Kyuubi
> > 1.0,
> > > it
> > > > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes
> > > until
> > > > EOL. And it takes us about two months to finish a minor release with
> 3
> > ~
> > > 5
> > > > big improvements and various small features/bug fixes. We also will
> > start
> > > > with some release candidates for preview and test before the final.
> > This
> > > is
> > > > how we develop and release Kyuubi for now, but we do have a long way
> to
> > > go
> > > > and learn to match the ASF requirement.
> > > > And your committer list is very strange... 1. Why some of them are
> PPMC
> > > and
> > > > others are committers? Where this hierarchy comes from? The current
> > PPMC
> > > > candidates help us a lot build the Kyuubi project and community, not
> > > > directly to the codebase but as mentors(before Apache). Most of them
> > are
> > > > from Apache mature TLPs, which have close relationships with Kyuubi.
> > They
> > > > have sufficient Apache Way experience to guide us to become a more
> > > > Apache-like project before submitting this proposal in many ways. We
> > will
> > > > continue to promote more contributors to the PPMC/committer based on
> > > their
> > > > code or non-code contributions during incubation.
> > > > 2. Your core maintainers have 5, which only 4 PPMC? The current #.2
> - 5
> > > top
> > > > contributors will stay as committers, and they will join the PPMC
> > during
> > > > incubation.
> > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> have
> > 5
> > > > core developers? I didn't notice that 400 LOC is a bar for core
> > > developer,
> > > > will change 5 to 4 core developers accordingly
> > > > I think you don't have enough no-salaried developers, am I missing
> > > > anything? Yes.
> > > > Changed in the proposal, most of the contributors are salaried, and
> > some
> > > of
> > > > them are willing to help in the spare time. Thanks
> > > > This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA
> in
> > > > Proposal. All work here is approved by NetEase and can be forward.
> > > > At last, you are asking about
> > github.com/apache/incubator-kyuubi-shaded
> > > > repo,
> > > > what does this represent? I can only see you have only one source
> repo
> > > and
> > > > a doc repo. Removed this requirement in the proposal, and we will
> apply
> > > it
> > > > from IPMC if we do need this in the future.
> > > > ------------------------------
> > > >
> > > > Bests
> > > >
> > > > Kent Yao
> > > >
> > > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> > > >
> > > > > Hi
> > > > >
> > > > > I read the contributor list and the proposal
> > > > > > The risk of the Kyuubi project being abandoned is minimal. Many
> > > > > organizations are using Kyuubi to build critical big data pipelines
> > and
> > > > > willing to help develop Kyuubi's community if it becomes an ASF
> > > project.
> > > > >
> > > > > I think the team should evaluate the status again. You listed only
> 5
> > > core
> > > > > maintainers, but clearly only one has dominated contributions. I
> > agree
> > > > with
> > > > > Atri, orphaned is measurable.
> > > > > It is better to adjust this statement a little. Also, with this
> fact,
> > > the
> > > > > core maintainers should notice whether this small group of people
> to
> > > make
> > > > > sure your features iteration, bug fix, and release process working
> by
> > > > > following ASF's requirements.
> > > > > This would not be an easy job.
> > > > >
> > > > > And your committer list is very strange
> > > > > ```
> > > > > Kent Yao (yao@apache.org) (PPMC)
> > > > > Wenchen Fan (wenchen@apache.org) (PPMC)
> > > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > > > > Xun Liu (liuxun@apache.org) (PPMC)
> > > > > Fei Wang (cn.feiwang@gmail.com)
> > > > > Ulysses You (ulyssesyou18@gmail.com)
> > > > > Cheng Pan (pan3793@gmail.com)
> > > > > Hongxiang Jiang (jhx1008@gmail.com)
> > > > > Zen Jin(xinjingziranchan@gmail.com)
> > > > > ```
> > > > > 1. Why some of them are PPMC and others are committers? Where this
> > > > > hierarchy comes from?
> > > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy
> > > makes
> > > > > sense, most likely not)
> > > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> > have
> > > 5
> > > > > core developers?
> > > > >
> > > > > Another re-evaluation request is about `Reliance on Salaried
> > > Developers:`
> > > > > > They are all passionate about the project, and we are confident
> > that
> > > > the
> > > > > project will continue even if no salaried developers contribute to
> > the
> > > > > project.
> > > > >
> > > > > I think you don't have enough no-salaried developers, am I missing
> > > > > anything?
> > > > >
> > > > > About IP clearance, `Initial Source and Intellectual Property
> > > Submission
> > > > > Plan`
> > > > > > As soon as Kyuubi is approved to join Apache Incubator, our
> initial
> > > > > committers will submit iCLA(s) and CCLA(s). The codebase is already
> > > > > licensed under the Apache License 2.0.
> > > > >
> > > > > This project belongs to NetEase Inc. clearly, why NO SGA?
> > > > >
> > > > > At last, you are asking about `
> > > > github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> > <http://github.com/apache/incubator-kyuubi-shaded>
> > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > > repo, what does this represent? I can only see you have only one
> > source
> > > > > repo and a doc repo.
> > > > >
> > > > > Sheng Wu 吴晟
> > > > > Twitter, wusheng1108
> > > > >
> > > > >
> > > > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > I looked at the link you advised on and if I am reading it
> > correctly,
> > > > top
> > > > > > committer has 630ish commits and the next committer has 73
> commits?
> > > > > >
> > > > > > IMO this is an indication of the community not being active
> enough
> > > and
> > > > > > hence the risk of project getting orphaned is measurable.
> > > > > >
> > > > > > Atri
> > > > > >
> > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Atri Sharma,
> > > > > > >
> > > > > > > It's easier to check out the recent contribution of kyuubi from
> > > this
> > > > > > > link[1].
> > > > > > > We already identified the one person contribution issue of
> kyuubi
> > > > when
> > > > > > > Kent came to us and we kept reminding him to put more effort
> into
> > > > > > > building the community.
> > > > > > > Last week I had a long F2F conversation with Kent, I think he
> > > already
> > > > > > > knows about it and there have been some improvements in recent
> > > months
> > > > > > > but there is still a long way to go if Kyuubi is accepted as a
> > > > podling
> > > > > > > in the Apache Incubator.
> > > > > > >
> > > > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > > > > >
> > > > > > > Willem Jiang
> > > > > > >
> > > > > > > Twitter: willemjiang
> > > > > > > Weibo: 姜宁willem
> > > > > > >
> > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I took a look at commit log and seems like all the commits
> for
> > > > last 3
> > > > > > > > months have been done by a single person.
> > > > > > > >
> > > > > > > > How active are the remaining committers?
> > > > > > > >
> > > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > > > > > > >
> > > > > > > > > Hi Alexander Alten,
> > > > > > > > >
> > > > > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > > > > >
> > > > > > > > > Not being very familiar with Dremel, I just took a quick
> look
> > > at
> > > > > the
> > > > > > > > > profile of a very similar and exciting project, Apache
> Drill.
> > > If
> > > > I
> > > > > > > > > understand correctly, Kyuubi is built on top of the compute
> > > layer
> > > > > > > > > (Apache Spark), while Apache Drill is built on top of the
> > > storage
> > > > > > > > > layer.
> > > > > > > > >
> > > > > > > > > Apache Spark has a large user base and usage scenarios all
> > over
> > > > the
> > > > > > > > > world. When users have some basic Spark programming and
> > tuning
> > > > > > skills,
> > > > > > > > > things are easy for them. But many times, most users do not
> > > have
> > > > > all
> > > > > > > > > of these abilities when they use Spark directly or
> > indirectly.
> > > We
> > > > > > want
> > > > > > > > > to build a more easy-to-use platform on top of it to help
> > these
> > > > > > users.
> > > > > > > > > On the one hand, we will work on the interface layer to
> make
> > it
> > > > > easy
> > > > > > > > > to access (of course, this effort will make our project
> look
> > > very
> > > > > > > > > similar to other projects, such as HiveServer2). On the
> other
> > > > hand,
> > > > > > we
> > > > > > > > > will work under Kyuubi's multi-tenant architecture to
> extend
> > > our
> > > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it
> more
> > > > > stable
> > > > > > > > > and efficient in our scenario.
> > > > > > > > >
> > > > > > > > > What's more, we are going to add more on-prem Kyuubi engine
> > > > types,
> > > > > > e.g.
> > > > > > > > > Flink.
> > > > > > > > >
> > > > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hey,
> > > > > > > > > >
> > > > > > > > > > Looks like an interesting project, competition to
> > Starburst.
> > > > How
> > > > > > > does it
> > > > > > > > > > compare to Apache Dremel?
> > > > > > > > > >
> > > > > > > > > > +1 (non-binding) for incubation.
> > > > > > > > > >
> > > > > > > > > > Thanks and stay safe,
> > > > > > > > > > --alex
> > > > > > > > > >
> > > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org>
> > wrote:
> > > > > > > > > >
> > > > > > > > > > > Dear all,
> > > > > > > > > > >
> > > > > > > > > > > We want to propose Kyuubi as a new Apache Incubator
> > > project.
> > > > > > > > > > >
> > > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC
> > > server
> > > > > for
> > > > > > > > > > > large-scale data management, processing, and analytics,
> > > built
> > > > > on
> > > > > > > top
> > > > > > > > > > > of Apache Spark and designed to support more engines
> > (i.e.,
> > > > > > Apache
> > > > > > > > > > > Flink). We are aiming to make Kyuubi an
> "out-of-the-box"
> > > tool
> > > > > for
> > > > > > > data
> > > > > > > > > > > warehouses and data lakes. Kyuubi concentrates on
> > improving
> > > > the
> > > > > > > > > > > experience for non-Spark users, making it easy for them
> > to
> > > > use
> > > > > > > Spark
> > > > > > > > > > > as a service and focus on business data of their own,
> not
> > > > where
> > > > > > it
> > > > > > > > > > > stores, how it computes. NetEase has open-sourced
> Kyuubi
> > > > since
> > > > > > > 2018,
> > > > > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > > > > >
> > > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo
> > > > Zhang,
> > > > > > > Akira
> > > > > > > > > > > Ajisaka for being the mentors of the Kyuubi project and
> > > > helping
> > > > > > us
> > > > > > > set
> > > > > > > > > > > up this proposal. The proposal can be found at
> > > > > > > > > > >
> > > > > > >
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> > > > .
> > > > > > > > > > >
> > > > > > > > > > > Looking forward to your feedback and thank you very
> much.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Kent Yao
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail:
> > > > > general-unsubscribe@incubator.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> > > > > > general-help@incubator.apache.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Kent Yao
> > > > > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
> > > Corp.
> > > > > > > > > a spark enthusiast
> > > > > > > > > kyuubiis a unified multi-tenant JDBC interface for
> > large-scale
> > > > data
> > > > > > > > > processing and analytics, built on top of Apache Spark.
> > > > > > > > >
> > > > > > > > > spark-authorizerA Spark SQL extension which provides SQL
> > > Standard
> > > > > > > > > Authorization for Apache Spark.
> > > > > > > > > spark-postgres A library for reading data from and
> > transferring
> > > > > data
> > > > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames,
> > 10~100x
> > > > > > faster.
> > > > > > > > > itatchiA library that brings useful functions from various
> > > modern
> > > > > > > > > database management systems to Apache Spark.
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail:
> > > general-unsubscribe@incubator.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Kent Yao *
> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > a spark enthusiast
> > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> > > JDBC
> > > > interface for large-scale data processing and analytics, built on top
> > of
> > > > Apache
> > > > Spark <http://spark.apache.org/>.
> > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
> Spark
> > > SQL
> > > > extension which provides SQL Standard Authorization for Apache Spark
> > > > <http://spark.apache.org/>.
> > > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A
> library
> > > for
> > > > reading data from and transferring data to Postgres / Greenplum with
> > > Spark
> > > > SQL and DataFrames, 10~100x faster.
> > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library
> that
> > > > brings useful functions from various modern database management
> > > > systems to Apache
> > > > Spark <http://spark.apache.org/>.
> > > >
> > >
> >
> >
> > --
> >
> > *Kent Yao *
> > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > a spark enthusiast
> > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> JDBC
> > interface for large-scale data processing and analytics, built on top of
> > Apache
> > Spark <http://spark.apache.org/>.
> > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> SQL
> > extension which provides SQL Standard Authorization for Apache Spark
> > <http://spark.apache.org/>.
> > spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
> for
> > reading data from and transferring data to Postgres / Greenplum with
> Spark
> > SQL and DataFrames, 10~100x faster.
> > itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> > brings useful functions from various modern database management
> > systems to Apache
> > Spark <http://spark.apache.org/>.
> >
>
--
*Kent Yao *
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of Apache
Spark <http://spark.apache.org/>.
spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
extension which provides SQL Standard Authorization for Apache Spark
<http://spark.apache.org/>.
spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.
itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
brings useful functions from various modern database management
systems to Apache
Spark <http://spark.apache.org/>.
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Sheng Wu <wu...@gmail.com>.
Kent
Thanks.
The proposal is generally better this time.
If your initial committers and your mentors agree, you are ready to embrace
the ASF culture, I am good with that.
Just from my experience, your community looks a little too young, like
Atri's concern, which is reasonable.
But I would like to trust you and your mentors' judgment and determination.
Sheng Wu 吴晟
Twitter, wusheng1108
Kent Yao <ya...@apache.org> 于2021年6月3日周四 上午12:54写道：
> Hi Sheng,
>
> Truly sorry for the format issue.
>
> Please trust me I did not mean to hurt anybody and push any contributor
> away. We drafted this proposal together based on the New Podling Proposal.
> We never tried to use PPMC and committer to measure whose contribution is
> higher. This mistake has been fixed in the proposal.
>
> About releases, we will follow
> https://www.apache.org/legal/release-policy.html and the guide of our
> mentors. We will also take the initiative to learn other documents.
>
> > This is not a block, the point I mentioned this is, you will need more
> people.
>
> We will keep working on this.
>
> Bests,
>
> Kent Yao
>
>
> Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
>
> > Hi Kent
> >
> > Thanks for replying, but to be honest, your mail format is chaos.
> > Take a look at bellowing, which is what I saw in the gmail.
> >
> > This may be an issue of your mail client or some else.
> >
> > > The current #.2 - 5 top
> > > contributors will stay as committers, and they will join the PPMC
> during
> > > incubation.
> >
> > PPMC usually is the default for all initial committers, considering your
> > community is still small and not very active.
> > You should invite all people you trusted to join, rather than pushing
> them
> > away.
> > This is not just a too-high bar issue. Excluding 4 out of 5 is strange.
> > I can't guess what is the root cause, why the other 4 PPMC members don't
> > touch codes but are trusted completely. Meanwhile, people with easily
> > measurable code contributions are just as committers?
> > We(ASF) always remind the community to recognize no-code contribution,
> but
> > in this case, it seems the opposite? Don't recognize code contributors
> > enough?
> > Please reach your mentors, this seems not a typical ASF style. If you
> have
> > other people already involved in the ASF, asking them too.
> >
> > And from your description of the release process, I think you should talk
> > more with your mentors.
> > Such as what is a qualified ASF release, what should be checked.
> > This is not a block, the point I mentioned this is, you will need more
> > people.
> >
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
> >
> > > Hi Sheng,
> > >
> > > Thanks for your detailed check and valuable comments. They are
> > > accommodating. I have addressed some of them directly in the proposal.
> > > Please also check the table below for some Q&As.
> > > QA
> > > You listed only 5 core maintainers, but clearly only one has dominated
> > > contributions. The distortion and skewness of the contributions are
> > mainly
> > > caused by the way how we merge PRs changed(mentioned in another email
> on
> > > the list).
> > > I agree with Atri, orphaned is measurable. It is better to adjust this
> > > statement a little. Address the comment to notify the risk in the
> Kyuubi
> > > proposal.
> > > Also, with this fact, the core maintainers should notice whether this
> > small
> > > group of people to make sure your features iteration, bug fix, and
> > release
> > > process working by following ASF's requirements.This would not be an
> easy
> > > job. Currently, we learn the release process from some Apache projects,
> > > using major.minor.patch for versioning. As we can see, since Kyuubi
> 1.0,
> > it
> > > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes
> > until
> > > EOL. And it takes us about two months to finish a minor release with 3
> ~
> > 5
> > > big improvements and various small features/bug fixes. We also will
> start
> > > with some release candidates for preview and test before the final.
> This
> > is
> > > how we develop and release Kyuubi for now, but we do have a long way to
> > go
> > > and learn to match the ASF requirement.
> > > And your committer list is very strange... 1. Why some of them are PPMC
> > and
> > > others are committers? Where this hierarchy comes from? The current
> PPMC
> > > candidates help us a lot build the Kyuubi project and community, not
> > > directly to the codebase but as mentors(before Apache). Most of them
> are
> > > from Apache mature TLPs, which have close relationships with Kyuubi.
> They
> > > have sufficient Apache Way experience to guide us to become a more
> > > Apache-like project before submitting this proposal in many ways. We
> will
> > > continue to promote more contributors to the PPMC/committer based on
> > their
> > > code or non-code contributions during incubation.
> > > 2. Your core maintainers have 5, which only 4 PPMC? The current #.2 - 5
> > top
> > > contributors will stay as committers, and they will join the PPMC
> during
> > > incubation.
> > > 3. I think only the top 4 have over 400 LOC, why you declared you have
> 5
> > > core developers? I didn't notice that 400 LOC is a bar for core
> > developer,
> > > will change 5 to 4 core developers accordingly
> > > I think you don't have enough no-salaried developers, am I missing
> > > anything? Yes.
> > > Changed in the proposal, most of the contributors are salaried, and
> some
> > of
> > > them are willing to help in the spare time. Thanks
> > > This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA in
> > > Proposal. All work here is approved by NetEase and can be forward.
> > > At last, you are asking about
> github.com/apache/incubator-kyuubi-shaded
> > > repo,
> > > what does this represent? I can only see you have only one source repo
> > and
> > > a doc repo. Removed this requirement in the proposal, and we will apply
> > it
> > > from IPMC if we do need this in the future.
> > > ------------------------------
> > >
> > > Bests
> > >
> > > Kent Yao
> > >
> > > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> > >
> > > > Hi
> > > >
> > > > I read the contributor list and the proposal
> > > > > The risk of the Kyuubi project being abandoned is minimal. Many
> > > > organizations are using Kyuubi to build critical big data pipelines
> and
> > > > willing to help develop Kyuubi's community if it becomes an ASF
> > project.
> > > >
> > > > I think the team should evaluate the status again. You listed only 5
> > core
> > > > maintainers, but clearly only one has dominated contributions. I
> agree
> > > with
> > > > Atri, orphaned is measurable.
> > > > It is better to adjust this statement a little. Also, with this fact,
> > the
> > > > core maintainers should notice whether this small group of people to
> > make
> > > > sure your features iteration, bug fix, and release process working by
> > > > following ASF's requirements.
> > > > This would not be an easy job.
> > > >
> > > > And your committer list is very strange
> > > > ```
> > > > Kent Yao (yao@apache.org) (PPMC)
> > > > Wenchen Fan (wenchen@apache.org) (PPMC)
> > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > > > Xun Liu (liuxun@apache.org) (PPMC)
> > > > Fei Wang (cn.feiwang@gmail.com)
> > > > Ulysses You (ulyssesyou18@gmail.com)
> > > > Cheng Pan (pan3793@gmail.com)
> > > > Hongxiang Jiang (jhx1008@gmail.com)
> > > > Zen Jin(xinjingziranchan@gmail.com)
> > > > ```
> > > > 1. Why some of them are PPMC and others are committers? Where this
> > > > hierarchy comes from?
> > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy
> > makes
> > > > sense, most likely not)
> > > > 3. I think only the top 4 have over 400 LOC, why you declared you
> have
> > 5
> > > > core developers?
> > > >
> > > > Another re-evaluation request is about `Reliance on Salaried
> > Developers:`
> > > > > They are all passionate about the project, and we are confident
> that
> > > the
> > > > project will continue even if no salaried developers contribute to
> the
> > > > project.
> > > >
> > > > I think you don't have enough no-salaried developers, am I missing
> > > > anything?
> > > >
> > > > About IP clearance, `Initial Source and Intellectual Property
> > Submission
> > > > Plan`
> > > > > As soon as Kyuubi is approved to join Apache Incubator, our initial
> > > > committers will submit iCLA(s) and CCLA(s). The codebase is already
> > > > licensed under the Apache License 2.0.
> > > >
> > > > This project belongs to NetEase Inc. clearly, why NO SGA?
> > > >
> > > > At last, you are asking about `
> > > github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> > <http://github.com/apache/incubator-kyuubi-shaded>
> > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > > repo, what does this represent? I can only see you have only one
> source
> > > > repo and a doc repo.
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> > > >
> > > > > Thanks.
> > > > >
> > > > > I looked at the link you advised on and if I am reading it
> correctly,
> > > top
> > > > > committer has 630ish commits and the next committer has 73 commits?
> > > > >
> > > > > IMO this is an indication of the community not being active enough
> > and
> > > > > hence the risk of project getting orphaned is measurable.
> > > > >
> > > > > Atri
> > > > >
> > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Atri Sharma,
> > > > > >
> > > > > > It's easier to check out the recent contribution of kyuubi from
> > this
> > > > > > link[1].
> > > > > > We already identified the one person contribution issue of kyuubi
> > > when
> > > > > > Kent came to us and we kept reminding him to put more effort into
> > > > > > building the community.
> > > > > > Last week I had a long F2F conversation with Kent, I think he
> > already
> > > > > > knows about it and there have been some improvements in recent
> > months
> > > > > > but there is still a long way to go if Kyuubi is accepted as a
> > > podling
> > > > > > in the Apache Incubator.
> > > > > >
> > > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > > > >
> > > > > > Willem Jiang
> > > > > >
> > > > > > Twitter: willemjiang
> > > > > > Weibo: 姜宁willem
> > > > > >
> > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I took a look at commit log and seems like all the commits for
> > > last 3
> > > > > > > months have been done by a single person.
> > > > > > >
> > > > > > > How active are the remaining committers?
> > > > > > >
> > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > > > > > >
> > > > > > > > Hi Alexander Alten,
> > > > > > > >
> > > > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > > > >
> > > > > > > > Not being very familiar with Dremel, I just took a quick look
> > at
> > > > the
> > > > > > > > profile of a very similar and exciting project, Apache Drill.
> > If
> > > I
> > > > > > > > understand correctly, Kyuubi is built on top of the compute
> > layer
> > > > > > > > (Apache Spark), while Apache Drill is built on top of the
> > storage
> > > > > > > > layer.
> > > > > > > >
> > > > > > > > Apache Spark has a large user base and usage scenarios all
> over
> > > the
> > > > > > > > world. When users have some basic Spark programming and
> tuning
> > > > > skills,
> > > > > > > > things are easy for them. But many times, most users do not
> > have
> > > > all
> > > > > > > > of these abilities when they use Spark directly or
> indirectly.
> > We
> > > > > want
> > > > > > > > to build a more easy-to-use platform on top of it to help
> these
> > > > > users.
> > > > > > > > On the one hand, we will work on the interface layer to make
> it
> > > > easy
> > > > > > > > to access (of course, this effort will make our project look
> > very
> > > > > > > > similar to other projects, such as HiveServer2). On the other
> > > hand,
> > > > > we
> > > > > > > > will work under Kyuubi's multi-tenant architecture to extend
> > our
> > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more
> > > > stable
> > > > > > > > and efficient in our scenario.
> > > > > > > >
> > > > > > > > What's more, we are going to add more on-prem Kyuubi engine
> > > types,
> > > > > e.g.
> > > > > > > > Flink.
> > > > > > > >
> > > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Hey,
> > > > > > > > >
> > > > > > > > > Looks like an interesting project, competition to
> Starburst.
> > > How
> > > > > > does it
> > > > > > > > > compare to Apache Dremel?
> > > > > > > > >
> > > > > > > > > +1 (non-binding) for incubation.
> > > > > > > > >
> > > > > > > > > Thanks and stay safe,
> > > > > > > > > --alex
> > > > > > > > >
> > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org>
> wrote:
> > > > > > > > >
> > > > > > > > > > Dear all,
> > > > > > > > > >
> > > > > > > > > > We want to propose Kyuubi as a new Apache Incubator
> > project.
> > > > > > > > > >
> > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC
> > server
> > > > for
> > > > > > > > > > large-scale data management, processing, and analytics,
> > built
> > > > on
> > > > > > top
> > > > > > > > > > of Apache Spark and designed to support more engines
> (i.e.,
> > > > > Apache
> > > > > > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box"
> > tool
> > > > for
> > > > > > data
> > > > > > > > > > warehouses and data lakes. Kyuubi concentrates on
> improving
> > > the
> > > > > > > > > > experience for non-Spark users, making it easy for them
> to
> > > use
> > > > > > Spark
> > > > > > > > > > as a service and focus on business data of their own, not
> > > where
> > > > > it
> > > > > > > > > > stores, how it computes. NetEase has open-sourced Kyuubi
> > > since
> > > > > > 2018,
> > > > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > > > >
> > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo
> > > Zhang,
> > > > > > Akira
> > > > > > > > > > Ajisaka for being the mentors of the Kyuubi project and
> > > helping
> > > > > us
> > > > > > set
> > > > > > > > > > up this proposal. The proposal can be found at
> > > > > > > > > >
> > > > > >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> > > .
> > > > > > > > > >
> > > > > > > > > > Looking forward to your feedback and thank you very much.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Kent Yao
> > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail:
> > > > general-unsubscribe@incubator.apache.org
> > > > > > > > > > For additional commands, e-mail:
> > > > > general-help@incubator.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Kent Yao
> > > > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
> > Corp.
> > > > > > > > a spark enthusiast
> > > > > > > > kyuubiis a unified multi-tenant JDBC interface for
> large-scale
> > > data
> > > > > > > > processing and analytics, built on top of Apache Spark.
> > > > > > > >
> > > > > > > > spark-authorizerA Spark SQL extension which provides SQL
> > Standard
> > > > > > > > Authorization for Apache Spark.
> > > > > > > > spark-postgres A library for reading data from and
> transferring
> > > > data
> > > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames,
> 10~100x
> > > > > faster.
> > > > > > > > itatchiA library that brings useful functions from various
> > modern
> > > > > > > > database management systems to Apache Spark.
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail:
> > general-unsubscribe@incubator.apache.org
> > > > > > > > For additional commands, e-mail:
> > > general-help@incubator.apache.org
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > *Kent Yao *
> > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > a spark enthusiast
> > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> > JDBC
> > > interface for large-scale data processing and analytics, built on top
> of
> > > Apache
> > > Spark <http://spark.apache.org/>.
> > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> > SQL
> > > extension which provides SQL Standard Authorization for Apache Spark
> > > <http://spark.apache.org/>.
> > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
> > for
> > > reading data from and transferring data to Postgres / Greenplum with
> > Spark
> > > SQL and DataFrames, 10~100x faster.
> > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> > > brings useful functions from various modern database management
> > > systems to Apache
> > > Spark <http://spark.apache.org/>.
> > >
> >
>
>
> --
>
> *Kent Yao *
> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> a spark enthusiast
> kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
> interface for large-scale data processing and analytics, built on top of
> Apache
> Spark <http://spark.apache.org/>.
> spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
> extension which provides SQL Standard Authorization for Apache Spark
> <http://spark.apache.org/>.
> spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
> reading data from and transferring data to Postgres / Greenplum with Spark
> SQL and DataFrames, 10~100x faster.
> itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> brings useful functions from various modern database management
> systems to Apache
> Spark <http://spark.apache.org/>.
>
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Hi Sheng,
Truly sorry for the format issue.
Please trust me I did not mean to hurt anybody and push any contributor
away. We drafted this proposal together based on the New Podling Proposal.
We never tried to use PPMC and committer to measure whose contribution is
higher. This mistake has been fixed in the proposal.
About releases, we will follow
https://www.apache.org/legal/release-policy.html and the guide of our
mentors. We will also take the initiative to learn other documents.
> This is not a block, the point I mentioned this is, you will need more
people.
We will keep working on this.
Bests,
Kent Yao
Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午10:53写道：
> Hi Kent
>
> Thanks for replying, but to be honest, your mail format is chaos.
> Take a look at bellowing, which is what I saw in the gmail.
>
> This may be an issue of your mail client or some else.
>
> > The current #.2 - 5 top
> > contributors will stay as committers, and they will join the PPMC during
> > incubation.
>
> PPMC usually is the default for all initial committers, considering your
> community is still small and not very active.
> You should invite all people you trusted to join, rather than pushing them
> away.
> This is not just a too-high bar issue. Excluding 4 out of 5 is strange.
> I can't guess what is the root cause, why the other 4 PPMC members don't
> touch codes but are trusted completely. Meanwhile, people with easily
> measurable code contributions are just as committers?
> We(ASF) always remind the community to recognize no-code contribution, but
> in this case, it seems the opposite? Don't recognize code contributors
> enough?
> Please reach your mentors, this seems not a typical ASF style. If you have
> other people already involved in the ASF, asking them too.
>
> And from your description of the release process, I think you should talk
> more with your mentors.
> Such as what is a qualified ASF release, what should be checked.
> This is not a block, the point I mentioned this is, you will need more
> people.
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
>
> > Hi Sheng,
> >
> > Thanks for your detailed check and valuable comments. They are
> > accommodating. I have addressed some of them directly in the proposal.
> > Please also check the table below for some Q&As.
> > QA
> > You listed only 5 core maintainers, but clearly only one has dominated
> > contributions. The distortion and skewness of the contributions are
> mainly
> > caused by the way how we merge PRs changed(mentioned in another email on
> > the list).
> > I agree with Atri, orphaned is measurable. It is better to adjust this
> > statement a little. Address the comment to notify the risk in the Kyuubi
> > proposal.
> > Also, with this fact, the core maintainers should notice whether this
> small
> > group of people to make sure your features iteration, bug fix, and
> release
> > process working by following ASF's requirements.This would not be an easy
> > job. Currently, we learn the release process from some Apache projects,
> > using major.minor.patch for versioning. As we can see, since Kyuubi 1.0,
> it
> > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes
> until
> > EOL. And it takes us about two months to finish a minor release with 3 ~
> 5
> > big improvements and various small features/bug fixes. We also will start
> > with some release candidates for preview and test before the final. This
> is
> > how we develop and release Kyuubi for now, but we do have a long way to
> go
> > and learn to match the ASF requirement.
> > And your committer list is very strange... 1. Why some of them are PPMC
> and
> > others are committers? Where this hierarchy comes from? The current PPMC
> > candidates help us a lot build the Kyuubi project and community, not
> > directly to the codebase but as mentors(before Apache). Most of them are
> > from Apache mature TLPs, which have close relationships with Kyuubi. They
> > have sufficient Apache Way experience to guide us to become a more
> > Apache-like project before submitting this proposal in many ways. We will
> > continue to promote more contributors to the PPMC/committer based on
> their
> > code or non-code contributions during incubation.
> > 2. Your core maintainers have 5, which only 4 PPMC? The current #.2 - 5
> top
> > contributors will stay as committers, and they will join the PPMC during
> > incubation.
> > 3. I think only the top 4 have over 400 LOC, why you declared you have 5
> > core developers? I didn't notice that 400 LOC is a bar for core
> developer,
> > will change 5 to 4 core developers accordingly
> > I think you don't have enough no-salaried developers, am I missing
> > anything? Yes.
> > Changed in the proposal, most of the contributors are salaried, and some
> of
> > them are willing to help in the spare time. Thanks
> > This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA in
> > Proposal. All work here is approved by NetEase and can be forward.
> > At last, you are asking about github.com/apache/incubator-kyuubi-shaded
> > repo,
> > what does this represent? I can only see you have only one source repo
> and
> > a doc repo. Removed this requirement in the proposal, and we will apply
> it
> > from IPMC if we do need this in the future.
> > ------------------------------
> >
> > Bests
> >
> > Kent Yao
> >
> > Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> >
> > > Hi
> > >
> > > I read the contributor list and the proposal
> > > > The risk of the Kyuubi project being abandoned is minimal. Many
> > > organizations are using Kyuubi to build critical big data pipelines and
> > > willing to help develop Kyuubi's community if it becomes an ASF
> project.
> > >
> > > I think the team should evaluate the status again. You listed only 5
> core
> > > maintainers, but clearly only one has dominated contributions. I agree
> > with
> > > Atri, orphaned is measurable.
> > > It is better to adjust this statement a little. Also, with this fact,
> the
> > > core maintainers should notice whether this small group of people to
> make
> > > sure your features iteration, bug fix, and release process working by
> > > following ASF's requirements.
> > > This would not be an easy job.
> > >
> > > And your committer list is very strange
> > > ```
> > > Kent Yao (yao@apache.org) (PPMC)
> > > Wenchen Fan (wenchen@apache.org) (PPMC)
> > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > > Xun Liu (liuxun@apache.org) (PPMC)
> > > Fei Wang (cn.feiwang@gmail.com)
> > > Ulysses You (ulyssesyou18@gmail.com)
> > > Cheng Pan (pan3793@gmail.com)
> > > Hongxiang Jiang (jhx1008@gmail.com)
> > > Zen Jin(xinjingziranchan@gmail.com)
> > > ```
> > > 1. Why some of them are PPMC and others are committers? Where this
> > > hierarchy comes from?
> > > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy
> makes
> > > sense, most likely not)
> > > 3. I think only the top 4 have over 400 LOC, why you declared you have
> 5
> > > core developers?
> > >
> > > Another re-evaluation request is about `Reliance on Salaried
> Developers:`
> > > > They are all passionate about the project, and we are confident that
> > the
> > > project will continue even if no salaried developers contribute to the
> > > project.
> > >
> > > I think you don't have enough no-salaried developers, am I missing
> > > anything?
> > >
> > > About IP clearance, `Initial Source and Intellectual Property
> Submission
> > > Plan`
> > > > As soon as Kyuubi is approved to join Apache Incubator, our initial
> > > committers will submit iCLA(s) and CCLA(s). The codebase is already
> > > licensed under the Apache License 2.0.
> > >
> > > This project belongs to NetEase Inc. clearly, why NO SGA?
> > >
> > > At last, you are asking about `
> > github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> > <http://github.com/apache/incubator-kyuubi-shaded>
> > > <http://github.com/apache/incubator-kyuubi-shaded>
> > > repo, what does this represent? I can only see you have only one source
> > > repo and a doc repo.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> > >
> > > > Thanks.
> > > >
> > > > I looked at the link you advised on and if I am reading it correctly,
> > top
> > > > committer has 630ish commits and the next committer has 73 commits?
> > > >
> > > > IMO this is an indication of the community not being active enough
> and
> > > > hence the risk of project getting orphaned is measurable.
> > > >
> > > > Atri
> > > >
> > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Atri Sharma,
> > > > >
> > > > > It's easier to check out the recent contribution of kyuubi from
> this
> > > > > link[1].
> > > > > We already identified the one person contribution issue of kyuubi
> > when
> > > > > Kent came to us and we kept reminding him to put more effort into
> > > > > building the community.
> > > > > Last week I had a long F2F conversation with Kent, I think he
> already
> > > > > knows about it and there have been some improvements in recent
> months
> > > > > but there is still a long way to go if Kyuubi is accepted as a
> > podling
> > > > > in the Apache Incubator.
> > > > >
> > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > > >
> > > > > Willem Jiang
> > > > >
> > > > > Twitter: willemjiang
> > > > > Weibo: 姜宁willem
> > > > >
> > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org>
> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I took a look at commit log and seems like all the commits for
> > last 3
> > > > > > months have been done by a single person.
> > > > > >
> > > > > > How active are the remaining committers?
> > > > > >
> > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > > > > >
> > > > > > > Hi Alexander Alten,
> > > > > > >
> > > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > > >
> > > > > > > Not being very familiar with Dremel, I just took a quick look
> at
> > > the
> > > > > > > profile of a very similar and exciting project, Apache Drill.
> If
> > I
> > > > > > > understand correctly, Kyuubi is built on top of the compute
> layer
> > > > > > > (Apache Spark), while Apache Drill is built on top of the
> storage
> > > > > > > layer.
> > > > > > >
> > > > > > > Apache Spark has a large user base and usage scenarios all over
> > the
> > > > > > > world. When users have some basic Spark programming and tuning
> > > > skills,
> > > > > > > things are easy for them. But many times, most users do not
> have
> > > all
> > > > > > > of these abilities when they use Spark directly or indirectly.
> We
> > > > want
> > > > > > > to build a more easy-to-use platform on top of it to help these
> > > > users.
> > > > > > > On the one hand, we will work on the interface layer to make it
> > > easy
> > > > > > > to access (of course, this effort will make our project look
> very
> > > > > > > similar to other projects, such as HiveServer2). On the other
> > hand,
> > > > we
> > > > > > > will work under Kyuubi's multi-tenant architecture to extend
> our
> > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more
> > > stable
> > > > > > > and efficient in our scenario.
> > > > > > >
> > > > > > > What's more, we are going to add more on-prem Kyuubi engine
> > types,
> > > > e.g.
> > > > > > > Flink.
> > > > > > >
> > > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > > > >
> > > > > > > >
> > > > > > > > Hey,
> > > > > > > >
> > > > > > > > Looks like an interesting project, competition to Starburst.
> > How
> > > > > does it
> > > > > > > > compare to Apache Dremel?
> > > > > > > >
> > > > > > > > +1 (non-binding) for incubation.
> > > > > > > >
> > > > > > > > Thanks and stay safe,
> > > > > > > > --alex
> > > > > > > >
> > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > > > > > > >
> > > > > > > > > Dear all,
> > > > > > > > >
> > > > > > > > > We want to propose Kyuubi as a new Apache Incubator
> project.
> > > > > > > > >
> > > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC
> server
> > > for
> > > > > > > > > large-scale data management, processing, and analytics,
> built
> > > on
> > > > > top
> > > > > > > > > of Apache Spark and designed to support more engines (i.e.,
> > > > Apache
> > > > > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box"
> tool
> > > for
> > > > > data
> > > > > > > > > warehouses and data lakes. Kyuubi concentrates on improving
> > the
> > > > > > > > > experience for non-Spark users, making it easy for them to
> > use
> > > > > Spark
> > > > > > > > > as a service and focus on business data of their own, not
> > where
> > > > it
> > > > > > > > > stores, how it computes. NetEase has open-sourced Kyuubi
> > since
> > > > > 2018,
> > > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > > >
> > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo
> > Zhang,
> > > > > Akira
> > > > > > > > > Ajisaka for being the mentors of the Kyuubi project and
> > helping
> > > > us
> > > > > set
> > > > > > > > > up this proposal. The proposal can be found at
> > > > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> > .
> > > > > > > > >
> > > > > > > > > Looking forward to your feedback and thank you very much.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Kent Yao
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail:
> > > general-unsubscribe@incubator.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Kent Yao
> > > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase
> Corp.
> > > > > > > a spark enthusiast
> > > > > > > kyuubiis a unified multi-tenant JDBC interface for large-scale
> > data
> > > > > > > processing and analytics, built on top of Apache Spark.
> > > > > > >
> > > > > > > spark-authorizerA Spark SQL extension which provides SQL
> Standard
> > > > > > > Authorization for Apache Spark.
> > > > > > > spark-postgres A library for reading data from and transferring
> > > data
> > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x
> > > > faster.
> > > > > > > itatchiA library that brings useful functions from various
> modern
> > > > > > > database management systems to Apache Spark.
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > *Kent Yao *
> > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > a spark enthusiast
> > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant
> JDBC
> > interface for large-scale data processing and analytics, built on top of
> > Apache
> > Spark <http://spark.apache.org/>.
> > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> SQL
> > extension which provides SQL Standard Authorization for Apache Spark
> > <http://spark.apache.org/>.
> > spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
> for
> > reading data from and transferring data to Postgres / Greenplum with
> Spark
> > SQL and DataFrames, 10~100x faster.
> > itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> > brings useful functions from various modern database management
> > systems to Apache
> > Spark <http://spark.apache.org/>.
> >
>
--
*Kent Yao *
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of Apache
Spark <http://spark.apache.org/>.
spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
extension which provides SQL Standard Authorization for Apache Spark
<http://spark.apache.org/>.
spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.
itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
brings useful functions from various modern database management
systems to Apache
Spark <http://spark.apache.org/>.
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Sheng Wu <wu...@gmail.com>.
Hi Kent
Thanks for replying, but to be honest, your mail format is chaos.
Take a look at bellowing, which is what I saw in the gmail.
This may be an issue of your mail client or some else.
> The current #.2 - 5 top
> contributors will stay as committers, and they will join the PPMC during
> incubation.
PPMC usually is the default for all initial committers, considering your
community is still small and not very active.
You should invite all people you trusted to join, rather than pushing them
away.
This is not just a too-high bar issue. Excluding 4 out of 5 is strange.
I can't guess what is the root cause, why the other 4 PPMC members don't
touch codes but are trusted completely. Meanwhile, people with easily
measurable code contributions are just as committers?
We(ASF) always remind the community to recognize no-code contribution, but
in this case, it seems the opposite? Don't recognize code contributors
enough?
Please reach your mentors, this seems not a typical ASF style. If you have
other people already involved in the ASF, asking them too.
And from your description of the release process, I think you should talk
more with your mentors.
Such as what is a qualified ASF release, what should be checked.
This is not a block, the point I mentioned this is, you will need more
people.
Sheng Wu 吴晟
Twitter, wusheng1108
Kent Yao <ya...@apache.org> 于2021年6月2日周三 下午10:37写道：
> Hi Sheng,
>
> Thanks for your detailed check and valuable comments. They are
> accommodating. I have addressed some of them directly in the proposal.
> Please also check the table below for some Q&As.
> QA
> You listed only 5 core maintainers, but clearly only one has dominated
> contributions. The distortion and skewness of the contributions are mainly
> caused by the way how we merge PRs changed(mentioned in another email on
> the list).
> I agree with Atri, orphaned is measurable. It is better to adjust this
> statement a little. Address the comment to notify the risk in the Kyuubi
> proposal.
> Also, with this fact, the core maintainers should notice whether this small
> group of people to make sure your features iteration, bug fix, and release
> process working by following ASF's requirements.This would not be an easy
> job. Currently, we learn the release process from some Apache projects,
> using major.minor.patch for versioning. As we can see, since Kyuubi 1.0, it
> takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes until
> EOL. And it takes us about two months to finish a minor release with 3 ~ 5
> big improvements and various small features/bug fixes. We also will start
> with some release candidates for preview and test before the final. This is
> how we develop and release Kyuubi for now, but we do have a long way to go
> and learn to match the ASF requirement.
> And your committer list is very strange... 1. Why some of them are PPMC and
> others are committers? Where this hierarchy comes from? The current PPMC
> candidates help us a lot build the Kyuubi project and community, not
> directly to the codebase but as mentors(before Apache). Most of them are
> from Apache mature TLPs, which have close relationships with Kyuubi. They
> have sufficient Apache Way experience to guide us to become a more
> Apache-like project before submitting this proposal in many ways. We will
> continue to promote more contributors to the PPMC/committer based on their
> code or non-code contributions during incubation.
> 2. Your core maintainers have 5, which only 4 PPMC? The current #.2 - 5 top
> contributors will stay as committers, and they will join the PPMC during
> incubation.
> 3. I think only the top 4 have over 400 LOC, why you declared you have 5
> core developers? I didn't notice that 400 LOC is a bar for core developer,
> will change 5 to 4 core developers accordingly
> I think you don't have enough no-salaried developers, am I missing
> anything? Yes.
> Changed in the proposal, most of the contributors are salaried, and some of
> them are willing to help in the spare time. Thanks
> This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA in
> Proposal. All work here is approved by NetEase and can be forward.
> At last, you are asking about github.com/apache/incubator-kyuubi-shaded
> repo,
> what does this represent? I can only see you have only one source repo and
> a doc repo. Removed this requirement in the proposal, and we will apply it
> from IPMC if we do need this in the future.
> ------------------------------
>
> Bests
>
> Kent Yao
>
> Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
>
> > Hi
> >
> > I read the contributor list and the proposal
> > > The risk of the Kyuubi project being abandoned is minimal. Many
> > organizations are using Kyuubi to build critical big data pipelines and
> > willing to help develop Kyuubi's community if it becomes an ASF project.
> >
> > I think the team should evaluate the status again. You listed only 5 core
> > maintainers, but clearly only one has dominated contributions. I agree
> with
> > Atri, orphaned is measurable.
> > It is better to adjust this statement a little. Also, with this fact, the
> > core maintainers should notice whether this small group of people to make
> > sure your features iteration, bug fix, and release process working by
> > following ASF's requirements.
> > This would not be an easy job.
> >
> > And your committer list is very strange
> > ```
> > Kent Yao (yao@apache.org) (PPMC)
> > Wenchen Fan (wenchen@apache.org) (PPMC)
> > Wanqiang Ji (jiwq@apache.org) (PPMC)
> > Xun Liu (liuxun@apache.org) (PPMC)
> > Fei Wang (cn.feiwang@gmail.com)
> > Ulysses You (ulyssesyou18@gmail.com)
> > Cheng Pan (pan3793@gmail.com)
> > Hongxiang Jiang (jhx1008@gmail.com)
> > Zen Jin(xinjingziranchan@gmail.com)
> > ```
> > 1. Why some of them are PPMC and others are committers? Where this
> > hierarchy comes from?
> > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy makes
> > sense, most likely not)
> > 3. I think only the top 4 have over 400 LOC, why you declared you have 5
> > core developers?
> >
> > Another re-evaluation request is about `Reliance on Salaried Developers:`
> > > They are all passionate about the project, and we are confident that
> the
> > project will continue even if no salaried developers contribute to the
> > project.
> >
> > I think you don't have enough no-salaried developers, am I missing
> > anything?
> >
> > About IP clearance, `Initial Source and Intellectual Property Submission
> > Plan`
> > > As soon as Kyuubi is approved to join Apache Incubator, our initial
> > committers will submit iCLA(s) and CCLA(s). The codebase is already
> > licensed under the Apache License 2.0.
> >
> > This project belongs to NetEase Inc. clearly, why NO SGA?
> >
> > At last, you are asking about `
> github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> > <http://github.com/apache/incubator-kyuubi-shaded>
> > repo, what does this represent? I can only see you have only one source
> > repo and a doc repo.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> >
> > > Thanks.
> > >
> > > I looked at the link you advised on and if I am reading it correctly,
> top
> > > committer has 630ish commits and the next committer has 73 commits?
> > >
> > > IMO this is an indication of the community not being active enough and
> > > hence the risk of project getting orphaned is measurable.
> > >
> > > Atri
> > >
> > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com>
> wrote:
> > >
> > > > Hi Atri Sharma,
> > > >
> > > > It's easier to check out the recent contribution of kyuubi from this
> > > > link[1].
> > > > We already identified the one person contribution issue of kyuubi
> when
> > > > Kent came to us and we kept reminding him to put more effort into
> > > > building the community.
> > > > Last week I had a long F2F conversation with Kent, I think he already
> > > > knows about it and there have been some improvements in recent months
> > > > but there is still a long way to go if Kyuubi is accepted as a
> podling
> > > > in the Apache Incubator.
> > > >
> > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > > >
> > > > Willem Jiang
> > > >
> > > > Twitter: willemjiang
> > > > Weibo: 姜宁willem
> > > >
> > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I took a look at commit log and seems like all the commits for
> last 3
> > > > > months have been done by a single person.
> > > > >
> > > > > How active are the remaining committers?
> > > > >
> > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > > > >
> > > > > > Hi Alexander Alten,
> > > > > >
> > > > > > Thanks for your interest and +1 for Kyuubi.
> > > > > >
> > > > > > Not being very familiar with Dremel, I just took a quick look at
> > the
> > > > > > profile of a very similar and exciting project, Apache Drill. If
> I
> > > > > > understand correctly, Kyuubi is built on top of the compute layer
> > > > > > (Apache Spark), while Apache Drill is built on top of the storage
> > > > > > layer.
> > > > > >
> > > > > > Apache Spark has a large user base and usage scenarios all over
> the
> > > > > > world. When users have some basic Spark programming and tuning
> > > skills,
> > > > > > things are easy for them. But many times, most users do not have
> > all
> > > > > > of these abilities when they use Spark directly or indirectly. We
> > > want
> > > > > > to build a more easy-to-use platform on top of it to help these
> > > users.
> > > > > > On the one hand, we will work on the interface layer to make it
> > easy
> > > > > > to access (of course, this effort will make our project look very
> > > > > > similar to other projects, such as HiveServer2). On the other
> hand,
> > > we
> > > > > > will work under Kyuubi's multi-tenant architecture to extend our
> > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more
> > stable
> > > > > > and efficient in our scenario.
> > > > > >
> > > > > > What's more, we are going to add more on-prem Kyuubi engine
> types,
> > > e.g.
> > > > > > Flink.
> > > > > >
> > > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > > >
> > > > > > >
> > > > > > > Hey,
> > > > > > >
> > > > > > > Looks like an interesting project, competition to Starburst.
> How
> > > > does it
> > > > > > > compare to Apache Dremel?
> > > > > > >
> > > > > > > +1 (non-binding) for incubation.
> > > > > > >
> > > > > > > Thanks and stay safe,
> > > > > > > --alex
> > > > > > >
> > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > > > > > >
> > > > > > > > Dear all,
> > > > > > > >
> > > > > > > > We want to propose Kyuubi as a new Apache Incubator project.
> > > > > > > >
> > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server
> > for
> > > > > > > > large-scale data management, processing, and analytics, built
> > on
> > > > top
> > > > > > > > of Apache Spark and designed to support more engines (i.e.,
> > > Apache
> > > > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool
> > for
> > > > data
> > > > > > > > warehouses and data lakes. Kyuubi concentrates on improving
> the
> > > > > > > > experience for non-Spark users, making it easy for them to
> use
> > > > Spark
> > > > > > > > as a service and focus on business data of their own, not
> where
> > > it
> > > > > > > > stores, how it computes. NetEase has open-sourced Kyuubi
> since
> > > > 2018,
> > > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > > >
> > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo
> Zhang,
> > > > Akira
> > > > > > > > Ajisaka for being the mentors of the Kyuubi project and
> helping
> > > us
> > > > set
> > > > > > > > up this proposal. The proposal can be found at
> > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> .
> > > > > > > >
> > > > > > > > Looking forward to your feedback and thank you very much.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Kent Yao
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail:
> > general-unsubscribe@incubator.apache.org
> > > > > > > > For additional commands, e-mail:
> > > general-help@incubator.apache.org
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Kent Yao
> > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > > > a spark enthusiast
> > > > > > kyuubiis a unified multi-tenant JDBC interface for large-scale
> data
> > > > > > processing and analytics, built on top of Apache Spark.
> > > > > >
> > > > > > spark-authorizerA Spark SQL extension which provides SQL Standard
> > > > > > Authorization for Apache Spark.
> > > > > > spark-postgres A library for reading data from and transferring
> > data
> > > > > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x
> > > faster.
> > > > > > itatchiA library that brings useful functions from various modern
> > > > > > database management systems to Apache Spark.
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> > >
> >
>
>
> --
>
> *Kent Yao *
> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> a spark enthusiast
> kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
> interface for large-scale data processing and analytics, built on top of
> Apache
> Spark <http://spark.apache.org/>.
> spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
> extension which provides SQL Standard Authorization for Apache Spark
> <http://spark.apache.org/>.
> spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
> reading data from and transferring data to Postgres / Greenplum with Spark
> SQL and DataFrames, 10~100x faster.
> itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
> brings useful functions from various modern database management
> systems to Apache
> Spark <http://spark.apache.org/>.
>
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Hi Sheng,
Thanks for your detailed check and valuable comments. They are
accommodating. I have addressed some of them directly in the proposal.
Please also check the table below for some Q&As.
QA
You listed only 5 core maintainers, but clearly only one has dominated
contributions. The distortion and skewness of the contributions are mainly
caused by the way how we merge PRs changed(mentioned in another email on
the list).
I agree with Atri, orphaned is measurable. It is better to adjust this
statement a little. Address the comment to notify the risk in the Kyuubi
proposal.
Also, with this fact, the core maintainers should notice whether this small
group of people to make sure your features iteration, bug fix, and release
process working by following ASF's requirements.This would not be an easy
job. Currently, we learn the release process from some Apache projects,
using major.minor.patch for versioning. As we can see, since Kyuubi 1.0, it
takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes until
EOL. And it takes us about two months to finish a minor release with 3 ~ 5
big improvements and various small features/bug fixes. We also will start
with some release candidates for preview and test before the final. This is
how we develop and release Kyuubi for now, but we do have a long way to go
and learn to match the ASF requirement.
And your committer list is very strange... 1. Why some of them are PPMC and
others are committers? Where this hierarchy comes from? The current PPMC
candidates help us a lot build the Kyuubi project and community, not
directly to the codebase but as mentors(before Apache). Most of them are
from Apache mature TLPs, which have close relationships with Kyuubi. They
have sufficient Apache Way experience to guide us to become a more
Apache-like project before submitting this proposal in many ways. We will
continue to promote more contributors to the PPMC/committer based on their
code or non-code contributions during incubation.
2. Your core maintainers have 5, which only 4 PPMC? The current #.2 - 5 top
contributors will stay as committers, and they will join the PPMC during
incubation.
3. I think only the top 4 have over 400 LOC, why you declared you have 5
core developers? I didn't notice that 400 LOC is a bar for core developer,
will change 5 to 4 core developers accordingly
I think you don't have enough no-salaried developers, am I missing
anything? Yes.
Changed in the proposal, most of the contributors are salaried, and some of
them are willing to help in the spare time. Thanks
This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA in
Proposal. All work here is approved by NetEase and can be forward.
At last, you are asking about github.com/apache/incubator-kyuubi-shaded repo,
what does this represent? I can only see you have only one source repo and
a doc repo. Removed this requirement in the proposal, and we will apply it
from IPMC if we do need this in the future.
------------------------------
Bests
Kent Yao
Sheng Wu <wu...@gmail.com> 于2021年6月2日周三 下午5:45写道：
> Hi
>
> I read the contributor list and the proposal
> > The risk of the Kyuubi project being abandoned is minimal. Many
> organizations are using Kyuubi to build critical big data pipelines and
> willing to help develop Kyuubi's community if it becomes an ASF project.
>
> I think the team should evaluate the status again. You listed only 5 core
> maintainers, but clearly only one has dominated contributions. I agree with
> Atri, orphaned is measurable.
> It is better to adjust this statement a little. Also, with this fact, the
> core maintainers should notice whether this small group of people to make
> sure your features iteration, bug fix, and release process working by
> following ASF's requirements.
> This would not be an easy job.
>
> And your committer list is very strange
> ```
> Kent Yao (yao@apache.org) (PPMC)
> Wenchen Fan (wenchen@apache.org) (PPMC)
> Wanqiang Ji (jiwq@apache.org) (PPMC)
> Xun Liu (liuxun@apache.org) (PPMC)
> Fei Wang (cn.feiwang@gmail.com)
> Ulysses You (ulyssesyou18@gmail.com)
> Cheng Pan (pan3793@gmail.com)
> Hongxiang Jiang (jhx1008@gmail.com)
> Zen Jin(xinjingziranchan@gmail.com)
> ```
> 1. Why some of them are PPMC and others are committers? Where this
> hierarchy comes from?
> 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy makes
> sense, most likely not)
> 3. I think only the top 4 have over 400 LOC, why you declared you have 5
> core developers?
>
> Another re-evaluation request is about `Reliance on Salaried Developers:`
> > They are all passionate about the project, and we are confident that the
> project will continue even if no salaried developers contribute to the
> project.
>
> I think you don't have enough no-salaried developers, am I missing
> anything?
>
> About IP clearance, `Initial Source and Intellectual Property Submission
> Plan`
> > As soon as Kyuubi is approved to join Apache Incubator, our initial
> committers will submit iCLA(s) and CCLA(s). The codebase is already
> licensed under the Apache License 2.0.
>
> This project belongs to NetEase Inc. clearly, why NO SGA?
>
> At last, you are asking about `github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> repo, what does this represent? I can only see you have only one source
> repo and a doc repo.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
>
> > Thanks.
> >
> > I looked at the link you advised on and if I am reading it correctly, top
> > committer has 630ish commits and the next committer has 73 commits?
> >
> > IMO this is an indication of the community not being active enough and
> > hence the risk of project getting orphaned is measurable.
> >
> > Atri
> >
> > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com> wrote:
> >
> > > Hi Atri Sharma,
> > >
> > > It's easier to check out the recent contribution of kyuubi from this
> > > link[1].
> > > We already identified the one person contribution issue of kyuubi when
> > > Kent came to us and we kept reminding him to put more effort into
> > > building the community.
> > > Last week I had a long F2F conversation with Kent, I think he already
> > > knows about it and there have been some improvements in recent months
> > > but there is still a long way to go if Kyuubi is accepted as a podling
> > > in the Apache Incubator.
> > >
> > > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> > >
> > > Willem Jiang
> > >
> > > Twitter: willemjiang
> > > Weibo: 姜宁willem
> > >
> > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I took a look at commit log and seems like all the commits for last 3
> > > > months have been done by a single person.
> > > >
> > > > How active are the remaining committers?
> > > >
> > > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > > >
> > > > > Hi Alexander Alten,
> > > > >
> > > > > Thanks for your interest and +1 for Kyuubi.
> > > > >
> > > > > Not being very familiar with Dremel, I just took a quick look at
> the
> > > > > profile of a very similar and exciting project, Apache Drill. If I
> > > > > understand correctly, Kyuubi is built on top of the compute layer
> > > > > (Apache Spark), while Apache Drill is built on top of the storage
> > > > > layer.
> > > > >
> > > > > Apache Spark has a large user base and usage scenarios all over the
> > > > > world. When users have some basic Spark programming and tuning
> > skills,
> > > > > things are easy for them. But many times, most users do not have
> all
> > > > > of these abilities when they use Spark directly or indirectly. We
> > want
> > > > > to build a more easy-to-use platform on top of it to help these
> > users.
> > > > > On the one hand, we will work on the interface layer to make it
> easy
> > > > > to access (of course, this effort will make our project look very
> > > > > similar to other projects, such as HiveServer2). On the other hand,
> > we
> > > > > will work under Kyuubi's multi-tenant architecture to extend our
> > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more
> stable
> > > > > and efficient in our scenario.
> > > > >
> > > > > What's more, we are going to add more on-prem Kyuubi engine types,
> > e.g.
> > > > > Flink.
> > > > >
> > > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > > >
> > > > > >
> > > > > > Hey,
> > > > > >
> > > > > > Looks like an interesting project, competition to Starburst. How
> > > does it
> > > > > > compare to Apache Dremel?
> > > > > >
> > > > > > +1 (non-binding) for incubation.
> > > > > >
> > > > > > Thanks and stay safe,
> > > > > > --alex
> > > > > >
> > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > > > > >
> > > > > > > Dear all,
> > > > > > >
> > > > > > > We want to propose Kyuubi as a new Apache Incubator project.
> > > > > > >
> > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server
> for
> > > > > > > large-scale data management, processing, and analytics, built
> on
> > > top
> > > > > > > of Apache Spark and designed to support more engines (i.e.,
> > Apache
> > > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool
> for
> > > data
> > > > > > > warehouses and data lakes. Kyuubi concentrates on improving the
> > > > > > > experience for non-Spark users, making it easy for them to use
> > > Spark
> > > > > > > as a service and focus on business data of their own, not where
> > it
> > > > > > > stores, how it computes. NetEase has open-sourced Kyuubi since
> > > 2018,
> > > > > > > https://github.com/NetEase/kyuubi.
> > > > > > >
> > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang,
> > > Akira
> > > > > > > Ajisaka for being the mentors of the Kyuubi project and helping
> > us
> > > set
> > > > > > > up this proposal. The proposal can be found at
> > > > > > >
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> > > > > > >
> > > > > > > Looking forward to your feedback and thank you very much.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Kent Yao
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Kent Yao
> > > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > > a spark enthusiast
> > > > > kyuubiis a unified multi-tenant JDBC interface for large-scale data
> > > > > processing and analytics, built on top of Apache Spark.
> > > > >
> > > > > spark-authorizerA Spark SQL extension which provides SQL Standard
> > > > > Authorization for Apache Spark.
> > > > > spark-postgres A library for reading data from and transferring
> data
> > > > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x
> > faster.
> > > > > itatchiA library that brings useful functions from various modern
> > > > > database management systems to Apache Spark.
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
>
--
*Kent Yao *
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of Apache
Spark <http://spark.apache.org/>.
spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
extension which provides SQL Standard Authorization for Apache Spark
<http://spark.apache.org/>.
spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.
itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
brings useful functions from various modern database management
systems to Apache
Spark <http://spark.apache.org/>.
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Sheng Wu <wu...@gmail.com>.
Hi
I read the contributor list and the proposal
> The risk of the Kyuubi project being abandoned is minimal. Many
organizations are using Kyuubi to build critical big data pipelines and
willing to help develop Kyuubi's community if it becomes an ASF project.
I think the team should evaluate the status again. You listed only 5 core
maintainers, but clearly only one has dominated contributions. I agree with
Atri, orphaned is measurable.
It is better to adjust this statement a little. Also, with this fact, the
core maintainers should notice whether this small group of people to make
sure your features iteration, bug fix, and release process working by
following ASF's requirements.
This would not be an easy job.
And your committer list is very strange
```
Kent Yao (yao@apache.org) (PPMC)
Wenchen Fan (wenchen@apache.org) (PPMC)
Wanqiang Ji (jiwq@apache.org) (PPMC)
Xun Liu (liuxun@apache.org) (PPMC)
Fei Wang (cn.feiwang@gmail.com)
Ulysses You (ulyssesyou18@gmail.com)
Cheng Pan (pan3793@gmail.com)
Hongxiang Jiang (jhx1008@gmail.com)
Zen Jin(xinjingziranchan@gmail.com)
```
1. Why some of them are PPMC and others are committers? Where this
hierarchy comes from?
2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy makes
sense, most likely not)
3. I think only the top 4 have over 400 LOC, why you declared you have 5
core developers?
Another re-evaluation request is about `Reliance on Salaried Developers:`
> They are all passionate about the project, and we are confident that the
project will continue even if no salaried developers contribute to the
project.
I think you don't have enough no-salaried developers, am I missing anything?
About IP clearance, `Initial Source and Intellectual Property Submission
Plan`
> As soon as Kyuubi is approved to join Apache Incubator, our initial
committers will submit iCLA(s) and CCLA(s). The codebase is already
licensed under the Apache License 2.0.
This project belongs to NetEase Inc. clearly, why NO SGA?
At last, you are asking about `github.com/apache/incubator-kyuubi-shaded`
repo, what does this represent? I can only see you have only one source
repo and a doc repo.
Sheng Wu 吴晟
Twitter, wusheng1108
Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:49写道：
> Thanks.
>
> I looked at the link you advised on and if I am reading it correctly, top
> committer has 630ish commits and the next committer has 73 commits?
>
> IMO this is an indication of the community not being active enough and
> hence the risk of project getting orphaned is measurable.
>
> Atri
>
> On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com> wrote:
>
> > Hi Atri Sharma,
> >
> > It's easier to check out the recent contribution of kyuubi from this
> > link[1].
> > We already identified the one person contribution issue of kyuubi when
> > Kent came to us and we kept reminding him to put more effort into
> > building the community.
> > Last week I had a long F2F conversation with Kent, I think he already
> > knows about it and there have been some improvements in recent months
> > but there is still a long way to go if Kyuubi is accepted as a podling
> > in the Apache Incubator.
> >
> > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org> wrote:
> > >
> > > Hi,
> > >
> > > I took a look at commit log and seems like all the commits for last 3
> > > months have been done by a single person.
> > >
> > > How active are the remaining committers?
> > >
> > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > >
> > > > Hi Alexander Alten,
> > > >
> > > > Thanks for your interest and +1 for Kyuubi.
> > > >
> > > > Not being very familiar with Dremel, I just took a quick look at the
> > > > profile of a very similar and exciting project, Apache Drill. If I
> > > > understand correctly, Kyuubi is built on top of the compute layer
> > > > (Apache Spark), while Apache Drill is built on top of the storage
> > > > layer.
> > > >
> > > > Apache Spark has a large user base and usage scenarios all over the
> > > > world. When users have some basic Spark programming and tuning
> skills,
> > > > things are easy for them. But many times, most users do not have all
> > > > of these abilities when they use Spark directly or indirectly. We
> want
> > > > to build a more easy-to-use platform on top of it to help these
> users.
> > > > On the one hand, we will work on the interface layer to make it easy
> > > > to access (of course, this effort will make our project look very
> > > > similar to other projects, such as HiveServer2). On the other hand,
> we
> > > > will work under Kyuubi's multi-tenant architecture to extend our
> > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more stable
> > > > and efficient in our scenario.
> > > >
> > > > What's more, we are going to add more on-prem Kyuubi engine types,
> e.g.
> > > > Flink.
> > > >
> > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > >
> > > > >
> > > > > Hey,
> > > > >
> > > > > Looks like an interesting project, competition to Starburst. How
> > does it
> > > > > compare to Apache Dremel?
> > > > >
> > > > > +1 (non-binding) for incubation.
> > > > >
> > > > > Thanks and stay safe,
> > > > > --alex
> > > > >
> > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > > > >
> > > > > > Dear all,
> > > > > >
> > > > > > We want to propose Kyuubi as a new Apache Incubator project.
> > > > > >
> > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > > > > > large-scale data management, processing, and analytics, built on
> > top
> > > > > > of Apache Spark and designed to support more engines (i.e.,
> Apache
> > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for
> > data
> > > > > > warehouses and data lakes. Kyuubi concentrates on improving the
> > > > > > experience for non-Spark users, making it easy for them to use
> > Spark
> > > > > > as a service and focus on business data of their own, not where
> it
> > > > > > stores, how it computes. NetEase has open-sourced Kyuubi since
> > 2018,
> > > > > > https://github.com/NetEase/kyuubi.
> > > > > >
> > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang,
> > Akira
> > > > > > Ajisaka for being the mentors of the Kyuubi project and helping
> us
> > set
> > > > > > up this proposal. The proposal can be found at
> > > > > >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> > > > > >
> > > > > > Looking forward to your feedback and thank you very much.
> > > > > >
> > > > > > Best regards,
> > > > > > Kent Yao
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Kent Yao
> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > a spark enthusiast
> > > > kyuubiis a unified multi-tenant JDBC interface for large-scale data
> > > > processing and analytics, built on top of Apache Spark.
> > > >
> > > > spark-authorizerA Spark SQL extension which provides SQL Standard
> > > > Authorization for Apache Spark.
> > > > spark-postgres A library for reading data from and transferring data
> > > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x
> faster.
> > > > itatchiA library that brings useful functions from various modern
> > > > database management systems to Apache Spark.
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Hi Atri Sharma,
I just notice that you might mean '3 years' not '3 months' in your
first mail, LOL. Sorry for the confusion.
After borrowed the PR merge tool[1] from one Apache project to learn how
they manage PR and commits, we start to use it to merge PRs, which squashes
all commits in one single PR before gets merged to the main codebase. So
the gap here is not that big as it looks on the Github contributors page if
we don't switch to use that tool and merge all the PR commits to the main
codebase.
Sorry for the confusion about this too.
Bests,
Kent Yao
[1]
https://github.com/NetEase/kyuubi/commit/aa4c8bb6e5286fd1d226fd8e03841f33948ebeb4
Atri Sharma <at...@apache.org> 于2021年6月2日周三 下午4:48写道：
> Thanks.
>
> I looked at the link you advised on and if I am reading it correctly, top
> committer has 630ish commits and the next committer has 73 commits?
>
> IMO this is an indication of the community not being active enough and
> hence the risk of project getting orphaned is measurable.
>
> Atri
>
> On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com> wrote:
>
> > Hi Atri Sharma,
> >
> > It's easier to check out the recent contribution of kyuubi from this
> > link[1].
> > We already identified the one person contribution issue of kyuubi when
> > Kent came to us and we kept reminding him to put more effort into
> > building the community.
> > Last week I had a long F2F conversation with Kent, I think he already
> > knows about it and there have been some improvements in recent months
> > but there is still a long way to go if Kyuubi is accepted as a podling
> > in the Apache Incubator.
> >
> > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org> wrote:
> > >
> > > Hi,
> > >
> > > I took a look at commit log and seems like all the commits for last 3
> > > months have been done by a single person.
> > >
> > > How active are the remaining committers?
> > >
> > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > >
> > > > Hi Alexander Alten,
> > > >
> > > > Thanks for your interest and +1 for Kyuubi.
> > > >
> > > > Not being very familiar with Dremel, I just took a quick look at the
> > > > profile of a very similar and exciting project, Apache Drill. If I
> > > > understand correctly, Kyuubi is built on top of the compute layer
> > > > (Apache Spark), while Apache Drill is built on top of the storage
> > > > layer.
> > > >
> > > > Apache Spark has a large user base and usage scenarios all over the
> > > > world. When users have some basic Spark programming and tuning
> skills,
> > > > things are easy for them. But many times, most users do not have all
> > > > of these abilities when they use Spark directly or indirectly. We
> want
> > > > to build a more easy-to-use platform on top of it to help these
> users.
> > > > On the one hand, we will work on the interface layer to make it easy
> > > > to access (of course, this effort will make our project look very
> > > > similar to other projects, such as HiveServer2). On the other hand,
> we
> > > > will work under Kyuubi's multi-tenant architecture to extend our
> > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more stable
> > > > and efficient in our scenario.
> > > >
> > > > What's more, we are going to add more on-prem Kyuubi engine types,
> e.g.
> > > > Flink.
> > > >
> > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > >
> > > > >
> > > > > Hey,
> > > > >
> > > > > Looks like an interesting project, competition to Starburst. How
> > does it
> > > > > compare to Apache Dremel?
> > > > >
> > > > > +1 (non-binding) for incubation.
> > > > >
> > > > > Thanks and stay safe,
> > > > > --alex
> > > > >
> > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > > > >
> > > > > > Dear all,
> > > > > >
> > > > > > We want to propose Kyuubi as a new Apache Incubator project.
> > > > > >
> > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > > > > > large-scale data management, processing, and analytics, built on
> > top
> > > > > > of Apache Spark and designed to support more engines (i.e.,
> Apache
> > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for
> > data
> > > > > > warehouses and data lakes. Kyuubi concentrates on improving the
> > > > > > experience for non-Spark users, making it easy for them to use
> > Spark
> > > > > > as a service and focus on business data of their own, not where
> it
> > > > > > stores, how it computes. NetEase has open-sourced Kyuubi since
> > 2018,
> > > > > > https://github.com/NetEase/kyuubi.
> > > > > >
> > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang,
> > Akira
> > > > > > Ajisaka for being the mentors of the Kyuubi project and helping
> us
> > set
> > > > > > up this proposal. The proposal can be found at
> > > > > >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> > > > > >
> > > > > > Looking forward to your feedback and thank you very much.
> > > > > >
> > > > > > Best regards,
> > > > > > Kent Yao
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Kent Yao
> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > a spark enthusiast
> > > > kyuubiis a unified multi-tenant JDBC interface for large-scale data
> > > > processing and analytics, built on top of Apache Spark.
> > > >
> > > > spark-authorizerA Spark SQL extension which provides SQL Standard
> > > > Authorization for Apache Spark.
> > > > spark-postgres A library for reading data from and transferring data
> > > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x
> faster.
> > > > itatchiA library that brings useful functions from various modern
> > > > database management systems to Apache Spark.
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
--
*Kent Yao *
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of Apache
Spark <http://spark.apache.org/>.
spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
extension which provides SQL Standard Authorization for Apache Spark
<http://spark.apache.org/>.
spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.
itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
brings useful functions from various modern database management
systems to Apache
Spark <http://spark.apache.org/>.
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Willem Jiang <wi...@gmail.com>.
Hi Atri,
I shared the same concern with you when Kent Yao came to me two month ago.
I think he already realized it and made some improvements recently.
He told me he want to build up the community around Sparks ecosystem
by donating this project into ASF last week.
He did a presentation about it in China Open Source Hackathon
2012[1], and hold a meetup about Sparks.
From my mentoring experience, I saw the similar situation, if the
project main developer realize the community over code and will to do
some change, the community will be built soon. So I'd like to give
this project a try.
[1]https://www.163.com/dy/article/GAPRPV840518DLIO.html
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Wed, Jun 2, 2021 at 4:48 PM Atri Sharma <at...@apache.org> wrote:
>
> Thanks.
>
> I looked at the link you advised on and if I am reading it correctly, top
> committer has 630ish commits and the next committer has 73 commits?
>
> IMO this is an indication of the community not being active enough and
> hence the risk of project getting orphaned is measurable.
>
> Atri
>
> On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com> wrote:
>
> > Hi Atri Sharma,
> >
> > It's easier to check out the recent contribution of kyuubi from this
> > link[1].
> > We already identified the one person contribution issue of kyuubi when
> > Kent came to us and we kept reminding him to put more effort into
> > building the community.
> > Last week I had a long F2F conversation with Kent, I think he already
> > knows about it and there have been some improvements in recent months
> > but there is still a long way to go if Kyuubi is accepted as a podling
> > in the Apache Incubator.
> >
> > [1]https://github.com/NetEase/kyuubi/graphs/contributors
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org> wrote:
> > >
> > > Hi,
> > >
> > > I took a look at commit log and seems like all the commits for last 3
> > > months have been done by a single person.
> > >
> > > How active are the remaining committers?
> > >
> > > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> > >
> > > > Hi Alexander Alten,
> > > >
> > > > Thanks for your interest and +1 for Kyuubi.
> > > >
> > > > Not being very familiar with Dremel, I just took a quick look at the
> > > > profile of a very similar and exciting project, Apache Drill. If I
> > > > understand correctly, Kyuubi is built on top of the compute layer
> > > > (Apache Spark), while Apache Drill is built on top of the storage
> > > > layer.
> > > >
> > > > Apache Spark has a large user base and usage scenarios all over the
> > > > world. When users have some basic Spark programming and tuning skills,
> > > > things are easy for them. But many times, most users do not have all
> > > > of these abilities when they use Spark directly or indirectly. We want
> > > > to build a more easy-to-use platform on top of it to help these users.
> > > > On the one hand, we will work on the interface layer to make it easy
> > > > to access (of course, this effort will make our project look very
> > > > similar to other projects, such as HiveServer2). On the other hand, we
> > > > will work under Kyuubi's multi-tenant architecture to extend our
> > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more stable
> > > > and efficient in our scenario.
> > > >
> > > > What's more, we are going to add more on-prem Kyuubi engine types, e.g.
> > > > Flink.
> > > >
> > > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > > >
> > > > >
> > > > > Hey,
> > > > >
> > > > > Looks like an interesting project, competition to Starburst. How
> > does it
> > > > > compare to Apache Dremel?
> > > > >
> > > > > +1 (non-binding) for incubation.
> > > > >
> > > > > Thanks and stay safe,
> > > > > --alex
> > > > >
> > > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > > > >
> > > > > > Dear all,
> > > > > >
> > > > > > We want to propose Kyuubi as a new Apache Incubator project.
> > > > > >
> > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > > > > > large-scale data management, processing, and analytics, built on
> > top
> > > > > > of Apache Spark and designed to support more engines (i.e., Apache
> > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for
> > data
> > > > > > warehouses and data lakes. Kyuubi concentrates on improving the
> > > > > > experience for non-Spark users, making it easy for them to use
> > Spark
> > > > > > as a service and focus on business data of their own, not where it
> > > > > > stores, how it computes. NetEase has open-sourced Kyuubi since
> > 2018,
> > > > > > https://github.com/NetEase/kyuubi.
> > > > > >
> > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang,
> > Akira
> > > > > > Ajisaka for being the mentors of the Kyuubi project and helping us
> > set
> > > > > > up this proposal. The proposal can be found at
> > > > > >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> > > > > >
> > > > > > Looking forward to your feedback and thank you very much.
> > > > > >
> > > > > > Best regards,
> > > > > > Kent Yao
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Kent Yao
> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > > a spark enthusiast
> > > > kyuubiis a unified multi-tenant JDBC interface for large-scale data
> > > > processing and analytics, built on top of Apache Spark.
> > > >
> > > > spark-authorizerA Spark SQL extension which provides SQL Standard
> > > > Authorization for Apache Spark.
> > > > spark-postgres A library for reading data from and transferring data
> > > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
> > > > itatchiA library that brings useful functions from various modern
> > > > database management systems to Apache Spark.
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Atri Sharma <at...@apache.org>.
Thanks.
I looked at the link you advised on and if I am reading it correctly, top
committer has 630ish commits and the next committer has 73 commits?
IMO this is an indication of the community not being active enough and
hence the risk of project getting orphaned is measurable.
Atri
On Wed, 2 Jun 2021, 06:57 Willem Jiang, <wi...@gmail.com> wrote:
> Hi Atri Sharma,
>
> It's easier to check out the recent contribution of kyuubi from this
> link[1].
> We already identified the one person contribution issue of kyuubi when
> Kent came to us and we kept reminding him to put more effort into
> building the community.
> Last week I had a long F2F conversation with Kent, I think he already
> knows about it and there have been some improvements in recent months
> but there is still a long way to go if Kyuubi is accepted as a podling
> in the Apache Incubator.
>
> [1]https://github.com/NetEase/kyuubi/graphs/contributors
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org> wrote:
> >
> > Hi,
> >
> > I took a look at commit log and seems like all the commits for last 3
> > months have been done by a single person.
> >
> > How active are the remaining committers?
> >
> > On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> >
> > > Hi Alexander Alten,
> > >
> > > Thanks for your interest and +1 for Kyuubi.
> > >
> > > Not being very familiar with Dremel, I just took a quick look at the
> > > profile of a very similar and exciting project, Apache Drill. If I
> > > understand correctly, Kyuubi is built on top of the compute layer
> > > (Apache Spark), while Apache Drill is built on top of the storage
> > > layer.
> > >
> > > Apache Spark has a large user base and usage scenarios all over the
> > > world. When users have some basic Spark programming and tuning skills,
> > > things are easy for them. But many times, most users do not have all
> > > of these abilities when they use Spark directly or indirectly. We want
> > > to build a more easy-to-use platform on top of it to help these users.
> > > On the one hand, we will work on the interface layer to make it easy
> > > to access (of course, this effort will make our project look very
> > > similar to other projects, such as HiveServer2). On the other hand, we
> > > will work under Kyuubi's multi-tenant architecture to extend our
> > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more stable
> > > and efficient in our scenario.
> > >
> > > What's more, we are going to add more on-prem Kyuubi engine types, e.g.
> > > Flink.
> > >
> > > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> > >
> > > >
> > > > Hey,
> > > >
> > > > Looks like an interesting project, competition to Starburst. How
> does it
> > > > compare to Apache Dremel?
> > > >
> > > > +1 (non-binding) for incubation.
> > > >
> > > > Thanks and stay safe,
> > > > --alex
> > > >
> > > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > We want to propose Kyuubi as a new Apache Incubator project.
> > > > >
> > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > > > > large-scale data management, processing, and analytics, built on
> top
> > > > > of Apache Spark and designed to support more engines (i.e., Apache
> > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for
> data
> > > > > warehouses and data lakes. Kyuubi concentrates on improving the
> > > > > experience for non-Spark users, making it easy for them to use
> Spark
> > > > > as a service and focus on business data of their own, not where it
> > > > > stores, how it computes. NetEase has open-sourced Kyuubi since
> 2018,
> > > > > https://github.com/NetEase/kyuubi.
> > > > >
> > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang,
> Akira
> > > > > Ajisaka for being the mentors of the Kyuubi project and helping us
> set
> > > > > up this proposal. The proposal can be found at
> > > > >
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> > > > >
> > > > > Looking forward to your feedback and thank you very much.
> > > > >
> > > > > Best regards,
> > > > > Kent Yao
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > >
> > > Kent Yao
> > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > > a spark enthusiast
> > > kyuubiis a unified multi-tenant JDBC interface for large-scale data
> > > processing and analytics, built on top of Apache Spark.
> > >
> > > spark-authorizerA Spark SQL extension which provides SQL Standard
> > > Authorization for Apache Spark.
> > > spark-postgres A library for reading data from and transferring data
> > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
> > > itatchiA library that brings useful functions from various modern
> > > database management systems to Apache Spark.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Willem Jiang <wi...@gmail.com>.
Hi Atri Sharma,
It's easier to check out the recent contribution of kyuubi from this link[1].
We already identified the one person contribution issue of kyuubi when
Kent came to us and we kept reminding him to put more effort into
building the community.
Last week I had a long F2F conversation with Kent, I think he already
knows about it and there have been some improvements in recent months
but there is still a long way to go if Kyuubi is accepted as a podling
in the Apache Incubator.
[1]https://github.com/NetEase/kyuubi/graphs/contributors
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <at...@apache.org> wrote:
>
> Hi,
>
> I took a look at commit log and seems like all the commits for last 3
> months have been done by a single person.
>
> How active are the remaining committers?
>
> On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
>
> > Hi Alexander Alten,
> >
> > Thanks for your interest and +1 for Kyuubi.
> >
> > Not being very familiar with Dremel, I just took a quick look at the
> > profile of a very similar and exciting project, Apache Drill. If I
> > understand correctly, Kyuubi is built on top of the compute layer
> > (Apache Spark), while Apache Drill is built on top of the storage
> > layer.
> >
> > Apache Spark has a large user base and usage scenarios all over the
> > world. When users have some basic Spark programming and tuning skills,
> > things are easy for them. But many times, most users do not have all
> > of these abilities when they use Spark directly or indirectly. We want
> > to build a more easy-to-use platform on top of it to help these users.
> > On the one hand, we will work on the interface layer to make it easy
> > to access (of course, this effort will make our project look very
> > similar to other projects, such as HiveServer2). On the other hand, we
> > will work under Kyuubi's multi-tenant architecture to extend our
> > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more stable
> > and efficient in our scenario.
> >
> > What's more, we are going to add more on-prem Kyuubi engine types, e.g.
> > Flink.
> >
> > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> >
> > >
> > > Hey,
> > >
> > > Looks like an interesting project, competition to Starburst. How does it
> > > compare to Apache Dremel?
> > >
> > > +1 (non-binding) for incubation.
> > >
> > > Thanks and stay safe,
> > > --alex
> > >
> > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > >
> > > > Dear all,
> > > >
> > > > We want to propose Kyuubi as a new Apache Incubator project.
> > > >
> > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > > > large-scale data management, processing, and analytics, built on top
> > > > of Apache Spark and designed to support more engines (i.e., Apache
> > > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> > > > warehouses and data lakes. Kyuubi concentrates on improving the
> > > > experience for non-Spark users, making it easy for them to use Spark
> > > > as a service and focus on business data of their own, not where it
> > > > stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> > > > https://github.com/NetEase/kyuubi.
> > > >
> > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> > > > Ajisaka for being the mentors of the Kyuubi project and helping us set
> > > > up this proposal. The proposal can be found at
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> > > >
> > > > Looking forward to your feedback and thank you very much.
> > > >
> > > > Best regards,
> > > > Kent Yao
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> >
> >
> >
> > --
> >
> > Kent Yao
> > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > a spark enthusiast
> > kyuubiis a unified multi-tenant JDBC interface for large-scale data
> > processing and analytics, built on top of Apache Spark.
> >
> > spark-authorizerA Spark SQL extension which provides SQL Standard
> > Authorization for Apache Spark.
> > spark-postgres A library for reading data from and transferring data
> > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
> > itatchiA library that brings useful functions from various modern
> > database management systems to Apache Spark.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Hi Justion，
Thanks for your suggestions.
We won't copy their bar blindly from the Spark community. We appreciate all
the help from our contributors and will try our best to help them when they
help the community. We will continue to develop more committers among them
to make the community grow better.
Bests,
Kent Yao
Justin Mclean <ju...@classsoftware.com> 于2021年6月3日周四 上午7:08写道：
> HI,
>
> > We've open-sourced Kyuubi under Apache License v2.0 and tried out best to
> > learn the Apache Way from many mature ASF TLPs, mostly Apache Spark.
>
> Spark is a sightly unusual project that has high activity and a very high
> committer bar, I wouldn’t recommend that you copy that feature of their
> community. Some projects make people with committers after a couple of PRs
> or other involvement.
>
> Kind Regards,
> Justin
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
*Kent Yao *
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of Apache
Spark <http://spark.apache.org/>.
spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
extension which provides SQL Standard Authorization for Apache Spark
<http://spark.apache.org/>.
spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.
itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
brings useful functions from various modern database management
systems to Apache
Spark <http://spark.apache.org/>.
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Justin Mclean <ju...@classsoftware.com>.
HI,
> We've open-sourced Kyuubi under Apache License v2.0 and tried out best to
> learn the Apache Way from many mature ASF TLPs, mostly Apache Spark.
Spark is a sightly unusual project that has high activity and a very high committer bar, I wouldn’t recommend that you copy that feature of their community. Some projects make people with committers after a couple of PRs or other involvement.
Kind Regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Hi Atri Sharma,
Thanks for your interest and review.
We've open-sourced Kyuubi under Apache License v2.0 and tried out best to
learn the Apache Way from many mature ASF TLPs, mostly Apache Spark. And
many of the current Kyuubi committers have contributing history for one or
more Apache projects. We use Github a lot, so most things happen on these
open lists:
- Issue track: https://github.com/NetEase/kyuubi/issues
- Pull request: https://github.com/NetEase/kyuubi/pulls
- Other staffs: https://github.com/NetEase/kyuubi/discussions
And particularly for your question about the commit history of Kyuubi for
the last 3 months, you can find it and the activities of our committers
from https://github.com/NetEase/kyuubi/commits/master, and we have kept
both the original authors(leading-, co-) and the committer who signed-off
the commit. Another to check this is using `git log
--pretty=format:"rev:%h,author:%an,commiter:%cn,date:%ar"` with our
codebase forked to your local environment if the Github page does not offer
enough information for you.
I have to admit that the Kyuubi and its community are still young. One of
the major efforts we should make in the near future is to increase the size
and diversity of our community, hopefully, with the help of Apache
Incubator.
Bests,
Kent Yao
Atri Sharma <at...@apache.org> 于2021年6月1日周二 下午10:52写道：
> Hi,
>
> I took a look at commit log and seems like all the commits for last 3
> months have been done by a single person.
>
> How active are the remaining committers?
>
> On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
>
> > Hi Alexander Alten,
> >
> > Thanks for your interest and +1 for Kyuubi.
> >
> > Not being very familiar with Dremel, I just took a quick look at the
> > profile of a very similar and exciting project, Apache Drill. If I
> > understand correctly, Kyuubi is built on top of the compute layer
> > (Apache Spark), while Apache Drill is built on top of the storage
> > layer.
> >
> > Apache Spark has a large user base and usage scenarios all over the
> > world. When users have some basic Spark programming and tuning skills,
> > things are easy for them. But many times, most users do not have all
> > of these abilities when they use Spark directly or indirectly. We want
> > to build a more easy-to-use platform on top of it to help these users.
> > On the one hand, we will work on the interface layer to make it easy
> > to access (of course, this effort will make our project look very
> > similar to other projects, such as HiveServer2). On the other hand, we
> > will work under Kyuubi's multi-tenant architecture to extend our
> > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more stable
> > and efficient in our scenario.
> >
> > What's more, we are going to add more on-prem Kyuubi engine types, e.g.
> > Flink.
> >
> > Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
> >
> > >
> > > Hey,
> > >
> > > Looks like an interesting project, competition to Starburst. How does
> it
> > > compare to Apache Dremel?
> > >
> > > +1 (non-binding) for incubation.
> > >
> > > Thanks and stay safe,
> > > --alex
> > >
> > > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> > >
> > > > Dear all,
> > > >
> > > > We want to propose Kyuubi as a new Apache Incubator project.
> > > >
> > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > > > large-scale data management, processing, and analytics, built on top
> > > > of Apache Spark and designed to support more engines (i.e., Apache
> > > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for
> data
> > > > warehouses and data lakes. Kyuubi concentrates on improving the
> > > > experience for non-Spark users, making it easy for them to use Spark
> > > > as a service and focus on business data of their own, not where it
> > > > stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> > > > https://github.com/NetEase/kyuubi.
> > > >
> > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> > > > Ajisaka for being the mentors of the Kyuubi project and helping us
> set
> > > > up this proposal. The proposal can be found at
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> .
> > > >
> > > > Looking forward to your feedback and thank you very much.
> > > >
> > > > Best regards,
> > > > Kent Yao
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> >
> >
> >
> > --
> >
> > Kent Yao
> > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> > a spark enthusiast
> > kyuubiis a unified multi-tenant JDBC interface for large-scale data
> > processing and analytics, built on top of Apache Spark.
> >
> > spark-authorizerA Spark SQL extension which provides SQL Standard
> > Authorization for Apache Spark.
> > spark-postgres A library for reading data from and transferring data
> > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
> > itatchiA library that brings useful functions from various modern
> > database management systems to Apache Spark.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
--
*Kent Yao *
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of Apache
Spark <http://spark.apache.org/>.
spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
extension which provides SQL Standard Authorization for Apache Spark
<http://spark.apache.org/>.
spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.
itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
brings useful functions from various modern database management
systems to Apache
Spark <http://spark.apache.org/>.
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Atri Sharma <at...@apache.org>.
Hi,
I took a look at commit log and seems like all the commits for last 3
months have been done by a single person.
How active are the remaining committers?
On Mon, 31 May 2021, 12:59 Kent Yao, <ya...@apache.org> wrote:
> Hi Alexander Alten,
>
> Thanks for your interest and +1 for Kyuubi.
>
> Not being very familiar with Dremel, I just took a quick look at the
> profile of a very similar and exciting project, Apache Drill. If I
> understand correctly, Kyuubi is built on top of the compute layer
> (Apache Spark), while Apache Drill is built on top of the storage
> layer.
>
> Apache Spark has a large user base and usage scenarios all over the
> world. When users have some basic Spark programming and tuning skills,
> things are easy for them. But many times, most users do not have all
> of these abilities when they use Spark directly or indirectly. We want
> to build a more easy-to-use platform on top of it to help these users.
> On the one hand, we will work on the interface layer to make it easy
> to access (of course, this effort will make our project look very
> similar to other projects, such as HiveServer2). On the other hand, we
> will work under Kyuubi's multi-tenant architecture to extend our
> on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more stable
> and efficient in our scenario.
>
> What's more, we are going to add more on-prem Kyuubi engine types, e.g.
> Flink.
>
> Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
>
> >
> > Hey,
> >
> > Looks like an interesting project, competition to Starburst. How does it
> > compare to Apache Dremel?
> >
> > +1 (non-binding) for incubation.
> >
> > Thanks and stay safe,
> > --alex
> >
> > On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> >
> > > Dear all,
> > >
> > > We want to propose Kyuubi as a new Apache Incubator project.
> > >
> > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > > large-scale data management, processing, and analytics, built on top
> > > of Apache Spark and designed to support more engines (i.e., Apache
> > > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> > > warehouses and data lakes. Kyuubi concentrates on improving the
> > > experience for non-Spark users, making it easy for them to use Spark
> > > as a service and focus on business data of their own, not where it
> > > stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> > > https://github.com/NetEase/kyuubi.
> > >
> > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> > > Ajisaka for being the mentors of the Kyuubi project and helping us set
> > > up this proposal. The proposal can be found at
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> > >
> > > Looking forward to your feedback and thank you very much.
> > >
> > > Best regards,
> > > Kent Yao
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
>
>
>
> --
>
> Kent Yao
> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> a spark enthusiast
> kyuubiis a unified multi-tenant JDBC interface for large-scale data
> processing and analytics, built on top of Apache Spark.
>
> spark-authorizerA Spark SQL extension which provides SQL Standard
> Authorization for Apache Spark.
> spark-postgres A library for reading data from and transferring data
> to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
> itatchiA library that brings useful functions from various modern
> database management systems to Apache Spark.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Kent Yao <ya...@apache.org>.
Hi Alexander Alten,
Thanks for your interest and +1 for Kyuubi.
Not being very familiar with Dremel, I just took a quick look at the
profile of a very similar and exciting project, Apache Drill. If I
understand correctly, Kyuubi is built on top of the compute layer
(Apache Spark), while Apache Drill is built on top of the storage
layer.
Apache Spark has a large user base and usage scenarios all over the
world. When users have some basic Spark programming and tuning skills,
things are easy for them. But many times, most users do not have all
of these abilities when they use Spark directly or indirectly. We want
to build a more easy-to-use platform on top of it to help these users.
On the one hand, we will work on the interface layer to make it easy
to access (of course, this effort will make our project look very
similar to other projects, such as HiveServer2). On the other hand, we
will work under Kyuubi's multi-tenant architecture to extend our
on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more stable
and efficient in our scenario.
What's more, we are going to add more on-prem Kyuubi engine types, e.g. Flink.
Alexander Alten <al...@scalytics.io> 于2021年5月31日周一 下午2:35写道：
>
> Hey,
>
> Looks like an interesting project, competition to Starburst. How does it
> compare to Apache Dremel?
>
> +1 (non-binding) for incubation.
>
> Thanks and stay safe,
> --alex
>
> On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
>
> > Dear all,
> >
> > We want to propose Kyuubi as a new Apache Incubator project.
> >
> > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> > large-scale data management, processing, and analytics, built on top
> > of Apache Spark and designed to support more engines (i.e., Apache
> > Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> > warehouses and data lakes. Kyuubi concentrates on improving the
> > experience for non-Spark users, making it easy for them to use Spark
> > as a service and focus on business data of their own, not where it
> > stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> > https://github.com/NetEase/kyuubi.
> >
> > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> > Ajisaka for being the mentors of the Kyuubi project and helping us set
> > up this proposal. The proposal can be found at
> > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
> >
> > Looking forward to your feedback and thank you very much.
> >
> > Best regards,
> > Kent Yao
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
--
Kent Yao
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubiis a unified multi-tenant JDBC interface for large-scale data
processing and analytics, built on top of Apache Spark.
spark-authorizerA Spark SQL extension which provides SQL Standard
Authorization for Apache Spark.
spark-postgres A library for reading data from and transferring data
to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
itatchiA library that brings useful functions from various modern
database management systems to Apache Spark.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Incubating Proposal of Kyuubi
Posted by Alexander Alten <al...@scalytics.io>.
Hey,
Looks like an interesting project, competition to Starburst. How does it
compare to Apache Dremel?
+1 (non-binding) for incubation.
Thanks and stay safe,
--alex
On Mon, May 31, 2021, 08:21 Kent Yao <ya...@apache.org> wrote:
> Dear all,
>
> We want to propose Kyuubi as a new Apache Incubator project.
>
> Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for
> large-scale data management, processing, and analytics, built on top
> of Apache Spark and designed to support more engines (i.e., Apache
> Flink). We are aiming to make Kyuubi an "out-of-the-box" tool for data
> warehouses and data lakes. Kyuubi concentrates on improving the
> experience for non-Spark users, making it easy for them to use Spark
> as a service and focus on business data of their own, not where it
> stores, how it computes. NetEase has open-sourced Kyuubi since 2018,
> https://github.com/NetEase/kyuubi.
>
> Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo Zhang, Akira
> Ajisaka for being the mentors of the Kyuubi project and helping us set
> up this proposal. The proposal can be found at
> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal.
>
> Looking forward to your feedback and thank you very much.
>
> Best regards,
> Kent Yao
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>