You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Manikumar <ma...@gmail.com> on 2016/09/28 07:04:51 UTC
[DISCUSS] KIP-80: Kafka REST Server
Hi Kafka Devs,
I created KIP-80 to add Kafka REST Server to Kafka Repository.
There are already open-source alternatives are available. But we would
like to add REST server that
many users ask for under Apache Kafka repo. Many data Infra tools comes up
with Rest Interface.
It is useful to have inbuilt Rest API support for Produce, Consume messages
and admin interface for
integrating with external management and provisioning tools.This will also
allow the maintenance of
REST server and adding new features makes it easy because apache community.
The KIP wiki is the following:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
80%3A+Kafka+Rest+Server
Your comments and feedback are welcome.
Thanks,
Manikumar
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Harsha Ch <ha...@gmail.com>.
Ofir,
" personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka community, so there is a chance to
work this out."
It is important for the code to be under Apache guidelines for
contributions so that it is driven by the community.
As part of Kafka community, we would like to improve the project and add
new features. If kafka-rest repo can be donated to Apache Kafka project
we will be more than happy to welcome that and add our features on top of
it.
Thanks,
harsha
On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps <ja...@confluent.io> wrote:
Hi Manikumar,
I agree totally agree that REST is important. What I don't understand is
why we'd duplicate the existing REST interface inside the Kafka project.
That seems to needlessly fragment things.
-Jay
On Sat, Oct 1, 2016 at 5:38 AM, Manikumar <ma...@gmail.com> wrote:
> Hi Jay,
>
> Thanks for your reply.
>
> I agree that we can not add all the clients/tools available in ecosystem
> page to Kafka repo itself. But we feel REST Interface is different from
> other clients/tools. Since any language that can work with HTTP can
> easily integrate with this interface, Having an "official" REST
> interface helps user community. This also helps us to integrate well
> with external management and provisioning tools. Apache Kafka release
> with Java clients + REST interface is sufficient for most of the user
> deployments/requirements. This helps users to deal with less number
> of distributions/builds.
>
> Thanks,
> Manikumar
>
>
> On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > Hey guys,
> >
> > There's already a REST interface maintained as a separate project--it's
> > open source and apache licensed and actively maintained (
> > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> You
> > mentioned that there was some compatibility concern, but compatibility
> has
> > to do with the consumer protocol guarantees not the repo the code is in,
> > right? Not sure that concern makes sense.
> >
> > We could argue for adding pretty much anything and everything in the
> > ecosystem page in Kafka itself but I'm not sure that would make the
> project
> > more agile.
> >
> > -Jay
> >
> > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi Kafka Devs,
> > >
> > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > >
> > > There are already open-source alternatives are available. But we
would
> > > like to add REST server that
> > > many users ask for under Apache Kafka repo. Many data Infra tools
comes
> > up
> > > with Rest Interface.
> > > It is useful to have inbuilt Rest API support for Produce, Consume
> > messages
> > > and admin interface for
> > > integrating with external management and provisioning tools.This will
> > also
> > > allow the maintenance of
> > > REST server and adding new features makes it easy because apache
> > community.
> > >
> > > The KIP wiki is the following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 80%3A+Kafka+Rest+Server
> > >
> > > Your comments and feedback are welcome.
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Neha Narkhede <ne...@confluent.io>.
Harsha/Mani,
I completely agree that adding admin API support and security are important
features for the Kafka REST proxy. Luckily the roadmap items that you
mentioned as being important for a Kafka REST proxy server are exactly the
ones the community working on this REST proxy want to add to it :-)
But I haven't seen any community emails or patches being submitted by you
guys, so I'm wondering why you are concerned about whether the community is
open to accepting patches or not.
Does that make sense?
On Fri, Oct 7, 2016 at 12:03 PM Harsha Chintalapani <ka...@harsha.io> wrote:
> Ofir,
> …
> " personally think it would be quite wasteful to re-implement the REST
> gateway just because that an actively-maintained piece of Apache-licensed
> software is not governed directly by the Apache Kafka community... While
> kafka-rest repo is owned by Confluent, the contributors including the main
> one are also part of the Apache Kafka community, so there is a chance to
> work this out."
> It is important for the code to be under Apache guidelines for
> contributions so that it is driven by the community.
> As part of Kafka community, we would like to improve the project and add
> new features. If kafka-rest repo can be donated to Apache Kafka project
> we will be more than happy to welcome that and add our features on top of
> it.
>
> Thanks,
> harsha
>
> On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps <ja...@confluent.io> wrote:
>
> > Hi Manikumar,
> >
> > I agree totally agree that REST is important. What I don't understand is
> > why we'd duplicate the existing REST interface inside the Kafka project.
> > That seems to needlessly fragment things.
> >
> > -Jay
> >
> > On Sat, Oct 1, 2016 at 5:38 AM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Thanks for your reply.
> > >
> > > I agree that we can not add all the clients/tools available in
> ecosystem
> > > page to Kafka repo itself. But we feel REST Interface is different from
> > > other clients/tools. Since any language that can work with HTTP can
> > > easily integrate with this interface, Having an "official" REST
> > > interface helps user community. This also helps us to integrate well
> > > with external management and provisioning tools. Apache Kafka release
> > > with Java clients + REST interface is sufficient for most of the user
> > > deployments/requirements. This helps users to deal with less number
> > > of distributions/builds.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > >
> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > Hey guys,
> > > >
> > > > There's already a REST interface maintained as a separate
> project--it's
> > > > open source and apache licensed and actively maintained (
> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
> that?
> > > You
> > > > mentioned that there was some compatibility concern, but
> compatibility
> > > has
> > > > to do with the consumer protocol guarantees not the repo the code is
> > in,
> > > > right? Not sure that concern makes sense.
> > > >
> > > > We could argue for adding pretty much anything and everything in the
> > > > ecosystem page in Kafka itself but I'm not sure that would make the
> > > project
> > > > more agile.
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> manikumar.reddy@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Kafka Devs,
> > > > >
> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > > >
> > > > > There are already open-source alternatives are available. But we
> > would
> > > > > like to add REST server that
> > > > > many users ask for under Apache Kafka repo. Many data Infra tools
> > comes
> > > > up
> > > > > with Rest Interface.
> > > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > > messages
> > > > > and admin interface for
> > > > > integrating with external management and provisioning tools.This
> will
> > > > also
> > > > > allow the maintenance of
> > > > > REST server and adding new features makes it easy because apache
> > > > community.
> > > > >
> > > > > The KIP wiki is the following:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 80%3A+Kafka+Rest+Server
> > > > >
> > > > > Your comments and feedback are welcome.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> >
>
--
Thanks,
Neha
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Harsha Chintalapani <ka...@harsha.io>.
Ofir,
…
" personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka community, so there is a chance to
work this out."
It is important for the code to be under Apache guidelines for
contributions so that it is driven by the community.
As part of Kafka community, we would like to improve the project and add
new features. If kafka-rest repo can be donated to Apache Kafka project
we will be more than happy to welcome that and add our features on top of
it.
Thanks,
harsha
On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps <ja...@confluent.io> wrote:
> Hi Manikumar,
>
> I agree totally agree that REST is important. What I don't understand is
> why we'd duplicate the existing REST interface inside the Kafka project.
> That seems to needlessly fragment things.
>
> -Jay
>
> On Sat, Oct 1, 2016 at 5:38 AM, Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi Jay,
> >
> > Thanks for your reply.
> >
> > I agree that we can not add all the clients/tools available in ecosystem
> > page to Kafka repo itself. But we feel REST Interface is different from
> > other clients/tools. Since any language that can work with HTTP can
> > easily integrate with this interface, Having an "official" REST
> > interface helps user community. This also helps us to integrate well
> > with external management and provisioning tools. Apache Kafka release
> > with Java clients + REST interface is sufficient for most of the user
> > deployments/requirements. This helps users to deal with less number
> > of distributions/builds.
> >
> > Thanks,
> > Manikumar
> >
> >
> > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> >
> > > Hey guys,
> > >
> > > There's already a REST interface maintained as a separate project--it's
> > > open source and apache licensed and actively maintained (
> > > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> > You
> > > mentioned that there was some compatibility concern, but compatibility
> > has
> > > to do with the consumer protocol guarantees not the repo the code is
> in,
> > > right? Not sure that concern makes sense.
> > >
> > > We could argue for adding pretty much anything and everything in the
> > > ecosystem page in Kafka itself but I'm not sure that would make the
> > project
> > > more agile.
> > >
> > > -Jay
> > >
> > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <manikumar.reddy@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Kafka Devs,
> > > >
> > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > >
> > > > There are already open-source alternatives are available. But we
> would
> > > > like to add REST server that
> > > > many users ask for under Apache Kafka repo. Many data Infra tools
> comes
> > > up
> > > > with Rest Interface.
> > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > messages
> > > > and admin interface for
> > > > integrating with external management and provisioning tools.This will
> > > also
> > > > allow the maintenance of
> > > > REST server and adding new features makes it easy because apache
> > > community.
> > > >
> > > > The KIP wiki is the following:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 80%3A+Kafka+Rest+Server
> > > >
> > > > Your comments and feedback are welcome.
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Jay Kreps <ja...@confluent.io>.
Hi Manikumar,
I agree totally agree that REST is important. What I don't understand is
why we'd duplicate the existing REST interface inside the Kafka project.
That seems to needlessly fragment things.
-Jay
On Sat, Oct 1, 2016 at 5:38 AM, Manikumar <ma...@gmail.com> wrote:
> Hi Jay,
>
> Thanks for your reply.
>
> I agree that we can not add all the clients/tools available in ecosystem
> page to Kafka repo itself. But we feel REST Interface is different from
> other clients/tools. Since any language that can work with HTTP can
> easily integrate with this interface, Having an "official" REST
> interface helps user community. This also helps us to integrate well
> with external management and provisioning tools. Apache Kafka release
> with Java clients + REST interface is sufficient for most of the user
> deployments/requirements. This helps users to deal with less number
> of distributions/builds.
>
> Thanks,
> Manikumar
>
>
> On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > Hey guys,
> >
> > There's already a REST interface maintained as a separate project--it's
> > open source and apache licensed and actively maintained (
> > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> You
> > mentioned that there was some compatibility concern, but compatibility
> has
> > to do with the consumer protocol guarantees not the repo the code is in,
> > right? Not sure that concern makes sense.
> >
> > We could argue for adding pretty much anything and everything in the
> > ecosystem page in Kafka itself but I'm not sure that would make the
> project
> > more agile.
> >
> > -Jay
> >
> > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi Kafka Devs,
> > >
> > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > >
> > > There are already open-source alternatives are available. But we would
> > > like to add REST server that
> > > many users ask for under Apache Kafka repo. Many data Infra tools comes
> > up
> > > with Rest Interface.
> > > It is useful to have inbuilt Rest API support for Produce, Consume
> > messages
> > > and admin interface for
> > > integrating with external management and provisioning tools.This will
> > also
> > > allow the maintenance of
> > > REST server and adding new features makes it easy because apache
> > community.
> > >
> > > The KIP wiki is the following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 80%3A+Kafka+Rest+Server
> > >
> > > Your comments and feedback are welcome.
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Ben Davison <be...@7digital.com>.
+ 1 to rest client (don't mind if it's the current confluent version or
something else)
We are a multi language company and the quality of the other clients that
are not Java are really hit and miss. A rest endpoint a user could just
pump messages into or subscribe to would be amazing.
On Sun, Oct 2, 2016 at 9:23 PM, Harsha Chintalapani <ka...@harsha.io> wrote:
> Neha & Jay,
> We did look at the open source alternatives. Our concern
> is what's the patch acceptance and adding features/ bug-fixes to the
> existing project under a Github (although it's licensed under Apache 2.0).
> It would be great if that project made available under Apache and driven by
> the community. Adding to the above, not all Kafka users are interested in
> using the Java client API, they would like to have simple REST API where
> they can code against using any language. I do believe this adds value to
> Apache Kafka in itself.
>
> "For 1, I don't think there is value in giving in to the NIH syndrome and
> reinventing the wheel. What I'm looking for is a detailed comparison of the
> gaps and why those can't be improved in the REST proxy that already exists
> and is actively maintained."
>
> We are not looking at this as NIH. What we are worried about is a project
> that's not maintained in a community. So the process of accepting patches
> and priorities is not clear, and it's not developed in Apache community.
> Not only that, existing REST API project doesn't support new client API and
> hence there is no security support either.
> We don't know the timeline when that's made available. We would like to add
> admin functionality into the REST API. So the Roadmap of that project is
> not driven by Apache.
>
>
> "This doesn't materially have an impact on expanding the usability of
> Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
> Kafka users, and maybe 10% of those are the ones who will use the REST
> proxy. The remaining 50% are non-java client users (C, python, go, node
> etc)."
>
> REST API is most often asked feature in my interactions with Kafka users.
> In an organization, There will be independent teams who will write their
> Kafka clients using different language libraries available today, and
> there is no way to standardize this. Instead of supporting several
> different client libraries users will be interested in using a REST API
> server. The need for a REST API will only increase as more and more users
> start using Kafka.
>
> "More surface area means more work to keep things consistent. Failure
> to do that has, in fact, hurt the user experience."
> Having myriad Kafka client GitHub projects that support different languages
> hurts the user experience and pushes burden to maintain these libraries.
> REST API is a simple code base that uses existing java client libraries to
> make life easier for the users.
>
> Thanks,
> Harsha
>
> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io> wrote:
>
> > Manikumar,
> >
> > Thanks for sharing the proposal. I think there are 2 parts to this
> > discussion -
> >
> > 1. Should we rewrite a REST proxy given that there is a feature-complete,
> > open-source and actively maintained REST proxy in the community?
> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> maintain
> > the high-quality experience that Kafka users have today?
> >
> > For 1, I don't think there is value in giving in to the NIH syndrome and
> > reinventing the wheel. What I'm looking for is a detailed comparison of
> the
> > gaps and why those can't be improved in the REST proxy that already
> exists
> > and is actively maintained. For example, we depend on zkClient and have
> > found as well as fixed several bugs by working closely with the people
> who
> > maintain zkClient. This should be possible for REST proxy as well, right?
> >
> > For 2, I'd like us to review our history of expanding the surface area to
> > add more clients in the past. Here is a summary -
> >
> > - This doesn't materially have an impact on expanding the usability of
> > Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> > all
> > Kafka users, and maybe 10% of those are the ones who will use the REST
> > proxy. The remaining 50% are non-java client users (C, python, go,
> node
> > etc).
> > - People are a lot more excited about promising to contribute while
> > adding the surface area but not on an ongoing basis down the line.
> > - More surface area means more work to keep things consistent. Failure
> > to do that has, in fact, hurt the user experience.
> > - More surface area hurts agility. We want to do a few things really
> > well as well as be agile to be able to build on our core competency.
> >
> >
> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Thanks for your reply.
> > >
> > > I agree that we can not add all the clients/tools available in
> ecosystem
> > > page to Kafka repo itself. But we feel REST Interface is different from
> > > other clients/tools. Since any language that can work with HTTP can
> > > easily integrate with this interface, Having an "official" REST
> > > interface helps user community. This also helps us to integrate well
> > > with external management and provisioning tools. Apache Kafka release
> > > with Java clients + REST interface is sufficient for most of the user
> > > deployments/requirements. This helps users to deal with less number
> > > of distributions/builds.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > >
> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > Hey guys,
> > > >
> > > > There's already a REST interface maintained as a separate
> project--it's
> > > > open source and apache licensed and actively maintained (
> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
> that?
> > > You
> > > > mentioned that there was some compatibility concern, but
> compatibility
> > > has
> > > > to do with the consumer protocol guarantees not the repo the code is
> > in,
> > > > right? Not sure that concern makes sense.
> > > >
> > > > We could argue for adding pretty much anything and everything in the
> > > > ecosystem page in Kafka itself but I'm not sure that would make the
> > > project
> > > > more agile.
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> manikumar.reddy@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Kafka Devs,
> > > > >
> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > > >
> > > > > There are already open-source alternatives are available. But we
> > would
> > > > > like to add REST server that
> > > > > many users ask for under Apache Kafka repo. Many data Infra tools
> > comes
> > > > up
> > > > > with Rest Interface.
> > > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > > messages
> > > > > and admin interface for
> > > > > integrating with external management and provisioning tools.This
> will
> > > > also
> > > > > allow the maintenance of
> > > > > REST server and adding new features makes it easy because apache
> > > > community.
> > > > >
> > > > > The KIP wiki is the following:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 80%3A+Kafka+Rest+Server
> > > > >
> > > > > Your comments and feedback are welcome.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> > --
> > Thanks,
> > Neha
> >
>
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Jason Gustafson <ja...@confluent.io>.
I've been a little reluctant to wade into this discussion since I've spent
a lot of my own time on Confluent's kafka-rest project myself. Seems
obvious I would hope that time is not wasted and the project succeeds,
right? I think it's the same for a lot of others who have submitted patches
to it. Admittedly, progress has not been as rapid as we would like. It
would be great if we could get more contributions, especially from the
companies that seem interested in offering a service like it. Nevertheless,
I'd like to see an attempt to resolve any problems with that project in its
respective community first before the Kafka project takes over.
Thanks,
Jason
On Thu, Oct 20, 2016 at 2:26 PM, Harsha Ch <ha...@gmail.com> wrote:
> Jay,
> REST API is something every user is in need of. If the argument is to
> clone and write your API, this will do a disservice to the users as they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
>
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
>
> Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this REST
> server to Kafka will affect the agility of the project doesn't mak sense.
>
> It looks like your argument is either we develop all these small tools or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka could
> have been left in their own Github projects based on your arguments. What
> about the REST API is so different that such that it should stay out of the
> Kafka project? From my experience, more users are asking for the REST API.
>
> Thanks,
> Harsha
>
>
>
>
>
> On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
>
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not really
> > make sense to me. If in the future we disagree, that is the beauty of
> open
> > source, you can always fork off a copy of the code and start an
> independent
> > project either in Apache or elsewhere. Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be done
> > and we have an existing code base for HTTP/kafka access that is heavily
> > used in production seems quite silly.
> >
> > Let me give some background on how I at least think about these things.
> > I've participated in open source projects out of LinkedIn via github as
> > well as via the ASF. I don't think there is a "right" answer to how to do
> > these but rather some tradeoffs. We thought about this quite a lot in the
> > context of Kafka based on the experience with the Hadoop ecosystem as
> well
> > as from other open source communities.
> >
> > There is a rich ecosystem around Kafka. Many of the projects are quite
> > small--single clients or tools that do a single thing well--and almost
> none
> > of them are top level apache projects. I don't think trying to force each
> > of these to turn into independent Apache projects is necessarily the best
> > thing for the ecosystem.
> >
> > My observation of how this can go wrong is really what I think has
> happened
> > to the Hadoop ecosystem. There you see quite a zoo of projects which all
> > drift apart and don't quite work together well. Coordinating even simple
> > changes and standardization across these is exceptionally difficult. The
> > result is a bit of a mess for users--the pieces just don't really come
> > together very well. This makes sense for independent infrastructure
> systems
> > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > little tool or helper library has lead to a desirable state. I think the
> > mode of operating where the Hadoop vendors spawn off a few new Apache
> > projects for each new product initiative, especially since often that
> > project is only valued by that vendor (and the other vendor has a
> different
> > competing Apache project) doesn't necessarily do a better job at
> producing
> > high quality communities or high quality software.
> >
> > These tools/connects/clients/proxies and other integration pieces can
> take
> > many forms, but my take of what makes one of these things good is that it
> > remains simple, does its one thing well, and cleaves as closely as
> possible
> > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > monitoring, configuring, etc. For the tools we've contributed we've tried
> > really hard to make them consistent with Kafka as well as with each other
> > in how testing, configuration, monitoring, etc works.
> >
> > I think what Apache does superbly well is create a community for
> managing a
> > large infrastructure layer like Kafka in a vendor independent way. What I
> > think is less successful is attempting to form full and independent
> apache
> > communities around very simple single purpose tools, especially if you
> hope
> > for these to come together into a cohesive toolset across multiple such
> > tools. Much of what Apache does--create a collective decision making
> > process for resolving disagreement, help to trademark and protect the
> marks
> > of the project, etc just isn't that relevant for simple single-purpose
> > tools.
> >
> > So my take is there are a couple of options:
> >
> > 1. We can try to put all the small tools into the Apache Project. I
> > think this is not the right approach as there is simply too many of
> > them,
> > many in different languages, serving different protocols, integrating
> > with
> > particular systems, and a single community can't effectively maintain
> > them
> > all. Doing this would significantly slow the progress of the Kafka
> > project.
> > As a protocol for messaging, I don't really see a case for including
> > REST
> > but not MQTT or AMQP which are technically much better suited to
> > messaging
> > and both are widely used for that.
> > 2. We can treat ecosystem projects that aren't top level Apache
> projects
> > as invalid and try to recreate them all as Apache projects. Honestly,
> > though, if you go to the Kafka ecosystem page virtually none of the
> most
> > popular add-ons to Kafka are Apache projects. The most successful
> > things in
> > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> other
> > clients, as well as the existing REST layer have succeeded at
> developing
> > communities that actively contribute and use these pieces and I don't
> > know
> > that that is a bad thing unless that community proves to be
> uninclusive,
> > unresponsive, or goes in a bad technical direction--and those are
> > failure
> > modes that all open source efforts face.
> > 3. We can do what I think makes the most sense and try to work with
> the
> > projects that exist in the ecosystem and if the project doesn't have a
> > responsive community or wants to go in a different direction fork or
> > recreate that work.
> >
> > Of course any person can choose whatever of these options they want. But
> > from my point of view, option (3) has been the path of the community so
> far
> > and I think it has been quite successful.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Neha,
> > > "But I haven't seen any community emails or patches being submitted by
> > you
> > > guys, so I'm wondering why you are concerned about whether the
> community
> > is
> > > open to accepting patches or not."
> > >
> > > I think you are talking about contributing patches to this repository
> > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > the guidelines/governance model is not clear on the project and I guess
> > its
> > > driven by opening a github issue request. Its the repository owned by
> > > confluent and as much I appreciate that the features we mentioned are
> in
> > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > gurantee what we want to add in the furture will be in your roadmap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot as
> > other
> > > users can participate in the discussions. We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project time
> > > within
> > > > my internal projects to developers contributing to another commercial
> > > > companies project even so technically open source, as then there is
> > that
> > > > commercial companies interest will always prevail and essentially can
> > > > always have a final vote where disagreement. Im sure they never
> intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint on
> > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages
> in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
> more
> > > > surface area for every client. But something like REST is universal
> and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > ________________________________________
> > > > From: Andrew Schofield <an...@outlook.com>
> > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > There's a massive difference between the governance of Kafka and the
> > > > governance of the REST proxy.
> > > >
> > > > In Kafka, there is a broad community of people contributing their
> > > opinions
> > > > about future enhancements in the form of KIPs. There's some really
> deep
> > > > consideration that goes into some of the trickier KIPs. There are
> > people
> > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > reputations to become committers. I get the impression that the
> roadmap
> > > of
> > > > Kafka is not really community-owned (what's the big feature for Kafka
> > > 0.11,
> > > > for example), but the conveyor belt of smaller features in the form
> of
> > > KIPs
> > > > works nicely. It's a good example of open-source working well.
> > > >
> > > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > > roadmap is less clear. There's not really a community properly
> engaged
> > in
> > > > the way that there is with Kafka. So, you could say that it's clear
> > that
> > > > fewer people are interested, but I think the whole governance thing
> > is a
> > > > big barrier to engagement. And it's looking like it's getting out of
> > > date.
> > > >
> > > > In technical terms, I can think of two big improvements to the REST
> > > proxy.
> > > > First, it needs to use the new consumer API so that it's possible to
> > > secure
> > > > connections between the REST proxy and Kafka. I don't care too much
> > which
> > > > method calls it uses actually uses to consume messages, but I do
> care
> > > that
> > > > I cannot secure connections because of network security rules.
> Second,
> > > > there's an affinity between a consumer and the instance of the REST
> > proxy
> > > > to which it first connected. Kafka itself avoids this kind of
> affinity
> > > for
> > > > good reason, and in the name of availability the REST proxy should
> too.
> > > > These are natural KIPs.
> > > >
> > > > I think it would be good to have the code for the REST proxy
> > contributed
> > > > to Apache Kafka so that it would be able to be developed in the same
> > way.
> > > >
> > > > Andrew Schofield
> > > >
> > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > Sent: 07 October 2016 22:41:52
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > ASF already gives us a clear framework and governance model for
> > community
> > > > development. This is already understood by the people contributing to
> > > > Apache Kafka project, and they are the same people who want to
> > contribute
> > > > to the REST server capability as well. Everyone is in agreement on
> the
> > > > need for collaborating on this effort. So why not contribute the code
> > to
> > > > Apache Kafka. This will help avoid duplication of effort and forks
> that
> > > > may crop up, hugely benefitting the user community. This will also
> > avoid
> > > > having to define a process similar to ASF on a GitHub project and
> > instead
> > > > there is a single community with clear understanding community
> process
> > as
> > > > defined in ASF.
> > > >
> > > > As others have said, this is an important capability for Apache
> Kafka.
> > It
> > > > is worth maintaining this as a part of the project.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > >
> > > > >I personally think it would be quite wasteful to re-implement the
> REST
> > > > >gateway just because that an actively-maintained piece of
> > > Apache-licensed
> > > > >software is not governed directly by the Apache Kafka community...
> > While
> > > > >kafka-rest repo is owned by Confluent, the contributors including
> the
> > > main
> > > > >one are also part of the Apache Kafka community, so there is a
> chance
> > > to
> > > > >work this out.
> > > > >
> > > > >However, there are two valid concerns here that could be addressed,
> > > around
> > > > >community and accessibility:
> > > > >>> What we are worried about is a project
> > > > >>> that's not maintained in a community. So the process of accepting
> > > > >>>patches
> > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > >>>community.
> > > > >>> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >and
> > > > >>> hence there is no security support either.
> > > > >
> > > > >This might be easy to fix. Maybe Confluent / kafka-rest community
> can
> > > > >clarify that - what is their contribution policy, dev style, roadmap
> > > etc.
> > > > >If they want, they can make an effort to encourage participation
> from
> > > > >people outside Confluent (easily accept contributions, invite
> external
> > > > >commiters or have open dev process similar to Apache Kafka etc), as
> > > there
> > > > >is definitely seems to be some interest on the list. That might
> clear
> > > the
> > > > >community concern and help kafka-rest project (but that is a
> > calculation
> > > > >Confluent will have to make).
> > > > >
> > > > >The other, independent, concern is that REST is something that is
> > > expected
> > > > >to be available out of the box with Kafka. I personally don't feel
> > > > >strongly
> > > > >about it (better use proper, efficient APIs from day one), though it
> > is
> > > > >definitely way smaller than adding a stream processing engine to the
> > > > >project :)
> > > > >Again,the kafka-rest "community" could take steps to make it even
> > easier
> > > > >to
> > > > >install, configure and run kafka-rest for new users on vanilla
> Apache
> > > > >Kafka
> > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > >contributions to that end), but that is up to them.
> > > > >Finally, if after the above steps were taken there would still a
> > strong
> > > > >desire to include a great rest gateway with Apache Kafka, I assume
> the
> > > > >community could hypothetically fork the existing kafka-rest into an
> > > Apache
> > > > >Kafka subproject and maintain it "within Apache" instead of
> > implementing
> > > > >it
> > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> > > > >happen
> > > > >without Confluent blessing, and I think that is likely much less
> > optimal
> > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > having
> > > a
> > > > >separate external community around kafka-rest.
> > > > >
> > > > >
> > > > >Just my two cents,
> > > > >
> > > > >
> > > > >Ofir Manor
> > > > >
> > > > >Co-Founder & CTO | Equalum
> > > > >
> > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> <+972%2054-780-1286> |
> > Email:
> > > > ofir.manor@equalum.io
> > > > >
> > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > > > >wrote:
> > > > >
> > > > >> Neha & Jay,
> > > > >> We did look at the open source alternatives. Our
> > > > >>concern
> > > > >> is what's the patch acceptance and adding features/ bug-fixes to
> the
> > > > >> existing project under a Github (although it's licensed under
> Apache
> > > > >>2.0).
> > > > >> It would be great if that project made available under Apache and
> > > > >>driven by
> > > > >> the community. Adding to the above, not all Kafka users are
> > > interested
> > > > >>in
> > > > >> using the Java client API, they would like to have simple REST API
> > > where
> > > > >> they can code against using any language. I do believe this adds
> > value
> > > > >>to
> > > > >> Apache Kafka in itself.
> > > > >>
> > > > >> "For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> reinventing the wheel. What I'm looking for is a detailed
> comparison
> > > of
> > > > >>the
> > > > >> gaps and why those can't be improved in the REST proxy that
> already
> > > > >>exists
> > > > >> and is actively maintained."
> > > > >>
> > > > >> We are not looking at this as NIH. What we are worried about is a
> > > > >>project
> > > > >> that's not maintained in a community. So the process of accepting
> > > > >>patches
> > > > >> and priorities is not clear, and it's not developed in Apache
> > > community.
> > > > >> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >>and
> > > > >> hence there is no security support either.
> > > > >> We don't know the timeline when that's made available. We would
> like
> > > to
> > > > >>add
> > > > >> admin functionality into the REST API. So the Roadmap of that
> > project
> > > is
> > > > >> not driven by Apache.
> > > > >>
> > > > >>
> > > > >> "This doesn't materially have an impact on expanding the usability
> > of
> > > > >> Kafka. In my experience, REST proxy + Java clients only cover
> > ~50%
> > > of
> > > > >> all
> > > > >> Kafka users, and maybe 10% of those are the ones who will use
> the
> > > > >>REST
> > > > >> proxy. The remaining 50% are non-java client users (C, python,
> > go,
> > > > >>node
> > > > >> etc)."
> > > > >>
> > > > >> REST API is most often asked feature in my interactions with Kafka
> > > > >>users.
> > > > >> In an organization, There will be independent teams who will write
> > > their
> > > > >> Kafka clients using different language libraries available today,
> > and
> > > > >> there is no way to standardize this. Instead of supporting several
> > > > >> different client libraries users will be interested in using a
> REST
> > > API
> > > > >> server. The need for a REST API will only increase as more and
> more
> > > > >>users
> > > > >> start using Kafka.
> > > > >>
> > > > >> "More surface area means more work to keep things consistent.
> > Failure
> > > > >> to do that has, in fact, hurt the user experience."
> > > > >> Having myriad Kafka client GitHub projects that support different
> > > > >>languages
> > > > >> hurts the user experience and pushes burden to maintain these
> > > libraries.
> > > > >> REST API is a simple code base that uses existing java client
> > > libraries
> > > > >>to
> > > > >> make life easier for the users.
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > > > wrote:
> > > > >>
> > > > >> > Manikumar,
> > > > >> >
> > > > >> > Thanks for sharing the proposal. I think there are 2 parts to
> this
> > > > >> > discussion -
> > > > >> >
> > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > >>feature-complete,
> > > > >> > open-source and actively maintained REST proxy in the community?
> > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile
> and
> > > > >> maintain
> > > > >> > the high-quality experience that Kafka users have today?
> > > > >> >
> > > > >> > For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > comparison
> > > > >>of
> > > > >> the
> > > > >> > gaps and why those can't be improved in the REST proxy that
> > already
> > > > >> exists
> > > > >> > and is actively maintained. For example, we depend on zkClient
> and
> > > > >>have
> > > > >> > found as well as fixed several bugs by working closely with the
> > > people
> > > > >> who
> > > > >> > maintain zkClient. This should be possible for REST proxy as
> well,
> > > > >>right?
> > > > >> >
> > > > >> > For 2, I'd like us to review our history of expanding the
> surface
> > > > >>area to
> > > > >> > add more clients in the past. Here is a summary -
> > > > >> >
> > > > >> > - This doesn't materially have an impact on expanding the
> > > > >>usability of
> > > > >> > Kafka. In my experience, REST proxy + Java clients only cover
> > > ~50%
> > > > >>of
> > > > >> > all
> > > > >> > Kafka users, and maybe 10% of those are the ones who will use
> > the
> > > > >>REST
> > > > >> > proxy. The remaining 50% are non-java client users (C,
> python,
> > > go,
> > > > >> node
> > > > >> > etc).
> > > > >> > - People are a lot more excited about promising to contribute
> > > while
> > > > >> > adding the surface area but not on an ongoing basis down the
> > > line.
> > > > >> > - More surface area means more work to keep things
> consistent.
> > > > >>Failure
> > > > >> > to do that has, in fact, hurt the user experience.
> > > > >> > - More surface area hurts agility. We want to do a few things
> > > > >>really
> > > > >> > well as well as be agile to be able to build on our core
> > > > >>competency.
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Jay,
> > > > >> > >
> > > > >> > > Thanks for your reply.
> > > > >> > >
> > > > >> > > I agree that we can not add all the clients/tools available in
> > > > >> ecosystem
> > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > different
> > > > >>from
> > > > >> > > other clients/tools. Since any language that can work with
> HTTP
> > > can
> > > > >> > > easily integrate with this interface, Having an "official"
> REST
> > > > >> > > interface helps user community. This also helps us to
> integrate
> > > well
> > > > >> > > with external management and provisioning tools. Apache Kafka
> > > > >>release
> > > > >> > > with Java clients + REST interface is sufficient for most of
> the
> > > > >>user
> > > > >> > > deployments/requirements. This helps users to deal with less
> > > number
> > > > >> > > of distributions/builds.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Manikumar
> > > > >> > >
> > > > >> > >
> > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> > > wrote:
> > > > >> > >
> > > > >> > > > Hey guys,
> > > > >> > > >
> > > > >> > > > There's already a REST interface maintained as a separate
> > > > >> project--it's
> > > > >> > > > open source and apache licensed and actively maintained (
> > > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong
> > with
> > > >
> > > >
> > > >
> > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > github.com
> > > > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
> > It
> > > > makes it easy to produce and consume messages, view the state of the
> > > > cluster, and ...
> > > >
> > > > >> that?
> > > > >> > > You
> > > > >> > > > mentioned that there was some compatibility concern, but
> > > > >> compatibility
> > > > >> > > has
> > > > >> > > > to do with the consumer protocol guarantees not the repo the
> > > code
> > > > >>is
> > > > >> > in,
> > > > >> > > > right? Not sure that concern makes sense.
> > > > >> > > >
> > > > >> > > > We could argue for adding pretty much anything and
> everything
> > in
> > > > >>the
> > > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> > make
> > > > >>the
> > > > >> > > project
> > > > >> > > > more agile.
> > > > >> > > >
> > > > >> > > > -Jay
> > > > >> > > >
> > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > >> manikumar.reddy@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Kafka Devs,
> > > > >> > > > >
> > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > Repository.
> > > > >> > > > >
> > > > >> > > > > There are already open-source alternatives are available.
> > But
> > > > >>we
> > > > >> > would
> > > > >> > > > > like to add REST server that
> > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> Infra
> > > > >>tools
> > > > >> > comes
> > > > >> > > > up
> > > > >> > > > > with Rest Interface.
> > > > >> > > > > It is useful to have inbuilt Rest API support for Produce,
> > > > >>Consume
> > > > >> > > > messages
> > > > >> > > > > and admin interface for
> > > > >> > > > > integrating with external management and provisioning
> > > tools.This
> > > > >> will
> > > > >> > > > also
> > > > >> > > > > allow the maintenance of
> > > > >> > > > > REST server and adding new features makes it easy because
> > > apache
> > > > >> > > > community.
> > > > >> > > > >
> > > > >> > > > > The KIP wiki is the following:
> > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > >> > > > >
> > > > >> > > > > Your comments and feedback are welcome.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Manikumar
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > --
> > > > >> > Thanks,
> > > > >> > Neha
> > > > >> >
> > > > >>
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Edoardo Comar <EC...@uk.ibm.com>.
Harsha Ch <ha...@gmail.com> wrote on 20/10/2016 22:26:53:
> The Streams and Connect projects that were pushed into Kafka
could
> have been left in their own Github projects based on your arguments.
What
> about the REST API is so different that such that it should stay out of
the
> Kafka project? From my experience, more users are asking for the REST
API.
Thanks Harsha
for keeping pushing this issue.
Your experience about users matches ours !
cheers
Edo
--------------------------------------------------
Edoardo Comar
IBM MessageHub
ecomar@uk.ibm.com
IBM UK Ltd, Hursley Park, SO21 2JN
IBM United Kingdom Limited Registered in England and Wales with number
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
3AU
Harsha Ch <ha...@gmail.com> wrote on 20/10/2016 22:26:53:
> From: Harsha Ch <ha...@gmail.com>
> To: dev@kafka.apache.org
> Date: 20/10/2016 22:32
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> Jay,
> REST API is something every user is in need of. If the argument is
to
> clone and write your API, this will do a disservice to the users as
they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
>
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be
done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
>
> Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and
creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this
REST
> server to Kafka will affect the agility of the project doesn't mak
sense.
>
> It looks like your argument is either we develop all these small tools
or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka
could
> have been left in their own Github projects based on your arguments.
What
> about the REST API is so different that such that it should stay out of
the
> Kafka project? From my experience, more users are asking for the REST
API.
>
> Thanks,
> Harsha
>
>
>
>
>
> On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
>
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not
really
> > make sense to me. If in the future we disagree, that is the beauty of
open
> > source, you can always fork off a copy of the code and start an
independent
> > project either in Apache or elsewhere. Pre-emptively re-creating
another
> > REST layer when it seems like we all quite agree on what needs to be
done
> > and we have an existing code base for HTTP/kafka access that is
heavily
> > used in production seems quite silly.
> >
> > Let me give some background on how I at least think about these
things.
> > I've participated in open source projects out of LinkedIn via github
as
> > well as via the ASF. I don't think there is a "right" answer to how to
do
> > these but rather some tradeoffs. We thought about this quite a lot in
the
> > context of Kafka based on the experience with the Hadoop ecosystem as
well
> > as from other open source communities.
> >
> > There is a rich ecosystem around Kafka. Many of the projects are quite
> > small--single clients or tools that do a single thing well--and almost
none
> > of them are top level apache projects. I don't think trying to force
each
> > of these to turn into independent Apache projects is necessarily the
best
> > thing for the ecosystem.
> >
> > My observation of how this can go wrong is really what I think has
happened
> > to the Hadoop ecosystem. There you see quite a zoo of projects which
all
> > drift apart and don't quite work together well. Coordinating even
simple
> > changes and standardization across these is exceptionally difficult.
The
> > result is a bit of a mess for users--the pieces just don't really come
> > together very well. This makes sense for independent infrastructure
systems
> > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > little tool or helper library has lead to a desirable state. I think
the
> > mode of operating where the Hadoop vendors spawn off a few new Apache
> > projects for each new product initiative, especially since often that
> > project is only valued by that vendor (and the other vendor has a
different
> > competing Apache project) doesn't necessarily do a better job at
producing
> > high quality communities or high quality software.
> >
> > These tools/connects/clients/proxies and other integration pieces can
take
> > many forms, but my take of what makes one of these things good is that
it
> > remains simple, does its one thing well, and cleaves as closely as
possible
> > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > monitoring, configuring, etc. For the tools we've contributed we've
tried
> > really hard to make them consistent with Kafka as well as with each
other
> > in how testing, configuration, monitoring, etc works.
> >
> > I think what Apache does superbly well is create a community for
managing a
> > large infrastructure layer like Kafka in a vendor independent way.
What I
> > think is less successful is attempting to form full and independent
apache
> > communities around very simple single purpose tools, especially if you
hope
> > for these to come together into a cohesive toolset across multiple
such
> > tools. Much of what Apache does--create a collective decision making
> > process for resolving disagreement, help to trademark and protect the
marks
> > of the project, etc just isn't that relevant for simple single-purpose
> > tools.
> >
> > So my take is there are a couple of options:
> >
> > 1. We can try to put all the small tools into the Apache Project. I
> > think this is not the right approach as there is simply too many of
> > them,
> > many in different languages, serving different protocols,
integrating
> > with
> > particular systems, and a single community can't effectively
maintain
> > them
> > all. Doing this would significantly slow the progress of the Kafka
> > project.
> > As a protocol for messaging, I don't really see a case for
including
> > REST
> > but not MQTT or AMQP which are technically much better suited to
> > messaging
> > and both are widely used for that.
> > 2. We can treat ecosystem projects that aren't top level Apache
projects
> > as invalid and try to recreate them all as Apache projects.
Honestly,
> > though, if you go to the Kafka ecosystem page virtually none of the
most
> > popular add-ons to Kafka are Apache projects. The most successful
> > things in
> > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
other
> > clients, as well as the existing REST layer have succeeded at
developing
> > communities that actively contribute and use these pieces and I
don't
> > know
> > that that is a bad thing unless that community proves to be
uninclusive,
> > unresponsive, or goes in a bad technical direction--and those are
> > failure
> > modes that all open source efforts face.
> > 3. We can do what I think makes the most sense and try to work with
the
> > projects that exist in the ecosystem and if the project doesn't
have a
> > responsive community or wants to go in a different direction fork
or
> > recreate that work.
> >
> > Of course any person can choose whatever of these options they want.
But
> > from my point of view, option (3) has been the path of the community
so far
> > and I think it has been quite successful.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani
<ka...@harsha.io>
> > wrote:
> >
> > > Neha,
> > > "But I haven't seen any community emails or patches being submitted
by
> > you
> > > guys, so I'm wondering why you are concerned about whether the
community
> > is
> > > open to accepting patches or not."
> > >
> > > I think you are talking about contributing patches to this
repository
> > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > the guidelines/governance model is not clear on the project and I
guess
> > its
> > > driven by opening a github issue request. Its the repository owned
by
> > > confluent and as much I appreciate that the features we mentioned
are in
> > > the roadmap and welcoming us to contribute to the project. It
doesn't
> > > gurantee what we want to add in the furture will be in your roadmap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot
as
> > other
> > > users can participate in the discussions. We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce
<Mi...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project
time
> > > within
> > > > my internal projects to developers contributing to another
commercial
> > > > companies project even so technically open source, as then there
is
> > that
> > > > commercial companies interest will always prevail and essentially
can
> > > > always have a final vote where disagreement. Im sure they never
intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint
on
> > > > GitHub, BitBucket etc. Each one has their benefits and
disadvantages in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the
best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
more
> > > > surface area for every client. But something like REST is
universal and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > ________________________________________
> > > > From: Andrew Schofield <an...@outlook.com>
> > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > There's a massive difference between the governance of Kafka and
the
> > > > governance of the REST proxy.
> > > >
> > > > In Kafka, there is a broad community of people contributing their
> > > opinions
> > > > about future enhancements in the form of KIPs. There's some really
deep
> > > > consideration that goes into some of the trickier KIPs. There are
> > people
> > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > reputations to become committers. I get the impression that the
roadmap
> > > of
> > > > Kafka is not really community-owned (what's the big feature for
Kafka
> > > 0.11,
> > > > for example), but the conveyor belt of smaller features in the
form of
> > > KIPs
> > > > works nicely. It's a good example of open-source working well.
> > > >
> > > > The equivalent for the REST proxy is basically issues on GitHub.
The
> > > > roadmap is less clear. There's not really a community properly
engaged
> > in
> > > > the way that there is with Kafka. So, you could say that it's
clear
> > that
> > > > fewer people are interested, but I think the whole governance
thing
> > is a
> > > > big barrier to engagement. And it's looking like it's getting out
of
> > > date.
> > > >
> > > > In technical terms, I can think of two big improvements to the
REST
> > > proxy.
> > > > First, it needs to use the new consumer API so that it's possible
to
> > > secure
> > > > connections between the REST proxy and Kafka. I don't care too
much
> > which
> > > > method calls it uses actually uses to consume messages, but I do
care
> > > that
> > > > I cannot secure connections because of network security rules.
Second,
> > > > there's an affinity between a consumer and the instance of the
REST
> > proxy
> > > > to which it first connected. Kafka itself avoids this kind of
affinity
> > > for
> > > > good reason, and in the name of availability the REST proxy should
too.
> > > > These are natural KIPs.
> > > >
> > > > I think it would be good to have the code for the REST proxy
> > contributed
> > > > to Apache Kafka so that it would be able to be developed in the
same
> > way.
> > > >
> > > > Andrew Schofield
> > > >
> > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > Sent: 07 October 2016 22:41:52
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > ASF already gives us a clear framework and governance model for
> > community
> > > > development. This is already understood by the people contributing
to
> > > > Apache Kafka project, and they are the same people who want to
> > contribute
> > > > to the REST server capability as well. Everyone is in agreement on
the
> > > > need for collaborating on this effort. So why not contribute the
code
> > to
> > > > Apache Kafka. This will help avoid duplication of effort and forks
that
> > > > may crop up, hugely benefitting the user community. This will also
> > avoid
> > > > having to define a process similar to ASF on a GitHub project and
> > instead
> > > > there is a single community with clear understanding community
process
> > as
> > > > defined in ASF.
> > > >
> > > > As others have said, this is an important capability for Apache
Kafka.
> > It
> > > > is worth maintaining this as a part of the project.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > >
> > > > >I personally think it would be quite wasteful to re-implement the
REST
> > > > >gateway just because that an actively-maintained piece of
> > > Apache-licensed
> > > > >software is not governed directly by the Apache Kafka
community...
> > While
> > > > >kafka-rest repo is owned by Confluent, the contributors including
the
> > > main
> > > > >one are also part of the Apache Kafka community, so there is a
chance
> > > to
> > > > >work this out.
> > > > >
> > > > >However, there are two valid concerns here that could be
addressed,
> > > around
> > > > >community and accessibility:
> > > > >>> What we are worried about is a project
> > > > >>> that's not maintained in a community. So the process of
accepting
> > > > >>>patches
> > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > >>>community.
> > > > >>> Not only that, existing REST API project doesn't support new
client
> > > API
> > > > >and
> > > > >>> hence there is no security support either.
> > > > >
> > > > >This might be easy to fix. Maybe Confluent / kafka-rest community
can
> > > > >clarify that - what is their contribution policy, dev style,
roadmap
> > > etc.
> > > > >If they want, they can make an effort to encourage participation
from
> > > > >people outside Confluent (easily accept contributions, invite
external
> > > > >commiters or have open dev process similar to Apache Kafka etc),
as
> > > there
> > > > >is definitely seems to be some interest on the list. That might
clear
> > > the
> > > > >community concern and help kafka-rest project (but that is a
> > calculation
> > > > >Confluent will have to make).
> > > > >
> > > > >The other, independent, concern is that REST is something that is
> > > expected
> > > > >to be available out of the box with Kafka. I personally don't
feel
> > > > >strongly
> > > > >about it (better use proper, efficient APIs from day one), though
it
> > is
> > > > >definitely way smaller than adding a stream processing engine to
the
> > > > >project :)
> > > > >Again,the kafka-rest "community" could take steps to make it even
> > easier
> > > > >to
> > > > >install, configure and run kafka-rest for new users on vanilla
Apache
> > > > >Kafka
> > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > >contributions to that end), but that is up to them.
> > > > >Finally, if after the above steps were taken there would still a
> > strong
> > > > >desire to include a great rest gateway with Apache Kafka, I
assume the
> > > > >community could hypothetically fork the existing kafka-rest into
an
> > > Apache
> > > > >Kafka subproject and maintain it "within Apache" instead of
> > implementing
> > > > >it
> > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine
it
> > > > >happen
> > > > >without Confluent blessing, and I think that is likely much less
> > optimal
> > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > having
> > > a
> > > > >separate external community around kafka-rest.
> > > > >
> > > > >
> > > > >Just my two cents,
> > > > >
> > > > >
> > > > >Ofir Manor
> > > > >
> > > > >Co-Founder & CTO | Equalum
> > > > >
> > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> <+972%2054-780-1286>
|
> > Email:
> > > > ofir.manor@equalum.io
> > > > >
> > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani
<kafka@harsha.io
> > >
> > > > >wrote:
> > > > >
> > > > >> Neha & Jay,
> > > > >> We did look at the open source alternatives.
Our
> > > > >>concern
> > > > >> is what's the patch acceptance and adding features/ bug-fixes
to the
> > > > >> existing project under a Github (although it's licensed under
Apache
> > > > >>2.0).
> > > > >> It would be great if that project made available under Apache
and
> > > > >>driven by
> > > > >> the community. Adding to the above, not all Kafka users are
> > > interested
> > > > >>in
> > > > >> using the Java client API, they would like to have simple REST
API
> > > where
> > > > >> they can code against using any language. I do believe this
adds
> > value
> > > > >>to
> > > > >> Apache Kafka in itself.
> > > > >>
> > > > >> "For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> reinventing the wheel. What I'm looking for is a detailed
comparison
> > > of
> > > > >>the
> > > > >> gaps and why those can't be improved in the REST proxy that
already
> > > > >>exists
> > > > >> and is actively maintained."
> > > > >>
> > > > >> We are not looking at this as NIH. What we are worried about
is a
> > > > >>project
> > > > >> that's not maintained in a community. So the process of
accepting
> > > > >>patches
> > > > >> and priorities is not clear, and it's not developed in Apache
> > > community.
> > > > >> Not only that, existing REST API project doesn't support new
client
> > > API
> > > > >>and
> > > > >> hence there is no security support either.
> > > > >> We don't know the timeline when that's made available. We would
like
> > > to
> > > > >>add
> > > > >> admin functionality into the REST API. So the Roadmap of that
> > project
> > > is
> > > > >> not driven by Apache.
> > > > >>
> > > > >>
> > > > >> "This doesn't materially have an impact on expanding the
usability
> > of
> > > > >> Kafka. In my experience, REST proxy + Java clients only
cover
> > ~50%
> > > of
> > > > >> all
> > > > >> Kafka users, and maybe 10% of those are the ones who will
use the
> > > > >>REST
> > > > >> proxy. The remaining 50% are non-java client users (C,
python,
> > go,
> > > > >>node
> > > > >> etc)."
> > > > >>
> > > > >> REST API is most often asked feature in my interactions with
Kafka
> > > > >>users.
> > > > >> In an organization, There will be independent teams who will
write
> > > their
> > > > >> Kafka clients using different language libraries available
today,
> > and
> > > > >> there is no way to standardize this. Instead of supporting
several
> > > > >> different client libraries users will be interested in using a
REST
> > > API
> > > > >> server. The need for a REST API will only increase as more and
more
> > > > >>users
> > > > >> start using Kafka.
> > > > >>
> > > > >> "More surface area means more work to keep things consistent.
> > Failure
> > > > >> to do that has, in fact, hurt the user experience."
> > > > >> Having myriad Kafka client GitHub projects that support
different
> > > > >>languages
> > > > >> hurts the user experience and pushes burden to maintain these
> > > libraries.
> > > > >> REST API is a simple code base that uses existing java client
> > > libraries
> > > > >>to
> > > > >> make life easier for the users.
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede
<ne...@confluent.io>
> > > > wrote:
> > > > >>
> > > > >> > Manikumar,
> > > > >> >
> > > > >> > Thanks for sharing the proposal. I think there are 2 parts to
this
> > > > >> > discussion -
> > > > >> >
> > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > >>feature-complete,
> > > > >> > open-source and actively maintained REST proxy in the
community?
> > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more
agile and
> > > > >> maintain
> > > > >> > the high-quality experience that Kafka users have today?
> > > > >> >
> > > > >> > For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > comparison
> > > > >>of
> > > > >> the
> > > > >> > gaps and why those can't be improved in the REST proxy that
> > already
> > > > >> exists
> > > > >> > and is actively maintained. For example, we depend on
zkClient and
> > > > >>have
> > > > >> > found as well as fixed several bugs by working closely with
the
> > > people
> > > > >> who
> > > > >> > maintain zkClient. This should be possible for REST proxy as
well,
> > > > >>right?
> > > > >> >
> > > > >> > For 2, I'd like us to review our history of expanding the
surface
> > > > >>area to
> > > > >> > add more clients in the past. Here is a summary -
> > > > >> >
> > > > >> > - This doesn't materially have an impact on expanding the
> > > > >>usability of
> > > > >> > Kafka. In my experience, REST proxy + Java clients only
cover
> > > ~50%
> > > > >>of
> > > > >> > all
> > > > >> > Kafka users, and maybe 10% of those are the ones who will
use
> > the
> > > > >>REST
> > > > >> > proxy. The remaining 50% are non-java client users (C,
python,
> > > go,
> > > > >> node
> > > > >> > etc).
> > > > >> > - People are a lot more excited about promising to
contribute
> > > while
> > > > >> > adding the surface area but not on an ongoing basis down
the
> > > line.
> > > > >> > - More surface area means more work to keep things
consistent.
> > > > >>Failure
> > > > >> > to do that has, in fact, hurt the user experience.
> > > > >> > - More surface area hurts agility. We want to do a few
things
> > > > >>really
> > > > >> > well as well as be agile to be able to build on our core
> > > > >>competency.
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Jay,
> > > > >> > >
> > > > >> > > Thanks for your reply.
> > > > >> > >
> > > > >> > > I agree that we can not add all the clients/tools available
in
> > > > >> ecosystem
> > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > different
> > > > >>from
> > > > >> > > other clients/tools. Since any language that can work with
HTTP
> > > can
> > > > >> > > easily integrate with this interface, Having an "official"
REST
> > > > >> > > interface helps user community. This also helps us to
integrate
> > > well
> > > > >> > > with external management and provisioning tools. Apache
Kafka
> > > > >>release
> > > > >> > > with Java clients + REST interface is sufficient for most
of the
> > > > >>user
> > > > >> > > deployments/requirements. This helps users to deal with
less
> > > number
> > > > >> > > of distributions/builds.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Manikumar
> > > > >> > >
> > > > >> > >
> > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps
<ja...@confluent.io>
> > > wrote:
> > > > >> > >
> > > > >> > > > Hey guys,
> > > > >> > > >
> > > > >> > > > There's already a REST interface maintained as a separate
> > > > >> project--it's
> > > > >> > > > open source and apache licensed and actively maintained (
> > > > >> > > > https://github.com/confluentinc/kafka-rest). What is
wrong
> > with
> > > >
> > > >
> > > >
> > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > github.com
> > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
cluster.
> > It
> > > > makes it easy to produce and consume messages, view the state of
the
> > > > cluster, and ...
> > > >
> > > > >> that?
> > > > >> > > You
> > > > >> > > > mentioned that there was some compatibility concern, but
> > > > >> compatibility
> > > > >> > > has
> > > > >> > > > to do with the consumer protocol guarantees not the repo
the
> > > code
> > > > >>is
> > > > >> > in,
> > > > >> > > > right? Not sure that concern makes sense.
> > > > >> > > >
> > > > >> > > > We could argue for adding pretty much anything and
everything
> > in
> > > > >>the
> > > > >> > > > ecosystem page in Kafka itself but I'm not sure that
would
> > make
> > > > >>the
> > > > >> > > project
> > > > >> > > > more agile.
> > > > >> > > >
> > > > >> > > > -Jay
> > > > >> > > >
> > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > >> manikumar.reddy@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Kafka Devs,
> > > > >> > > > >
> > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > Repository.
> > > > >> > > > >
> > > > >> > > > > There are already open-source alternatives are
available.
> > But
> > > > >>we
> > > > >> > would
> > > > >> > > > > like to add REST server that
> > > > >> > > > > many users ask for under Apache Kafka repo. Many data
Infra
> > > > >>tools
> > > > >> > comes
> > > > >> > > > up
> > > > >> > > > > with Rest Interface.
> > > > >> > > > > It is useful to have inbuilt Rest API support for
Produce,
> > > > >>Consume
> > > > >> > > > messages
> > > > >> > > > > and admin interface for
> > > > >> > > > > integrating with external management and provisioning
> > > tools.This
> > > > >> will
> > > > >> > > > also
> > > > >> > > > > allow the maintenance of
> > > > >> > > > > REST server and adding new features makes it easy
because
> > > apache
> > > > >> > > > community.
> > > > >> > > > >
> > > > >> > > > > The KIP wiki is the following:
> > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > >> > > > >
> > > > >> > > > > Your comments and feedback are welcome.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Manikumar
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > --
> > > > >> > Thanks,
> > > > >> > Neha
> > > > >> >
> > > > >>
> > > > The information contained in this email is strictly confidential
and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you
are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose
to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete
the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not
relate to
> > > the
> > > > official business of this company shall be understood as neither
given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
company
> > > > registered in England and Wales, company number 04008957) and IG
Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
and
> > IG
> > > > Index Limited (register number 114059) are authorised and
regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Nacho Solis <ns...@linkedin.com.INVALID>.
To be perfectly clear, I'm actually not in favor of adding a REST module to
Kafka, but that's because I think other things should be separated as well.
Again, I'm new and don't know the history, and I admit it's purely personal
preference, but I'm on the Tea Party side of things... the microkernel
sub-culture. In other words, I would rather have things divided into
components. If it was up to me I would split the broker from the client
libraries from the protocol definition. In that same vein I would split
Connect, Streams, Mirrormaker, etc.
I'm not sure what you mean by "data integration". Is that something that
you can't do from an external module? Do we have a high overhead to use the
Kafka facilities from outside our own source code? If so, we probably need
to rethink our APIs and data structures; or they way we have modularized
our code (callbacks, interceptors, etc).
I'm assuming we're not trying to be like the closed source companies that
give their own applications access to a more efficient API to have an
advantage over the competition, right?
So, if a streaming platform is what we want, then maybe what we need is a
set of modules that deliver that platform vision. A good example of this is
the connector set that Confluent provides. These are not in the core
source, they create an offering. The same could be true for KStreams,
Connect and REST.
Do we have any code that calls Connect and KStreams from inside core? Or is
the code able to live on it's own? (Same for Mirror maker).
Nacho
On Fri, Oct 21, 2016 at 2:31 PM, Sriram Subramanian <ra...@confluent.io>
wrote:
> FWIW, Apache Kafka has evolved a lot from where it started. It did start as
> a messaging system. Over time we realized that that the vision for Kafka is
> to build a streaming platform and not just a messaging system. You can take
> a look at the site for more description about what comprises the streaming
> platform http://kafka.apache.org/ and http://kafka.apache.org/intro.
>
> Can the streaming platform exist without Connect? - No. Data integration is
> fundamental to building an end to end platform
>
> Can the streaming platform exist without stream processing? - No.
> Processing stream data again is a core part of streaming platform.
>
> Can the streaming platform exist without clients? - We at least need one
> client library to complete the platform. Our Java clients help us to
> complete the platform story. The rest of the clients are built and
> maintained outside the project.
>
> Can the platform exist without the rest proxy? - Yes. The proxy does not
> complete the platform vision in anyway. It is just a good to have tool that
> might be required by quite a few users and there is an active project that
> works on this - https://github.com/confluentinc/kafka-rest
>
>
>
>
> On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis <nsolis@linkedin.com.invalid
> >
> wrote:
>
> > Are you saying Kafka REST is subjective but Kafka Streams and Kafka
> Connect
> > are not subjective?
> >
> > > "there are likely places that can live without a rest proxy"
> >
> > There are also places that can live without Kafka Streams and Kafka
> > Connect.
> >
> > Nacho
> >
> > On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > At the high level, I think ideally it makes sense to add a component to
> > > Apache Kafka if (1) it's widely needed and (2) it needs tight
> integration
> > > with Kafka core. For Kafka Stream, we do expect stream processing will
> be
> > > used widely in the future. Implementation wise, Kafka Stream only
> > supports
> > > getting data from Kafka and leverages quite a few of the core
> > > functionalities in Kafka core. For example, it uses customized
> rebalance
> > > callback in the consumer and uses the compacted topic heavily. So,
> having
> > > Kafka Stream in the same repo makes it easier for testing when those
> core
> > > functionalities evolve over time. Kafka Connect is in the same
> situation.
> > >
> > > For rest proxy, whether it's widely used or not is going to be a bit
> > > subjective. However, there are likely places that can live without a
> rest
> > > proxy. The rest proxy is just a proxy for the regular clients and
> doesn't
> > > need to be tightly integrated with Kafka core. So, the case for
> including
> > > rest proxy in Apache Kafka is probably not as strong as Kafka Stream
> and
> > > Kafka Connect.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > So from my reading essentially the first question needs to
> answered/and
> > > > voted on is:
> > > >
> > > > Is Apache Kafka Community only about the Core or does the apache
> > > community
> > > > also support some subprojects (and just we need some better way to
> > manage
> > > > this)
> > > >
> > > > If vote for Core only wins, then the following should be removed:
> > > > Kafka Connect
> > > > Kafka Stream
> > > >
> > > > If vote for Core only loses (aka we will support subprojects) then:
> > > > We should look to add Kafka Rest
> > > >
> > > > And we should look to see how we can manage better govern and manage
> > > > submodules.
> > > >
> > > > A good example which id propose here is how some other communities in
> > > > Apache do this.
> > > >
> > > > Each Module has a Module Management Committee(MMC), this is like
> almost
> > > > the PMC but at a per module basis.
> > > >
> > > > This MMC should essentially hold the binding votes for that module.
> > > > The MMC should be made up of a single representative from each
> > > > organisation (so no single organisation can fully veto the community
> > it
> > > > has to a genuine consenus)
> > > > The MMC requires at least 3 members (so there cant be a tied vote on
> 2)
> > > > For a new Module to be added a MMC committee should be sought
> > > > A new Module is only capable of being added if the above requirements
> > can
> > > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
> that
> > > > only actively support modules would be added
> > > >
> > > > The PMC reviews each module every 6months or Year. If MMC is
> inactive,
> > a
> > > > vote/call to find replacements if raised, if none are forthcoming
> > > dropping
> > > > the MMC to less than 3 then the module moves to "the attic" (very
> much
> > > like
> > > > apache attic but a little more aggressively)
> > > >
> > > > This way the PMC does not need to micro manage every module
> > > > We only add modules where some amount of active support and
> maintenance
> > > > and use is provided by the community
> > > > We have an automatic way to retire old or inactive projects.
> > > >
> > > > Thoughts?
> > > > Mike
> > > >
> > > >
> > > > ________________________________________
> > > > From: Harsha Ch <ha...@gmail.com>
> > > > Sent: Thursday, October 20, 2016 10:26 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > Jay,
> > > > REST API is something every user is in need of. If the argument
> > is
> > > to
> > > > clone and write your API, this will do a disservice to the users as
> > they
> > > > now have to choose one vs. others instead of keeping one API that is
> > > > supported in Kafka community.
> > > >
> > > > "Pre-emptively re-creating another
> > > > REST layer when it seems like we all quite agree on what needs to be
> > done
> > > > and we have an existing code base for HTTP/Kafka access that is
> heavily
> > > > used in production seems quite silly."
> > > >
> > > > Exactly our point. Why can't we develop this in Apache Kafka
> > > > community? Instead of us open sourcing another GitHub project and
> > > creating
> > > > a divide in users and another version of API. Let's build this in
> Kafka
> > > > Community and use the governance model that is proven to provide
> vendor
> > > > free user driven consensus features. The argument that is adding this
> > > REST
> > > > server to Kafka will affect the agility of the project doesn't mak
> > sense.
> > > >
> > > > It looks like your argument is either we develop all these small
> tools
> > or
> > > > none at all. We as a community need to look at supporting critical
> > > > tools/API. Instead of dividing this project into individual external
> > > > communities. We should build this as part of Kafka which best serves
> > the
> > > > needs of users.
> > > > The Streams and Connect projects that were pushed into Kafka
> > > could
> > > > have been left in their own Github projects based on your arguments.
> > What
> > > > about the REST API is so different that such that it should stay out
> of
> > > the
> > > > Kafka project? From my experience, more users are asking for the REST
> > > API.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
> > > >
> > > > > I think the questions around governance make sense, I think we
> should
> > > > > really clarify that to make the process more clear so it can be
> fully
> > > > > inclusive.
> > > > >
> > > > > The idea that we should not collaborate on what is there now,
> though,
> > > > > because in the future we might disagree about direction does not
> > really
> > > > > make sense to me. If in the future we disagree, that is the beauty
> of
> > > > open
> > > > > source, you can always fork off a copy of the code and start an
> > > > independent
> > > > > project either in Apache or elsewhere. Pre-emptively re-creating
> > > another
> > > > > REST layer when it seems like we all quite agree on what needs to
> be
> > > done
> > > > > and we have an existing code base for HTTP/kafka access that is
> > heavily
> > > > > used in production seems quite silly.
> > > > >
> > > > > Let me give some background on how I at least think about these
> > things.
> > > > > I've participated in open source projects out of LinkedIn via
> github
> > as
> > > > > well as via the ASF. I don't think there is a "right" answer to how
> > to
> > > do
> > > > > these but rather some tradeoffs. We thought about this quite a lot
> in
> > > the
> > > > > context of Kafka based on the experience with the Hadoop ecosystem
> as
> > > > well
> > > > > as from other open source communities.
> > > > >
> > > > > There is a rich ecosystem around Kafka. Many of the projects are
> > quite
> > > > > small--single clients or tools that do a single thing well--and
> > almost
> > > > none
> > > > > of them are top level apache projects. I don't think trying to
> force
> > > each
> > > > > of these to turn into independent Apache projects is necessarily
> the
> > > best
> > > > > thing for the ecosystem.
> > > > >
> > > > > My observation of how this can go wrong is really what I think has
> > > > happened
> > > > > to the Hadoop ecosystem. There you see quite a zoo of projects
> which
> > > all
> > > > > drift apart and don't quite work together well. Coordinating even
> > > simple
> > > > > changes and standardization across these is exceptionally
> difficult.
> > > The
> > > > > result is a bit of a mess for users--the pieces just don't really
> > come
> > > > > together very well. This makes sense for independent infrastructure
> > > > systems
> > > > > (Kudu vs HDFS) but I'm not at all convinced that doing this for
> every
> > > > > little tool or helper library has lead to a desirable state. I
> think
> > > the
> > > > > mode of operating where the Hadoop vendors spawn off a few new
> Apache
> > > > > projects for each new product initiative, especially since often
> that
> > > > > project is only valued by that vendor (and the other vendor has a
> > > > different
> > > > > competing Apache project) doesn't necessarily do a better job at
> > > > producing
> > > > > high quality communities or high quality software.
> > > > >
> > > > > These tools/connects/clients/proxies and other integration pieces
> can
> > > > take
> > > > > many forms, but my take of what makes one of these things good is
> > that
> > > it
> > > > > remains simple, does its one thing well, and cleaves as closely as
> > > > possible
> > > > > to the conventions for Kafka itself--i.e. doesn't invent new ways
> of
> > > > > monitoring, configuring, etc. For the tools we've contributed we've
> > > tried
> > > > > really hard to make them consistent with Kafka as well as with each
> > > other
> > > > > in how testing, configuration, monitoring, etc works.
> > > > >
> > > > > I think what Apache does superbly well is create a community for
> > > > managing a
> > > > > large infrastructure layer like Kafka in a vendor independent way.
> > > What I
> > > > > think is less successful is attempting to form full and independent
> > > > apache
> > > > > communities around very simple single purpose tools, especially if
> > you
> > > > hope
> > > > > for these to come together into a cohesive toolset across multiple
> > such
> > > > > tools. Much of what Apache does--create a collective decision
> making
> > > > > process for resolving disagreement, help to trademark and protect
> the
> > > > marks
> > > > > of the project, etc just isn't that relevant for simple
> > single-purpose
> > > > > tools.
> > > > >
> > > > > So my take is there are a couple of options:
> > > > >
> > > > > 1. We can try to put all the small tools into the Apache
> Project.
> > I
> > > > > think this is not the right approach as there is simply too many
> > of
> > > > > them,
> > > > > many in different languages, serving different protocols,
> > > integrating
> > > > > with
> > > > > particular systems, and a single community can't effectively
> > > maintain
> > > > > them
> > > > > all. Doing this would significantly slow the progress of the
> Kafka
> > > > > project.
> > > > > As a protocol for messaging, I don't really see a case for
> > including
> > > > > REST
> > > > > but not MQTT or AMQP which are technically much better suited to
> > > > > messaging
> > > > > and both are widely used for that.
> > > > > 2. We can treat ecosystem projects that aren't top level Apache
> > > > projects
> > > > > as invalid and try to recreate them all as Apache projects.
> > > Honestly,
> > > > > though, if you go to the Kafka ecosystem page virtually none of
> > the
> > > > most
> > > > > popular add-ons to Kafka are Apache projects. The most
> successful
> > > > > things in
> > > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number
> of
> > > > other
> > > > > clients, as well as the existing REST layer have succeeded at
> > > > developing
> > > > > communities that actively contribute and use these pieces and I
> > > don't
> > > > > know
> > > > > that that is a bad thing unless that community proves to be
> > > > uninclusive,
> > > > > unresponsive, or goes in a bad technical direction--and those
> are
> > > > > failure
> > > > > modes that all open source efforts face.
> > > > > 3. We can do what I think makes the most sense and try to work
> > with
> > > > the
> > > > > projects that exist in the ecosystem and if the project doesn't
> > > have a
> > > > > responsive community or wants to go in a different direction
> fork
> > or
> > > > > recreate that work.
> > > > >
> > > > > Of course any person can choose whatever of these options they
> want.
> > > But
> > > > > from my point of view, option (3) has been the path of the
> community
> > so
> > > > far
> > > > > and I think it has been quite successful.
> > > > >
> > > > > -Jay
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> > kafka@harsha.io
> > > >
> > > > > wrote:
> > > > >
> > > > > > Neha,
> > > > > > "But I haven't seen any community emails or patches being
> submitted
> > > by
> > > > > you
> > > > > > guys, so I'm wondering why you are concerned about whether the
> > > > community
> > > > > is
> > > > > > open to accepting patches or not."
> > > > > >
> > > > > > I think you are talking about contributing patches to this
> > repository
> > > > > > right? https://github.com/confluentinc/kafka-rest . All I am
> > saying
> > > > > > the guidelines/governance model is not clear on the project and I
> > > guess
> > > > > its
> > > > > > driven by opening a github issue request. Its the repository
> owned
> > > by
> > > > > > confluent and as much I appreciate that the features we mentioned
> > are
> > > > in
> > > > > > the roadmap and welcoming us to contribute to the project. It
> > doesn't
> > > > > > gurantee what we want to add in the furture will be in your
> > roadmap.
> > > > > >
> > > > > > Hence the reason having it part of Kafka community will help a
> lot
> > as
> > > > > other
> > > > > > users can participate in the discussions. We are happy to drive
> > any
> > > > > > feature additions through KIPs which gives everyone a chance to
> > > > > participate
> > > > > > and add to the discussions.
> > > > > >
> > > > > > Thanks,
> > > > > > Harsha
> > > > > >
> > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > I agree on the governance comments whole heartedly.
> > > > > > >
> > > > > > > Also i agree about the contribution comments made earlier in
> the
> > > > > thread,
> > > > > > i
> > > > > > > personally am less likely to spend any of mine, or give project
> > > time
> > > > > > within
> > > > > > > my internal projects to developers contributing to another
> > > commercial
> > > > > > > companies project even so technically open source, as then
> there
> > is
> > > > > that
> > > > > > > commercial companies interest will always prevail and
> essentially
> > > can
> > > > > > > always have a final vote where disagreement. Im sure they never
> > > > intend
> > > > > > to,
> > > > > > > but there is that true reality. This is why we have community
> > open
> > > > > source
> > > > > > > projects.
> > > > > > >
> > > > > > > I can find many different implementations now of a rest
> endpoint
> > on
> > > > > > > GitHub, BitBucket etc. Each one has their benefits and
> > > disadvantages
> > > > in
> > > > > > > their implementation. By making / providing one this would
> bring
> > > > > together
> > > > > > > these solutions, unifying those developers and also bringing
> the
> > > best
> > > > > of
> > > > > > > all.
> > > > > > >
> > > > > > > I understand the concern on the community burden
> > adding/supporting
> > > > more
> > > > > > > surface area for every client. But something like REST is
> > universal
> > > > and
> > > > > > > worthy to be owned by the community.
> > > > > > >
> > > > > > > Mike
> > > > > > >
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Andrew Schofield <an...@outlook.com>
> > > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > > >
> > > > > > > There's a massive difference between the governance of Kafka
> and
> > > the
> > > > > > > governance of the REST proxy.
> > > > > > >
> > > > > > > In Kafka, there is a broad community of people contributing
> their
> > > > > > opinions
> > > > > > > about future enhancements in the form of KIPs. There's some
> > really
> > > > deep
> > > > > > > consideration that goes into some of the trickier KIPs. There
> are
> > > > > people
> > > > > > > outside Confluent deeply knowledgeable in Kafka and building
> the
> > > > > > > reputations to become committers. I get the impression that the
> > > > roadmap
> > > > > > of
> > > > > > > Kafka is not really community-owned (what's the big feature for
> > > Kafka
> > > > > > 0.11,
> > > > > > > for example), but the conveyor belt of smaller features in the
> > form
> > > > of
> > > > > > KIPs
> > > > > > > works nicely. It's a good example of open-source working well.
> > > > > > >
> > > > > > > The equivalent for the REST proxy is basically issues on
> GitHub.
> > > The
> > > > > > > roadmap is less clear. There's not really a community properly
> > > > engaged
> > > > > in
> > > > > > > the way that there is with Kafka. So, you could say that it's
> > clear
> > > > > that
> > > > > > > fewer people are interested, but I think the whole governance
> > > thing
> > > > > is a
> > > > > > > big barrier to engagement. And it's looking like it's getting
> out
> > > of
> > > > > > date.
> > > > > > >
> > > > > > > In technical terms, I can think of two big improvements to the
> > REST
> > > > > > proxy.
> > > > > > > First, it needs to use the new consumer API so that it's
> possible
> > > to
> > > > > > secure
> > > > > > > connections between the REST proxy and Kafka. I don't care too
> > much
> > > > > which
> > > > > > > method calls it uses actually uses to consume messages, but I
> do
> > > > care
> > > > > > that
> > > > > > > I cannot secure connections because of network security rules.
> > > > Second,
> > > > > > > there's an affinity between a consumer and the instance of the
> > REST
> > > > > proxy
> > > > > > > to which it first connected. Kafka itself avoids this kind of
> > > > affinity
> > > > > > for
> > > > > > > good reason, and in the name of availability the REST proxy
> > should
> > > > too.
> > > > > > > These are natural KIPs.
> > > > > > >
> > > > > > > I think it would be good to have the code for the REST proxy
> > > > > contributed
> > > > > > > to Apache Kafka so that it would be able to be developed in the
> > > same
> > > > > way.
> > > > > > >
> > > > > > > Andrew Schofield
> > > > > > >
> > > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > > > > Sent: 07 October 2016 22:41:52
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > > >
> > > > > > > ASF already gives us a clear framework and governance model for
> > > > > community
> > > > > > > development. This is already understood by the people
> > contributing
> > > to
> > > > > > > Apache Kafka project, and they are the same people who want to
> > > > > contribute
> > > > > > > to the REST server capability as well. Everyone is in agreement
> > on
> > > > the
> > > > > > > need for collaborating on this effort. So why not contribute
> the
> > > code
> > > > > to
> > > > > > > Apache Kafka. This will help avoid duplication of effort and
> > forks
> > > > that
> > > > > > > may crop up, hugely benefitting the user community. This will
> > also
> > > > > avoid
> > > > > > > having to define a process similar to ASF on a GitHub project
> and
> > > > > instead
> > > > > > > there is a single community with clear understanding community
> > > > process
> > > > > as
> > > > > > > defined in ASF.
> > > > > > >
> > > > > > > As others have said, this is an important capability for Apache
> > > > Kafka.
> > > > > It
> > > > > > > is worth maintaining this as a part of the project.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Suresh
> > > > > > >
> > > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io>
> wrote:
> > > > > > >
> > > > > > > >I personally think it would be quite wasteful to re-implement
> > the
> > > > REST
> > > > > > > >gateway just because that an actively-maintained piece of
> > > > > > Apache-licensed
> > > > > > > >software is not governed directly by the Apache Kafka
> > community...
> > > > > While
> > > > > > > >kafka-rest repo is owned by Confluent, the contributors
> > including
> > > > the
> > > > > > main
> > > > > > > >one are also part of the Apache Kafka community, so there is
> a
> > > > chance
> > > > > > to
> > > > > > > >work this out.
> > > > > > > >
> > > > > > > >However, there are two valid concerns here that could be
> > > addressed,
> > > > > > around
> > > > > > > >community and accessibility:
> > > > > > > >>> What we are worried about is a project
> > > > > > > >>> that's not maintained in a community. So the process of
> > > accepting
> > > > > > > >>>patches
> > > > > > > >>> and priorities is not clear, and it's not developed in
> Apache
> > > > > > > >>>community.
> > > > > > > >>> Not only that, existing REST API project doesn't support
> new
> > > > client
> > > > > > API
> > > > > > > >and
> > > > > > > >>> hence there is no security support either.
> > > > > > > >
> > > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
> > community
> > > > can
> > > > > > > >clarify that - what is their contribution policy, dev style,
> > > roadmap
> > > > > > etc.
> > > > > > > >If they want, they can make an effort to encourage
> participation
> > > > from
> > > > > > > >people outside Confluent (easily accept contributions, invite
> > > > external
> > > > > > > >commiters or have open dev process similar to Apache Kafka
> etc),
> > > as
> > > > > > there
> > > > > > > >is definitely seems to be some interest on the list. That
> might
> > > > clear
> > > > > > the
> > > > > > > >community concern and help kafka-rest project (but that is a
> > > > > calculation
> > > > > > > >Confluent will have to make).
> > > > > > > >
> > > > > > > >The other, independent, concern is that REST is something that
> > is
> > > > > > expected
> > > > > > > >to be available out of the box with Kafka. I personally don't
> > feel
> > > > > > > >strongly
> > > > > > > >about it (better use proper, efficient APIs from day one),
> > though
> > > it
> > > > > is
> > > > > > > >definitely way smaller than adding a stream processing engine
> to
> > > the
> > > > > > > >project :)
> > > > > > > >Again,the kafka-rest "community" could take steps to make it
> > even
> > > > > easier
> > > > > > > >to
> > > > > > > >install, configure and run kafka-rest for new users on vanilla
> > > > Apache
> > > > > > > >Kafka
> > > > > > > >(outside the Confluent platform), if they wish that (or
> welcome
> > > > > > > >contributions to that end), but that is up to them.
> > > > > > > >Finally, if after the above steps were taken there would
> still a
> > > > > strong
> > > > > > > >desire to include a great rest gateway with Apache Kafka, I
> > assume
> > > > the
> > > > > > > >community could hypothetically fork the existing kafka-rest
> into
> > > an
> > > > > > Apache
> > > > > > > >Kafka subproject and maintain it "within Apache" instead of
> > > > > implementing
> > > > > > > >it
> > > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot
> > imagine
> > > it
> > > > > > > >happen
> > > > > > > >without Confluent blessing, and I think that is likely much
> less
> > > > > optimal
> > > > > > > >(pulling in other Confluent / Apache licensed dependencies)
> than
> > > > > having
> > > > > > a
> > > > > > > >separate external community around kafka-rest.
> > > > > > > >
> > > > > > > >
> > > > > > > >Just my two cents,
> > > > > > > >
> > > > > > > >
> > > > > > > >Ofir Manor
> > > > > > > >
> > > > > > > >Co-Founder & CTO | Equalum
> > > > > > > >
> > > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > > <+972%2054-780-1286> |
> > > > > Email:
> > > > > > > ofir.manor@equalum.io
> > > > > > > >
> > > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > > > kafka@harsha.io
> > > > > >
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >> Neha & Jay,
> > > > > > > >> We did look at the open source
> alternatives.
> > > Our
> > > > > > > >>concern
> > > > > > > >> is what's the patch acceptance and adding features/
> bug-fixes
> > to
> > > > the
> > > > > > > >> existing project under a Github (although it's licensed
> under
> > > > Apache
> > > > > > > >>2.0).
> > > > > > > >> It would be great if that project made available under
> Apache
> > > and
> > > > > > > >>driven by
> > > > > > > >> the community. Adding to the above, not all Kafka users are
> > > > > > interested
> > > > > > > >>in
> > > > > > > >> using the Java client API, they would like to have simple
> REST
> > > API
> > > > > > where
> > > > > > > >> they can code against using any language. I do believe this
> > adds
> > > > > value
> > > > > > > >>to
> > > > > > > >> Apache Kafka in itself.
> > > > > > > >>
> > > > > > > >> "For 1, I don't think there is value in giving in to the NIH
> > > > > syndrome
> > > > > > > >>and
> > > > > > > >> reinventing the wheel. What I'm looking for is a detailed
> > > > comparison
> > > > > > of
> > > > > > > >>the
> > > > > > > >> gaps and why those can't be improved in the REST proxy that
> > > > already
> > > > > > > >>exists
> > > > > > > >> and is actively maintained."
> > > > > > > >>
> > > > > > > >> We are not looking at this as NIH. What we are worried
> about
> > > is a
> > > > > > > >>project
> > > > > > > >> that's not maintained in a community. So the process of
> > > accepting
> > > > > > > >>patches
> > > > > > > >> and priorities is not clear, and it's not developed in
> Apache
> > > > > > community.
> > > > > > > >> Not only that, existing REST API project doesn't support new
> > > > client
> > > > > > API
> > > > > > > >>and
> > > > > > > >> hence there is no security support either.
> > > > > > > >> We don't know the timeline when that's made available. We
> > would
> > > > like
> > > > > > to
> > > > > > > >>add
> > > > > > > >> admin functionality into the REST API. So the Roadmap of
> that
> > > > > project
> > > > > > is
> > > > > > > >> not driven by Apache.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> "This doesn't materially have an impact on expanding the
> > > usability
> > > > > of
> > > > > > > >> Kafka. In my experience, REST proxy + Java clients only
> > cover
> > > > > ~50%
> > > > > > of
> > > > > > > >> all
> > > > > > > >> Kafka users, and maybe 10% of those are the ones who will
> > use
> > > > the
> > > > > > > >>REST
> > > > > > > >> proxy. The remaining 50% are non-java client users (C,
> > > python,
> > > > > go,
> > > > > > > >>node
> > > > > > > >> etc)."
> > > > > > > >>
> > > > > > > >> REST API is most often asked feature in my interactions with
> > > Kafka
> > > > > > > >>users.
> > > > > > > >> In an organization, There will be independent teams who will
> > > write
> > > > > > their
> > > > > > > >> Kafka clients using different language libraries available
> > > today,
> > > > > and
> > > > > > > >> there is no way to standardize this. Instead of supporting
> > > several
> > > > > > > >> different client libraries users will be interested in
> using a
> > > > REST
> > > > > > API
> > > > > > > >> server. The need for a REST API will only increase as more
> and
> > > > more
> > > > > > > >>users
> > > > > > > >> start using Kafka.
> > > > > > > >>
> > > > > > > >> "More surface area means more work to keep things
> consistent.
> > > > > Failure
> > > > > > > >> to do that has, in fact, hurt the user experience."
> > > > > > > >> Having myriad Kafka client GitHub projects that support
> > > different
> > > > > > > >>languages
> > > > > > > >> hurts the user experience and pushes burden to maintain
> these
> > > > > > libraries.
> > > > > > > >> REST API is a simple code base that uses existing java
> client
> > > > > > libraries
> > > > > > > >>to
> > > > > > > >> make life easier for the users.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Harsha
> > > > > > > >>
> > > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > > neha@confluent.io>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Manikumar,
> > > > > > > >> >
> > > > > > > >> > Thanks for sharing the proposal. I think there are 2 parts
> > to
> > > > this
> > > > > > > >> > discussion -
> > > > > > > >> >
> > > > > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > > > > >>feature-complete,
> > > > > > > >> > open-source and actively maintained REST proxy in the
> > > community?
> > > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more
> > agile
> > > > and
> > > > > > > >> maintain
> > > > > > > >> > the high-quality experience that Kafka users have today?
> > > > > > > >> >
> > > > > > > >> > For 1, I don't think there is value in giving in to the
> NIH
> > > > > syndrome
> > > > > > > >>and
> > > > > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > > > > comparison
> > > > > > > >>of
> > > > > > > >> the
> > > > > > > >> > gaps and why those can't be improved in the REST proxy
> that
> > > > > already
> > > > > > > >> exists
> > > > > > > >> > and is actively maintained. For example, we depend on
> > zkClient
> > > > and
> > > > > > > >>have
> > > > > > > >> > found as well as fixed several bugs by working closely
> with
> > > the
> > > > > > people
> > > > > > > >> who
> > > > > > > >> > maintain zkClient. This should be possible for REST proxy
> as
> > > > well,
> > > > > > > >>right?
> > > > > > > >> >
> > > > > > > >> > For 2, I'd like us to review our history of expanding the
> > > > surface
> > > > > > > >>area to
> > > > > > > >> > add more clients in the past. Here is a summary -
> > > > > > > >> >
> > > > > > > >> > - This doesn't materially have an impact on expanding
> the
> > > > > > > >>usability of
> > > > > > > >> > Kafka. In my experience, REST proxy + Java clients only
> > > cover
> > > > > > ~50%
> > > > > > > >>of
> > > > > > > >> > all
> > > > > > > >> > Kafka users, and maybe 10% of those are the ones who
> will
> > > use
> > > > > the
> > > > > > > >>REST
> > > > > > > >> > proxy. The remaining 50% are non-java client users (C,
> > > > python,
> > > > > > go,
> > > > > > > >> node
> > > > > > > >> > etc).
> > > > > > > >> > - People are a lot more excited about promising to
> > > contribute
> > > > > > while
> > > > > > > >> > adding the surface area but not on an ongoing basis
> down
> > > the
> > > > > > line.
> > > > > > > >> > - More surface area means more work to keep things
> > > > consistent.
> > > > > > > >>Failure
> > > > > > > >> > to do that has, in fact, hurt the user experience.
> > > > > > > >> > - More surface area hurts agility. We want to do a few
> > > things
> > > > > > > >>really
> > > > > > > >> > well as well as be agile to be able to build on our
> core
> > > > > > > >>competency.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > > > manikumar.reddy@gmail.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Jay,
> > > > > > > >> > >
> > > > > > > >> > > Thanks for your reply.
> > > > > > > >> > >
> > > > > > > >> > > I agree that we can not add all the clients/tools
> > available
> > > in
> > > > > > > >> ecosystem
> > > > > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > > > > different
> > > > > > > >>from
> > > > > > > >> > > other clients/tools. Since any language that can work
> with
> > > > HTTP
> > > > > > can
> > > > > > > >> > > easily integrate with this interface, Having an
> "official"
> > > > REST
> > > > > > > >> > > interface helps user community. This also helps us to
> > > > integrate
> > > > > > well
> > > > > > > >> > > with external management and provisioning tools. Apache
> > > Kafka
> > > > > > > >>release
> > > > > > > >> > > with Java clients + REST interface is sufficient for
> most
> > of
> > > > the
> > > > > > > >>user
> > > > > > > >> > > deployments/requirements. This helps users to deal with
> > less
> > > > > > number
> > > > > > > >> > > of distributions/builds.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Manikumar
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> > jay@confluent.io
> > > >
> > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Hey guys,
> > > > > > > >> > > >
> > > > > > > >> > > > There's already a REST interface maintained as a
> > separate
> > > > > > > >> project--it's
> > > > > > > >> > > > open source and apache licensed and actively
> maintained
> > (
> > > > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is
> > > wrong
> > > > > with
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > > > > github.com
> > > > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
> > > cluster.
> > > > > It
> > > > > > > makes it easy to produce and consume messages, view the state
> of
> > > the
> > > > > > > cluster, and ...
> > > > > > >
> > > > > > > >> that?
> > > > > > > >> > > You
> > > > > > > >> > > > mentioned that there was some compatibility concern,
> but
> > > > > > > >> compatibility
> > > > > > > >> > > has
> > > > > > > >> > > > to do with the consumer protocol guarantees not the
> repo
> > > the
> > > > > > code
> > > > > > > >>is
> > > > > > > >> > in,
> > > > > > > >> > > > right? Not sure that concern makes sense.
> > > > > > > >> > > >
> > > > > > > >> > > > We could argue for adding pretty much anything and
> > > > everything
> > > > > in
> > > > > > > >>the
> > > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that
> > would
> > > > > make
> > > > > > > >>the
> > > > > > > >> > > project
> > > > > > > >> > > > more agile.
> > > > > > > >> > > >
> > > > > > > >> > > > -Jay
> > > > > > > >> > > >
> > > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > > > > >> manikumar.reddy@gmail.com
> > > > > > > >> > >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Hi Kafka Devs,
> > > > > > > >> > > > >
> > > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > > > > Repository.
> > > > > > > >> > > > >
> > > > > > > >> > > > > There are already open-source alternatives are
> > > available.
> > > > > But
> > > > > > > >>we
> > > > > > > >> > would
> > > > > > > >> > > > > like to add REST server that
> > > > > > > >> > > > > many users ask for under Apache Kafka repo. Many
> data
> > > > Infra
> > > > > > > >>tools
> > > > > > > >> > comes
> > > > > > > >> > > > up
> > > > > > > >> > > > > with Rest Interface.
> > > > > > > >> > > > > It is useful to have inbuilt Rest API support for
> > > Produce,
> > > > > > > >>Consume
> > > > > > > >> > > > messages
> > > > > > > >> > > > > and admin interface for
> > > > > > > >> > > > > integrating with external management and
> provisioning
> > > > > > tools.This
> > > > > > > >> will
> > > > > > > >> > > > also
> > > > > > > >> > > > > allow the maintenance of
> > > > > > > >> > > > > REST server and adding new features makes it easy
> > > because
> > > > > > apache
> > > > > > > >> > > > community.
> > > > > > > >> > > > >
> > > > > > > >> > > > > The KIP wiki is the following:
> > > > > > > >> > > > > https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-
> > > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > > > > >> > > > >
> > > > > > > >> > > > > Your comments and feedback are welcome.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > Manikumar
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > --
> > > > > > > >> > Thanks,
> > > > > > > >> > Neha
> > > > > > > >> >
> > > > > > > >>
> > > > > > > The information contained in this email is strictly
> confidential
> > > and
> > > > > for
> > > > > > > the use of the addressee only, unless otherwise indicated. If
> you
> > > are
> > > > > not
> > > > > > > the intended recipient, please do not read, copy, use or
> disclose
> > > to
> > > > > > others
> > > > > > > this message or any attachment. Please also notify the sender
> by
> > > > > replying
> > > > > > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > > the
> > > > > > email
> > > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> > > > to
> > > > > > the
> > > > > > > official business of this company shall be understood as
> neither
> > > > given
> > > > > > nor
> > > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > > > > > > registered in England and Wales, company number 04008957) and
> IG
> > > > Index
> > > > > > > Limited (a company registered in England and Wales, company
> > number
> > > > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > > > Hill,
> > > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > > and
> > > > > IG
> > > > > > > Index Limited (register number 114059) are authorised and
> > regulated
> > > > by
> > > > > > the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > >
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
> >
> >
> > --
> > Nacho (Ignacio) Solis
> > Kafka
> > nsolis@linkedin.com
> >
>
--
Nacho (Ignacio) Solis
Kafka
nsolis@linkedin.com
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Manikumar <ma...@gmail.com>.
Hi Evan,
Thanks for the detailed comments. Sorry for the late reply.
As most of the community members are not in favor of of adding Rest proxy to
Kafka, we decided to drop the KIP proposal. We will work with Confluent's
kafka-rest
project community to contribute new features.
Also, thanks for raising valid queries and helping me to improve the KIP
details.
I will discuss some of your notes (new consumer API, HTTP/2, multi-REST-proxy
etc) at
kafka-rest project issues page .
Thanks you all for participating in this discussion.
Thanks,
Manikumar
On Wed, Oct 26, 2016 at 11:56 AM, Guozhang Wang <wa...@gmail.com> wrote:
> I find it a bit hard to just discuss "whether or not we should add a REST
> proxy into Apache Kafka repo" without discussing about "what should be
> included in the Apache Kafka repo", which, as people mentioned, was a
> grey-area and deserves ongoing discussions. So I would like to first throw
> my thoughts for the second question before the first one:
>
> 1. Reading on this email I saw two ideas on extreme sides of the tradeoff,
> i.e. "we should just keep Kafka broker and Kafka protocol (or even just the
> Kafka protocol) in the core Apache Kafka and leave everything else in other
> repos that depend on it", v.s. "we should consider including any
> eco-systems and tools into Apache Kafka, as long as we believe they are
> commonly used along with Kafka". For myself, I am leaning towards not
> having all-small-projects-style of Apache project, but at the same time I
> am not convinced about going to the other extreme as well. Instead I would
> love to propose something in between: specifically, I think the Kafka
> protocol, the Kafka broker implementation, plus a JVM-based implementation
> of the integrated clients as default and reference implementations: admin
> (although not many people have talked about that, I feel it is one kind of
> clients that should really be considered to add, as proposed in KIP-4),
> producer, consumer, processor (streams), and native pipe (connect and MM),
> to be included in Apache Kafka. Additional clients implementations,
> potentially in other languages -- note we have actually seen other Java
> implementations of producer / consumer and MM as well besides the AK
> provided implementations -- or frameworks, and eco systems and tools such
> like Kafka manager, monitoring facilities, integration with resource
> managers / metrics collectors, to be included in separate repos.
>
> 2. A lot have been said about "whether or not adding a REST proxy", so I
> would not try to restate them again. Just following my thoughts around
> "what we should add", I'd think of a REST proxy as a non-JVM pipe client
> since it usually is used as an extra hop to the Kafka brokers from user
> code or other systems, like MM and connect. Personally I would prefer
> keeping such components in a separate repo instead of adding them into
> Apache Kafka.
>
>
>
> Guozhang
>
> On Tue, Oct 25, 2016 at 9:12 PM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > Sorry that I've stayed quiet on this for a bit. My reason for doing so is
> > well summed up by Jason's notes.
> >
> > First, I do want to say thanks for referencing the Confluent REST Proxy
> in
> > the proposal! It makes me proud that it's being used as an important
> > reference point.
> >
> > Second, Nacho, I want to say thanks to you because as I read this
> thread, I
> > think you have brought the most nuanced, gray-area view of this, which is
> > where it really lies. It *is* fuzzy, and an ongoing discussion. See my
> > answer (somewhere below) re: why I would consider Streams & Connect
> > different than a REST proxy (key word being proxy...). My taste
> definitely
> > skews towards yours wrt keeping inclusion of code in the core project
> > minimal. Jay referenced this a bit as well and I think it's partly a
> > comfort with the lots-of-small-projects style of open source.
> >
> > I want to address a few different areas of concern: the motivation from
> the
> > proposal, a few general observations re: inclusion, and more concrete
> notes
> > on the details of the proposal. email, unfortunately, is not ideal for
> > this, but hopefully this won't be too much for one email...
> >
> > ---
> >
> > On the motivation section, I want to address the 3 key points:
> >
> > > 1) Many data Infra tools comes up with Rest Interface. It is useful to
> > have inbuilt Rest API support for Produce, Consume messages and admin
> > interface for integrating with external management and provisioning
> tools.
> >
> > This doesn't explain why including a REST proxy is a good thing (also,
> REST
> > proxy and REST interface are different things). Just because many tools
> do
> > this doesn't mean its the best choice for Kafka. Presumably there are
> > reasons they choose to, some of why might apply to Kafka, some of which
> may
> > not. It would be helpful to explicitly enumerate these.
> >
> >
> > > 2) Shipping Kafka Rest Server as part of a normal Kafka release makes
> it
> > immediately available to every user that downloads Kafka.
> >
> > I would argue this is a bad thing. There are historical reasons why it
> was
> > challenging to write native clients for each language. It's still
> > non-trivial (see also: Confluent's focus on wrappers of librdkafka rather
> > than writing clients from scratch). But I personally view the REST proxy
> as
> > a stop gap for when there isn't a good "real" client (at least from the
> > perspective of most users). While it is sometimes a necessity given other
> > decisions (I wouldn't expect there to be a high quality Haskell client,
> if
> > that's your language of choice), I also don't think using HTTP here
> should
> > be encouraged by default. It:
> >
> > a) is less efficient (see: encoding overhead, 2x bandwidth overhead,
> > translation overhead, reliance on each client to do good batching)
> > b) has a pretty significant protocol-level impedance mismatch (see:
> > consumers aren't really RESTful)
> > c) still requires extra wrappers (in practice, nobody wants to write a
> > substantial amount of code without a language-specific API)
> >
> > and probably more drawbacks that I am not thinking of off the top of my
> > head.
> >
> > There is *always* a tradeoff between a low-cost, immediately-available
> > solution and the "right" solution. But I'm not convinced we want to
> > showcase what I'd consider a suboptimal (but workable) solution as part
> of
> > the core Kafka offerinag. (Which is why, despite being the original
> author
> > of Confluent's REST proxy, I invest a bunch of my time working on our
> > effort to build better native clients and would love to see REST proxy
> > usage dwindle, despite it being an important part of Confluent's open
> > source offering).
> >
> > I specifically want to call out a comment from earlier in the thread:
> >
> > > Adding to the above, not all Kafka users are interested in using the
> Java
> > client API, they would like to have simple REST API where they can code
> > against using any language. I do believe this adds value to Apache Kafka
> in
> > itself.
> >
> > This incorrectly conflates two things that I think are relevant here. The
> > first is that not everyone wants to use the Java API. That's true. The
> > second is that they want a REST API. That's true for some folks, but
> really
> > it's just a fallback. I think you'd be hard pressed to find anyone who
> > would prefer just a REST API to a native library in their language that
> > they could just install and use naturally.
> >
> >
> > > 3) Helps to maintain the version compatibility between Kafka and Rest
> > Server.
> >
> > The design in the doc never explains how this is true and as far as I can
> > tell it wouldn't be. KIP-35 support in the java clients can help fix
> this,
> > but that doesn't have to do with the REST proxy. As it stands, it is
> > probably *more* painful to do this as part of Kafka (as we have to deal
> > with new build layout organization and some internal/some external
> > dependencies on Kafka clients since you presumably want the REST proxy to
> > depend on an older client version, i.e. lots of extra build system
> > gymnastics that are avoided if it is maintained separately).
> >
> > The one thing I case I can think of where this might be true is that we
> > already have upgrade system tests for Kafka itself and compatibility
> tests
> > with different versions of clients in Kafka today. We could probably get
> > compatibility tests between different versions of a REST Proxy/Kafka
> pretty
> > easily as well. This is true outside of Apache Kafka too, so this doesn't
> > seem to be a strong argument.
> >
> >
> > ---
> >
> > A few general observations re: inclusion of a REST proxy:
> >
> > * Streams and Connect are different clients, but not different
> protocols. I
> > think this is an important differentiation. They offer different
> > abstractions for working with Kafka while retaining all the benefits of
> > actually working *directly* with Kafka. They try to enhance/simplify the
> > "low-level" clients, but don't fundamentally change the interaction with
> > the brokers. All the proxy layers people request are more of like "I like
> > Kafka but would really prefer if it had this different interface". (I
> would
> > love to see more innovation of the type "adds an innovative new API for
> > interacting with Kafka but can be mapped to the normal Kafka protocol
> > without significant loss due to impedance mismatch" and with the right
> APIs
> > I'd easily be supportive of including them in Kafka).
> >
> > * Other protocols are far more prevalent (and efficient) with traditional
> > messaging systems. Why an HTTP interface first? It certainly covers a
> broad
> > set of languages, but is it the best type of proxy to provide? And if
> there
> > are different tradeoffs, why only one? Why not an MQTT proxy? An AMQP
> > proxy?
> >
> > * I think admin APIs are one of the few subsets of the functionality
> where
> > the current state of affairs (CLI tools or protocol with custom
> > serialization as the only public API) is frustrating and a REST API
> > probably *does* work pretty well -- they aren't bandwidth intensive or
> > anything, a REST API would be easier to work with than the low-level
> > interface without having to write support for a bunch of different
> > languages. That said, there are still some tradeoffs here. One example
> that
> > springs to mind is security -- you need to figure out how to forward all
> > sorts of security credentials and probably make a new KIP-4 AdminClient
> per
> > request.
> >
> >
> > ----
> >
> > And a few notes on the proposal:
> >
> > 1. The proposed consumer API: this, along with much of the proposal,
> > mirrors the current implementation of Confluent's kafka-rest project. I
> > agree with many design points, but the current consumer API was designed
> > around the old consumer. We've already started work on replacing it with
> > one better suited to the (quite different) new consumer -- see
> > https://github.com/confluentinc/kafka-rest/wiki/New-Consumer-API-Design
> > for
> > the very basics of the API. Given that the new consumer is the long term
> > commitment for the project, I'd like to see something designed for it
> > instead.
> >
> > 2. Scope: Are we limiting to normal HTTP requests? How about Websockets
> or
> > HTTP/2 PUSH? The latter actually fit with the consumer model much better.
> > Or how about persistent connections that push data, ala streaming APIs
> like
> > Twitter's?
> >
> > You'll get these questions as soon as we make a basic REST API available
> (I
> > know because we've already seen all of these...). Of course the position
> of
> > the project could change over time, but I think this is important because
> > its the difference between committing to maintaining a relatively small
> API
> > to provide reasonable, usable access to users without good clients and
> > providing a full-fledged, feature complete client supporting all types of
> > streaming technologies that work over HTTP are *very* different goals
> with
> > *very* different levels of effort required.
> >
> > The scope, both in terms of initial implementation cost and ongoing
> > maintenance, grow *substantially* larger if you're trying to provide the
> > latter. The community should understand what they are signing up for.
> >
> > 3. Blessed serialization formats. This proposal is supporting binary and
> > JSON. I think people tend to make things overly extensible by default in
> > support of unknown future requirements, but while Confluent could make a
> > decision to support binary & Avro, then eventually add JSON and maybe add
> > support for other serialization formats (which we're willing to do! PRs
> > welcome!), putting something into AK will have different requirements.
> Why
> > not Thrift? Why not protobufs? Why not msgpack?
> >
> > By the way, we also took this into account with Kafka Connect (nee
> Copycat)
> > when submitting it for inclusion in Apache Kafka. We spent quite a bit of
> > time figuring out how to make things work well with pluggable
> serialization
> > -- it didn't start out this way when we were unsure where we thought it
> > would be best for Connect to live. It'll make the proposal larger, but
> I'd
> > question a proposal that blesses certain formats over others given that
> > Kafka is otherwise completely agnostic to format.
> >
> > 4. Related to that, there are some known gaps that don't seem to be
> > addressed here that, with a small amount of GH issue digging, you'd
> > probably find easily. For example, combining simple serializers for keys
> > and "complex" serializers for values. This requires some sort of complex
> > Content-Type and/or Jersey annotation/parsing magic if you go the Jersey
> > route. This is not infrequently requested (e.g. string keys, Avro value)
> > and it'd be good to plan for a design that allows it.
> >
> > The particular issues aren't the point, though. It concerns me is lessons
> > learned, especially re: customer needs, are not being addressed either as
> > future work (with reasonable compatibility suggestion) or argued as out
> of
> > scope.
> >
> > 5. The multi-REST-proxy story is very hand wavy. This is more complicated
> > than it appears at first given a lot of different ways people deploy and
> do
> > service discovery. If we're committing to a particular approach
> (especially
> > re: consumers), we should actually explore the options and constraints
> > here. (Confluent is happy to help out in this regard, we explored lots of
> > options when coming up with our REST proxy design and have learned plenty
> > since that first iteration!)
> >
> > 6. Simple consumer mode. We added this to Confluent's REST proxy
> (actually
> > it was a nice, big, 3rd party addition! Open source works even outside of
> > Apache!). Where does this fit in the APIs? Are we leaving it to future
> > KIPs? If so, we should at least have an idea of where it could fit.
> >
> > 7. The APIs/URLs provided are a bit confusing since they seem to be a
> > subset of what Confluent's proxy provides, yet based on it. In
> particular,
> > the way topics/partitions are organized is confusing if you only have 2
> > produce APIs for them without any support for getting metadata about
> them.
> > At a minimum, this should be documented as leaving room for future API
> > growth.
> >
> > 8. The proposal includes lots of the 5-digit error codes that Confluent's
> > proxy uses. Confluent's proxy assumes a standardized JSON error response
> > body with a message and extended error code since HTTP status codes are
> > quite limiting. Where is this in the proposal?
> >
> > 9. There were some choices (e.g. around offset commit) that Confluent
> made
> > to keep the API simple at the cost of flexibility for REST users. Are
> these
> > preserved because you think they are the right choice? For example, why
> > can't the user include the (topic-partition, offset)s to commit?
> >
> > 10. This is really more of a general comment on KIPs, but almost all KIPs
> > have far too few rejected alternatives, including this one. KIPs should
> be
> > an exploration of the design space that help us arrive on the solution we
> > think is closest to optimal given the information at the time. Is it
> really
> > possible for us, as a community, to just accept that most of what ended
> up
> > in the Confluent REST proxy as the ideal? I certainly don't trust myself
> > that much. As a really small example: I remember having quite a bit of
> > discussion with Neha around how to express "embedded" serialization
> formats
> > (e.g. Content-Type vs query parameter vs in URL). I can't believe we've
> > explored the options in this KIP thoroughly if there are only 2 rejected
> > alternative points and one of them is just about whether it lives inside
> or
> > outside of Kafka...
> >
> >
> > -Ewen
> >
> > On Tue, Oct 25, 2016 at 2:41 AM, Ben Davison <be...@7digital.com>
> > wrote:
> >
> > > Good point Ismael :D
> > >
> > > On Mon, Oct 24, 2016 at 11:07 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > If we want to start a vote on this, I suggest starting a [VOTE]
> thread
> > > > instead of mentioning it in the middle of a very long [DISCUSS]
> thread.
> > > :)
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Oct 24, 2016 at 9:46 PM, Ben Davison <
> ben.davison@7digital.com
> > >
> > > > wrote:
> > > >
> > > > > This KIP has got muddy on differing opinions of what should be part
> > of
> > > > > Kafka etc. I agree with Suresh, let's have a vote on if we actually
> > > want
> > > > a
> > > > > REST client in Kafka core, and then we can work out what it
> actually
> > > > looks
> > > > > like (personally I would like to reuse Confluents, great REST
> client
> > if
> > > > > they donate it to ASF)
> > > > >
> > > > > For me +1 on REST client, this is a fundamental feature that's
> > missing
> > > > from
> > > > > Kafka.
> > > > >
> > > > > On Mon, Oct 24, 2016 at 9:22 PM, Jay Kreps <ja...@confluent.io>
> wrote:
> > > > >
> > > > > > Hey Suresh,
> > > > > >
> > > > > > I think we agree that REST apis are useful. I don't think we
> agree
> > > that
> > > > > > they need to be part of the Kafka project. We've had this
> > discussion
> > > a
> > > > > > dozen odd times for different protocols---AMQP, REST, MQTT. Going
> > > back
> > > > > the
> > > > > > last five years we've always rejected these. That doesn't mean
> they
> > > > > aren't
> > > > > > useful, I think having these as separate is fine, they don't have
> > any
> > > > > > complex interaction with Kafka, they just use the vanilla APIs
> like
> > > any
> > > > > of
> > > > > > the dozens of things on the ecosystem page.
> > > > > >
> > > > > > In terms of how they're developed. I think there are two
> > discussions
> > > > > here:
> > > > > > 1. Separate project or not
> > > > > > 2. Standalone Apache project or github
> > > > > >
> > > > > > The first of those I just talked about---I think this makes sense
> > as
> > > an
> > > > > > independent project.
> > > > > >
> > > > > > For the second of these, I actually don't think that spawning
> off a
> > > > bunch
> > > > > > of itty-bitty independent Apache projects is a good thing. I
> think
> > > you
> > > > > can
> > > > > > see this in the Hadoop ecosystem where the parts kind of all
> evolve
> > > off
> > > > > in
> > > > > > different directions and don't really work together as a whole. I
> > > think
> > > > > > that makes sense for deep infrastructure projects, but not for
> > these
> > > > > little
> > > > > > helper projects. I think a hub and spoke model where you have a
> > > central
> > > > > > project (Kafka) and then a bunch of helper tools that strive to
> fit
> > > in
> > > > > with
> > > > > > it (in terms of config, monitoring, apis, and every other
> > > conventions)
> > > > is
> > > > > > actually much better. In any case there are already so many of
> > these
> > > > > tools
> > > > > > (we capture maybe 20% of them on the ecosystem page), made by so
> > many
> > > > > > people, and virtually all developed in this style, it is a bit
> late
> > > to
> > > > > put
> > > > > > the cat back in the bag.
> > > > > >
> > > > > > Perhaps it's just a difference in background. For my part I had
> > many
> > > > > years
> > > > > > successfully running github-style projects, and i think that
> model
> > > can
> > > > > work
> > > > > > quite well for small things. I do think it is important for these
> > > > > projects
> > > > > > to clarify governance, which we should absolutely do, but
> > > realistically
> > > > > it
> > > > > > is a pretty simple tool, there isn't a huge governance challenge
> > for
> > > > > > something like this because its scope is so limited ("take http
> > > > requests,
> > > > > > make Kafka requests").
> > > > > >
> > > > > > More specifically, I don't think there is an actual problem being
> > > > solved
> > > > > > here. I haven't heard any complaint about direction or patches
> not
> > > > > getting
> > > > > > added. The only complaint I've heard is missing features where
> the
> > > > normal
> > > > > > "patches accepted" rule applies. I would urge people to actually
> > get
> > > > > > involved in contribution here. In the future if there is a
> > situation
> > > > > where
> > > > > > people don't like the direction of a given tool, they can fork it
> > and
> > > > > > either turn it into an independent Apache project or develop it
> > > > > > independently, trying to do that preemptively seems a bit
> hostile.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas <
> > > > > suresh@hortonworks.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I am dividing this discussion into two parts:
> > > > > > > 1. REST APIs as core Apache Kafka capability
> > > > > > > This should be a core Kafka functionality. Same view has been
> > > > reflected
> > > > > > by
> > > > > > > others (users and developers) as well. While we can debate
> > whether
> > > > > other
> > > > > > > capabilities are core Kafka (Streams, Connect), it would be
> good
> > > > > separate
> > > > > > > that out and to keep this discussion focussed on REST APIs as
> > > > proposed
> > > > > by
> > > > > > > this KIP. If there is ambivalence about the need for this in
> core
> > > > > kafka,
> > > > > > > we could have voting in the mailing list.
> > > > > > >
> > > > > > > Can we get an agreement on this? I am +1 on REST API in Apache
> > > Kafka.
> > > > > > >
> > > > > > > 2. Community where Kafka REST APIs need to be collaborated on
> > > > > > > There is already a GitHub project that caters to Kafka REST
> APIs.
> > > > But a
> > > > > > > company owned Github is less than ideal for collaboration due
> to
> > > lack
> > > > > of
> > > > > > > governance, open community and roadmap. This is reflected by
> many
> > > > > people
> > > > > > > interested in this functionality and still not contributing to
> > and
> > > > > > > adopting the code base in the GitHub. I think trying overlay
> the
> > > ASF
> > > > > > > governance model on GitHub project, which is what the need is,
> > > seems
> > > > > > > unnecessary, if the code can be part of Apache Kafka.
> > > > > > >
> > > > > > > The question is, would Confluent be okay with
> > > licensing/contributing
> > > > > the
> > > > > > > code to Kafka project (assuming either Confluent or another
> > > > contributor
> > > > > > > can work on it)? I see clear intent in collaborating with
> others
> > on
> > > > > REST
> > > > > > > APIs from confluent. Why not do it in Kafka project under ASF?
> > This
> > > > > takes
> > > > > > > away all the barrier to collaboration? Alternatively, if
> > Confluent
> > > is
> > > > > not
> > > > > > > willing to contribute the code from GitHub, would anyone veto
> > > > building
> > > > > a
> > > > > > > separate REST API functionality in ASF Kafka? There are enough
> > > people
> > > > > who
> > > > > > > want to work on this and maintain it.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Suresh
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 10/21/16, 9:41 PM, "Harsha Chintalapani" <ka...@harsha.io>
> > > wrote:
> > > > > > >
> > > > > > > >Sriram,
> > > > > > > > "Can the streaming platform exist without stream
> processing?
> > -
> > > > No.
> > > > > > > >Processing stream data again is a core part of streaming
> > > platform."
> > > > > > > >
> > > > > > > >Yes, it can. There are no.of Stream processing frameworks out
> > > there,
> > > > > and
> > > > > > > >they all have integration into Kafka.
> > > > > > > >It doesn't need to be developed within Kafka.
> > > > > > > >
> > > > > > > >
> > > > > > > >"Can the platform exist without the rest proxy? - Yes. The
> proxy
> > > > does
> > > > > > not
> > > > > > > >complete the platform vision in anyway. It is just a good to
> > have
> > > > tool
> > > > > > > >that
> > > > > > > >might be required by quite a few users and there is an active
> > > > project
> > > > > > that
> > > > > > > >works on this - https://github.com/confluentinc/kafka-rest"
> > > > > > > >
> > > > > > > >The rest proxy is as important as any API. The vision that
> shown
> > > > here
> > > > > > > >http://kafka.apache.org/intro
> > > > > > > >require users to write the producers and consumers to get
> their
> > > data
> > > > > > into
> > > > > > > >and out of Kafka, without which having Streams or Connect
> won't
> > > help
> > > > > > > >anyone.
> > > > > > > >The rest proxy makes easier for users get their data into
> Kafka.
> > > > > > > >Adding the rest proxy to the project doesn't invalidate the
> > > current
> > > > > > > >vision,
> > > > > > > >it only strengthens it.
> > > > > > > >
> > > > > > > >Thanks,
> > > > > > > >Harsha
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <
> > > > ram@confluent.io>
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >FWIW, Apache Kafka has evolved a lot from where it started. It
> > did
> > > > > start
> > > > > > > >as
> > > > > > > >a messaging system. Over time we realized that that the vision
> > for
> > > > > Kafka
> > > > > > > >is
> > > > > > > >to build a streaming platform and not just a messaging system.
> > You
> > > > can
> > > > > > > >take
> > > > > > > >a look at the site for more description about what comprises
> the
> > > > > > streaming
> > > > > > > >platform http://kafka.apache.org/ and
> > > http://kafka.apache.org/intro
> > > > .
> > > > > > > >
> > > > > > > >Can the streaming platform exist without Connect? - No. Data
> > > > > integration
> > > > > > > >is
> > > > > > > >fundamental to building an end to end platform
> > > > > > > >
> > > > > > > >Can the streaming platform exist without stream processing? -
> > No.
> > > > > > > >Processing stream data again is a core part of streaming
> > platform.
> > > > > > > >
> > > > > > > >Can the streaming platform exist without clients? - We at
> least
> > > need
> > > > > one
> > > > > > > >client library to complete the platform. Our Java clients help
> > us
> > > to
> > > > > > > >complete the platform story. The rest of the clients are built
> > and
> > > > > > > >maintained outside the project.
> > > > > > > >
> > > > > > > >Can the platform exist without the rest proxy? - Yes. The
> proxy
> > > does
> > > > > not
> > > > > > > >complete the platform vision in anyway. It is just a good to
> > have
> > > > tool
> > > > > > > >that
> > > > > > > >might be required by quite a few users and there is an active
> > > > project
> > > > > > that
> > > > > > > >works on this - https://github.com/confluentinc/kafka-rest
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > > > > > ><ns...@linkedin.com.invalid>
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >> Are you saying Kafka REST is subjective but Kafka Streams
> and
> > > > Kafka
> > > > > > > >Connect
> > > > > > > >> are not subjective?
> > > > > > > >>
> > > > > > > >> > "there are likely places that can live without a rest
> proxy"
> > > > > > > >>
> > > > > > > >> There are also places that can live without Kafka Streams
> and
> > > > Kafka
> > > > > > > >> Connect.
> > > > > > > >>
> > > > > > > >> Nacho
> > > > > > > >>
> > > > > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <jun@confluent.io
> >
> > > > wrote:
> > > > > > > >>
> > > > > > > >> > At the high level, I think ideally it makes sense to add a
> > > > > component
> > > > > > > >>to
> > > > > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs
> > tight
> > > > > > > >integration
> > > > > > > >> > with Kafka core. For Kafka Stream, we do expect stream
> > > > processing
> > > > > > will
> > > > > > > >be
> > > > > > > >> > used widely in the future. Implementation wise, Kafka
> Stream
> > > > only
> > > > > > > >> supports
> > > > > > > >> > getting data from Kafka and leverages quite a few of the
> > core
> > > > > > > >> > functionalities in Kafka core. For example, it uses
> > customized
> > > > > > > >>rebalance
> > > > > > > >> > callback in the consumer and uses the compacted topic
> > heavily.
> > > > So,
> > > > > > > >having
> > > > > > > >> > Kafka Stream in the same repo makes it easier for testing
> > when
> > > > > those
> > > > > > > >core
> > > > > > > >> > functionalities evolve over time. Kafka Connect is in the
> > same
> > > > > > > >situation.
> > > > > > > >> >
> > > > > > > >> > For rest proxy, whether it's widely used or not is going
> to
> > > be a
> > > > > bit
> > > > > > > >> > subjective. However, there are likely places that can live
> > > > > without a
> > > > > > > >rest
> > > > > > > >> > proxy. The rest proxy is just a proxy for the regular
> > clients
> > > > and
> > > > > > > >doesn't
> > > > > > > >> > need to be tightly integrated with Kafka core. So, the
> case
> > > for
> > > > > > > >including
> > > > > > > >> > rest proxy in Apache Kafka is probably not as strong as
> > Kafka
> > > > > Stream
> > > > > > > >>and
> > > > > > > >> > Kafka Connect.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> >
> > > > > > > >> > Jun
> > > > > > > >> >
> > > > > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > > > > > >><Mi...@ig.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > So from my reading essentially the first question needs
> to
> > > > > > > >answered/and
> > > > > > > >> > > voted on is:
> > > > > > > >> > >
> > > > > > > >> > > Is Apache Kafka Community only about the Core or does
> the
> > > > apache
> > > > > > > >> > community
> > > > > > > >> > > also support some subprojects (and just we need some
> > better
> > > > way
> > > > > to
> > > > > > > >> manage
> > > > > > > >> > > this)
> > > > > > > >> > >
> > > > > > > >> > > If vote for Core only wins, then the following should be
> > > > > removed:
> > > > > > > >> > > Kafka Connect
> > > > > > > >> > > Kafka Stream
> > > > > > > >> > >
> > > > > > > >> > > If vote for Core only loses (aka we will support
> > > subprojects)
> > > > > > then:
> > > > > > > >> > > We should look to add Kafka Rest
> > > > > > > >> > >
> > > > > > > >> > > And we should look to see how we can manage better
> govern
> > > and
> > > > > > manage
> > > > > > > >> > > submodules.
> > > > > > > >> > >
> > > > > > > >> > > A good example which id propose here is how some other
> > > > > communities
> > > > > > > >>in
> > > > > > > >> > > Apache do this.
> > > > > > > >> > >
> > > > > > > >> > > Each Module has a Module Management Committee(MMC), this
> > is
> > > > like
> > > > > > > >almost
> > > > > > > >> > > the PMC but at a per module basis.
> > > > > > > >> > >
> > > > > > > >> > > This MMC should essentially hold the binding votes for
> > that
> > > > > > module.
> > > > > > > >> > > The MMC should be made up of a single representative
> from
> > > each
> > > > > > > >> > > organisation (so no single organisation can fully veto
> > the
> > > > > > > >>community
> > > > > > > >> it
> > > > > > > >> > > has to a genuine consenus)
> > > > > > > >> > > The MMC requires at least 3 members (so there cant be a
> > tied
> > > > > vote
> > > > > > on
> > > > > > > >2)
> > > > > > > >> > > For a new Module to be added a MMC committee should be
> > > sought
> > > > > > > >> > > A new Module is only capable of being added if the above
> > > > > > > >>requirements
> > > > > > > >> can
> > > > > > > >> > > be met (e.g. 3 people wishing to step up, from 3
> > > > organisations)
> > > > > so
> > > > > > > >that
> > > > > > > >> > > only actively support modules would be added
> > > > > > > >> > >
> > > > > > > >> > > The PMC reviews each module every 6months or Year. If
> MMC
> > is
> > > > > > > >>inactive,
> > > > > > > >> a
> > > > > > > >> > > vote/call to find replacements if raised, if none are
> > > > > forthcoming
> > > > > > > >> > dropping
> > > > > > > >> > > the MMC to less than 3 then the module moves to "the
> > attic"
> > > > > (very
> > > > > > > >>much
> > > > > > > >> > like
> > > > > > > >> > > apache attic but a little more aggressively)
> > > > > > > >> > >
> > > > > > > >> > > This way the PMC does not need to micro manage every
> > module
> > > > > > > >> > > We only add modules where some amount of active support
> > and
> > > > > > > >maintenance
> > > > > > > >> > > and use is provided by the community
> > > > > > > >> > > We have an automatic way to retire old or inactive
> > projects.
> > > > > > > >> > >
> > > > > > > >> > > Thoughts?
> > > > > > > >> > > Mike
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > ________________________________________
> > > > > > > >> > > From: Harsha Ch <ha...@gmail.com>
> > > > > > > >> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > > > > > >> > > To: dev@kafka.apache.org
> > > > > > > >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > > > >> > >
> > > > > > > >> > > Jay,
> > > > > > > >> > > REST API is something every user is in need of. If
> > the
> > > > > > > >>argument
> > > > > > > >> is
> > > > > > > >> > to
> > > > > > > >> > > clone and write your API, this will do a disservice to
> > the
> > > > > users
> > > > > > as
> > > > > > > >> they
> > > > > > > >> > > now have to choose one vs. others instead of keeping one
> > API
> > > > > that
> > > > > > is
> > > > > > > >> > > supported in Kafka community.
> > > > > > > >> > >
> > > > > > > >> > > "Pre-emptively re-creating another
> > > > > > > >> > > REST layer when it seems like we all quite agree on what
> > > needs
> > > > > to
> > > > > > be
> > > > > > > >> done
> > > > > > > >> > > and we have an existing code base for HTTP/Kafka access
> > that
> > > > is
> > > > > > > >heavily
> > > > > > > >> > > used in production seems quite silly."
> > > > > > > >> > >
> > > > > > > >> > > Exactly our point. Why can't we develop this in
> > > Apache
> > > > > > Kafka
> > > > > > > >> > > community? Instead of us open sourcing another GitHub
> > > project
> > > > > and
> > > > > > > >> > creating
> > > > > > > >> > > a divide in users and another version of API. Let's
> build
> > > this
> > > > > in
> > > > > > > >Kafka
> > > > > > > >> > > Community and use the governance model that is proven to
> > > > provide
> > > > > > > >vendor
> > > > > > > >> > > free user driven consensus features. The argument that
> is
> > > > adding
> > > > > > > >>this
> > > > > > > >> > REST
> > > > > > > >> > > server to Kafka will affect the agility of the project
> > > doesn't
> > > > > mak
> > > > > > > >> sense.
> > > > > > > >> > >
> > > > > > > >> > > It looks like your argument is either we develop all
> these
> > > > small
> > > > > > > >>tools
> > > > > > > >> or
> > > > > > > >> > > none at all. We as a community need to look at
> supporting
> > > > > critical
> > > > > > > >> > > tools/API. Instead of dividing this project into
> > individual
> > > > > > external
> > > > > > > >> > > communities. We should build this as part of Kafka which
> > > best
> > > > > > serves
> > > > > > > >> the
> > > > > > > >> > > needs of users.
> > > > > > > >> > > The Streams and Connect projects that were
> pushed
> > > into
> > > > > > Kafka
> > > > > > > >> > could
> > > > > > > >> > > have been left in their own Github projects based on
> your
> > > > > > arguments.
> > > > > > > >> What
> > > > > > > >> > > about the REST API is so different that such that it
> > should
> > > > stay
> > > > > > out
> > > > > > > >of
> > > > > > > >> > the
> > > > > > > >> > > Kafka project? From my experience, more users are asking
> > for
> > > > the
> > > > > > > >>REST
> > > > > > > >> > API.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Harsha
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <
> > jay@confluent.io
> > > >
> > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > I think the questions around governance make sense, I
> > > think
> > > > we
> > > > > > > >should
> > > > > > > >> > > > really clarify that to make the process more clear so
> it
> > > can
> > > > > be
> > > > > > > >fully
> > > > > > > >> > > > inclusive.
> > > > > > > >> > > >
> > > > > > > >> > > > The idea that we should not collaborate on what is
> there
> > > > now,
> > > > > > > >though,
> > > > > > > >> > > > because in the future we might disagree about
> direction
> > > does
> > > > > not
> > > > > > > >> really
> > > > > > > >> > > > make sense to me. If in the future we disagree, that
> is
> > > the
> > > > > > beauty
> > > > > > > >of
> > > > > > > >> > > open
> > > > > > > >> > > > source, you can always fork off a copy of the code and
> > > start
> > > > > an
> > > > > > > >> > > independent
> > > > > > > >> > > > project either in Apache or elsewhere. Pre-emptively
> > > > > re-creating
> > > > > > > >> > another
> > > > > > > >> > > > REST layer when it seems like we all quite agree on
> what
> > > > needs
> > > > > > to
> > > > > > > >>be
> > > > > > > >> > done
> > > > > > > >> > > > and we have an existing code base for HTTP/kafka
> access
> > > that
> > > > > is
> > > > > > > >> heavily
> > > > > > > >> > > > used in production seems quite silly.
> > > > > > > >> > > >
> > > > > > > >> > > > Let me give some background on how I at least think
> > about
> > > > > these
> > > > > > > >> things.
> > > > > > > >> > > > I've participated in open source projects out of
> > LinkedIn
> > > > via
> > > > > > > >>github
> > > > > > > >> as
> > > > > > > >> > > > well as via the ASF. I don't think there is a "right"
> > > answer
> > > > > to
> > > > > > > >>how
> > > > > > > >> to
> > > > > > > >> > do
> > > > > > > >> > > > these but rather some tradeoffs. We thought about this
> > > > quite a
> > > > > > lot
> > > > > > > >in
> > > > > > > >> > the
> > > > > > > >> > > > context of Kafka based on the experience with the
> Hadoop
> > > > > > ecosystem
> > > > > > > >as
> > > > > > > >> > > well
> > > > > > > >> > > > as from other open source communities.
> > > > > > > >> > > >
> > > > > > > >> > > > There is a rich ecosystem around Kafka. Many of the
> > > projects
> > > > > are
> > > > > > > >> quite
> > > > > > > >> > > > small--single clients or tools that do a single thing
> > > > > well--and
> > > > > > > >> almost
> > > > > > > >> > > none
> > > > > > > >> > > > of them are top level apache projects. I don't think
> > > trying
> > > > to
> > > > > > > >>force
> > > > > > > >> > each
> > > > > > > >> > > > of these to turn into independent Apache projects is
> > > > > necessarily
> > > > > > > >>the
> > > > > > > >> > best
> > > > > > > >> > > > thing for the ecosystem.
> > > > > > > >> > > >
> > > > > > > >> > > > My observation of how this can go wrong is really
> what I
> > > > think
> > > > > > has
> > > > > > > >> > > happened
> > > > > > > >> > > > to the Hadoop ecosystem. There you see quite a zoo of
> > > > projects
> > > > > > > >>which
> > > > > > > >> > all
> > > > > > > >> > > > drift apart and don't quite work together well.
> > > Coordinating
> > > > > > even
> > > > > > > >> > simple
> > > > > > > >> > > > changes and standardization across these is
> > exceptionally
> > > > > > > >>difficult.
> > > > > > > >> > The
> > > > > > > >> > > > result is a bit of a mess for users--the pieces just
> > don't
> > > > > > really
> > > > > > > >> come
> > > > > > > >> > > > together very well. This makes sense for independent
> > > > > > > >>infrastructure
> > > > > > > >> > > systems
> > > > > > > >> > > > (Kudu vs HDFS) but I'm not at all convinced that doing
> > > this
> > > > > for
> > > > > > > >every
> > > > > > > >> > > > little tool or helper library has lead to a desirable
> > > > state. I
> > > > > > > >>think
> > > > > > > >> > the
> > > > > > > >> > > > mode of operating where the Hadoop vendors spawn off a
> > few
> > > > new
> > > > > > > >Apache
> > > > > > > >> > > > projects for each new product initiative, especially
> > since
> > > > > often
> > > > > > > >that
> > > > > > > >> > > > project is only valued by that vendor (and the other
> > > vendor
> > > > > has
> > > > > > a
> > > > > > > >> > > different
> > > > > > > >> > > > competing Apache project) doesn't necessarily do a
> > better
> > > > job
> > > > > at
> > > > > > > >> > > producing
> > > > > > > >> > > > high quality communities or high quality software.
> > > > > > > >> > > >
> > > > > > > >> > > > These tools/connects/clients/proxies and other
> > integration
> > > > > > pieces
> > > > > > > >can
> > > > > > > >> > > take
> > > > > > > >> > > > many forms, but my take of what makes one of these
> > things
> > > > good
> > > > > > is
> > > > > > > >> that
> > > > > > > >> > it
> > > > > > > >> > > > remains simple, does its one thing well, and cleaves
> as
> > > > > closely
> > > > > > as
> > > > > > > >> > > possible
> > > > > > > >> > > > to the conventions for Kafka itself--i.e. doesn't
> invent
> > > new
> > > > > > ways
> > > > > > > >>of
> > > > > > > >> > > > monitoring, configuring, etc. For the tools we've
> > > > contributed
> > > > > > > >>we've
> > > > > > > >> > tried
> > > > > > > >> > > > really hard to make them consistent with Kafka as well
> > as
> > > > with
> > > > > > > >>each
> > > > > > > >> > other
> > > > > > > >> > > > in how testing, configuration, monitoring, etc works.
> > > > > > > >> > > >
> > > > > > > >> > > > I think what Apache does superbly well is create a
> > > community
> > > > > for
> > > > > > > >> > > managing a
> > > > > > > >> > > > large infrastructure layer like Kafka in a vendor
> > > > independent
> > > > > > way.
> > > > > > > >> > What I
> > > > > > > >> > > > think is less successful is attempting to form full
> and
> > > > > > > >>independent
> > > > > > > >> > > apache
> > > > > > > >> > > > communities around very simple single purpose tools,
> > > > > especially
> > > > > > if
> > > > > > > >> you
> > > > > > > >> > > hope
> > > > > > > >> > > > for these to come together into a cohesive toolset
> > across
> > > > > > multiple
> > > > > > > >> such
> > > > > > > >> > > > tools. Much of what Apache does--create a collective
> > > > decision
> > > > > > > >>making
> > > > > > > >> > > > process for resolving disagreement, help to trademark
> > and
> > > > > > protect
> > > > > > > >the
> > > > > > > >> > > marks
> > > > > > > >> > > > of the project, etc just isn't that relevant for
> simple
> > > > > > > >> single-purpose
> > > > > > > >> > > > tools.
> > > > > > > >> > > >
> > > > > > > >> > > > So my take is there are a couple of options:
> > > > > > > >> > > >
> > > > > > > >> > > > 1. We can try to put all the small tools into the
> > > Apache
> > > > > > > >>Project.
> > > > > > > >> I
> > > > > > > >> > > > think this is not the right approach as there is
> > simply
> > > > too
> > > > > > > >>many
> > > > > > > >> of
> > > > > > > >> > > > them,
> > > > > > > >> > > > many in different languages, serving different
> > > protocols,
> > > > > > > >> > integrating
> > > > > > > >> > > > with
> > > > > > > >> > > > particular systems, and a single community can't
> > > > > effectively
> > > > > > > >> > maintain
> > > > > > > >> > > > them
> > > > > > > >> > > > all. Doing this would significantly slow the
> progress
> > > of
> > > > > the
> > > > > > > >Kafka
> > > > > > > >> > > > project.
> > > > > > > >> > > > As a protocol for messaging, I don't really see a
> > case
> > > > for
> > > > > > > >> including
> > > > > > > >> > > > REST
> > > > > > > >> > > > but not MQTT or AMQP which are technically much
> > better
> > > > > suited
> > > > > > > >>to
> > > > > > > >> > > > messaging
> > > > > > > >> > > > and both are widely used for that.
> > > > > > > >> > > > 2. We can treat ecosystem projects that aren't top
> > > level
> > > > > > Apache
> > > > > > > >> > > projects
> > > > > > > >> > > > as invalid and try to recreate them all as Apache
> > > > projects.
> > > > > > > >> > Honestly,
> > > > > > > >> > > > though, if you go to the Kafka ecosystem page
> > virtually
> > > > > none
> > > > > > of
> > > > > > > >> the
> > > > > > > >> > > most
> > > > > > > >> > > > popular add-ons to Kafka are Apache projects. The
> > most
> > > > > > > >>successful
> > > > > > > >> > > > things in
> > > > > > > >> > > > the Kafka ecosystem such as Yahoo Manager,
> > librdkafka,
> > > a
> > > > > > number
> > > > > > > >of
> > > > > > > >> > > other
> > > > > > > >> > > > clients, as well as the existing REST layer have
> > > > succeeded
> > > > > at
> > > > > > > >> > > developing
> > > > > > > >> > > > communities that actively contribute and use these
> > > pieces
> > > > > > and I
> > > > > > > >> > don't
> > > > > > > >> > > > know
> > > > > > > >> > > > that that is a bad thing unless that community
> proves
> > > to
> > > > be
> > > > > > > >> > > uninclusive,
> > > > > > > >> > > > unresponsive, or goes in a bad technical
> > direction--and
> > > > > those
> > > > > > > >>are
> > > > > > > >> > > > failure
> > > > > > > >> > > > modes that all open source efforts face.
> > > > > > > >> > > > 3. We can do what I think makes the most sense and
> > try
> > > to
> > > > > > work
> > > > > > > >> with
> > > > > > > >> > > the
> > > > > > > >> > > > projects that exist in the ecosystem and if the
> > project
> > > > > > doesn't
> > > > > > > >> > have a
> > > > > > > >> > > > responsive community or wants to go in a different
> > > > > direction
> > > > > > > >>fork
> > > > > > > >> or
> > > > > > > >> > > > recreate that work.
> > > > > > > >> > > >
> > > > > > > >> > > > Of course any person can choose whatever of these
> > options
> > > > they
> > > > > > > >>want.
> > > > > > > >> > But
> > > > > > > >> > > > from my point of view, option (3) has been the path of
> > the
> > > > > > > >>community
> > > > > > > >> so
> > > > > > > >> > > far
> > > > > > > >> > > > and I think it has been quite successful.
> > > > > > > >> > > >
> > > > > > > >> > > > -Jay
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani
> <
> > > > > > > >> kafka@harsha.io
> > > > > > > >> > >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Neha,
> > > > > > > >> > > > > "But I haven't seen any community emails or patches
> > > being
> > > > > > > >submitted
> > > > > > > >> > by
> > > > > > > >> > > > you
> > > > > > > >> > > > > guys, so I'm wondering why you are concerned about
> > > whether
> > > > > the
> > > > > > > >> > > community
> > > > > > > >> > > > is
> > > > > > > >> > > > > open to accepting patches or not."
> > > > > > > >> > > > >
> > > > > > > >> > > > > I think you are talking about contributing patches
> to
> > > this
> > > > > > > >> repository
> > > > > > > >> > > > > right? https://github.com/confluentinc/kafka-rest .
> > > All I
> > > > > am
> > > > > > > >> saying
> > > > > > > >> > > > > the guidelines/governance model is not clear on the
> > > > project
> > > > > > and
> > > > > > > >>I
> > > > > > > >> > guess
> > > > > > > >> > > > its
> > > > > > > >> > > > > driven by opening a github issue request. Its the
> > > > > repository
> > > > > > > >owned
> > > > > > > >> > by
> > > > > > > >> > > > > confluent and as much I appreciate that the features
> > we
> > > > > > > >>mentioned
> > > > > > > >> are
> > > > > > > >> > > in
> > > > > > > >> > > > > the roadmap and welcoming us to contribute to the
> > > project.
> > > > > It
> > > > > > > >> doesn't
> > > > > > > >> > > > > gurantee what we want to add in the furture will be
> in
> > > > your
> > > > > > > >> roadmap.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Hence the reason having it part of Kafka community
> > will
> > > > > help a
> > > > > > > >>lot
> > > > > > > >> as
> > > > > > > >> > > > other
> > > > > > > >> > > > > users can participate in the discussions. We are
> > happy
> > > to
> > > > > > drive
> > > > > > > >> any
> > > > > > > >> > > > > feature additions through KIPs which gives everyone
> a
> > > > chance
> > > > > > to
> > > > > > > >> > > > participate
> > > > > > > >> > > > > and add to the discussions.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > Harsha
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > > > > > > >> > Michael.Pearce@ig.com>
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > +1
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I agree on the governance comments whole
> heartedly.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Also i agree about the contribution comments made
> > > > earlier
> > > > > in
> > > > > > > >>the
> > > > > > > >> > > > thread,
> > > > > > > >> > > > > i
> > > > > > > >> > > > > > personally am less likely to spend any of mine, or
> > > give
> > > > > > > >>project
> > > > > > > >> > time
> > > > > > > >> > > > > within
> > > > > > > >> > > > > > my internal projects to developers contributing to
> > > > another
> > > > > > > >> > commercial
> > > > > > > >> > > > > > companies project even so technically open source,
> > as
> > > > then
> > > > > > > >>there
> > > > > > > >> is
> > > > > > > >> > > > that
> > > > > > > >> > > > > > commercial companies interest will always prevail
> > and
> > > > > > > >essentially
> > > > > > > >> > can
> > > > > > > >> > > > > > always have a final vote where disagreement. Im
> sure
> > > > they
> > > > > > > >>never
> > > > > > > >> > > intend
> > > > > > > >> > > > > to,
> > > > > > > >> > > > > > but there is that true reality. This is why we
> have
> > > > > > community
> > > > > > > >> open
> > > > > > > >> > > > source
> > > > > > > >> > > > > > projects.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I can find many different implementations now of a
> > > rest
> > > > > > > >>endpoint
> > > > > > > >> on
> > > > > > > >> > > > > > GitHub, BitBucket etc. Each one has their benefits
> > and
> > > > > > > >> > disadvantages
> > > > > > > >> > > in
> > > > > > > >> > > > > > their implementation. By making / providing one
> this
> > > > would
> > > > > > > >>bring
> > > > > > > >> > > > together
> > > > > > > >> > > > > > these solutions, unifying those developers and
> also
> > > > > bringing
> > > > > > > >>the
> > > > > > > >> > best
> > > > > > > >> > > > of
> > > > > > > >> > > > > > all.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I understand the concern on the community burden
> > > > > > > >> adding/supporting
> > > > > > > >> > > more
> > > > > > > >> > > > > > surface area for every client. But something like
> > REST
> > > > is
> > > > > > > >> universal
> > > > > > > >> > > and
> > > > > > > >> > > > > > worthy to be owned by the community.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Mike
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > ________________________________________
> > > > > > > >> > > > > > From: Andrew Schofield <andrew_schofield_jira@
> > > > outlook.com
> > > > > >
> > > > > > > >> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > > > >> > > > > > To: dev@kafka.apache.org
> > > > > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > There's a massive difference between the
> governance
> > of
> > > > > Kafka
> > > > > > > >>and
> > > > > > > >> > the
> > > > > > > >> > > > > > governance of the REST proxy.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > In Kafka, there is a broad community of people
> > > > > contributing
> > > > > > > >their
> > > > > > > >> > > > > opinions
> > > > > > > >> > > > > > about future enhancements in the form of KIPs.
> > There's
> > > > > some
> > > > > > > >> really
> > > > > > > >> > > deep
> > > > > > > >> > > > > > consideration that goes into some of the trickier
> > > KIPs.
> > > > > > There
> > > > > > > >are
> > > > > > > >> > > > people
> > > > > > > >> > > > > > outside Confluent deeply knowledgeable in Kafka
> and
> > > > > > building
> > > > > > > >the
> > > > > > > >> > > > > > reputations to become committers. I get the
> > impression
> > > > > that
> > > > > > > >>the
> > > > > > > >> > > roadmap
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > Kafka is not really community-owned (what's the
> big
> > > > > feature
> > > > > > > >>for
> > > > > > > >> > Kafka
> > > > > > > >> > > > > 0.11,
> > > > > > > >> > > > > > for example), but the conveyor belt of smaller
> > > features
> > > > in
> > > > > > the
> > > > > > > >> form
> > > > > > > >> > > of
> > > > > > > >> > > > > KIPs
> > > > > > > >> > > > > > works nicely. It's a good example of open-source
> > > > working
> > > > > > > >>well.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > The equivalent for the REST proxy is basically
> > issues
> > > on
> > > > > > > >>GitHub.
> > > > > > > >> > The
> > > > > > > >> > > > > > roadmap is less clear. There's not really a
> > community
> > > > > > properly
> > > > > > > >> > > engaged
> > > > > > > >> > > > in
> > > > > > > >> > > > > > the way that there is with Kafka. So, you could
> say
> > > that
> > > > > > it's
> > > > > > > >> clear
> > > > > > > >> > > > that
> > > > > > > >> > > > > > fewer people are interested, but I think the
> whole
> > > > > > governance
> > > > > > > >> > thing
> > > > > > > >> > > > is a
> > > > > > > >> > > > > > big barrier to engagement. And it's looking like
> > it's
> > > > > > getting
> > > > > > > >out
> > > > > > > >> > of
> > > > > > > >> > > > > date.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > In technical terms, I can think of two big
> > > improvements
> > > > to
> > > > > > the
> > > > > > > >> REST
> > > > > > > >> > > > > proxy.
> > > > > > > >> > > > > > First, it needs to use the new consumer API so
> that
> > > it's
> > > > > > > >possible
> > > > > > > >> > to
> > > > > > > >> > > > > secure
> > > > > > > >> > > > > > connections between the REST proxy and Kafka. I
> > don't
> > > > care
> > > > > > too
> > > > > > > >> much
> > > > > > > >> > > > which
> > > > > > > >> > > > > > method calls it uses actually uses to consume
> > > messages,
> > > > > > but I
> > > > > > > >do
> > > > > > > >> > > care
> > > > > > > >> > > > > that
> > > > > > > >> > > > > > I cannot secure connections because of network
> > > security
> > > > > > rules.
> > > > > > > >> > > Second,
> > > > > > > >> > > > > > there's an affinity between a consumer and the
> > > instance
> > > > of
> > > > > > the
> > > > > > > >> REST
> > > > > > > >> > > > proxy
> > > > > > > >> > > > > > to which it first connected. Kafka itself avoids
> > this
> > > > kind
> > > > > > of
> > > > > > > >> > > affinity
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > good reason, and in the name of availability the
> > REST
> > > > > proxy
> > > > > > > >> should
> > > > > > > >> > > too.
> > > > > > > >> > > > > > These are natural KIPs.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I think it would be good to have the code for the
> > REST
> > > > > proxy
> > > > > > > >> > > > contributed
> > > > > > > >> > > > > > to Apache Kafka so that it would be able to be
> > > developed
> > > > > in
> > > > > > > >>the
> > > > > > > >> > same
> > > > > > > >> > > > way.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Andrew Schofield
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > > > > >> > > > > > Sent: 07 October 2016 22:41:52
> > > > > > > >> > > > > > To: dev@kafka.apache.org
> > > > > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > ASF already gives us a clear framework and
> > governance
> > > > > model
> > > > > > > >>for
> > > > > > > >> > > > community
> > > > > > > >> > > > > > development. This is already understood by the
> > people
> > > > > > > >> contributing
> > > > > > > >> > to
> > > > > > > >> > > > > > Apache Kafka project, and they are the same people
> > who
> > > > > want
> > > > > > to
> > > > > > > >> > > > contribute
> > > > > > > >> > > > > > to the REST server capability as well. Everyone is
> > in
> > > > > > > >>agreement
> > > > > > > >> on
> > > > > > > >> > > the
> > > > > > > >> > > > > > need for collaborating on this effort. So why not
> > > > > contribute
> > > > > > > >>the
> > > > > > > >> > code
> > > > > > > >> > > > to
> > > > > > > >> > > > > > Apache Kafka. This will help avoid duplication of
> > > effort
> > > > > and
> > > > > > > >> forks
> > > > > > > >> > > that
> > > > > > > >> > > > > > may crop up, hugely benefitting the user
> community.
> > > This
> > > > > > will
> > > > > > > >> also
> > > > > > > >> > > > avoid
> > > > > > > >> > > > > > having to define a process similar to ASF on a
> > GitHub
> > > > > > project
> > > > > > > >and
> > > > > > > >> > > > instead
> > > > > > > >> > > > > > there is a single community with clear
> understanding
> > > > > > community
> > > > > > > >> > > process
> > > > > > > >> > > > as
> > > > > > > >> > > > > > defined in ASF.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > As others have said, this is an important
> capability
> > > for
> > > > > > > >>Apache
> > > > > > > >> > > Kafka.
> > > > > > > >> > > > It
> > > > > > > >> > > > > > is worth maintaining this as a part of the
> project.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Regards,
> > > > > > > >> > > > > > Suresh
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <
> > > > ofir.manor@equalum.io>
> > > > > > > >>wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > >I personally think it would be quite wasteful to
> > > > > > re-implement
> > > > > > > >> the
> > > > > > > >> > > REST
> > > > > > > >> > > > > > >gateway just because that an actively-maintained
> > > piece
> > > > of
> > > > > > > >> > > > > Apache-licensed
> > > > > > > >> > > > > > >software is not governed directly by the Apache
> > Kafka
> > > > > > > >> community...
> > > > > > > >> > > > While
> > > > > > > >> > > > > > >kafka-rest repo is owned by Confluent, the
> > > contributors
> > > > > > > >> including
> > > > > > > >> > > the
> > > > > > > >> > > > > main
> > > > > > > >> > > > > > >one are also part of the Apache Kafka community,
> > so
> > > > > there
> > > > > > > >>is a
> > > > > > > >> > > chance
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > >work this out.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >However, there are two valid concerns here that
> > could
> > > > be
> > > > > > > >> > addressed,
> > > > > > > >> > > > > around
> > > > > > > >> > > > > > >community and accessibility:
> > > > > > > >> > > > > > >>> What we are worried about is a project
> > > > > > > >> > > > > > >>> that's not maintained in a community. So the
> > > process
> > > > > of
> > > > > > > >> > accepting
> > > > > > > >> > > > > > >>>patches
> > > > > > > >> > > > > > >>> and priorities is not clear, and it's not
> > > developed
> > > > in
> > > > > > > >Apache
> > > > > > > >> > > > > > >>>community.
> > > > > > > >> > > > > > >>> Not only that, existing REST API project
> doesn't
> > > > > support
> > > > > > > >>new
> > > > > > > >> > > client
> > > > > > > >> > > > > API
> > > > > > > >> > > > > > >and
> > > > > > > >> > > > > > >>> hence there is no security support either.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >This might be easy to fix. Maybe Confluent /
> > > kafka-rest
> > > > > > > >> community
> > > > > > > >> > > can
> > > > > > > >> > > > > > >clarify that - what is their contribution policy,
> > dev
> > > > > > style,
> > > > > > > >> > roadmap
> > > > > > > >> > > > > etc.
> > > > > > > >> > > > > > >If they want, they can make an effort to
> encourage
> > > > > > > >participation
> > > > > > > >> > > from
> > > > > > > >> > > > > > >people outside Confluent (easily accept
> > > contributions,
> > > > > > invite
> > > > > > > >> > > external
> > > > > > > >> > > > > > >commiters or have open dev process similar to
> > Apache
> > > > > Kafka
> > > > > > > >etc),
> > > > > > > >> > as
> > > > > > > >> > > > > there
> > > > > > > >> > > > > > >is definitely seems to be some interest on the
> > list.
> > > > That
> > > > > > > >>might
> > > > > > > >> > > clear
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >community concern and help kafka-rest project
> (but
> > > that
> > > > > is
> > > > > > a
> > > > > > > >> > > > calculation
> > > > > > > >> > > > > > >Confluent will have to make).
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >The other, independent, concern is that REST is
> > > > something
> > > > > > > >>that
> > > > > > > >> is
> > > > > > > >> > > > > expected
> > > > > > > >> > > > > > >to be available out of the box with Kafka. I
> > > personally
> > > > > > don't
> > > > > > > >> feel
> > > > > > > >> > > > > > >strongly
> > > > > > > >> > > > > > >about it (better use proper, efficient APIs from
> > day
> > > > > one),
> > > > > > > >> though
> > > > > > > >> > it
> > > > > > > >> > > > is
> > > > > > > >> > > > > > >definitely way smaller than adding a stream
> > > processing
> > > > > > engine
> > > > > > > >to
> > > > > > > >> > the
> > > > > > > >> > > > > > >project :)
> > > > > > > >> > > > > > >Again,the kafka-rest "community" could take steps
> > to
> > > > make
> > > > > > it
> > > > > > > >> even
> > > > > > > >> > > > easier
> > > > > > > >> > > > > > >to
> > > > > > > >> > > > > > >install, configure and run kafka-rest for new
> users
> > > on
> > > > > > > >>vanilla
> > > > > > > >> > > Apache
> > > > > > > >> > > > > > >Kafka
> > > > > > > >> > > > > > >(outside the Confluent platform), if they wish
> that
> > > (or
> > > > > > > >>welcome
> > > > > > > >> > > > > > >contributions to that end), but that is up to
> them.
> > > > > > > >> > > > > > >Finally, if after the above steps were taken
> there
> > > > would
> > > > > > > >>still
> > > > > > > >a
> > > > > > > >> > > > strong
> > > > > > > >> > > > > > >desire to include a great rest gateway with
> Apache
> > > > > Kafka, I
> > > > > > > >> assume
> > > > > > > >> > > the
> > > > > > > >> > > > > > >community could hypothetically fork the existing
> > > > > kafka-rest
> > > > > > > >into
> > > > > > > >> > an
> > > > > > > >> > > > > Apache
> > > > > > > >> > > > > > >Kafka subproject and maintain it "within Apache"
> > > > instead
> > > > > of
> > > > > > > >> > > > implementing
> > > > > > > >> > > > > > >it
> > > > > > > >> > > > > > >from scratch (though I'm not a lawyer etc) - but
> I
> > > > cannot
> > > > > > > >> imagine
> > > > > > > >> > it
> > > > > > > >> > > > > > >happen
> > > > > > > >> > > > > > >without Confluent blessing, and I think that is
> > > likely
> > > > > much
> > > > > > > >less
> > > > > > > >> > > > optimal
> > > > > > > >> > > > > > >(pulling in other Confluent / Apache licensed
> > > > > dependencies)
> > > > > > > >than
> > > > > > > >> > > > having
> > > > > > > >> > > > > a
> > > > > > > >> > > > > > >separate external community around kafka-rest.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >Just my two cents,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >Ofir Manor
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >Co-Founder & CTO | Equalum
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > > > > > > ><+972%2054-780-1286>
> > > > > > > >> > <+972%2054-780-1286> |
> > > > > > > >> > > > Email:
> > > > > > > >> > > > > > ofir.manor@equalum.io
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha
> > Chintalapani
> > > <
> > > > > > > >> > > kafka@harsha.io
> > > > > > > >> > > > >
> > > > > > > >> > > > > > >wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> Neha & Jay,
> > > > > > > >> > > > > > >> We did look at the open source
> > > > > > > >>alternatives.
> > > > > > > >> > Our
> > > > > > > >> > > > > > >>concern
> > > > > > > >> > > > > > >> is what's the patch acceptance and adding
> > features/
> > > > > > > >>bug-fixes
> > > > > > > >> to
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> existing project under a Github (although it's
> > > > licensed
> > > > > > > >>under
> > > > > > > >> > > Apache
> > > > > > > >> > > > > > >>2.0).
> > > > > > > >> > > > > > >> It would be great if that project made
> available
> > > > under
> > > > > > > >>Apache
> > > > > > > >> > and
> > > > > > > >> > > > > > >>driven by
> > > > > > > >> > > > > > >> the community. Adding to the above, not all
> > Kafka
> > > > > users
> > > > > > > >>are
> > > > > > > >> > > > > interested
> > > > > > > >> > > > > > >>in
> > > > > > > >> > > > > > >> using the Java client API, they would like to
> > have
> > > > > simple
> > > > > > > >REST
> > > > > > > >> > API
> > > > > > > >> > > > > where
> > > > > > > >> > > > > > >> they can code against using any language. I do
> > > > believe
> > > > > > this
> > > > > > > >> adds
> > > > > > > >> > > > value
> > > > > > > >> > > > > > >>to
> > > > > > > >> > > > > > >> Apache Kafka in itself.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> "For 1, I don't think there is value in giving
> in
> > > to
> > > > > the
> > > > > > > >>NIH
> > > > > > > >> > > > syndrome
> > > > > > > >> > > > > > >>and
> > > > > > > >> > > > > > >> reinventing the wheel. What I'm looking for is
> a
> > > > > detailed
> > > > > > > >> > > comparison
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > >>the
> > > > > > > >> > > > > > >> gaps and why those can't be improved in the
> REST
> > > > proxy
> > > > > > that
> > > > > > > >> > > already
> > > > > > > >> > > > > > >>exists
> > > > > > > >> > > > > > >> and is actively maintained."
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> We are not looking at this as NIH. What we are
> > > > worried
> > > > > > > >>about
> > > > > > > >> > is a
> > > > > > > >> > > > > > >>project
> > > > > > > >> > > > > > >> that's not maintained in a community. So the
> > > process
> > > > of
> > > > > > > >> > accepting
> > > > > > > >> > > > > > >>patches
> > > > > > > >> > > > > > >> and priorities is not clear, and it's not
> > developed
> > > > in
> > > > > > > >>Apache
> > > > > > > >> > > > > community.
> > > > > > > >> > > > > > >> Not only that, existing REST API project
> doesn't
> > > > > support
> > > > > > > >>new
> > > > > > > >> > > client
> > > > > > > >> > > > > API
> > > > > > > >> > > > > > >>and
> > > > > > > >> > > > > > >> hence there is no security support either.
> > > > > > > >> > > > > > >> We don't know the timeline when that's made
> > > > available.
> > > > > We
> > > > > > > >> would
> > > > > > > >> > > like
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > >>add
> > > > > > > >> > > > > > >> admin functionality into the REST API. So the
> > > Roadmap
> > > > > of
> > > > > > > >>that
> > > > > > > >> > > > project
> > > > > > > >> > > > > is
> > > > > > > >> > > > > > >> not driven by Apache.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> "This doesn't materially have an impact on
> > > expanding
> > > > > the
> > > > > > > >> > usability
> > > > > > > >> > > > of
> > > > > > > >> > > > > > >> Kafka. In my experience, REST proxy + Java
> > > clients
> > > > > > only
> > > > > > > >> cover
> > > > > > > >> > > > ~50%
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > >> all
> > > > > > > >> > > > > > >> Kafka users, and maybe 10% of those are the
> > ones
> > > > who
> > > > > > > >>will
> > > > > > > >> use
> > > > > > > >> > > the
> > > > > > > >> > > > > > >>REST
> > > > > > > >> > > > > > >> proxy. The remaining 50% are non-java client
> > > users
> > > > > (C,
> > > > > > > >> > python,
> > > > > > > >> > > > go,
> > > > > > > >> > > > > > >>node
> > > > > > > >> > > > > > >> etc)."
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> REST API is most often asked feature in my
> > > > interactions
> > > > > > > >>with
> > > > > > > >> > Kafka
> > > > > > > >> > > > > > >>users.
> > > > > > > >> > > > > > >> In an organization, There will be independent
> > teams
> > > > who
> > > > > > > >>will
> > > > > > > >> > write
> > > > > > > >> > > > > their
> > > > > > > >> > > > > > >> Kafka clients using different language
> libraries
> > > > > > available
> > > > > > > >> > today,
> > > > > > > >> > > > and
> > > > > > > >> > > > > > >> there is no way to standardize this. Instead of
> > > > > > supporting
> > > > > > > >> > several
> > > > > > > >> > > > > > >> different client libraries users will be
> > interested
> > > > in
> > > > > > > >>using
> > > > > > > >a
> > > > > > > >> > > REST
> > > > > > > >> > > > > API
> > > > > > > >> > > > > > >> server. The need for a REST API will only
> > increase
> > > as
> > > > > > more
> > > > > > > >and
> > > > > > > >> > > more
> > > > > > > >> > > > > > >>users
> > > > > > > >> > > > > > >> start using Kafka.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> "More surface area means more work to keep
> things
> > > > > > > >>consistent.
> > > > > > > >> > > > Failure
> > > > > > > >> > > > > > >> to do that has, in fact, hurt the user
> > > > experience."
> > > > > > > >> > > > > > >> Having myriad Kafka client GitHub projects that
> > > > support
> > > > > > > >> > different
> > > > > > > >> > > > > > >>languages
> > > > > > > >> > > > > > >> hurts the user experience and pushes burden to
> > > > maintain
> > > > > > > >>these
> > > > > > > >> > > > > libraries.
> > > > > > > >> > > > > > >> REST API is a simple code base that uses
> existing
> > > > java
> > > > > > > >>client
> > > > > > > >> > > > > libraries
> > > > > > > >> > > > > > >>to
> > > > > > > >> > > > > > >> make life easier for the users.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> Thanks,
> > > > > > > >> > > > > > >> Harsha
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > > > > > > >> > neha@confluent.io>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> > Manikumar,
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > Thanks for sharing the proposal. I think
> there
> > > are
> > > > 2
> > > > > > > >>parts
> > > > > > > >> to
> > > > > > > >> > > this
> > > > > > > >> > > > > > >> > discussion -
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > 1. Should we rewrite a REST proxy given that
> > > there
> > > > > is a
> > > > > > > >> > > > > > >>feature-complete,
> > > > > > > >> > > > > > >> > open-source and actively maintained REST
> proxy
> > in
> > > > the
> > > > > > > >> > community?
> > > > > > > >> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka
> > make
> > > us
> > > > > > more
> > > > > > > >> agile
> > > > > > > >> > > and
> > > > > > > >> > > > > > >> maintain
> > > > > > > >> > > > > > >> > the high-quality experience that Kafka users
> > have
> > > > > > today?
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > For 1, I don't think there is value in giving
> > in
> > > to
> > > > > the
> > > > > > > >>NIH
> > > > > > > >> > > > syndrome
> > > > > > > >> > > > > > >>and
> > > > > > > >> > > > > > >> > reinventing the wheel. What I'm looking for
> is
> > a
> > > > > > detailed
> > > > > > > >> > > > comparison
> > > > > > > >> > > > > > >>of
> > > > > > > >> > > > > > >> the
> > > > > > > >> > > > > > >> > gaps and why those can't be improved in the
> > REST
> > > > > proxy
> > > > > > > >>that
> > > > > > > >> > > > already
> > > > > > > >> > > > > > >> exists
> > > > > > > >> > > > > > >> > and is actively maintained. For example, we
> > > depend
> > > > on
> > > > > > > >> zkClient
> > > > > > > >> > > and
> > > > > > > >> > > > > > >>have
> > > > > > > >> > > > > > >> > found as well as fixed several bugs by
> working
> > > > > closely
> > > > > > > >>with
> > > > > > > >> > the
> > > > > > > >> > > > > people
> > > > > > > >> > > > > > >> who
> > > > > > > >> > > > > > >> > maintain zkClient. This should be possible
> for
> > > REST
> > > > > > proxy
> > > > > > > >as
> > > > > > > >> > > well,
> > > > > > > >> > > > > > >>right?
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > For 2, I'd like us to review our history of
> > > > expanding
> > > > > > the
> > > > > > > >> > > surface
> > > > > > > >> > > > > > >>area to
> > > > > > > >> > > > > > >> > add more clients in the past. Here is a
> > summary -
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > - This doesn't materially have an impact
> on
> > > > > > expanding
> > > > > > > >the
> > > > > > > >> > > > > > >>usability of
> > > > > > > >> > > > > > >> > Kafka. In my experience, REST proxy + Java
> > > > clients
> > > > > > > >>only
> > > > > > > >> > cover
> > > > > > > >> > > > > ~50%
> > > > > > > >> > > > > > >>of
> > > > > > > >> > > > > > >> > all
> > > > > > > >> > > > > > >> > Kafka users, and maybe 10% of those are
> the
> > > ones
> > > > > who
> > > > > > > >will
> > > > > > > >> > use
> > > > > > > >> > > > the
> > > > > > > >> > > > > > >>REST
> > > > > > > >> > > > > > >> > proxy. The remaining 50% are non-java
> client
> > > > users
> > > > > > (C,
> > > > > > > >> > > python,
> > > > > > > >> > > > > go,
> > > > > > > >> > > > > > >> node
> > > > > > > >> > > > > > >> > etc).
> > > > > > > >> > > > > > >> > - People are a lot more excited about
> > > promising
> > > > to
> > > > > > > >> > contribute
> > > > > > > >> > > > > while
> > > > > > > >> > > > > > >> > adding the surface area but not on an
> > ongoing
> > > > > basis
> > > > > > > >>down
> > > > > > > >> > the
> > > > > > > >> > > > > line.
> > > > > > > >> > > > > > >> > - More surface area means more work to
> keep
> > > > things
> > > > > > > >> > > consistent.
> > > > > > > >> > > > > > >>Failure
> > > > > > > >> > > > > > >> > to do that has, in fact, hurt the user
> > > > experience.
> > > > > > > >> > > > > > >> > - More surface area hurts agility. We want
> > to
> > > > do a
> > > > > > few
> > > > > > > >> > things
> > > > > > > >> > > > > > >>really
> > > > > > > >> > > > > > >> > well as well as be agile to be able to
> build
> > > on
> > > > > our
> > > > > > > >>core
> > > > > > > >> > > > > > >>competency.
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > > > > > >> > > > manikumar.reddy@gmail.com
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > wrote:
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > > Hi Jay,
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > Thanks for your reply.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > I agree that we can not add all the
> > > clients/tools
> > > > > > > >> available
> > > > > > > >> > in
> > > > > > > >> > > > > > >> ecosystem
> > > > > > > >> > > > > > >> > > page to Kafka repo itself. But we feel REST
> > > > > Interface
> > > > > > > >>is
> > > > > > > >> > > > different
> > > > > > > >> > > > > > >>from
> > > > > > > >> > > > > > >> > > other clients/tools. Since any language
> that
> > > can
> > > > > work
> > > > > > > >with
> > > > > > > >> > > HTTP
> > > > > > > >> > > > > can
> > > > > > > >> > > > > > >> > > easily integrate with this interface,
> Having
> > an
> > > > > > > >"official"
> > > > > > > >> > > REST
> > > > > > > >> > > > > > >> > > interface helps user community. This also
> > helps
> > > > us
> > > > > to
> > > > > > > >> > > integrate
> > > > > > > >> > > > > well
> > > > > > > >> > > > > > >> > > with external management and provisioning
> > > tools.
> > > > > > > >>Apache
> > > > > > > >> > Kafka
> > > > > > > >> > > > > > >>release
> > > > > > > >> > > > > > >> > > with Java clients + REST interface is
> > > sufficient
> > > > > for
> > > > > > > >>most
> > > > > > > >> of
> > > > > > > >> > > the
> > > > > > > >> > > > > > >>user
> > > > > > > >> > > > > > >> > > deployments/requirements. This helps users
> to
> > > > deal
> > > > > > with
> > > > > > > >> less
> > > > > > > >> > > > > number
> > > > > > > >> > > > > > >> > > of distributions/builds.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > Thanks,
> > > > > > > >> > > > > > >> > > Manikumar
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> > > > > > > >> jay@confluent.io
> > > > > > > >> > >
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > > Hey guys,
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > There's already a REST interface
> maintained
> > > as
> > > > a
> > > > > > > >> separate
> > > > > > > >> > > > > > >> project--it's
> > > > > > > >> > > > > > >> > > > open source and apache licensed and
> > actively
> > > > > > > >>maintained
> > > > > > > >> (
> > > > > > > >> > > > > > >> > > > https://github.com/
> confluentinc/kafka-rest
> > ).
> > > > > What
> > > > > > is
> > > > > > > >> > wrong
> > > > > > > >> > > > with
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for
> > Kafka
> > > > > > > >> > > > > > github.com
> > > > > > > >> > > > > > The Kafka REST Proxy provides a RESTful interface
> > to a
> > > > > Kafka
> > > > > > > >> > cluster.
> > > > > > > >> > > > It
> > > > > > > >> > > > > > makes it easy to produce and consume messages,
> view
> > > the
> > > > > > state
> > > > > > > >>of
> > > > > > > >> > the
> > > > > > > >> > > > > > cluster, and ...
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > >> that?
> > > > > > > >> > > > > > >> > > You
> > > > > > > >> > > > > > >> > > > mentioned that there was some
> compatibility
> > > > > > concern,
> > > > > > > >but
> > > > > > > >> > > > > > >> compatibility
> > > > > > > >> > > > > > >> > > has
> > > > > > > >> > > > > > >> > > > to do with the consumer protocol
> guarantees
> > > not
> > > > > the
> > > > > > > >repo
> > > > > > > >> > the
> > > > > > > >> > > > > code
> > > > > > > >> > > > > > >>is
> > > > > > > >> > > > > > >> > in,
> > > > > > > >> > > > > > >> > > > right? Not sure that concern makes sense.
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > We could argue for adding pretty much
> > > anything
> > > > > and
> > > > > > > >> > > everything
> > > > > > > >> > > > in
> > > > > > > >> > > > > > >>the
> > > > > > > >> > > > > > >> > > > ecosystem page in Kafka itself but I'm
> not
> > > sure
> > > > > > that
> > > > > > > >> would
> > > > > > > >> > > > make
> > > > > > > >> > > > > > >>the
> > > > > > > >> > > > > > >> > > project
> > > > > > > >> > > > > > >> > > > more agile.
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > -Jay
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM,
> > Manikumar <
> > > > > > > >> > > > > > >> manikumar.reddy@gmail.com
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > > wrote:
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > > Hi Kafka Devs,
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > I created KIP-80 to add Kafka REST
> Server
> > > to
> > > > > > Kafka
> > > > > > > >> > > > Repository.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > There are already open-source
> > alternatives
> > > > are
> > > > > > > >> > available.
> > > > > > > >> > > > But
> > > > > > > >> > > > > > >>we
> > > > > > > >> > > > > > >> > would
> > > > > > > >> > > > > > >> > > > > like to add REST server that
> > > > > > > >> > > > > > >> > > > > many users ask for under Apache Kafka
> > repo.
> > > > > Many
> > > > > > > >>data
> > > > > > > >> > > Infra
> > > > > > > >> > > > > > >>tools
> > > > > > > >> > > > > > >> > comes
> > > > > > > >> > > > > > >> > > > up
> > > > > > > >> > > > > > >> > > > > with Rest Interface.
> > > > > > > >> > > > > > >> > > > > It is useful to have inbuilt Rest API
> > > support
> > > > > for
> > > > > > > >> > Produce,
> > > > > > > >> > > > > > >>Consume
> > > > > > > >> > > > > > >> > > > messages
> > > > > > > >> > > > > > >> > > > > and admin interface for
> > > > > > > >> > > > > > >> > > > > integrating with external management
> and
> > > > > > > >>provisioning
> > > > > > > >> > > > > tools.This
> > > > > > > >> > > > > > >> will
> > > > > > > >> > > > > > >> > > > also
> > > > > > > >> > > > > > >> > > > > allow the maintenance of
> > > > > > > >> > > > > > >> > > > > REST server and adding new features
> makes
> > > it
> > > > > easy
> > > > > > > >> > because
> > > > > > > >> > > > > apache
> > > > > > > >> > > > > > >> > > > community.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > The KIP wiki is the following:
> > > > > > > >> > > > > > >> > > > > https://cwiki.apache.org/
> > > > > > > >> confluence/display/KAFKA/KIP-
> > > > > > > >> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > Your comments and feedback are welcome.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > > >> > > > > Manikumar
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > --
> > > > > > > >> > > > > > >> > Thanks,
> > > > > > > >> > > > > > >> > Neha
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > The information contained in this email is
> strictly
> > > > > > > >>confidential
> > > > > > > >> > and
> > > > > > > >> > > > for
> > > > > > > >> > > > > > the use of the addressee only, unless otherwise
> > > > indicated.
> > > > > > If
> > > > > > > >you
> > > > > > > >> > are
> > > > > > > >> > > > not
> > > > > > > >> > > > > > the intended recipient, please do not read, copy,
> > use
> > > or
> > > > > > > >disclose
> > > > > > > >> > to
> > > > > > > >> > > > > others
> > > > > > > >> > > > > > this message or any attachment. Please also notify
> > the
> > > > > > sender
> > > > > > > >>by
> > > > > > > >> > > > replying
> > > > > > > >> > > > > > to this email or by telephone (+44(020 7896 0011)
> > and
> > > > then
> > > > > > > >delete
> > > > > > > >> > the
> > > > > > > >> > > > > email
> > > > > > > >> > > > > > and any copies of it. Opinions, conclusion (etc)
> > that
> > > do
> > > > > not
> > > > > > > >> relate
> > > > > > > >> > > to
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > official business of this company shall be
> > understood
> > > as
> > > > > > > >>neither
> > > > > > > >> > > given
> > > > > > > >> > > > > nor
> > > > > > > >> > > > > > endorsed by it. IG is a trading name of IG Markets
> > > > Limited
> > > > > > (a
> > > > > > > >> > company
> > > > > > > >> > > > > > registered in England and Wales, company number
> > > > 04008957)
> > > > > > and
> > > > > > > >>IG
> > > > > > > >> > > Index
> > > > > > > >> > > > > > Limited (a company registered in England and
> Wales,
> > > > > company
> > > > > > > >> number
> > > > > > > >> > > > > > 01190902). Registered address at Cannon Bridge
> > House,
> > > 25
> > > > > > > >>Dowgate
> > > > > > > >> > > Hill,
> > > > > > > >> > > > > > London EC4R 2YA. Both IG Markets Limited (register
> > > > number
> > > > > > > >195355)
> > > > > > > >> > and
> > > > > > > >> > > > IG
> > > > > > > >> > > > > > Index Limited (register number 114059) are
> > authorised
> > > > and
> > > > > > > >> regulated
> > > > > > > >> > > by
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > Financial Conduct Authority.
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > > The information contained in this email is strictly
> > > > confidential
> > > > > > and
> > > > > > > >> for
> > > > > > > >> > > the use of the addressee only, unless otherwise
> indicated.
> > > If
> > > > > you
> > > > > > > >>are
> > > > > > > >> not
> > > > > > > >> > > the intended recipient, please do not read, copy, use or
> > > > > disclose
> > > > > > to
> > > > > > > >> > others
> > > > > > > >> > > this message or any attachment. Please also notify the
> > > sender
> > > > by
> > > > > > > >> replying
> > > > > > > >> > > to this email or by telephone (+44(020 7896 0011) and
> then
> > > > > delete
> > > > > > > >>the
> > > > > > > >> > email
> > > > > > > >> > > and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > > > > relate
> > > > > > > >>to
> > > > > > > >> > the
> > > > > > > >> > > official business of this company shall be understood as
> > > > neither
> > > > > > > >>given
> > > > > > > >> > nor
> > > > > > > >> > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > > (a
> > > > > > > >>company
> > > > > > > >> > > registered in England and Wales, company number
> 04008957)
> > > and
> > > > IG
> > > > > > > >>Index
> > > > > > > >> > > Limited (a company registered in England and Wales,
> > company
> > > > > number
> > > > > > > >> > > 01190902). Registered address at Cannon Bridge House, 25
> > > > Dowgate
> > > > > > > >>Hill,
> > > > > > > >> > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > > > 195355)
> > > > > > > >>and
> > > > > > > >> IG
> > > > > > > >> > > Index Limited (register number 114059) are authorised
> and
> > > > > > regulated
> > > > > > > >>by
> > > > > > > >> > the
> > > > > > > >> > > Financial Conduct Authority.
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Nacho (Ignacio) Solis
> > > > > > > >> Kafka
> > > > > > > >> nsolis@linkedin.com
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > > > This email, including attachments, is private and confidential. If
> > you
> > > > have
> > > > > received this email in error please notify the sender and delete it
> > > from
> > > > > your system. Emails are not secure and may contain viruses. No
> > > liability
> > > > > can be accepted for viruses that might be transferred by this email
> > or
> > > > any
> > > > > attachment. Any unauthorised copying of this message or
> unauthorised
> > > > > distribution and publication of the information contained herein
> are
> > > > > prohibited.
> > > > >
> > > > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A
> > 2BB.
> > > > > Registered in England and Wales. Registered No. 04843573.
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> > > This email, including attachments, is private and confidential. If you
> > have
> > > received this email in error please notify the sender and delete it
> from
> > > your system. Emails are not secure and may contain viruses. No
> liability
> > > can be accepted for viruses that might be transferred by this email or
> > any
> > > attachment. Any unauthorised copying of this message or unauthorised
> > > distribution and publication of the information contained herein are
> > > prohibited.
> > >
> > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> > > Registered in England and Wales. Registered No. 04843573.
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>
>
>
> --
> -- Guozhang
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Guozhang Wang <wa...@gmail.com>.
I find it a bit hard to just discuss "whether or not we should add a REST
proxy into Apache Kafka repo" without discussing about "what should be
included in the Apache Kafka repo", which, as people mentioned, was a
grey-area and deserves ongoing discussions. So I would like to first throw
my thoughts for the second question before the first one:
1. Reading on this email I saw two ideas on extreme sides of the tradeoff,
i.e. "we should just keep Kafka broker and Kafka protocol (or even just the
Kafka protocol) in the core Apache Kafka and leave everything else in other
repos that depend on it", v.s. "we should consider including any
eco-systems and tools into Apache Kafka, as long as we believe they are
commonly used along with Kafka". For myself, I am leaning towards not
having all-small-projects-style of Apache project, but at the same time I
am not convinced about going to the other extreme as well. Instead I would
love to propose something in between: specifically, I think the Kafka
protocol, the Kafka broker implementation, plus a JVM-based implementation
of the integrated clients as default and reference implementations: admin
(although not many people have talked about that, I feel it is one kind of
clients that should really be considered to add, as proposed in KIP-4),
producer, consumer, processor (streams), and native pipe (connect and MM),
to be included in Apache Kafka. Additional clients implementations,
potentially in other languages -- note we have actually seen other Java
implementations of producer / consumer and MM as well besides the AK
provided implementations -- or frameworks, and eco systems and tools such
like Kafka manager, monitoring facilities, integration with resource
managers / metrics collectors, to be included in separate repos.
2. A lot have been said about "whether or not adding a REST proxy", so I
would not try to restate them again. Just following my thoughts around
"what we should add", I'd think of a REST proxy as a non-JVM pipe client
since it usually is used as an extra hop to the Kafka brokers from user
code or other systems, like MM and connect. Personally I would prefer
keeping such components in a separate repo instead of adding them into
Apache Kafka.
Guozhang
On Tue, Oct 25, 2016 at 9:12 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Sorry that I've stayed quiet on this for a bit. My reason for doing so is
> well summed up by Jason's notes.
>
> First, I do want to say thanks for referencing the Confluent REST Proxy in
> the proposal! It makes me proud that it's being used as an important
> reference point.
>
> Second, Nacho, I want to say thanks to you because as I read this thread, I
> think you have brought the most nuanced, gray-area view of this, which is
> where it really lies. It *is* fuzzy, and an ongoing discussion. See my
> answer (somewhere below) re: why I would consider Streams & Connect
> different than a REST proxy (key word being proxy...). My taste definitely
> skews towards yours wrt keeping inclusion of code in the core project
> minimal. Jay referenced this a bit as well and I think it's partly a
> comfort with the lots-of-small-projects style of open source.
>
> I want to address a few different areas of concern: the motivation from the
> proposal, a few general observations re: inclusion, and more concrete notes
> on the details of the proposal. email, unfortunately, is not ideal for
> this, but hopefully this won't be too much for one email...
>
> ---
>
> On the motivation section, I want to address the 3 key points:
>
> > 1) Many data Infra tools comes up with Rest Interface. It is useful to
> have inbuilt Rest API support for Produce, Consume messages and admin
> interface for integrating with external management and provisioning tools.
>
> This doesn't explain why including a REST proxy is a good thing (also, REST
> proxy and REST interface are different things). Just because many tools do
> this doesn't mean its the best choice for Kafka. Presumably there are
> reasons they choose to, some of why might apply to Kafka, some of which may
> not. It would be helpful to explicitly enumerate these.
>
>
> > 2) Shipping Kafka Rest Server as part of a normal Kafka release makes it
> immediately available to every user that downloads Kafka.
>
> I would argue this is a bad thing. There are historical reasons why it was
> challenging to write native clients for each language. It's still
> non-trivial (see also: Confluent's focus on wrappers of librdkafka rather
> than writing clients from scratch). But I personally view the REST proxy as
> a stop gap for when there isn't a good "real" client (at least from the
> perspective of most users). While it is sometimes a necessity given other
> decisions (I wouldn't expect there to be a high quality Haskell client, if
> that's your language of choice), I also don't think using HTTP here should
> be encouraged by default. It:
>
> a) is less efficient (see: encoding overhead, 2x bandwidth overhead,
> translation overhead, reliance on each client to do good batching)
> b) has a pretty significant protocol-level impedance mismatch (see:
> consumers aren't really RESTful)
> c) still requires extra wrappers (in practice, nobody wants to write a
> substantial amount of code without a language-specific API)
>
> and probably more drawbacks that I am not thinking of off the top of my
> head.
>
> There is *always* a tradeoff between a low-cost, immediately-available
> solution and the "right" solution. But I'm not convinced we want to
> showcase what I'd consider a suboptimal (but workable) solution as part of
> the core Kafka offerinag. (Which is why, despite being the original author
> of Confluent's REST proxy, I invest a bunch of my time working on our
> effort to build better native clients and would love to see REST proxy
> usage dwindle, despite it being an important part of Confluent's open
> source offering).
>
> I specifically want to call out a comment from earlier in the thread:
>
> > Adding to the above, not all Kafka users are interested in using the Java
> client API, they would like to have simple REST API where they can code
> against using any language. I do believe this adds value to Apache Kafka in
> itself.
>
> This incorrectly conflates two things that I think are relevant here. The
> first is that not everyone wants to use the Java API. That's true. The
> second is that they want a REST API. That's true for some folks, but really
> it's just a fallback. I think you'd be hard pressed to find anyone who
> would prefer just a REST API to a native library in their language that
> they could just install and use naturally.
>
>
> > 3) Helps to maintain the version compatibility between Kafka and Rest
> Server.
>
> The design in the doc never explains how this is true and as far as I can
> tell it wouldn't be. KIP-35 support in the java clients can help fix this,
> but that doesn't have to do with the REST proxy. As it stands, it is
> probably *more* painful to do this as part of Kafka (as we have to deal
> with new build layout organization and some internal/some external
> dependencies on Kafka clients since you presumably want the REST proxy to
> depend on an older client version, i.e. lots of extra build system
> gymnastics that are avoided if it is maintained separately).
>
> The one thing I case I can think of where this might be true is that we
> already have upgrade system tests for Kafka itself and compatibility tests
> with different versions of clients in Kafka today. We could probably get
> compatibility tests between different versions of a REST Proxy/Kafka pretty
> easily as well. This is true outside of Apache Kafka too, so this doesn't
> seem to be a strong argument.
>
>
> ---
>
> A few general observations re: inclusion of a REST proxy:
>
> * Streams and Connect are different clients, but not different protocols. I
> think this is an important differentiation. They offer different
> abstractions for working with Kafka while retaining all the benefits of
> actually working *directly* with Kafka. They try to enhance/simplify the
> "low-level" clients, but don't fundamentally change the interaction with
> the brokers. All the proxy layers people request are more of like "I like
> Kafka but would really prefer if it had this different interface". (I would
> love to see more innovation of the type "adds an innovative new API for
> interacting with Kafka but can be mapped to the normal Kafka protocol
> without significant loss due to impedance mismatch" and with the right APIs
> I'd easily be supportive of including them in Kafka).
>
> * Other protocols are far more prevalent (and efficient) with traditional
> messaging systems. Why an HTTP interface first? It certainly covers a broad
> set of languages, but is it the best type of proxy to provide? And if there
> are different tradeoffs, why only one? Why not an MQTT proxy? An AMQP
> proxy?
>
> * I think admin APIs are one of the few subsets of the functionality where
> the current state of affairs (CLI tools or protocol with custom
> serialization as the only public API) is frustrating and a REST API
> probably *does* work pretty well -- they aren't bandwidth intensive or
> anything, a REST API would be easier to work with than the low-level
> interface without having to write support for a bunch of different
> languages. That said, there are still some tradeoffs here. One example that
> springs to mind is security -- you need to figure out how to forward all
> sorts of security credentials and probably make a new KIP-4 AdminClient per
> request.
>
>
> ----
>
> And a few notes on the proposal:
>
> 1. The proposed consumer API: this, along with much of the proposal,
> mirrors the current implementation of Confluent's kafka-rest project. I
> agree with many design points, but the current consumer API was designed
> around the old consumer. We've already started work on replacing it with
> one better suited to the (quite different) new consumer -- see
> https://github.com/confluentinc/kafka-rest/wiki/New-Consumer-API-Design
> for
> the very basics of the API. Given that the new consumer is the long term
> commitment for the project, I'd like to see something designed for it
> instead.
>
> 2. Scope: Are we limiting to normal HTTP requests? How about Websockets or
> HTTP/2 PUSH? The latter actually fit with the consumer model much better.
> Or how about persistent connections that push data, ala streaming APIs like
> Twitter's?
>
> You'll get these questions as soon as we make a basic REST API available (I
> know because we've already seen all of these...). Of course the position of
> the project could change over time, but I think this is important because
> its the difference between committing to maintaining a relatively small API
> to provide reasonable, usable access to users without good clients and
> providing a full-fledged, feature complete client supporting all types of
> streaming technologies that work over HTTP are *very* different goals with
> *very* different levels of effort required.
>
> The scope, both in terms of initial implementation cost and ongoing
> maintenance, grow *substantially* larger if you're trying to provide the
> latter. The community should understand what they are signing up for.
>
> 3. Blessed serialization formats. This proposal is supporting binary and
> JSON. I think people tend to make things overly extensible by default in
> support of unknown future requirements, but while Confluent could make a
> decision to support binary & Avro, then eventually add JSON and maybe add
> support for other serialization formats (which we're willing to do! PRs
> welcome!), putting something into AK will have different requirements. Why
> not Thrift? Why not protobufs? Why not msgpack?
>
> By the way, we also took this into account with Kafka Connect (nee Copycat)
> when submitting it for inclusion in Apache Kafka. We spent quite a bit of
> time figuring out how to make things work well with pluggable serialization
> -- it didn't start out this way when we were unsure where we thought it
> would be best for Connect to live. It'll make the proposal larger, but I'd
> question a proposal that blesses certain formats over others given that
> Kafka is otherwise completely agnostic to format.
>
> 4. Related to that, there are some known gaps that don't seem to be
> addressed here that, with a small amount of GH issue digging, you'd
> probably find easily. For example, combining simple serializers for keys
> and "complex" serializers for values. This requires some sort of complex
> Content-Type and/or Jersey annotation/parsing magic if you go the Jersey
> route. This is not infrequently requested (e.g. string keys, Avro value)
> and it'd be good to plan for a design that allows it.
>
> The particular issues aren't the point, though. It concerns me is lessons
> learned, especially re: customer needs, are not being addressed either as
> future work (with reasonable compatibility suggestion) or argued as out of
> scope.
>
> 5. The multi-REST-proxy story is very hand wavy. This is more complicated
> than it appears at first given a lot of different ways people deploy and do
> service discovery. If we're committing to a particular approach (especially
> re: consumers), we should actually explore the options and constraints
> here. (Confluent is happy to help out in this regard, we explored lots of
> options when coming up with our REST proxy design and have learned plenty
> since that first iteration!)
>
> 6. Simple consumer mode. We added this to Confluent's REST proxy (actually
> it was a nice, big, 3rd party addition! Open source works even outside of
> Apache!). Where does this fit in the APIs? Are we leaving it to future
> KIPs? If so, we should at least have an idea of where it could fit.
>
> 7. The APIs/URLs provided are a bit confusing since they seem to be a
> subset of what Confluent's proxy provides, yet based on it. In particular,
> the way topics/partitions are organized is confusing if you only have 2
> produce APIs for them without any support for getting metadata about them.
> At a minimum, this should be documented as leaving room for future API
> growth.
>
> 8. The proposal includes lots of the 5-digit error codes that Confluent's
> proxy uses. Confluent's proxy assumes a standardized JSON error response
> body with a message and extended error code since HTTP status codes are
> quite limiting. Where is this in the proposal?
>
> 9. There were some choices (e.g. around offset commit) that Confluent made
> to keep the API simple at the cost of flexibility for REST users. Are these
> preserved because you think they are the right choice? For example, why
> can't the user include the (topic-partition, offset)s to commit?
>
> 10. This is really more of a general comment on KIPs, but almost all KIPs
> have far too few rejected alternatives, including this one. KIPs should be
> an exploration of the design space that help us arrive on the solution we
> think is closest to optimal given the information at the time. Is it really
> possible for us, as a community, to just accept that most of what ended up
> in the Confluent REST proxy as the ideal? I certainly don't trust myself
> that much. As a really small example: I remember having quite a bit of
> discussion with Neha around how to express "embedded" serialization formats
> (e.g. Content-Type vs query parameter vs in URL). I can't believe we've
> explored the options in this KIP thoroughly if there are only 2 rejected
> alternative points and one of them is just about whether it lives inside or
> outside of Kafka...
>
>
> -Ewen
>
> On Tue, Oct 25, 2016 at 2:41 AM, Ben Davison <be...@7digital.com>
> wrote:
>
> > Good point Ismael :D
> >
> > On Mon, Oct 24, 2016 at 11:07 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > If we want to start a vote on this, I suggest starting a [VOTE] thread
> > > instead of mentioning it in the middle of a very long [DISCUSS] thread.
> > :)
> > >
> > > Ismael
> > >
> > > On Mon, Oct 24, 2016 at 9:46 PM, Ben Davison <ben.davison@7digital.com
> >
> > > wrote:
> > >
> > > > This KIP has got muddy on differing opinions of what should be part
> of
> > > > Kafka etc. I agree with Suresh, let's have a vote on if we actually
> > want
> > > a
> > > > REST client in Kafka core, and then we can work out what it actually
> > > looks
> > > > like (personally I would like to reuse Confluents, great REST client
> if
> > > > they donate it to ASF)
> > > >
> > > > For me +1 on REST client, this is a fundamental feature that's
> missing
> > > from
> > > > Kafka.
> > > >
> > > > On Mon, Oct 24, 2016 at 9:22 PM, Jay Kreps <ja...@confluent.io> wrote:
> > > >
> > > > > Hey Suresh,
> > > > >
> > > > > I think we agree that REST apis are useful. I don't think we agree
> > that
> > > > > they need to be part of the Kafka project. We've had this
> discussion
> > a
> > > > > dozen odd times for different protocols---AMQP, REST, MQTT. Going
> > back
> > > > the
> > > > > last five years we've always rejected these. That doesn't mean they
> > > > aren't
> > > > > useful, I think having these as separate is fine, they don't have
> any
> > > > > complex interaction with Kafka, they just use the vanilla APIs like
> > any
> > > > of
> > > > > the dozens of things on the ecosystem page.
> > > > >
> > > > > In terms of how they're developed. I think there are two
> discussions
> > > > here:
> > > > > 1. Separate project or not
> > > > > 2. Standalone Apache project or github
> > > > >
> > > > > The first of those I just talked about---I think this makes sense
> as
> > an
> > > > > independent project.
> > > > >
> > > > > For the second of these, I actually don't think that spawning off a
> > > bunch
> > > > > of itty-bitty independent Apache projects is a good thing. I think
> > you
> > > > can
> > > > > see this in the Hadoop ecosystem where the parts kind of all evolve
> > off
> > > > in
> > > > > different directions and don't really work together as a whole. I
> > think
> > > > > that makes sense for deep infrastructure projects, but not for
> these
> > > > little
> > > > > helper projects. I think a hub and spoke model where you have a
> > central
> > > > > project (Kafka) and then a bunch of helper tools that strive to fit
> > in
> > > > with
> > > > > it (in terms of config, monitoring, apis, and every other
> > conventions)
> > > is
> > > > > actually much better. In any case there are already so many of
> these
> > > > tools
> > > > > (we capture maybe 20% of them on the ecosystem page), made by so
> many
> > > > > people, and virtually all developed in this style, it is a bit late
> > to
> > > > put
> > > > > the cat back in the bag.
> > > > >
> > > > > Perhaps it's just a difference in background. For my part I had
> many
> > > > years
> > > > > successfully running github-style projects, and i think that model
> > can
> > > > work
> > > > > quite well for small things. I do think it is important for these
> > > > projects
> > > > > to clarify governance, which we should absolutely do, but
> > realistically
> > > > it
> > > > > is a pretty simple tool, there isn't a huge governance challenge
> for
> > > > > something like this because its scope is so limited ("take http
> > > requests,
> > > > > make Kafka requests").
> > > > >
> > > > > More specifically, I don't think there is an actual problem being
> > > solved
> > > > > here. I haven't heard any complaint about direction or patches not
> > > > getting
> > > > > added. The only complaint I've heard is missing features where the
> > > normal
> > > > > "patches accepted" rule applies. I would urge people to actually
> get
> > > > > involved in contribution here. In the future if there is a
> situation
> > > > where
> > > > > people don't like the direction of a given tool, they can fork it
> and
> > > > > either turn it into an independent Apache project or develop it
> > > > > independently, trying to do that preemptively seems a bit hostile.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas <
> > > > suresh@hortonworks.com>
> > > > > wrote:
> > > > >
> > > > > > I am dividing this discussion into two parts:
> > > > > > 1. REST APIs as core Apache Kafka capability
> > > > > > This should be a core Kafka functionality. Same view has been
> > > reflected
> > > > > by
> > > > > > others (users and developers) as well. While we can debate
> whether
> > > > other
> > > > > > capabilities are core Kafka (Streams, Connect), it would be good
> > > > separate
> > > > > > that out and to keep this discussion focussed on REST APIs as
> > > proposed
> > > > by
> > > > > > this KIP. If there is ambivalence about the need for this in core
> > > > kafka,
> > > > > > we could have voting in the mailing list.
> > > > > >
> > > > > > Can we get an agreement on this? I am +1 on REST API in Apache
> > Kafka.
> > > > > >
> > > > > > 2. Community where Kafka REST APIs need to be collaborated on
> > > > > > There is already a GitHub project that caters to Kafka REST APIs.
> > > But a
> > > > > > company owned Github is less than ideal for collaboration due to
> > lack
> > > > of
> > > > > > governance, open community and roadmap. This is reflected by many
> > > > people
> > > > > > interested in this functionality and still not contributing to
> and
> > > > > > adopting the code base in the GitHub. I think trying overlay the
> > ASF
> > > > > > governance model on GitHub project, which is what the need is,
> > seems
> > > > > > unnecessary, if the code can be part of Apache Kafka.
> > > > > >
> > > > > > The question is, would Confluent be okay with
> > licensing/contributing
> > > > the
> > > > > > code to Kafka project (assuming either Confluent or another
> > > contributor
> > > > > > can work on it)? I see clear intent in collaborating with others
> on
> > > > REST
> > > > > > APIs from confluent. Why not do it in Kafka project under ASF?
> This
> > > > takes
> > > > > > away all the barrier to collaboration? Alternatively, if
> Confluent
> > is
> > > > not
> > > > > > willing to contribute the code from GitHub, would anyone veto
> > > building
> > > > a
> > > > > > separate REST API functionality in ASF Kafka? There are enough
> > people
> > > > who
> > > > > > want to work on this and maintain it.
> > > > > >
> > > > > > Regards,
> > > > > > Suresh
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 10/21/16, 9:41 PM, "Harsha Chintalapani" <ka...@harsha.io>
> > wrote:
> > > > > >
> > > > > > >Sriram,
> > > > > > > "Can the streaming platform exist without stream processing?
> -
> > > No.
> > > > > > >Processing stream data again is a core part of streaming
> > platform."
> > > > > > >
> > > > > > >Yes, it can. There are no.of Stream processing frameworks out
> > there,
> > > > and
> > > > > > >they all have integration into Kafka.
> > > > > > >It doesn't need to be developed within Kafka.
> > > > > > >
> > > > > > >
> > > > > > >"Can the platform exist without the rest proxy? - Yes. The proxy
> > > does
> > > > > not
> > > > > > >complete the platform vision in anyway. It is just a good to
> have
> > > tool
> > > > > > >that
> > > > > > >might be required by quite a few users and there is an active
> > > project
> > > > > that
> > > > > > >works on this - https://github.com/confluentinc/kafka-rest"
> > > > > > >
> > > > > > >The rest proxy is as important as any API. The vision that shown
> > > here
> > > > > > >http://kafka.apache.org/intro
> > > > > > >require users to write the producers and consumers to get their
> > data
> > > > > into
> > > > > > >and out of Kafka, without which having Streams or Connect won't
> > help
> > > > > > >anyone.
> > > > > > >The rest proxy makes easier for users get their data into Kafka.
> > > > > > >Adding the rest proxy to the project doesn't invalidate the
> > current
> > > > > > >vision,
> > > > > > >it only strengthens it.
> > > > > > >
> > > > > > >Thanks,
> > > > > > >Harsha
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <
> > > ram@confluent.io>
> > > > > > >wrote:
> > > > > > >
> > > > > > >FWIW, Apache Kafka has evolved a lot from where it started. It
> did
> > > > start
> > > > > > >as
> > > > > > >a messaging system. Over time we realized that that the vision
> for
> > > > Kafka
> > > > > > >is
> > > > > > >to build a streaming platform and not just a messaging system.
> You
> > > can
> > > > > > >take
> > > > > > >a look at the site for more description about what comprises the
> > > > > streaming
> > > > > > >platform http://kafka.apache.org/ and
> > http://kafka.apache.org/intro
> > > .
> > > > > > >
> > > > > > >Can the streaming platform exist without Connect? - No. Data
> > > > integration
> > > > > > >is
> > > > > > >fundamental to building an end to end platform
> > > > > > >
> > > > > > >Can the streaming platform exist without stream processing? -
> No.
> > > > > > >Processing stream data again is a core part of streaming
> platform.
> > > > > > >
> > > > > > >Can the streaming platform exist without clients? - We at least
> > need
> > > > one
> > > > > > >client library to complete the platform. Our Java clients help
> us
> > to
> > > > > > >complete the platform story. The rest of the clients are built
> and
> > > > > > >maintained outside the project.
> > > > > > >
> > > > > > >Can the platform exist without the rest proxy? - Yes. The proxy
> > does
> > > > not
> > > > > > >complete the platform vision in anyway. It is just a good to
> have
> > > tool
> > > > > > >that
> > > > > > >might be required by quite a few users and there is an active
> > > project
> > > > > that
> > > > > > >works on this - https://github.com/confluentinc/kafka-rest
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > > > > ><ns...@linkedin.com.invalid>
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Are you saying Kafka REST is subjective but Kafka Streams and
> > > Kafka
> > > > > > >Connect
> > > > > > >> are not subjective?
> > > > > > >>
> > > > > > >> > "there are likely places that can live without a rest proxy"
> > > > > > >>
> > > > > > >> There are also places that can live without Kafka Streams and
> > > Kafka
> > > > > > >> Connect.
> > > > > > >>
> > > > > > >> Nacho
> > > > > > >>
> > > > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > > >>
> > > > > > >> > At the high level, I think ideally it makes sense to add a
> > > > component
> > > > > > >>to
> > > > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs
> tight
> > > > > > >integration
> > > > > > >> > with Kafka core. For Kafka Stream, we do expect stream
> > > processing
> > > > > will
> > > > > > >be
> > > > > > >> > used widely in the future. Implementation wise, Kafka Stream
> > > only
> > > > > > >> supports
> > > > > > >> > getting data from Kafka and leverages quite a few of the
> core
> > > > > > >> > functionalities in Kafka core. For example, it uses
> customized
> > > > > > >>rebalance
> > > > > > >> > callback in the consumer and uses the compacted topic
> heavily.
> > > So,
> > > > > > >having
> > > > > > >> > Kafka Stream in the same repo makes it easier for testing
> when
> > > > those
> > > > > > >core
> > > > > > >> > functionalities evolve over time. Kafka Connect is in the
> same
> > > > > > >situation.
> > > > > > >> >
> > > > > > >> > For rest proxy, whether it's widely used or not is going to
> > be a
> > > > bit
> > > > > > >> > subjective. However, there are likely places that can live
> > > > without a
> > > > > > >rest
> > > > > > >> > proxy. The rest proxy is just a proxy for the regular
> clients
> > > and
> > > > > > >doesn't
> > > > > > >> > need to be tightly integrated with Kafka core. So, the case
> > for
> > > > > > >including
> > > > > > >> > rest proxy in Apache Kafka is probably not as strong as
> Kafka
> > > > Stream
> > > > > > >>and
> > > > > > >> > Kafka Connect.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> >
> > > > > > >> > Jun
> > > > > > >> >
> > > > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > > > > >><Mi...@ig.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > So from my reading essentially the first question needs to
> > > > > > >answered/and
> > > > > > >> > > voted on is:
> > > > > > >> > >
> > > > > > >> > > Is Apache Kafka Community only about the Core or does the
> > > apache
> > > > > > >> > community
> > > > > > >> > > also support some subprojects (and just we need some
> better
> > > way
> > > > to
> > > > > > >> manage
> > > > > > >> > > this)
> > > > > > >> > >
> > > > > > >> > > If vote for Core only wins, then the following should be
> > > > removed:
> > > > > > >> > > Kafka Connect
> > > > > > >> > > Kafka Stream
> > > > > > >> > >
> > > > > > >> > > If vote for Core only loses (aka we will support
> > subprojects)
> > > > > then:
> > > > > > >> > > We should look to add Kafka Rest
> > > > > > >> > >
> > > > > > >> > > And we should look to see how we can manage better govern
> > and
> > > > > manage
> > > > > > >> > > submodules.
> > > > > > >> > >
> > > > > > >> > > A good example which id propose here is how some other
> > > > communities
> > > > > > >>in
> > > > > > >> > > Apache do this.
> > > > > > >> > >
> > > > > > >> > > Each Module has a Module Management Committee(MMC), this
> is
> > > like
> > > > > > >almost
> > > > > > >> > > the PMC but at a per module basis.
> > > > > > >> > >
> > > > > > >> > > This MMC should essentially hold the binding votes for
> that
> > > > > module.
> > > > > > >> > > The MMC should be made up of a single representative from
> > each
> > > > > > >> > > organisation (so no single organisation can fully veto
> the
> > > > > > >>community
> > > > > > >> it
> > > > > > >> > > has to a genuine consenus)
> > > > > > >> > > The MMC requires at least 3 members (so there cant be a
> tied
> > > > vote
> > > > > on
> > > > > > >2)
> > > > > > >> > > For a new Module to be added a MMC committee should be
> > sought
> > > > > > >> > > A new Module is only capable of being added if the above
> > > > > > >>requirements
> > > > > > >> can
> > > > > > >> > > be met (e.g. 3 people wishing to step up, from 3
> > > organisations)
> > > > so
> > > > > > >that
> > > > > > >> > > only actively support modules would be added
> > > > > > >> > >
> > > > > > >> > > The PMC reviews each module every 6months or Year. If MMC
> is
> > > > > > >>inactive,
> > > > > > >> a
> > > > > > >> > > vote/call to find replacements if raised, if none are
> > > > forthcoming
> > > > > > >> > dropping
> > > > > > >> > > the MMC to less than 3 then the module moves to "the
> attic"
> > > > (very
> > > > > > >>much
> > > > > > >> > like
> > > > > > >> > > apache attic but a little more aggressively)
> > > > > > >> > >
> > > > > > >> > > This way the PMC does not need to micro manage every
> module
> > > > > > >> > > We only add modules where some amount of active support
> and
> > > > > > >maintenance
> > > > > > >> > > and use is provided by the community
> > > > > > >> > > We have an automatic way to retire old or inactive
> projects.
> > > > > > >> > >
> > > > > > >> > > Thoughts?
> > > > > > >> > > Mike
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > ________________________________________
> > > > > > >> > > From: Harsha Ch <ha...@gmail.com>
> > > > > > >> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > > > > >> > > To: dev@kafka.apache.org
> > > > > > >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > > >> > >
> > > > > > >> > > Jay,
> > > > > > >> > > REST API is something every user is in need of. If
> the
> > > > > > >>argument
> > > > > > >> is
> > > > > > >> > to
> > > > > > >> > > clone and write your API, this will do a disservice to
> the
> > > > users
> > > > > as
> > > > > > >> they
> > > > > > >> > > now have to choose one vs. others instead of keeping one
> API
> > > > that
> > > > > is
> > > > > > >> > > supported in Kafka community.
> > > > > > >> > >
> > > > > > >> > > "Pre-emptively re-creating another
> > > > > > >> > > REST layer when it seems like we all quite agree on what
> > needs
> > > > to
> > > > > be
> > > > > > >> done
> > > > > > >> > > and we have an existing code base for HTTP/Kafka access
> that
> > > is
> > > > > > >heavily
> > > > > > >> > > used in production seems quite silly."
> > > > > > >> > >
> > > > > > >> > > Exactly our point. Why can't we develop this in
> > Apache
> > > > > Kafka
> > > > > > >> > > community? Instead of us open sourcing another GitHub
> > project
> > > > and
> > > > > > >> > creating
> > > > > > >> > > a divide in users and another version of API. Let's build
> > this
> > > > in
> > > > > > >Kafka
> > > > > > >> > > Community and use the governance model that is proven to
> > > provide
> > > > > > >vendor
> > > > > > >> > > free user driven consensus features. The argument that is
> > > adding
> > > > > > >>this
> > > > > > >> > REST
> > > > > > >> > > server to Kafka will affect the agility of the project
> > doesn't
> > > > mak
> > > > > > >> sense.
> > > > > > >> > >
> > > > > > >> > > It looks like your argument is either we develop all these
> > > small
> > > > > > >>tools
> > > > > > >> or
> > > > > > >> > > none at all. We as a community need to look at supporting
> > > > critical
> > > > > > >> > > tools/API. Instead of dividing this project into
> individual
> > > > > external
> > > > > > >> > > communities. We should build this as part of Kafka which
> > best
> > > > > serves
> > > > > > >> the
> > > > > > >> > > needs of users.
> > > > > > >> > > The Streams and Connect projects that were pushed
> > into
> > > > > Kafka
> > > > > > >> > could
> > > > > > >> > > have been left in their own Github projects based on your
> > > > > arguments.
> > > > > > >> What
> > > > > > >> > > about the REST API is so different that such that it
> should
> > > stay
> > > > > out
> > > > > > >of
> > > > > > >> > the
> > > > > > >> > > Kafka project? From my experience, more users are asking
> for
> > > the
> > > > > > >>REST
> > > > > > >> > API.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Harsha
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <
> jay@confluent.io
> > >
> > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > I think the questions around governance make sense, I
> > think
> > > we
> > > > > > >should
> > > > > > >> > > > really clarify that to make the process more clear so it
> > can
> > > > be
> > > > > > >fully
> > > > > > >> > > > inclusive.
> > > > > > >> > > >
> > > > > > >> > > > The idea that we should not collaborate on what is there
> > > now,
> > > > > > >though,
> > > > > > >> > > > because in the future we might disagree about direction
> > does
> > > > not
> > > > > > >> really
> > > > > > >> > > > make sense to me. If in the future we disagree, that is
> > the
> > > > > beauty
> > > > > > >of
> > > > > > >> > > open
> > > > > > >> > > > source, you can always fork off a copy of the code and
> > start
> > > > an
> > > > > > >> > > independent
> > > > > > >> > > > project either in Apache or elsewhere. Pre-emptively
> > > > re-creating
> > > > > > >> > another
> > > > > > >> > > > REST layer when it seems like we all quite agree on what
> > > needs
> > > > > to
> > > > > > >>be
> > > > > > >> > done
> > > > > > >> > > > and we have an existing code base for HTTP/kafka access
> > that
> > > > is
> > > > > > >> heavily
> > > > > > >> > > > used in production seems quite silly.
> > > > > > >> > > >
> > > > > > >> > > > Let me give some background on how I at least think
> about
> > > > these
> > > > > > >> things.
> > > > > > >> > > > I've participated in open source projects out of
> LinkedIn
> > > via
> > > > > > >>github
> > > > > > >> as
> > > > > > >> > > > well as via the ASF. I don't think there is a "right"
> > answer
> > > > to
> > > > > > >>how
> > > > > > >> to
> > > > > > >> > do
> > > > > > >> > > > these but rather some tradeoffs. We thought about this
> > > quite a
> > > > > lot
> > > > > > >in
> > > > > > >> > the
> > > > > > >> > > > context of Kafka based on the experience with the Hadoop
> > > > > ecosystem
> > > > > > >as
> > > > > > >> > > well
> > > > > > >> > > > as from other open source communities.
> > > > > > >> > > >
> > > > > > >> > > > There is a rich ecosystem around Kafka. Many of the
> > projects
> > > > are
> > > > > > >> quite
> > > > > > >> > > > small--single clients or tools that do a single thing
> > > > well--and
> > > > > > >> almost
> > > > > > >> > > none
> > > > > > >> > > > of them are top level apache projects. I don't think
> > trying
> > > to
> > > > > > >>force
> > > > > > >> > each
> > > > > > >> > > > of these to turn into independent Apache projects is
> > > > necessarily
> > > > > > >>the
> > > > > > >> > best
> > > > > > >> > > > thing for the ecosystem.
> > > > > > >> > > >
> > > > > > >> > > > My observation of how this can go wrong is really what I
> > > think
> > > > > has
> > > > > > >> > > happened
> > > > > > >> > > > to the Hadoop ecosystem. There you see quite a zoo of
> > > projects
> > > > > > >>which
> > > > > > >> > all
> > > > > > >> > > > drift apart and don't quite work together well.
> > Coordinating
> > > > > even
> > > > > > >> > simple
> > > > > > >> > > > changes and standardization across these is
> exceptionally
> > > > > > >>difficult.
> > > > > > >> > The
> > > > > > >> > > > result is a bit of a mess for users--the pieces just
> don't
> > > > > really
> > > > > > >> come
> > > > > > >> > > > together very well. This makes sense for independent
> > > > > > >>infrastructure
> > > > > > >> > > systems
> > > > > > >> > > > (Kudu vs HDFS) but I'm not at all convinced that doing
> > this
> > > > for
> > > > > > >every
> > > > > > >> > > > little tool or helper library has lead to a desirable
> > > state. I
> > > > > > >>think
> > > > > > >> > the
> > > > > > >> > > > mode of operating where the Hadoop vendors spawn off a
> few
> > > new
> > > > > > >Apache
> > > > > > >> > > > projects for each new product initiative, especially
> since
> > > > often
> > > > > > >that
> > > > > > >> > > > project is only valued by that vendor (and the other
> > vendor
> > > > has
> > > > > a
> > > > > > >> > > different
> > > > > > >> > > > competing Apache project) doesn't necessarily do a
> better
> > > job
> > > > at
> > > > > > >> > > producing
> > > > > > >> > > > high quality communities or high quality software.
> > > > > > >> > > >
> > > > > > >> > > > These tools/connects/clients/proxies and other
> integration
> > > > > pieces
> > > > > > >can
> > > > > > >> > > take
> > > > > > >> > > > many forms, but my take of what makes one of these
> things
> > > good
> > > > > is
> > > > > > >> that
> > > > > > >> > it
> > > > > > >> > > > remains simple, does its one thing well, and cleaves as
> > > > closely
> > > > > as
> > > > > > >> > > possible
> > > > > > >> > > > to the conventions for Kafka itself--i.e. doesn't invent
> > new
> > > > > ways
> > > > > > >>of
> > > > > > >> > > > monitoring, configuring, etc. For the tools we've
> > > contributed
> > > > > > >>we've
> > > > > > >> > tried
> > > > > > >> > > > really hard to make them consistent with Kafka as well
> as
> > > with
> > > > > > >>each
> > > > > > >> > other
> > > > > > >> > > > in how testing, configuration, monitoring, etc works.
> > > > > > >> > > >
> > > > > > >> > > > I think what Apache does superbly well is create a
> > community
> > > > for
> > > > > > >> > > managing a
> > > > > > >> > > > large infrastructure layer like Kafka in a vendor
> > > independent
> > > > > way.
> > > > > > >> > What I
> > > > > > >> > > > think is less successful is attempting to form full and
> > > > > > >>independent
> > > > > > >> > > apache
> > > > > > >> > > > communities around very simple single purpose tools,
> > > > especially
> > > > > if
> > > > > > >> you
> > > > > > >> > > hope
> > > > > > >> > > > for these to come together into a cohesive toolset
> across
> > > > > multiple
> > > > > > >> such
> > > > > > >> > > > tools. Much of what Apache does--create a collective
> > > decision
> > > > > > >>making
> > > > > > >> > > > process for resolving disagreement, help to trademark
> and
> > > > > protect
> > > > > > >the
> > > > > > >> > > marks
> > > > > > >> > > > of the project, etc just isn't that relevant for simple
> > > > > > >> single-purpose
> > > > > > >> > > > tools.
> > > > > > >> > > >
> > > > > > >> > > > So my take is there are a couple of options:
> > > > > > >> > > >
> > > > > > >> > > > 1. We can try to put all the small tools into the
> > Apache
> > > > > > >>Project.
> > > > > > >> I
> > > > > > >> > > > think this is not the right approach as there is
> simply
> > > too
> > > > > > >>many
> > > > > > >> of
> > > > > > >> > > > them,
> > > > > > >> > > > many in different languages, serving different
> > protocols,
> > > > > > >> > integrating
> > > > > > >> > > > with
> > > > > > >> > > > particular systems, and a single community can't
> > > > effectively
> > > > > > >> > maintain
> > > > > > >> > > > them
> > > > > > >> > > > all. Doing this would significantly slow the progress
> > of
> > > > the
> > > > > > >Kafka
> > > > > > >> > > > project.
> > > > > > >> > > > As a protocol for messaging, I don't really see a
> case
> > > for
> > > > > > >> including
> > > > > > >> > > > REST
> > > > > > >> > > > but not MQTT or AMQP which are technically much
> better
> > > > suited
> > > > > > >>to
> > > > > > >> > > > messaging
> > > > > > >> > > > and both are widely used for that.
> > > > > > >> > > > 2. We can treat ecosystem projects that aren't top
> > level
> > > > > Apache
> > > > > > >> > > projects
> > > > > > >> > > > as invalid and try to recreate them all as Apache
> > > projects.
> > > > > > >> > Honestly,
> > > > > > >> > > > though, if you go to the Kafka ecosystem page
> virtually
> > > > none
> > > > > of
> > > > > > >> the
> > > > > > >> > > most
> > > > > > >> > > > popular add-ons to Kafka are Apache projects. The
> most
> > > > > > >>successful
> > > > > > >> > > > things in
> > > > > > >> > > > the Kafka ecosystem such as Yahoo Manager,
> librdkafka,
> > a
> > > > > number
> > > > > > >of
> > > > > > >> > > other
> > > > > > >> > > > clients, as well as the existing REST layer have
> > > succeeded
> > > > at
> > > > > > >> > > developing
> > > > > > >> > > > communities that actively contribute and use these
> > pieces
> > > > > and I
> > > > > > >> > don't
> > > > > > >> > > > know
> > > > > > >> > > > that that is a bad thing unless that community proves
> > to
> > > be
> > > > > > >> > > uninclusive,
> > > > > > >> > > > unresponsive, or goes in a bad technical
> direction--and
> > > > those
> > > > > > >>are
> > > > > > >> > > > failure
> > > > > > >> > > > modes that all open source efforts face.
> > > > > > >> > > > 3. We can do what I think makes the most sense and
> try
> > to
> > > > > work
> > > > > > >> with
> > > > > > >> > > the
> > > > > > >> > > > projects that exist in the ecosystem and if the
> project
> > > > > doesn't
> > > > > > >> > have a
> > > > > > >> > > > responsive community or wants to go in a different
> > > > direction
> > > > > > >>fork
> > > > > > >> or
> > > > > > >> > > > recreate that work.
> > > > > > >> > > >
> > > > > > >> > > > Of course any person can choose whatever of these
> options
> > > they
> > > > > > >>want.
> > > > > > >> > But
> > > > > > >> > > > from my point of view, option (3) has been the path of
> the
> > > > > > >>community
> > > > > > >> so
> > > > > > >> > > far
> > > > > > >> > > > and I think it has been quite successful.
> > > > > > >> > > >
> > > > > > >> > > > -Jay
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> > > > > > >> kafka@harsha.io
> > > > > > >> > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Neha,
> > > > > > >> > > > > "But I haven't seen any community emails or patches
> > being
> > > > > > >submitted
> > > > > > >> > by
> > > > > > >> > > > you
> > > > > > >> > > > > guys, so I'm wondering why you are concerned about
> > whether
> > > > the
> > > > > > >> > > community
> > > > > > >> > > > is
> > > > > > >> > > > > open to accepting patches or not."
> > > > > > >> > > > >
> > > > > > >> > > > > I think you are talking about contributing patches to
> > this
> > > > > > >> repository
> > > > > > >> > > > > right? https://github.com/confluentinc/kafka-rest .
> > All I
> > > > am
> > > > > > >> saying
> > > > > > >> > > > > the guidelines/governance model is not clear on the
> > > project
> > > > > and
> > > > > > >>I
> > > > > > >> > guess
> > > > > > >> > > > its
> > > > > > >> > > > > driven by opening a github issue request. Its the
> > > > repository
> > > > > > >owned
> > > > > > >> > by
> > > > > > >> > > > > confluent and as much I appreciate that the features
> we
> > > > > > >>mentioned
> > > > > > >> are
> > > > > > >> > > in
> > > > > > >> > > > > the roadmap and welcoming us to contribute to the
> > project.
> > > > It
> > > > > > >> doesn't
> > > > > > >> > > > > gurantee what we want to add in the furture will be in
> > > your
> > > > > > >> roadmap.
> > > > > > >> > > > >
> > > > > > >> > > > > Hence the reason having it part of Kafka community
> will
> > > > help a
> > > > > > >>lot
> > > > > > >> as
> > > > > > >> > > > other
> > > > > > >> > > > > users can participate in the discussions. We are
> happy
> > to
> > > > > drive
> > > > > > >> any
> > > > > > >> > > > > feature additions through KIPs which gives everyone a
> > > chance
> > > > > to
> > > > > > >> > > > participate
> > > > > > >> > > > > and add to the discussions.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Harsha
> > > > > > >> > > > >
> > > > > > >> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > > > > > >> > Michael.Pearce@ig.com>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > +1
> > > > > > >> > > > > >
> > > > > > >> > > > > > I agree on the governance comments whole heartedly.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Also i agree about the contribution comments made
> > > earlier
> > > > in
> > > > > > >>the
> > > > > > >> > > > thread,
> > > > > > >> > > > > i
> > > > > > >> > > > > > personally am less likely to spend any of mine, or
> > give
> > > > > > >>project
> > > > > > >> > time
> > > > > > >> > > > > within
> > > > > > >> > > > > > my internal projects to developers contributing to
> > > another
> > > > > > >> > commercial
> > > > > > >> > > > > > companies project even so technically open source,
> as
> > > then
> > > > > > >>there
> > > > > > >> is
> > > > > > >> > > > that
> > > > > > >> > > > > > commercial companies interest will always prevail
> and
> > > > > > >essentially
> > > > > > >> > can
> > > > > > >> > > > > > always have a final vote where disagreement. Im sure
> > > they
> > > > > > >>never
> > > > > > >> > > intend
> > > > > > >> > > > > to,
> > > > > > >> > > > > > but there is that true reality. This is why we have
> > > > > community
> > > > > > >> open
> > > > > > >> > > > source
> > > > > > >> > > > > > projects.
> > > > > > >> > > > > >
> > > > > > >> > > > > > I can find many different implementations now of a
> > rest
> > > > > > >>endpoint
> > > > > > >> on
> > > > > > >> > > > > > GitHub, BitBucket etc. Each one has their benefits
> and
> > > > > > >> > disadvantages
> > > > > > >> > > in
> > > > > > >> > > > > > their implementation. By making / providing one this
> > > would
> > > > > > >>bring
> > > > > > >> > > > together
> > > > > > >> > > > > > these solutions, unifying those developers and also
> > > > bringing
> > > > > > >>the
> > > > > > >> > best
> > > > > > >> > > > of
> > > > > > >> > > > > > all.
> > > > > > >> > > > > >
> > > > > > >> > > > > > I understand the concern on the community burden
> > > > > > >> adding/supporting
> > > > > > >> > > more
> > > > > > >> > > > > > surface area for every client. But something like
> REST
> > > is
> > > > > > >> universal
> > > > > > >> > > and
> > > > > > >> > > > > > worthy to be owned by the community.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Mike
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > ________________________________________
> > > > > > >> > > > > > From: Andrew Schofield <andrew_schofield_jira@
> > > outlook.com
> > > > >
> > > > > > >> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > > >> > > > > > To: dev@kafka.apache.org
> > > > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > > >> > > > > >
> > > > > > >> > > > > > There's a massive difference between the governance
> of
> > > > Kafka
> > > > > > >>and
> > > > > > >> > the
> > > > > > >> > > > > > governance of the REST proxy.
> > > > > > >> > > > > >
> > > > > > >> > > > > > In Kafka, there is a broad community of people
> > > > contributing
> > > > > > >their
> > > > > > >> > > > > opinions
> > > > > > >> > > > > > about future enhancements in the form of KIPs.
> There's
> > > > some
> > > > > > >> really
> > > > > > >> > > deep
> > > > > > >> > > > > > consideration that goes into some of the trickier
> > KIPs.
> > > > > There
> > > > > > >are
> > > > > > >> > > > people
> > > > > > >> > > > > > outside Confluent deeply knowledgeable in Kafka and
> > > > > building
> > > > > > >the
> > > > > > >> > > > > > reputations to become committers. I get the
> impression
> > > > that
> > > > > > >>the
> > > > > > >> > > roadmap
> > > > > > >> > > > > of
> > > > > > >> > > > > > Kafka is not really community-owned (what's the big
> > > > feature
> > > > > > >>for
> > > > > > >> > Kafka
> > > > > > >> > > > > 0.11,
> > > > > > >> > > > > > for example), but the conveyor belt of smaller
> > features
> > > in
> > > > > the
> > > > > > >> form
> > > > > > >> > > of
> > > > > > >> > > > > KIPs
> > > > > > >> > > > > > works nicely. It's a good example of open-source
> > > working
> > > > > > >>well.
> > > > > > >> > > > > >
> > > > > > >> > > > > > The equivalent for the REST proxy is basically
> issues
> > on
> > > > > > >>GitHub.
> > > > > > >> > The
> > > > > > >> > > > > > roadmap is less clear. There's not really a
> community
> > > > > properly
> > > > > > >> > > engaged
> > > > > > >> > > > in
> > > > > > >> > > > > > the way that there is with Kafka. So, you could say
> > that
> > > > > it's
> > > > > > >> clear
> > > > > > >> > > > that
> > > > > > >> > > > > > fewer people are interested, but I think the whole
> > > > > governance
> > > > > > >> > thing
> > > > > > >> > > > is a
> > > > > > >> > > > > > big barrier to engagement. And it's looking like
> it's
> > > > > getting
> > > > > > >out
> > > > > > >> > of
> > > > > > >> > > > > date.
> > > > > > >> > > > > >
> > > > > > >> > > > > > In technical terms, I can think of two big
> > improvements
> > > to
> > > > > the
> > > > > > >> REST
> > > > > > >> > > > > proxy.
> > > > > > >> > > > > > First, it needs to use the new consumer API so that
> > it's
> > > > > > >possible
> > > > > > >> > to
> > > > > > >> > > > > secure
> > > > > > >> > > > > > connections between the REST proxy and Kafka. I
> don't
> > > care
> > > > > too
> > > > > > >> much
> > > > > > >> > > > which
> > > > > > >> > > > > > method calls it uses actually uses to consume
> > messages,
> > > > > but I
> > > > > > >do
> > > > > > >> > > care
> > > > > > >> > > > > that
> > > > > > >> > > > > > I cannot secure connections because of network
> > security
> > > > > rules.
> > > > > > >> > > Second,
> > > > > > >> > > > > > there's an affinity between a consumer and the
> > instance
> > > of
> > > > > the
> > > > > > >> REST
> > > > > > >> > > > proxy
> > > > > > >> > > > > > to which it first connected. Kafka itself avoids
> this
> > > kind
> > > > > of
> > > > > > >> > > affinity
> > > > > > >> > > > > for
> > > > > > >> > > > > > good reason, and in the name of availability the
> REST
> > > > proxy
> > > > > > >> should
> > > > > > >> > > too.
> > > > > > >> > > > > > These are natural KIPs.
> > > > > > >> > > > > >
> > > > > > >> > > > > > I think it would be good to have the code for the
> REST
> > > > proxy
> > > > > > >> > > > contributed
> > > > > > >> > > > > > to Apache Kafka so that it would be able to be
> > developed
> > > > in
> > > > > > >>the
> > > > > > >> > same
> > > > > > >> > > > way.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Andrew Schofield
> > > > > > >> > > > > >
> > > > > > >> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > > > >> > > > > > Sent: 07 October 2016 22:41:52
> > > > > > >> > > > > > To: dev@kafka.apache.org
> > > > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > > >> > > > > >
> > > > > > >> > > > > > ASF already gives us a clear framework and
> governance
> > > > model
> > > > > > >>for
> > > > > > >> > > > community
> > > > > > >> > > > > > development. This is already understood by the
> people
> > > > > > >> contributing
> > > > > > >> > to
> > > > > > >> > > > > > Apache Kafka project, and they are the same people
> who
> > > > want
> > > > > to
> > > > > > >> > > > contribute
> > > > > > >> > > > > > to the REST server capability as well. Everyone is
> in
> > > > > > >>agreement
> > > > > > >> on
> > > > > > >> > > the
> > > > > > >> > > > > > need for collaborating on this effort. So why not
> > > > contribute
> > > > > > >>the
> > > > > > >> > code
> > > > > > >> > > > to
> > > > > > >> > > > > > Apache Kafka. This will help avoid duplication of
> > effort
> > > > and
> > > > > > >> forks
> > > > > > >> > > that
> > > > > > >> > > > > > may crop up, hugely benefitting the user community.
> > This
> > > > > will
> > > > > > >> also
> > > > > > >> > > > avoid
> > > > > > >> > > > > > having to define a process similar to ASF on a
> GitHub
> > > > > project
> > > > > > >and
> > > > > > >> > > > instead
> > > > > > >> > > > > > there is a single community with clear understanding
> > > > > community
> > > > > > >> > > process
> > > > > > >> > > > as
> > > > > > >> > > > > > defined in ASF.
> > > > > > >> > > > > >
> > > > > > >> > > > > > As others have said, this is an important capability
> > for
> > > > > > >>Apache
> > > > > > >> > > Kafka.
> > > > > > >> > > > It
> > > > > > >> > > > > > is worth maintaining this as a part of the project.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Regards,
> > > > > > >> > > > > > Suresh
> > > > > > >> > > > > >
> > > > > > >> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <
> > > ofir.manor@equalum.io>
> > > > > > >>wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > >I personally think it would be quite wasteful to
> > > > > re-implement
> > > > > > >> the
> > > > > > >> > > REST
> > > > > > >> > > > > > >gateway just because that an actively-maintained
> > piece
> > > of
> > > > > > >> > > > > Apache-licensed
> > > > > > >> > > > > > >software is not governed directly by the Apache
> Kafka
> > > > > > >> community...
> > > > > > >> > > > While
> > > > > > >> > > > > > >kafka-rest repo is owned by Confluent, the
> > contributors
> > > > > > >> including
> > > > > > >> > > the
> > > > > > >> > > > > main
> > > > > > >> > > > > > >one are also part of the Apache Kafka community,
> so
> > > > there
> > > > > > >>is a
> > > > > > >> > > chance
> > > > > > >> > > > > to
> > > > > > >> > > > > > >work this out.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >However, there are two valid concerns here that
> could
> > > be
> > > > > > >> > addressed,
> > > > > > >> > > > > around
> > > > > > >> > > > > > >community and accessibility:
> > > > > > >> > > > > > >>> What we are worried about is a project
> > > > > > >> > > > > > >>> that's not maintained in a community. So the
> > process
> > > > of
> > > > > > >> > accepting
> > > > > > >> > > > > > >>>patches
> > > > > > >> > > > > > >>> and priorities is not clear, and it's not
> > developed
> > > in
> > > > > > >Apache
> > > > > > >> > > > > > >>>community.
> > > > > > >> > > > > > >>> Not only that, existing REST API project doesn't
> > > > support
> > > > > > >>new
> > > > > > >> > > client
> > > > > > >> > > > > API
> > > > > > >> > > > > > >and
> > > > > > >> > > > > > >>> hence there is no security support either.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >This might be easy to fix. Maybe Confluent /
> > kafka-rest
> > > > > > >> community
> > > > > > >> > > can
> > > > > > >> > > > > > >clarify that - what is their contribution policy,
> dev
> > > > > style,
> > > > > > >> > roadmap
> > > > > > >> > > > > etc.
> > > > > > >> > > > > > >If they want, they can make an effort to encourage
> > > > > > >participation
> > > > > > >> > > from
> > > > > > >> > > > > > >people outside Confluent (easily accept
> > contributions,
> > > > > invite
> > > > > > >> > > external
> > > > > > >> > > > > > >commiters or have open dev process similar to
> Apache
> > > > Kafka
> > > > > > >etc),
> > > > > > >> > as
> > > > > > >> > > > > there
> > > > > > >> > > > > > >is definitely seems to be some interest on the
> list.
> > > That
> > > > > > >>might
> > > > > > >> > > clear
> > > > > > >> > > > > the
> > > > > > >> > > > > > >community concern and help kafka-rest project (but
> > that
> > > > is
> > > > > a
> > > > > > >> > > > calculation
> > > > > > >> > > > > > >Confluent will have to make).
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >The other, independent, concern is that REST is
> > > something
> > > > > > >>that
> > > > > > >> is
> > > > > > >> > > > > expected
> > > > > > >> > > > > > >to be available out of the box with Kafka. I
> > personally
> > > > > don't
> > > > > > >> feel
> > > > > > >> > > > > > >strongly
> > > > > > >> > > > > > >about it (better use proper, efficient APIs from
> day
> > > > one),
> > > > > > >> though
> > > > > > >> > it
> > > > > > >> > > > is
> > > > > > >> > > > > > >definitely way smaller than adding a stream
> > processing
> > > > > engine
> > > > > > >to
> > > > > > >> > the
> > > > > > >> > > > > > >project :)
> > > > > > >> > > > > > >Again,the kafka-rest "community" could take steps
> to
> > > make
> > > > > it
> > > > > > >> even
> > > > > > >> > > > easier
> > > > > > >> > > > > > >to
> > > > > > >> > > > > > >install, configure and run kafka-rest for new users
> > on
> > > > > > >>vanilla
> > > > > > >> > > Apache
> > > > > > >> > > > > > >Kafka
> > > > > > >> > > > > > >(outside the Confluent platform), if they wish that
> > (or
> > > > > > >>welcome
> > > > > > >> > > > > > >contributions to that end), but that is up to them.
> > > > > > >> > > > > > >Finally, if after the above steps were taken there
> > > would
> > > > > > >>still
> > > > > > >a
> > > > > > >> > > > strong
> > > > > > >> > > > > > >desire to include a great rest gateway with Apache
> > > > Kafka, I
> > > > > > >> assume
> > > > > > >> > > the
> > > > > > >> > > > > > >community could hypothetically fork the existing
> > > > kafka-rest
> > > > > > >into
> > > > > > >> > an
> > > > > > >> > > > > Apache
> > > > > > >> > > > > > >Kafka subproject and maintain it "within Apache"
> > > instead
> > > > of
> > > > > > >> > > > implementing
> > > > > > >> > > > > > >it
> > > > > > >> > > > > > >from scratch (though I'm not a lawyer etc) - but I
> > > cannot
> > > > > > >> imagine
> > > > > > >> > it
> > > > > > >> > > > > > >happen
> > > > > > >> > > > > > >without Confluent blessing, and I think that is
> > likely
> > > > much
> > > > > > >less
> > > > > > >> > > > optimal
> > > > > > >> > > > > > >(pulling in other Confluent / Apache licensed
> > > > dependencies)
> > > > > > >than
> > > > > > >> > > > having
> > > > > > >> > > > > a
> > > > > > >> > > > > > >separate external community around kafka-rest.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >Just my two cents,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >Ofir Manor
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >Co-Founder & CTO | Equalum
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > > > > > ><+972%2054-780-1286>
> > > > > > >> > <+972%2054-780-1286> |
> > > > > > >> > > > Email:
> > > > > > >> > > > > > ofir.manor@equalum.io
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha
> Chintalapani
> > <
> > > > > > >> > > kafka@harsha.io
> > > > > > >> > > > >
> > > > > > >> > > > > > >wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> Neha & Jay,
> > > > > > >> > > > > > >> We did look at the open source
> > > > > > >>alternatives.
> > > > > > >> > Our
> > > > > > >> > > > > > >>concern
> > > > > > >> > > > > > >> is what's the patch acceptance and adding
> features/
> > > > > > >>bug-fixes
> > > > > > >> to
> > > > > > >> > > the
> > > > > > >> > > > > > >> existing project under a Github (although it's
> > > licensed
> > > > > > >>under
> > > > > > >> > > Apache
> > > > > > >> > > > > > >>2.0).
> > > > > > >> > > > > > >> It would be great if that project made available
> > > under
> > > > > > >>Apache
> > > > > > >> > and
> > > > > > >> > > > > > >>driven by
> > > > > > >> > > > > > >> the community. Adding to the above, not all
> Kafka
> > > > users
> > > > > > >>are
> > > > > > >> > > > > interested
> > > > > > >> > > > > > >>in
> > > > > > >> > > > > > >> using the Java client API, they would like to
> have
> > > > simple
> > > > > > >REST
> > > > > > >> > API
> > > > > > >> > > > > where
> > > > > > >> > > > > > >> they can code against using any language. I do
> > > believe
> > > > > this
> > > > > > >> adds
> > > > > > >> > > > value
> > > > > > >> > > > > > >>to
> > > > > > >> > > > > > >> Apache Kafka in itself.
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> "For 1, I don't think there is value in giving in
> > to
> > > > the
> > > > > > >>NIH
> > > > > > >> > > > syndrome
> > > > > > >> > > > > > >>and
> > > > > > >> > > > > > >> reinventing the wheel. What I'm looking for is a
> > > > detailed
> > > > > > >> > > comparison
> > > > > > >> > > > > of
> > > > > > >> > > > > > >>the
> > > > > > >> > > > > > >> gaps and why those can't be improved in the REST
> > > proxy
> > > > > that
> > > > > > >> > > already
> > > > > > >> > > > > > >>exists
> > > > > > >> > > > > > >> and is actively maintained."
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> We are not looking at this as NIH. What we are
> > > worried
> > > > > > >>about
> > > > > > >> > is a
> > > > > > >> > > > > > >>project
> > > > > > >> > > > > > >> that's not maintained in a community. So the
> > process
> > > of
> > > > > > >> > accepting
> > > > > > >> > > > > > >>patches
> > > > > > >> > > > > > >> and priorities is not clear, and it's not
> developed
> > > in
> > > > > > >>Apache
> > > > > > >> > > > > community.
> > > > > > >> > > > > > >> Not only that, existing REST API project doesn't
> > > > support
> > > > > > >>new
> > > > > > >> > > client
> > > > > > >> > > > > API
> > > > > > >> > > > > > >>and
> > > > > > >> > > > > > >> hence there is no security support either.
> > > > > > >> > > > > > >> We don't know the timeline when that's made
> > > available.
> > > > We
> > > > > > >> would
> > > > > > >> > > like
> > > > > > >> > > > > to
> > > > > > >> > > > > > >>add
> > > > > > >> > > > > > >> admin functionality into the REST API. So the
> > Roadmap
> > > > of
> > > > > > >>that
> > > > > > >> > > > project
> > > > > > >> > > > > is
> > > > > > >> > > > > > >> not driven by Apache.
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> "This doesn't materially have an impact on
> > expanding
> > > > the
> > > > > > >> > usability
> > > > > > >> > > > of
> > > > > > >> > > > > > >> Kafka. In my experience, REST proxy + Java
> > clients
> > > > > only
> > > > > > >> cover
> > > > > > >> > > > ~50%
> > > > > > >> > > > > of
> > > > > > >> > > > > > >> all
> > > > > > >> > > > > > >> Kafka users, and maybe 10% of those are the
> ones
> > > who
> > > > > > >>will
> > > > > > >> use
> > > > > > >> > > the
> > > > > > >> > > > > > >>REST
> > > > > > >> > > > > > >> proxy. The remaining 50% are non-java client
> > users
> > > > (C,
> > > > > > >> > python,
> > > > > > >> > > > go,
> > > > > > >> > > > > > >>node
> > > > > > >> > > > > > >> etc)."
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> REST API is most often asked feature in my
> > > interactions
> > > > > > >>with
> > > > > > >> > Kafka
> > > > > > >> > > > > > >>users.
> > > > > > >> > > > > > >> In an organization, There will be independent
> teams
> > > who
> > > > > > >>will
> > > > > > >> > write
> > > > > > >> > > > > their
> > > > > > >> > > > > > >> Kafka clients using different language libraries
> > > > > available
> > > > > > >> > today,
> > > > > > >> > > > and
> > > > > > >> > > > > > >> there is no way to standardize this. Instead of
> > > > > supporting
> > > > > > >> > several
> > > > > > >> > > > > > >> different client libraries users will be
> interested
> > > in
> > > > > > >>using
> > > > > > >a
> > > > > > >> > > REST
> > > > > > >> > > > > API
> > > > > > >> > > > > > >> server. The need for a REST API will only
> increase
> > as
> > > > > more
> > > > > > >and
> > > > > > >> > > more
> > > > > > >> > > > > > >>users
> > > > > > >> > > > > > >> start using Kafka.
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> "More surface area means more work to keep things
> > > > > > >>consistent.
> > > > > > >> > > > Failure
> > > > > > >> > > > > > >> to do that has, in fact, hurt the user
> > > experience."
> > > > > > >> > > > > > >> Having myriad Kafka client GitHub projects that
> > > support
> > > > > > >> > different
> > > > > > >> > > > > > >>languages
> > > > > > >> > > > > > >> hurts the user experience and pushes burden to
> > > maintain
> > > > > > >>these
> > > > > > >> > > > > libraries.
> > > > > > >> > > > > > >> REST API is a simple code base that uses existing
> > > java
> > > > > > >>client
> > > > > > >> > > > > libraries
> > > > > > >> > > > > > >>to
> > > > > > >> > > > > > >> make life easier for the users.
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> Thanks,
> > > > > > >> > > > > > >> Harsha
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > > > > > >> > neha@confluent.io>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> > Manikumar,
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > Thanks for sharing the proposal. I think there
> > are
> > > 2
> > > > > > >>parts
> > > > > > >> to
> > > > > > >> > > this
> > > > > > >> > > > > > >> > discussion -
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > 1. Should we rewrite a REST proxy given that
> > there
> > > > is a
> > > > > > >> > > > > > >>feature-complete,
> > > > > > >> > > > > > >> > open-source and actively maintained REST proxy
> in
> > > the
> > > > > > >> > community?
> > > > > > >> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka
> make
> > us
> > > > > more
> > > > > > >> agile
> > > > > > >> > > and
> > > > > > >> > > > > > >> maintain
> > > > > > >> > > > > > >> > the high-quality experience that Kafka users
> have
> > > > > today?
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > For 1, I don't think there is value in giving
> in
> > to
> > > > the
> > > > > > >>NIH
> > > > > > >> > > > syndrome
> > > > > > >> > > > > > >>and
> > > > > > >> > > > > > >> > reinventing the wheel. What I'm looking for is
> a
> > > > > detailed
> > > > > > >> > > > comparison
> > > > > > >> > > > > > >>of
> > > > > > >> > > > > > >> the
> > > > > > >> > > > > > >> > gaps and why those can't be improved in the
> REST
> > > > proxy
> > > > > > >>that
> > > > > > >> > > > already
> > > > > > >> > > > > > >> exists
> > > > > > >> > > > > > >> > and is actively maintained. For example, we
> > depend
> > > on
> > > > > > >> zkClient
> > > > > > >> > > and
> > > > > > >> > > > > > >>have
> > > > > > >> > > > > > >> > found as well as fixed several bugs by working
> > > > closely
> > > > > > >>with
> > > > > > >> > the
> > > > > > >> > > > > people
> > > > > > >> > > > > > >> who
> > > > > > >> > > > > > >> > maintain zkClient. This should be possible for
> > REST
> > > > > proxy
> > > > > > >as
> > > > > > >> > > well,
> > > > > > >> > > > > > >>right?
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > For 2, I'd like us to review our history of
> > > expanding
> > > > > the
> > > > > > >> > > surface
> > > > > > >> > > > > > >>area to
> > > > > > >> > > > > > >> > add more clients in the past. Here is a
> summary -
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > - This doesn't materially have an impact on
> > > > > expanding
> > > > > > >the
> > > > > > >> > > > > > >>usability of
> > > > > > >> > > > > > >> > Kafka. In my experience, REST proxy + Java
> > > clients
> > > > > > >>only
> > > > > > >> > cover
> > > > > > >> > > > > ~50%
> > > > > > >> > > > > > >>of
> > > > > > >> > > > > > >> > all
> > > > > > >> > > > > > >> > Kafka users, and maybe 10% of those are the
> > ones
> > > > who
> > > > > > >will
> > > > > > >> > use
> > > > > > >> > > > the
> > > > > > >> > > > > > >>REST
> > > > > > >> > > > > > >> > proxy. The remaining 50% are non-java client
> > > users
> > > > > (C,
> > > > > > >> > > python,
> > > > > > >> > > > > go,
> > > > > > >> > > > > > >> node
> > > > > > >> > > > > > >> > etc).
> > > > > > >> > > > > > >> > - People are a lot more excited about
> > promising
> > > to
> > > > > > >> > contribute
> > > > > > >> > > > > while
> > > > > > >> > > > > > >> > adding the surface area but not on an
> ongoing
> > > > basis
> > > > > > >>down
> > > > > > >> > the
> > > > > > >> > > > > line.
> > > > > > >> > > > > > >> > - More surface area means more work to keep
> > > things
> > > > > > >> > > consistent.
> > > > > > >> > > > > > >>Failure
> > > > > > >> > > > > > >> > to do that has, in fact, hurt the user
> > > experience.
> > > > > > >> > > > > > >> > - More surface area hurts agility. We want
> to
> > > do a
> > > > > few
> > > > > > >> > things
> > > > > > >> > > > > > >>really
> > > > > > >> > > > > > >> > well as well as be agile to be able to build
> > on
> > > > our
> > > > > > >>core
> > > > > > >> > > > > > >>competency.
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > > > > >> > > > manikumar.reddy@gmail.com
> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > wrote:
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > > Hi Jay,
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > Thanks for your reply.
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > I agree that we can not add all the
> > clients/tools
> > > > > > >> available
> > > > > > >> > in
> > > > > > >> > > > > > >> ecosystem
> > > > > > >> > > > > > >> > > page to Kafka repo itself. But we feel REST
> > > > Interface
> > > > > > >>is
> > > > > > >> > > > different
> > > > > > >> > > > > > >>from
> > > > > > >> > > > > > >> > > other clients/tools. Since any language that
> > can
> > > > work
> > > > > > >with
> > > > > > >> > > HTTP
> > > > > > >> > > > > can
> > > > > > >> > > > > > >> > > easily integrate with this interface, Having
> an
> > > > > > >"official"
> > > > > > >> > > REST
> > > > > > >> > > > > > >> > > interface helps user community. This also
> helps
> > > us
> > > > to
> > > > > > >> > > integrate
> > > > > > >> > > > > well
> > > > > > >> > > > > > >> > > with external management and provisioning
> > tools.
> > > > > > >>Apache
> > > > > > >> > Kafka
> > > > > > >> > > > > > >>release
> > > > > > >> > > > > > >> > > with Java clients + REST interface is
> > sufficient
> > > > for
> > > > > > >>most
> > > > > > >> of
> > > > > > >> > > the
> > > > > > >> > > > > > >>user
> > > > > > >> > > > > > >> > > deployments/requirements. This helps users to
> > > deal
> > > > > with
> > > > > > >> less
> > > > > > >> > > > > number
> > > > > > >> > > > > > >> > > of distributions/builds.
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > Thanks,
> > > > > > >> > > > > > >> > > Manikumar
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> > > > > > >> jay@confluent.io
> > > > > > >> > >
> > > > > > >> > > > > wrote:
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > > Hey guys,
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > > > There's already a REST interface maintained
> > as
> > > a
> > > > > > >> separate
> > > > > > >> > > > > > >> project--it's
> > > > > > >> > > > > > >> > > > open source and apache licensed and
> actively
> > > > > > >>maintained
> > > > > > >> (
> > > > > > >> > > > > > >> > > > https://github.com/confluentinc/kafka-rest
> ).
> > > > What
> > > > > is
> > > > > > >> > wrong
> > > > > > >> > > > with
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for
> Kafka
> > > > > > >> > > > > > github.com
> > > > > > >> > > > > > The Kafka REST Proxy provides a RESTful interface
> to a
> > > > Kafka
> > > > > > >> > cluster.
> > > > > > >> > > > It
> > > > > > >> > > > > > makes it easy to produce and consume messages, view
> > the
> > > > > state
> > > > > > >>of
> > > > > > >> > the
> > > > > > >> > > > > > cluster, and ...
> > > > > > >> > > > > >
> > > > > > >> > > > > > >> that?
> > > > > > >> > > > > > >> > > You
> > > > > > >> > > > > > >> > > > mentioned that there was some compatibility
> > > > > concern,
> > > > > > >but
> > > > > > >> > > > > > >> compatibility
> > > > > > >> > > > > > >> > > has
> > > > > > >> > > > > > >> > > > to do with the consumer protocol guarantees
> > not
> > > > the
> > > > > > >repo
> > > > > > >> > the
> > > > > > >> > > > > code
> > > > > > >> > > > > > >>is
> > > > > > >> > > > > > >> > in,
> > > > > > >> > > > > > >> > > > right? Not sure that concern makes sense.
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > > > We could argue for adding pretty much
> > anything
> > > > and
> > > > > > >> > > everything
> > > > > > >> > > > in
> > > > > > >> > > > > > >>the
> > > > > > >> > > > > > >> > > > ecosystem page in Kafka itself but I'm not
> > sure
> > > > > that
> > > > > > >> would
> > > > > > >> > > > make
> > > > > > >> > > > > > >>the
> > > > > > >> > > > > > >> > > project
> > > > > > >> > > > > > >> > > > more agile.
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > > > -Jay
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM,
> Manikumar <
> > > > > > >> > > > > > >> manikumar.reddy@gmail.com
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > > wrote:
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > > > > Hi Kafka Devs,
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server
> > to
> > > > > Kafka
> > > > > > >> > > > Repository.
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > There are already open-source
> alternatives
> > > are
> > > > > > >> > available.
> > > > > > >> > > > But
> > > > > > >> > > > > > >>we
> > > > > > >> > > > > > >> > would
> > > > > > >> > > > > > >> > > > > like to add REST server that
> > > > > > >> > > > > > >> > > > > many users ask for under Apache Kafka
> repo.
> > > > Many
> > > > > > >>data
> > > > > > >> > > Infra
> > > > > > >> > > > > > >>tools
> > > > > > >> > > > > > >> > comes
> > > > > > >> > > > > > >> > > > up
> > > > > > >> > > > > > >> > > > > with Rest Interface.
> > > > > > >> > > > > > >> > > > > It is useful to have inbuilt Rest API
> > support
> > > > for
> > > > > > >> > Produce,
> > > > > > >> > > > > > >>Consume
> > > > > > >> > > > > > >> > > > messages
> > > > > > >> > > > > > >> > > > > and admin interface for
> > > > > > >> > > > > > >> > > > > integrating with external management and
> > > > > > >>provisioning
> > > > > > >> > > > > tools.This
> > > > > > >> > > > > > >> will
> > > > > > >> > > > > > >> > > > also
> > > > > > >> > > > > > >> > > > > allow the maintenance of
> > > > > > >> > > > > > >> > > > > REST server and adding new features makes
> > it
> > > > easy
> > > > > > >> > because
> > > > > > >> > > > > apache
> > > > > > >> > > > > > >> > > > community.
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > The KIP wiki is the following:
> > > > > > >> > > > > > >> > > > > https://cwiki.apache.org/
> > > > > > >> confluence/display/KAFKA/KIP-
> > > > > > >> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > Your comments and feedback are welcome.
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > > >> > > > > Manikumar
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > --
> > > > > > >> > > > > > >> > Thanks,
> > > > > > >> > > > > > >> > Neha
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > The information contained in this email is strictly
> > > > > > >>confidential
> > > > > > >> > and
> > > > > > >> > > > for
> > > > > > >> > > > > > the use of the addressee only, unless otherwise
> > > indicated.
> > > > > If
> > > > > > >you
> > > > > > >> > are
> > > > > > >> > > > not
> > > > > > >> > > > > > the intended recipient, please do not read, copy,
> use
> > or
> > > > > > >disclose
> > > > > > >> > to
> > > > > > >> > > > > others
> > > > > > >> > > > > > this message or any attachment. Please also notify
> the
> > > > > sender
> > > > > > >>by
> > > > > > >> > > > replying
> > > > > > >> > > > > > to this email or by telephone (+44(020 7896 0011)
> and
> > > then
> > > > > > >delete
> > > > > > >> > the
> > > > > > >> > > > > email
> > > > > > >> > > > > > and any copies of it. Opinions, conclusion (etc)
> that
> > do
> > > > not
> > > > > > >> relate
> > > > > > >> > > to
> > > > > > >> > > > > the
> > > > > > >> > > > > > official business of this company shall be
> understood
> > as
> > > > > > >>neither
> > > > > > >> > > given
> > > > > > >> > > > > nor
> > > > > > >> > > > > > endorsed by it. IG is a trading name of IG Markets
> > > Limited
> > > > > (a
> > > > > > >> > company
> > > > > > >> > > > > > registered in England and Wales, company number
> > > 04008957)
> > > > > and
> > > > > > >>IG
> > > > > > >> > > Index
> > > > > > >> > > > > > Limited (a company registered in England and Wales,
> > > > company
> > > > > > >> number
> > > > > > >> > > > > > 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > > > > > >>Dowgate
> > > > > > >> > > Hill,
> > > > > > >> > > > > > London EC4R 2YA. Both IG Markets Limited (register
> > > number
> > > > > > >195355)
> > > > > > >> > and
> > > > > > >> > > > IG
> > > > > > >> > > > > > Index Limited (register number 114059) are
> authorised
> > > and
> > > > > > >> regulated
> > > > > > >> > > by
> > > > > > >> > > > > the
> > > > > > >> > > > > > Financial Conduct Authority.
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > > The information contained in this email is strictly
> > > confidential
> > > > > and
> > > > > > >> for
> > > > > > >> > > the use of the addressee only, unless otherwise indicated.
> > If
> > > > you
> > > > > > >>are
> > > > > > >> not
> > > > > > >> > > the intended recipient, please do not read, copy, use or
> > > > disclose
> > > > > to
> > > > > > >> > others
> > > > > > >> > > this message or any attachment. Please also notify the
> > sender
> > > by
> > > > > > >> replying
> > > > > > >> > > to this email or by telephone (+44(020 7896 0011) and then
> > > > delete
> > > > > > >>the
> > > > > > >> > email
> > > > > > >> > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > > > relate
> > > > > > >>to
> > > > > > >> > the
> > > > > > >> > > official business of this company shall be understood as
> > > neither
> > > > > > >>given
> > > > > > >> > nor
> > > > > > >> > > endorsed by it. IG is a trading name of IG Markets Limited
> > (a
> > > > > > >>company
> > > > > > >> > > registered in England and Wales, company number 04008957)
> > and
> > > IG
> > > > > > >>Index
> > > > > > >> > > Limited (a company registered in England and Wales,
> company
> > > > number
> > > > > > >> > > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > > > > >>Hill,
> > > > > > >> > > London EC4R 2YA. Both IG Markets Limited (register number
> > > > 195355)
> > > > > > >>and
> > > > > > >> IG
> > > > > > >> > > Index Limited (register number 114059) are authorised and
> > > > > regulated
> > > > > > >>by
> > > > > > >> > the
> > > > > > >> > > Financial Conduct Authority.
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Nacho (Ignacio) Solis
> > > > > > >> Kafka
> > > > > > >> nsolis@linkedin.com
> > > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > > > This email, including attachments, is private and confidential. If
> you
> > > have
> > > > received this email in error please notify the sender and delete it
> > from
> > > > your system. Emails are not secure and may contain viruses. No
> > liability
> > > > can be accepted for viruses that might be transferred by this email
> or
> > > any
> > > > attachment. Any unauthorised copying of this message or unauthorised
> > > > distribution and publication of the information contained herein are
> > > > prohibited.
> > > >
> > > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A
> 2BB.
> > > > Registered in England and Wales. Registered No. 04843573.
> > > >
> > >
> >
> > --
> >
> >
> > This email, including attachments, is private and confidential. If you
> have
> > received this email in error please notify the sender and delete it from
> > your system. Emails are not secure and may contain viruses. No liability
> > can be accepted for viruses that might be transferred by this email or
> any
> > attachment. Any unauthorised copying of this message or unauthorised
> > distribution and publication of the information contained herein are
> > prohibited.
> >
> > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> > Registered in England and Wales. Registered No. 04843573.
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
-- Guozhang
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Sorry that I've stayed quiet on this for a bit. My reason for doing so is
well summed up by Jason's notes.
First, I do want to say thanks for referencing the Confluent REST Proxy in
the proposal! It makes me proud that it's being used as an important
reference point.
Second, Nacho, I want to say thanks to you because as I read this thread, I
think you have brought the most nuanced, gray-area view of this, which is
where it really lies. It *is* fuzzy, and an ongoing discussion. See my
answer (somewhere below) re: why I would consider Streams & Connect
different than a REST proxy (key word being proxy...). My taste definitely
skews towards yours wrt keeping inclusion of code in the core project
minimal. Jay referenced this a bit as well and I think it's partly a
comfort with the lots-of-small-projects style of open source.
I want to address a few different areas of concern: the motivation from the
proposal, a few general observations re: inclusion, and more concrete notes
on the details of the proposal. email, unfortunately, is not ideal for
this, but hopefully this won't be too much for one email...
---
On the motivation section, I want to address the 3 key points:
> 1) Many data Infra tools comes up with Rest Interface. It is useful to
have inbuilt Rest API support for Produce, Consume messages and admin
interface for integrating with external management and provisioning tools.
This doesn't explain why including a REST proxy is a good thing (also, REST
proxy and REST interface are different things). Just because many tools do
this doesn't mean its the best choice for Kafka. Presumably there are
reasons they choose to, some of why might apply to Kafka, some of which may
not. It would be helpful to explicitly enumerate these.
> 2) Shipping Kafka Rest Server as part of a normal Kafka release makes it
immediately available to every user that downloads Kafka.
I would argue this is a bad thing. There are historical reasons why it was
challenging to write native clients for each language. It's still
non-trivial (see also: Confluent's focus on wrappers of librdkafka rather
than writing clients from scratch). But I personally view the REST proxy as
a stop gap for when there isn't a good "real" client (at least from the
perspective of most users). While it is sometimes a necessity given other
decisions (I wouldn't expect there to be a high quality Haskell client, if
that's your language of choice), I also don't think using HTTP here should
be encouraged by default. It:
a) is less efficient (see: encoding overhead, 2x bandwidth overhead,
translation overhead, reliance on each client to do good batching)
b) has a pretty significant protocol-level impedance mismatch (see:
consumers aren't really RESTful)
c) still requires extra wrappers (in practice, nobody wants to write a
substantial amount of code without a language-specific API)
and probably more drawbacks that I am not thinking of off the top of my
head.
There is *always* a tradeoff between a low-cost, immediately-available
solution and the "right" solution. But I'm not convinced we want to
showcase what I'd consider a suboptimal (but workable) solution as part of
the core Kafka offerinag. (Which is why, despite being the original author
of Confluent's REST proxy, I invest a bunch of my time working on our
effort to build better native clients and would love to see REST proxy
usage dwindle, despite it being an important part of Confluent's open
source offering).
I specifically want to call out a comment from earlier in the thread:
> Adding to the above, not all Kafka users are interested in using the Java
client API, they would like to have simple REST API where they can code
against using any language. I do believe this adds value to Apache Kafka in
itself.
This incorrectly conflates two things that I think are relevant here. The
first is that not everyone wants to use the Java API. That's true. The
second is that they want a REST API. That's true for some folks, but really
it's just a fallback. I think you'd be hard pressed to find anyone who
would prefer just a REST API to a native library in their language that
they could just install and use naturally.
> 3) Helps to maintain the version compatibility between Kafka and Rest
Server.
The design in the doc never explains how this is true and as far as I can
tell it wouldn't be. KIP-35 support in the java clients can help fix this,
but that doesn't have to do with the REST proxy. As it stands, it is
probably *more* painful to do this as part of Kafka (as we have to deal
with new build layout organization and some internal/some external
dependencies on Kafka clients since you presumably want the REST proxy to
depend on an older client version, i.e. lots of extra build system
gymnastics that are avoided if it is maintained separately).
The one thing I case I can think of where this might be true is that we
already have upgrade system tests for Kafka itself and compatibility tests
with different versions of clients in Kafka today. We could probably get
compatibility tests between different versions of a REST Proxy/Kafka pretty
easily as well. This is true outside of Apache Kafka too, so this doesn't
seem to be a strong argument.
---
A few general observations re: inclusion of a REST proxy:
* Streams and Connect are different clients, but not different protocols. I
think this is an important differentiation. They offer different
abstractions for working with Kafka while retaining all the benefits of
actually working *directly* with Kafka. They try to enhance/simplify the
"low-level" clients, but don't fundamentally change the interaction with
the brokers. All the proxy layers people request are more of like "I like
Kafka but would really prefer if it had this different interface". (I would
love to see more innovation of the type "adds an innovative new API for
interacting with Kafka but can be mapped to the normal Kafka protocol
without significant loss due to impedance mismatch" and with the right APIs
I'd easily be supportive of including them in Kafka).
* Other protocols are far more prevalent (and efficient) with traditional
messaging systems. Why an HTTP interface first? It certainly covers a broad
set of languages, but is it the best type of proxy to provide? And if there
are different tradeoffs, why only one? Why not an MQTT proxy? An AMQP proxy?
* I think admin APIs are one of the few subsets of the functionality where
the current state of affairs (CLI tools or protocol with custom
serialization as the only public API) is frustrating and a REST API
probably *does* work pretty well -- they aren't bandwidth intensive or
anything, a REST API would be easier to work with than the low-level
interface without having to write support for a bunch of different
languages. That said, there are still some tradeoffs here. One example that
springs to mind is security -- you need to figure out how to forward all
sorts of security credentials and probably make a new KIP-4 AdminClient per
request.
----
And a few notes on the proposal:
1. The proposed consumer API: this, along with much of the proposal,
mirrors the current implementation of Confluent's kafka-rest project. I
agree with many design points, but the current consumer API was designed
around the old consumer. We've already started work on replacing it with
one better suited to the (quite different) new consumer -- see
https://github.com/confluentinc/kafka-rest/wiki/New-Consumer-API-Design for
the very basics of the API. Given that the new consumer is the long term
commitment for the project, I'd like to see something designed for it
instead.
2. Scope: Are we limiting to normal HTTP requests? How about Websockets or
HTTP/2 PUSH? The latter actually fit with the consumer model much better.
Or how about persistent connections that push data, ala streaming APIs like
Twitter's?
You'll get these questions as soon as we make a basic REST API available (I
know because we've already seen all of these...). Of course the position of
the project could change over time, but I think this is important because
its the difference between committing to maintaining a relatively small API
to provide reasonable, usable access to users without good clients and
providing a full-fledged, feature complete client supporting all types of
streaming technologies that work over HTTP are *very* different goals with
*very* different levels of effort required.
The scope, both in terms of initial implementation cost and ongoing
maintenance, grow *substantially* larger if you're trying to provide the
latter. The community should understand what they are signing up for.
3. Blessed serialization formats. This proposal is supporting binary and
JSON. I think people tend to make things overly extensible by default in
support of unknown future requirements, but while Confluent could make a
decision to support binary & Avro, then eventually add JSON and maybe add
support for other serialization formats (which we're willing to do! PRs
welcome!), putting something into AK will have different requirements. Why
not Thrift? Why not protobufs? Why not msgpack?
By the way, we also took this into account with Kafka Connect (nee Copycat)
when submitting it for inclusion in Apache Kafka. We spent quite a bit of
time figuring out how to make things work well with pluggable serialization
-- it didn't start out this way when we were unsure where we thought it
would be best for Connect to live. It'll make the proposal larger, but I'd
question a proposal that blesses certain formats over others given that
Kafka is otherwise completely agnostic to format.
4. Related to that, there are some known gaps that don't seem to be
addressed here that, with a small amount of GH issue digging, you'd
probably find easily. For example, combining simple serializers for keys
and "complex" serializers for values. This requires some sort of complex
Content-Type and/or Jersey annotation/parsing magic if you go the Jersey
route. This is not infrequently requested (e.g. string keys, Avro value)
and it'd be good to plan for a design that allows it.
The particular issues aren't the point, though. It concerns me is lessons
learned, especially re: customer needs, are not being addressed either as
future work (with reasonable compatibility suggestion) or argued as out of
scope.
5. The multi-REST-proxy story is very hand wavy. This is more complicated
than it appears at first given a lot of different ways people deploy and do
service discovery. If we're committing to a particular approach (especially
re: consumers), we should actually explore the options and constraints
here. (Confluent is happy to help out in this regard, we explored lots of
options when coming up with our REST proxy design and have learned plenty
since that first iteration!)
6. Simple consumer mode. We added this to Confluent's REST proxy (actually
it was a nice, big, 3rd party addition! Open source works even outside of
Apache!). Where does this fit in the APIs? Are we leaving it to future
KIPs? If so, we should at least have an idea of where it could fit.
7. The APIs/URLs provided are a bit confusing since they seem to be a
subset of what Confluent's proxy provides, yet based on it. In particular,
the way topics/partitions are organized is confusing if you only have 2
produce APIs for them without any support for getting metadata about them.
At a minimum, this should be documented as leaving room for future API
growth.
8. The proposal includes lots of the 5-digit error codes that Confluent's
proxy uses. Confluent's proxy assumes a standardized JSON error response
body with a message and extended error code since HTTP status codes are
quite limiting. Where is this in the proposal?
9. There were some choices (e.g. around offset commit) that Confluent made
to keep the API simple at the cost of flexibility for REST users. Are these
preserved because you think they are the right choice? For example, why
can't the user include the (topic-partition, offset)s to commit?
10. This is really more of a general comment on KIPs, but almost all KIPs
have far too few rejected alternatives, including this one. KIPs should be
an exploration of the design space that help us arrive on the solution we
think is closest to optimal given the information at the time. Is it really
possible for us, as a community, to just accept that most of what ended up
in the Confluent REST proxy as the ideal? I certainly don't trust myself
that much. As a really small example: I remember having quite a bit of
discussion with Neha around how to express "embedded" serialization formats
(e.g. Content-Type vs query parameter vs in URL). I can't believe we've
explored the options in this KIP thoroughly if there are only 2 rejected
alternative points and one of them is just about whether it lives inside or
outside of Kafka...
-Ewen
On Tue, Oct 25, 2016 at 2:41 AM, Ben Davison <be...@7digital.com>
wrote:
> Good point Ismael :D
>
> On Mon, Oct 24, 2016 at 11:07 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > If we want to start a vote on this, I suggest starting a [VOTE] thread
> > instead of mentioning it in the middle of a very long [DISCUSS] thread.
> :)
> >
> > Ismael
> >
> > On Mon, Oct 24, 2016 at 9:46 PM, Ben Davison <be...@7digital.com>
> > wrote:
> >
> > > This KIP has got muddy on differing opinions of what should be part of
> > > Kafka etc. I agree with Suresh, let's have a vote on if we actually
> want
> > a
> > > REST client in Kafka core, and then we can work out what it actually
> > looks
> > > like (personally I would like to reuse Confluents, great REST client if
> > > they donate it to ASF)
> > >
> > > For me +1 on REST client, this is a fundamental feature that's missing
> > from
> > > Kafka.
> > >
> > > On Mon, Oct 24, 2016 at 9:22 PM, Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > Hey Suresh,
> > > >
> > > > I think we agree that REST apis are useful. I don't think we agree
> that
> > > > they need to be part of the Kafka project. We've had this discussion
> a
> > > > dozen odd times for different protocols---AMQP, REST, MQTT. Going
> back
> > > the
> > > > last five years we've always rejected these. That doesn't mean they
> > > aren't
> > > > useful, I think having these as separate is fine, they don't have any
> > > > complex interaction with Kafka, they just use the vanilla APIs like
> any
> > > of
> > > > the dozens of things on the ecosystem page.
> > > >
> > > > In terms of how they're developed. I think there are two discussions
> > > here:
> > > > 1. Separate project or not
> > > > 2. Standalone Apache project or github
> > > >
> > > > The first of those I just talked about---I think this makes sense as
> an
> > > > independent project.
> > > >
> > > > For the second of these, I actually don't think that spawning off a
> > bunch
> > > > of itty-bitty independent Apache projects is a good thing. I think
> you
> > > can
> > > > see this in the Hadoop ecosystem where the parts kind of all evolve
> off
> > > in
> > > > different directions and don't really work together as a whole. I
> think
> > > > that makes sense for deep infrastructure projects, but not for these
> > > little
> > > > helper projects. I think a hub and spoke model where you have a
> central
> > > > project (Kafka) and then a bunch of helper tools that strive to fit
> in
> > > with
> > > > it (in terms of config, monitoring, apis, and every other
> conventions)
> > is
> > > > actually much better. In any case there are already so many of these
> > > tools
> > > > (we capture maybe 20% of them on the ecosystem page), made by so many
> > > > people, and virtually all developed in this style, it is a bit late
> to
> > > put
> > > > the cat back in the bag.
> > > >
> > > > Perhaps it's just a difference in background. For my part I had many
> > > years
> > > > successfully running github-style projects, and i think that model
> can
> > > work
> > > > quite well for small things. I do think it is important for these
> > > projects
> > > > to clarify governance, which we should absolutely do, but
> realistically
> > > it
> > > > is a pretty simple tool, there isn't a huge governance challenge for
> > > > something like this because its scope is so limited ("take http
> > requests,
> > > > make Kafka requests").
> > > >
> > > > More specifically, I don't think there is an actual problem being
> > solved
> > > > here. I haven't heard any complaint about direction or patches not
> > > getting
> > > > added. The only complaint I've heard is missing features where the
> > normal
> > > > "patches accepted" rule applies. I would urge people to actually get
> > > > involved in contribution here. In the future if there is a situation
> > > where
> > > > people don't like the direction of a given tool, they can fork it and
> > > > either turn it into an independent Apache project or develop it
> > > > independently, trying to do that preemptively seems a bit hostile.
> > > >
> > > > -Jay
> > > >
> > > > On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas <
> > > suresh@hortonworks.com>
> > > > wrote:
> > > >
> > > > > I am dividing this discussion into two parts:
> > > > > 1. REST APIs as core Apache Kafka capability
> > > > > This should be a core Kafka functionality. Same view has been
> > reflected
> > > > by
> > > > > others (users and developers) as well. While we can debate whether
> > > other
> > > > > capabilities are core Kafka (Streams, Connect), it would be good
> > > separate
> > > > > that out and to keep this discussion focussed on REST APIs as
> > proposed
> > > by
> > > > > this KIP. If there is ambivalence about the need for this in core
> > > kafka,
> > > > > we could have voting in the mailing list.
> > > > >
> > > > > Can we get an agreement on this? I am +1 on REST API in Apache
> Kafka.
> > > > >
> > > > > 2. Community where Kafka REST APIs need to be collaborated on
> > > > > There is already a GitHub project that caters to Kafka REST APIs.
> > But a
> > > > > company owned Github is less than ideal for collaboration due to
> lack
> > > of
> > > > > governance, open community and roadmap. This is reflected by many
> > > people
> > > > > interested in this functionality and still not contributing to and
> > > > > adopting the code base in the GitHub. I think trying overlay the
> ASF
> > > > > governance model on GitHub project, which is what the need is,
> seems
> > > > > unnecessary, if the code can be part of Apache Kafka.
> > > > >
> > > > > The question is, would Confluent be okay with
> licensing/contributing
> > > the
> > > > > code to Kafka project (assuming either Confluent or another
> > contributor
> > > > > can work on it)? I see clear intent in collaborating with others on
> > > REST
> > > > > APIs from confluent. Why not do it in Kafka project under ASF? This
> > > takes
> > > > > away all the barrier to collaboration? Alternatively, if Confluent
> is
> > > not
> > > > > willing to contribute the code from GitHub, would anyone veto
> > building
> > > a
> > > > > separate REST API functionality in ASF Kafka? There are enough
> people
> > > who
> > > > > want to work on this and maintain it.
> > > > >
> > > > > Regards,
> > > > > Suresh
> > > > >
> > > > >
> > > > >
> > > > > On 10/21/16, 9:41 PM, "Harsha Chintalapani" <ka...@harsha.io>
> wrote:
> > > > >
> > > > > >Sriram,
> > > > > > "Can the streaming platform exist without stream processing? -
> > No.
> > > > > >Processing stream data again is a core part of streaming
> platform."
> > > > > >
> > > > > >Yes, it can. There are no.of Stream processing frameworks out
> there,
> > > and
> > > > > >they all have integration into Kafka.
> > > > > >It doesn't need to be developed within Kafka.
> > > > > >
> > > > > >
> > > > > >"Can the platform exist without the rest proxy? - Yes. The proxy
> > does
> > > > not
> > > > > >complete the platform vision in anyway. It is just a good to have
> > tool
> > > > > >that
> > > > > >might be required by quite a few users and there is an active
> > project
> > > > that
> > > > > >works on this - https://github.com/confluentinc/kafka-rest"
> > > > > >
> > > > > >The rest proxy is as important as any API. The vision that shown
> > here
> > > > > >http://kafka.apache.org/intro
> > > > > >require users to write the producers and consumers to get their
> data
> > > > into
> > > > > >and out of Kafka, without which having Streams or Connect won't
> help
> > > > > >anyone.
> > > > > >The rest proxy makes easier for users get their data into Kafka.
> > > > > >Adding the rest proxy to the project doesn't invalidate the
> current
> > > > > >vision,
> > > > > >it only strengthens it.
> > > > > >
> > > > > >Thanks,
> > > > > >Harsha
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <
> > ram@confluent.io>
> > > > > >wrote:
> > > > > >
> > > > > >FWIW, Apache Kafka has evolved a lot from where it started. It did
> > > start
> > > > > >as
> > > > > >a messaging system. Over time we realized that that the vision for
> > > Kafka
> > > > > >is
> > > > > >to build a streaming platform and not just a messaging system. You
> > can
> > > > > >take
> > > > > >a look at the site for more description about what comprises the
> > > > streaming
> > > > > >platform http://kafka.apache.org/ and
> http://kafka.apache.org/intro
> > .
> > > > > >
> > > > > >Can the streaming platform exist without Connect? - No. Data
> > > integration
> > > > > >is
> > > > > >fundamental to building an end to end platform
> > > > > >
> > > > > >Can the streaming platform exist without stream processing? - No.
> > > > > >Processing stream data again is a core part of streaming platform.
> > > > > >
> > > > > >Can the streaming platform exist without clients? - We at least
> need
> > > one
> > > > > >client library to complete the platform. Our Java clients help us
> to
> > > > > >complete the platform story. The rest of the clients are built and
> > > > > >maintained outside the project.
> > > > > >
> > > > > >Can the platform exist without the rest proxy? - Yes. The proxy
> does
> > > not
> > > > > >complete the platform vision in anyway. It is just a good to have
> > tool
> > > > > >that
> > > > > >might be required by quite a few users and there is an active
> > project
> > > > that
> > > > > >works on this - https://github.com/confluentinc/kafka-rest
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > > > ><ns...@linkedin.com.invalid>
> > > > > >wrote:
> > > > > >
> > > > > >> Are you saying Kafka REST is subjective but Kafka Streams and
> > Kafka
> > > > > >Connect
> > > > > >> are not subjective?
> > > > > >>
> > > > > >> > "there are likely places that can live without a rest proxy"
> > > > > >>
> > > > > >> There are also places that can live without Kafka Streams and
> > Kafka
> > > > > >> Connect.
> > > > > >>
> > > > > >> Nacho
> > > > > >>
> > > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > >>
> > > > > >> > At the high level, I think ideally it makes sense to add a
> > > component
> > > > > >>to
> > > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight
> > > > > >integration
> > > > > >> > with Kafka core. For Kafka Stream, we do expect stream
> > processing
> > > > will
> > > > > >be
> > > > > >> > used widely in the future. Implementation wise, Kafka Stream
> > only
> > > > > >> supports
> > > > > >> > getting data from Kafka and leverages quite a few of the core
> > > > > >> > functionalities in Kafka core. For example, it uses customized
> > > > > >>rebalance
> > > > > >> > callback in the consumer and uses the compacted topic heavily.
> > So,
> > > > > >having
> > > > > >> > Kafka Stream in the same repo makes it easier for testing when
> > > those
> > > > > >core
> > > > > >> > functionalities evolve over time. Kafka Connect is in the same
> > > > > >situation.
> > > > > >> >
> > > > > >> > For rest proxy, whether it's widely used or not is going to
> be a
> > > bit
> > > > > >> > subjective. However, there are likely places that can live
> > > without a
> > > > > >rest
> > > > > >> > proxy. The rest proxy is just a proxy for the regular clients
> > and
> > > > > >doesn't
> > > > > >> > need to be tightly integrated with Kafka core. So, the case
> for
> > > > > >including
> > > > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka
> > > Stream
> > > > > >>and
> > > > > >> > Kafka Connect.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> >
> > > > > >> > Jun
> > > > > >> >
> > > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > > > >><Mi...@ig.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > So from my reading essentially the first question needs to
> > > > > >answered/and
> > > > > >> > > voted on is:
> > > > > >> > >
> > > > > >> > > Is Apache Kafka Community only about the Core or does the
> > apache
> > > > > >> > community
> > > > > >> > > also support some subprojects (and just we need some better
> > way
> > > to
> > > > > >> manage
> > > > > >> > > this)
> > > > > >> > >
> > > > > >> > > If vote for Core only wins, then the following should be
> > > removed:
> > > > > >> > > Kafka Connect
> > > > > >> > > Kafka Stream
> > > > > >> > >
> > > > > >> > > If vote for Core only loses (aka we will support
> subprojects)
> > > > then:
> > > > > >> > > We should look to add Kafka Rest
> > > > > >> > >
> > > > > >> > > And we should look to see how we can manage better govern
> and
> > > > manage
> > > > > >> > > submodules.
> > > > > >> > >
> > > > > >> > > A good example which id propose here is how some other
> > > communities
> > > > > >>in
> > > > > >> > > Apache do this.
> > > > > >> > >
> > > > > >> > > Each Module has a Module Management Committee(MMC), this is
> > like
> > > > > >almost
> > > > > >> > > the PMC but at a per module basis.
> > > > > >> > >
> > > > > >> > > This MMC should essentially hold the binding votes for that
> > > > module.
> > > > > >> > > The MMC should be made up of a single representative from
> each
> > > > > >> > > organisation (so no single organisation can fully veto the
> > > > > >>community
> > > > > >> it
> > > > > >> > > has to a genuine consenus)
> > > > > >> > > The MMC requires at least 3 members (so there cant be a tied
> > > vote
> > > > on
> > > > > >2)
> > > > > >> > > For a new Module to be added a MMC committee should be
> sought
> > > > > >> > > A new Module is only capable of being added if the above
> > > > > >>requirements
> > > > > >> can
> > > > > >> > > be met (e.g. 3 people wishing to step up, from 3
> > organisations)
> > > so
> > > > > >that
> > > > > >> > > only actively support modules would be added
> > > > > >> > >
> > > > > >> > > The PMC reviews each module every 6months or Year. If MMC is
> > > > > >>inactive,
> > > > > >> a
> > > > > >> > > vote/call to find replacements if raised, if none are
> > > forthcoming
> > > > > >> > dropping
> > > > > >> > > the MMC to less than 3 then the module moves to "the attic"
> > > (very
> > > > > >>much
> > > > > >> > like
> > > > > >> > > apache attic but a little more aggressively)
> > > > > >> > >
> > > > > >> > > This way the PMC does not need to micro manage every module
> > > > > >> > > We only add modules where some amount of active support and
> > > > > >maintenance
> > > > > >> > > and use is provided by the community
> > > > > >> > > We have an automatic way to retire old or inactive projects.
> > > > > >> > >
> > > > > >> > > Thoughts?
> > > > > >> > > Mike
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > ________________________________________
> > > > > >> > > From: Harsha Ch <ha...@gmail.com>
> > > > > >> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > > > >> > > To: dev@kafka.apache.org
> > > > > >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > >> > >
> > > > > >> > > Jay,
> > > > > >> > > REST API is something every user is in need of. If the
> > > > > >>argument
> > > > > >> is
> > > > > >> > to
> > > > > >> > > clone and write your API, this will do a disservice to the
> > > users
> > > > as
> > > > > >> they
> > > > > >> > > now have to choose one vs. others instead of keeping one API
> > > that
> > > > is
> > > > > >> > > supported in Kafka community.
> > > > > >> > >
> > > > > >> > > "Pre-emptively re-creating another
> > > > > >> > > REST layer when it seems like we all quite agree on what
> needs
> > > to
> > > > be
> > > > > >> done
> > > > > >> > > and we have an existing code base for HTTP/Kafka access that
> > is
> > > > > >heavily
> > > > > >> > > used in production seems quite silly."
> > > > > >> > >
> > > > > >> > > Exactly our point. Why can't we develop this in
> Apache
> > > > Kafka
> > > > > >> > > community? Instead of us open sourcing another GitHub
> project
> > > and
> > > > > >> > creating
> > > > > >> > > a divide in users and another version of API. Let's build
> this
> > > in
> > > > > >Kafka
> > > > > >> > > Community and use the governance model that is proven to
> > provide
> > > > > >vendor
> > > > > >> > > free user driven consensus features. The argument that is
> > adding
> > > > > >>this
> > > > > >> > REST
> > > > > >> > > server to Kafka will affect the agility of the project
> doesn't
> > > mak
> > > > > >> sense.
> > > > > >> > >
> > > > > >> > > It looks like your argument is either we develop all these
> > small
> > > > > >>tools
> > > > > >> or
> > > > > >> > > none at all. We as a community need to look at supporting
> > > critical
> > > > > >> > > tools/API. Instead of dividing this project into individual
> > > > external
> > > > > >> > > communities. We should build this as part of Kafka which
> best
> > > > serves
> > > > > >> the
> > > > > >> > > needs of users.
> > > > > >> > > The Streams and Connect projects that were pushed
> into
> > > > Kafka
> > > > > >> > could
> > > > > >> > > have been left in their own Github projects based on your
> > > > arguments.
> > > > > >> What
> > > > > >> > > about the REST API is so different that such that it should
> > stay
> > > > out
> > > > > >of
> > > > > >> > the
> > > > > >> > > Kafka project? From my experience, more users are asking for
> > the
> > > > > >>REST
> > > > > >> > API.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Harsha
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <jay@confluent.io
> >
> > > > wrote:
> > > > > >> > >
> > > > > >> > > > I think the questions around governance make sense, I
> think
> > we
> > > > > >should
> > > > > >> > > > really clarify that to make the process more clear so it
> can
> > > be
> > > > > >fully
> > > > > >> > > > inclusive.
> > > > > >> > > >
> > > > > >> > > > The idea that we should not collaborate on what is there
> > now,
> > > > > >though,
> > > > > >> > > > because in the future we might disagree about direction
> does
> > > not
> > > > > >> really
> > > > > >> > > > make sense to me. If in the future we disagree, that is
> the
> > > > beauty
> > > > > >of
> > > > > >> > > open
> > > > > >> > > > source, you can always fork off a copy of the code and
> start
> > > an
> > > > > >> > > independent
> > > > > >> > > > project either in Apache or elsewhere. Pre-emptively
> > > re-creating
> > > > > >> > another
> > > > > >> > > > REST layer when it seems like we all quite agree on what
> > needs
> > > > to
> > > > > >>be
> > > > > >> > done
> > > > > >> > > > and we have an existing code base for HTTP/kafka access
> that
> > > is
> > > > > >> heavily
> > > > > >> > > > used in production seems quite silly.
> > > > > >> > > >
> > > > > >> > > > Let me give some background on how I at least think about
> > > these
> > > > > >> things.
> > > > > >> > > > I've participated in open source projects out of LinkedIn
> > via
> > > > > >>github
> > > > > >> as
> > > > > >> > > > well as via the ASF. I don't think there is a "right"
> answer
> > > to
> > > > > >>how
> > > > > >> to
> > > > > >> > do
> > > > > >> > > > these but rather some tradeoffs. We thought about this
> > quite a
> > > > lot
> > > > > >in
> > > > > >> > the
> > > > > >> > > > context of Kafka based on the experience with the Hadoop
> > > > ecosystem
> > > > > >as
> > > > > >> > > well
> > > > > >> > > > as from other open source communities.
> > > > > >> > > >
> > > > > >> > > > There is a rich ecosystem around Kafka. Many of the
> projects
> > > are
> > > > > >> quite
> > > > > >> > > > small--single clients or tools that do a single thing
> > > well--and
> > > > > >> almost
> > > > > >> > > none
> > > > > >> > > > of them are top level apache projects. I don't think
> trying
> > to
> > > > > >>force
> > > > > >> > each
> > > > > >> > > > of these to turn into independent Apache projects is
> > > necessarily
> > > > > >>the
> > > > > >> > best
> > > > > >> > > > thing for the ecosystem.
> > > > > >> > > >
> > > > > >> > > > My observation of how this can go wrong is really what I
> > think
> > > > has
> > > > > >> > > happened
> > > > > >> > > > to the Hadoop ecosystem. There you see quite a zoo of
> > projects
> > > > > >>which
> > > > > >> > all
> > > > > >> > > > drift apart and don't quite work together well.
> Coordinating
> > > > even
> > > > > >> > simple
> > > > > >> > > > changes and standardization across these is exceptionally
> > > > > >>difficult.
> > > > > >> > The
> > > > > >> > > > result is a bit of a mess for users--the pieces just don't
> > > > really
> > > > > >> come
> > > > > >> > > > together very well. This makes sense for independent
> > > > > >>infrastructure
> > > > > >> > > systems
> > > > > >> > > > (Kudu vs HDFS) but I'm not at all convinced that doing
> this
> > > for
> > > > > >every
> > > > > >> > > > little tool or helper library has lead to a desirable
> > state. I
> > > > > >>think
> > > > > >> > the
> > > > > >> > > > mode of operating where the Hadoop vendors spawn off a few
> > new
> > > > > >Apache
> > > > > >> > > > projects for each new product initiative, especially since
> > > often
> > > > > >that
> > > > > >> > > > project is only valued by that vendor (and the other
> vendor
> > > has
> > > > a
> > > > > >> > > different
> > > > > >> > > > competing Apache project) doesn't necessarily do a better
> > job
> > > at
> > > > > >> > > producing
> > > > > >> > > > high quality communities or high quality software.
> > > > > >> > > >
> > > > > >> > > > These tools/connects/clients/proxies and other integration
> > > > pieces
> > > > > >can
> > > > > >> > > take
> > > > > >> > > > many forms, but my take of what makes one of these things
> > good
> > > > is
> > > > > >> that
> > > > > >> > it
> > > > > >> > > > remains simple, does its one thing well, and cleaves as
> > > closely
> > > > as
> > > > > >> > > possible
> > > > > >> > > > to the conventions for Kafka itself--i.e. doesn't invent
> new
> > > > ways
> > > > > >>of
> > > > > >> > > > monitoring, configuring, etc. For the tools we've
> > contributed
> > > > > >>we've
> > > > > >> > tried
> > > > > >> > > > really hard to make them consistent with Kafka as well as
> > with
> > > > > >>each
> > > > > >> > other
> > > > > >> > > > in how testing, configuration, monitoring, etc works.
> > > > > >> > > >
> > > > > >> > > > I think what Apache does superbly well is create a
> community
> > > for
> > > > > >> > > managing a
> > > > > >> > > > large infrastructure layer like Kafka in a vendor
> > independent
> > > > way.
> > > > > >> > What I
> > > > > >> > > > think is less successful is attempting to form full and
> > > > > >>independent
> > > > > >> > > apache
> > > > > >> > > > communities around very simple single purpose tools,
> > > especially
> > > > if
> > > > > >> you
> > > > > >> > > hope
> > > > > >> > > > for these to come together into a cohesive toolset across
> > > > multiple
> > > > > >> such
> > > > > >> > > > tools. Much of what Apache does--create a collective
> > decision
> > > > > >>making
> > > > > >> > > > process for resolving disagreement, help to trademark and
> > > > protect
> > > > > >the
> > > > > >> > > marks
> > > > > >> > > > of the project, etc just isn't that relevant for simple
> > > > > >> single-purpose
> > > > > >> > > > tools.
> > > > > >> > > >
> > > > > >> > > > So my take is there are a couple of options:
> > > > > >> > > >
> > > > > >> > > > 1. We can try to put all the small tools into the
> Apache
> > > > > >>Project.
> > > > > >> I
> > > > > >> > > > think this is not the right approach as there is simply
> > too
> > > > > >>many
> > > > > >> of
> > > > > >> > > > them,
> > > > > >> > > > many in different languages, serving different
> protocols,
> > > > > >> > integrating
> > > > > >> > > > with
> > > > > >> > > > particular systems, and a single community can't
> > > effectively
> > > > > >> > maintain
> > > > > >> > > > them
> > > > > >> > > > all. Doing this would significantly slow the progress
> of
> > > the
> > > > > >Kafka
> > > > > >> > > > project.
> > > > > >> > > > As a protocol for messaging, I don't really see a case
> > for
> > > > > >> including
> > > > > >> > > > REST
> > > > > >> > > > but not MQTT or AMQP which are technically much better
> > > suited
> > > > > >>to
> > > > > >> > > > messaging
> > > > > >> > > > and both are widely used for that.
> > > > > >> > > > 2. We can treat ecosystem projects that aren't top
> level
> > > > Apache
> > > > > >> > > projects
> > > > > >> > > > as invalid and try to recreate them all as Apache
> > projects.
> > > > > >> > Honestly,
> > > > > >> > > > though, if you go to the Kafka ecosystem page virtually
> > > none
> > > > of
> > > > > >> the
> > > > > >> > > most
> > > > > >> > > > popular add-ons to Kafka are Apache projects. The most
> > > > > >>successful
> > > > > >> > > > things in
> > > > > >> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka,
> a
> > > > number
> > > > > >of
> > > > > >> > > other
> > > > > >> > > > clients, as well as the existing REST layer have
> > succeeded
> > > at
> > > > > >> > > developing
> > > > > >> > > > communities that actively contribute and use these
> pieces
> > > > and I
> > > > > >> > don't
> > > > > >> > > > know
> > > > > >> > > > that that is a bad thing unless that community proves
> to
> > be
> > > > > >> > > uninclusive,
> > > > > >> > > > unresponsive, or goes in a bad technical direction--and
> > > those
> > > > > >>are
> > > > > >> > > > failure
> > > > > >> > > > modes that all open source efforts face.
> > > > > >> > > > 3. We can do what I think makes the most sense and try
> to
> > > > work
> > > > > >> with
> > > > > >> > > the
> > > > > >> > > > projects that exist in the ecosystem and if the project
> > > > doesn't
> > > > > >> > have a
> > > > > >> > > > responsive community or wants to go in a different
> > > direction
> > > > > >>fork
> > > > > >> or
> > > > > >> > > > recreate that work.
> > > > > >> > > >
> > > > > >> > > > Of course any person can choose whatever of these options
> > they
> > > > > >>want.
> > > > > >> > But
> > > > > >> > > > from my point of view, option (3) has been the path of the
> > > > > >>community
> > > > > >> so
> > > > > >> > > far
> > > > > >> > > > and I think it has been quite successful.
> > > > > >> > > >
> > > > > >> > > > -Jay
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> > > > > >> kafka@harsha.io
> > > > > >> > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Neha,
> > > > > >> > > > > "But I haven't seen any community emails or patches
> being
> > > > > >submitted
> > > > > >> > by
> > > > > >> > > > you
> > > > > >> > > > > guys, so I'm wondering why you are concerned about
> whether
> > > the
> > > > > >> > > community
> > > > > >> > > > is
> > > > > >> > > > > open to accepting patches or not."
> > > > > >> > > > >
> > > > > >> > > > > I think you are talking about contributing patches to
> this
> > > > > >> repository
> > > > > >> > > > > right? https://github.com/confluentinc/kafka-rest .
> All I
> > > am
> > > > > >> saying
> > > > > >> > > > > the guidelines/governance model is not clear on the
> > project
> > > > and
> > > > > >>I
> > > > > >> > guess
> > > > > >> > > > its
> > > > > >> > > > > driven by opening a github issue request. Its the
> > > repository
> > > > > >owned
> > > > > >> > by
> > > > > >> > > > > confluent and as much I appreciate that the features we
> > > > > >>mentioned
> > > > > >> are
> > > > > >> > > in
> > > > > >> > > > > the roadmap and welcoming us to contribute to the
> project.
> > > It
> > > > > >> doesn't
> > > > > >> > > > > gurantee what we want to add in the furture will be in
> > your
> > > > > >> roadmap.
> > > > > >> > > > >
> > > > > >> > > > > Hence the reason having it part of Kafka community will
> > > help a
> > > > > >>lot
> > > > > >> as
> > > > > >> > > > other
> > > > > >> > > > > users can participate in the discussions. We are happy
> to
> > > > drive
> > > > > >> any
> > > > > >> > > > > feature additions through KIPs which gives everyone a
> > chance
> > > > to
> > > > > >> > > > participate
> > > > > >> > > > > and add to the discussions.
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Harsha
> > > > > >> > > > >
> > > > > >> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > > > > >> > Michael.Pearce@ig.com>
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > +1
> > > > > >> > > > > >
> > > > > >> > > > > > I agree on the governance comments whole heartedly.
> > > > > >> > > > > >
> > > > > >> > > > > > Also i agree about the contribution comments made
> > earlier
> > > in
> > > > > >>the
> > > > > >> > > > thread,
> > > > > >> > > > > i
> > > > > >> > > > > > personally am less likely to spend any of mine, or
> give
> > > > > >>project
> > > > > >> > time
> > > > > >> > > > > within
> > > > > >> > > > > > my internal projects to developers contributing to
> > another
> > > > > >> > commercial
> > > > > >> > > > > > companies project even so technically open source, as
> > then
> > > > > >>there
> > > > > >> is
> > > > > >> > > > that
> > > > > >> > > > > > commercial companies interest will always prevail and
> > > > > >essentially
> > > > > >> > can
> > > > > >> > > > > > always have a final vote where disagreement. Im sure
> > they
> > > > > >>never
> > > > > >> > > intend
> > > > > >> > > > > to,
> > > > > >> > > > > > but there is that true reality. This is why we have
> > > > community
> > > > > >> open
> > > > > >> > > > source
> > > > > >> > > > > > projects.
> > > > > >> > > > > >
> > > > > >> > > > > > I can find many different implementations now of a
> rest
> > > > > >>endpoint
> > > > > >> on
> > > > > >> > > > > > GitHub, BitBucket etc. Each one has their benefits and
> > > > > >> > disadvantages
> > > > > >> > > in
> > > > > >> > > > > > their implementation. By making / providing one this
> > would
> > > > > >>bring
> > > > > >> > > > together
> > > > > >> > > > > > these solutions, unifying those developers and also
> > > bringing
> > > > > >>the
> > > > > >> > best
> > > > > >> > > > of
> > > > > >> > > > > > all.
> > > > > >> > > > > >
> > > > > >> > > > > > I understand the concern on the community burden
> > > > > >> adding/supporting
> > > > > >> > > more
> > > > > >> > > > > > surface area for every client. But something like REST
> > is
> > > > > >> universal
> > > > > >> > > and
> > > > > >> > > > > > worthy to be owned by the community.
> > > > > >> > > > > >
> > > > > >> > > > > > Mike
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > ________________________________________
> > > > > >> > > > > > From: Andrew Schofield <andrew_schofield_jira@
> > outlook.com
> > > >
> > > > > >> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > >> > > > > > To: dev@kafka.apache.org
> > > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > >> > > > > >
> > > > > >> > > > > > There's a massive difference between the governance of
> > > Kafka
> > > > > >>and
> > > > > >> > the
> > > > > >> > > > > > governance of the REST proxy.
> > > > > >> > > > > >
> > > > > >> > > > > > In Kafka, there is a broad community of people
> > > contributing
> > > > > >their
> > > > > >> > > > > opinions
> > > > > >> > > > > > about future enhancements in the form of KIPs. There's
> > > some
> > > > > >> really
> > > > > >> > > deep
> > > > > >> > > > > > consideration that goes into some of the trickier
> KIPs.
> > > > There
> > > > > >are
> > > > > >> > > > people
> > > > > >> > > > > > outside Confluent deeply knowledgeable in Kafka and
> > > > building
> > > > > >the
> > > > > >> > > > > > reputations to become committers. I get the impression
> > > that
> > > > > >>the
> > > > > >> > > roadmap
> > > > > >> > > > > of
> > > > > >> > > > > > Kafka is not really community-owned (what's the big
> > > feature
> > > > > >>for
> > > > > >> > Kafka
> > > > > >> > > > > 0.11,
> > > > > >> > > > > > for example), but the conveyor belt of smaller
> features
> > in
> > > > the
> > > > > >> form
> > > > > >> > > of
> > > > > >> > > > > KIPs
> > > > > >> > > > > > works nicely. It's a good example of open-source
> > working
> > > > > >>well.
> > > > > >> > > > > >
> > > > > >> > > > > > The equivalent for the REST proxy is basically issues
> on
> > > > > >>GitHub.
> > > > > >> > The
> > > > > >> > > > > > roadmap is less clear. There's not really a community
> > > > properly
> > > > > >> > > engaged
> > > > > >> > > > in
> > > > > >> > > > > > the way that there is with Kafka. So, you could say
> that
> > > > it's
> > > > > >> clear
> > > > > >> > > > that
> > > > > >> > > > > > fewer people are interested, but I think the whole
> > > > governance
> > > > > >> > thing
> > > > > >> > > > is a
> > > > > >> > > > > > big barrier to engagement. And it's looking like it's
> > > > getting
> > > > > >out
> > > > > >> > of
> > > > > >> > > > > date.
> > > > > >> > > > > >
> > > > > >> > > > > > In technical terms, I can think of two big
> improvements
> > to
> > > > the
> > > > > >> REST
> > > > > >> > > > > proxy.
> > > > > >> > > > > > First, it needs to use the new consumer API so that
> it's
> > > > > >possible
> > > > > >> > to
> > > > > >> > > > > secure
> > > > > >> > > > > > connections between the REST proxy and Kafka. I don't
> > care
> > > > too
> > > > > >> much
> > > > > >> > > > which
> > > > > >> > > > > > method calls it uses actually uses to consume
> messages,
> > > > but I
> > > > > >do
> > > > > >> > > care
> > > > > >> > > > > that
> > > > > >> > > > > > I cannot secure connections because of network
> security
> > > > rules.
> > > > > >> > > Second,
> > > > > >> > > > > > there's an affinity between a consumer and the
> instance
> > of
> > > > the
> > > > > >> REST
> > > > > >> > > > proxy
> > > > > >> > > > > > to which it first connected. Kafka itself avoids this
> > kind
> > > > of
> > > > > >> > > affinity
> > > > > >> > > > > for
> > > > > >> > > > > > good reason, and in the name of availability the REST
> > > proxy
> > > > > >> should
> > > > > >> > > too.
> > > > > >> > > > > > These are natural KIPs.
> > > > > >> > > > > >
> > > > > >> > > > > > I think it would be good to have the code for the REST
> > > proxy
> > > > > >> > > > contributed
> > > > > >> > > > > > to Apache Kafka so that it would be able to be
> developed
> > > in
> > > > > >>the
> > > > > >> > same
> > > > > >> > > > way.
> > > > > >> > > > > >
> > > > > >> > > > > > Andrew Schofield
> > > > > >> > > > > >
> > > > > >> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > > >> > > > > > Sent: 07 October 2016 22:41:52
> > > > > >> > > > > > To: dev@kafka.apache.org
> > > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > >> > > > > >
> > > > > >> > > > > > ASF already gives us a clear framework and governance
> > > model
> > > > > >>for
> > > > > >> > > > community
> > > > > >> > > > > > development. This is already understood by the people
> > > > > >> contributing
> > > > > >> > to
> > > > > >> > > > > > Apache Kafka project, and they are the same people who
> > > want
> > > > to
> > > > > >> > > > contribute
> > > > > >> > > > > > to the REST server capability as well. Everyone is in
> > > > > >>agreement
> > > > > >> on
> > > > > >> > > the
> > > > > >> > > > > > need for collaborating on this effort. So why not
> > > contribute
> > > > > >>the
> > > > > >> > code
> > > > > >> > > > to
> > > > > >> > > > > > Apache Kafka. This will help avoid duplication of
> effort
> > > and
> > > > > >> forks
> > > > > >> > > that
> > > > > >> > > > > > may crop up, hugely benefitting the user community.
> This
> > > > will
> > > > > >> also
> > > > > >> > > > avoid
> > > > > >> > > > > > having to define a process similar to ASF on a GitHub
> > > > project
> > > > > >and
> > > > > >> > > > instead
> > > > > >> > > > > > there is a single community with clear understanding
> > > > community
> > > > > >> > > process
> > > > > >> > > > as
> > > > > >> > > > > > defined in ASF.
> > > > > >> > > > > >
> > > > > >> > > > > > As others have said, this is an important capability
> for
> > > > > >>Apache
> > > > > >> > > Kafka.
> > > > > >> > > > It
> > > > > >> > > > > > is worth maintaining this as a part of the project.
> > > > > >> > > > > >
> > > > > >> > > > > > Regards,
> > > > > >> > > > > > Suresh
> > > > > >> > > > > >
> > > > > >> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <
> > ofir.manor@equalum.io>
> > > > > >>wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > >I personally think it would be quite wasteful to
> > > > re-implement
> > > > > >> the
> > > > > >> > > REST
> > > > > >> > > > > > >gateway just because that an actively-maintained
> piece
> > of
> > > > > >> > > > > Apache-licensed
> > > > > >> > > > > > >software is not governed directly by the Apache Kafka
> > > > > >> community...
> > > > > >> > > > While
> > > > > >> > > > > > >kafka-rest repo is owned by Confluent, the
> contributors
> > > > > >> including
> > > > > >> > > the
> > > > > >> > > > > main
> > > > > >> > > > > > >one are also part of the Apache Kafka community, so
> > > there
> > > > > >>is a
> > > > > >> > > chance
> > > > > >> > > > > to
> > > > > >> > > > > > >work this out.
> > > > > >> > > > > > >
> > > > > >> > > > > > >However, there are two valid concerns here that could
> > be
> > > > > >> > addressed,
> > > > > >> > > > > around
> > > > > >> > > > > > >community and accessibility:
> > > > > >> > > > > > >>> What we are worried about is a project
> > > > > >> > > > > > >>> that's not maintained in a community. So the
> process
> > > of
> > > > > >> > accepting
> > > > > >> > > > > > >>>patches
> > > > > >> > > > > > >>> and priorities is not clear, and it's not
> developed
> > in
> > > > > >Apache
> > > > > >> > > > > > >>>community.
> > > > > >> > > > > > >>> Not only that, existing REST API project doesn't
> > > support
> > > > > >>new
> > > > > >> > > client
> > > > > >> > > > > API
> > > > > >> > > > > > >and
> > > > > >> > > > > > >>> hence there is no security support either.
> > > > > >> > > > > > >
> > > > > >> > > > > > >This might be easy to fix. Maybe Confluent /
> kafka-rest
> > > > > >> community
> > > > > >> > > can
> > > > > >> > > > > > >clarify that - what is their contribution policy, dev
> > > > style,
> > > > > >> > roadmap
> > > > > >> > > > > etc.
> > > > > >> > > > > > >If they want, they can make an effort to encourage
> > > > > >participation
> > > > > >> > > from
> > > > > >> > > > > > >people outside Confluent (easily accept
> contributions,
> > > > invite
> > > > > >> > > external
> > > > > >> > > > > > >commiters or have open dev process similar to Apache
> > > Kafka
> > > > > >etc),
> > > > > >> > as
> > > > > >> > > > > there
> > > > > >> > > > > > >is definitely seems to be some interest on the list.
> > That
> > > > > >>might
> > > > > >> > > clear
> > > > > >> > > > > the
> > > > > >> > > > > > >community concern and help kafka-rest project (but
> that
> > > is
> > > > a
> > > > > >> > > > calculation
> > > > > >> > > > > > >Confluent will have to make).
> > > > > >> > > > > > >
> > > > > >> > > > > > >The other, independent, concern is that REST is
> > something
> > > > > >>that
> > > > > >> is
> > > > > >> > > > > expected
> > > > > >> > > > > > >to be available out of the box with Kafka. I
> personally
> > > > don't
> > > > > >> feel
> > > > > >> > > > > > >strongly
> > > > > >> > > > > > >about it (better use proper, efficient APIs from day
> > > one),
> > > > > >> though
> > > > > >> > it
> > > > > >> > > > is
> > > > > >> > > > > > >definitely way smaller than adding a stream
> processing
> > > > engine
> > > > > >to
> > > > > >> > the
> > > > > >> > > > > > >project :)
> > > > > >> > > > > > >Again,the kafka-rest "community" could take steps to
> > make
> > > > it
> > > > > >> even
> > > > > >> > > > easier
> > > > > >> > > > > > >to
> > > > > >> > > > > > >install, configure and run kafka-rest for new users
> on
> > > > > >>vanilla
> > > > > >> > > Apache
> > > > > >> > > > > > >Kafka
> > > > > >> > > > > > >(outside the Confluent platform), if they wish that
> (or
> > > > > >>welcome
> > > > > >> > > > > > >contributions to that end), but that is up to them.
> > > > > >> > > > > > >Finally, if after the above steps were taken there
> > would
> > > > > >>still
> > > > > >a
> > > > > >> > > > strong
> > > > > >> > > > > > >desire to include a great rest gateway with Apache
> > > Kafka, I
> > > > > >> assume
> > > > > >> > > the
> > > > > >> > > > > > >community could hypothetically fork the existing
> > > kafka-rest
> > > > > >into
> > > > > >> > an
> > > > > >> > > > > Apache
> > > > > >> > > > > > >Kafka subproject and maintain it "within Apache"
> > instead
> > > of
> > > > > >> > > > implementing
> > > > > >> > > > > > >it
> > > > > >> > > > > > >from scratch (though I'm not a lawyer etc) - but I
> > cannot
> > > > > >> imagine
> > > > > >> > it
> > > > > >> > > > > > >happen
> > > > > >> > > > > > >without Confluent blessing, and I think that is
> likely
> > > much
> > > > > >less
> > > > > >> > > > optimal
> > > > > >> > > > > > >(pulling in other Confluent / Apache licensed
> > > dependencies)
> > > > > >than
> > > > > >> > > > having
> > > > > >> > > > > a
> > > > > >> > > > > > >separate external community around kafka-rest.
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >Just my two cents,
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >Ofir Manor
> > > > > >> > > > > > >
> > > > > >> > > > > > >Co-Founder & CTO | Equalum
> > > > > >> > > > > > >
> > > > > >> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > > > > ><+972%2054-780-1286>
> > > > > >> > <+972%2054-780-1286> |
> > > > > >> > > > Email:
> > > > > >> > > > > > ofir.manor@equalum.io
> > > > > >> > > > > > >
> > > > > >> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani
> <
> > > > > >> > > kafka@harsha.io
> > > > > >> > > > >
> > > > > >> > > > > > >wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > >> Neha & Jay,
> > > > > >> > > > > > >> We did look at the open source
> > > > > >>alternatives.
> > > > > >> > Our
> > > > > >> > > > > > >>concern
> > > > > >> > > > > > >> is what's the patch acceptance and adding features/
> > > > > >>bug-fixes
> > > > > >> to
> > > > > >> > > the
> > > > > >> > > > > > >> existing project under a Github (although it's
> > licensed
> > > > > >>under
> > > > > >> > > Apache
> > > > > >> > > > > > >>2.0).
> > > > > >> > > > > > >> It would be great if that project made available
> > under
> > > > > >>Apache
> > > > > >> > and
> > > > > >> > > > > > >>driven by
> > > > > >> > > > > > >> the community. Adding to the above, not all Kafka
> > > users
> > > > > >>are
> > > > > >> > > > > interested
> > > > > >> > > > > > >>in
> > > > > >> > > > > > >> using the Java client API, they would like to have
> > > simple
> > > > > >REST
> > > > > >> > API
> > > > > >> > > > > where
> > > > > >> > > > > > >> they can code against using any language. I do
> > believe
> > > > this
> > > > > >> adds
> > > > > >> > > > value
> > > > > >> > > > > > >>to
> > > > > >> > > > > > >> Apache Kafka in itself.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> "For 1, I don't think there is value in giving in
> to
> > > the
> > > > > >>NIH
> > > > > >> > > > syndrome
> > > > > >> > > > > > >>and
> > > > > >> > > > > > >> reinventing the wheel. What I'm looking for is a
> > > detailed
> > > > > >> > > comparison
> > > > > >> > > > > of
> > > > > >> > > > > > >>the
> > > > > >> > > > > > >> gaps and why those can't be improved in the REST
> > proxy
> > > > that
> > > > > >> > > already
> > > > > >> > > > > > >>exists
> > > > > >> > > > > > >> and is actively maintained."
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> We are not looking at this as NIH. What we are
> > worried
> > > > > >>about
> > > > > >> > is a
> > > > > >> > > > > > >>project
> > > > > >> > > > > > >> that's not maintained in a community. So the
> process
> > of
> > > > > >> > accepting
> > > > > >> > > > > > >>patches
> > > > > >> > > > > > >> and priorities is not clear, and it's not developed
> > in
> > > > > >>Apache
> > > > > >> > > > > community.
> > > > > >> > > > > > >> Not only that, existing REST API project doesn't
> > > support
> > > > > >>new
> > > > > >> > > client
> > > > > >> > > > > API
> > > > > >> > > > > > >>and
> > > > > >> > > > > > >> hence there is no security support either.
> > > > > >> > > > > > >> We don't know the timeline when that's made
> > available.
> > > We
> > > > > >> would
> > > > > >> > > like
> > > > > >> > > > > to
> > > > > >> > > > > > >>add
> > > > > >> > > > > > >> admin functionality into the REST API. So the
> Roadmap
> > > of
> > > > > >>that
> > > > > >> > > > project
> > > > > >> > > > > is
> > > > > >> > > > > > >> not driven by Apache.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> "This doesn't materially have an impact on
> expanding
> > > the
> > > > > >> > usability
> > > > > >> > > > of
> > > > > >> > > > > > >> Kafka. In my experience, REST proxy + Java
> clients
> > > > only
> > > > > >> cover
> > > > > >> > > > ~50%
> > > > > >> > > > > of
> > > > > >> > > > > > >> all
> > > > > >> > > > > > >> Kafka users, and maybe 10% of those are the ones
> > who
> > > > > >>will
> > > > > >> use
> > > > > >> > > the
> > > > > >> > > > > > >>REST
> > > > > >> > > > > > >> proxy. The remaining 50% are non-java client
> users
> > > (C,
> > > > > >> > python,
> > > > > >> > > > go,
> > > > > >> > > > > > >>node
> > > > > >> > > > > > >> etc)."
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> REST API is most often asked feature in my
> > interactions
> > > > > >>with
> > > > > >> > Kafka
> > > > > >> > > > > > >>users.
> > > > > >> > > > > > >> In an organization, There will be independent teams
> > who
> > > > > >>will
> > > > > >> > write
> > > > > >> > > > > their
> > > > > >> > > > > > >> Kafka clients using different language libraries
> > > > available
> > > > > >> > today,
> > > > > >> > > > and
> > > > > >> > > > > > >> there is no way to standardize this. Instead of
> > > > supporting
> > > > > >> > several
> > > > > >> > > > > > >> different client libraries users will be interested
> > in
> > > > > >>using
> > > > > >a
> > > > > >> > > REST
> > > > > >> > > > > API
> > > > > >> > > > > > >> server. The need for a REST API will only increase
> as
> > > > more
> > > > > >and
> > > > > >> > > more
> > > > > >> > > > > > >>users
> > > > > >> > > > > > >> start using Kafka.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> "More surface area means more work to keep things
> > > > > >>consistent.
> > > > > >> > > > Failure
> > > > > >> > > > > > >> to do that has, in fact, hurt the user
> > experience."
> > > > > >> > > > > > >> Having myriad Kafka client GitHub projects that
> > support
> > > > > >> > different
> > > > > >> > > > > > >>languages
> > > > > >> > > > > > >> hurts the user experience and pushes burden to
> > maintain
> > > > > >>these
> > > > > >> > > > > libraries.
> > > > > >> > > > > > >> REST API is a simple code base that uses existing
> > java
> > > > > >>client
> > > > > >> > > > > libraries
> > > > > >> > > > > > >>to
> > > > > >> > > > > > >> make life easier for the users.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> Thanks,
> > > > > >> > > > > > >> Harsha
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > > > > >> > neha@confluent.io>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> > Manikumar,
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > Thanks for sharing the proposal. I think there
> are
> > 2
> > > > > >>parts
> > > > > >> to
> > > > > >> > > this
> > > > > >> > > > > > >> > discussion -
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > 1. Should we rewrite a REST proxy given that
> there
> > > is a
> > > > > >> > > > > > >>feature-complete,
> > > > > >> > > > > > >> > open-source and actively maintained REST proxy in
> > the
> > > > > >> > community?
> > > > > >> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make
> us
> > > > more
> > > > > >> agile
> > > > > >> > > and
> > > > > >> > > > > > >> maintain
> > > > > >> > > > > > >> > the high-quality experience that Kafka users have
> > > > today?
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > For 1, I don't think there is value in giving in
> to
> > > the
> > > > > >>NIH
> > > > > >> > > > syndrome
> > > > > >> > > > > > >>and
> > > > > >> > > > > > >> > reinventing the wheel. What I'm looking for is a
> > > > detailed
> > > > > >> > > > comparison
> > > > > >> > > > > > >>of
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > gaps and why those can't be improved in the REST
> > > proxy
> > > > > >>that
> > > > > >> > > > already
> > > > > >> > > > > > >> exists
> > > > > >> > > > > > >> > and is actively maintained. For example, we
> depend
> > on
> > > > > >> zkClient
> > > > > >> > > and
> > > > > >> > > > > > >>have
> > > > > >> > > > > > >> > found as well as fixed several bugs by working
> > > closely
> > > > > >>with
> > > > > >> > the
> > > > > >> > > > > people
> > > > > >> > > > > > >> who
> > > > > >> > > > > > >> > maintain zkClient. This should be possible for
> REST
> > > > proxy
> > > > > >as
> > > > > >> > > well,
> > > > > >> > > > > > >>right?
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > For 2, I'd like us to review our history of
> > expanding
> > > > the
> > > > > >> > > surface
> > > > > >> > > > > > >>area to
> > > > > >> > > > > > >> > add more clients in the past. Here is a summary -
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > - This doesn't materially have an impact on
> > > > expanding
> > > > > >the
> > > > > >> > > > > > >>usability of
> > > > > >> > > > > > >> > Kafka. In my experience, REST proxy + Java
> > clients
> > > > > >>only
> > > > > >> > cover
> > > > > >> > > > > ~50%
> > > > > >> > > > > > >>of
> > > > > >> > > > > > >> > all
> > > > > >> > > > > > >> > Kafka users, and maybe 10% of those are the
> ones
> > > who
> > > > > >will
> > > > > >> > use
> > > > > >> > > > the
> > > > > >> > > > > > >>REST
> > > > > >> > > > > > >> > proxy. The remaining 50% are non-java client
> > users
> > > > (C,
> > > > > >> > > python,
> > > > > >> > > > > go,
> > > > > >> > > > > > >> node
> > > > > >> > > > > > >> > etc).
> > > > > >> > > > > > >> > - People are a lot more excited about
> promising
> > to
> > > > > >> > contribute
> > > > > >> > > > > while
> > > > > >> > > > > > >> > adding the surface area but not on an ongoing
> > > basis
> > > > > >>down
> > > > > >> > the
> > > > > >> > > > > line.
> > > > > >> > > > > > >> > - More surface area means more work to keep
> > things
> > > > > >> > > consistent.
> > > > > >> > > > > > >>Failure
> > > > > >> > > > > > >> > to do that has, in fact, hurt the user
> > experience.
> > > > > >> > > > > > >> > - More surface area hurts agility. We want to
> > do a
> > > > few
> > > > > >> > things
> > > > > >> > > > > > >>really
> > > > > >> > > > > > >> > well as well as be agile to be able to build
> on
> > > our
> > > > > >>core
> > > > > >> > > > > > >>competency.
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > > > >> > > > manikumar.reddy@gmail.com
> > > > > >> > > > > >
> > > > > >> > > > > > >> > wrote:
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > > Hi Jay,
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > Thanks for your reply.
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > I agree that we can not add all the
> clients/tools
> > > > > >> available
> > > > > >> > in
> > > > > >> > > > > > >> ecosystem
> > > > > >> > > > > > >> > > page to Kafka repo itself. But we feel REST
> > > Interface
> > > > > >>is
> > > > > >> > > > different
> > > > > >> > > > > > >>from
> > > > > >> > > > > > >> > > other clients/tools. Since any language that
> can
> > > work
> > > > > >with
> > > > > >> > > HTTP
> > > > > >> > > > > can
> > > > > >> > > > > > >> > > easily integrate with this interface, Having an
> > > > > >"official"
> > > > > >> > > REST
> > > > > >> > > > > > >> > > interface helps user community. This also helps
> > us
> > > to
> > > > > >> > > integrate
> > > > > >> > > > > well
> > > > > >> > > > > > >> > > with external management and provisioning
> tools.
> > > > > >>Apache
> > > > > >> > Kafka
> > > > > >> > > > > > >>release
> > > > > >> > > > > > >> > > with Java clients + REST interface is
> sufficient
> > > for
> > > > > >>most
> > > > > >> of
> > > > > >> > > the
> > > > > >> > > > > > >>user
> > > > > >> > > > > > >> > > deployments/requirements. This helps users to
> > deal
> > > > with
> > > > > >> less
> > > > > >> > > > > number
> > > > > >> > > > > > >> > > of distributions/builds.
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > Thanks,
> > > > > >> > > > > > >> > > Manikumar
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> > > > > >> jay@confluent.io
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > > Hey guys,
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > There's already a REST interface maintained
> as
> > a
> > > > > >> separate
> > > > > >> > > > > > >> project--it's
> > > > > >> > > > > > >> > > > open source and apache licensed and actively
> > > > > >>maintained
> > > > > >> (
> > > > > >> > > > > > >> > > > https://github.com/confluentinc/kafka-rest).
> > > What
> > > > is
> > > > > >> > wrong
> > > > > >> > > > with
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > > >> > > > > > github.com
> > > > > >> > > > > > The Kafka REST Proxy provides a RESTful interface to a
> > > Kafka
> > > > > >> > cluster.
> > > > > >> > > > It
> > > > > >> > > > > > makes it easy to produce and consume messages, view
> the
> > > > state
> > > > > >>of
> > > > > >> > the
> > > > > >> > > > > > cluster, and ...
> > > > > >> > > > > >
> > > > > >> > > > > > >> that?
> > > > > >> > > > > > >> > > You
> > > > > >> > > > > > >> > > > mentioned that there was some compatibility
> > > > concern,
> > > > > >but
> > > > > >> > > > > > >> compatibility
> > > > > >> > > > > > >> > > has
> > > > > >> > > > > > >> > > > to do with the consumer protocol guarantees
> not
> > > the
> > > > > >repo
> > > > > >> > the
> > > > > >> > > > > code
> > > > > >> > > > > > >>is
> > > > > >> > > > > > >> > in,
> > > > > >> > > > > > >> > > > right? Not sure that concern makes sense.
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > We could argue for adding pretty much
> anything
> > > and
> > > > > >> > > everything
> > > > > >> > > > in
> > > > > >> > > > > > >>the
> > > > > >> > > > > > >> > > > ecosystem page in Kafka itself but I'm not
> sure
> > > > that
> > > > > >> would
> > > > > >> > > > make
> > > > > >> > > > > > >>the
> > > > > >> > > > > > >> > > project
> > > > > >> > > > > > >> > > > more agile.
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > -Jay
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > > >> > > > > > >> manikumar.reddy@gmail.com
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > > wrote:
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > > Hi Kafka Devs,
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server
> to
> > > > Kafka
> > > > > >> > > > Repository.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > There are already open-source alternatives
> > are
> > > > > >> > available.
> > > > > >> > > > But
> > > > > >> > > > > > >>we
> > > > > >> > > > > > >> > would
> > > > > >> > > > > > >> > > > > like to add REST server that
> > > > > >> > > > > > >> > > > > many users ask for under Apache Kafka repo.
> > > Many
> > > > > >>data
> > > > > >> > > Infra
> > > > > >> > > > > > >>tools
> > > > > >> > > > > > >> > comes
> > > > > >> > > > > > >> > > > up
> > > > > >> > > > > > >> > > > > with Rest Interface.
> > > > > >> > > > > > >> > > > > It is useful to have inbuilt Rest API
> support
> > > for
> > > > > >> > Produce,
> > > > > >> > > > > > >>Consume
> > > > > >> > > > > > >> > > > messages
> > > > > >> > > > > > >> > > > > and admin interface for
> > > > > >> > > > > > >> > > > > integrating with external management and
> > > > > >>provisioning
> > > > > >> > > > > tools.This
> > > > > >> > > > > > >> will
> > > > > >> > > > > > >> > > > also
> > > > > >> > > > > > >> > > > > allow the maintenance of
> > > > > >> > > > > > >> > > > > REST server and adding new features makes
> it
> > > easy
> > > > > >> > because
> > > > > >> > > > > apache
> > > > > >> > > > > > >> > > > community.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > The KIP wiki is the following:
> > > > > >> > > > > > >> > > > > https://cwiki.apache.org/
> > > > > >> confluence/display/KAFKA/KIP-
> > > > > >> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > Your comments and feedback are welcome.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > Thanks,
> > > > > >> > > > > > >> > > > > Manikumar
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > --
> > > > > >> > > > > > >> > Thanks,
> > > > > >> > > > > > >> > Neha
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > > The information contained in this email is strictly
> > > > > >>confidential
> > > > > >> > and
> > > > > >> > > > for
> > > > > >> > > > > > the use of the addressee only, unless otherwise
> > indicated.
> > > > If
> > > > > >you
> > > > > >> > are
> > > > > >> > > > not
> > > > > >> > > > > > the intended recipient, please do not read, copy, use
> or
> > > > > >disclose
> > > > > >> > to
> > > > > >> > > > > others
> > > > > >> > > > > > this message or any attachment. Please also notify the
> > > > sender
> > > > > >>by
> > > > > >> > > > replying
> > > > > >> > > > > > to this email or by telephone (+44(020 7896 0011) and
> > then
> > > > > >delete
> > > > > >> > the
> > > > > >> > > > > email
> > > > > >> > > > > > and any copies of it. Opinions, conclusion (etc) that
> do
> > > not
> > > > > >> relate
> > > > > >> > > to
> > > > > >> > > > > the
> > > > > >> > > > > > official business of this company shall be understood
> as
> > > > > >>neither
> > > > > >> > > given
> > > > > >> > > > > nor
> > > > > >> > > > > > endorsed by it. IG is a trading name of IG Markets
> > Limited
> > > > (a
> > > > > >> > company
> > > > > >> > > > > > registered in England and Wales, company number
> > 04008957)
> > > > and
> > > > > >>IG
> > > > > >> > > Index
> > > > > >> > > > > > Limited (a company registered in England and Wales,
> > > company
> > > > > >> number
> > > > > >> > > > > > 01190902). Registered address at Cannon Bridge House,
> 25
> > > > > >>Dowgate
> > > > > >> > > Hill,
> > > > > >> > > > > > London EC4R 2YA. Both IG Markets Limited (register
> > number
> > > > > >195355)
> > > > > >> > and
> > > > > >> > > > IG
> > > > > >> > > > > > Index Limited (register number 114059) are authorised
> > and
> > > > > >> regulated
> > > > > >> > > by
> > > > > >> > > > > the
> > > > > >> > > > > > Financial Conduct Authority.
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > > The information contained in this email is strictly
> > confidential
> > > > and
> > > > > >> for
> > > > > >> > > the use of the addressee only, unless otherwise indicated.
> If
> > > you
> > > > > >>are
> > > > > >> not
> > > > > >> > > the intended recipient, please do not read, copy, use or
> > > disclose
> > > > to
> > > > > >> > others
> > > > > >> > > this message or any attachment. Please also notify the
> sender
> > by
> > > > > >> replying
> > > > > >> > > to this email or by telephone (+44(020 7896 0011) and then
> > > delete
> > > > > >>the
> > > > > >> > email
> > > > > >> > > and any copies of it. Opinions, conclusion (etc) that do not
> > > > relate
> > > > > >>to
> > > > > >> > the
> > > > > >> > > official business of this company shall be understood as
> > neither
> > > > > >>given
> > > > > >> > nor
> > > > > >> > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > > > > >>company
> > > > > >> > > registered in England and Wales, company number 04008957)
> and
> > IG
> > > > > >>Index
> > > > > >> > > Limited (a company registered in England and Wales, company
> > > number
> > > > > >> > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > > > >>Hill,
> > > > > >> > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355)
> > > > > >>and
> > > > > >> IG
> > > > > >> > > Index Limited (register number 114059) are authorised and
> > > > regulated
> > > > > >>by
> > > > > >> > the
> > > > > >> > > Financial Conduct Authority.
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Nacho (Ignacio) Solis
> > > > > >> Kafka
> > > > > >> nsolis@linkedin.com
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> > > This email, including attachments, is private and confidential. If you
> > have
> > > received this email in error please notify the sender and delete it
> from
> > > your system. Emails are not secure and may contain viruses. No
> liability
> > > can be accepted for viruses that might be transferred by this email or
> > any
> > > attachment. Any unauthorised copying of this message or unauthorised
> > > distribution and publication of the information contained herein are
> > > prohibited.
> > >
> > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> > > Registered in England and Wales. Registered No. 04843573.
> > >
> >
>
> --
>
>
> This email, including attachments, is private and confidential. If you have
> received this email in error please notify the sender and delete it from
> your system. Emails are not secure and may contain viruses. No liability
> can be accepted for viruses that might be transferred by this email or any
> attachment. Any unauthorised copying of this message or unauthorised
> distribution and publication of the information contained herein are
> prohibited.
>
> 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> Registered in England and Wales. Registered No. 04843573.
>
--
Thanks,
Ewen
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Ben Davison <be...@7digital.com>.
Good point Ismael :D
On Mon, Oct 24, 2016 at 11:07 PM, Ismael Juma <is...@juma.me.uk> wrote:
> If we want to start a vote on this, I suggest starting a [VOTE] thread
> instead of mentioning it in the middle of a very long [DISCUSS] thread. :)
>
> Ismael
>
> On Mon, Oct 24, 2016 at 9:46 PM, Ben Davison <be...@7digital.com>
> wrote:
>
> > This KIP has got muddy on differing opinions of what should be part of
> > Kafka etc. I agree with Suresh, let's have a vote on if we actually want
> a
> > REST client in Kafka core, and then we can work out what it actually
> looks
> > like (personally I would like to reuse Confluents, great REST client if
> > they donate it to ASF)
> >
> > For me +1 on REST client, this is a fundamental feature that's missing
> from
> > Kafka.
> >
> > On Mon, Oct 24, 2016 at 9:22 PM, Jay Kreps <ja...@confluent.io> wrote:
> >
> > > Hey Suresh,
> > >
> > > I think we agree that REST apis are useful. I don't think we agree that
> > > they need to be part of the Kafka project. We've had this discussion a
> > > dozen odd times for different protocols---AMQP, REST, MQTT. Going back
> > the
> > > last five years we've always rejected these. That doesn't mean they
> > aren't
> > > useful, I think having these as separate is fine, they don't have any
> > > complex interaction with Kafka, they just use the vanilla APIs like any
> > of
> > > the dozens of things on the ecosystem page.
> > >
> > > In terms of how they're developed. I think there are two discussions
> > here:
> > > 1. Separate project or not
> > > 2. Standalone Apache project or github
> > >
> > > The first of those I just talked about---I think this makes sense as an
> > > independent project.
> > >
> > > For the second of these, I actually don't think that spawning off a
> bunch
> > > of itty-bitty independent Apache projects is a good thing. I think you
> > can
> > > see this in the Hadoop ecosystem where the parts kind of all evolve off
> > in
> > > different directions and don't really work together as a whole. I think
> > > that makes sense for deep infrastructure projects, but not for these
> > little
> > > helper projects. I think a hub and spoke model where you have a central
> > > project (Kafka) and then a bunch of helper tools that strive to fit in
> > with
> > > it (in terms of config, monitoring, apis, and every other conventions)
> is
> > > actually much better. In any case there are already so many of these
> > tools
> > > (we capture maybe 20% of them on the ecosystem page), made by so many
> > > people, and virtually all developed in this style, it is a bit late to
> > put
> > > the cat back in the bag.
> > >
> > > Perhaps it's just a difference in background. For my part I had many
> > years
> > > successfully running github-style projects, and i think that model can
> > work
> > > quite well for small things. I do think it is important for these
> > projects
> > > to clarify governance, which we should absolutely do, but realistically
> > it
> > > is a pretty simple tool, there isn't a huge governance challenge for
> > > something like this because its scope is so limited ("take http
> requests,
> > > make Kafka requests").
> > >
> > > More specifically, I don't think there is an actual problem being
> solved
> > > here. I haven't heard any complaint about direction or patches not
> > getting
> > > added. The only complaint I've heard is missing features where the
> normal
> > > "patches accepted" rule applies. I would urge people to actually get
> > > involved in contribution here. In the future if there is a situation
> > where
> > > people don't like the direction of a given tool, they can fork it and
> > > either turn it into an independent Apache project or develop it
> > > independently, trying to do that preemptively seems a bit hostile.
> > >
> > > -Jay
> > >
> > > On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas <
> > suresh@hortonworks.com>
> > > wrote:
> > >
> > > > I am dividing this discussion into two parts:
> > > > 1. REST APIs as core Apache Kafka capability
> > > > This should be a core Kafka functionality. Same view has been
> reflected
> > > by
> > > > others (users and developers) as well. While we can debate whether
> > other
> > > > capabilities are core Kafka (Streams, Connect), it would be good
> > separate
> > > > that out and to keep this discussion focussed on REST APIs as
> proposed
> > by
> > > > this KIP. If there is ambivalence about the need for this in core
> > kafka,
> > > > we could have voting in the mailing list.
> > > >
> > > > Can we get an agreement on this? I am +1 on REST API in Apache Kafka.
> > > >
> > > > 2. Community where Kafka REST APIs need to be collaborated on
> > > > There is already a GitHub project that caters to Kafka REST APIs.
> But a
> > > > company owned Github is less than ideal for collaboration due to lack
> > of
> > > > governance, open community and roadmap. This is reflected by many
> > people
> > > > interested in this functionality and still not contributing to and
> > > > adopting the code base in the GitHub. I think trying overlay the ASF
> > > > governance model on GitHub project, which is what the need is, seems
> > > > unnecessary, if the code can be part of Apache Kafka.
> > > >
> > > > The question is, would Confluent be okay with licensing/contributing
> > the
> > > > code to Kafka project (assuming either Confluent or another
> contributor
> > > > can work on it)? I see clear intent in collaborating with others on
> > REST
> > > > APIs from confluent. Why not do it in Kafka project under ASF? This
> > takes
> > > > away all the barrier to collaboration? Alternatively, if Confluent is
> > not
> > > > willing to contribute the code from GitHub, would anyone veto
> building
> > a
> > > > separate REST API functionality in ASF Kafka? There are enough people
> > who
> > > > want to work on this and maintain it.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > >
> > > >
> > > > On 10/21/16, 9:41 PM, "Harsha Chintalapani" <ka...@harsha.io> wrote:
> > > >
> > > > >Sriram,
> > > > > "Can the streaming platform exist without stream processing? -
> No.
> > > > >Processing stream data again is a core part of streaming platform."
> > > > >
> > > > >Yes, it can. There are no.of Stream processing frameworks out there,
> > and
> > > > >they all have integration into Kafka.
> > > > >It doesn't need to be developed within Kafka.
> > > > >
> > > > >
> > > > >"Can the platform exist without the rest proxy? - Yes. The proxy
> does
> > > not
> > > > >complete the platform vision in anyway. It is just a good to have
> tool
> > > > >that
> > > > >might be required by quite a few users and there is an active
> project
> > > that
> > > > >works on this - https://github.com/confluentinc/kafka-rest"
> > > > >
> > > > >The rest proxy is as important as any API. The vision that shown
> here
> > > > >http://kafka.apache.org/intro
> > > > >require users to write the producers and consumers to get their data
> > > into
> > > > >and out of Kafka, without which having Streams or Connect won't help
> > > > >anyone.
> > > > >The rest proxy makes easier for users get their data into Kafka.
> > > > >Adding the rest proxy to the project doesn't invalidate the current
> > > > >vision,
> > > > >it only strengthens it.
> > > > >
> > > > >Thanks,
> > > > >Harsha
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <
> ram@confluent.io>
> > > > >wrote:
> > > > >
> > > > >FWIW, Apache Kafka has evolved a lot from where it started. It did
> > start
> > > > >as
> > > > >a messaging system. Over time we realized that that the vision for
> > Kafka
> > > > >is
> > > > >to build a streaming platform and not just a messaging system. You
> can
> > > > >take
> > > > >a look at the site for more description about what comprises the
> > > streaming
> > > > >platform http://kafka.apache.org/ and http://kafka.apache.org/intro
> .
> > > > >
> > > > >Can the streaming platform exist without Connect? - No. Data
> > integration
> > > > >is
> > > > >fundamental to building an end to end platform
> > > > >
> > > > >Can the streaming platform exist without stream processing? - No.
> > > > >Processing stream data again is a core part of streaming platform.
> > > > >
> > > > >Can the streaming platform exist without clients? - We at least need
> > one
> > > > >client library to complete the platform. Our Java clients help us to
> > > > >complete the platform story. The rest of the clients are built and
> > > > >maintained outside the project.
> > > > >
> > > > >Can the platform exist without the rest proxy? - Yes. The proxy does
> > not
> > > > >complete the platform vision in anyway. It is just a good to have
> tool
> > > > >that
> > > > >might be required by quite a few users and there is an active
> project
> > > that
> > > > >works on this - https://github.com/confluentinc/kafka-rest
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > > ><ns...@linkedin.com.invalid>
> > > > >wrote:
> > > > >
> > > > >> Are you saying Kafka REST is subjective but Kafka Streams and
> Kafka
> > > > >Connect
> > > > >> are not subjective?
> > > > >>
> > > > >> > "there are likely places that can live without a rest proxy"
> > > > >>
> > > > >> There are also places that can live without Kafka Streams and
> Kafka
> > > > >> Connect.
> > > > >>
> > > > >> Nacho
> > > > >>
> > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > >>
> > > > >> > At the high level, I think ideally it makes sense to add a
> > component
> > > > >>to
> > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight
> > > > >integration
> > > > >> > with Kafka core. For Kafka Stream, we do expect stream
> processing
> > > will
> > > > >be
> > > > >> > used widely in the future. Implementation wise, Kafka Stream
> only
> > > > >> supports
> > > > >> > getting data from Kafka and leverages quite a few of the core
> > > > >> > functionalities in Kafka core. For example, it uses customized
> > > > >>rebalance
> > > > >> > callback in the consumer and uses the compacted topic heavily.
> So,
> > > > >having
> > > > >> > Kafka Stream in the same repo makes it easier for testing when
> > those
> > > > >core
> > > > >> > functionalities evolve over time. Kafka Connect is in the same
> > > > >situation.
> > > > >> >
> > > > >> > For rest proxy, whether it's widely used or not is going to be a
> > bit
> > > > >> > subjective. However, there are likely places that can live
> > without a
> > > > >rest
> > > > >> > proxy. The rest proxy is just a proxy for the regular clients
> and
> > > > >doesn't
> > > > >> > need to be tightly integrated with Kafka core. So, the case for
> > > > >including
> > > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka
> > Stream
> > > > >>and
> > > > >> > Kafka Connect.
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Jun
> > > > >> >
> > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > > >><Mi...@ig.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > So from my reading essentially the first question needs to
> > > > >answered/and
> > > > >> > > voted on is:
> > > > >> > >
> > > > >> > > Is Apache Kafka Community only about the Core or does the
> apache
> > > > >> > community
> > > > >> > > also support some subprojects (and just we need some better
> way
> > to
> > > > >> manage
> > > > >> > > this)
> > > > >> > >
> > > > >> > > If vote for Core only wins, then the following should be
> > removed:
> > > > >> > > Kafka Connect
> > > > >> > > Kafka Stream
> > > > >> > >
> > > > >> > > If vote for Core only loses (aka we will support subprojects)
> > > then:
> > > > >> > > We should look to add Kafka Rest
> > > > >> > >
> > > > >> > > And we should look to see how we can manage better govern and
> > > manage
> > > > >> > > submodules.
> > > > >> > >
> > > > >> > > A good example which id propose here is how some other
> > communities
> > > > >>in
> > > > >> > > Apache do this.
> > > > >> > >
> > > > >> > > Each Module has a Module Management Committee(MMC), this is
> like
> > > > >almost
> > > > >> > > the PMC but at a per module basis.
> > > > >> > >
> > > > >> > > This MMC should essentially hold the binding votes for that
> > > module.
> > > > >> > > The MMC should be made up of a single representative from each
> > > > >> > > organisation (so no single organisation can fully veto the
> > > > >>community
> > > > >> it
> > > > >> > > has to a genuine consenus)
> > > > >> > > The MMC requires at least 3 members (so there cant be a tied
> > vote
> > > on
> > > > >2)
> > > > >> > > For a new Module to be added a MMC committee should be sought
> > > > >> > > A new Module is only capable of being added if the above
> > > > >>requirements
> > > > >> can
> > > > >> > > be met (e.g. 3 people wishing to step up, from 3
> organisations)
> > so
> > > > >that
> > > > >> > > only actively support modules would be added
> > > > >> > >
> > > > >> > > The PMC reviews each module every 6months or Year. If MMC is
> > > > >>inactive,
> > > > >> a
> > > > >> > > vote/call to find replacements if raised, if none are
> > forthcoming
> > > > >> > dropping
> > > > >> > > the MMC to less than 3 then the module moves to "the attic"
> > (very
> > > > >>much
> > > > >> > like
> > > > >> > > apache attic but a little more aggressively)
> > > > >> > >
> > > > >> > > This way the PMC does not need to micro manage every module
> > > > >> > > We only add modules where some amount of active support and
> > > > >maintenance
> > > > >> > > and use is provided by the community
> > > > >> > > We have an automatic way to retire old or inactive projects.
> > > > >> > >
> > > > >> > > Thoughts?
> > > > >> > > Mike
> > > > >> > >
> > > > >> > >
> > > > >> > > ________________________________________
> > > > >> > > From: Harsha Ch <ha...@gmail.com>
> > > > >> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > > >> > > To: dev@kafka.apache.org
> > > > >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >> > >
> > > > >> > > Jay,
> > > > >> > > REST API is something every user is in need of. If the
> > > > >>argument
> > > > >> is
> > > > >> > to
> > > > >> > > clone and write your API, this will do a disservice to the
> > users
> > > as
> > > > >> they
> > > > >> > > now have to choose one vs. others instead of keeping one API
> > that
> > > is
> > > > >> > > supported in Kafka community.
> > > > >> > >
> > > > >> > > "Pre-emptively re-creating another
> > > > >> > > REST layer when it seems like we all quite agree on what needs
> > to
> > > be
> > > > >> done
> > > > >> > > and we have an existing code base for HTTP/Kafka access that
> is
> > > > >heavily
> > > > >> > > used in production seems quite silly."
> > > > >> > >
> > > > >> > > Exactly our point. Why can't we develop this in Apache
> > > Kafka
> > > > >> > > community? Instead of us open sourcing another GitHub project
> > and
> > > > >> > creating
> > > > >> > > a divide in users and another version of API. Let's build this
> > in
> > > > >Kafka
> > > > >> > > Community and use the governance model that is proven to
> provide
> > > > >vendor
> > > > >> > > free user driven consensus features. The argument that is
> adding
> > > > >>this
> > > > >> > REST
> > > > >> > > server to Kafka will affect the agility of the project doesn't
> > mak
> > > > >> sense.
> > > > >> > >
> > > > >> > > It looks like your argument is either we develop all these
> small
> > > > >>tools
> > > > >> or
> > > > >> > > none at all. We as a community need to look at supporting
> > critical
> > > > >> > > tools/API. Instead of dividing this project into individual
> > > external
> > > > >> > > communities. We should build this as part of Kafka which best
> > > serves
> > > > >> the
> > > > >> > > needs of users.
> > > > >> > > The Streams and Connect projects that were pushed into
> > > Kafka
> > > > >> > could
> > > > >> > > have been left in their own Github projects based on your
> > > arguments.
> > > > >> What
> > > > >> > > about the REST API is so different that such that it should
> stay
> > > out
> > > > >of
> > > > >> > the
> > > > >> > > Kafka project? From my experience, more users are asking for
> the
> > > > >>REST
> > > > >> > API.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Harsha
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io>
> > > wrote:
> > > > >> > >
> > > > >> > > > I think the questions around governance make sense, I think
> we
> > > > >should
> > > > >> > > > really clarify that to make the process more clear so it can
> > be
> > > > >fully
> > > > >> > > > inclusive.
> > > > >> > > >
> > > > >> > > > The idea that we should not collaborate on what is there
> now,
> > > > >though,
> > > > >> > > > because in the future we might disagree about direction does
> > not
> > > > >> really
> > > > >> > > > make sense to me. If in the future we disagree, that is the
> > > beauty
> > > > >of
> > > > >> > > open
> > > > >> > > > source, you can always fork off a copy of the code and start
> > an
> > > > >> > > independent
> > > > >> > > > project either in Apache or elsewhere. Pre-emptively
> > re-creating
> > > > >> > another
> > > > >> > > > REST layer when it seems like we all quite agree on what
> needs
> > > to
> > > > >>be
> > > > >> > done
> > > > >> > > > and we have an existing code base for HTTP/kafka access that
> > is
> > > > >> heavily
> > > > >> > > > used in production seems quite silly.
> > > > >> > > >
> > > > >> > > > Let me give some background on how I at least think about
> > these
> > > > >> things.
> > > > >> > > > I've participated in open source projects out of LinkedIn
> via
> > > > >>github
> > > > >> as
> > > > >> > > > well as via the ASF. I don't think there is a "right" answer
> > to
> > > > >>how
> > > > >> to
> > > > >> > do
> > > > >> > > > these but rather some tradeoffs. We thought about this
> quite a
> > > lot
> > > > >in
> > > > >> > the
> > > > >> > > > context of Kafka based on the experience with the Hadoop
> > > ecosystem
> > > > >as
> > > > >> > > well
> > > > >> > > > as from other open source communities.
> > > > >> > > >
> > > > >> > > > There is a rich ecosystem around Kafka. Many of the projects
> > are
> > > > >> quite
> > > > >> > > > small--single clients or tools that do a single thing
> > well--and
> > > > >> almost
> > > > >> > > none
> > > > >> > > > of them are top level apache projects. I don't think trying
> to
> > > > >>force
> > > > >> > each
> > > > >> > > > of these to turn into independent Apache projects is
> > necessarily
> > > > >>the
> > > > >> > best
> > > > >> > > > thing for the ecosystem.
> > > > >> > > >
> > > > >> > > > My observation of how this can go wrong is really what I
> think
> > > has
> > > > >> > > happened
> > > > >> > > > to the Hadoop ecosystem. There you see quite a zoo of
> projects
> > > > >>which
> > > > >> > all
> > > > >> > > > drift apart and don't quite work together well. Coordinating
> > > even
> > > > >> > simple
> > > > >> > > > changes and standardization across these is exceptionally
> > > > >>difficult.
> > > > >> > The
> > > > >> > > > result is a bit of a mess for users--the pieces just don't
> > > really
> > > > >> come
> > > > >> > > > together very well. This makes sense for independent
> > > > >>infrastructure
> > > > >> > > systems
> > > > >> > > > (Kudu vs HDFS) but I'm not at all convinced that doing this
> > for
> > > > >every
> > > > >> > > > little tool or helper library has lead to a desirable
> state. I
> > > > >>think
> > > > >> > the
> > > > >> > > > mode of operating where the Hadoop vendors spawn off a few
> new
> > > > >Apache
> > > > >> > > > projects for each new product initiative, especially since
> > often
> > > > >that
> > > > >> > > > project is only valued by that vendor (and the other vendor
> > has
> > > a
> > > > >> > > different
> > > > >> > > > competing Apache project) doesn't necessarily do a better
> job
> > at
> > > > >> > > producing
> > > > >> > > > high quality communities or high quality software.
> > > > >> > > >
> > > > >> > > > These tools/connects/clients/proxies and other integration
> > > pieces
> > > > >can
> > > > >> > > take
> > > > >> > > > many forms, but my take of what makes one of these things
> good
> > > is
> > > > >> that
> > > > >> > it
> > > > >> > > > remains simple, does its one thing well, and cleaves as
> > closely
> > > as
> > > > >> > > possible
> > > > >> > > > to the conventions for Kafka itself--i.e. doesn't invent new
> > > ways
> > > > >>of
> > > > >> > > > monitoring, configuring, etc. For the tools we've
> contributed
> > > > >>we've
> > > > >> > tried
> > > > >> > > > really hard to make them consistent with Kafka as well as
> with
> > > > >>each
> > > > >> > other
> > > > >> > > > in how testing, configuration, monitoring, etc works.
> > > > >> > > >
> > > > >> > > > I think what Apache does superbly well is create a community
> > for
> > > > >> > > managing a
> > > > >> > > > large infrastructure layer like Kafka in a vendor
> independent
> > > way.
> > > > >> > What I
> > > > >> > > > think is less successful is attempting to form full and
> > > > >>independent
> > > > >> > > apache
> > > > >> > > > communities around very simple single purpose tools,
> > especially
> > > if
> > > > >> you
> > > > >> > > hope
> > > > >> > > > for these to come together into a cohesive toolset across
> > > multiple
> > > > >> such
> > > > >> > > > tools. Much of what Apache does--create a collective
> decision
> > > > >>making
> > > > >> > > > process for resolving disagreement, help to trademark and
> > > protect
> > > > >the
> > > > >> > > marks
> > > > >> > > > of the project, etc just isn't that relevant for simple
> > > > >> single-purpose
> > > > >> > > > tools.
> > > > >> > > >
> > > > >> > > > So my take is there are a couple of options:
> > > > >> > > >
> > > > >> > > > 1. We can try to put all the small tools into the Apache
> > > > >>Project.
> > > > >> I
> > > > >> > > > think this is not the right approach as there is simply
> too
> > > > >>many
> > > > >> of
> > > > >> > > > them,
> > > > >> > > > many in different languages, serving different protocols,
> > > > >> > integrating
> > > > >> > > > with
> > > > >> > > > particular systems, and a single community can't
> > effectively
> > > > >> > maintain
> > > > >> > > > them
> > > > >> > > > all. Doing this would significantly slow the progress of
> > the
> > > > >Kafka
> > > > >> > > > project.
> > > > >> > > > As a protocol for messaging, I don't really see a case
> for
> > > > >> including
> > > > >> > > > REST
> > > > >> > > > but not MQTT or AMQP which are technically much better
> > suited
> > > > >>to
> > > > >> > > > messaging
> > > > >> > > > and both are widely used for that.
> > > > >> > > > 2. We can treat ecosystem projects that aren't top level
> > > Apache
> > > > >> > > projects
> > > > >> > > > as invalid and try to recreate them all as Apache
> projects.
> > > > >> > Honestly,
> > > > >> > > > though, if you go to the Kafka ecosystem page virtually
> > none
> > > of
> > > > >> the
> > > > >> > > most
> > > > >> > > > popular add-ons to Kafka are Apache projects. The most
> > > > >>successful
> > > > >> > > > things in
> > > > >> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a
> > > number
> > > > >of
> > > > >> > > other
> > > > >> > > > clients, as well as the existing REST layer have
> succeeded
> > at
> > > > >> > > developing
> > > > >> > > > communities that actively contribute and use these pieces
> > > and I
> > > > >> > don't
> > > > >> > > > know
> > > > >> > > > that that is a bad thing unless that community proves to
> be
> > > > >> > > uninclusive,
> > > > >> > > > unresponsive, or goes in a bad technical direction--and
> > those
> > > > >>are
> > > > >> > > > failure
> > > > >> > > > modes that all open source efforts face.
> > > > >> > > > 3. We can do what I think makes the most sense and try to
> > > work
> > > > >> with
> > > > >> > > the
> > > > >> > > > projects that exist in the ecosystem and if the project
> > > doesn't
> > > > >> > have a
> > > > >> > > > responsive community or wants to go in a different
> > direction
> > > > >>fork
> > > > >> or
> > > > >> > > > recreate that work.
> > > > >> > > >
> > > > >> > > > Of course any person can choose whatever of these options
> they
> > > > >>want.
> > > > >> > But
> > > > >> > > > from my point of view, option (3) has been the path of the
> > > > >>community
> > > > >> so
> > > > >> > > far
> > > > >> > > > and I think it has been quite successful.
> > > > >> > > >
> > > > >> > > > -Jay
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> > > > >> kafka@harsha.io
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Neha,
> > > > >> > > > > "But I haven't seen any community emails or patches being
> > > > >submitted
> > > > >> > by
> > > > >> > > > you
> > > > >> > > > > guys, so I'm wondering why you are concerned about whether
> > the
> > > > >> > > community
> > > > >> > > > is
> > > > >> > > > > open to accepting patches or not."
> > > > >> > > > >
> > > > >> > > > > I think you are talking about contributing patches to this
> > > > >> repository
> > > > >> > > > > right? https://github.com/confluentinc/kafka-rest . All I
> > am
> > > > >> saying
> > > > >> > > > > the guidelines/governance model is not clear on the
> project
> > > and
> > > > >>I
> > > > >> > guess
> > > > >> > > > its
> > > > >> > > > > driven by opening a github issue request. Its the
> > repository
> > > > >owned
> > > > >> > by
> > > > >> > > > > confluent and as much I appreciate that the features we
> > > > >>mentioned
> > > > >> are
> > > > >> > > in
> > > > >> > > > > the roadmap and welcoming us to contribute to the project.
> > It
> > > > >> doesn't
> > > > >> > > > > gurantee what we want to add in the furture will be in
> your
> > > > >> roadmap.
> > > > >> > > > >
> > > > >> > > > > Hence the reason having it part of Kafka community will
> > help a
> > > > >>lot
> > > > >> as
> > > > >> > > > other
> > > > >> > > > > users can participate in the discussions. We are happy to
> > > drive
> > > > >> any
> > > > >> > > > > feature additions through KIPs which gives everyone a
> chance
> > > to
> > > > >> > > > participate
> > > > >> > > > > and add to the discussions.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Harsha
> > > > >> > > > >
> > > > >> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > > > >> > Michael.Pearce@ig.com>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > +1
> > > > >> > > > > >
> > > > >> > > > > > I agree on the governance comments whole heartedly.
> > > > >> > > > > >
> > > > >> > > > > > Also i agree about the contribution comments made
> earlier
> > in
> > > > >>the
> > > > >> > > > thread,
> > > > >> > > > > i
> > > > >> > > > > > personally am less likely to spend any of mine, or give
> > > > >>project
> > > > >> > time
> > > > >> > > > > within
> > > > >> > > > > > my internal projects to developers contributing to
> another
> > > > >> > commercial
> > > > >> > > > > > companies project even so technically open source, as
> then
> > > > >>there
> > > > >> is
> > > > >> > > > that
> > > > >> > > > > > commercial companies interest will always prevail and
> > > > >essentially
> > > > >> > can
> > > > >> > > > > > always have a final vote where disagreement. Im sure
> they
> > > > >>never
> > > > >> > > intend
> > > > >> > > > > to,
> > > > >> > > > > > but there is that true reality. This is why we have
> > > community
> > > > >> open
> > > > >> > > > source
> > > > >> > > > > > projects.
> > > > >> > > > > >
> > > > >> > > > > > I can find many different implementations now of a rest
> > > > >>endpoint
> > > > >> on
> > > > >> > > > > > GitHub, BitBucket etc. Each one has their benefits and
> > > > >> > disadvantages
> > > > >> > > in
> > > > >> > > > > > their implementation. By making / providing one this
> would
> > > > >>bring
> > > > >> > > > together
> > > > >> > > > > > these solutions, unifying those developers and also
> > bringing
> > > > >>the
> > > > >> > best
> > > > >> > > > of
> > > > >> > > > > > all.
> > > > >> > > > > >
> > > > >> > > > > > I understand the concern on the community burden
> > > > >> adding/supporting
> > > > >> > > more
> > > > >> > > > > > surface area for every client. But something like REST
> is
> > > > >> universal
> > > > >> > > and
> > > > >> > > > > > worthy to be owned by the community.
> > > > >> > > > > >
> > > > >> > > > > > Mike
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > ________________________________________
> > > > >> > > > > > From: Andrew Schofield <andrew_schofield_jira@
> outlook.com
> > >
> > > > >> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > >> > > > > > To: dev@kafka.apache.org
> > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >> > > > > >
> > > > >> > > > > > There's a massive difference between the governance of
> > Kafka
> > > > >>and
> > > > >> > the
> > > > >> > > > > > governance of the REST proxy.
> > > > >> > > > > >
> > > > >> > > > > > In Kafka, there is a broad community of people
> > contributing
> > > > >their
> > > > >> > > > > opinions
> > > > >> > > > > > about future enhancements in the form of KIPs. There's
> > some
> > > > >> really
> > > > >> > > deep
> > > > >> > > > > > consideration that goes into some of the trickier KIPs.
> > > There
> > > > >are
> > > > >> > > > people
> > > > >> > > > > > outside Confluent deeply knowledgeable in Kafka and
> > > building
> > > > >the
> > > > >> > > > > > reputations to become committers. I get the impression
> > that
> > > > >>the
> > > > >> > > roadmap
> > > > >> > > > > of
> > > > >> > > > > > Kafka is not really community-owned (what's the big
> > feature
> > > > >>for
> > > > >> > Kafka
> > > > >> > > > > 0.11,
> > > > >> > > > > > for example), but the conveyor belt of smaller features
> in
> > > the
> > > > >> form
> > > > >> > > of
> > > > >> > > > > KIPs
> > > > >> > > > > > works nicely. It's a good example of open-source
> working
> > > > >>well.
> > > > >> > > > > >
> > > > >> > > > > > The equivalent for the REST proxy is basically issues on
> > > > >>GitHub.
> > > > >> > The
> > > > >> > > > > > roadmap is less clear. There's not really a community
> > > properly
> > > > >> > > engaged
> > > > >> > > > in
> > > > >> > > > > > the way that there is with Kafka. So, you could say that
> > > it's
> > > > >> clear
> > > > >> > > > that
> > > > >> > > > > > fewer people are interested, but I think the whole
> > > governance
> > > > >> > thing
> > > > >> > > > is a
> > > > >> > > > > > big barrier to engagement. And it's looking like it's
> > > getting
> > > > >out
> > > > >> > of
> > > > >> > > > > date.
> > > > >> > > > > >
> > > > >> > > > > > In technical terms, I can think of two big improvements
> to
> > > the
> > > > >> REST
> > > > >> > > > > proxy.
> > > > >> > > > > > First, it needs to use the new consumer API so that it's
> > > > >possible
> > > > >> > to
> > > > >> > > > > secure
> > > > >> > > > > > connections between the REST proxy and Kafka. I don't
> care
> > > too
> > > > >> much
> > > > >> > > > which
> > > > >> > > > > > method calls it uses actually uses to consume messages,
> > > but I
> > > > >do
> > > > >> > > care
> > > > >> > > > > that
> > > > >> > > > > > I cannot secure connections because of network security
> > > rules.
> > > > >> > > Second,
> > > > >> > > > > > there's an affinity between a consumer and the instance
> of
> > > the
> > > > >> REST
> > > > >> > > > proxy
> > > > >> > > > > > to which it first connected. Kafka itself avoids this
> kind
> > > of
> > > > >> > > affinity
> > > > >> > > > > for
> > > > >> > > > > > good reason, and in the name of availability the REST
> > proxy
> > > > >> should
> > > > >> > > too.
> > > > >> > > > > > These are natural KIPs.
> > > > >> > > > > >
> > > > >> > > > > > I think it would be good to have the code for the REST
> > proxy
> > > > >> > > > contributed
> > > > >> > > > > > to Apache Kafka so that it would be able to be developed
> > in
> > > > >>the
> > > > >> > same
> > > > >> > > > way.
> > > > >> > > > > >
> > > > >> > > > > > Andrew Schofield
> > > > >> > > > > >
> > > > >> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > >> > > > > > Sent: 07 October 2016 22:41:52
> > > > >> > > > > > To: dev@kafka.apache.org
> > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >> > > > > >
> > > > >> > > > > > ASF already gives us a clear framework and governance
> > model
> > > > >>for
> > > > >> > > > community
> > > > >> > > > > > development. This is already understood by the people
> > > > >> contributing
> > > > >> > to
> > > > >> > > > > > Apache Kafka project, and they are the same people who
> > want
> > > to
> > > > >> > > > contribute
> > > > >> > > > > > to the REST server capability as well. Everyone is in
> > > > >>agreement
> > > > >> on
> > > > >> > > the
> > > > >> > > > > > need for collaborating on this effort. So why not
> > contribute
> > > > >>the
> > > > >> > code
> > > > >> > > > to
> > > > >> > > > > > Apache Kafka. This will help avoid duplication of effort
> > and
> > > > >> forks
> > > > >> > > that
> > > > >> > > > > > may crop up, hugely benefitting the user community. This
> > > will
> > > > >> also
> > > > >> > > > avoid
> > > > >> > > > > > having to define a process similar to ASF on a GitHub
> > > project
> > > > >and
> > > > >> > > > instead
> > > > >> > > > > > there is a single community with clear understanding
> > > community
> > > > >> > > process
> > > > >> > > > as
> > > > >> > > > > > defined in ASF.
> > > > >> > > > > >
> > > > >> > > > > > As others have said, this is an important capability for
> > > > >>Apache
> > > > >> > > Kafka.
> > > > >> > > > It
> > > > >> > > > > > is worth maintaining this as a part of the project.
> > > > >> > > > > >
> > > > >> > > > > > Regards,
> > > > >> > > > > > Suresh
> > > > >> > > > > >
> > > > >> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <
> ofir.manor@equalum.io>
> > > > >>wrote:
> > > > >> > > > > >
> > > > >> > > > > > >I personally think it would be quite wasteful to
> > > re-implement
> > > > >> the
> > > > >> > > REST
> > > > >> > > > > > >gateway just because that an actively-maintained piece
> of
> > > > >> > > > > Apache-licensed
> > > > >> > > > > > >software is not governed directly by the Apache Kafka
> > > > >> community...
> > > > >> > > > While
> > > > >> > > > > > >kafka-rest repo is owned by Confluent, the contributors
> > > > >> including
> > > > >> > > the
> > > > >> > > > > main
> > > > >> > > > > > >one are also part of the Apache Kafka community, so
> > there
> > > > >>is a
> > > > >> > > chance
> > > > >> > > > > to
> > > > >> > > > > > >work this out.
> > > > >> > > > > > >
> > > > >> > > > > > >However, there are two valid concerns here that could
> be
> > > > >> > addressed,
> > > > >> > > > > around
> > > > >> > > > > > >community and accessibility:
> > > > >> > > > > > >>> What we are worried about is a project
> > > > >> > > > > > >>> that's not maintained in a community. So the process
> > of
> > > > >> > accepting
> > > > >> > > > > > >>>patches
> > > > >> > > > > > >>> and priorities is not clear, and it's not developed
> in
> > > > >Apache
> > > > >> > > > > > >>>community.
> > > > >> > > > > > >>> Not only that, existing REST API project doesn't
> > support
> > > > >>new
> > > > >> > > client
> > > > >> > > > > API
> > > > >> > > > > > >and
> > > > >> > > > > > >>> hence there is no security support either.
> > > > >> > > > > > >
> > > > >> > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
> > > > >> community
> > > > >> > > can
> > > > >> > > > > > >clarify that - what is their contribution policy, dev
> > > style,
> > > > >> > roadmap
> > > > >> > > > > etc.
> > > > >> > > > > > >If they want, they can make an effort to encourage
> > > > >participation
> > > > >> > > from
> > > > >> > > > > > >people outside Confluent (easily accept contributions,
> > > invite
> > > > >> > > external
> > > > >> > > > > > >commiters or have open dev process similar to Apache
> > Kafka
> > > > >etc),
> > > > >> > as
> > > > >> > > > > there
> > > > >> > > > > > >is definitely seems to be some interest on the list.
> That
> > > > >>might
> > > > >> > > clear
> > > > >> > > > > the
> > > > >> > > > > > >community concern and help kafka-rest project (but that
> > is
> > > a
> > > > >> > > > calculation
> > > > >> > > > > > >Confluent will have to make).
> > > > >> > > > > > >
> > > > >> > > > > > >The other, independent, concern is that REST is
> something
> > > > >>that
> > > > >> is
> > > > >> > > > > expected
> > > > >> > > > > > >to be available out of the box with Kafka. I personally
> > > don't
> > > > >> feel
> > > > >> > > > > > >strongly
> > > > >> > > > > > >about it (better use proper, efficient APIs from day
> > one),
> > > > >> though
> > > > >> > it
> > > > >> > > > is
> > > > >> > > > > > >definitely way smaller than adding a stream processing
> > > engine
> > > > >to
> > > > >> > the
> > > > >> > > > > > >project :)
> > > > >> > > > > > >Again,the kafka-rest "community" could take steps to
> make
> > > it
> > > > >> even
> > > > >> > > > easier
> > > > >> > > > > > >to
> > > > >> > > > > > >install, configure and run kafka-rest for new users on
> > > > >>vanilla
> > > > >> > > Apache
> > > > >> > > > > > >Kafka
> > > > >> > > > > > >(outside the Confluent platform), if they wish that (or
> > > > >>welcome
> > > > >> > > > > > >contributions to that end), but that is up to them.
> > > > >> > > > > > >Finally, if after the above steps were taken there
> would
> > > > >>still
> > > > >a
> > > > >> > > > strong
> > > > >> > > > > > >desire to include a great rest gateway with Apache
> > Kafka, I
> > > > >> assume
> > > > >> > > the
> > > > >> > > > > > >community could hypothetically fork the existing
> > kafka-rest
> > > > >into
> > > > >> > an
> > > > >> > > > > Apache
> > > > >> > > > > > >Kafka subproject and maintain it "within Apache"
> instead
> > of
> > > > >> > > > implementing
> > > > >> > > > > > >it
> > > > >> > > > > > >from scratch (though I'm not a lawyer etc) - but I
> cannot
> > > > >> imagine
> > > > >> > it
> > > > >> > > > > > >happen
> > > > >> > > > > > >without Confluent blessing, and I think that is likely
> > much
> > > > >less
> > > > >> > > > optimal
> > > > >> > > > > > >(pulling in other Confluent / Apache licensed
> > dependencies)
> > > > >than
> > > > >> > > > having
> > > > >> > > > > a
> > > > >> > > > > > >separate external community around kafka-rest.
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >Just my two cents,
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >Ofir Manor
> > > > >> > > > > > >
> > > > >> > > > > > >Co-Founder & CTO | Equalum
> > > > >> > > > > > >
> > > > >> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > > > ><+972%2054-780-1286>
> > > > >> > <+972%2054-780-1286> |
> > > > >> > > > Email:
> > > > >> > > > > > ofir.manor@equalum.io
> > > > >> > > > > > >
> > > > >> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > > > >> > > kafka@harsha.io
> > > > >> > > > >
> > > > >> > > > > > >wrote:
> > > > >> > > > > > >
> > > > >> > > > > > >> Neha & Jay,
> > > > >> > > > > > >> We did look at the open source
> > > > >>alternatives.
> > > > >> > Our
> > > > >> > > > > > >>concern
> > > > >> > > > > > >> is what's the patch acceptance and adding features/
> > > > >>bug-fixes
> > > > >> to
> > > > >> > > the
> > > > >> > > > > > >> existing project under a Github (although it's
> licensed
> > > > >>under
> > > > >> > > Apache
> > > > >> > > > > > >>2.0).
> > > > >> > > > > > >> It would be great if that project made available
> under
> > > > >>Apache
> > > > >> > and
> > > > >> > > > > > >>driven by
> > > > >> > > > > > >> the community. Adding to the above, not all Kafka
> > users
> > > > >>are
> > > > >> > > > > interested
> > > > >> > > > > > >>in
> > > > >> > > > > > >> using the Java client API, they would like to have
> > simple
> > > > >REST
> > > > >> > API
> > > > >> > > > > where
> > > > >> > > > > > >> they can code against using any language. I do
> believe
> > > this
> > > > >> adds
> > > > >> > > > value
> > > > >> > > > > > >>to
> > > > >> > > > > > >> Apache Kafka in itself.
> > > > >> > > > > > >>
> > > > >> > > > > > >> "For 1, I don't think there is value in giving in to
> > the
> > > > >>NIH
> > > > >> > > > syndrome
> > > > >> > > > > > >>and
> > > > >> > > > > > >> reinventing the wheel. What I'm looking for is a
> > detailed
> > > > >> > > comparison
> > > > >> > > > > of
> > > > >> > > > > > >>the
> > > > >> > > > > > >> gaps and why those can't be improved in the REST
> proxy
> > > that
> > > > >> > > already
> > > > >> > > > > > >>exists
> > > > >> > > > > > >> and is actively maintained."
> > > > >> > > > > > >>
> > > > >> > > > > > >> We are not looking at this as NIH. What we are
> worried
> > > > >>about
> > > > >> > is a
> > > > >> > > > > > >>project
> > > > >> > > > > > >> that's not maintained in a community. So the process
> of
> > > > >> > accepting
> > > > >> > > > > > >>patches
> > > > >> > > > > > >> and priorities is not clear, and it's not developed
> in
> > > > >>Apache
> > > > >> > > > > community.
> > > > >> > > > > > >> Not only that, existing REST API project doesn't
> > support
> > > > >>new
> > > > >> > > client
> > > > >> > > > > API
> > > > >> > > > > > >>and
> > > > >> > > > > > >> hence there is no security support either.
> > > > >> > > > > > >> We don't know the timeline when that's made
> available.
> > We
> > > > >> would
> > > > >> > > like
> > > > >> > > > > to
> > > > >> > > > > > >>add
> > > > >> > > > > > >> admin functionality into the REST API. So the Roadmap
> > of
> > > > >>that
> > > > >> > > > project
> > > > >> > > > > is
> > > > >> > > > > > >> not driven by Apache.
> > > > >> > > > > > >>
> > > > >> > > > > > >>
> > > > >> > > > > > >> "This doesn't materially have an impact on expanding
> > the
> > > > >> > usability
> > > > >> > > > of
> > > > >> > > > > > >> Kafka. In my experience, REST proxy + Java clients
> > > only
> > > > >> cover
> > > > >> > > > ~50%
> > > > >> > > > > of
> > > > >> > > > > > >> all
> > > > >> > > > > > >> Kafka users, and maybe 10% of those are the ones
> who
> > > > >>will
> > > > >> use
> > > > >> > > the
> > > > >> > > > > > >>REST
> > > > >> > > > > > >> proxy. The remaining 50% are non-java client users
> > (C,
> > > > >> > python,
> > > > >> > > > go,
> > > > >> > > > > > >>node
> > > > >> > > > > > >> etc)."
> > > > >> > > > > > >>
> > > > >> > > > > > >> REST API is most often asked feature in my
> interactions
> > > > >>with
> > > > >> > Kafka
> > > > >> > > > > > >>users.
> > > > >> > > > > > >> In an organization, There will be independent teams
> who
> > > > >>will
> > > > >> > write
> > > > >> > > > > their
> > > > >> > > > > > >> Kafka clients using different language libraries
> > > available
> > > > >> > today,
> > > > >> > > > and
> > > > >> > > > > > >> there is no way to standardize this. Instead of
> > > supporting
> > > > >> > several
> > > > >> > > > > > >> different client libraries users will be interested
> in
> > > > >>using
> > > > >a
> > > > >> > > REST
> > > > >> > > > > API
> > > > >> > > > > > >> server. The need for a REST API will only increase as
> > > more
> > > > >and
> > > > >> > > more
> > > > >> > > > > > >>users
> > > > >> > > > > > >> start using Kafka.
> > > > >> > > > > > >>
> > > > >> > > > > > >> "More surface area means more work to keep things
> > > > >>consistent.
> > > > >> > > > Failure
> > > > >> > > > > > >> to do that has, in fact, hurt the user
> experience."
> > > > >> > > > > > >> Having myriad Kafka client GitHub projects that
> support
> > > > >> > different
> > > > >> > > > > > >>languages
> > > > >> > > > > > >> hurts the user experience and pushes burden to
> maintain
> > > > >>these
> > > > >> > > > > libraries.
> > > > >> > > > > > >> REST API is a simple code base that uses existing
> java
> > > > >>client
> > > > >> > > > > libraries
> > > > >> > > > > > >>to
> > > > >> > > > > > >> make life easier for the users.
> > > > >> > > > > > >>
> > > > >> > > > > > >> Thanks,
> > > > >> > > > > > >> Harsha
> > > > >> > > > > > >>
> > > > >> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > > > >> > neha@confluent.io>
> > > > >> > > > > > wrote:
> > > > >> > > > > > >>
> > > > >> > > > > > >> > Manikumar,
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > Thanks for sharing the proposal. I think there are
> 2
> > > > >>parts
> > > > >> to
> > > > >> > > this
> > > > >> > > > > > >> > discussion -
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > 1. Should we rewrite a REST proxy given that there
> > is a
> > > > >> > > > > > >>feature-complete,
> > > > >> > > > > > >> > open-source and actively maintained REST proxy in
> the
> > > > >> > community?
> > > > >> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us
> > > more
> > > > >> agile
> > > > >> > > and
> > > > >> > > > > > >> maintain
> > > > >> > > > > > >> > the high-quality experience that Kafka users have
> > > today?
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > For 1, I don't think there is value in giving in to
> > the
> > > > >>NIH
> > > > >> > > > syndrome
> > > > >> > > > > > >>and
> > > > >> > > > > > >> > reinventing the wheel. What I'm looking for is a
> > > detailed
> > > > >> > > > comparison
> > > > >> > > > > > >>of
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > gaps and why those can't be improved in the REST
> > proxy
> > > > >>that
> > > > >> > > > already
> > > > >> > > > > > >> exists
> > > > >> > > > > > >> > and is actively maintained. For example, we depend
> on
> > > > >> zkClient
> > > > >> > > and
> > > > >> > > > > > >>have
> > > > >> > > > > > >> > found as well as fixed several bugs by working
> > closely
> > > > >>with
> > > > >> > the
> > > > >> > > > > people
> > > > >> > > > > > >> who
> > > > >> > > > > > >> > maintain zkClient. This should be possible for REST
> > > proxy
> > > > >as
> > > > >> > > well,
> > > > >> > > > > > >>right?
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > For 2, I'd like us to review our history of
> expanding
> > > the
> > > > >> > > surface
> > > > >> > > > > > >>area to
> > > > >> > > > > > >> > add more clients in the past. Here is a summary -
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > - This doesn't materially have an impact on
> > > expanding
> > > > >the
> > > > >> > > > > > >>usability of
> > > > >> > > > > > >> > Kafka. In my experience, REST proxy + Java
> clients
> > > > >>only
> > > > >> > cover
> > > > >> > > > > ~50%
> > > > >> > > > > > >>of
> > > > >> > > > > > >> > all
> > > > >> > > > > > >> > Kafka users, and maybe 10% of those are the ones
> > who
> > > > >will
> > > > >> > use
> > > > >> > > > the
> > > > >> > > > > > >>REST
> > > > >> > > > > > >> > proxy. The remaining 50% are non-java client
> users
> > > (C,
> > > > >> > > python,
> > > > >> > > > > go,
> > > > >> > > > > > >> node
> > > > >> > > > > > >> > etc).
> > > > >> > > > > > >> > - People are a lot more excited about promising
> to
> > > > >> > contribute
> > > > >> > > > > while
> > > > >> > > > > > >> > adding the surface area but not on an ongoing
> > basis
> > > > >>down
> > > > >> > the
> > > > >> > > > > line.
> > > > >> > > > > > >> > - More surface area means more work to keep
> things
> > > > >> > > consistent.
> > > > >> > > > > > >>Failure
> > > > >> > > > > > >> > to do that has, in fact, hurt the user
> experience.
> > > > >> > > > > > >> > - More surface area hurts agility. We want to
> do a
> > > few
> > > > >> > things
> > > > >> > > > > > >>really
> > > > >> > > > > > >> > well as well as be agile to be able to build on
> > our
> > > > >>core
> > > > >> > > > > > >>competency.
> > > > >> > > > > > >> >
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > > >> > > > manikumar.reddy@gmail.com
> > > > >> > > > > >
> > > > >> > > > > > >> > wrote:
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > > Hi Jay,
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > Thanks for your reply.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > I agree that we can not add all the clients/tools
> > > > >> available
> > > > >> > in
> > > > >> > > > > > >> ecosystem
> > > > >> > > > > > >> > > page to Kafka repo itself. But we feel REST
> > Interface
> > > > >>is
> > > > >> > > > different
> > > > >> > > > > > >>from
> > > > >> > > > > > >> > > other clients/tools. Since any language that can
> > work
> > > > >with
> > > > >> > > HTTP
> > > > >> > > > > can
> > > > >> > > > > > >> > > easily integrate with this interface, Having an
> > > > >"official"
> > > > >> > > REST
> > > > >> > > > > > >> > > interface helps user community. This also helps
> us
> > to
> > > > >> > > integrate
> > > > >> > > > > well
> > > > >> > > > > > >> > > with external management and provisioning tools.
> > > > >>Apache
> > > > >> > Kafka
> > > > >> > > > > > >>release
> > > > >> > > > > > >> > > with Java clients + REST interface is sufficient
> > for
> > > > >>most
> > > > >> of
> > > > >> > > the
> > > > >> > > > > > >>user
> > > > >> > > > > > >> > > deployments/requirements. This helps users to
> deal
> > > with
> > > > >> less
> > > > >> > > > > number
> > > > >> > > > > > >> > > of distributions/builds.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > Thanks,
> > > > >> > > > > > >> > > Manikumar
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> > > > >> jay@confluent.io
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > > Hey guys,
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > There's already a REST interface maintained as
> a
> > > > >> separate
> > > > >> > > > > > >> project--it's
> > > > >> > > > > > >> > > > open source and apache licensed and actively
> > > > >>maintained
> > > > >> (
> > > > >> > > > > > >> > > > https://github.com/confluentinc/kafka-rest).
> > What
> > > is
> > > > >> > wrong
> > > > >> > > > with
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > >> > > > > > github.com
> > > > >> > > > > > The Kafka REST Proxy provides a RESTful interface to a
> > Kafka
> > > > >> > cluster.
> > > > >> > > > It
> > > > >> > > > > > makes it easy to produce and consume messages, view the
> > > state
> > > > >>of
> > > > >> > the
> > > > >> > > > > > cluster, and ...
> > > > >> > > > > >
> > > > >> > > > > > >> that?
> > > > >> > > > > > >> > > You
> > > > >> > > > > > >> > > > mentioned that there was some compatibility
> > > concern,
> > > > >but
> > > > >> > > > > > >> compatibility
> > > > >> > > > > > >> > > has
> > > > >> > > > > > >> > > > to do with the consumer protocol guarantees not
> > the
> > > > >repo
> > > > >> > the
> > > > >> > > > > code
> > > > >> > > > > > >>is
> > > > >> > > > > > >> > in,
> > > > >> > > > > > >> > > > right? Not sure that concern makes sense.
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > We could argue for adding pretty much anything
> > and
> > > > >> > > everything
> > > > >> > > > in
> > > > >> > > > > > >>the
> > > > >> > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure
> > > that
> > > > >> would
> > > > >> > > > make
> > > > >> > > > > > >>the
> > > > >> > > > > > >> > > project
> > > > >> > > > > > >> > > > more agile.
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > -Jay
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > >> > > > > > >> manikumar.reddy@gmail.com
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > > wrote:
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > > Hi Kafka Devs,
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to
> > > Kafka
> > > > >> > > > Repository.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > There are already open-source alternatives
> are
> > > > >> > available.
> > > > >> > > > But
> > > > >> > > > > > >>we
> > > > >> > > > > > >> > would
> > > > >> > > > > > >> > > > > like to add REST server that
> > > > >> > > > > > >> > > > > many users ask for under Apache Kafka repo.
> > Many
> > > > >>data
> > > > >> > > Infra
> > > > >> > > > > > >>tools
> > > > >> > > > > > >> > comes
> > > > >> > > > > > >> > > > up
> > > > >> > > > > > >> > > > > with Rest Interface.
> > > > >> > > > > > >> > > > > It is useful to have inbuilt Rest API support
> > for
> > > > >> > Produce,
> > > > >> > > > > > >>Consume
> > > > >> > > > > > >> > > > messages
> > > > >> > > > > > >> > > > > and admin interface for
> > > > >> > > > > > >> > > > > integrating with external management and
> > > > >>provisioning
> > > > >> > > > > tools.This
> > > > >> > > > > > >> will
> > > > >> > > > > > >> > > > also
> > > > >> > > > > > >> > > > > allow the maintenance of
> > > > >> > > > > > >> > > > > REST server and adding new features makes it
> > easy
> > > > >> > because
> > > > >> > > > > apache
> > > > >> > > > > > >> > > > community.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > The KIP wiki is the following:
> > > > >> > > > > > >> > > > > https://cwiki.apache.org/
> > > > >> confluence/display/KAFKA/KIP-
> > > > >> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Your comments and feedback are welcome.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Thanks,
> > > > >> > > > > > >> > > > > Manikumar
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > --
> > > > >> > > > > > >> > Thanks,
> > > > >> > > > > > >> > Neha
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > > The information contained in this email is strictly
> > > > >>confidential
> > > > >> > and
> > > > >> > > > for
> > > > >> > > > > > the use of the addressee only, unless otherwise
> indicated.
> > > If
> > > > >you
> > > > >> > are
> > > > >> > > > not
> > > > >> > > > > > the intended recipient, please do not read, copy, use or
> > > > >disclose
> > > > >> > to
> > > > >> > > > > others
> > > > >> > > > > > this message or any attachment. Please also notify the
> > > sender
> > > > >>by
> > > > >> > > > replying
> > > > >> > > > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > > > >delete
> > > > >> > the
> > > > >> > > > > email
> > > > >> > > > > > and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > > >> relate
> > > > >> > > to
> > > > >> > > > > the
> > > > >> > > > > > official business of this company shall be understood as
> > > > >>neither
> > > > >> > > given
> > > > >> > > > > nor
> > > > >> > > > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > > (a
> > > > >> > company
> > > > >> > > > > > registered in England and Wales, company number
> 04008957)
> > > and
> > > > >>IG
> > > > >> > > Index
> > > > >> > > > > > Limited (a company registered in England and Wales,
> > company
> > > > >> number
> > > > >> > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > > > >>Dowgate
> > > > >> > > Hill,
> > > > >> > > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > > >195355)
> > > > >> > and
> > > > >> > > > IG
> > > > >> > > > > > Index Limited (register number 114059) are authorised
> and
> > > > >> regulated
> > > > >> > > by
> > > > >> > > > > the
> > > > >> > > > > > Financial Conduct Authority.
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > The information contained in this email is strictly
> confidential
> > > and
> > > > >> for
> > > > >> > > the use of the addressee only, unless otherwise indicated. If
> > you
> > > > >>are
> > > > >> not
> > > > >> > > the intended recipient, please do not read, copy, use or
> > disclose
> > > to
> > > > >> > others
> > > > >> > > this message or any attachment. Please also notify the sender
> by
> > > > >> replying
> > > > >> > > to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > > >>the
> > > > >> > email
> > > > >> > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate
> > > > >>to
> > > > >> > the
> > > > >> > > official business of this company shall be understood as
> neither
> > > > >>given
> > > > >> > nor
> > > > >> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > > >>company
> > > > >> > > registered in England and Wales, company number 04008957) and
> IG
> > > > >>Index
> > > > >> > > Limited (a company registered in England and Wales, company
> > number
> > > > >> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > > > >>Hill,
> > > > >> > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > >>and
> > > > >> IG
> > > > >> > > Index Limited (register number 114059) are authorised and
> > > regulated
> > > > >>by
> > > > >> > the
> > > > >> > > Financial Conduct Authority.
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Nacho (Ignacio) Solis
> > > > >> Kafka
> > > > >> nsolis@linkedin.com
> > > > >>
> > > >
> > > >
> > > >
> > >
> >
> > --
> >
> >
> > This email, including attachments, is private and confidential. If you
> have
> > received this email in error please notify the sender and delete it from
> > your system. Emails are not secure and may contain viruses. No liability
> > can be accepted for viruses that might be transferred by this email or
> any
> > attachment. Any unauthorised copying of this message or unauthorised
> > distribution and publication of the information contained herein are
> > prohibited.
> >
> > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> > Registered in England and Wales. Registered No. 04843573.
> >
>
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Ismael Juma <is...@juma.me.uk>.
If we want to start a vote on this, I suggest starting a [VOTE] thread
instead of mentioning it in the middle of a very long [DISCUSS] thread. :)
Ismael
On Mon, Oct 24, 2016 at 9:46 PM, Ben Davison <be...@7digital.com>
wrote:
> This KIP has got muddy on differing opinions of what should be part of
> Kafka etc. I agree with Suresh, let's have a vote on if we actually want a
> REST client in Kafka core, and then we can work out what it actually looks
> like (personally I would like to reuse Confluents, great REST client if
> they donate it to ASF)
>
> For me +1 on REST client, this is a fundamental feature that's missing from
> Kafka.
>
> On Mon, Oct 24, 2016 at 9:22 PM, Jay Kreps <ja...@confluent.io> wrote:
>
> > Hey Suresh,
> >
> > I think we agree that REST apis are useful. I don't think we agree that
> > they need to be part of the Kafka project. We've had this discussion a
> > dozen odd times for different protocols---AMQP, REST, MQTT. Going back
> the
> > last five years we've always rejected these. That doesn't mean they
> aren't
> > useful, I think having these as separate is fine, they don't have any
> > complex interaction with Kafka, they just use the vanilla APIs like any
> of
> > the dozens of things on the ecosystem page.
> >
> > In terms of how they're developed. I think there are two discussions
> here:
> > 1. Separate project or not
> > 2. Standalone Apache project or github
> >
> > The first of those I just talked about---I think this makes sense as an
> > independent project.
> >
> > For the second of these, I actually don't think that spawning off a bunch
> > of itty-bitty independent Apache projects is a good thing. I think you
> can
> > see this in the Hadoop ecosystem where the parts kind of all evolve off
> in
> > different directions and don't really work together as a whole. I think
> > that makes sense for deep infrastructure projects, but not for these
> little
> > helper projects. I think a hub and spoke model where you have a central
> > project (Kafka) and then a bunch of helper tools that strive to fit in
> with
> > it (in terms of config, monitoring, apis, and every other conventions) is
> > actually much better. In any case there are already so many of these
> tools
> > (we capture maybe 20% of them on the ecosystem page), made by so many
> > people, and virtually all developed in this style, it is a bit late to
> put
> > the cat back in the bag.
> >
> > Perhaps it's just a difference in background. For my part I had many
> years
> > successfully running github-style projects, and i think that model can
> work
> > quite well for small things. I do think it is important for these
> projects
> > to clarify governance, which we should absolutely do, but realistically
> it
> > is a pretty simple tool, there isn't a huge governance challenge for
> > something like this because its scope is so limited ("take http requests,
> > make Kafka requests").
> >
> > More specifically, I don't think there is an actual problem being solved
> > here. I haven't heard any complaint about direction or patches not
> getting
> > added. The only complaint I've heard is missing features where the normal
> > "patches accepted" rule applies. I would urge people to actually get
> > involved in contribution here. In the future if there is a situation
> where
> > people don't like the direction of a given tool, they can fork it and
> > either turn it into an independent Apache project or develop it
> > independently, trying to do that preemptively seems a bit hostile.
> >
> > -Jay
> >
> > On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas <
> suresh@hortonworks.com>
> > wrote:
> >
> > > I am dividing this discussion into two parts:
> > > 1. REST APIs as core Apache Kafka capability
> > > This should be a core Kafka functionality. Same view has been reflected
> > by
> > > others (users and developers) as well. While we can debate whether
> other
> > > capabilities are core Kafka (Streams, Connect), it would be good
> separate
> > > that out and to keep this discussion focussed on REST APIs as proposed
> by
> > > this KIP. If there is ambivalence about the need for this in core
> kafka,
> > > we could have voting in the mailing list.
> > >
> > > Can we get an agreement on this? I am +1 on REST API in Apache Kafka.
> > >
> > > 2. Community where Kafka REST APIs need to be collaborated on
> > > There is already a GitHub project that caters to Kafka REST APIs. But a
> > > company owned Github is less than ideal for collaboration due to lack
> of
> > > governance, open community and roadmap. This is reflected by many
> people
> > > interested in this functionality and still not contributing to and
> > > adopting the code base in the GitHub. I think trying overlay the ASF
> > > governance model on GitHub project, which is what the need is, seems
> > > unnecessary, if the code can be part of Apache Kafka.
> > >
> > > The question is, would Confluent be okay with licensing/contributing
> the
> > > code to Kafka project (assuming either Confluent or another contributor
> > > can work on it)? I see clear intent in collaborating with others on
> REST
> > > APIs from confluent. Why not do it in Kafka project under ASF? This
> takes
> > > away all the barrier to collaboration? Alternatively, if Confluent is
> not
> > > willing to contribute the code from GitHub, would anyone veto building
> a
> > > separate REST API functionality in ASF Kafka? There are enough people
> who
> > > want to work on this and maintain it.
> > >
> > > Regards,
> > > Suresh
> > >
> > >
> > >
> > > On 10/21/16, 9:41 PM, "Harsha Chintalapani" <ka...@harsha.io> wrote:
> > >
> > > >Sriram,
> > > > "Can the streaming platform exist without stream processing? - No.
> > > >Processing stream data again is a core part of streaming platform."
> > > >
> > > >Yes, it can. There are no.of Stream processing frameworks out there,
> and
> > > >they all have integration into Kafka.
> > > >It doesn't need to be developed within Kafka.
> > > >
> > > >
> > > >"Can the platform exist without the rest proxy? - Yes. The proxy does
> > not
> > > >complete the platform vision in anyway. It is just a good to have tool
> > > >that
> > > >might be required by quite a few users and there is an active project
> > that
> > > >works on this - https://github.com/confluentinc/kafka-rest"
> > > >
> > > >The rest proxy is as important as any API. The vision that shown here
> > > >http://kafka.apache.org/intro
> > > >require users to write the producers and consumers to get their data
> > into
> > > >and out of Kafka, without which having Streams or Connect won't help
> > > >anyone.
> > > >The rest proxy makes easier for users get their data into Kafka.
> > > >Adding the rest proxy to the project doesn't invalidate the current
> > > >vision,
> > > >it only strengthens it.
> > > >
> > > >Thanks,
> > > >Harsha
> > > >
> > > >
> > > >
> > > >
> > > >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <ra...@confluent.io>
> > > >wrote:
> > > >
> > > >FWIW, Apache Kafka has evolved a lot from where it started. It did
> start
> > > >as
> > > >a messaging system. Over time we realized that that the vision for
> Kafka
> > > >is
> > > >to build a streaming platform and not just a messaging system. You can
> > > >take
> > > >a look at the site for more description about what comprises the
> > streaming
> > > >platform http://kafka.apache.org/ and http://kafka.apache.org/intro.
> > > >
> > > >Can the streaming platform exist without Connect? - No. Data
> integration
> > > >is
> > > >fundamental to building an end to end platform
> > > >
> > > >Can the streaming platform exist without stream processing? - No.
> > > >Processing stream data again is a core part of streaming platform.
> > > >
> > > >Can the streaming platform exist without clients? - We at least need
> one
> > > >client library to complete the platform. Our Java clients help us to
> > > >complete the platform story. The rest of the clients are built and
> > > >maintained outside the project.
> > > >
> > > >Can the platform exist without the rest proxy? - Yes. The proxy does
> not
> > > >complete the platform vision in anyway. It is just a good to have tool
> > > >that
> > > >might be required by quite a few users and there is an active project
> > that
> > > >works on this - https://github.com/confluentinc/kafka-rest
> > > >
> > > >
> > > >
> > > >
> > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > ><ns...@linkedin.com.invalid>
> > > >wrote:
> > > >
> > > >> Are you saying Kafka REST is subjective but Kafka Streams and Kafka
> > > >Connect
> > > >> are not subjective?
> > > >>
> > > >> > "there are likely places that can live without a rest proxy"
> > > >>
> > > >> There are also places that can live without Kafka Streams and Kafka
> > > >> Connect.
> > > >>
> > > >> Nacho
> > > >>
> > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io> wrote:
> > > >>
> > > >> > At the high level, I think ideally it makes sense to add a
> component
> > > >>to
> > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight
> > > >integration
> > > >> > with Kafka core. For Kafka Stream, we do expect stream processing
> > will
> > > >be
> > > >> > used widely in the future. Implementation wise, Kafka Stream only
> > > >> supports
> > > >> > getting data from Kafka and leverages quite a few of the core
> > > >> > functionalities in Kafka core. For example, it uses customized
> > > >>rebalance
> > > >> > callback in the consumer and uses the compacted topic heavily. So,
> > > >having
> > > >> > Kafka Stream in the same repo makes it easier for testing when
> those
> > > >core
> > > >> > functionalities evolve over time. Kafka Connect is in the same
> > > >situation.
> > > >> >
> > > >> > For rest proxy, whether it's widely used or not is going to be a
> bit
> > > >> > subjective. However, there are likely places that can live
> without a
> > > >rest
> > > >> > proxy. The rest proxy is just a proxy for the regular clients and
> > > >doesn't
> > > >> > need to be tightly integrated with Kafka core. So, the case for
> > > >including
> > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka
> Stream
> > > >>and
> > > >> > Kafka Connect.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Jun
> > > >> >
> > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > >><Mi...@ig.com>
> > > >> > wrote:
> > > >> >
> > > >> > > So from my reading essentially the first question needs to
> > > >answered/and
> > > >> > > voted on is:
> > > >> > >
> > > >> > > Is Apache Kafka Community only about the Core or does the apache
> > > >> > community
> > > >> > > also support some subprojects (and just we need some better way
> to
> > > >> manage
> > > >> > > this)
> > > >> > >
> > > >> > > If vote for Core only wins, then the following should be
> removed:
> > > >> > > Kafka Connect
> > > >> > > Kafka Stream
> > > >> > >
> > > >> > > If vote for Core only loses (aka we will support subprojects)
> > then:
> > > >> > > We should look to add Kafka Rest
> > > >> > >
> > > >> > > And we should look to see how we can manage better govern and
> > manage
> > > >> > > submodules.
> > > >> > >
> > > >> > > A good example which id propose here is how some other
> communities
> > > >>in
> > > >> > > Apache do this.
> > > >> > >
> > > >> > > Each Module has a Module Management Committee(MMC), this is like
> > > >almost
> > > >> > > the PMC but at a per module basis.
> > > >> > >
> > > >> > > This MMC should essentially hold the binding votes for that
> > module.
> > > >> > > The MMC should be made up of a single representative from each
> > > >> > > organisation (so no single organisation can fully veto the
> > > >>community
> > > >> it
> > > >> > > has to a genuine consenus)
> > > >> > > The MMC requires at least 3 members (so there cant be a tied
> vote
> > on
> > > >2)
> > > >> > > For a new Module to be added a MMC committee should be sought
> > > >> > > A new Module is only capable of being added if the above
> > > >>requirements
> > > >> can
> > > >> > > be met (e.g. 3 people wishing to step up, from 3 organisations)
> so
> > > >that
> > > >> > > only actively support modules would be added
> > > >> > >
> > > >> > > The PMC reviews each module every 6months or Year. If MMC is
> > > >>inactive,
> > > >> a
> > > >> > > vote/call to find replacements if raised, if none are
> forthcoming
> > > >> > dropping
> > > >> > > the MMC to less than 3 then the module moves to "the attic"
> (very
> > > >>much
> > > >> > like
> > > >> > > apache attic but a little more aggressively)
> > > >> > >
> > > >> > > This way the PMC does not need to micro manage every module
> > > >> > > We only add modules where some amount of active support and
> > > >maintenance
> > > >> > > and use is provided by the community
> > > >> > > We have an automatic way to retire old or inactive projects.
> > > >> > >
> > > >> > > Thoughts?
> > > >> > > Mike
> > > >> > >
> > > >> > >
> > > >> > > ________________________________________
> > > >> > > From: Harsha Ch <ha...@gmail.com>
> > > >> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > >> > > To: dev@kafka.apache.org
> > > >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >> > >
> > > >> > > Jay,
> > > >> > > REST API is something every user is in need of. If the
> > > >>argument
> > > >> is
> > > >> > to
> > > >> > > clone and write your API, this will do a disservice to the
> users
> > as
> > > >> they
> > > >> > > now have to choose one vs. others instead of keeping one API
> that
> > is
> > > >> > > supported in Kafka community.
> > > >> > >
> > > >> > > "Pre-emptively re-creating another
> > > >> > > REST layer when it seems like we all quite agree on what needs
> to
> > be
> > > >> done
> > > >> > > and we have an existing code base for HTTP/Kafka access that is
> > > >heavily
> > > >> > > used in production seems quite silly."
> > > >> > >
> > > >> > > Exactly our point. Why can't we develop this in Apache
> > Kafka
> > > >> > > community? Instead of us open sourcing another GitHub project
> and
> > > >> > creating
> > > >> > > a divide in users and another version of API. Let's build this
> in
> > > >Kafka
> > > >> > > Community and use the governance model that is proven to provide
> > > >vendor
> > > >> > > free user driven consensus features. The argument that is adding
> > > >>this
> > > >> > REST
> > > >> > > server to Kafka will affect the agility of the project doesn't
> mak
> > > >> sense.
> > > >> > >
> > > >> > > It looks like your argument is either we develop all these small
> > > >>tools
> > > >> or
> > > >> > > none at all. We as a community need to look at supporting
> critical
> > > >> > > tools/API. Instead of dividing this project into individual
> > external
> > > >> > > communities. We should build this as part of Kafka which best
> > serves
> > > >> the
> > > >> > > needs of users.
> > > >> > > The Streams and Connect projects that were pushed into
> > Kafka
> > > >> > could
> > > >> > > have been left in their own Github projects based on your
> > arguments.
> > > >> What
> > > >> > > about the REST API is so different that such that it should stay
> > out
> > > >of
> > > >> > the
> > > >> > > Kafka project? From my experience, more users are asking for the
> > > >>REST
> > > >> > API.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Harsha
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io>
> > wrote:
> > > >> > >
> > > >> > > > I think the questions around governance make sense, I think we
> > > >should
> > > >> > > > really clarify that to make the process more clear so it can
> be
> > > >fully
> > > >> > > > inclusive.
> > > >> > > >
> > > >> > > > The idea that we should not collaborate on what is there now,
> > > >though,
> > > >> > > > because in the future we might disagree about direction does
> not
> > > >> really
> > > >> > > > make sense to me. If in the future we disagree, that is the
> > beauty
> > > >of
> > > >> > > open
> > > >> > > > source, you can always fork off a copy of the code and start
> an
> > > >> > > independent
> > > >> > > > project either in Apache or elsewhere. Pre-emptively
> re-creating
> > > >> > another
> > > >> > > > REST layer when it seems like we all quite agree on what needs
> > to
> > > >>be
> > > >> > done
> > > >> > > > and we have an existing code base for HTTP/kafka access that
> is
> > > >> heavily
> > > >> > > > used in production seems quite silly.
> > > >> > > >
> > > >> > > > Let me give some background on how I at least think about
> these
> > > >> things.
> > > >> > > > I've participated in open source projects out of LinkedIn via
> > > >>github
> > > >> as
> > > >> > > > well as via the ASF. I don't think there is a "right" answer
> to
> > > >>how
> > > >> to
> > > >> > do
> > > >> > > > these but rather some tradeoffs. We thought about this quite a
> > lot
> > > >in
> > > >> > the
> > > >> > > > context of Kafka based on the experience with the Hadoop
> > ecosystem
> > > >as
> > > >> > > well
> > > >> > > > as from other open source communities.
> > > >> > > >
> > > >> > > > There is a rich ecosystem around Kafka. Many of the projects
> are
> > > >> quite
> > > >> > > > small--single clients or tools that do a single thing
> well--and
> > > >> almost
> > > >> > > none
> > > >> > > > of them are top level apache projects. I don't think trying to
> > > >>force
> > > >> > each
> > > >> > > > of these to turn into independent Apache projects is
> necessarily
> > > >>the
> > > >> > best
> > > >> > > > thing for the ecosystem.
> > > >> > > >
> > > >> > > > My observation of how this can go wrong is really what I think
> > has
> > > >> > > happened
> > > >> > > > to the Hadoop ecosystem. There you see quite a zoo of projects
> > > >>which
> > > >> > all
> > > >> > > > drift apart and don't quite work together well. Coordinating
> > even
> > > >> > simple
> > > >> > > > changes and standardization across these is exceptionally
> > > >>difficult.
> > > >> > The
> > > >> > > > result is a bit of a mess for users--the pieces just don't
> > really
> > > >> come
> > > >> > > > together very well. This makes sense for independent
> > > >>infrastructure
> > > >> > > systems
> > > >> > > > (Kudu vs HDFS) but I'm not at all convinced that doing this
> for
> > > >every
> > > >> > > > little tool or helper library has lead to a desirable state. I
> > > >>think
> > > >> > the
> > > >> > > > mode of operating where the Hadoop vendors spawn off a few new
> > > >Apache
> > > >> > > > projects for each new product initiative, especially since
> often
> > > >that
> > > >> > > > project is only valued by that vendor (and the other vendor
> has
> > a
> > > >> > > different
> > > >> > > > competing Apache project) doesn't necessarily do a better job
> at
> > > >> > > producing
> > > >> > > > high quality communities or high quality software.
> > > >> > > >
> > > >> > > > These tools/connects/clients/proxies and other integration
> > pieces
> > > >can
> > > >> > > take
> > > >> > > > many forms, but my take of what makes one of these things good
> > is
> > > >> that
> > > >> > it
> > > >> > > > remains simple, does its one thing well, and cleaves as
> closely
> > as
> > > >> > > possible
> > > >> > > > to the conventions for Kafka itself--i.e. doesn't invent new
> > ways
> > > >>of
> > > >> > > > monitoring, configuring, etc. For the tools we've contributed
> > > >>we've
> > > >> > tried
> > > >> > > > really hard to make them consistent with Kafka as well as with
> > > >>each
> > > >> > other
> > > >> > > > in how testing, configuration, monitoring, etc works.
> > > >> > > >
> > > >> > > > I think what Apache does superbly well is create a community
> for
> > > >> > > managing a
> > > >> > > > large infrastructure layer like Kafka in a vendor independent
> > way.
> > > >> > What I
> > > >> > > > think is less successful is attempting to form full and
> > > >>independent
> > > >> > > apache
> > > >> > > > communities around very simple single purpose tools,
> especially
> > if
> > > >> you
> > > >> > > hope
> > > >> > > > for these to come together into a cohesive toolset across
> > multiple
> > > >> such
> > > >> > > > tools. Much of what Apache does--create a collective decision
> > > >>making
> > > >> > > > process for resolving disagreement, help to trademark and
> > protect
> > > >the
> > > >> > > marks
> > > >> > > > of the project, etc just isn't that relevant for simple
> > > >> single-purpose
> > > >> > > > tools.
> > > >> > > >
> > > >> > > > So my take is there are a couple of options:
> > > >> > > >
> > > >> > > > 1. We can try to put all the small tools into the Apache
> > > >>Project.
> > > >> I
> > > >> > > > think this is not the right approach as there is simply too
> > > >>many
> > > >> of
> > > >> > > > them,
> > > >> > > > many in different languages, serving different protocols,
> > > >> > integrating
> > > >> > > > with
> > > >> > > > particular systems, and a single community can't
> effectively
> > > >> > maintain
> > > >> > > > them
> > > >> > > > all. Doing this would significantly slow the progress of
> the
> > > >Kafka
> > > >> > > > project.
> > > >> > > > As a protocol for messaging, I don't really see a case for
> > > >> including
> > > >> > > > REST
> > > >> > > > but not MQTT or AMQP which are technically much better
> suited
> > > >>to
> > > >> > > > messaging
> > > >> > > > and both are widely used for that.
> > > >> > > > 2. We can treat ecosystem projects that aren't top level
> > Apache
> > > >> > > projects
> > > >> > > > as invalid and try to recreate them all as Apache projects.
> > > >> > Honestly,
> > > >> > > > though, if you go to the Kafka ecosystem page virtually
> none
> > of
> > > >> the
> > > >> > > most
> > > >> > > > popular add-ons to Kafka are Apache projects. The most
> > > >>successful
> > > >> > > > things in
> > > >> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a
> > number
> > > >of
> > > >> > > other
> > > >> > > > clients, as well as the existing REST layer have succeeded
> at
> > > >> > > developing
> > > >> > > > communities that actively contribute and use these pieces
> > and I
> > > >> > don't
> > > >> > > > know
> > > >> > > > that that is a bad thing unless that community proves to be
> > > >> > > uninclusive,
> > > >> > > > unresponsive, or goes in a bad technical direction--and
> those
> > > >>are
> > > >> > > > failure
> > > >> > > > modes that all open source efforts face.
> > > >> > > > 3. We can do what I think makes the most sense and try to
> > work
> > > >> with
> > > >> > > the
> > > >> > > > projects that exist in the ecosystem and if the project
> > doesn't
> > > >> > have a
> > > >> > > > responsive community or wants to go in a different
> direction
> > > >>fork
> > > >> or
> > > >> > > > recreate that work.
> > > >> > > >
> > > >> > > > Of course any person can choose whatever of these options they
> > > >>want.
> > > >> > But
> > > >> > > > from my point of view, option (3) has been the path of the
> > > >>community
> > > >> so
> > > >> > > far
> > > >> > > > and I think it has been quite successful.
> > > >> > > >
> > > >> > > > -Jay
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> > > >> kafka@harsha.io
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Neha,
> > > >> > > > > "But I haven't seen any community emails or patches being
> > > >submitted
> > > >> > by
> > > >> > > > you
> > > >> > > > > guys, so I'm wondering why you are concerned about whether
> the
> > > >> > > community
> > > >> > > > is
> > > >> > > > > open to accepting patches or not."
> > > >> > > > >
> > > >> > > > > I think you are talking about contributing patches to this
> > > >> repository
> > > >> > > > > right? https://github.com/confluentinc/kafka-rest . All I
> am
> > > >> saying
> > > >> > > > > the guidelines/governance model is not clear on the project
> > and
> > > >>I
> > > >> > guess
> > > >> > > > its
> > > >> > > > > driven by opening a github issue request. Its the
> repository
> > > >owned
> > > >> > by
> > > >> > > > > confluent and as much I appreciate that the features we
> > > >>mentioned
> > > >> are
> > > >> > > in
> > > >> > > > > the roadmap and welcoming us to contribute to the project.
> It
> > > >> doesn't
> > > >> > > > > gurantee what we want to add in the furture will be in your
> > > >> roadmap.
> > > >> > > > >
> > > >> > > > > Hence the reason having it part of Kafka community will
> help a
> > > >>lot
> > > >> as
> > > >> > > > other
> > > >> > > > > users can participate in the discussions. We are happy to
> > drive
> > > >> any
> > > >> > > > > feature additions through KIPs which gives everyone a chance
> > to
> > > >> > > > participate
> > > >> > > > > and add to the discussions.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Harsha
> > > >> > > > >
> > > >> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > > >> > Michael.Pearce@ig.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > +1
> > > >> > > > > >
> > > >> > > > > > I agree on the governance comments whole heartedly.
> > > >> > > > > >
> > > >> > > > > > Also i agree about the contribution comments made earlier
> in
> > > >>the
> > > >> > > > thread,
> > > >> > > > > i
> > > >> > > > > > personally am less likely to spend any of mine, or give
> > > >>project
> > > >> > time
> > > >> > > > > within
> > > >> > > > > > my internal projects to developers contributing to another
> > > >> > commercial
> > > >> > > > > > companies project even so technically open source, as then
> > > >>there
> > > >> is
> > > >> > > > that
> > > >> > > > > > commercial companies interest will always prevail and
> > > >essentially
> > > >> > can
> > > >> > > > > > always have a final vote where disagreement. Im sure they
> > > >>never
> > > >> > > intend
> > > >> > > > > to,
> > > >> > > > > > but there is that true reality. This is why we have
> > community
> > > >> open
> > > >> > > > source
> > > >> > > > > > projects.
> > > >> > > > > >
> > > >> > > > > > I can find many different implementations now of a rest
> > > >>endpoint
> > > >> on
> > > >> > > > > > GitHub, BitBucket etc. Each one has their benefits and
> > > >> > disadvantages
> > > >> > > in
> > > >> > > > > > their implementation. By making / providing one this would
> > > >>bring
> > > >> > > > together
> > > >> > > > > > these solutions, unifying those developers and also
> bringing
> > > >>the
> > > >> > best
> > > >> > > > of
> > > >> > > > > > all.
> > > >> > > > > >
> > > >> > > > > > I understand the concern on the community burden
> > > >> adding/supporting
> > > >> > > more
> > > >> > > > > > surface area for every client. But something like REST is
> > > >> universal
> > > >> > > and
> > > >> > > > > > worthy to be owned by the community.
> > > >> > > > > >
> > > >> > > > > > Mike
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > ________________________________________
> > > >> > > > > > From: Andrew Schofield <andrew_schofield_jira@outlook.com
> >
> > > >> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > >> > > > > > To: dev@kafka.apache.org
> > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >> > > > > >
> > > >> > > > > > There's a massive difference between the governance of
> Kafka
> > > >>and
> > > >> > the
> > > >> > > > > > governance of the REST proxy.
> > > >> > > > > >
> > > >> > > > > > In Kafka, there is a broad community of people
> contributing
> > > >their
> > > >> > > > > opinions
> > > >> > > > > > about future enhancements in the form of KIPs. There's
> some
> > > >> really
> > > >> > > deep
> > > >> > > > > > consideration that goes into some of the trickier KIPs.
> > There
> > > >are
> > > >> > > > people
> > > >> > > > > > outside Confluent deeply knowledgeable in Kafka and
> > building
> > > >the
> > > >> > > > > > reputations to become committers. I get the impression
> that
> > > >>the
> > > >> > > roadmap
> > > >> > > > > of
> > > >> > > > > > Kafka is not really community-owned (what's the big
> feature
> > > >>for
> > > >> > Kafka
> > > >> > > > > 0.11,
> > > >> > > > > > for example), but the conveyor belt of smaller features in
> > the
> > > >> form
> > > >> > > of
> > > >> > > > > KIPs
> > > >> > > > > > works nicely. It's a good example of open-source working
> > > >>well.
> > > >> > > > > >
> > > >> > > > > > The equivalent for the REST proxy is basically issues on
> > > >>GitHub.
> > > >> > The
> > > >> > > > > > roadmap is less clear. There's not really a community
> > properly
> > > >> > > engaged
> > > >> > > > in
> > > >> > > > > > the way that there is with Kafka. So, you could say that
> > it's
> > > >> clear
> > > >> > > > that
> > > >> > > > > > fewer people are interested, but I think the whole
> > governance
> > > >> > thing
> > > >> > > > is a
> > > >> > > > > > big barrier to engagement. And it's looking like it's
> > getting
> > > >out
> > > >> > of
> > > >> > > > > date.
> > > >> > > > > >
> > > >> > > > > > In technical terms, I can think of two big improvements to
> > the
> > > >> REST
> > > >> > > > > proxy.
> > > >> > > > > > First, it needs to use the new consumer API so that it's
> > > >possible
> > > >> > to
> > > >> > > > > secure
> > > >> > > > > > connections between the REST proxy and Kafka. I don't care
> > too
> > > >> much
> > > >> > > > which
> > > >> > > > > > method calls it uses actually uses to consume messages,
> > but I
> > > >do
> > > >> > > care
> > > >> > > > > that
> > > >> > > > > > I cannot secure connections because of network security
> > rules.
> > > >> > > Second,
> > > >> > > > > > there's an affinity between a consumer and the instance of
> > the
> > > >> REST
> > > >> > > > proxy
> > > >> > > > > > to which it first connected. Kafka itself avoids this kind
> > of
> > > >> > > affinity
> > > >> > > > > for
> > > >> > > > > > good reason, and in the name of availability the REST
> proxy
> > > >> should
> > > >> > > too.
> > > >> > > > > > These are natural KIPs.
> > > >> > > > > >
> > > >> > > > > > I think it would be good to have the code for the REST
> proxy
> > > >> > > > contributed
> > > >> > > > > > to Apache Kafka so that it would be able to be developed
> in
> > > >>the
> > > >> > same
> > > >> > > > way.
> > > >> > > > > >
> > > >> > > > > > Andrew Schofield
> > > >> > > > > >
> > > >> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > >> > > > > > Sent: 07 October 2016 22:41:52
> > > >> > > > > > To: dev@kafka.apache.org
> > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >> > > > > >
> > > >> > > > > > ASF already gives us a clear framework and governance
> model
> > > >>for
> > > >> > > > community
> > > >> > > > > > development. This is already understood by the people
> > > >> contributing
> > > >> > to
> > > >> > > > > > Apache Kafka project, and they are the same people who
> want
> > to
> > > >> > > > contribute
> > > >> > > > > > to the REST server capability as well. Everyone is in
> > > >>agreement
> > > >> on
> > > >> > > the
> > > >> > > > > > need for collaborating on this effort. So why not
> contribute
> > > >>the
> > > >> > code
> > > >> > > > to
> > > >> > > > > > Apache Kafka. This will help avoid duplication of effort
> and
> > > >> forks
> > > >> > > that
> > > >> > > > > > may crop up, hugely benefitting the user community. This
> > will
> > > >> also
> > > >> > > > avoid
> > > >> > > > > > having to define a process similar to ASF on a GitHub
> > project
> > > >and
> > > >> > > > instead
> > > >> > > > > > there is a single community with clear understanding
> > community
> > > >> > > process
> > > >> > > > as
> > > >> > > > > > defined in ASF.
> > > >> > > > > >
> > > >> > > > > > As others have said, this is an important capability for
> > > >>Apache
> > > >> > > Kafka.
> > > >> > > > It
> > > >> > > > > > is worth maintaining this as a part of the project.
> > > >> > > > > >
> > > >> > > > > > Regards,
> > > >> > > > > > Suresh
> > > >> > > > > >
> > > >> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io>
> > > >>wrote:
> > > >> > > > > >
> > > >> > > > > > >I personally think it would be quite wasteful to
> > re-implement
> > > >> the
> > > >> > > REST
> > > >> > > > > > >gateway just because that an actively-maintained piece of
> > > >> > > > > Apache-licensed
> > > >> > > > > > >software is not governed directly by the Apache Kafka
> > > >> community...
> > > >> > > > While
> > > >> > > > > > >kafka-rest repo is owned by Confluent, the contributors
> > > >> including
> > > >> > > the
> > > >> > > > > main
> > > >> > > > > > >one are also part of the Apache Kafka community, so
> there
> > > >>is a
> > > >> > > chance
> > > >> > > > > to
> > > >> > > > > > >work this out.
> > > >> > > > > > >
> > > >> > > > > > >However, there are two valid concerns here that could be
> > > >> > addressed,
> > > >> > > > > around
> > > >> > > > > > >community and accessibility:
> > > >> > > > > > >>> What we are worried about is a project
> > > >> > > > > > >>> that's not maintained in a community. So the process
> of
> > > >> > accepting
> > > >> > > > > > >>>patches
> > > >> > > > > > >>> and priorities is not clear, and it's not developed in
> > > >Apache
> > > >> > > > > > >>>community.
> > > >> > > > > > >>> Not only that, existing REST API project doesn't
> support
> > > >>new
> > > >> > > client
> > > >> > > > > API
> > > >> > > > > > >and
> > > >> > > > > > >>> hence there is no security support either.
> > > >> > > > > > >
> > > >> > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
> > > >> community
> > > >> > > can
> > > >> > > > > > >clarify that - what is their contribution policy, dev
> > style,
> > > >> > roadmap
> > > >> > > > > etc.
> > > >> > > > > > >If they want, they can make an effort to encourage
> > > >participation
> > > >> > > from
> > > >> > > > > > >people outside Confluent (easily accept contributions,
> > invite
> > > >> > > external
> > > >> > > > > > >commiters or have open dev process similar to Apache
> Kafka
> > > >etc),
> > > >> > as
> > > >> > > > > there
> > > >> > > > > > >is definitely seems to be some interest on the list. That
> > > >>might
> > > >> > > clear
> > > >> > > > > the
> > > >> > > > > > >community concern and help kafka-rest project (but that
> is
> > a
> > > >> > > > calculation
> > > >> > > > > > >Confluent will have to make).
> > > >> > > > > > >
> > > >> > > > > > >The other, independent, concern is that REST is something
> > > >>that
> > > >> is
> > > >> > > > > expected
> > > >> > > > > > >to be available out of the box with Kafka. I personally
> > don't
> > > >> feel
> > > >> > > > > > >strongly
> > > >> > > > > > >about it (better use proper, efficient APIs from day
> one),
> > > >> though
> > > >> > it
> > > >> > > > is
> > > >> > > > > > >definitely way smaller than adding a stream processing
> > engine
> > > >to
> > > >> > the
> > > >> > > > > > >project :)
> > > >> > > > > > >Again,the kafka-rest "community" could take steps to make
> > it
> > > >> even
> > > >> > > > easier
> > > >> > > > > > >to
> > > >> > > > > > >install, configure and run kafka-rest for new users on
> > > >>vanilla
> > > >> > > Apache
> > > >> > > > > > >Kafka
> > > >> > > > > > >(outside the Confluent platform), if they wish that (or
> > > >>welcome
> > > >> > > > > > >contributions to that end), but that is up to them.
> > > >> > > > > > >Finally, if after the above steps were taken there would
> > > >>still
> > > >a
> > > >> > > > strong
> > > >> > > > > > >desire to include a great rest gateway with Apache
> Kafka, I
> > > >> assume
> > > >> > > the
> > > >> > > > > > >community could hypothetically fork the existing
> kafka-rest
> > > >into
> > > >> > an
> > > >> > > > > Apache
> > > >> > > > > > >Kafka subproject and maintain it "within Apache" instead
> of
> > > >> > > > implementing
> > > >> > > > > > >it
> > > >> > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot
> > > >> imagine
> > > >> > it
> > > >> > > > > > >happen
> > > >> > > > > > >without Confluent blessing, and I think that is likely
> much
> > > >less
> > > >> > > > optimal
> > > >> > > > > > >(pulling in other Confluent / Apache licensed
> dependencies)
> > > >than
> > > >> > > > having
> > > >> > > > > a
> > > >> > > > > > >separate external community around kafka-rest.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >Just my two cents,
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >Ofir Manor
> > > >> > > > > > >
> > > >> > > > > > >Co-Founder & CTO | Equalum
> > > >> > > > > > >
> > > >> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > > ><+972%2054-780-1286>
> > > >> > <+972%2054-780-1286> |
> > > >> > > > Email:
> > > >> > > > > > ofir.manor@equalum.io
> > > >> > > > > > >
> > > >> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > > >> > > kafka@harsha.io
> > > >> > > > >
> > > >> > > > > > >wrote:
> > > >> > > > > > >
> > > >> > > > > > >> Neha & Jay,
> > > >> > > > > > >> We did look at the open source
> > > >>alternatives.
> > > >> > Our
> > > >> > > > > > >>concern
> > > >> > > > > > >> is what's the patch acceptance and adding features/
> > > >>bug-fixes
> > > >> to
> > > >> > > the
> > > >> > > > > > >> existing project under a Github (although it's licensed
> > > >>under
> > > >> > > Apache
> > > >> > > > > > >>2.0).
> > > >> > > > > > >> It would be great if that project made available under
> > > >>Apache
> > > >> > and
> > > >> > > > > > >>driven by
> > > >> > > > > > >> the community. Adding to the above, not all Kafka
> users
> > > >>are
> > > >> > > > > interested
> > > >> > > > > > >>in
> > > >> > > > > > >> using the Java client API, they would like to have
> simple
> > > >REST
> > > >> > API
> > > >> > > > > where
> > > >> > > > > > >> they can code against using any language. I do believe
> > this
> > > >> adds
> > > >> > > > value
> > > >> > > > > > >>to
> > > >> > > > > > >> Apache Kafka in itself.
> > > >> > > > > > >>
> > > >> > > > > > >> "For 1, I don't think there is value in giving in to
> the
> > > >>NIH
> > > >> > > > syndrome
> > > >> > > > > > >>and
> > > >> > > > > > >> reinventing the wheel. What I'm looking for is a
> detailed
> > > >> > > comparison
> > > >> > > > > of
> > > >> > > > > > >>the
> > > >> > > > > > >> gaps and why those can't be improved in the REST proxy
> > that
> > > >> > > already
> > > >> > > > > > >>exists
> > > >> > > > > > >> and is actively maintained."
> > > >> > > > > > >>
> > > >> > > > > > >> We are not looking at this as NIH. What we are worried
> > > >>about
> > > >> > is a
> > > >> > > > > > >>project
> > > >> > > > > > >> that's not maintained in a community. So the process of
> > > >> > accepting
> > > >> > > > > > >>patches
> > > >> > > > > > >> and priorities is not clear, and it's not developed in
> > > >>Apache
> > > >> > > > > community.
> > > >> > > > > > >> Not only that, existing REST API project doesn't
> support
> > > >>new
> > > >> > > client
> > > >> > > > > API
> > > >> > > > > > >>and
> > > >> > > > > > >> hence there is no security support either.
> > > >> > > > > > >> We don't know the timeline when that's made available.
> We
> > > >> would
> > > >> > > like
> > > >> > > > > to
> > > >> > > > > > >>add
> > > >> > > > > > >> admin functionality into the REST API. So the Roadmap
> of
> > > >>that
> > > >> > > > project
> > > >> > > > > is
> > > >> > > > > > >> not driven by Apache.
> > > >> > > > > > >>
> > > >> > > > > > >>
> > > >> > > > > > >> "This doesn't materially have an impact on expanding
> the
> > > >> > usability
> > > >> > > > of
> > > >> > > > > > >> Kafka. In my experience, REST proxy + Java clients
> > only
> > > >> cover
> > > >> > > > ~50%
> > > >> > > > > of
> > > >> > > > > > >> all
> > > >> > > > > > >> Kafka users, and maybe 10% of those are the ones who
> > > >>will
> > > >> use
> > > >> > > the
> > > >> > > > > > >>REST
> > > >> > > > > > >> proxy. The remaining 50% are non-java client users
> (C,
> > > >> > python,
> > > >> > > > go,
> > > >> > > > > > >>node
> > > >> > > > > > >> etc)."
> > > >> > > > > > >>
> > > >> > > > > > >> REST API is most often asked feature in my interactions
> > > >>with
> > > >> > Kafka
> > > >> > > > > > >>users.
> > > >> > > > > > >> In an organization, There will be independent teams who
> > > >>will
> > > >> > write
> > > >> > > > > their
> > > >> > > > > > >> Kafka clients using different language libraries
> > available
> > > >> > today,
> > > >> > > > and
> > > >> > > > > > >> there is no way to standardize this. Instead of
> > supporting
> > > >> > several
> > > >> > > > > > >> different client libraries users will be interested in
> > > >>using
> > > >a
> > > >> > > REST
> > > >> > > > > API
> > > >> > > > > > >> server. The need for a REST API will only increase as
> > more
> > > >and
> > > >> > > more
> > > >> > > > > > >>users
> > > >> > > > > > >> start using Kafka.
> > > >> > > > > > >>
> > > >> > > > > > >> "More surface area means more work to keep things
> > > >>consistent.
> > > >> > > > Failure
> > > >> > > > > > >> to do that has, in fact, hurt the user experience."
> > > >> > > > > > >> Having myriad Kafka client GitHub projects that support
> > > >> > different
> > > >> > > > > > >>languages
> > > >> > > > > > >> hurts the user experience and pushes burden to maintain
> > > >>these
> > > >> > > > > libraries.
> > > >> > > > > > >> REST API is a simple code base that uses existing java
> > > >>client
> > > >> > > > > libraries
> > > >> > > > > > >>to
> > > >> > > > > > >> make life easier for the users.
> > > >> > > > > > >>
> > > >> > > > > > >> Thanks,
> > > >> > > > > > >> Harsha
> > > >> > > > > > >>
> > > >> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > > >> > neha@confluent.io>
> > > >> > > > > > wrote:
> > > >> > > > > > >>
> > > >> > > > > > >> > Manikumar,
> > > >> > > > > > >> >
> > > >> > > > > > >> > Thanks for sharing the proposal. I think there are 2
> > > >>parts
> > > >> to
> > > >> > > this
> > > >> > > > > > >> > discussion -
> > > >> > > > > > >> >
> > > >> > > > > > >> > 1. Should we rewrite a REST proxy given that there
> is a
> > > >> > > > > > >>feature-complete,
> > > >> > > > > > >> > open-source and actively maintained REST proxy in the
> > > >> > community?
> > > >> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us
> > more
> > > >> agile
> > > >> > > and
> > > >> > > > > > >> maintain
> > > >> > > > > > >> > the high-quality experience that Kafka users have
> > today?
> > > >> > > > > > >> >
> > > >> > > > > > >> > For 1, I don't think there is value in giving in to
> the
> > > >>NIH
> > > >> > > > syndrome
> > > >> > > > > > >>and
> > > >> > > > > > >> > reinventing the wheel. What I'm looking for is a
> > detailed
> > > >> > > > comparison
> > > >> > > > > > >>of
> > > >> > > > > > >> the
> > > >> > > > > > >> > gaps and why those can't be improved in the REST
> proxy
> > > >>that
> > > >> > > > already
> > > >> > > > > > >> exists
> > > >> > > > > > >> > and is actively maintained. For example, we depend on
> > > >> zkClient
> > > >> > > and
> > > >> > > > > > >>have
> > > >> > > > > > >> > found as well as fixed several bugs by working
> closely
> > > >>with
> > > >> > the
> > > >> > > > > people
> > > >> > > > > > >> who
> > > >> > > > > > >> > maintain zkClient. This should be possible for REST
> > proxy
> > > >as
> > > >> > > well,
> > > >> > > > > > >>right?
> > > >> > > > > > >> >
> > > >> > > > > > >> > For 2, I'd like us to review our history of expanding
> > the
> > > >> > > surface
> > > >> > > > > > >>area to
> > > >> > > > > > >> > add more clients in the past. Here is a summary -
> > > >> > > > > > >> >
> > > >> > > > > > >> > - This doesn't materially have an impact on
> > expanding
> > > >the
> > > >> > > > > > >>usability of
> > > >> > > > > > >> > Kafka. In my experience, REST proxy + Java clients
> > > >>only
> > > >> > cover
> > > >> > > > > ~50%
> > > >> > > > > > >>of
> > > >> > > > > > >> > all
> > > >> > > > > > >> > Kafka users, and maybe 10% of those are the ones
> who
> > > >will
> > > >> > use
> > > >> > > > the
> > > >> > > > > > >>REST
> > > >> > > > > > >> > proxy. The remaining 50% are non-java client users
> > (C,
> > > >> > > python,
> > > >> > > > > go,
> > > >> > > > > > >> node
> > > >> > > > > > >> > etc).
> > > >> > > > > > >> > - People are a lot more excited about promising to
> > > >> > contribute
> > > >> > > > > while
> > > >> > > > > > >> > adding the surface area but not on an ongoing
> basis
> > > >>down
> > > >> > the
> > > >> > > > > line.
> > > >> > > > > > >> > - More surface area means more work to keep things
> > > >> > > consistent.
> > > >> > > > > > >>Failure
> > > >> > > > > > >> > to do that has, in fact, hurt the user experience.
> > > >> > > > > > >> > - More surface area hurts agility. We want to do a
> > few
> > > >> > things
> > > >> > > > > > >>really
> > > >> > > > > > >> > well as well as be agile to be able to build on
> our
> > > >>core
> > > >> > > > > > >>competency.
> > > >> > > > > > >> >
> > > >> > > > > > >> >
> > > >> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > >> > > > manikumar.reddy@gmail.com
> > > >> > > > > >
> > > >> > > > > > >> > wrote:
> > > >> > > > > > >> >
> > > >> > > > > > >> > > Hi Jay,
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > Thanks for your reply.
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > I agree that we can not add all the clients/tools
> > > >> available
> > > >> > in
> > > >> > > > > > >> ecosystem
> > > >> > > > > > >> > > page to Kafka repo itself. But we feel REST
> Interface
> > > >>is
> > > >> > > > different
> > > >> > > > > > >>from
> > > >> > > > > > >> > > other clients/tools. Since any language that can
> work
> > > >with
> > > >> > > HTTP
> > > >> > > > > can
> > > >> > > > > > >> > > easily integrate with this interface, Having an
> > > >"official"
> > > >> > > REST
> > > >> > > > > > >> > > interface helps user community. This also helps us
> to
> > > >> > > integrate
> > > >> > > > > well
> > > >> > > > > > >> > > with external management and provisioning tools.
> > > >>Apache
> > > >> > Kafka
> > > >> > > > > > >>release
> > > >> > > > > > >> > > with Java clients + REST interface is sufficient
> for
> > > >>most
> > > >> of
> > > >> > > the
> > > >> > > > > > >>user
> > > >> > > > > > >> > > deployments/requirements. This helps users to deal
> > with
> > > >> less
> > > >> > > > > number
> > > >> > > > > > >> > > of distributions/builds.
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > Thanks,
> > > >> > > > > > >> > > Manikumar
> > > >> > > > > > >> > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> > > >> jay@confluent.io
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > > Hey guys,
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > There's already a REST interface maintained as a
> > > >> separate
> > > >> > > > > > >> project--it's
> > > >> > > > > > >> > > > open source and apache licensed and actively
> > > >>maintained
> > > >> (
> > > >> > > > > > >> > > > https://github.com/confluentinc/kafka-rest).
> What
> > is
> > > >> > wrong
> > > >> > > > with
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > >> > > > > > github.com
> > > >> > > > > > The Kafka REST Proxy provides a RESTful interface to a
> Kafka
> > > >> > cluster.
> > > >> > > > It
> > > >> > > > > > makes it easy to produce and consume messages, view the
> > state
> > > >>of
> > > >> > the
> > > >> > > > > > cluster, and ...
> > > >> > > > > >
> > > >> > > > > > >> that?
> > > >> > > > > > >> > > You
> > > >> > > > > > >> > > > mentioned that there was some compatibility
> > concern,
> > > >but
> > > >> > > > > > >> compatibility
> > > >> > > > > > >> > > has
> > > >> > > > > > >> > > > to do with the consumer protocol guarantees not
> the
> > > >repo
> > > >> > the
> > > >> > > > > code
> > > >> > > > > > >>is
> > > >> > > > > > >> > in,
> > > >> > > > > > >> > > > right? Not sure that concern makes sense.
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > We could argue for adding pretty much anything
> and
> > > >> > > everything
> > > >> > > > in
> > > >> > > > > > >>the
> > > >> > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure
> > that
> > > >> would
> > > >> > > > make
> > > >> > > > > > >>the
> > > >> > > > > > >> > > project
> > > >> > > > > > >> > > > more agile.
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > -Jay
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > >> > > > > > >> manikumar.reddy@gmail.com
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > > wrote:
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > > Hi Kafka Devs,
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to
> > Kafka
> > > >> > > > Repository.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > There are already open-source alternatives are
> > > >> > available.
> > > >> > > > But
> > > >> > > > > > >>we
> > > >> > > > > > >> > would
> > > >> > > > > > >> > > > > like to add REST server that
> > > >> > > > > > >> > > > > many users ask for under Apache Kafka repo.
> Many
> > > >>data
> > > >> > > Infra
> > > >> > > > > > >>tools
> > > >> > > > > > >> > comes
> > > >> > > > > > >> > > > up
> > > >> > > > > > >> > > > > with Rest Interface.
> > > >> > > > > > >> > > > > It is useful to have inbuilt Rest API support
> for
> > > >> > Produce,
> > > >> > > > > > >>Consume
> > > >> > > > > > >> > > > messages
> > > >> > > > > > >> > > > > and admin interface for
> > > >> > > > > > >> > > > > integrating with external management and
> > > >>provisioning
> > > >> > > > > tools.This
> > > >> > > > > > >> will
> > > >> > > > > > >> > > > also
> > > >> > > > > > >> > > > > allow the maintenance of
> > > >> > > > > > >> > > > > REST server and adding new features makes it
> easy
> > > >> > because
> > > >> > > > > apache
> > > >> > > > > > >> > > > community.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > The KIP wiki is the following:
> > > >> > > > > > >> > > > > https://cwiki.apache.org/
> > > >> confluence/display/KAFKA/KIP-
> > > >> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > Your comments and feedback are welcome.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > Thanks,
> > > >> > > > > > >> > > > > Manikumar
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> > --
> > > >> > > > > > >> > Thanks,
> > > >> > > > > > >> > Neha
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > > The information contained in this email is strictly
> > > >>confidential
> > > >> > and
> > > >> > > > for
> > > >> > > > > > the use of the addressee only, unless otherwise indicated.
> > If
> > > >you
> > > >> > are
> > > >> > > > not
> > > >> > > > > > the intended recipient, please do not read, copy, use or
> > > >disclose
> > > >> > to
> > > >> > > > > others
> > > >> > > > > > this message or any attachment. Please also notify the
> > sender
> > > >>by
> > > >> > > > replying
> > > >> > > > > > to this email or by telephone (+44(020 7896 0011) and then
> > > >delete
> > > >> > the
> > > >> > > > > email
> > > >> > > > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > >> relate
> > > >> > > to
> > > >> > > > > the
> > > >> > > > > > official business of this company shall be understood as
> > > >>neither
> > > >> > > given
> > > >> > > > > nor
> > > >> > > > > > endorsed by it. IG is a trading name of IG Markets Limited
> > (a
> > > >> > company
> > > >> > > > > > registered in England and Wales, company number 04008957)
> > and
> > > >>IG
> > > >> > > Index
> > > >> > > > > > Limited (a company registered in England and Wales,
> company
> > > >> number
> > > >> > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > > >>Dowgate
> > > >> > > Hill,
> > > >> > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > >195355)
> > > >> > and
> > > >> > > > IG
> > > >> > > > > > Index Limited (register number 114059) are authorised and
> > > >> regulated
> > > >> > > by
> > > >> > > > > the
> > > >> > > > > > Financial Conduct Authority.
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > The information contained in this email is strictly confidential
> > and
> > > >> for
> > > >> > > the use of the addressee only, unless otherwise indicated. If
> you
> > > >>are
> > > >> not
> > > >> > > the intended recipient, please do not read, copy, use or
> disclose
> > to
> > > >> > others
> > > >> > > this message or any attachment. Please also notify the sender by
> > > >> replying
> > > >> > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > > >>the
> > > >> > email
> > > >> > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> > > >>to
> > > >> > the
> > > >> > > official business of this company shall be understood as neither
> > > >>given
> > > >> > nor
> > > >> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > >>company
> > > >> > > registered in England and Wales, company number 04008957) and IG
> > > >>Index
> > > >> > > Limited (a company registered in England and Wales, company
> number
> > > >> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > >>Hill,
> > > >> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > > >>and
> > > >> IG
> > > >> > > Index Limited (register number 114059) are authorised and
> > regulated
> > > >>by
> > > >> > the
> > > >> > > Financial Conduct Authority.
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Nacho (Ignacio) Solis
> > > >> Kafka
> > > >> nsolis@linkedin.com
> > > >>
> > >
> > >
> > >
> >
>
> --
>
>
> This email, including attachments, is private and confidential. If you have
> received this email in error please notify the sender and delete it from
> your system. Emails are not secure and may contain viruses. No liability
> can be accepted for viruses that might be transferred by this email or any
> attachment. Any unauthorised copying of this message or unauthorised
> distribution and publication of the information contained herein are
> prohibited.
>
> 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> Registered in England and Wales. Registered No. 04843573.
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Ben Davison <be...@7digital.com>.
This KIP has got muddy on differing opinions of what should be part of
Kafka etc. I agree with Suresh, let's have a vote on if we actually want a
REST client in Kafka core, and then we can work out what it actually looks
like (personally I would like to reuse Confluents, great REST client if
they donate it to ASF)
For me +1 on REST client, this is a fundamental feature that's missing from
Kafka.
On Mon, Oct 24, 2016 at 9:22 PM, Jay Kreps <ja...@confluent.io> wrote:
> Hey Suresh,
>
> I think we agree that REST apis are useful. I don't think we agree that
> they need to be part of the Kafka project. We've had this discussion a
> dozen odd times for different protocols---AMQP, REST, MQTT. Going back the
> last five years we've always rejected these. That doesn't mean they aren't
> useful, I think having these as separate is fine, they don't have any
> complex interaction with Kafka, they just use the vanilla APIs like any of
> the dozens of things on the ecosystem page.
>
> In terms of how they're developed. I think there are two discussions here:
> 1. Separate project or not
> 2. Standalone Apache project or github
>
> The first of those I just talked about---I think this makes sense as an
> independent project.
>
> For the second of these, I actually don't think that spawning off a bunch
> of itty-bitty independent Apache projects is a good thing. I think you can
> see this in the Hadoop ecosystem where the parts kind of all evolve off in
> different directions and don't really work together as a whole. I think
> that makes sense for deep infrastructure projects, but not for these little
> helper projects. I think a hub and spoke model where you have a central
> project (Kafka) and then a bunch of helper tools that strive to fit in with
> it (in terms of config, monitoring, apis, and every other conventions) is
> actually much better. In any case there are already so many of these tools
> (we capture maybe 20% of them on the ecosystem page), made by so many
> people, and virtually all developed in this style, it is a bit late to put
> the cat back in the bag.
>
> Perhaps it's just a difference in background. For my part I had many years
> successfully running github-style projects, and i think that model can work
> quite well for small things. I do think it is important for these projects
> to clarify governance, which we should absolutely do, but realistically it
> is a pretty simple tool, there isn't a huge governance challenge for
> something like this because its scope is so limited ("take http requests,
> make Kafka requests").
>
> More specifically, I don't think there is an actual problem being solved
> here. I haven't heard any complaint about direction or patches not getting
> added. The only complaint I've heard is missing features where the normal
> "patches accepted" rule applies. I would urge people to actually get
> involved in contribution here. In the future if there is a situation where
> people don't like the direction of a given tool, they can fork it and
> either turn it into an independent Apache project or develop it
> independently, trying to do that preemptively seems a bit hostile.
>
> -Jay
>
> On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
> > I am dividing this discussion into two parts:
> > 1. REST APIs as core Apache Kafka capability
> > This should be a core Kafka functionality. Same view has been reflected
> by
> > others (users and developers) as well. While we can debate whether other
> > capabilities are core Kafka (Streams, Connect), it would be good separate
> > that out and to keep this discussion focussed on REST APIs as proposed by
> > this KIP. If there is ambivalence about the need for this in core kafka,
> > we could have voting in the mailing list.
> >
> > Can we get an agreement on this? I am +1 on REST API in Apache Kafka.
> >
> > 2. Community where Kafka REST APIs need to be collaborated on
> > There is already a GitHub project that caters to Kafka REST APIs. But a
> > company owned Github is less than ideal for collaboration due to lack of
> > governance, open community and roadmap. This is reflected by many people
> > interested in this functionality and still not contributing to and
> > adopting the code base in the GitHub. I think trying overlay the ASF
> > governance model on GitHub project, which is what the need is, seems
> > unnecessary, if the code can be part of Apache Kafka.
> >
> > The question is, would Confluent be okay with licensing/contributing the
> > code to Kafka project (assuming either Confluent or another contributor
> > can work on it)? I see clear intent in collaborating with others on REST
> > APIs from confluent. Why not do it in Kafka project under ASF? This takes
> > away all the barrier to collaboration? Alternatively, if Confluent is not
> > willing to contribute the code from GitHub, would anyone veto building a
> > separate REST API functionality in ASF Kafka? There are enough people who
> > want to work on this and maintain it.
> >
> > Regards,
> > Suresh
> >
> >
> >
> > On 10/21/16, 9:41 PM, "Harsha Chintalapani" <ka...@harsha.io> wrote:
> >
> > >Sriram,
> > > "Can the streaming platform exist without stream processing? - No.
> > >Processing stream data again is a core part of streaming platform."
> > >
> > >Yes, it can. There are no.of Stream processing frameworks out there, and
> > >they all have integration into Kafka.
> > >It doesn't need to be developed within Kafka.
> > >
> > >
> > >"Can the platform exist without the rest proxy? - Yes. The proxy does
> not
> > >complete the platform vision in anyway. It is just a good to have tool
> > >that
> > >might be required by quite a few users and there is an active project
> that
> > >works on this - https://github.com/confluentinc/kafka-rest"
> > >
> > >The rest proxy is as important as any API. The vision that shown here
> > >http://kafka.apache.org/intro
> > >require users to write the producers and consumers to get their data
> into
> > >and out of Kafka, without which having Streams or Connect won't help
> > >anyone.
> > >The rest proxy makes easier for users get their data into Kafka.
> > >Adding the rest proxy to the project doesn't invalidate the current
> > >vision,
> > >it only strengthens it.
> > >
> > >Thanks,
> > >Harsha
> > >
> > >
> > >
> > >
> > >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <ra...@confluent.io>
> > >wrote:
> > >
> > >FWIW, Apache Kafka has evolved a lot from where it started. It did start
> > >as
> > >a messaging system. Over time we realized that that the vision for Kafka
> > >is
> > >to build a streaming platform and not just a messaging system. You can
> > >take
> > >a look at the site for more description about what comprises the
> streaming
> > >platform http://kafka.apache.org/ and http://kafka.apache.org/intro.
> > >
> > >Can the streaming platform exist without Connect? - No. Data integration
> > >is
> > >fundamental to building an end to end platform
> > >
> > >Can the streaming platform exist without stream processing? - No.
> > >Processing stream data again is a core part of streaming platform.
> > >
> > >Can the streaming platform exist without clients? - We at least need one
> > >client library to complete the platform. Our Java clients help us to
> > >complete the platform story. The rest of the clients are built and
> > >maintained outside the project.
> > >
> > >Can the platform exist without the rest proxy? - Yes. The proxy does not
> > >complete the platform vision in anyway. It is just a good to have tool
> > >that
> > >might be required by quite a few users and there is an active project
> that
> > >works on this - https://github.com/confluentinc/kafka-rest
> > >
> > >
> > >
> > >
> > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > ><ns...@linkedin.com.invalid>
> > >wrote:
> > >
> > >> Are you saying Kafka REST is subjective but Kafka Streams and Kafka
> > >Connect
> > >> are not subjective?
> > >>
> > >> > "there are likely places that can live without a rest proxy"
> > >>
> > >> There are also places that can live without Kafka Streams and Kafka
> > >> Connect.
> > >>
> > >> Nacho
> > >>
> > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io> wrote:
> > >>
> > >> > At the high level, I think ideally it makes sense to add a component
> > >>to
> > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight
> > >integration
> > >> > with Kafka core. For Kafka Stream, we do expect stream processing
> will
> > >be
> > >> > used widely in the future. Implementation wise, Kafka Stream only
> > >> supports
> > >> > getting data from Kafka and leverages quite a few of the core
> > >> > functionalities in Kafka core. For example, it uses customized
> > >>rebalance
> > >> > callback in the consumer and uses the compacted topic heavily. So,
> > >having
> > >> > Kafka Stream in the same repo makes it easier for testing when those
> > >core
> > >> > functionalities evolve over time. Kafka Connect is in the same
> > >situation.
> > >> >
> > >> > For rest proxy, whether it's widely used or not is going to be a bit
> > >> > subjective. However, there are likely places that can live without a
> > >rest
> > >> > proxy. The rest proxy is just a proxy for the regular clients and
> > >doesn't
> > >> > need to be tightly integrated with Kafka core. So, the case for
> > >including
> > >> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream
> > >>and
> > >> > Kafka Connect.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > >><Mi...@ig.com>
> > >> > wrote:
> > >> >
> > >> > > So from my reading essentially the first question needs to
> > >answered/and
> > >> > > voted on is:
> > >> > >
> > >> > > Is Apache Kafka Community only about the Core or does the apache
> > >> > community
> > >> > > also support some subprojects (and just we need some better way to
> > >> manage
> > >> > > this)
> > >> > >
> > >> > > If vote for Core only wins, then the following should be removed:
> > >> > > Kafka Connect
> > >> > > Kafka Stream
> > >> > >
> > >> > > If vote for Core only loses (aka we will support subprojects)
> then:
> > >> > > We should look to add Kafka Rest
> > >> > >
> > >> > > And we should look to see how we can manage better govern and
> manage
> > >> > > submodules.
> > >> > >
> > >> > > A good example which id propose here is how some other communities
> > >>in
> > >> > > Apache do this.
> > >> > >
> > >> > > Each Module has a Module Management Committee(MMC), this is like
> > >almost
> > >> > > the PMC but at a per module basis.
> > >> > >
> > >> > > This MMC should essentially hold the binding votes for that
> module.
> > >> > > The MMC should be made up of a single representative from each
> > >> > > organisation (so no single organisation can fully veto the
> > >>community
> > >> it
> > >> > > has to a genuine consenus)
> > >> > > The MMC requires at least 3 members (so there cant be a tied vote
> on
> > >2)
> > >> > > For a new Module to be added a MMC committee should be sought
> > >> > > A new Module is only capable of being added if the above
> > >>requirements
> > >> can
> > >> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
> > >that
> > >> > > only actively support modules would be added
> > >> > >
> > >> > > The PMC reviews each module every 6months or Year. If MMC is
> > >>inactive,
> > >> a
> > >> > > vote/call to find replacements if raised, if none are forthcoming
> > >> > dropping
> > >> > > the MMC to less than 3 then the module moves to "the attic" (very
> > >>much
> > >> > like
> > >> > > apache attic but a little more aggressively)
> > >> > >
> > >> > > This way the PMC does not need to micro manage every module
> > >> > > We only add modules where some amount of active support and
> > >maintenance
> > >> > > and use is provided by the community
> > >> > > We have an automatic way to retire old or inactive projects.
> > >> > >
> > >> > > Thoughts?
> > >> > > Mike
> > >> > >
> > >> > >
> > >> > > ________________________________________
> > >> > > From: Harsha Ch <ha...@gmail.com>
> > >> > > Sent: Thursday, October 20, 2016 10:26 PM
> > >> > > To: dev@kafka.apache.org
> > >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >> > >
> > >> > > Jay,
> > >> > > REST API is something every user is in need of. If the
> > >>argument
> > >> is
> > >> > to
> > >> > > clone and write your API, this will do a disservice to the users
> as
> > >> they
> > >> > > now have to choose one vs. others instead of keeping one API that
> is
> > >> > > supported in Kafka community.
> > >> > >
> > >> > > "Pre-emptively re-creating another
> > >> > > REST layer when it seems like we all quite agree on what needs to
> be
> > >> done
> > >> > > and we have an existing code base for HTTP/Kafka access that is
> > >heavily
> > >> > > used in production seems quite silly."
> > >> > >
> > >> > > Exactly our point. Why can't we develop this in Apache
> Kafka
> > >> > > community? Instead of us open sourcing another GitHub project and
> > >> > creating
> > >> > > a divide in users and another version of API. Let's build this in
> > >Kafka
> > >> > > Community and use the governance model that is proven to provide
> > >vendor
> > >> > > free user driven consensus features. The argument that is adding
> > >>this
> > >> > REST
> > >> > > server to Kafka will affect the agility of the project doesn't mak
> > >> sense.
> > >> > >
> > >> > > It looks like your argument is either we develop all these small
> > >>tools
> > >> or
> > >> > > none at all. We as a community need to look at supporting critical
> > >> > > tools/API. Instead of dividing this project into individual
> external
> > >> > > communities. We should build this as part of Kafka which best
> serves
> > >> the
> > >> > > needs of users.
> > >> > > The Streams and Connect projects that were pushed into
> Kafka
> > >> > could
> > >> > > have been left in their own Github projects based on your
> arguments.
> > >> What
> > >> > > about the REST API is so different that such that it should stay
> out
> > >of
> > >> > the
> > >> > > Kafka project? From my experience, more users are asking for the
> > >>REST
> > >> > API.
> > >> > >
> > >> > > Thanks,
> > >> > > Harsha
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io>
> wrote:
> > >> > >
> > >> > > > I think the questions around governance make sense, I think we
> > >should
> > >> > > > really clarify that to make the process more clear so it can be
> > >fully
> > >> > > > inclusive.
> > >> > > >
> > >> > > > The idea that we should not collaborate on what is there now,
> > >though,
> > >> > > > because in the future we might disagree about direction does not
> > >> really
> > >> > > > make sense to me. If in the future we disagree, that is the
> beauty
> > >of
> > >> > > open
> > >> > > > source, you can always fork off a copy of the code and start an
> > >> > > independent
> > >> > > > project either in Apache or elsewhere. Pre-emptively re-creating
> > >> > another
> > >> > > > REST layer when it seems like we all quite agree on what needs
> to
> > >>be
> > >> > done
> > >> > > > and we have an existing code base for HTTP/kafka access that is
> > >> heavily
> > >> > > > used in production seems quite silly.
> > >> > > >
> > >> > > > Let me give some background on how I at least think about these
> > >> things.
> > >> > > > I've participated in open source projects out of LinkedIn via
> > >>github
> > >> as
> > >> > > > well as via the ASF. I don't think there is a "right" answer to
> > >>how
> > >> to
> > >> > do
> > >> > > > these but rather some tradeoffs. We thought about this quite a
> lot
> > >in
> > >> > the
> > >> > > > context of Kafka based on the experience with the Hadoop
> ecosystem
> > >as
> > >> > > well
> > >> > > > as from other open source communities.
> > >> > > >
> > >> > > > There is a rich ecosystem around Kafka. Many of the projects are
> > >> quite
> > >> > > > small--single clients or tools that do a single thing well--and
> > >> almost
> > >> > > none
> > >> > > > of them are top level apache projects. I don't think trying to
> > >>force
> > >> > each
> > >> > > > of these to turn into independent Apache projects is necessarily
> > >>the
> > >> > best
> > >> > > > thing for the ecosystem.
> > >> > > >
> > >> > > > My observation of how this can go wrong is really what I think
> has
> > >> > > happened
> > >> > > > to the Hadoop ecosystem. There you see quite a zoo of projects
> > >>which
> > >> > all
> > >> > > > drift apart and don't quite work together well. Coordinating
> even
> > >> > simple
> > >> > > > changes and standardization across these is exceptionally
> > >>difficult.
> > >> > The
> > >> > > > result is a bit of a mess for users--the pieces just don't
> really
> > >> come
> > >> > > > together very well. This makes sense for independent
> > >>infrastructure
> > >> > > systems
> > >> > > > (Kudu vs HDFS) but I'm not at all convinced that doing this for
> > >every
> > >> > > > little tool or helper library has lead to a desirable state. I
> > >>think
> > >> > the
> > >> > > > mode of operating where the Hadoop vendors spawn off a few new
> > >Apache
> > >> > > > projects for each new product initiative, especially since often
> > >that
> > >> > > > project is only valued by that vendor (and the other vendor has
> a
> > >> > > different
> > >> > > > competing Apache project) doesn't necessarily do a better job at
> > >> > > producing
> > >> > > > high quality communities or high quality software.
> > >> > > >
> > >> > > > These tools/connects/clients/proxies and other integration
> pieces
> > >can
> > >> > > take
> > >> > > > many forms, but my take of what makes one of these things good
> is
> > >> that
> > >> > it
> > >> > > > remains simple, does its one thing well, and cleaves as closely
> as
> > >> > > possible
> > >> > > > to the conventions for Kafka itself--i.e. doesn't invent new
> ways
> > >>of
> > >> > > > monitoring, configuring, etc. For the tools we've contributed
> > >>we've
> > >> > tried
> > >> > > > really hard to make them consistent with Kafka as well as with
> > >>each
> > >> > other
> > >> > > > in how testing, configuration, monitoring, etc works.
> > >> > > >
> > >> > > > I think what Apache does superbly well is create a community for
> > >> > > managing a
> > >> > > > large infrastructure layer like Kafka in a vendor independent
> way.
> > >> > What I
> > >> > > > think is less successful is attempting to form full and
> > >>independent
> > >> > > apache
> > >> > > > communities around very simple single purpose tools, especially
> if
> > >> you
> > >> > > hope
> > >> > > > for these to come together into a cohesive toolset across
> multiple
> > >> such
> > >> > > > tools. Much of what Apache does--create a collective decision
> > >>making
> > >> > > > process for resolving disagreement, help to trademark and
> protect
> > >the
> > >> > > marks
> > >> > > > of the project, etc just isn't that relevant for simple
> > >> single-purpose
> > >> > > > tools.
> > >> > > >
> > >> > > > So my take is there are a couple of options:
> > >> > > >
> > >> > > > 1. We can try to put all the small tools into the Apache
> > >>Project.
> > >> I
> > >> > > > think this is not the right approach as there is simply too
> > >>many
> > >> of
> > >> > > > them,
> > >> > > > many in different languages, serving different protocols,
> > >> > integrating
> > >> > > > with
> > >> > > > particular systems, and a single community can't effectively
> > >> > maintain
> > >> > > > them
> > >> > > > all. Doing this would significantly slow the progress of the
> > >Kafka
> > >> > > > project.
> > >> > > > As a protocol for messaging, I don't really see a case for
> > >> including
> > >> > > > REST
> > >> > > > but not MQTT or AMQP which are technically much better suited
> > >>to
> > >> > > > messaging
> > >> > > > and both are widely used for that.
> > >> > > > 2. We can treat ecosystem projects that aren't top level
> Apache
> > >> > > projects
> > >> > > > as invalid and try to recreate them all as Apache projects.
> > >> > Honestly,
> > >> > > > though, if you go to the Kafka ecosystem page virtually none
> of
> > >> the
> > >> > > most
> > >> > > > popular add-ons to Kafka are Apache projects. The most
> > >>successful
> > >> > > > things in
> > >> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a
> number
> > >of
> > >> > > other
> > >> > > > clients, as well as the existing REST layer have succeeded at
> > >> > > developing
> > >> > > > communities that actively contribute and use these pieces
> and I
> > >> > don't
> > >> > > > know
> > >> > > > that that is a bad thing unless that community proves to be
> > >> > > uninclusive,
> > >> > > > unresponsive, or goes in a bad technical direction--and those
> > >>are
> > >> > > > failure
> > >> > > > modes that all open source efforts face.
> > >> > > > 3. We can do what I think makes the most sense and try to
> work
> > >> with
> > >> > > the
> > >> > > > projects that exist in the ecosystem and if the project
> doesn't
> > >> > have a
> > >> > > > responsive community or wants to go in a different direction
> > >>fork
> > >> or
> > >> > > > recreate that work.
> > >> > > >
> > >> > > > Of course any person can choose whatever of these options they
> > >>want.
> > >> > But
> > >> > > > from my point of view, option (3) has been the path of the
> > >>community
> > >> so
> > >> > > far
> > >> > > > and I think it has been quite successful.
> > >> > > >
> > >> > > > -Jay
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> > >> kafka@harsha.io
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Neha,
> > >> > > > > "But I haven't seen any community emails or patches being
> > >submitted
> > >> > by
> > >> > > > you
> > >> > > > > guys, so I'm wondering why you are concerned about whether the
> > >> > > community
> > >> > > > is
> > >> > > > > open to accepting patches or not."
> > >> > > > >
> > >> > > > > I think you are talking about contributing patches to this
> > >> repository
> > >> > > > > right? https://github.com/confluentinc/kafka-rest . All I am
> > >> saying
> > >> > > > > the guidelines/governance model is not clear on the project
> and
> > >>I
> > >> > guess
> > >> > > > its
> > >> > > > > driven by opening a github issue request. Its the repository
> > >owned
> > >> > by
> > >> > > > > confluent and as much I appreciate that the features we
> > >>mentioned
> > >> are
> > >> > > in
> > >> > > > > the roadmap and welcoming us to contribute to the project. It
> > >> doesn't
> > >> > > > > gurantee what we want to add in the furture will be in your
> > >> roadmap.
> > >> > > > >
> > >> > > > > Hence the reason having it part of Kafka community will help a
> > >>lot
> > >> as
> > >> > > > other
> > >> > > > > users can participate in the discussions. We are happy to
> drive
> > >> any
> > >> > > > > feature additions through KIPs which gives everyone a chance
> to
> > >> > > > participate
> > >> > > > > and add to the discussions.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Harsha
> > >> > > > >
> > >> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > >> > Michael.Pearce@ig.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > +1
> > >> > > > > >
> > >> > > > > > I agree on the governance comments whole heartedly.
> > >> > > > > >
> > >> > > > > > Also i agree about the contribution comments made earlier in
> > >>the
> > >> > > > thread,
> > >> > > > > i
> > >> > > > > > personally am less likely to spend any of mine, or give
> > >>project
> > >> > time
> > >> > > > > within
> > >> > > > > > my internal projects to developers contributing to another
> > >> > commercial
> > >> > > > > > companies project even so technically open source, as then
> > >>there
> > >> is
> > >> > > > that
> > >> > > > > > commercial companies interest will always prevail and
> > >essentially
> > >> > can
> > >> > > > > > always have a final vote where disagreement. Im sure they
> > >>never
> > >> > > intend
> > >> > > > > to,
> > >> > > > > > but there is that true reality. This is why we have
> community
> > >> open
> > >> > > > source
> > >> > > > > > projects.
> > >> > > > > >
> > >> > > > > > I can find many different implementations now of a rest
> > >>endpoint
> > >> on
> > >> > > > > > GitHub, BitBucket etc. Each one has their benefits and
> > >> > disadvantages
> > >> > > in
> > >> > > > > > their implementation. By making / providing one this would
> > >>bring
> > >> > > > together
> > >> > > > > > these solutions, unifying those developers and also bringing
> > >>the
> > >> > best
> > >> > > > of
> > >> > > > > > all.
> > >> > > > > >
> > >> > > > > > I understand the concern on the community burden
> > >> adding/supporting
> > >> > > more
> > >> > > > > > surface area for every client. But something like REST is
> > >> universal
> > >> > > and
> > >> > > > > > worthy to be owned by the community.
> > >> > > > > >
> > >> > > > > > Mike
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > ________________________________________
> > >> > > > > > From: Andrew Schofield <an...@outlook.com>
> > >> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > >> > > > > > To: dev@kafka.apache.org
> > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >> > > > > >
> > >> > > > > > There's a massive difference between the governance of Kafka
> > >>and
> > >> > the
> > >> > > > > > governance of the REST proxy.
> > >> > > > > >
> > >> > > > > > In Kafka, there is a broad community of people contributing
> > >their
> > >> > > > > opinions
> > >> > > > > > about future enhancements in the form of KIPs. There's some
> > >> really
> > >> > > deep
> > >> > > > > > consideration that goes into some of the trickier KIPs.
> There
> > >are
> > >> > > > people
> > >> > > > > > outside Confluent deeply knowledgeable in Kafka and
> building
> > >the
> > >> > > > > > reputations to become committers. I get the impression that
> > >>the
> > >> > > roadmap
> > >> > > > > of
> > >> > > > > > Kafka is not really community-owned (what's the big feature
> > >>for
> > >> > Kafka
> > >> > > > > 0.11,
> > >> > > > > > for example), but the conveyor belt of smaller features in
> the
> > >> form
> > >> > > of
> > >> > > > > KIPs
> > >> > > > > > works nicely. It's a good example of open-source working
> > >>well.
> > >> > > > > >
> > >> > > > > > The equivalent for the REST proxy is basically issues on
> > >>GitHub.
> > >> > The
> > >> > > > > > roadmap is less clear. There's not really a community
> properly
> > >> > > engaged
> > >> > > > in
> > >> > > > > > the way that there is with Kafka. So, you could say that
> it's
> > >> clear
> > >> > > > that
> > >> > > > > > fewer people are interested, but I think the whole
> governance
> > >> > thing
> > >> > > > is a
> > >> > > > > > big barrier to engagement. And it's looking like it's
> getting
> > >out
> > >> > of
> > >> > > > > date.
> > >> > > > > >
> > >> > > > > > In technical terms, I can think of two big improvements to
> the
> > >> REST
> > >> > > > > proxy.
> > >> > > > > > First, it needs to use the new consumer API so that it's
> > >possible
> > >> > to
> > >> > > > > secure
> > >> > > > > > connections between the REST proxy and Kafka. I don't care
> too
> > >> much
> > >> > > > which
> > >> > > > > > method calls it uses actually uses to consume messages,
> but I
> > >do
> > >> > > care
> > >> > > > > that
> > >> > > > > > I cannot secure connections because of network security
> rules.
> > >> > > Second,
> > >> > > > > > there's an affinity between a consumer and the instance of
> the
> > >> REST
> > >> > > > proxy
> > >> > > > > > to which it first connected. Kafka itself avoids this kind
> of
> > >> > > affinity
> > >> > > > > for
> > >> > > > > > good reason, and in the name of availability the REST proxy
> > >> should
> > >> > > too.
> > >> > > > > > These are natural KIPs.
> > >> > > > > >
> > >> > > > > > I think it would be good to have the code for the REST proxy
> > >> > > > contributed
> > >> > > > > > to Apache Kafka so that it would be able to be developed in
> > >>the
> > >> > same
> > >> > > > way.
> > >> > > > > >
> > >> > > > > > Andrew Schofield
> > >> > > > > >
> > >> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > >> > > > > > Sent: 07 October 2016 22:41:52
> > >> > > > > > To: dev@kafka.apache.org
> > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >> > > > > >
> > >> > > > > > ASF already gives us a clear framework and governance model
> > >>for
> > >> > > > community
> > >> > > > > > development. This is already understood by the people
> > >> contributing
> > >> > to
> > >> > > > > > Apache Kafka project, and they are the same people who want
> to
> > >> > > > contribute
> > >> > > > > > to the REST server capability as well. Everyone is in
> > >>agreement
> > >> on
> > >> > > the
> > >> > > > > > need for collaborating on this effort. So why not contribute
> > >>the
> > >> > code
> > >> > > > to
> > >> > > > > > Apache Kafka. This will help avoid duplication of effort and
> > >> forks
> > >> > > that
> > >> > > > > > may crop up, hugely benefitting the user community. This
> will
> > >> also
> > >> > > > avoid
> > >> > > > > > having to define a process similar to ASF on a GitHub
> project
> > >and
> > >> > > > instead
> > >> > > > > > there is a single community with clear understanding
> community
> > >> > > process
> > >> > > > as
> > >> > > > > > defined in ASF.
> > >> > > > > >
> > >> > > > > > As others have said, this is an important capability for
> > >>Apache
> > >> > > Kafka.
> > >> > > > It
> > >> > > > > > is worth maintaining this as a part of the project.
> > >> > > > > >
> > >> > > > > > Regards,
> > >> > > > > > Suresh
> > >> > > > > >
> > >> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io>
> > >>wrote:
> > >> > > > > >
> > >> > > > > > >I personally think it would be quite wasteful to
> re-implement
> > >> the
> > >> > > REST
> > >> > > > > > >gateway just because that an actively-maintained piece of
> > >> > > > > Apache-licensed
> > >> > > > > > >software is not governed directly by the Apache Kafka
> > >> community...
> > >> > > > While
> > >> > > > > > >kafka-rest repo is owned by Confluent, the contributors
> > >> including
> > >> > > the
> > >> > > > > main
> > >> > > > > > >one are also part of the Apache Kafka community, so there
> > >>is a
> > >> > > chance
> > >> > > > > to
> > >> > > > > > >work this out.
> > >> > > > > > >
> > >> > > > > > >However, there are two valid concerns here that could be
> > >> > addressed,
> > >> > > > > around
> > >> > > > > > >community and accessibility:
> > >> > > > > > >>> What we are worried about is a project
> > >> > > > > > >>> that's not maintained in a community. So the process of
> > >> > accepting
> > >> > > > > > >>>patches
> > >> > > > > > >>> and priorities is not clear, and it's not developed in
> > >Apache
> > >> > > > > > >>>community.
> > >> > > > > > >>> Not only that, existing REST API project doesn't support
> > >>new
> > >> > > client
> > >> > > > > API
> > >> > > > > > >and
> > >> > > > > > >>> hence there is no security support either.
> > >> > > > > > >
> > >> > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
> > >> community
> > >> > > can
> > >> > > > > > >clarify that - what is their contribution policy, dev
> style,
> > >> > roadmap
> > >> > > > > etc.
> > >> > > > > > >If they want, they can make an effort to encourage
> > >participation
> > >> > > from
> > >> > > > > > >people outside Confluent (easily accept contributions,
> invite
> > >> > > external
> > >> > > > > > >commiters or have open dev process similar to Apache Kafka
> > >etc),
> > >> > as
> > >> > > > > there
> > >> > > > > > >is definitely seems to be some interest on the list. That
> > >>might
> > >> > > clear
> > >> > > > > the
> > >> > > > > > >community concern and help kafka-rest project (but that is
> a
> > >> > > > calculation
> > >> > > > > > >Confluent will have to make).
> > >> > > > > > >
> > >> > > > > > >The other, independent, concern is that REST is something
> > >>that
> > >> is
> > >> > > > > expected
> > >> > > > > > >to be available out of the box with Kafka. I personally
> don't
> > >> feel
> > >> > > > > > >strongly
> > >> > > > > > >about it (better use proper, efficient APIs from day one),
> > >> though
> > >> > it
> > >> > > > is
> > >> > > > > > >definitely way smaller than adding a stream processing
> engine
> > >to
> > >> > the
> > >> > > > > > >project :)
> > >> > > > > > >Again,the kafka-rest "community" could take steps to make
> it
> > >> even
> > >> > > > easier
> > >> > > > > > >to
> > >> > > > > > >install, configure and run kafka-rest for new users on
> > >>vanilla
> > >> > > Apache
> > >> > > > > > >Kafka
> > >> > > > > > >(outside the Confluent platform), if they wish that (or
> > >>welcome
> > >> > > > > > >contributions to that end), but that is up to them.
> > >> > > > > > >Finally, if after the above steps were taken there would
> > >>still
> > >a
> > >> > > > strong
> > >> > > > > > >desire to include a great rest gateway with Apache Kafka, I
> > >> assume
> > >> > > the
> > >> > > > > > >community could hypothetically fork the existing kafka-rest
> > >into
> > >> > an
> > >> > > > > Apache
> > >> > > > > > >Kafka subproject and maintain it "within Apache" instead of
> > >> > > > implementing
> > >> > > > > > >it
> > >> > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot
> > >> imagine
> > >> > it
> > >> > > > > > >happen
> > >> > > > > > >without Confluent blessing, and I think that is likely much
> > >less
> > >> > > > optimal
> > >> > > > > > >(pulling in other Confluent / Apache licensed dependencies)
> > >than
> > >> > > > having
> > >> > > > > a
> > >> > > > > > >separate external community around kafka-rest.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >Just my two cents,
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >Ofir Manor
> > >> > > > > > >
> > >> > > > > > >Co-Founder & CTO | Equalum
> > >> > > > > > >
> > >> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > ><+972%2054-780-1286>
> > >> > <+972%2054-780-1286> |
> > >> > > > Email:
> > >> > > > > > ofir.manor@equalum.io
> > >> > > > > > >
> > >> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > >> > > kafka@harsha.io
> > >> > > > >
> > >> > > > > > >wrote:
> > >> > > > > > >
> > >> > > > > > >> Neha & Jay,
> > >> > > > > > >> We did look at the open source
> > >>alternatives.
> > >> > Our
> > >> > > > > > >>concern
> > >> > > > > > >> is what's the patch acceptance and adding features/
> > >>bug-fixes
> > >> to
> > >> > > the
> > >> > > > > > >> existing project under a Github (although it's licensed
> > >>under
> > >> > > Apache
> > >> > > > > > >>2.0).
> > >> > > > > > >> It would be great if that project made available under
> > >>Apache
> > >> > and
> > >> > > > > > >>driven by
> > >> > > > > > >> the community. Adding to the above, not all Kafka users
> > >>are
> > >> > > > > interested
> > >> > > > > > >>in
> > >> > > > > > >> using the Java client API, they would like to have simple
> > >REST
> > >> > API
> > >> > > > > where
> > >> > > > > > >> they can code against using any language. I do believe
> this
> > >> adds
> > >> > > > value
> > >> > > > > > >>to
> > >> > > > > > >> Apache Kafka in itself.
> > >> > > > > > >>
> > >> > > > > > >> "For 1, I don't think there is value in giving in to the
> > >>NIH
> > >> > > > syndrome
> > >> > > > > > >>and
> > >> > > > > > >> reinventing the wheel. What I'm looking for is a detailed
> > >> > > comparison
> > >> > > > > of
> > >> > > > > > >>the
> > >> > > > > > >> gaps and why those can't be improved in the REST proxy
> that
> > >> > > already
> > >> > > > > > >>exists
> > >> > > > > > >> and is actively maintained."
> > >> > > > > > >>
> > >> > > > > > >> We are not looking at this as NIH. What we are worried
> > >>about
> > >> > is a
> > >> > > > > > >>project
> > >> > > > > > >> that's not maintained in a community. So the process of
> > >> > accepting
> > >> > > > > > >>patches
> > >> > > > > > >> and priorities is not clear, and it's not developed in
> > >>Apache
> > >> > > > > community.
> > >> > > > > > >> Not only that, existing REST API project doesn't support
> > >>new
> > >> > > client
> > >> > > > > API
> > >> > > > > > >>and
> > >> > > > > > >> hence there is no security support either.
> > >> > > > > > >> We don't know the timeline when that's made available. We
> > >> would
> > >> > > like
> > >> > > > > to
> > >> > > > > > >>add
> > >> > > > > > >> admin functionality into the REST API. So the Roadmap of
> > >>that
> > >> > > > project
> > >> > > > > is
> > >> > > > > > >> not driven by Apache.
> > >> > > > > > >>
> > >> > > > > > >>
> > >> > > > > > >> "This doesn't materially have an impact on expanding the
> > >> > usability
> > >> > > > of
> > >> > > > > > >> Kafka. In my experience, REST proxy + Java clients
> only
> > >> cover
> > >> > > > ~50%
> > >> > > > > of
> > >> > > > > > >> all
> > >> > > > > > >> Kafka users, and maybe 10% of those are the ones who
> > >>will
> > >> use
> > >> > > the
> > >> > > > > > >>REST
> > >> > > > > > >> proxy. The remaining 50% are non-java client users (C,
> > >> > python,
> > >> > > > go,
> > >> > > > > > >>node
> > >> > > > > > >> etc)."
> > >> > > > > > >>
> > >> > > > > > >> REST API is most often asked feature in my interactions
> > >>with
> > >> > Kafka
> > >> > > > > > >>users.
> > >> > > > > > >> In an organization, There will be independent teams who
> > >>will
> > >> > write
> > >> > > > > their
> > >> > > > > > >> Kafka clients using different language libraries
> available
> > >> > today,
> > >> > > > and
> > >> > > > > > >> there is no way to standardize this. Instead of
> supporting
> > >> > several
> > >> > > > > > >> different client libraries users will be interested in
> > >>using
> > >a
> > >> > > REST
> > >> > > > > API
> > >> > > > > > >> server. The need for a REST API will only increase as
> more
> > >and
> > >> > > more
> > >> > > > > > >>users
> > >> > > > > > >> start using Kafka.
> > >> > > > > > >>
> > >> > > > > > >> "More surface area means more work to keep things
> > >>consistent.
> > >> > > > Failure
> > >> > > > > > >> to do that has, in fact, hurt the user experience."
> > >> > > > > > >> Having myriad Kafka client GitHub projects that support
> > >> > different
> > >> > > > > > >>languages
> > >> > > > > > >> hurts the user experience and pushes burden to maintain
> > >>these
> > >> > > > > libraries.
> > >> > > > > > >> REST API is a simple code base that uses existing java
> > >>client
> > >> > > > > libraries
> > >> > > > > > >>to
> > >> > > > > > >> make life easier for the users.
> > >> > > > > > >>
> > >> > > > > > >> Thanks,
> > >> > > > > > >> Harsha
> > >> > > > > > >>
> > >> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > >> > neha@confluent.io>
> > >> > > > > > wrote:
> > >> > > > > > >>
> > >> > > > > > >> > Manikumar,
> > >> > > > > > >> >
> > >> > > > > > >> > Thanks for sharing the proposal. I think there are 2
> > >>parts
> > >> to
> > >> > > this
> > >> > > > > > >> > discussion -
> > >> > > > > > >> >
> > >> > > > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > >> > > > > > >>feature-complete,
> > >> > > > > > >> > open-source and actively maintained REST proxy in the
> > >> > community?
> > >> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us
> more
> > >> agile
> > >> > > and
> > >> > > > > > >> maintain
> > >> > > > > > >> > the high-quality experience that Kafka users have
> today?
> > >> > > > > > >> >
> > >> > > > > > >> > For 1, I don't think there is value in giving in to the
> > >>NIH
> > >> > > > syndrome
> > >> > > > > > >>and
> > >> > > > > > >> > reinventing the wheel. What I'm looking for is a
> detailed
> > >> > > > comparison
> > >> > > > > > >>of
> > >> > > > > > >> the
> > >> > > > > > >> > gaps and why those can't be improved in the REST proxy
> > >>that
> > >> > > > already
> > >> > > > > > >> exists
> > >> > > > > > >> > and is actively maintained. For example, we depend on
> > >> zkClient
> > >> > > and
> > >> > > > > > >>have
> > >> > > > > > >> > found as well as fixed several bugs by working closely
> > >>with
> > >> > the
> > >> > > > > people
> > >> > > > > > >> who
> > >> > > > > > >> > maintain zkClient. This should be possible for REST
> proxy
> > >as
> > >> > > well,
> > >> > > > > > >>right?
> > >> > > > > > >> >
> > >> > > > > > >> > For 2, I'd like us to review our history of expanding
> the
> > >> > > surface
> > >> > > > > > >>area to
> > >> > > > > > >> > add more clients in the past. Here is a summary -
> > >> > > > > > >> >
> > >> > > > > > >> > - This doesn't materially have an impact on
> expanding
> > >the
> > >> > > > > > >>usability of
> > >> > > > > > >> > Kafka. In my experience, REST proxy + Java clients
> > >>only
> > >> > cover
> > >> > > > > ~50%
> > >> > > > > > >>of
> > >> > > > > > >> > all
> > >> > > > > > >> > Kafka users, and maybe 10% of those are the ones who
> > >will
> > >> > use
> > >> > > > the
> > >> > > > > > >>REST
> > >> > > > > > >> > proxy. The remaining 50% are non-java client users
> (C,
> > >> > > python,
> > >> > > > > go,
> > >> > > > > > >> node
> > >> > > > > > >> > etc).
> > >> > > > > > >> > - People are a lot more excited about promising to
> > >> > contribute
> > >> > > > > while
> > >> > > > > > >> > adding the surface area but not on an ongoing basis
> > >>down
> > >> > the
> > >> > > > > line.
> > >> > > > > > >> > - More surface area means more work to keep things
> > >> > > consistent.
> > >> > > > > > >>Failure
> > >> > > > > > >> > to do that has, in fact, hurt the user experience.
> > >> > > > > > >> > - More surface area hurts agility. We want to do a
> few
> > >> > things
> > >> > > > > > >>really
> > >> > > > > > >> > well as well as be agile to be able to build on our
> > >>core
> > >> > > > > > >>competency.
> > >> > > > > > >> >
> > >> > > > > > >> >
> > >> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > >> > > > manikumar.reddy@gmail.com
> > >> > > > > >
> > >> > > > > > >> > wrote:
> > >> > > > > > >> >
> > >> > > > > > >> > > Hi Jay,
> > >> > > > > > >> > >
> > >> > > > > > >> > > Thanks for your reply.
> > >> > > > > > >> > >
> > >> > > > > > >> > > I agree that we can not add all the clients/tools
> > >> available
> > >> > in
> > >> > > > > > >> ecosystem
> > >> > > > > > >> > > page to Kafka repo itself. But we feel REST Interface
> > >>is
> > >> > > > different
> > >> > > > > > >>from
> > >> > > > > > >> > > other clients/tools. Since any language that can work
> > >with
> > >> > > HTTP
> > >> > > > > can
> > >> > > > > > >> > > easily integrate with this interface, Having an
> > >"official"
> > >> > > REST
> > >> > > > > > >> > > interface helps user community. This also helps us to
> > >> > > integrate
> > >> > > > > well
> > >> > > > > > >> > > with external management and provisioning tools.
> > >>Apache
> > >> > Kafka
> > >> > > > > > >>release
> > >> > > > > > >> > > with Java clients + REST interface is sufficient for
> > >>most
> > >> of
> > >> > > the
> > >> > > > > > >>user
> > >> > > > > > >> > > deployments/requirements. This helps users to deal
> with
> > >> less
> > >> > > > > number
> > >> > > > > > >> > > of distributions/builds.
> > >> > > > > > >> > >
> > >> > > > > > >> > > Thanks,
> > >> > > > > > >> > > Manikumar
> > >> > > > > > >> > >
> > >> > > > > > >> > >
> > >> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> > >> jay@confluent.io
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > >> > >
> > >> > > > > > >> > > > Hey guys,
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > There's already a REST interface maintained as a
> > >> separate
> > >> > > > > > >> project--it's
> > >> > > > > > >> > > > open source and apache licensed and actively
> > >>maintained
> > >> (
> > >> > > > > > >> > > > https://github.com/confluentinc/kafka-rest). What
> is
> > >> > wrong
> > >> > > > with
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > >> > > > > > github.com
> > >> > > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
> > >> > cluster.
> > >> > > > It
> > >> > > > > > makes it easy to produce and consume messages, view the
> state
> > >>of
> > >> > the
> > >> > > > > > cluster, and ...
> > >> > > > > >
> > >> > > > > > >> that?
> > >> > > > > > >> > > You
> > >> > > > > > >> > > > mentioned that there was some compatibility
> concern,
> > >but
> > >> > > > > > >> compatibility
> > >> > > > > > >> > > has
> > >> > > > > > >> > > > to do with the consumer protocol guarantees not the
> > >repo
> > >> > the
> > >> > > > > code
> > >> > > > > > >>is
> > >> > > > > > >> > in,
> > >> > > > > > >> > > > right? Not sure that concern makes sense.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > We could argue for adding pretty much anything and
> > >> > > everything
> > >> > > > in
> > >> > > > > > >>the
> > >> > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure
> that
> > >> would
> > >> > > > make
> > >> > > > > > >>the
> > >> > > > > > >> > > project
> > >> > > > > > >> > > > more agile.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > -Jay
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > >> > > > > > >> manikumar.reddy@gmail.com
> > >> > > > > > >> > >
> > >> > > > > > >> > > > wrote:
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > > Hi Kafka Devs,
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to
> Kafka
> > >> > > > Repository.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > There are already open-source alternatives are
> > >> > available.
> > >> > > > But
> > >> > > > > > >>we
> > >> > > > > > >> > would
> > >> > > > > > >> > > > > like to add REST server that
> > >> > > > > > >> > > > > many users ask for under Apache Kafka repo. Many
> > >>data
> > >> > > Infra
> > >> > > > > > >>tools
> > >> > > > > > >> > comes
> > >> > > > > > >> > > > up
> > >> > > > > > >> > > > > with Rest Interface.
> > >> > > > > > >> > > > > It is useful to have inbuilt Rest API support for
> > >> > Produce,
> > >> > > > > > >>Consume
> > >> > > > > > >> > > > messages
> > >> > > > > > >> > > > > and admin interface for
> > >> > > > > > >> > > > > integrating with external management and
> > >>provisioning
> > >> > > > > tools.This
> > >> > > > > > >> will
> > >> > > > > > >> > > > also
> > >> > > > > > >> > > > > allow the maintenance of
> > >> > > > > > >> > > > > REST server and adding new features makes it easy
> > >> > because
> > >> > > > > apache
> > >> > > > > > >> > > > community.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > The KIP wiki is the following:
> > >> > > > > > >> > > > > https://cwiki.apache.org/
> > >> confluence/display/KAFKA/KIP-
> > >> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Your comments and feedback are welcome.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Thanks,
> > >> > > > > > >> > > > > Manikumar
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> > --
> > >> > > > > > >> > Thanks,
> > >> > > > > > >> > Neha
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > > The information contained in this email is strictly
> > >>confidential
> > >> > and
> > >> > > > for
> > >> > > > > > the use of the addressee only, unless otherwise indicated.
> If
> > >you
> > >> > are
> > >> > > > not
> > >> > > > > > the intended recipient, please do not read, copy, use or
> > >disclose
> > >> > to
> > >> > > > > others
> > >> > > > > > this message or any attachment. Please also notify the
> sender
> > >>by
> > >> > > > replying
> > >> > > > > > to this email or by telephone (+44(020 7896 0011) and then
> > >delete
> > >> > the
> > >> > > > > email
> > >> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> > >> relate
> > >> > > to
> > >> > > > > the
> > >> > > > > > official business of this company shall be understood as
> > >>neither
> > >> > > given
> > >> > > > > nor
> > >> > > > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > >> > company
> > >> > > > > > registered in England and Wales, company number 04008957)
> and
> > >>IG
> > >> > > Index
> > >> > > > > > Limited (a company registered in England and Wales, company
> > >> number
> > >> > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > >>Dowgate
> > >> > > Hill,
> > >> > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > >195355)
> > >> > and
> > >> > > > IG
> > >> > > > > > Index Limited (register number 114059) are authorised and
> > >> regulated
> > >> > > by
> > >> > > > > the
> > >> > > > > > Financial Conduct Authority.
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > > The information contained in this email is strictly confidential
> and
> > >> for
> > >> > > the use of the addressee only, unless otherwise indicated. If you
> > >>are
> > >> not
> > >> > > the intended recipient, please do not read, copy, use or disclose
> to
> > >> > others
> > >> > > this message or any attachment. Please also notify the sender by
> > >> replying
> > >> > > to this email or by telephone (+44(020 7896 0011) and then delete
> > >>the
> > >> > email
> > >> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > >>to
> > >> > the
> > >> > > official business of this company shall be understood as neither
> > >>given
> > >> > nor
> > >> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > >>company
> > >> > > registered in England and Wales, company number 04008957) and IG
> > >>Index
> > >> > > Limited (a company registered in England and Wales, company number
> > >> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > >>Hill,
> > >> > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > >>and
> > >> IG
> > >> > > Index Limited (register number 114059) are authorised and
> regulated
> > >>by
> > >> > the
> > >> > > Financial Conduct Authority.
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Nacho (Ignacio) Solis
> > >> Kafka
> > >> nsolis@linkedin.com
> > >>
> >
> >
> >
>
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Jay Kreps <ja...@confluent.io>.
Hey Suresh,
I think we agree that REST apis are useful. I don't think we agree that
they need to be part of the Kafka project. We've had this discussion a
dozen odd times for different protocols---AMQP, REST, MQTT. Going back the
last five years we've always rejected these. That doesn't mean they aren't
useful, I think having these as separate is fine, they don't have any
complex interaction with Kafka, they just use the vanilla APIs like any of
the dozens of things on the ecosystem page.
In terms of how they're developed. I think there are two discussions here:
1. Separate project or not
2. Standalone Apache project or github
The first of those I just talked about---I think this makes sense as an
independent project.
For the second of these, I actually don't think that spawning off a bunch
of itty-bitty independent Apache projects is a good thing. I think you can
see this in the Hadoop ecosystem where the parts kind of all evolve off in
different directions and don't really work together as a whole. I think
that makes sense for deep infrastructure projects, but not for these little
helper projects. I think a hub and spoke model where you have a central
project (Kafka) and then a bunch of helper tools that strive to fit in with
it (in terms of config, monitoring, apis, and every other conventions) is
actually much better. In any case there are already so many of these tools
(we capture maybe 20% of them on the ecosystem page), made by so many
people, and virtually all developed in this style, it is a bit late to put
the cat back in the bag.
Perhaps it's just a difference in background. For my part I had many years
successfully running github-style projects, and i think that model can work
quite well for small things. I do think it is important for these projects
to clarify governance, which we should absolutely do, but realistically it
is a pretty simple tool, there isn't a huge governance challenge for
something like this because its scope is so limited ("take http requests,
make Kafka requests").
More specifically, I don't think there is an actual problem being solved
here. I haven't heard any complaint about direction or patches not getting
added. The only complaint I've heard is missing features where the normal
"patches accepted" rule applies. I would urge people to actually get
involved in contribution here. In the future if there is a situation where
people don't like the direction of a given tool, they can fork it and
either turn it into an independent Apache project or develop it
independently, trying to do that preemptively seems a bit hostile.
-Jay
On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:
> I am dividing this discussion into two parts:
> 1. REST APIs as core Apache Kafka capability
> This should be a core Kafka functionality. Same view has been reflected by
> others (users and developers) as well. While we can debate whether other
> capabilities are core Kafka (Streams, Connect), it would be good separate
> that out and to keep this discussion focussed on REST APIs as proposed by
> this KIP. If there is ambivalence about the need for this in core kafka,
> we could have voting in the mailing list.
>
> Can we get an agreement on this? I am +1 on REST API in Apache Kafka.
>
> 2. Community where Kafka REST APIs need to be collaborated on
> There is already a GitHub project that caters to Kafka REST APIs. But a
> company owned Github is less than ideal for collaboration due to lack of
> governance, open community and roadmap. This is reflected by many people
> interested in this functionality and still not contributing to and
> adopting the code base in the GitHub. I think trying overlay the ASF
> governance model on GitHub project, which is what the need is, seems
> unnecessary, if the code can be part of Apache Kafka.
>
> The question is, would Confluent be okay with licensing/contributing the
> code to Kafka project (assuming either Confluent or another contributor
> can work on it)? I see clear intent in collaborating with others on REST
> APIs from confluent. Why not do it in Kafka project under ASF? This takes
> away all the barrier to collaboration? Alternatively, if Confluent is not
> willing to contribute the code from GitHub, would anyone veto building a
> separate REST API functionality in ASF Kafka? There are enough people who
> want to work on this and maintain it.
>
> Regards,
> Suresh
>
>
>
> On 10/21/16, 9:41 PM, "Harsha Chintalapani" <ka...@harsha.io> wrote:
>
> >Sriram,
> > "Can the streaming platform exist without stream processing? - No.
> >Processing stream data again is a core part of streaming platform."
> >
> >Yes, it can. There are no.of Stream processing frameworks out there, and
> >they all have integration into Kafka.
> >It doesn't need to be developed within Kafka.
> >
> >
> >"Can the platform exist without the rest proxy? - Yes. The proxy does not
> >complete the platform vision in anyway. It is just a good to have tool
> >that
> >might be required by quite a few users and there is an active project that
> >works on this - https://github.com/confluentinc/kafka-rest"
> >
> >The rest proxy is as important as any API. The vision that shown here
> >http://kafka.apache.org/intro
> >require users to write the producers and consumers to get their data into
> >and out of Kafka, without which having Streams or Connect won't help
> >anyone.
> >The rest proxy makes easier for users get their data into Kafka.
> >Adding the rest proxy to the project doesn't invalidate the current
> >vision,
> >it only strengthens it.
> >
> >Thanks,
> >Harsha
> >
> >
> >
> >
> >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <ra...@confluent.io>
> >wrote:
> >
> >FWIW, Apache Kafka has evolved a lot from where it started. It did start
> >as
> >a messaging system. Over time we realized that that the vision for Kafka
> >is
> >to build a streaming platform and not just a messaging system. You can
> >take
> >a look at the site for more description about what comprises the streaming
> >platform http://kafka.apache.org/ and http://kafka.apache.org/intro.
> >
> >Can the streaming platform exist without Connect? - No. Data integration
> >is
> >fundamental to building an end to end platform
> >
> >Can the streaming platform exist without stream processing? - No.
> >Processing stream data again is a core part of streaming platform.
> >
> >Can the streaming platform exist without clients? - We at least need one
> >client library to complete the platform. Our Java clients help us to
> >complete the platform story. The rest of the clients are built and
> >maintained outside the project.
> >
> >Can the platform exist without the rest proxy? - Yes. The proxy does not
> >complete the platform vision in anyway. It is just a good to have tool
> >that
> >might be required by quite a few users and there is an active project that
> >works on this - https://github.com/confluentinc/kafka-rest
> >
> >
> >
> >
> >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> ><ns...@linkedin.com.invalid>
> >wrote:
> >
> >> Are you saying Kafka REST is subjective but Kafka Streams and Kafka
> >Connect
> >> are not subjective?
> >>
> >> > "there are likely places that can live without a rest proxy"
> >>
> >> There are also places that can live without Kafka Streams and Kafka
> >> Connect.
> >>
> >> Nacho
> >>
> >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io> wrote:
> >>
> >> > At the high level, I think ideally it makes sense to add a component
> >>to
> >> > Apache Kafka if (1) it's widely needed and (2) it needs tight
> >integration
> >> > with Kafka core. For Kafka Stream, we do expect stream processing will
> >be
> >> > used widely in the future. Implementation wise, Kafka Stream only
> >> supports
> >> > getting data from Kafka and leverages quite a few of the core
> >> > functionalities in Kafka core. For example, it uses customized
> >>rebalance
> >> > callback in the consumer and uses the compacted topic heavily. So,
> >having
> >> > Kafka Stream in the same repo makes it easier for testing when those
> >core
> >> > functionalities evolve over time. Kafka Connect is in the same
> >situation.
> >> >
> >> > For rest proxy, whether it's widely used or not is going to be a bit
> >> > subjective. However, there are likely places that can live without a
> >rest
> >> > proxy. The rest proxy is just a proxy for the regular clients and
> >doesn't
> >> > need to be tightly integrated with Kafka core. So, the case for
> >including
> >> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream
> >>and
> >> > Kafka Connect.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> >><Mi...@ig.com>
> >> > wrote:
> >> >
> >> > > So from my reading essentially the first question needs to
> >answered/and
> >> > > voted on is:
> >> > >
> >> > > Is Apache Kafka Community only about the Core or does the apache
> >> > community
> >> > > also support some subprojects (and just we need some better way to
> >> manage
> >> > > this)
> >> > >
> >> > > If vote for Core only wins, then the following should be removed:
> >> > > Kafka Connect
> >> > > Kafka Stream
> >> > >
> >> > > If vote for Core only loses (aka we will support subprojects) then:
> >> > > We should look to add Kafka Rest
> >> > >
> >> > > And we should look to see how we can manage better govern and manage
> >> > > submodules.
> >> > >
> >> > > A good example which id propose here is how some other communities
> >>in
> >> > > Apache do this.
> >> > >
> >> > > Each Module has a Module Management Committee(MMC), this is like
> >almost
> >> > > the PMC but at a per module basis.
> >> > >
> >> > > This MMC should essentially hold the binding votes for that module.
> >> > > The MMC should be made up of a single representative from each
> >> > > organisation (so no single organisation can fully veto the
> >>community
> >> it
> >> > > has to a genuine consenus)
> >> > > The MMC requires at least 3 members (so there cant be a tied vote on
> >2)
> >> > > For a new Module to be added a MMC committee should be sought
> >> > > A new Module is only capable of being added if the above
> >>requirements
> >> can
> >> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
> >that
> >> > > only actively support modules would be added
> >> > >
> >> > > The PMC reviews each module every 6months or Year. If MMC is
> >>inactive,
> >> a
> >> > > vote/call to find replacements if raised, if none are forthcoming
> >> > dropping
> >> > > the MMC to less than 3 then the module moves to "the attic" (very
> >>much
> >> > like
> >> > > apache attic but a little more aggressively)
> >> > >
> >> > > This way the PMC does not need to micro manage every module
> >> > > We only add modules where some amount of active support and
> >maintenance
> >> > > and use is provided by the community
> >> > > We have an automatic way to retire old or inactive projects.
> >> > >
> >> > > Thoughts?
> >> > > Mike
> >> > >
> >> > >
> >> > > ________________________________________
> >> > > From: Harsha Ch <ha...@gmail.com>
> >> > > Sent: Thursday, October 20, 2016 10:26 PM
> >> > > To: dev@kafka.apache.org
> >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >> > >
> >> > > Jay,
> >> > > REST API is something every user is in need of. If the
> >>argument
> >> is
> >> > to
> >> > > clone and write your API, this will do a disservice to the users as
> >> they
> >> > > now have to choose one vs. others instead of keeping one API that is
> >> > > supported in Kafka community.
> >> > >
> >> > > "Pre-emptively re-creating another
> >> > > REST layer when it seems like we all quite agree on what needs to be
> >> done
> >> > > and we have an existing code base for HTTP/Kafka access that is
> >heavily
> >> > > used in production seems quite silly."
> >> > >
> >> > > Exactly our point. Why can't we develop this in Apache Kafka
> >> > > community? Instead of us open sourcing another GitHub project and
> >> > creating
> >> > > a divide in users and another version of API. Let's build this in
> >Kafka
> >> > > Community and use the governance model that is proven to provide
> >vendor
> >> > > free user driven consensus features. The argument that is adding
> >>this
> >> > REST
> >> > > server to Kafka will affect the agility of the project doesn't mak
> >> sense.
> >> > >
> >> > > It looks like your argument is either we develop all these small
> >>tools
> >> or
> >> > > none at all. We as a community need to look at supporting critical
> >> > > tools/API. Instead of dividing this project into individual external
> >> > > communities. We should build this as part of Kafka which best serves
> >> the
> >> > > needs of users.
> >> > > The Streams and Connect projects that were pushed into Kafka
> >> > could
> >> > > have been left in their own Github projects based on your arguments.
> >> What
> >> > > about the REST API is so different that such that it should stay out
> >of
> >> > the
> >> > > Kafka project? From my experience, more users are asking for the
> >>REST
> >> > API.
> >> > >
> >> > > Thanks,
> >> > > Harsha
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
> >> > >
> >> > > > I think the questions around governance make sense, I think we
> >should
> >> > > > really clarify that to make the process more clear so it can be
> >fully
> >> > > > inclusive.
> >> > > >
> >> > > > The idea that we should not collaborate on what is there now,
> >though,
> >> > > > because in the future we might disagree about direction does not
> >> really
> >> > > > make sense to me. If in the future we disagree, that is the beauty
> >of
> >> > > open
> >> > > > source, you can always fork off a copy of the code and start an
> >> > > independent
> >> > > > project either in Apache or elsewhere. Pre-emptively re-creating
> >> > another
> >> > > > REST layer when it seems like we all quite agree on what needs to
> >>be
> >> > done
> >> > > > and we have an existing code base for HTTP/kafka access that is
> >> heavily
> >> > > > used in production seems quite silly.
> >> > > >
> >> > > > Let me give some background on how I at least think about these
> >> things.
> >> > > > I've participated in open source projects out of LinkedIn via
> >>github
> >> as
> >> > > > well as via the ASF. I don't think there is a "right" answer to
> >>how
> >> to
> >> > do
> >> > > > these but rather some tradeoffs. We thought about this quite a lot
> >in
> >> > the
> >> > > > context of Kafka based on the experience with the Hadoop ecosystem
> >as
> >> > > well
> >> > > > as from other open source communities.
> >> > > >
> >> > > > There is a rich ecosystem around Kafka. Many of the projects are
> >> quite
> >> > > > small--single clients or tools that do a single thing well--and
> >> almost
> >> > > none
> >> > > > of them are top level apache projects. I don't think trying to
> >>force
> >> > each
> >> > > > of these to turn into independent Apache projects is necessarily
> >>the
> >> > best
> >> > > > thing for the ecosystem.
> >> > > >
> >> > > > My observation of how this can go wrong is really what I think has
> >> > > happened
> >> > > > to the Hadoop ecosystem. There you see quite a zoo of projects
> >>which
> >> > all
> >> > > > drift apart and don't quite work together well. Coordinating even
> >> > simple
> >> > > > changes and standardization across these is exceptionally
> >>difficult.
> >> > The
> >> > > > result is a bit of a mess for users--the pieces just don't really
> >> come
> >> > > > together very well. This makes sense for independent
> >>infrastructure
> >> > > systems
> >> > > > (Kudu vs HDFS) but I'm not at all convinced that doing this for
> >every
> >> > > > little tool or helper library has lead to a desirable state. I
> >>think
> >> > the
> >> > > > mode of operating where the Hadoop vendors spawn off a few new
> >Apache
> >> > > > projects for each new product initiative, especially since often
> >that
> >> > > > project is only valued by that vendor (and the other vendor has a
> >> > > different
> >> > > > competing Apache project) doesn't necessarily do a better job at
> >> > > producing
> >> > > > high quality communities or high quality software.
> >> > > >
> >> > > > These tools/connects/clients/proxies and other integration pieces
> >can
> >> > > take
> >> > > > many forms, but my take of what makes one of these things good is
> >> that
> >> > it
> >> > > > remains simple, does its one thing well, and cleaves as closely as
> >> > > possible
> >> > > > to the conventions for Kafka itself--i.e. doesn't invent new ways
> >>of
> >> > > > monitoring, configuring, etc. For the tools we've contributed
> >>we've
> >> > tried
> >> > > > really hard to make them consistent with Kafka as well as with
> >>each
> >> > other
> >> > > > in how testing, configuration, monitoring, etc works.
> >> > > >
> >> > > > I think what Apache does superbly well is create a community for
> >> > > managing a
> >> > > > large infrastructure layer like Kafka in a vendor independent way.
> >> > What I
> >> > > > think is less successful is attempting to form full and
> >>independent
> >> > > apache
> >> > > > communities around very simple single purpose tools, especially if
> >> you
> >> > > hope
> >> > > > for these to come together into a cohesive toolset across multiple
> >> such
> >> > > > tools. Much of what Apache does--create a collective decision
> >>making
> >> > > > process for resolving disagreement, help to trademark and protect
> >the
> >> > > marks
> >> > > > of the project, etc just isn't that relevant for simple
> >> single-purpose
> >> > > > tools.
> >> > > >
> >> > > > So my take is there are a couple of options:
> >> > > >
> >> > > > 1. We can try to put all the small tools into the Apache
> >>Project.
> >> I
> >> > > > think this is not the right approach as there is simply too
> >>many
> >> of
> >> > > > them,
> >> > > > many in different languages, serving different protocols,
> >> > integrating
> >> > > > with
> >> > > > particular systems, and a single community can't effectively
> >> > maintain
> >> > > > them
> >> > > > all. Doing this would significantly slow the progress of the
> >Kafka
> >> > > > project.
> >> > > > As a protocol for messaging, I don't really see a case for
> >> including
> >> > > > REST
> >> > > > but not MQTT or AMQP which are technically much better suited
> >>to
> >> > > > messaging
> >> > > > and both are widely used for that.
> >> > > > 2. We can treat ecosystem projects that aren't top level Apache
> >> > > projects
> >> > > > as invalid and try to recreate them all as Apache projects.
> >> > Honestly,
> >> > > > though, if you go to the Kafka ecosystem page virtually none of
> >> the
> >> > > most
> >> > > > popular add-ons to Kafka are Apache projects. The most
> >>successful
> >> > > > things in
> >> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number
> >of
> >> > > other
> >> > > > clients, as well as the existing REST layer have succeeded at
> >> > > developing
> >> > > > communities that actively contribute and use these pieces and I
> >> > don't
> >> > > > know
> >> > > > that that is a bad thing unless that community proves to be
> >> > > uninclusive,
> >> > > > unresponsive, or goes in a bad technical direction--and those
> >>are
> >> > > > failure
> >> > > > modes that all open source efforts face.
> >> > > > 3. We can do what I think makes the most sense and try to work
> >> with
> >> > > the
> >> > > > projects that exist in the ecosystem and if the project doesn't
> >> > have a
> >> > > > responsive community or wants to go in a different direction
> >>fork
> >> or
> >> > > > recreate that work.
> >> > > >
> >> > > > Of course any person can choose whatever of these options they
> >>want.
> >> > But
> >> > > > from my point of view, option (3) has been the path of the
> >>community
> >> so
> >> > > far
> >> > > > and I think it has been quite successful.
> >> > > >
> >> > > > -Jay
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> >> kafka@harsha.io
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > Neha,
> >> > > > > "But I haven't seen any community emails or patches being
> >submitted
> >> > by
> >> > > > you
> >> > > > > guys, so I'm wondering why you are concerned about whether the
> >> > > community
> >> > > > is
> >> > > > > open to accepting patches or not."
> >> > > > >
> >> > > > > I think you are talking about contributing patches to this
> >> repository
> >> > > > > right? https://github.com/confluentinc/kafka-rest . All I am
> >> saying
> >> > > > > the guidelines/governance model is not clear on the project and
> >>I
> >> > guess
> >> > > > its
> >> > > > > driven by opening a github issue request. Its the repository
> >owned
> >> > by
> >> > > > > confluent and as much I appreciate that the features we
> >>mentioned
> >> are
> >> > > in
> >> > > > > the roadmap and welcoming us to contribute to the project. It
> >> doesn't
> >> > > > > gurantee what we want to add in the furture will be in your
> >> roadmap.
> >> > > > >
> >> > > > > Hence the reason having it part of Kafka community will help a
> >>lot
> >> as
> >> > > > other
> >> > > > > users can participate in the discussions. We are happy to drive
> >> any
> >> > > > > feature additions through KIPs which gives everyone a chance to
> >> > > > participate
> >> > > > > and add to the discussions.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Harsha
> >> > > > >
> >> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> >> > Michael.Pearce@ig.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > +1
> >> > > > > >
> >> > > > > > I agree on the governance comments whole heartedly.
> >> > > > > >
> >> > > > > > Also i agree about the contribution comments made earlier in
> >>the
> >> > > > thread,
> >> > > > > i
> >> > > > > > personally am less likely to spend any of mine, or give
> >>project
> >> > time
> >> > > > > within
> >> > > > > > my internal projects to developers contributing to another
> >> > commercial
> >> > > > > > companies project even so technically open source, as then
> >>there
> >> is
> >> > > > that
> >> > > > > > commercial companies interest will always prevail and
> >essentially
> >> > can
> >> > > > > > always have a final vote where disagreement. Im sure they
> >>never
> >> > > intend
> >> > > > > to,
> >> > > > > > but there is that true reality. This is why we have community
> >> open
> >> > > > source
> >> > > > > > projects.
> >> > > > > >
> >> > > > > > I can find many different implementations now of a rest
> >>endpoint
> >> on
> >> > > > > > GitHub, BitBucket etc. Each one has their benefits and
> >> > disadvantages
> >> > > in
> >> > > > > > their implementation. By making / providing one this would
> >>bring
> >> > > > together
> >> > > > > > these solutions, unifying those developers and also bringing
> >>the
> >> > best
> >> > > > of
> >> > > > > > all.
> >> > > > > >
> >> > > > > > I understand the concern on the community burden
> >> adding/supporting
> >> > > more
> >> > > > > > surface area for every client. But something like REST is
> >> universal
> >> > > and
> >> > > > > > worthy to be owned by the community.
> >> > > > > >
> >> > > > > > Mike
> >> > > > > >
> >> > > > > >
> >> > > > > > ________________________________________
> >> > > > > > From: Andrew Schofield <an...@outlook.com>
> >> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> >> > > > > > To: dev@kafka.apache.org
> >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >> > > > > >
> >> > > > > > There's a massive difference between the governance of Kafka
> >>and
> >> > the
> >> > > > > > governance of the REST proxy.
> >> > > > > >
> >> > > > > > In Kafka, there is a broad community of people contributing
> >their
> >> > > > > opinions
> >> > > > > > about future enhancements in the form of KIPs. There's some
> >> really
> >> > > deep
> >> > > > > > consideration that goes into some of the trickier KIPs. There
> >are
> >> > > > people
> >> > > > > > outside Confluent deeply knowledgeable in Kafka and building
> >the
> >> > > > > > reputations to become committers. I get the impression that
> >>the
> >> > > roadmap
> >> > > > > of
> >> > > > > > Kafka is not really community-owned (what's the big feature
> >>for
> >> > Kafka
> >> > > > > 0.11,
> >> > > > > > for example), but the conveyor belt of smaller features in the
> >> form
> >> > > of
> >> > > > > KIPs
> >> > > > > > works nicely. It's a good example of open-source working
> >>well.
> >> > > > > >
> >> > > > > > The equivalent for the REST proxy is basically issues on
> >>GitHub.
> >> > The
> >> > > > > > roadmap is less clear. There's not really a community properly
> >> > > engaged
> >> > > > in
> >> > > > > > the way that there is with Kafka. So, you could say that it's
> >> clear
> >> > > > that
> >> > > > > > fewer people are interested, but I think the whole governance
> >> > thing
> >> > > > is a
> >> > > > > > big barrier to engagement. And it's looking like it's getting
> >out
> >> > of
> >> > > > > date.
> >> > > > > >
> >> > > > > > In technical terms, I can think of two big improvements to the
> >> REST
> >> > > > > proxy.
> >> > > > > > First, it needs to use the new consumer API so that it's
> >possible
> >> > to
> >> > > > > secure
> >> > > > > > connections between the REST proxy and Kafka. I don't care too
> >> much
> >> > > > which
> >> > > > > > method calls it uses actually uses to consume messages, but I
> >do
> >> > > care
> >> > > > > that
> >> > > > > > I cannot secure connections because of network security rules.
> >> > > Second,
> >> > > > > > there's an affinity between a consumer and the instance of the
> >> REST
> >> > > > proxy
> >> > > > > > to which it first connected. Kafka itself avoids this kind of
> >> > > affinity
> >> > > > > for
> >> > > > > > good reason, and in the name of availability the REST proxy
> >> should
> >> > > too.
> >> > > > > > These are natural KIPs.
> >> > > > > >
> >> > > > > > I think it would be good to have the code for the REST proxy
> >> > > > contributed
> >> > > > > > to Apache Kafka so that it would be able to be developed in
> >>the
> >> > same
> >> > > > way.
> >> > > > > >
> >> > > > > > Andrew Schofield
> >> > > > > >
> >> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> >> > > > > > Sent: 07 October 2016 22:41:52
> >> > > > > > To: dev@kafka.apache.org
> >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >> > > > > >
> >> > > > > > ASF already gives us a clear framework and governance model
> >>for
> >> > > > community
> >> > > > > > development. This is already understood by the people
> >> contributing
> >> > to
> >> > > > > > Apache Kafka project, and they are the same people who want to
> >> > > > contribute
> >> > > > > > to the REST server capability as well. Everyone is in
> >>agreement
> >> on
> >> > > the
> >> > > > > > need for collaborating on this effort. So why not contribute
> >>the
> >> > code
> >> > > > to
> >> > > > > > Apache Kafka. This will help avoid duplication of effort and
> >> forks
> >> > > that
> >> > > > > > may crop up, hugely benefitting the user community. This will
> >> also
> >> > > > avoid
> >> > > > > > having to define a process similar to ASF on a GitHub project
> >and
> >> > > > instead
> >> > > > > > there is a single community with clear understanding community
> >> > > process
> >> > > > as
> >> > > > > > defined in ASF.
> >> > > > > >
> >> > > > > > As others have said, this is an important capability for
> >>Apache
> >> > > Kafka.
> >> > > > It
> >> > > > > > is worth maintaining this as a part of the project.
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > > Suresh
> >> > > > > >
> >> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io>
> >>wrote:
> >> > > > > >
> >> > > > > > >I personally think it would be quite wasteful to re-implement
> >> the
> >> > > REST
> >> > > > > > >gateway just because that an actively-maintained piece of
> >> > > > > Apache-licensed
> >> > > > > > >software is not governed directly by the Apache Kafka
> >> community...
> >> > > > While
> >> > > > > > >kafka-rest repo is owned by Confluent, the contributors
> >> including
> >> > > the
> >> > > > > main
> >> > > > > > >one are also part of the Apache Kafka community, so there
> >>is a
> >> > > chance
> >> > > > > to
> >> > > > > > >work this out.
> >> > > > > > >
> >> > > > > > >However, there are two valid concerns here that could be
> >> > addressed,
> >> > > > > around
> >> > > > > > >community and accessibility:
> >> > > > > > >>> What we are worried about is a project
> >> > > > > > >>> that's not maintained in a community. So the process of
> >> > accepting
> >> > > > > > >>>patches
> >> > > > > > >>> and priorities is not clear, and it's not developed in
> >Apache
> >> > > > > > >>>community.
> >> > > > > > >>> Not only that, existing REST API project doesn't support
> >>new
> >> > > client
> >> > > > > API
> >> > > > > > >and
> >> > > > > > >>> hence there is no security support either.
> >> > > > > > >
> >> > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
> >> community
> >> > > can
> >> > > > > > >clarify that - what is their contribution policy, dev style,
> >> > roadmap
> >> > > > > etc.
> >> > > > > > >If they want, they can make an effort to encourage
> >participation
> >> > > from
> >> > > > > > >people outside Confluent (easily accept contributions, invite
> >> > > external
> >> > > > > > >commiters or have open dev process similar to Apache Kafka
> >etc),
> >> > as
> >> > > > > there
> >> > > > > > >is definitely seems to be some interest on the list. That
> >>might
> >> > > clear
> >> > > > > the
> >> > > > > > >community concern and help kafka-rest project (but that is a
> >> > > > calculation
> >> > > > > > >Confluent will have to make).
> >> > > > > > >
> >> > > > > > >The other, independent, concern is that REST is something
> >>that
> >> is
> >> > > > > expected
> >> > > > > > >to be available out of the box with Kafka. I personally don't
> >> feel
> >> > > > > > >strongly
> >> > > > > > >about it (better use proper, efficient APIs from day one),
> >> though
> >> > it
> >> > > > is
> >> > > > > > >definitely way smaller than adding a stream processing engine
> >to
> >> > the
> >> > > > > > >project :)
> >> > > > > > >Again,the kafka-rest "community" could take steps to make it
> >> even
> >> > > > easier
> >> > > > > > >to
> >> > > > > > >install, configure and run kafka-rest for new users on
> >>vanilla
> >> > > Apache
> >> > > > > > >Kafka
> >> > > > > > >(outside the Confluent platform), if they wish that (or
> >>welcome
> >> > > > > > >contributions to that end), but that is up to them.
> >> > > > > > >Finally, if after the above steps were taken there would
> >>still
> >a
> >> > > > strong
> >> > > > > > >desire to include a great rest gateway with Apache Kafka, I
> >> assume
> >> > > the
> >> > > > > > >community could hypothetically fork the existing kafka-rest
> >into
> >> > an
> >> > > > > Apache
> >> > > > > > >Kafka subproject and maintain it "within Apache" instead of
> >> > > > implementing
> >> > > > > > >it
> >> > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot
> >> imagine
> >> > it
> >> > > > > > >happen
> >> > > > > > >without Confluent blessing, and I think that is likely much
> >less
> >> > > > optimal
> >> > > > > > >(pulling in other Confluent / Apache licensed dependencies)
> >than
> >> > > > having
> >> > > > > a
> >> > > > > > >separate external community around kafka-rest.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >Just my two cents,
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >Ofir Manor
> >> > > > > > >
> >> > > > > > >Co-Founder & CTO | Equalum
> >> > > > > > >
> >> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> ><+972%2054-780-1286>
> >> > <+972%2054-780-1286> |
> >> > > > Email:
> >> > > > > > ofir.manor@equalum.io
> >> > > > > > >
> >> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> >> > > kafka@harsha.io
> >> > > > >
> >> > > > > > >wrote:
> >> > > > > > >
> >> > > > > > >> Neha & Jay,
> >> > > > > > >> We did look at the open source
> >>alternatives.
> >> > Our
> >> > > > > > >>concern
> >> > > > > > >> is what's the patch acceptance and adding features/
> >>bug-fixes
> >> to
> >> > > the
> >> > > > > > >> existing project under a Github (although it's licensed
> >>under
> >> > > Apache
> >> > > > > > >>2.0).
> >> > > > > > >> It would be great if that project made available under
> >>Apache
> >> > and
> >> > > > > > >>driven by
> >> > > > > > >> the community. Adding to the above, not all Kafka users
> >>are
> >> > > > > interested
> >> > > > > > >>in
> >> > > > > > >> using the Java client API, they would like to have simple
> >REST
> >> > API
> >> > > > > where
> >> > > > > > >> they can code against using any language. I do believe this
> >> adds
> >> > > > value
> >> > > > > > >>to
> >> > > > > > >> Apache Kafka in itself.
> >> > > > > > >>
> >> > > > > > >> "For 1, I don't think there is value in giving in to the
> >>NIH
> >> > > > syndrome
> >> > > > > > >>and
> >> > > > > > >> reinventing the wheel. What I'm looking for is a detailed
> >> > > comparison
> >> > > > > of
> >> > > > > > >>the
> >> > > > > > >> gaps and why those can't be improved in the REST proxy that
> >> > > already
> >> > > > > > >>exists
> >> > > > > > >> and is actively maintained."
> >> > > > > > >>
> >> > > > > > >> We are not looking at this as NIH. What we are worried
> >>about
> >> > is a
> >> > > > > > >>project
> >> > > > > > >> that's not maintained in a community. So the process of
> >> > accepting
> >> > > > > > >>patches
> >> > > > > > >> and priorities is not clear, and it's not developed in
> >>Apache
> >> > > > > community.
> >> > > > > > >> Not only that, existing REST API project doesn't support
> >>new
> >> > > client
> >> > > > > API
> >> > > > > > >>and
> >> > > > > > >> hence there is no security support either.
> >> > > > > > >> We don't know the timeline when that's made available. We
> >> would
> >> > > like
> >> > > > > to
> >> > > > > > >>add
> >> > > > > > >> admin functionality into the REST API. So the Roadmap of
> >>that
> >> > > > project
> >> > > > > is
> >> > > > > > >> not driven by Apache.
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > > >> "This doesn't materially have an impact on expanding the
> >> > usability
> >> > > > of
> >> > > > > > >> Kafka. In my experience, REST proxy + Java clients only
> >> cover
> >> > > > ~50%
> >> > > > > of
> >> > > > > > >> all
> >> > > > > > >> Kafka users, and maybe 10% of those are the ones who
> >>will
> >> use
> >> > > the
> >> > > > > > >>REST
> >> > > > > > >> proxy. The remaining 50% are non-java client users (C,
> >> > python,
> >> > > > go,
> >> > > > > > >>node
> >> > > > > > >> etc)."
> >> > > > > > >>
> >> > > > > > >> REST API is most often asked feature in my interactions
> >>with
> >> > Kafka
> >> > > > > > >>users.
> >> > > > > > >> In an organization, There will be independent teams who
> >>will
> >> > write
> >> > > > > their
> >> > > > > > >> Kafka clients using different language libraries available
> >> > today,
> >> > > > and
> >> > > > > > >> there is no way to standardize this. Instead of supporting
> >> > several
> >> > > > > > >> different client libraries users will be interested in
> >>using
> >a
> >> > > REST
> >> > > > > API
> >> > > > > > >> server. The need for a REST API will only increase as more
> >and
> >> > > more
> >> > > > > > >>users
> >> > > > > > >> start using Kafka.
> >> > > > > > >>
> >> > > > > > >> "More surface area means more work to keep things
> >>consistent.
> >> > > > Failure
> >> > > > > > >> to do that has, in fact, hurt the user experience."
> >> > > > > > >> Having myriad Kafka client GitHub projects that support
> >> > different
> >> > > > > > >>languages
> >> > > > > > >> hurts the user experience and pushes burden to maintain
> >>these
> >> > > > > libraries.
> >> > > > > > >> REST API is a simple code base that uses existing java
> >>client
> >> > > > > libraries
> >> > > > > > >>to
> >> > > > > > >> make life easier for the users.
> >> > > > > > >>
> >> > > > > > >> Thanks,
> >> > > > > > >> Harsha
> >> > > > > > >>
> >> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> >> > neha@confluent.io>
> >> > > > > > wrote:
> >> > > > > > >>
> >> > > > > > >> > Manikumar,
> >> > > > > > >> >
> >> > > > > > >> > Thanks for sharing the proposal. I think there are 2
> >>parts
> >> to
> >> > > this
> >> > > > > > >> > discussion -
> >> > > > > > >> >
> >> > > > > > >> > 1. Should we rewrite a REST proxy given that there is a
> >> > > > > > >>feature-complete,
> >> > > > > > >> > open-source and actively maintained REST proxy in the
> >> > community?
> >> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more
> >> agile
> >> > > and
> >> > > > > > >> maintain
> >> > > > > > >> > the high-quality experience that Kafka users have today?
> >> > > > > > >> >
> >> > > > > > >> > For 1, I don't think there is value in giving in to the
> >>NIH
> >> > > > syndrome
> >> > > > > > >>and
> >> > > > > > >> > reinventing the wheel. What I'm looking for is a detailed
> >> > > > comparison
> >> > > > > > >>of
> >> > > > > > >> the
> >> > > > > > >> > gaps and why those can't be improved in the REST proxy
> >>that
> >> > > > already
> >> > > > > > >> exists
> >> > > > > > >> > and is actively maintained. For example, we depend on
> >> zkClient
> >> > > and
> >> > > > > > >>have
> >> > > > > > >> > found as well as fixed several bugs by working closely
> >>with
> >> > the
> >> > > > > people
> >> > > > > > >> who
> >> > > > > > >> > maintain zkClient. This should be possible for REST proxy
> >as
> >> > > well,
> >> > > > > > >>right?
> >> > > > > > >> >
> >> > > > > > >> > For 2, I'd like us to review our history of expanding the
> >> > > surface
> >> > > > > > >>area to
> >> > > > > > >> > add more clients in the past. Here is a summary -
> >> > > > > > >> >
> >> > > > > > >> > - This doesn't materially have an impact on expanding
> >the
> >> > > > > > >>usability of
> >> > > > > > >> > Kafka. In my experience, REST proxy + Java clients
> >>only
> >> > cover
> >> > > > > ~50%
> >> > > > > > >>of
> >> > > > > > >> > all
> >> > > > > > >> > Kafka users, and maybe 10% of those are the ones who
> >will
> >> > use
> >> > > > the
> >> > > > > > >>REST
> >> > > > > > >> > proxy. The remaining 50% are non-java client users (C,
> >> > > python,
> >> > > > > go,
> >> > > > > > >> node
> >> > > > > > >> > etc).
> >> > > > > > >> > - People are a lot more excited about promising to
> >> > contribute
> >> > > > > while
> >> > > > > > >> > adding the surface area but not on an ongoing basis
> >>down
> >> > the
> >> > > > > line.
> >> > > > > > >> > - More surface area means more work to keep things
> >> > > consistent.
> >> > > > > > >>Failure
> >> > > > > > >> > to do that has, in fact, hurt the user experience.
> >> > > > > > >> > - More surface area hurts agility. We want to do a few
> >> > things
> >> > > > > > >>really
> >> > > > > > >> > well as well as be agile to be able to build on our
> >>core
> >> > > > > > >>competency.
> >> > > > > > >> >
> >> > > > > > >> >
> >> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> >> > > > manikumar.reddy@gmail.com
> >> > > > > >
> >> > > > > > >> > wrote:
> >> > > > > > >> >
> >> > > > > > >> > > Hi Jay,
> >> > > > > > >> > >
> >> > > > > > >> > > Thanks for your reply.
> >> > > > > > >> > >
> >> > > > > > >> > > I agree that we can not add all the clients/tools
> >> available
> >> > in
> >> > > > > > >> ecosystem
> >> > > > > > >> > > page to Kafka repo itself. But we feel REST Interface
> >>is
> >> > > > different
> >> > > > > > >>from
> >> > > > > > >> > > other clients/tools. Since any language that can work
> >with
> >> > > HTTP
> >> > > > > can
> >> > > > > > >> > > easily integrate with this interface, Having an
> >"official"
> >> > > REST
> >> > > > > > >> > > interface helps user community. This also helps us to
> >> > > integrate
> >> > > > > well
> >> > > > > > >> > > with external management and provisioning tools.
> >>Apache
> >> > Kafka
> >> > > > > > >>release
> >> > > > > > >> > > with Java clients + REST interface is sufficient for
> >>most
> >> of
> >> > > the
> >> > > > > > >>user
> >> > > > > > >> > > deployments/requirements. This helps users to deal with
> >> less
> >> > > > > number
> >> > > > > > >> > > of distributions/builds.
> >> > > > > > >> > >
> >> > > > > > >> > > Thanks,
> >> > > > > > >> > > Manikumar
> >> > > > > > >> > >
> >> > > > > > >> > >
> >> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> >> jay@confluent.io
> >> > >
> >> > > > > wrote:
> >> > > > > > >> > >
> >> > > > > > >> > > > Hey guys,
> >> > > > > > >> > > >
> >> > > > > > >> > > > There's already a REST interface maintained as a
> >> separate
> >> > > > > > >> project--it's
> >> > > > > > >> > > > open source and apache licensed and actively
> >>maintained
> >> (
> >> > > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is
> >> > wrong
> >> > > > with
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> >> > > > > > github.com
> >> > > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
> >> > cluster.
> >> > > > It
> >> > > > > > makes it easy to produce and consume messages, view the state
> >>of
> >> > the
> >> > > > > > cluster, and ...
> >> > > > > >
> >> > > > > > >> that?
> >> > > > > > >> > > You
> >> > > > > > >> > > > mentioned that there was some compatibility concern,
> >but
> >> > > > > > >> compatibility
> >> > > > > > >> > > has
> >> > > > > > >> > > > to do with the consumer protocol guarantees not the
> >repo
> >> > the
> >> > > > > code
> >> > > > > > >>is
> >> > > > > > >> > in,
> >> > > > > > >> > > > right? Not sure that concern makes sense.
> >> > > > > > >> > > >
> >> > > > > > >> > > > We could argue for adding pretty much anything and
> >> > > everything
> >> > > > in
> >> > > > > > >>the
> >> > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that
> >> would
> >> > > > make
> >> > > > > > >>the
> >> > > > > > >> > > project
> >> > > > > > >> > > > more agile.
> >> > > > > > >> > > >
> >> > > > > > >> > > > -Jay
> >> > > > > > >> > > >
> >> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> >> > > > > > >> manikumar.reddy@gmail.com
> >> > > > > > >> > >
> >> > > > > > >> > > > wrote:
> >> > > > > > >> > > >
> >> > > > > > >> > > > > Hi Kafka Devs,
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> >> > > > Repository.
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > There are already open-source alternatives are
> >> > available.
> >> > > > But
> >> > > > > > >>we
> >> > > > > > >> > would
> >> > > > > > >> > > > > like to add REST server that
> >> > > > > > >> > > > > many users ask for under Apache Kafka repo. Many
> >>data
> >> > > Infra
> >> > > > > > >>tools
> >> > > > > > >> > comes
> >> > > > > > >> > > > up
> >> > > > > > >> > > > > with Rest Interface.
> >> > > > > > >> > > > > It is useful to have inbuilt Rest API support for
> >> > Produce,
> >> > > > > > >>Consume
> >> > > > > > >> > > > messages
> >> > > > > > >> > > > > and admin interface for
> >> > > > > > >> > > > > integrating with external management and
> >>provisioning
> >> > > > > tools.This
> >> > > > > > >> will
> >> > > > > > >> > > > also
> >> > > > > > >> > > > > allow the maintenance of
> >> > > > > > >> > > > > REST server and adding new features makes it easy
> >> > because
> >> > > > > apache
> >> > > > > > >> > > > community.
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > The KIP wiki is the following:
> >> > > > > > >> > > > > https://cwiki.apache.org/
> >> confluence/display/KAFKA/KIP-
> >> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > Your comments and feedback are welcome.
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > Thanks,
> >> > > > > > >> > > > > Manikumar
> >> > > > > > >> > > > >
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> > --
> >> > > > > > >> > Thanks,
> >> > > > > > >> > Neha
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > > The information contained in this email is strictly
> >>confidential
> >> > and
> >> > > > for
> >> > > > > > the use of the addressee only, unless otherwise indicated. If
> >you
> >> > are
> >> > > > not
> >> > > > > > the intended recipient, please do not read, copy, use or
> >disclose
> >> > to
> >> > > > > others
> >> > > > > > this message or any attachment. Please also notify the sender
> >>by
> >> > > > replying
> >> > > > > > to this email or by telephone (+44(020 7896 0011) and then
> >delete
> >> > the
> >> > > > > email
> >> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> >> relate
> >> > > to
> >> > > > > the
> >> > > > > > official business of this company shall be understood as
> >>neither
> >> > > given
> >> > > > > nor
> >> > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> >> > company
> >> > > > > > registered in England and Wales, company number 04008957) and
> >>IG
> >> > > Index
> >> > > > > > Limited (a company registered in England and Wales, company
> >> number
> >> > > > > > 01190902). Registered address at Cannon Bridge House, 25
> >>Dowgate
> >> > > Hill,
> >> > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> >195355)
> >> > and
> >> > > > IG
> >> > > > > > Index Limited (register number 114059) are authorised and
> >> regulated
> >> > > by
> >> > > > > the
> >> > > > > > Financial Conduct Authority.
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > The information contained in this email is strictly confidential and
> >> for
> >> > > the use of the addressee only, unless otherwise indicated. If you
> >>are
> >> not
> >> > > the intended recipient, please do not read, copy, use or disclose to
> >> > others
> >> > > this message or any attachment. Please also notify the sender by
> >> replying
> >> > > to this email or by telephone (+44(020 7896 0011) and then delete
> >>the
> >> > email
> >> > > and any copies of it. Opinions, conclusion (etc) that do not relate
> >>to
> >> > the
> >> > > official business of this company shall be understood as neither
> >>given
> >> > nor
> >> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> >>company
> >> > > registered in England and Wales, company number 04008957) and IG
> >>Index
> >> > > Limited (a company registered in England and Wales, company number
> >> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> >>Hill,
> >> > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> >>and
> >> IG
> >> > > Index Limited (register number 114059) are authorised and regulated
> >>by
> >> > the
> >> > > Financial Conduct Authority.
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Nacho (Ignacio) Solis
> >> Kafka
> >> nsolis@linkedin.com
> >>
>
>
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Suresh Srinivas <su...@hortonworks.com>.
I am dividing this discussion into two parts:
1. REST APIs as core Apache Kafka capability
This should be a core Kafka functionality. Same view has been reflected by
others (users and developers) as well. While we can debate whether other
capabilities are core Kafka (Streams, Connect), it would be good separate
that out and to keep this discussion focussed on REST APIs as proposed by
this KIP. If there is ambivalence about the need for this in core kafka,
we could have voting in the mailing list.
Can we get an agreement on this? I am +1 on REST API in Apache Kafka.
2. Community where Kafka REST APIs need to be collaborated on
There is already a GitHub project that caters to Kafka REST APIs. But a
company owned Github is less than ideal for collaboration due to lack of
governance, open community and roadmap. This is reflected by many people
interested in this functionality and still not contributing to and
adopting the code base in the GitHub. I think trying overlay the ASF
governance model on GitHub project, which is what the need is, seems
unnecessary, if the code can be part of Apache Kafka.
The question is, would Confluent be okay with licensing/contributing the
code to Kafka project (assuming either Confluent or another contributor
can work on it)? I see clear intent in collaborating with others on REST
APIs from confluent. Why not do it in Kafka project under ASF? This takes
away all the barrier to collaboration? Alternatively, if Confluent is not
willing to contribute the code from GitHub, would anyone veto building a
separate REST API functionality in ASF Kafka? There are enough people who
want to work on this and maintain it.
Regards,
Suresh
On 10/21/16, 9:41 PM, "Harsha Chintalapani" <ka...@harsha.io> wrote:
>Sriram,
> "Can the streaming platform exist without stream processing? - No.
>Processing stream data again is a core part of streaming platform."
>
>Yes, it can. There are no.of Stream processing frameworks out there, and
>they all have integration into Kafka.
>It doesn't need to be developed within Kafka.
>
>
>"Can the platform exist without the rest proxy? - Yes. The proxy does not
>complete the platform vision in anyway. It is just a good to have tool
>that
>might be required by quite a few users and there is an active project that
>works on this - https://github.com/confluentinc/kafka-rest"
>
>The rest proxy is as important as any API. The vision that shown here
>http://kafka.apache.org/intro
>require users to write the producers and consumers to get their data into
>and out of Kafka, without which having Streams or Connect won't help
>anyone.
>The rest proxy makes easier for users get their data into Kafka.
>Adding the rest proxy to the project doesn't invalidate the current
>vision,
>it only strengthens it.
>
>Thanks,
>Harsha
>
>
>
>
>On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <ra...@confluent.io>
>wrote:
>
>FWIW, Apache Kafka has evolved a lot from where it started. It did start
>as
>a messaging system. Over time we realized that that the vision for Kafka
>is
>to build a streaming platform and not just a messaging system. You can
>take
>a look at the site for more description about what comprises the streaming
>platform http://kafka.apache.org/ and http://kafka.apache.org/intro.
>
>Can the streaming platform exist without Connect? - No. Data integration
>is
>fundamental to building an end to end platform
>
>Can the streaming platform exist without stream processing? - No.
>Processing stream data again is a core part of streaming platform.
>
>Can the streaming platform exist without clients? - We at least need one
>client library to complete the platform. Our Java clients help us to
>complete the platform story. The rest of the clients are built and
>maintained outside the project.
>
>Can the platform exist without the rest proxy? - Yes. The proxy does not
>complete the platform vision in anyway. It is just a good to have tool
>that
>might be required by quite a few users and there is an active project that
>works on this - https://github.com/confluentinc/kafka-rest
>
>
>
>
>On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
><ns...@linkedin.com.invalid>
>wrote:
>
>> Are you saying Kafka REST is subjective but Kafka Streams and Kafka
>Connect
>> are not subjective?
>>
>> > "there are likely places that can live without a rest proxy"
>>
>> There are also places that can live without Kafka Streams and Kafka
>> Connect.
>>
>> Nacho
>>
>> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>> > At the high level, I think ideally it makes sense to add a component
>>to
>> > Apache Kafka if (1) it's widely needed and (2) it needs tight
>integration
>> > with Kafka core. For Kafka Stream, we do expect stream processing will
>be
>> > used widely in the future. Implementation wise, Kafka Stream only
>> supports
>> > getting data from Kafka and leverages quite a few of the core
>> > functionalities in Kafka core. For example, it uses customized
>>rebalance
>> > callback in the consumer and uses the compacted topic heavily. So,
>having
>> > Kafka Stream in the same repo makes it easier for testing when those
>core
>> > functionalities evolve over time. Kafka Connect is in the same
>situation.
>> >
>> > For rest proxy, whether it's widely used or not is going to be a bit
>> > subjective. However, there are likely places that can live without a
>rest
>> > proxy. The rest proxy is just a proxy for the regular clients and
>doesn't
>> > need to be tightly integrated with Kafka core. So, the case for
>including
>> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream
>>and
>> > Kafka Connect.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
>><Mi...@ig.com>
>> > wrote:
>> >
>> > > So from my reading essentially the first question needs to
>answered/and
>> > > voted on is:
>> > >
>> > > Is Apache Kafka Community only about the Core or does the apache
>> > community
>> > > also support some subprojects (and just we need some better way to
>> manage
>> > > this)
>> > >
>> > > If vote for Core only wins, then the following should be removed:
>> > > Kafka Connect
>> > > Kafka Stream
>> > >
>> > > If vote for Core only loses (aka we will support subprojects) then:
>> > > We should look to add Kafka Rest
>> > >
>> > > And we should look to see how we can manage better govern and manage
>> > > submodules.
>> > >
>> > > A good example which id propose here is how some other communities
>>in
>> > > Apache do this.
>> > >
>> > > Each Module has a Module Management Committee(MMC), this is like
>almost
>> > > the PMC but at a per module basis.
>> > >
>> > > This MMC should essentially hold the binding votes for that module.
>> > > The MMC should be made up of a single representative from each
>> > > organisation (so no single organisation can fully veto the
>>community
>> it
>> > > has to a genuine consenus)
>> > > The MMC requires at least 3 members (so there cant be a tied vote on
>2)
>> > > For a new Module to be added a MMC committee should be sought
>> > > A new Module is only capable of being added if the above
>>requirements
>> can
>> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
>that
>> > > only actively support modules would be added
>> > >
>> > > The PMC reviews each module every 6months or Year. If MMC is
>>inactive,
>> a
>> > > vote/call to find replacements if raised, if none are forthcoming
>> > dropping
>> > > the MMC to less than 3 then the module moves to "the attic" (very
>>much
>> > like
>> > > apache attic but a little more aggressively)
>> > >
>> > > This way the PMC does not need to micro manage every module
>> > > We only add modules where some amount of active support and
>maintenance
>> > > and use is provided by the community
>> > > We have an automatic way to retire old or inactive projects.
>> > >
>> > > Thoughts?
>> > > Mike
>> > >
>> > >
>> > > ________________________________________
>> > > From: Harsha Ch <ha...@gmail.com>
>> > > Sent: Thursday, October 20, 2016 10:26 PM
>> > > To: dev@kafka.apache.org
>> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>> > >
>> > > Jay,
>> > > REST API is something every user is in need of. If the
>>argument
>> is
>> > to
>> > > clone and write your API, this will do a disservice to the users as
>> they
>> > > now have to choose one vs. others instead of keeping one API that is
>> > > supported in Kafka community.
>> > >
>> > > "Pre-emptively re-creating another
>> > > REST layer when it seems like we all quite agree on what needs to be
>> done
>> > > and we have an existing code base for HTTP/Kafka access that is
>heavily
>> > > used in production seems quite silly."
>> > >
>> > > Exactly our point. Why can't we develop this in Apache Kafka
>> > > community? Instead of us open sourcing another GitHub project and
>> > creating
>> > > a divide in users and another version of API. Let's build this in
>Kafka
>> > > Community and use the governance model that is proven to provide
>vendor
>> > > free user driven consensus features. The argument that is adding
>>this
>> > REST
>> > > server to Kafka will affect the agility of the project doesn't mak
>> sense.
>> > >
>> > > It looks like your argument is either we develop all these small
>>tools
>> or
>> > > none at all. We as a community need to look at supporting critical
>> > > tools/API. Instead of dividing this project into individual external
>> > > communities. We should build this as part of Kafka which best serves
>> the
>> > > needs of users.
>> > > The Streams and Connect projects that were pushed into Kafka
>> > could
>> > > have been left in their own Github projects based on your arguments.
>> What
>> > > about the REST API is so different that such that it should stay out
>of
>> > the
>> > > Kafka project? From my experience, more users are asking for the
>>REST
>> > API.
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
>> > >
>> > > > I think the questions around governance make sense, I think we
>should
>> > > > really clarify that to make the process more clear so it can be
>fully
>> > > > inclusive.
>> > > >
>> > > > The idea that we should not collaborate on what is there now,
>though,
>> > > > because in the future we might disagree about direction does not
>> really
>> > > > make sense to me. If in the future we disagree, that is the beauty
>of
>> > > open
>> > > > source, you can always fork off a copy of the code and start an
>> > > independent
>> > > > project either in Apache or elsewhere. Pre-emptively re-creating
>> > another
>> > > > REST layer when it seems like we all quite agree on what needs to
>>be
>> > done
>> > > > and we have an existing code base for HTTP/kafka access that is
>> heavily
>> > > > used in production seems quite silly.
>> > > >
>> > > > Let me give some background on how I at least think about these
>> things.
>> > > > I've participated in open source projects out of LinkedIn via
>>github
>> as
>> > > > well as via the ASF. I don't think there is a "right" answer to
>>how
>> to
>> > do
>> > > > these but rather some tradeoffs. We thought about this quite a lot
>in
>> > the
>> > > > context of Kafka based on the experience with the Hadoop ecosystem
>as
>> > > well
>> > > > as from other open source communities.
>> > > >
>> > > > There is a rich ecosystem around Kafka. Many of the projects are
>> quite
>> > > > small--single clients or tools that do a single thing well--and
>> almost
>> > > none
>> > > > of them are top level apache projects. I don't think trying to
>>force
>> > each
>> > > > of these to turn into independent Apache projects is necessarily
>>the
>> > best
>> > > > thing for the ecosystem.
>> > > >
>> > > > My observation of how this can go wrong is really what I think has
>> > > happened
>> > > > to the Hadoop ecosystem. There you see quite a zoo of projects
>>which
>> > all
>> > > > drift apart and don't quite work together well. Coordinating even
>> > simple
>> > > > changes and standardization across these is exceptionally
>>difficult.
>> > The
>> > > > result is a bit of a mess for users--the pieces just don't really
>> come
>> > > > together very well. This makes sense for independent
>>infrastructure
>> > > systems
>> > > > (Kudu vs HDFS) but I'm not at all convinced that doing this for
>every
>> > > > little tool or helper library has lead to a desirable state. I
>>think
>> > the
>> > > > mode of operating where the Hadoop vendors spawn off a few new
>Apache
>> > > > projects for each new product initiative, especially since often
>that
>> > > > project is only valued by that vendor (and the other vendor has a
>> > > different
>> > > > competing Apache project) doesn't necessarily do a better job at
>> > > producing
>> > > > high quality communities or high quality software.
>> > > >
>> > > > These tools/connects/clients/proxies and other integration pieces
>can
>> > > take
>> > > > many forms, but my take of what makes one of these things good is
>> that
>> > it
>> > > > remains simple, does its one thing well, and cleaves as closely as
>> > > possible
>> > > > to the conventions for Kafka itself--i.e. doesn't invent new ways
>>of
>> > > > monitoring, configuring, etc. For the tools we've contributed
>>we've
>> > tried
>> > > > really hard to make them consistent with Kafka as well as with
>>each
>> > other
>> > > > in how testing, configuration, monitoring, etc works.
>> > > >
>> > > > I think what Apache does superbly well is create a community for
>> > > managing a
>> > > > large infrastructure layer like Kafka in a vendor independent way.
>> > What I
>> > > > think is less successful is attempting to form full and
>>independent
>> > > apache
>> > > > communities around very simple single purpose tools, especially if
>> you
>> > > hope
>> > > > for these to come together into a cohesive toolset across multiple
>> such
>> > > > tools. Much of what Apache does--create a collective decision
>>making
>> > > > process for resolving disagreement, help to trademark and protect
>the
>> > > marks
>> > > > of the project, etc just isn't that relevant for simple
>> single-purpose
>> > > > tools.
>> > > >
>> > > > So my take is there are a couple of options:
>> > > >
>> > > > 1. We can try to put all the small tools into the Apache
>>Project.
>> I
>> > > > think this is not the right approach as there is simply too
>>many
>> of
>> > > > them,
>> > > > many in different languages, serving different protocols,
>> > integrating
>> > > > with
>> > > > particular systems, and a single community can't effectively
>> > maintain
>> > > > them
>> > > > all. Doing this would significantly slow the progress of the
>Kafka
>> > > > project.
>> > > > As a protocol for messaging, I don't really see a case for
>> including
>> > > > REST
>> > > > but not MQTT or AMQP which are technically much better suited
>>to
>> > > > messaging
>> > > > and both are widely used for that.
>> > > > 2. We can treat ecosystem projects that aren't top level Apache
>> > > projects
>> > > > as invalid and try to recreate them all as Apache projects.
>> > Honestly,
>> > > > though, if you go to the Kafka ecosystem page virtually none of
>> the
>> > > most
>> > > > popular add-ons to Kafka are Apache projects. The most
>>successful
>> > > > things in
>> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number
>of
>> > > other
>> > > > clients, as well as the existing REST layer have succeeded at
>> > > developing
>> > > > communities that actively contribute and use these pieces and I
>> > don't
>> > > > know
>> > > > that that is a bad thing unless that community proves to be
>> > > uninclusive,
>> > > > unresponsive, or goes in a bad technical direction--and those
>>are
>> > > > failure
>> > > > modes that all open source efforts face.
>> > > > 3. We can do what I think makes the most sense and try to work
>> with
>> > > the
>> > > > projects that exist in the ecosystem and if the project doesn't
>> > have a
>> > > > responsive community or wants to go in a different direction
>>fork
>> or
>> > > > recreate that work.
>> > > >
>> > > > Of course any person can choose whatever of these options they
>>want.
>> > But
>> > > > from my point of view, option (3) has been the path of the
>>community
>> so
>> > > far
>> > > > and I think it has been quite successful.
>> > > >
>> > > > -Jay
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
>> kafka@harsha.io
>> > >
>> > > > wrote:
>> > > >
>> > > > > Neha,
>> > > > > "But I haven't seen any community emails or patches being
>submitted
>> > by
>> > > > you
>> > > > > guys, so I'm wondering why you are concerned about whether the
>> > > community
>> > > > is
>> > > > > open to accepting patches or not."
>> > > > >
>> > > > > I think you are talking about contributing patches to this
>> repository
>> > > > > right? https://github.com/confluentinc/kafka-rest . All I am
>> saying
>> > > > > the guidelines/governance model is not clear on the project and
>>I
>> > guess
>> > > > its
>> > > > > driven by opening a github issue request. Its the repository
>owned
>> > by
>> > > > > confluent and as much I appreciate that the features we
>>mentioned
>> are
>> > > in
>> > > > > the roadmap and welcoming us to contribute to the project. It
>> doesn't
>> > > > > gurantee what we want to add in the furture will be in your
>> roadmap.
>> > > > >
>> > > > > Hence the reason having it part of Kafka community will help a
>>lot
>> as
>> > > > other
>> > > > > users can participate in the discussions. We are happy to drive
>> any
>> > > > > feature additions through KIPs which gives everyone a chance to
>> > > > participate
>> > > > > and add to the discussions.
>> > > > >
>> > > > > Thanks,
>> > > > > Harsha
>> > > > >
>> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
>> > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > >
>> > > > > > +1
>> > > > > >
>> > > > > > I agree on the governance comments whole heartedly.
>> > > > > >
>> > > > > > Also i agree about the contribution comments made earlier in
>>the
>> > > > thread,
>> > > > > i
>> > > > > > personally am less likely to spend any of mine, or give
>>project
>> > time
>> > > > > within
>> > > > > > my internal projects to developers contributing to another
>> > commercial
>> > > > > > companies project even so technically open source, as then
>>there
>> is
>> > > > that
>> > > > > > commercial companies interest will always prevail and
>essentially
>> > can
>> > > > > > always have a final vote where disagreement. Im sure they
>>never
>> > > intend
>> > > > > to,
>> > > > > > but there is that true reality. This is why we have community
>> open
>> > > > source
>> > > > > > projects.
>> > > > > >
>> > > > > > I can find many different implementations now of a rest
>>endpoint
>> on
>> > > > > > GitHub, BitBucket etc. Each one has their benefits and
>> > disadvantages
>> > > in
>> > > > > > their implementation. By making / providing one this would
>>bring
>> > > > together
>> > > > > > these solutions, unifying those developers and also bringing
>>the
>> > best
>> > > > of
>> > > > > > all.
>> > > > > >
>> > > > > > I understand the concern on the community burden
>> adding/supporting
>> > > more
>> > > > > > surface area for every client. But something like REST is
>> universal
>> > > and
>> > > > > > worthy to be owned by the community.
>> > > > > >
>> > > > > > Mike
>> > > > > >
>> > > > > >
>> > > > > > ________________________________________
>> > > > > > From: Andrew Schofield <an...@outlook.com>
>> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
>> > > > > > To: dev@kafka.apache.org
>> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>> > > > > >
>> > > > > > There's a massive difference between the governance of Kafka
>>and
>> > the
>> > > > > > governance of the REST proxy.
>> > > > > >
>> > > > > > In Kafka, there is a broad community of people contributing
>their
>> > > > > opinions
>> > > > > > about future enhancements in the form of KIPs. There's some
>> really
>> > > deep
>> > > > > > consideration that goes into some of the trickier KIPs. There
>are
>> > > > people
>> > > > > > outside Confluent deeply knowledgeable in Kafka and building
>the
>> > > > > > reputations to become committers. I get the impression that
>>the
>> > > roadmap
>> > > > > of
>> > > > > > Kafka is not really community-owned (what's the big feature
>>for
>> > Kafka
>> > > > > 0.11,
>> > > > > > for example), but the conveyor belt of smaller features in the
>> form
>> > > of
>> > > > > KIPs
>> > > > > > works nicely. It's a good example of open-source working
>>well.
>> > > > > >
>> > > > > > The equivalent for the REST proxy is basically issues on
>>GitHub.
>> > The
>> > > > > > roadmap is less clear. There's not really a community properly
>> > > engaged
>> > > > in
>> > > > > > the way that there is with Kafka. So, you could say that it's
>> clear
>> > > > that
>> > > > > > fewer people are interested, but I think the whole governance
>> > thing
>> > > > is a
>> > > > > > big barrier to engagement. And it's looking like it's getting
>out
>> > of
>> > > > > date.
>> > > > > >
>> > > > > > In technical terms, I can think of two big improvements to the
>> REST
>> > > > > proxy.
>> > > > > > First, it needs to use the new consumer API so that it's
>possible
>> > to
>> > > > > secure
>> > > > > > connections between the REST proxy and Kafka. I don't care too
>> much
>> > > > which
>> > > > > > method calls it uses actually uses to consume messages, but I
>do
>> > > care
>> > > > > that
>> > > > > > I cannot secure connections because of network security rules.
>> > > Second,
>> > > > > > there's an affinity between a consumer and the instance of the
>> REST
>> > > > proxy
>> > > > > > to which it first connected. Kafka itself avoids this kind of
>> > > affinity
>> > > > > for
>> > > > > > good reason, and in the name of availability the REST proxy
>> should
>> > > too.
>> > > > > > These are natural KIPs.
>> > > > > >
>> > > > > > I think it would be good to have the code for the REST proxy
>> > > > contributed
>> > > > > > to Apache Kafka so that it would be able to be developed in
>>the
>> > same
>> > > > way.
>> > > > > >
>> > > > > > Andrew Schofield
>> > > > > >
>> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
>> > > > > > Sent: 07 October 2016 22:41:52
>> > > > > > To: dev@kafka.apache.org
>> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>> > > > > >
>> > > > > > ASF already gives us a clear framework and governance model
>>for
>> > > > community
>> > > > > > development. This is already understood by the people
>> contributing
>> > to
>> > > > > > Apache Kafka project, and they are the same people who want to
>> > > > contribute
>> > > > > > to the REST server capability as well. Everyone is in
>>agreement
>> on
>> > > the
>> > > > > > need for collaborating on this effort. So why not contribute
>>the
>> > code
>> > > > to
>> > > > > > Apache Kafka. This will help avoid duplication of effort and
>> forks
>> > > that
>> > > > > > may crop up, hugely benefitting the user community. This will
>> also
>> > > > avoid
>> > > > > > having to define a process similar to ASF on a GitHub project
>and
>> > > > instead
>> > > > > > there is a single community with clear understanding community
>> > > process
>> > > > as
>> > > > > > defined in ASF.
>> > > > > >
>> > > > > > As others have said, this is an important capability for
>>Apache
>> > > Kafka.
>> > > > It
>> > > > > > is worth maintaining this as a part of the project.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Suresh
>> > > > > >
>> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io>
>>wrote:
>> > > > > >
>> > > > > > >I personally think it would be quite wasteful to re-implement
>> the
>> > > REST
>> > > > > > >gateway just because that an actively-maintained piece of
>> > > > > Apache-licensed
>> > > > > > >software is not governed directly by the Apache Kafka
>> community...
>> > > > While
>> > > > > > >kafka-rest repo is owned by Confluent, the contributors
>> including
>> > > the
>> > > > > main
>> > > > > > >one are also part of the Apache Kafka community, so there
>>is a
>> > > chance
>> > > > > to
>> > > > > > >work this out.
>> > > > > > >
>> > > > > > >However, there are two valid concerns here that could be
>> > addressed,
>> > > > > around
>> > > > > > >community and accessibility:
>> > > > > > >>> What we are worried about is a project
>> > > > > > >>> that's not maintained in a community. So the process of
>> > accepting
>> > > > > > >>>patches
>> > > > > > >>> and priorities is not clear, and it's not developed in
>Apache
>> > > > > > >>>community.
>> > > > > > >>> Not only that, existing REST API project doesn't support
>>new
>> > > client
>> > > > > API
>> > > > > > >and
>> > > > > > >>> hence there is no security support either.
>> > > > > > >
>> > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
>> community
>> > > can
>> > > > > > >clarify that - what is their contribution policy, dev style,
>> > roadmap
>> > > > > etc.
>> > > > > > >If they want, they can make an effort to encourage
>participation
>> > > from
>> > > > > > >people outside Confluent (easily accept contributions, invite
>> > > external
>> > > > > > >commiters or have open dev process similar to Apache Kafka
>etc),
>> > as
>> > > > > there
>> > > > > > >is definitely seems to be some interest on the list. That
>>might
>> > > clear
>> > > > > the
>> > > > > > >community concern and help kafka-rest project (but that is a
>> > > > calculation
>> > > > > > >Confluent will have to make).
>> > > > > > >
>> > > > > > >The other, independent, concern is that REST is something
>>that
>> is
>> > > > > expected
>> > > > > > >to be available out of the box with Kafka. I personally don't
>> feel
>> > > > > > >strongly
>> > > > > > >about it (better use proper, efficient APIs from day one),
>> though
>> > it
>> > > > is
>> > > > > > >definitely way smaller than adding a stream processing engine
>to
>> > the
>> > > > > > >project :)
>> > > > > > >Again,the kafka-rest "community" could take steps to make it
>> even
>> > > > easier
>> > > > > > >to
>> > > > > > >install, configure and run kafka-rest for new users on
>>vanilla
>> > > Apache
>> > > > > > >Kafka
>> > > > > > >(outside the Confluent platform), if they wish that (or
>>welcome
>> > > > > > >contributions to that end), but that is up to them.
>> > > > > > >Finally, if after the above steps were taken there would
>>still
>a
>> > > > strong
>> > > > > > >desire to include a great rest gateway with Apache Kafka, I
>> assume
>> > > the
>> > > > > > >community could hypothetically fork the existing kafka-rest
>into
>> > an
>> > > > > Apache
>> > > > > > >Kafka subproject and maintain it "within Apache" instead of
>> > > > implementing
>> > > > > > >it
>> > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot
>> imagine
>> > it
>> > > > > > >happen
>> > > > > > >without Confluent blessing, and I think that is likely much
>less
>> > > > optimal
>> > > > > > >(pulling in other Confluent / Apache licensed dependencies)
>than
>> > > > having
>> > > > > a
>> > > > > > >separate external community around kafka-rest.
>> > > > > > >
>> > > > > > >
>> > > > > > >Just my two cents,
>> > > > > > >
>> > > > > > >
>> > > > > > >Ofir Manor
>> > > > > > >
>> > > > > > >Co-Founder & CTO | Equalum
>> > > > > > >
>> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
><+972%2054-780-1286>
>> > <+972%2054-780-1286> |
>> > > > Email:
>> > > > > > ofir.manor@equalum.io
>> > > > > > >
>> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
>> > > kafka@harsha.io
>> > > > >
>> > > > > > >wrote:
>> > > > > > >
>> > > > > > >> Neha & Jay,
>> > > > > > >> We did look at the open source
>>alternatives.
>> > Our
>> > > > > > >>concern
>> > > > > > >> is what's the patch acceptance and adding features/
>>bug-fixes
>> to
>> > > the
>> > > > > > >> existing project under a Github (although it's licensed
>>under
>> > > Apache
>> > > > > > >>2.0).
>> > > > > > >> It would be great if that project made available under
>>Apache
>> > and
>> > > > > > >>driven by
>> > > > > > >> the community. Adding to the above, not all Kafka users
>>are
>> > > > > interested
>> > > > > > >>in
>> > > > > > >> using the Java client API, they would like to have simple
>REST
>> > API
>> > > > > where
>> > > > > > >> they can code against using any language. I do believe this
>> adds
>> > > > value
>> > > > > > >>to
>> > > > > > >> Apache Kafka in itself.
>> > > > > > >>
>> > > > > > >> "For 1, I don't think there is value in giving in to the
>>NIH
>> > > > syndrome
>> > > > > > >>and
>> > > > > > >> reinventing the wheel. What I'm looking for is a detailed
>> > > comparison
>> > > > > of
>> > > > > > >>the
>> > > > > > >> gaps and why those can't be improved in the REST proxy that
>> > > already
>> > > > > > >>exists
>> > > > > > >> and is actively maintained."
>> > > > > > >>
>> > > > > > >> We are not looking at this as NIH. What we are worried
>>about
>> > is a
>> > > > > > >>project
>> > > > > > >> that's not maintained in a community. So the process of
>> > accepting
>> > > > > > >>patches
>> > > > > > >> and priorities is not clear, and it's not developed in
>>Apache
>> > > > > community.
>> > > > > > >> Not only that, existing REST API project doesn't support
>>new
>> > > client
>> > > > > API
>> > > > > > >>and
>> > > > > > >> hence there is no security support either.
>> > > > > > >> We don't know the timeline when that's made available. We
>> would
>> > > like
>> > > > > to
>> > > > > > >>add
>> > > > > > >> admin functionality into the REST API. So the Roadmap of
>>that
>> > > > project
>> > > > > is
>> > > > > > >> not driven by Apache.
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> "This doesn't materially have an impact on expanding the
>> > usability
>> > > > of
>> > > > > > >> Kafka. In my experience, REST proxy + Java clients only
>> cover
>> > > > ~50%
>> > > > > of
>> > > > > > >> all
>> > > > > > >> Kafka users, and maybe 10% of those are the ones who
>>will
>> use
>> > > the
>> > > > > > >>REST
>> > > > > > >> proxy. The remaining 50% are non-java client users (C,
>> > python,
>> > > > go,
>> > > > > > >>node
>> > > > > > >> etc)."
>> > > > > > >>
>> > > > > > >> REST API is most often asked feature in my interactions
>>with
>> > Kafka
>> > > > > > >>users.
>> > > > > > >> In an organization, There will be independent teams who
>>will
>> > write
>> > > > > their
>> > > > > > >> Kafka clients using different language libraries available
>> > today,
>> > > > and
>> > > > > > >> there is no way to standardize this. Instead of supporting
>> > several
>> > > > > > >> different client libraries users will be interested in
>>using
>a
>> > > REST
>> > > > > API
>> > > > > > >> server. The need for a REST API will only increase as more
>and
>> > > more
>> > > > > > >>users
>> > > > > > >> start using Kafka.
>> > > > > > >>
>> > > > > > >> "More surface area means more work to keep things
>>consistent.
>> > > > Failure
>> > > > > > >> to do that has, in fact, hurt the user experience."
>> > > > > > >> Having myriad Kafka client GitHub projects that support
>> > different
>> > > > > > >>languages
>> > > > > > >> hurts the user experience and pushes burden to maintain
>>these
>> > > > > libraries.
>> > > > > > >> REST API is a simple code base that uses existing java
>>client
>> > > > > libraries
>> > > > > > >>to
>> > > > > > >> make life easier for the users.
>> > > > > > >>
>> > > > > > >> Thanks,
>> > > > > > >> Harsha
>> > > > > > >>
>> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
>> > neha@confluent.io>
>> > > > > > wrote:
>> > > > > > >>
>> > > > > > >> > Manikumar,
>> > > > > > >> >
>> > > > > > >> > Thanks for sharing the proposal. I think there are 2
>>parts
>> to
>> > > this
>> > > > > > >> > discussion -
>> > > > > > >> >
>> > > > > > >> > 1. Should we rewrite a REST proxy given that there is a
>> > > > > > >>feature-complete,
>> > > > > > >> > open-source and actively maintained REST proxy in the
>> > community?
>> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more
>> agile
>> > > and
>> > > > > > >> maintain
>> > > > > > >> > the high-quality experience that Kafka users have today?
>> > > > > > >> >
>> > > > > > >> > For 1, I don't think there is value in giving in to the
>>NIH
>> > > > syndrome
>> > > > > > >>and
>> > > > > > >> > reinventing the wheel. What I'm looking for is a detailed
>> > > > comparison
>> > > > > > >>of
>> > > > > > >> the
>> > > > > > >> > gaps and why those can't be improved in the REST proxy
>>that
>> > > > already
>> > > > > > >> exists
>> > > > > > >> > and is actively maintained. For example, we depend on
>> zkClient
>> > > and
>> > > > > > >>have
>> > > > > > >> > found as well as fixed several bugs by working closely
>>with
>> > the
>> > > > > people
>> > > > > > >> who
>> > > > > > >> > maintain zkClient. This should be possible for REST proxy
>as
>> > > well,
>> > > > > > >>right?
>> > > > > > >> >
>> > > > > > >> > For 2, I'd like us to review our history of expanding the
>> > > surface
>> > > > > > >>area to
>> > > > > > >> > add more clients in the past. Here is a summary -
>> > > > > > >> >
>> > > > > > >> > - This doesn't materially have an impact on expanding
>the
>> > > > > > >>usability of
>> > > > > > >> > Kafka. In my experience, REST proxy + Java clients
>>only
>> > cover
>> > > > > ~50%
>> > > > > > >>of
>> > > > > > >> > all
>> > > > > > >> > Kafka users, and maybe 10% of those are the ones who
>will
>> > use
>> > > > the
>> > > > > > >>REST
>> > > > > > >> > proxy. The remaining 50% are non-java client users (C,
>> > > python,
>> > > > > go,
>> > > > > > >> node
>> > > > > > >> > etc).
>> > > > > > >> > - People are a lot more excited about promising to
>> > contribute
>> > > > > while
>> > > > > > >> > adding the surface area but not on an ongoing basis
>>down
>> > the
>> > > > > line.
>> > > > > > >> > - More surface area means more work to keep things
>> > > consistent.
>> > > > > > >>Failure
>> > > > > > >> > to do that has, in fact, hurt the user experience.
>> > > > > > >> > - More surface area hurts agility. We want to do a few
>> > things
>> > > > > > >>really
>> > > > > > >> > well as well as be agile to be able to build on our
>>core
>> > > > > > >>competency.
>> > > > > > >> >
>> > > > > > >> >
>> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
>> > > > manikumar.reddy@gmail.com
>> > > > > >
>> > > > > > >> > wrote:
>> > > > > > >> >
>> > > > > > >> > > Hi Jay,
>> > > > > > >> > >
>> > > > > > >> > > Thanks for your reply.
>> > > > > > >> > >
>> > > > > > >> > > I agree that we can not add all the clients/tools
>> available
>> > in
>> > > > > > >> ecosystem
>> > > > > > >> > > page to Kafka repo itself. But we feel REST Interface
>>is
>> > > > different
>> > > > > > >>from
>> > > > > > >> > > other clients/tools. Since any language that can work
>with
>> > > HTTP
>> > > > > can
>> > > > > > >> > > easily integrate with this interface, Having an
>"official"
>> > > REST
>> > > > > > >> > > interface helps user community. This also helps us to
>> > > integrate
>> > > > > well
>> > > > > > >> > > with external management and provisioning tools.
>>Apache
>> > Kafka
>> > > > > > >>release
>> > > > > > >> > > with Java clients + REST interface is sufficient for
>>most
>> of
>> > > the
>> > > > > > >>user
>> > > > > > >> > > deployments/requirements. This helps users to deal with
>> less
>> > > > > number
>> > > > > > >> > > of distributions/builds.
>> > > > > > >> > >
>> > > > > > >> > > Thanks,
>> > > > > > >> > > Manikumar
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
>> jay@confluent.io
>> > >
>> > > > > wrote:
>> > > > > > >> > >
>> > > > > > >> > > > Hey guys,
>> > > > > > >> > > >
>> > > > > > >> > > > There's already a REST interface maintained as a
>> separate
>> > > > > > >> project--it's
>> > > > > > >> > > > open source and apache licensed and actively
>>maintained
>> (
>> > > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is
>> > wrong
>> > > > with
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
>> > > > > > github.com
>> > > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
>> > cluster.
>> > > > It
>> > > > > > makes it easy to produce and consume messages, view the state
>>of
>> > the
>> > > > > > cluster, and ...
>> > > > > >
>> > > > > > >> that?
>> > > > > > >> > > You
>> > > > > > >> > > > mentioned that there was some compatibility concern,
>but
>> > > > > > >> compatibility
>> > > > > > >> > > has
>> > > > > > >> > > > to do with the consumer protocol guarantees not the
>repo
>> > the
>> > > > > code
>> > > > > > >>is
>> > > > > > >> > in,
>> > > > > > >> > > > right? Not sure that concern makes sense.
>> > > > > > >> > > >
>> > > > > > >> > > > We could argue for adding pretty much anything and
>> > > everything
>> > > > in
>> > > > > > >>the
>> > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that
>> would
>> > > > make
>> > > > > > >>the
>> > > > > > >> > > project
>> > > > > > >> > > > more agile.
>> > > > > > >> > > >
>> > > > > > >> > > > -Jay
>> > > > > > >> > > >
>> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
>> > > > > > >> manikumar.reddy@gmail.com
>> > > > > > >> > >
>> > > > > > >> > > > wrote:
>> > > > > > >> > > >
>> > > > > > >> > > > > Hi Kafka Devs,
>> > > > > > >> > > > >
>> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
>> > > > Repository.
>> > > > > > >> > > > >
>> > > > > > >> > > > > There are already open-source alternatives are
>> > available.
>> > > > But
>> > > > > > >>we
>> > > > > > >> > would
>> > > > > > >> > > > > like to add REST server that
>> > > > > > >> > > > > many users ask for under Apache Kafka repo. Many
>>data
>> > > Infra
>> > > > > > >>tools
>> > > > > > >> > comes
>> > > > > > >> > > > up
>> > > > > > >> > > > > with Rest Interface.
>> > > > > > >> > > > > It is useful to have inbuilt Rest API support for
>> > Produce,
>> > > > > > >>Consume
>> > > > > > >> > > > messages
>> > > > > > >> > > > > and admin interface for
>> > > > > > >> > > > > integrating with external management and
>>provisioning
>> > > > > tools.This
>> > > > > > >> will
>> > > > > > >> > > > also
>> > > > > > >> > > > > allow the maintenance of
>> > > > > > >> > > > > REST server and adding new features makes it easy
>> > because
>> > > > > apache
>> > > > > > >> > > > community.
>> > > > > > >> > > > >
>> > > > > > >> > > > > The KIP wiki is the following:
>> > > > > > >> > > > > https://cwiki.apache.org/
>> confluence/display/KAFKA/KIP-
>> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
>> > > > > > >> > > > >
>> > > > > > >> > > > > Your comments and feedback are welcome.
>> > > > > > >> > > > >
>> > > > > > >> > > > > Thanks,
>> > > > > > >> > > > > Manikumar
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> > --
>> > > > > > >> > Thanks,
>> > > > > > >> > Neha
>> > > > > > >> >
>> > > > > > >>
>> > > > > > The information contained in this email is strictly
>>confidential
>> > and
>> > > > for
>> > > > > > the use of the addressee only, unless otherwise indicated. If
>you
>> > are
>> > > > not
>> > > > > > the intended recipient, please do not read, copy, use or
>disclose
>> > to
>> > > > > others
>> > > > > > this message or any attachment. Please also notify the sender
>>by
>> > > > replying
>> > > > > > to this email or by telephone (+44(020 7896 0011) and then
>delete
>> > the
>> > > > > email
>> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
>> relate
>> > > to
>> > > > > the
>> > > > > > official business of this company shall be understood as
>>neither
>> > > given
>> > > > > nor
>> > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> > company
>> > > > > > registered in England and Wales, company number 04008957) and
>>IG
>> > > Index
>> > > > > > Limited (a company registered in England and Wales, company
>> number
>> > > > > > 01190902). Registered address at Cannon Bridge House, 25
>>Dowgate
>> > > Hill,
>> > > > > > London EC4R 2YA. Both IG Markets Limited (register number
>195355)
>> > and
>> > > > IG
>> > > > > > Index Limited (register number 114059) are authorised and
>> regulated
>> > > by
>> > > > > the
>> > > > > > Financial Conduct Authority.
>> > > > > >
>> > > > >
>> > > >
>> > > The information contained in this email is strictly confidential and
>> for
>> > > the use of the addressee only, unless otherwise indicated. If you
>>are
>> not
>> > > the intended recipient, please do not read, copy, use or disclose to
>> > others
>> > > this message or any attachment. Please also notify the sender by
>> replying
>> > > to this email or by telephone (+44(020 7896 0011) and then delete
>>the
>> > email
>> > > and any copies of it. Opinions, conclusion (etc) that do not relate
>>to
>> > the
>> > > official business of this company shall be understood as neither
>>given
>> > nor
>> > > endorsed by it. IG is a trading name of IG Markets Limited (a
>>company
>> > > registered in England and Wales, company number 04008957) and IG
>>Index
>> > > Limited (a company registered in England and Wales, company number
>> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>>Hill,
>> > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
>>and
>> IG
>> > > Index Limited (register number 114059) are authorised and regulated
>>by
>> > the
>> > > Financial Conduct Authority.
>> > >
>> >
>>
>>
>>
>> --
>> Nacho (Ignacio) Solis
>> Kafka
>> nsolis@linkedin.com
>>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Harsha Chintalapani <ka...@harsha.io>.
Sriram,
"Can the streaming platform exist without stream processing? - No.
Processing stream data again is a core part of streaming platform."
Yes, it can. There are no.of Stream processing frameworks out there, and
they all have integration into Kafka.
It doesn't need to be developed within Kafka.
"Can the platform exist without the rest proxy? - Yes. The proxy does not
complete the platform vision in anyway. It is just a good to have tool that
might be required by quite a few users and there is an active project that
works on this - https://github.com/confluentinc/kafka-rest"
The rest proxy is as important as any API. The vision that shown here
http://kafka.apache.org/intro
require users to write the producers and consumers to get their data into
and out of Kafka, without which having Streams or Connect won't help
anyone.
The rest proxy makes easier for users get their data into Kafka.
Adding the rest proxy to the project doesn't invalidate the current vision,
it only strengthens it.
Thanks,
Harsha
On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <ra...@confluent.io> wrote:
FWIW, Apache Kafka has evolved a lot from where it started. It did start as
a messaging system. Over time we realized that that the vision for Kafka is
to build a streaming platform and not just a messaging system. You can take
a look at the site for more description about what comprises the streaming
platform http://kafka.apache.org/ and http://kafka.apache.org/intro.
Can the streaming platform exist without Connect? - No. Data integration is
fundamental to building an end to end platform
Can the streaming platform exist without stream processing? - No.
Processing stream data again is a core part of streaming platform.
Can the streaming platform exist without clients? - We at least need one
client library to complete the platform. Our Java clients help us to
complete the platform story. The rest of the clients are built and
maintained outside the project.
Can the platform exist without the rest proxy? - Yes. The proxy does not
complete the platform vision in anyway. It is just a good to have tool that
might be required by quite a few users and there is an active project that
works on this - https://github.com/confluentinc/kafka-rest
On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis <ns...@linkedin.com.invalid>
wrote:
> Are you saying Kafka REST is subjective but Kafka Streams and Kafka
Connect
> are not subjective?
>
> > "there are likely places that can live without a rest proxy"
>
> There are also places that can live without Kafka Streams and Kafka
> Connect.
>
> Nacho
>
> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > At the high level, I think ideally it makes sense to add a component to
> > Apache Kafka if (1) it's widely needed and (2) it needs tight
integration
> > with Kafka core. For Kafka Stream, we do expect stream processing will
be
> > used widely in the future. Implementation wise, Kafka Stream only
> supports
> > getting data from Kafka and leverages quite a few of the core
> > functionalities in Kafka core. For example, it uses customized rebalance
> > callback in the consumer and uses the compacted topic heavily. So,
having
> > Kafka Stream in the same repo makes it easier for testing when those
core
> > functionalities evolve over time. Kafka Connect is in the same
situation.
> >
> > For rest proxy, whether it's widely used or not is going to be a bit
> > subjective. However, there are likely places that can live without a
rest
> > proxy. The rest proxy is just a proxy for the regular clients and
doesn't
> > need to be tightly integrated with Kafka core. So, the case for
including
> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream and
> > Kafka Connect.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > So from my reading essentially the first question needs to
answered/and
> > > voted on is:
> > >
> > > Is Apache Kafka Community only about the Core or does the apache
> > community
> > > also support some subprojects (and just we need some better way to
> manage
> > > this)
> > >
> > > If vote for Core only wins, then the following should be removed:
> > > Kafka Connect
> > > Kafka Stream
> > >
> > > If vote for Core only loses (aka we will support subprojects) then:
> > > We should look to add Kafka Rest
> > >
> > > And we should look to see how we can manage better govern and manage
> > > submodules.
> > >
> > > A good example which id propose here is how some other communities in
> > > Apache do this.
> > >
> > > Each Module has a Module Management Committee(MMC), this is like
almost
> > > the PMC but at a per module basis.
> > >
> > > This MMC should essentially hold the binding votes for that module.
> > > The MMC should be made up of a single representative from each
> > > organisation (so no single organisation can fully veto the community
> it
> > > has to a genuine consenus)
> > > The MMC requires at least 3 members (so there cant be a tied vote on
2)
> > > For a new Module to be added a MMC committee should be sought
> > > A new Module is only capable of being added if the above requirements
> can
> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
that
> > > only actively support modules would be added
> > >
> > > The PMC reviews each module every 6months or Year. If MMC is inactive,
> a
> > > vote/call to find replacements if raised, if none are forthcoming
> > dropping
> > > the MMC to less than 3 then the module moves to "the attic" (very much
> > like
> > > apache attic but a little more aggressively)
> > >
> > > This way the PMC does not need to micro manage every module
> > > We only add modules where some amount of active support and
maintenance
> > > and use is provided by the community
> > > We have an automatic way to retire old or inactive projects.
> > >
> > > Thoughts?
> > > Mike
> > >
> > >
> > > ________________________________________
> > > From: Harsha Ch <ha...@gmail.com>
> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > Jay,
> > > REST API is something every user is in need of. If the argument
> is
> > to
> > > clone and write your API, this will do a disservice to the users as
> they
> > > now have to choose one vs. others instead of keeping one API that is
> > > supported in Kafka community.
> > >
> > > "Pre-emptively re-creating another
> > > REST layer when it seems like we all quite agree on what needs to be
> done
> > > and we have an existing code base for HTTP/Kafka access that is
heavily
> > > used in production seems quite silly."
> > >
> > > Exactly our point. Why can't we develop this in Apache Kafka
> > > community? Instead of us open sourcing another GitHub project and
> > creating
> > > a divide in users and another version of API. Let's build this in
Kafka
> > > Community and use the governance model that is proven to provide
vendor
> > > free user driven consensus features. The argument that is adding this
> > REST
> > > server to Kafka will affect the agility of the project doesn't mak
> sense.
> > >
> > > It looks like your argument is either we develop all these small tools
> or
> > > none at all. We as a community need to look at supporting critical
> > > tools/API. Instead of dividing this project into individual external
> > > communities. We should build this as part of Kafka which best serves
> the
> > > needs of users.
> > > The Streams and Connect projects that were pushed into Kafka
> > could
> > > have been left in their own Github projects based on your arguments.
> What
> > > about the REST API is so different that such that it should stay out
of
> > the
> > > Kafka project? From my experience, more users are asking for the REST
> > API.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > I think the questions around governance make sense, I think we
should
> > > > really clarify that to make the process more clear so it can be
fully
> > > > inclusive.
> > > >
> > > > The idea that we should not collaborate on what is there now,
though,
> > > > because in the future we might disagree about direction does not
> really
> > > > make sense to me. If in the future we disagree, that is the beauty
of
> > > open
> > > > source, you can always fork off a copy of the code and start an
> > > independent
> > > > project either in Apache or elsewhere. Pre-emptively re-creating
> > another
> > > > REST layer when it seems like we all quite agree on what needs to be
> > done
> > > > and we have an existing code base for HTTP/kafka access that is
> heavily
> > > > used in production seems quite silly.
> > > >
> > > > Let me give some background on how I at least think about these
> things.
> > > > I've participated in open source projects out of LinkedIn via github
> as
> > > > well as via the ASF. I don't think there is a "right" answer to how
> to
> > do
> > > > these but rather some tradeoffs. We thought about this quite a lot
in
> > the
> > > > context of Kafka based on the experience with the Hadoop ecosystem
as
> > > well
> > > > as from other open source communities.
> > > >
> > > > There is a rich ecosystem around Kafka. Many of the projects are
> quite
> > > > small--single clients or tools that do a single thing well--and
> almost
> > > none
> > > > of them are top level apache projects. I don't think trying to force
> > each
> > > > of these to turn into independent Apache projects is necessarily the
> > best
> > > > thing for the ecosystem.
> > > >
> > > > My observation of how this can go wrong is really what I think has
> > > happened
> > > > to the Hadoop ecosystem. There you see quite a zoo of projects which
> > all
> > > > drift apart and don't quite work together well. Coordinating even
> > simple
> > > > changes and standardization across these is exceptionally difficult.
> > The
> > > > result is a bit of a mess for users--the pieces just don't really
> come
> > > > together very well. This makes sense for independent infrastructure
> > > systems
> > > > (Kudu vs HDFS) but I'm not at all convinced that doing this for
every
> > > > little tool or helper library has lead to a desirable state. I think
> > the
> > > > mode of operating where the Hadoop vendors spawn off a few new
Apache
> > > > projects for each new product initiative, especially since often
that
> > > > project is only valued by that vendor (and the other vendor has a
> > > different
> > > > competing Apache project) doesn't necessarily do a better job at
> > > producing
> > > > high quality communities or high quality software.
> > > >
> > > > These tools/connects/clients/proxies and other integration pieces
can
> > > take
> > > > many forms, but my take of what makes one of these things good is
> that
> > it
> > > > remains simple, does its one thing well, and cleaves as closely as
> > > possible
> > > > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > > > monitoring, configuring, etc. For the tools we've contributed we've
> > tried
> > > > really hard to make them consistent with Kafka as well as with each
> > other
> > > > in how testing, configuration, monitoring, etc works.
> > > >
> > > > I think what Apache does superbly well is create a community for
> > > managing a
> > > > large infrastructure layer like Kafka in a vendor independent way.
> > What I
> > > > think is less successful is attempting to form full and independent
> > > apache
> > > > communities around very simple single purpose tools, especially if
> you
> > > hope
> > > > for these to come together into a cohesive toolset across multiple
> such
> > > > tools. Much of what Apache does--create a collective decision making
> > > > process for resolving disagreement, help to trademark and protect
the
> > > marks
> > > > of the project, etc just isn't that relevant for simple
> single-purpose
> > > > tools.
> > > >
> > > > So my take is there are a couple of options:
> > > >
> > > > 1. We can try to put all the small tools into the Apache Project.
> I
> > > > think this is not the right approach as there is simply too many
> of
> > > > them,
> > > > many in different languages, serving different protocols,
> > integrating
> > > > with
> > > > particular systems, and a single community can't effectively
> > maintain
> > > > them
> > > > all. Doing this would significantly slow the progress of the
Kafka
> > > > project.
> > > > As a protocol for messaging, I don't really see a case for
> including
> > > > REST
> > > > but not MQTT or AMQP which are technically much better suited to
> > > > messaging
> > > > and both are widely used for that.
> > > > 2. We can treat ecosystem projects that aren't top level Apache
> > > projects
> > > > as invalid and try to recreate them all as Apache projects.
> > Honestly,
> > > > though, if you go to the Kafka ecosystem page virtually none of
> the
> > > most
> > > > popular add-ons to Kafka are Apache projects. The most successful
> > > > things in
> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number
of
> > > other
> > > > clients, as well as the existing REST layer have succeeded at
> > > developing
> > > > communities that actively contribute and use these pieces and I
> > don't
> > > > know
> > > > that that is a bad thing unless that community proves to be
> > > uninclusive,
> > > > unresponsive, or goes in a bad technical direction--and those are
> > > > failure
> > > > modes that all open source efforts face.
> > > > 3. We can do what I think makes the most sense and try to work
> with
> > > the
> > > > projects that exist in the ecosystem and if the project doesn't
> > have a
> > > > responsive community or wants to go in a different direction fork
> or
> > > > recreate that work.
> > > >
> > > > Of course any person can choose whatever of these options they want.
> > But
> > > > from my point of view, option (3) has been the path of the community
> so
> > > far
> > > > and I think it has been quite successful.
> > > >
> > > > -Jay
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > > > wrote:
> > > >
> > > > > Neha,
> > > > > "But I haven't seen any community emails or patches being
submitted
> > by
> > > > you
> > > > > guys, so I'm wondering why you are concerned about whether the
> > > community
> > > > is
> > > > > open to accepting patches or not."
> > > > >
> > > > > I think you are talking about contributing patches to this
> repository
> > > > > right? https://github.com/confluentinc/kafka-rest . All I am
> saying
> > > > > the guidelines/governance model is not clear on the project and I
> > guess
> > > > its
> > > > > driven by opening a github issue request. Its the repository
owned
> > by
> > > > > confluent and as much I appreciate that the features we mentioned
> are
> > > in
> > > > > the roadmap and welcoming us to contribute to the project. It
> doesn't
> > > > > gurantee what we want to add in the furture will be in your
> roadmap.
> > > > >
> > > > > Hence the reason having it part of Kafka community will help a lot
> as
> > > > other
> > > > > users can participate in the discussions. We are happy to drive
> any
> > > > > feature additions through KIPs which gives everyone a chance to
> > > > participate
> > > > > and add to the discussions.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > I agree on the governance comments whole heartedly.
> > > > > >
> > > > > > Also i agree about the contribution comments made earlier in the
> > > > thread,
> > > > > i
> > > > > > personally am less likely to spend any of mine, or give project
> > time
> > > > > within
> > > > > > my internal projects to developers contributing to another
> > commercial
> > > > > > companies project even so technically open source, as then there
> is
> > > > that
> > > > > > commercial companies interest will always prevail and
essentially
> > can
> > > > > > always have a final vote where disagreement. Im sure they never
> > > intend
> > > > > to,
> > > > > > but there is that true reality. This is why we have community
> open
> > > > source
> > > > > > projects.
> > > > > >
> > > > > > I can find many different implementations now of a rest endpoint
> on
> > > > > > GitHub, BitBucket etc. Each one has their benefits and
> > disadvantages
> > > in
> > > > > > their implementation. By making / providing one this would bring
> > > > together
> > > > > > these solutions, unifying those developers and also bringing the
> > best
> > > > of
> > > > > > all.
> > > > > >
> > > > > > I understand the concern on the community burden
> adding/supporting
> > > more
> > > > > > surface area for every client. But something like REST is
> universal
> > > and
> > > > > > worthy to be owned by the community.
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Andrew Schofield <an...@outlook.com>
> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > >
> > > > > > There's a massive difference between the governance of Kafka and
> > the
> > > > > > governance of the REST proxy.
> > > > > >
> > > > > > In Kafka, there is a broad community of people contributing
their
> > > > > opinions
> > > > > > about future enhancements in the form of KIPs. There's some
> really
> > > deep
> > > > > > consideration that goes into some of the trickier KIPs. There
are
> > > > people
> > > > > > outside Confluent deeply knowledgeable in Kafka and building
the
> > > > > > reputations to become committers. I get the impression that the
> > > roadmap
> > > > > of
> > > > > > Kafka is not really community-owned (what's the big feature for
> > Kafka
> > > > > 0.11,
> > > > > > for example), but the conveyor belt of smaller features in the
> form
> > > of
> > > > > KIPs
> > > > > > works nicely. It's a good example of open-source working well.
> > > > > >
> > > > > > The equivalent for the REST proxy is basically issues on GitHub.
> > The
> > > > > > roadmap is less clear. There's not really a community properly
> > > engaged
> > > > in
> > > > > > the way that there is with Kafka. So, you could say that it's
> clear
> > > > that
> > > > > > fewer people are interested, but I think the whole governance
> > thing
> > > > is a
> > > > > > big barrier to engagement. And it's looking like it's getting
out
> > of
> > > > > date.
> > > > > >
> > > > > > In technical terms, I can think of two big improvements to the
> REST
> > > > > proxy.
> > > > > > First, it needs to use the new consumer API so that it's
possible
> > to
> > > > > secure
> > > > > > connections between the REST proxy and Kafka. I don't care too
> much
> > > > which
> > > > > > method calls it uses actually uses to consume messages, but I
do
> > > care
> > > > > that
> > > > > > I cannot secure connections because of network security rules.
> > > Second,
> > > > > > there's an affinity between a consumer and the instance of the
> REST
> > > > proxy
> > > > > > to which it first connected. Kafka itself avoids this kind of
> > > affinity
> > > > > for
> > > > > > good reason, and in the name of availability the REST proxy
> should
> > > too.
> > > > > > These are natural KIPs.
> > > > > >
> > > > > > I think it would be good to have the code for the REST proxy
> > > > contributed
> > > > > > to Apache Kafka so that it would be able to be developed in the
> > same
> > > > way.
> > > > > >
> > > > > > Andrew Schofield
> > > > > >
> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > > > Sent: 07 October 2016 22:41:52
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > >
> > > > > > ASF already gives us a clear framework and governance model for
> > > > community
> > > > > > development. This is already understood by the people
> contributing
> > to
> > > > > > Apache Kafka project, and they are the same people who want to
> > > > contribute
> > > > > > to the REST server capability as well. Everyone is in agreement
> on
> > > the
> > > > > > need for collaborating on this effort. So why not contribute the
> > code
> > > > to
> > > > > > Apache Kafka. This will help avoid duplication of effort and
> forks
> > > that
> > > > > > may crop up, hugely benefitting the user community. This will
> also
> > > > avoid
> > > > > > having to define a process similar to ASF on a GitHub project
and
> > > > instead
> > > > > > there is a single community with clear understanding community
> > > process
> > > > as
> > > > > > defined in ASF.
> > > > > >
> > > > > > As others have said, this is an important capability for Apache
> > > Kafka.
> > > > It
> > > > > > is worth maintaining this as a part of the project.
> > > > > >
> > > > > > Regards,
> > > > > > Suresh
> > > > > >
> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > > > >
> > > > > > >I personally think it would be quite wasteful to re-implement
> the
> > > REST
> > > > > > >gateway just because that an actively-maintained piece of
> > > > > Apache-licensed
> > > > > > >software is not governed directly by the Apache Kafka
> community...
> > > > While
> > > > > > >kafka-rest repo is owned by Confluent, the contributors
> including
> > > the
> > > > > main
> > > > > > >one are also part of the Apache Kafka community, so there is a
> > > chance
> > > > > to
> > > > > > >work this out.
> > > > > > >
> > > > > > >However, there are two valid concerns here that could be
> > addressed,
> > > > > around
> > > > > > >community and accessibility:
> > > > > > >>> What we are worried about is a project
> > > > > > >>> that's not maintained in a community. So the process of
> > accepting
> > > > > > >>>patches
> > > > > > >>> and priorities is not clear, and it's not developed in
Apache
> > > > > > >>>community.
> > > > > > >>> Not only that, existing REST API project doesn't support new
> > > client
> > > > > API
> > > > > > >and
> > > > > > >>> hence there is no security support either.
> > > > > > >
> > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
> community
> > > can
> > > > > > >clarify that - what is their contribution policy, dev style,
> > roadmap
> > > > > etc.
> > > > > > >If they want, they can make an effort to encourage
participation
> > > from
> > > > > > >people outside Confluent (easily accept contributions, invite
> > > external
> > > > > > >commiters or have open dev process similar to Apache Kafka
etc),
> > as
> > > > > there
> > > > > > >is definitely seems to be some interest on the list. That might
> > > clear
> > > > > the
> > > > > > >community concern and help kafka-rest project (but that is a
> > > > calculation
> > > > > > >Confluent will have to make).
> > > > > > >
> > > > > > >The other, independent, concern is that REST is something that
> is
> > > > > expected
> > > > > > >to be available out of the box with Kafka. I personally don't
> feel
> > > > > > >strongly
> > > > > > >about it (better use proper, efficient APIs from day one),
> though
> > it
> > > > is
> > > > > > >definitely way smaller than adding a stream processing engine
to
> > the
> > > > > > >project :)
> > > > > > >Again,the kafka-rest "community" could take steps to make it
> even
> > > > easier
> > > > > > >to
> > > > > > >install, configure and run kafka-rest for new users on vanilla
> > > Apache
> > > > > > >Kafka
> > > > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > > > >contributions to that end), but that is up to them.
> > > > > > >Finally, if after the above steps were taken there would still
a
> > > > strong
> > > > > > >desire to include a great rest gateway with Apache Kafka, I
> assume
> > > the
> > > > > > >community could hypothetically fork the existing kafka-rest
into
> > an
> > > > > Apache
> > > > > > >Kafka subproject and maintain it "within Apache" instead of
> > > > implementing
> > > > > > >it
> > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot
> imagine
> > it
> > > > > > >happen
> > > > > > >without Confluent blessing, and I think that is likely much
less
> > > > optimal
> > > > > > >(pulling in other Confluent / Apache licensed dependencies)
than
> > > > having
> > > > > a
> > > > > > >separate external community around kafka-rest.
> > > > > > >
> > > > > > >
> > > > > > >Just my two cents,
> > > > > > >
> > > > > > >
> > > > > > >Ofir Manor
> > > > > > >
> > > > > > >Co-Founder & CTO | Equalum
> > > > > > >
> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
<+972%2054-780-1286>
> > <+972%2054-780-1286> |
> > > > Email:
> > > > > > ofir.manor@equalum.io
> > > > > > >
> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > > kafka@harsha.io
> > > > >
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Neha & Jay,
> > > > > > >> We did look at the open source alternatives.
> > Our
> > > > > > >>concern
> > > > > > >> is what's the patch acceptance and adding features/ bug-fixes
> to
> > > the
> > > > > > >> existing project under a Github (although it's licensed under
> > > Apache
> > > > > > >>2.0).
> > > > > > >> It would be great if that project made available under Apache
> > and
> > > > > > >>driven by
> > > > > > >> the community. Adding to the above, not all Kafka users are
> > > > > interested
> > > > > > >>in
> > > > > > >> using the Java client API, they would like to have simple
REST
> > API
> > > > > where
> > > > > > >> they can code against using any language. I do believe this
> adds
> > > > value
> > > > > > >>to
> > > > > > >> Apache Kafka in itself.
> > > > > > >>
> > > > > > >> "For 1, I don't think there is value in giving in to the NIH
> > > > syndrome
> > > > > > >>and
> > > > > > >> reinventing the wheel. What I'm looking for is a detailed
> > > comparison
> > > > > of
> > > > > > >>the
> > > > > > >> gaps and why those can't be improved in the REST proxy that
> > > already
> > > > > > >>exists
> > > > > > >> and is actively maintained."
> > > > > > >>
> > > > > > >> We are not looking at this as NIH. What we are worried about
> > is a
> > > > > > >>project
> > > > > > >> that's not maintained in a community. So the process of
> > accepting
> > > > > > >>patches
> > > > > > >> and priorities is not clear, and it's not developed in Apache
> > > > > community.
> > > > > > >> Not only that, existing REST API project doesn't support new
> > > client
> > > > > API
> > > > > > >>and
> > > > > > >> hence there is no security support either.
> > > > > > >> We don't know the timeline when that's made available. We
> would
> > > like
> > > > > to
> > > > > > >>add
> > > > > > >> admin functionality into the REST API. So the Roadmap of that
> > > > project
> > > > > is
> > > > > > >> not driven by Apache.
> > > > > > >>
> > > > > > >>
> > > > > > >> "This doesn't materially have an impact on expanding the
> > usability
> > > > of
> > > > > > >> Kafka. In my experience, REST proxy + Java clients only
> cover
> > > > ~50%
> > > > > of
> > > > > > >> all
> > > > > > >> Kafka users, and maybe 10% of those are the ones who will
> use
> > > the
> > > > > > >>REST
> > > > > > >> proxy. The remaining 50% are non-java client users (C,
> > python,
> > > > go,
> > > > > > >>node
> > > > > > >> etc)."
> > > > > > >>
> > > > > > >> REST API is most often asked feature in my interactions with
> > Kafka
> > > > > > >>users.
> > > > > > >> In an organization, There will be independent teams who will
> > write
> > > > > their
> > > > > > >> Kafka clients using different language libraries available
> > today,
> > > > and
> > > > > > >> there is no way to standardize this. Instead of supporting
> > several
> > > > > > >> different client libraries users will be interested in using
a
> > > REST
> > > > > API
> > > > > > >> server. The need for a REST API will only increase as more
and
> > > more
> > > > > > >>users
> > > > > > >> start using Kafka.
> > > > > > >>
> > > > > > >> "More surface area means more work to keep things consistent.
> > > > Failure
> > > > > > >> to do that has, in fact, hurt the user experience."
> > > > > > >> Having myriad Kafka client GitHub projects that support
> > different
> > > > > > >>languages
> > > > > > >> hurts the user experience and pushes burden to maintain these
> > > > > libraries.
> > > > > > >> REST API is a simple code base that uses existing java client
> > > > > libraries
> > > > > > >>to
> > > > > > >> make life easier for the users.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Harsha
> > > > > > >>
> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > neha@confluent.io>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Manikumar,
> > > > > > >> >
> > > > > > >> > Thanks for sharing the proposal. I think there are 2 parts
> to
> > > this
> > > > > > >> > discussion -
> > > > > > >> >
> > > > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > > > >>feature-complete,
> > > > > > >> > open-source and actively maintained REST proxy in the
> > community?
> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more
> agile
> > > and
> > > > > > >> maintain
> > > > > > >> > the high-quality experience that Kafka users have today?
> > > > > > >> >
> > > > > > >> > For 1, I don't think there is value in giving in to the NIH
> > > > syndrome
> > > > > > >>and
> > > > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > > > comparison
> > > > > > >>of
> > > > > > >> the
> > > > > > >> > gaps and why those can't be improved in the REST proxy that
> > > > already
> > > > > > >> exists
> > > > > > >> > and is actively maintained. For example, we depend on
> zkClient
> > > and
> > > > > > >>have
> > > > > > >> > found as well as fixed several bugs by working closely with
> > the
> > > > > people
> > > > > > >> who
> > > > > > >> > maintain zkClient. This should be possible for REST proxy
as
> > > well,
> > > > > > >>right?
> > > > > > >> >
> > > > > > >> > For 2, I'd like us to review our history of expanding the
> > > surface
> > > > > > >>area to
> > > > > > >> > add more clients in the past. Here is a summary -
> > > > > > >> >
> > > > > > >> > - This doesn't materially have an impact on expanding
the
> > > > > > >>usability of
> > > > > > >> > Kafka. In my experience, REST proxy + Java clients only
> > cover
> > > > > ~50%
> > > > > > >>of
> > > > > > >> > all
> > > > > > >> > Kafka users, and maybe 10% of those are the ones who
will
> > use
> > > > the
> > > > > > >>REST
> > > > > > >> > proxy. The remaining 50% are non-java client users (C,
> > > python,
> > > > > go,
> > > > > > >> node
> > > > > > >> > etc).
> > > > > > >> > - People are a lot more excited about promising to
> > contribute
> > > > > while
> > > > > > >> > adding the surface area but not on an ongoing basis down
> > the
> > > > > line.
> > > > > > >> > - More surface area means more work to keep things
> > > consistent.
> > > > > > >>Failure
> > > > > > >> > to do that has, in fact, hurt the user experience.
> > > > > > >> > - More surface area hurts agility. We want to do a few
> > things
> > > > > > >>really
> > > > > > >> > well as well as be agile to be able to build on our core
> > > > > > >>competency.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > > manikumar.reddy@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Jay,
> > > > > > >> > >
> > > > > > >> > > Thanks for your reply.
> > > > > > >> > >
> > > > > > >> > > I agree that we can not add all the clients/tools
> available
> > in
> > > > > > >> ecosystem
> > > > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > > > different
> > > > > > >>from
> > > > > > >> > > other clients/tools. Since any language that can work
with
> > > HTTP
> > > > > can
> > > > > > >> > > easily integrate with this interface, Having an
"official"
> > > REST
> > > > > > >> > > interface helps user community. This also helps us to
> > > integrate
> > > > > well
> > > > > > >> > > with external management and provisioning tools. Apache
> > Kafka
> > > > > > >>release
> > > > > > >> > > with Java clients + REST interface is sufficient for most
> of
> > > the
> > > > > > >>user
> > > > > > >> > > deployments/requirements. This helps users to deal with
> less
> > > > > number
> > > > > > >> > > of distributions/builds.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Manikumar
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> jay@confluent.io
> > >
> > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hey guys,
> > > > > > >> > > >
> > > > > > >> > > > There's already a REST interface maintained as a
> separate
> > > > > > >> project--it's
> > > > > > >> > > > open source and apache licensed and actively maintained
> (
> > > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is
> > wrong
> > > > with
> > > > > >
> > > > > >
> > > > > >
> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > > > github.com
> > > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
> > cluster.
> > > > It
> > > > > > makes it easy to produce and consume messages, view the state of
> > the
> > > > > > cluster, and ...
> > > > > >
> > > > > > >> that?
> > > > > > >> > > You
> > > > > > >> > > > mentioned that there was some compatibility concern,
but
> > > > > > >> compatibility
> > > > > > >> > > has
> > > > > > >> > > > to do with the consumer protocol guarantees not the
repo
> > the
> > > > > code
> > > > > > >>is
> > > > > > >> > in,
> > > > > > >> > > > right? Not sure that concern makes sense.
> > > > > > >> > > >
> > > > > > >> > > > We could argue for adding pretty much anything and
> > > everything
> > > > in
> > > > > > >>the
> > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that
> would
> > > > make
> > > > > > >>the
> > > > > > >> > > project
> > > > > > >> > > > more agile.
> > > > > > >> > > >
> > > > > > >> > > > -Jay
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > > > >> manikumar.reddy@gmail.com
> > > > > > >> > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi Kafka Devs,
> > > > > > >> > > > >
> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > > > Repository.
> > > > > > >> > > > >
> > > > > > >> > > > > There are already open-source alternatives are
> > available.
> > > > But
> > > > > > >>we
> > > > > > >> > would
> > > > > > >> > > > > like to add REST server that
> > > > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> > > Infra
> > > > > > >>tools
> > > > > > >> > comes
> > > > > > >> > > > up
> > > > > > >> > > > > with Rest Interface.
> > > > > > >> > > > > It is useful to have inbuilt Rest API support for
> > Produce,
> > > > > > >>Consume
> > > > > > >> > > > messages
> > > > > > >> > > > > and admin interface for
> > > > > > >> > > > > integrating with external management and provisioning
> > > > > tools.This
> > > > > > >> will
> > > > > > >> > > > also
> > > > > > >> > > > > allow the maintenance of
> > > > > > >> > > > > REST server and adding new features makes it easy
> > because
> > > > > apache
> > > > > > >> > > > community.
> > > > > > >> > > > >
> > > > > > >> > > > > The KIP wiki is the following:
> > > > > > >> > > > > https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-
> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > > > >> > > > >
> > > > > > >> > > > > Your comments and feedback are welcome.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Manikumar
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > --
> > > > > > >> > Thanks,
> > > > > > >> > Neha
> > > > > > >> >
> > > > > > >>
> > > > > > The information contained in this email is strictly confidential
> > and
> > > > for
> > > > > > the use of the addressee only, unless otherwise indicated. If
you
> > are
> > > > not
> > > > > > the intended recipient, please do not read, copy, use or
disclose
> > to
> > > > > others
> > > > > > this message or any attachment. Please also notify the sender by
> > > > replying
> > > > > > to this email or by telephone (+44(020 7896 0011) and then
delete
> > the
> > > > > email
> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > > to
> > > > > the
> > > > > > official business of this company shall be understood as neither
> > > given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > > > > > registered in England and Wales, company number 04008957) and IG
> > > Index
> > > > > > Limited (a company registered in England and Wales, company
> number
> > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited (register number
195355)
> > and
> > > > IG
> > > > > > Index Limited (register number 114059) are authorised and
> regulated
> > > by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
>
>
> --
> Nacho (Ignacio) Solis
> Kafka
> nsolis@linkedin.com
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Sriram Subramanian <ra...@confluent.io>.
FWIW, Apache Kafka has evolved a lot from where it started. It did start as
a messaging system. Over time we realized that that the vision for Kafka is
to build a streaming platform and not just a messaging system. You can take
a look at the site for more description about what comprises the streaming
platform http://kafka.apache.org/ and http://kafka.apache.org/intro.
Can the streaming platform exist without Connect? - No. Data integration is
fundamental to building an end to end platform
Can the streaming platform exist without stream processing? - No.
Processing stream data again is a core part of streaming platform.
Can the streaming platform exist without clients? - We at least need one
client library to complete the platform. Our Java clients help us to
complete the platform story. The rest of the clients are built and
maintained outside the project.
Can the platform exist without the rest proxy? - Yes. The proxy does not
complete the platform vision in anyway. It is just a good to have tool that
might be required by quite a few users and there is an active project that
works on this - https://github.com/confluentinc/kafka-rest
On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis <ns...@linkedin.com.invalid>
wrote:
> Are you saying Kafka REST is subjective but Kafka Streams and Kafka Connect
> are not subjective?
>
> > "there are likely places that can live without a rest proxy"
>
> There are also places that can live without Kafka Streams and Kafka
> Connect.
>
> Nacho
>
> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > At the high level, I think ideally it makes sense to add a component to
> > Apache Kafka if (1) it's widely needed and (2) it needs tight integration
> > with Kafka core. For Kafka Stream, we do expect stream processing will be
> > used widely in the future. Implementation wise, Kafka Stream only
> supports
> > getting data from Kafka and leverages quite a few of the core
> > functionalities in Kafka core. For example, it uses customized rebalance
> > callback in the consumer and uses the compacted topic heavily. So, having
> > Kafka Stream in the same repo makes it easier for testing when those core
> > functionalities evolve over time. Kafka Connect is in the same situation.
> >
> > For rest proxy, whether it's widely used or not is going to be a bit
> > subjective. However, there are likely places that can live without a rest
> > proxy. The rest proxy is just a proxy for the regular clients and doesn't
> > need to be tightly integrated with Kafka core. So, the case for including
> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream and
> > Kafka Connect.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > So from my reading essentially the first question needs to answered/and
> > > voted on is:
> > >
> > > Is Apache Kafka Community only about the Core or does the apache
> > community
> > > also support some subprojects (and just we need some better way to
> manage
> > > this)
> > >
> > > If vote for Core only wins, then the following should be removed:
> > > Kafka Connect
> > > Kafka Stream
> > >
> > > If vote for Core only loses (aka we will support subprojects) then:
> > > We should look to add Kafka Rest
> > >
> > > And we should look to see how we can manage better govern and manage
> > > submodules.
> > >
> > > A good example which id propose here is how some other communities in
> > > Apache do this.
> > >
> > > Each Module has a Module Management Committee(MMC), this is like almost
> > > the PMC but at a per module basis.
> > >
> > > This MMC should essentially hold the binding votes for that module.
> > > The MMC should be made up of a single representative from each
> > > organisation (so no single organisation can fully veto the community
> it
> > > has to a genuine consenus)
> > > The MMC requires at least 3 members (so there cant be a tied vote on 2)
> > > For a new Module to be added a MMC committee should be sought
> > > A new Module is only capable of being added if the above requirements
> can
> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> > > only actively support modules would be added
> > >
> > > The PMC reviews each module every 6months or Year. If MMC is inactive,
> a
> > > vote/call to find replacements if raised, if none are forthcoming
> > dropping
> > > the MMC to less than 3 then the module moves to "the attic" (very much
> > like
> > > apache attic but a little more aggressively)
> > >
> > > This way the PMC does not need to micro manage every module
> > > We only add modules where some amount of active support and maintenance
> > > and use is provided by the community
> > > We have an automatic way to retire old or inactive projects.
> > >
> > > Thoughts?
> > > Mike
> > >
> > >
> > > ________________________________________
> > > From: Harsha Ch <ha...@gmail.com>
> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > Jay,
> > > REST API is something every user is in need of. If the argument
> is
> > to
> > > clone and write your API, this will do a disservice to the users as
> they
> > > now have to choose one vs. others instead of keeping one API that is
> > > supported in Kafka community.
> > >
> > > "Pre-emptively re-creating another
> > > REST layer when it seems like we all quite agree on what needs to be
> done
> > > and we have an existing code base for HTTP/Kafka access that is heavily
> > > used in production seems quite silly."
> > >
> > > Exactly our point. Why can't we develop this in Apache Kafka
> > > community? Instead of us open sourcing another GitHub project and
> > creating
> > > a divide in users and another version of API. Let's build this in Kafka
> > > Community and use the governance model that is proven to provide vendor
> > > free user driven consensus features. The argument that is adding this
> > REST
> > > server to Kafka will affect the agility of the project doesn't mak
> sense.
> > >
> > > It looks like your argument is either we develop all these small tools
> or
> > > none at all. We as a community need to look at supporting critical
> > > tools/API. Instead of dividing this project into individual external
> > > communities. We should build this as part of Kafka which best serves
> the
> > > needs of users.
> > > The Streams and Connect projects that were pushed into Kafka
> > could
> > > have been left in their own Github projects based on your arguments.
> What
> > > about the REST API is so different that such that it should stay out of
> > the
> > > Kafka project? From my experience, more users are asking for the REST
> > API.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > I think the questions around governance make sense, I think we should
> > > > really clarify that to make the process more clear so it can be fully
> > > > inclusive.
> > > >
> > > > The idea that we should not collaborate on what is there now, though,
> > > > because in the future we might disagree about direction does not
> really
> > > > make sense to me. If in the future we disagree, that is the beauty of
> > > open
> > > > source, you can always fork off a copy of the code and start an
> > > independent
> > > > project either in Apache or elsewhere. Pre-emptively re-creating
> > another
> > > > REST layer when it seems like we all quite agree on what needs to be
> > done
> > > > and we have an existing code base for HTTP/kafka access that is
> heavily
> > > > used in production seems quite silly.
> > > >
> > > > Let me give some background on how I at least think about these
> things.
> > > > I've participated in open source projects out of LinkedIn via github
> as
> > > > well as via the ASF. I don't think there is a "right" answer to how
> to
> > do
> > > > these but rather some tradeoffs. We thought about this quite a lot in
> > the
> > > > context of Kafka based on the experience with the Hadoop ecosystem as
> > > well
> > > > as from other open source communities.
> > > >
> > > > There is a rich ecosystem around Kafka. Many of the projects are
> quite
> > > > small--single clients or tools that do a single thing well--and
> almost
> > > none
> > > > of them are top level apache projects. I don't think trying to force
> > each
> > > > of these to turn into independent Apache projects is necessarily the
> > best
> > > > thing for the ecosystem.
> > > >
> > > > My observation of how this can go wrong is really what I think has
> > > happened
> > > > to the Hadoop ecosystem. There you see quite a zoo of projects which
> > all
> > > > drift apart and don't quite work together well. Coordinating even
> > simple
> > > > changes and standardization across these is exceptionally difficult.
> > The
> > > > result is a bit of a mess for users--the pieces just don't really
> come
> > > > together very well. This makes sense for independent infrastructure
> > > systems
> > > > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > > > little tool or helper library has lead to a desirable state. I think
> > the
> > > > mode of operating where the Hadoop vendors spawn off a few new Apache
> > > > projects for each new product initiative, especially since often that
> > > > project is only valued by that vendor (and the other vendor has a
> > > different
> > > > competing Apache project) doesn't necessarily do a better job at
> > > producing
> > > > high quality communities or high quality software.
> > > >
> > > > These tools/connects/clients/proxies and other integration pieces can
> > > take
> > > > many forms, but my take of what makes one of these things good is
> that
> > it
> > > > remains simple, does its one thing well, and cleaves as closely as
> > > possible
> > > > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > > > monitoring, configuring, etc. For the tools we've contributed we've
> > tried
> > > > really hard to make them consistent with Kafka as well as with each
> > other
> > > > in how testing, configuration, monitoring, etc works.
> > > >
> > > > I think what Apache does superbly well is create a community for
> > > managing a
> > > > large infrastructure layer like Kafka in a vendor independent way.
> > What I
> > > > think is less successful is attempting to form full and independent
> > > apache
> > > > communities around very simple single purpose tools, especially if
> you
> > > hope
> > > > for these to come together into a cohesive toolset across multiple
> such
> > > > tools. Much of what Apache does--create a collective decision making
> > > > process for resolving disagreement, help to trademark and protect the
> > > marks
> > > > of the project, etc just isn't that relevant for simple
> single-purpose
> > > > tools.
> > > >
> > > > So my take is there are a couple of options:
> > > >
> > > > 1. We can try to put all the small tools into the Apache Project.
> I
> > > > think this is not the right approach as there is simply too many
> of
> > > > them,
> > > > many in different languages, serving different protocols,
> > integrating
> > > > with
> > > > particular systems, and a single community can't effectively
> > maintain
> > > > them
> > > > all. Doing this would significantly slow the progress of the Kafka
> > > > project.
> > > > As a protocol for messaging, I don't really see a case for
> including
> > > > REST
> > > > but not MQTT or AMQP which are technically much better suited to
> > > > messaging
> > > > and both are widely used for that.
> > > > 2. We can treat ecosystem projects that aren't top level Apache
> > > projects
> > > > as invalid and try to recreate them all as Apache projects.
> > Honestly,
> > > > though, if you go to the Kafka ecosystem page virtually none of
> the
> > > most
> > > > popular add-ons to Kafka are Apache projects. The most successful
> > > > things in
> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> > > other
> > > > clients, as well as the existing REST layer have succeeded at
> > > developing
> > > > communities that actively contribute and use these pieces and I
> > don't
> > > > know
> > > > that that is a bad thing unless that community proves to be
> > > uninclusive,
> > > > unresponsive, or goes in a bad technical direction--and those are
> > > > failure
> > > > modes that all open source efforts face.
> > > > 3. We can do what I think makes the most sense and try to work
> with
> > > the
> > > > projects that exist in the ecosystem and if the project doesn't
> > have a
> > > > responsive community or wants to go in a different direction fork
> or
> > > > recreate that work.
> > > >
> > > > Of course any person can choose whatever of these options they want.
> > But
> > > > from my point of view, option (3) has been the path of the community
> so
> > > far
> > > > and I think it has been quite successful.
> > > >
> > > > -Jay
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > > > wrote:
> > > >
> > > > > Neha,
> > > > > "But I haven't seen any community emails or patches being submitted
> > by
> > > > you
> > > > > guys, so I'm wondering why you are concerned about whether the
> > > community
> > > > is
> > > > > open to accepting patches or not."
> > > > >
> > > > > I think you are talking about contributing patches to this
> repository
> > > > > right? https://github.com/confluentinc/kafka-rest . All I am
> saying
> > > > > the guidelines/governance model is not clear on the project and I
> > guess
> > > > its
> > > > > driven by opening a github issue request. Its the repository owned
> > by
> > > > > confluent and as much I appreciate that the features we mentioned
> are
> > > in
> > > > > the roadmap and welcoming us to contribute to the project. It
> doesn't
> > > > > gurantee what we want to add in the furture will be in your
> roadmap.
> > > > >
> > > > > Hence the reason having it part of Kafka community will help a lot
> as
> > > > other
> > > > > users can participate in the discussions. We are happy to drive
> any
> > > > > feature additions through KIPs which gives everyone a chance to
> > > > participate
> > > > > and add to the discussions.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > I agree on the governance comments whole heartedly.
> > > > > >
> > > > > > Also i agree about the contribution comments made earlier in the
> > > > thread,
> > > > > i
> > > > > > personally am less likely to spend any of mine, or give project
> > time
> > > > > within
> > > > > > my internal projects to developers contributing to another
> > commercial
> > > > > > companies project even so technically open source, as then there
> is
> > > > that
> > > > > > commercial companies interest will always prevail and essentially
> > can
> > > > > > always have a final vote where disagreement. Im sure they never
> > > intend
> > > > > to,
> > > > > > but there is that true reality. This is why we have community
> open
> > > > source
> > > > > > projects.
> > > > > >
> > > > > > I can find many different implementations now of a rest endpoint
> on
> > > > > > GitHub, BitBucket etc. Each one has their benefits and
> > disadvantages
> > > in
> > > > > > their implementation. By making / providing one this would bring
> > > > together
> > > > > > these solutions, unifying those developers and also bringing the
> > best
> > > > of
> > > > > > all.
> > > > > >
> > > > > > I understand the concern on the community burden
> adding/supporting
> > > more
> > > > > > surface area for every client. But something like REST is
> universal
> > > and
> > > > > > worthy to be owned by the community.
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Andrew Schofield <an...@outlook.com>
> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > >
> > > > > > There's a massive difference between the governance of Kafka and
> > the
> > > > > > governance of the REST proxy.
> > > > > >
> > > > > > In Kafka, there is a broad community of people contributing their
> > > > > opinions
> > > > > > about future enhancements in the form of KIPs. There's some
> really
> > > deep
> > > > > > consideration that goes into some of the trickier KIPs. There are
> > > > people
> > > > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > > > reputations to become committers. I get the impression that the
> > > roadmap
> > > > > of
> > > > > > Kafka is not really community-owned (what's the big feature for
> > Kafka
> > > > > 0.11,
> > > > > > for example), but the conveyor belt of smaller features in the
> form
> > > of
> > > > > KIPs
> > > > > > works nicely. It's a good example of open-source working well.
> > > > > >
> > > > > > The equivalent for the REST proxy is basically issues on GitHub.
> > The
> > > > > > roadmap is less clear. There's not really a community properly
> > > engaged
> > > > in
> > > > > > the way that there is with Kafka. So, you could say that it's
> clear
> > > > that
> > > > > > fewer people are interested, but I think the whole governance
> > thing
> > > > is a
> > > > > > big barrier to engagement. And it's looking like it's getting out
> > of
> > > > > date.
> > > > > >
> > > > > > In technical terms, I can think of two big improvements to the
> REST
> > > > > proxy.
> > > > > > First, it needs to use the new consumer API so that it's possible
> > to
> > > > > secure
> > > > > > connections between the REST proxy and Kafka. I don't care too
> much
> > > > which
> > > > > > method calls it uses actually uses to consume messages, but I do
> > > care
> > > > > that
> > > > > > I cannot secure connections because of network security rules.
> > > Second,
> > > > > > there's an affinity between a consumer and the instance of the
> REST
> > > > proxy
> > > > > > to which it first connected. Kafka itself avoids this kind of
> > > affinity
> > > > > for
> > > > > > good reason, and in the name of availability the REST proxy
> should
> > > too.
> > > > > > These are natural KIPs.
> > > > > >
> > > > > > I think it would be good to have the code for the REST proxy
> > > > contributed
> > > > > > to Apache Kafka so that it would be able to be developed in the
> > same
> > > > way.
> > > > > >
> > > > > > Andrew Schofield
> > > > > >
> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > > > Sent: 07 October 2016 22:41:52
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > >
> > > > > > ASF already gives us a clear framework and governance model for
> > > > community
> > > > > > development. This is already understood by the people
> contributing
> > to
> > > > > > Apache Kafka project, and they are the same people who want to
> > > > contribute
> > > > > > to the REST server capability as well. Everyone is in agreement
> on
> > > the
> > > > > > need for collaborating on this effort. So why not contribute the
> > code
> > > > to
> > > > > > Apache Kafka. This will help avoid duplication of effort and
> forks
> > > that
> > > > > > may crop up, hugely benefitting the user community. This will
> also
> > > > avoid
> > > > > > having to define a process similar to ASF on a GitHub project and
> > > > instead
> > > > > > there is a single community with clear understanding community
> > > process
> > > > as
> > > > > > defined in ASF.
> > > > > >
> > > > > > As others have said, this is an important capability for Apache
> > > Kafka.
> > > > It
> > > > > > is worth maintaining this as a part of the project.
> > > > > >
> > > > > > Regards,
> > > > > > Suresh
> > > > > >
> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > > > >
> > > > > > >I personally think it would be quite wasteful to re-implement
> the
> > > REST
> > > > > > >gateway just because that an actively-maintained piece of
> > > > > Apache-licensed
> > > > > > >software is not governed directly by the Apache Kafka
> community...
> > > > While
> > > > > > >kafka-rest repo is owned by Confluent, the contributors
> including
> > > the
> > > > > main
> > > > > > >one are also part of the Apache Kafka community, so there is a
> > > chance
> > > > > to
> > > > > > >work this out.
> > > > > > >
> > > > > > >However, there are two valid concerns here that could be
> > addressed,
> > > > > around
> > > > > > >community and accessibility:
> > > > > > >>> What we are worried about is a project
> > > > > > >>> that's not maintained in a community. So the process of
> > accepting
> > > > > > >>>patches
> > > > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > > > >>>community.
> > > > > > >>> Not only that, existing REST API project doesn't support new
> > > client
> > > > > API
> > > > > > >and
> > > > > > >>> hence there is no security support either.
> > > > > > >
> > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
> community
> > > can
> > > > > > >clarify that - what is their contribution policy, dev style,
> > roadmap
> > > > > etc.
> > > > > > >If they want, they can make an effort to encourage participation
> > > from
> > > > > > >people outside Confluent (easily accept contributions, invite
> > > external
> > > > > > >commiters or have open dev process similar to Apache Kafka etc),
> > as
> > > > > there
> > > > > > >is definitely seems to be some interest on the list. That might
> > > clear
> > > > > the
> > > > > > >community concern and help kafka-rest project (but that is a
> > > > calculation
> > > > > > >Confluent will have to make).
> > > > > > >
> > > > > > >The other, independent, concern is that REST is something that
> is
> > > > > expected
> > > > > > >to be available out of the box with Kafka. I personally don't
> feel
> > > > > > >strongly
> > > > > > >about it (better use proper, efficient APIs from day one),
> though
> > it
> > > > is
> > > > > > >definitely way smaller than adding a stream processing engine to
> > the
> > > > > > >project :)
> > > > > > >Again,the kafka-rest "community" could take steps to make it
> even
> > > > easier
> > > > > > >to
> > > > > > >install, configure and run kafka-rest for new users on vanilla
> > > Apache
> > > > > > >Kafka
> > > > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > > > >contributions to that end), but that is up to them.
> > > > > > >Finally, if after the above steps were taken there would still a
> > > > strong
> > > > > > >desire to include a great rest gateway with Apache Kafka, I
> assume
> > > the
> > > > > > >community could hypothetically fork the existing kafka-rest into
> > an
> > > > > Apache
> > > > > > >Kafka subproject and maintain it "within Apache" instead of
> > > > implementing
> > > > > > >it
> > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot
> imagine
> > it
> > > > > > >happen
> > > > > > >without Confluent blessing, and I think that is likely much less
> > > > optimal
> > > > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > > > having
> > > > > a
> > > > > > >separate external community around kafka-rest.
> > > > > > >
> > > > > > >
> > > > > > >Just my two cents,
> > > > > > >
> > > > > > >
> > > > > > >Ofir Manor
> > > > > > >
> > > > > > >Co-Founder & CTO | Equalum
> > > > > > >
> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > <+972%2054-780-1286> |
> > > > Email:
> > > > > > ofir.manor@equalum.io
> > > > > > >
> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > > kafka@harsha.io
> > > > >
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Neha & Jay,
> > > > > > >> We did look at the open source alternatives.
> > Our
> > > > > > >>concern
> > > > > > >> is what's the patch acceptance and adding features/ bug-fixes
> to
> > > the
> > > > > > >> existing project under a Github (although it's licensed under
> > > Apache
> > > > > > >>2.0).
> > > > > > >> It would be great if that project made available under Apache
> > and
> > > > > > >>driven by
> > > > > > >> the community. Adding to the above, not all Kafka users are
> > > > > interested
> > > > > > >>in
> > > > > > >> using the Java client API, they would like to have simple REST
> > API
> > > > > where
> > > > > > >> they can code against using any language. I do believe this
> adds
> > > > value
> > > > > > >>to
> > > > > > >> Apache Kafka in itself.
> > > > > > >>
> > > > > > >> "For 1, I don't think there is value in giving in to the NIH
> > > > syndrome
> > > > > > >>and
> > > > > > >> reinventing the wheel. What I'm looking for is a detailed
> > > comparison
> > > > > of
> > > > > > >>the
> > > > > > >> gaps and why those can't be improved in the REST proxy that
> > > already
> > > > > > >>exists
> > > > > > >> and is actively maintained."
> > > > > > >>
> > > > > > >> We are not looking at this as NIH. What we are worried about
> > is a
> > > > > > >>project
> > > > > > >> that's not maintained in a community. So the process of
> > accepting
> > > > > > >>patches
> > > > > > >> and priorities is not clear, and it's not developed in Apache
> > > > > community.
> > > > > > >> Not only that, existing REST API project doesn't support new
> > > client
> > > > > API
> > > > > > >>and
> > > > > > >> hence there is no security support either.
> > > > > > >> We don't know the timeline when that's made available. We
> would
> > > like
> > > > > to
> > > > > > >>add
> > > > > > >> admin functionality into the REST API. So the Roadmap of that
> > > > project
> > > > > is
> > > > > > >> not driven by Apache.
> > > > > > >>
> > > > > > >>
> > > > > > >> "This doesn't materially have an impact on expanding the
> > usability
> > > > of
> > > > > > >> Kafka. In my experience, REST proxy + Java clients only
> cover
> > > > ~50%
> > > > > of
> > > > > > >> all
> > > > > > >> Kafka users, and maybe 10% of those are the ones who will
> use
> > > the
> > > > > > >>REST
> > > > > > >> proxy. The remaining 50% are non-java client users (C,
> > python,
> > > > go,
> > > > > > >>node
> > > > > > >> etc)."
> > > > > > >>
> > > > > > >> REST API is most often asked feature in my interactions with
> > Kafka
> > > > > > >>users.
> > > > > > >> In an organization, There will be independent teams who will
> > write
> > > > > their
> > > > > > >> Kafka clients using different language libraries available
> > today,
> > > > and
> > > > > > >> there is no way to standardize this. Instead of supporting
> > several
> > > > > > >> different client libraries users will be interested in using a
> > > REST
> > > > > API
> > > > > > >> server. The need for a REST API will only increase as more and
> > > more
> > > > > > >>users
> > > > > > >> start using Kafka.
> > > > > > >>
> > > > > > >> "More surface area means more work to keep things consistent.
> > > > Failure
> > > > > > >> to do that has, in fact, hurt the user experience."
> > > > > > >> Having myriad Kafka client GitHub projects that support
> > different
> > > > > > >>languages
> > > > > > >> hurts the user experience and pushes burden to maintain these
> > > > > libraries.
> > > > > > >> REST API is a simple code base that uses existing java client
> > > > > libraries
> > > > > > >>to
> > > > > > >> make life easier for the users.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Harsha
> > > > > > >>
> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > neha@confluent.io>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Manikumar,
> > > > > > >> >
> > > > > > >> > Thanks for sharing the proposal. I think there are 2 parts
> to
> > > this
> > > > > > >> > discussion -
> > > > > > >> >
> > > > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > > > >>feature-complete,
> > > > > > >> > open-source and actively maintained REST proxy in the
> > community?
> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more
> agile
> > > and
> > > > > > >> maintain
> > > > > > >> > the high-quality experience that Kafka users have today?
> > > > > > >> >
> > > > > > >> > For 1, I don't think there is value in giving in to the NIH
> > > > syndrome
> > > > > > >>and
> > > > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > > > comparison
> > > > > > >>of
> > > > > > >> the
> > > > > > >> > gaps and why those can't be improved in the REST proxy that
> > > > already
> > > > > > >> exists
> > > > > > >> > and is actively maintained. For example, we depend on
> zkClient
> > > and
> > > > > > >>have
> > > > > > >> > found as well as fixed several bugs by working closely with
> > the
> > > > > people
> > > > > > >> who
> > > > > > >> > maintain zkClient. This should be possible for REST proxy as
> > > well,
> > > > > > >>right?
> > > > > > >> >
> > > > > > >> > For 2, I'd like us to review our history of expanding the
> > > surface
> > > > > > >>area to
> > > > > > >> > add more clients in the past. Here is a summary -
> > > > > > >> >
> > > > > > >> > - This doesn't materially have an impact on expanding the
> > > > > > >>usability of
> > > > > > >> > Kafka. In my experience, REST proxy + Java clients only
> > cover
> > > > > ~50%
> > > > > > >>of
> > > > > > >> > all
> > > > > > >> > Kafka users, and maybe 10% of those are the ones who will
> > use
> > > > the
> > > > > > >>REST
> > > > > > >> > proxy. The remaining 50% are non-java client users (C,
> > > python,
> > > > > go,
> > > > > > >> node
> > > > > > >> > etc).
> > > > > > >> > - People are a lot more excited about promising to
> > contribute
> > > > > while
> > > > > > >> > adding the surface area but not on an ongoing basis down
> > the
> > > > > line.
> > > > > > >> > - More surface area means more work to keep things
> > > consistent.
> > > > > > >>Failure
> > > > > > >> > to do that has, in fact, hurt the user experience.
> > > > > > >> > - More surface area hurts agility. We want to do a few
> > things
> > > > > > >>really
> > > > > > >> > well as well as be agile to be able to build on our core
> > > > > > >>competency.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > > manikumar.reddy@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Jay,
> > > > > > >> > >
> > > > > > >> > > Thanks for your reply.
> > > > > > >> > >
> > > > > > >> > > I agree that we can not add all the clients/tools
> available
> > in
> > > > > > >> ecosystem
> > > > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > > > different
> > > > > > >>from
> > > > > > >> > > other clients/tools. Since any language that can work with
> > > HTTP
> > > > > can
> > > > > > >> > > easily integrate with this interface, Having an "official"
> > > REST
> > > > > > >> > > interface helps user community. This also helps us to
> > > integrate
> > > > > well
> > > > > > >> > > with external management and provisioning tools. Apache
> > Kafka
> > > > > > >>release
> > > > > > >> > > with Java clients + REST interface is sufficient for most
> of
> > > the
> > > > > > >>user
> > > > > > >> > > deployments/requirements. This helps users to deal with
> less
> > > > > number
> > > > > > >> > > of distributions/builds.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Manikumar
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> jay@confluent.io
> > >
> > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hey guys,
> > > > > > >> > > >
> > > > > > >> > > > There's already a REST interface maintained as a
> separate
> > > > > > >> project--it's
> > > > > > >> > > > open source and apache licensed and actively maintained
> (
> > > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is
> > wrong
> > > > with
> > > > > >
> > > > > >
> > > > > >
> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > > > github.com
> > > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
> > cluster.
> > > > It
> > > > > > makes it easy to produce and consume messages, view the state of
> > the
> > > > > > cluster, and ...
> > > > > >
> > > > > > >> that?
> > > > > > >> > > You
> > > > > > >> > > > mentioned that there was some compatibility concern, but
> > > > > > >> compatibility
> > > > > > >> > > has
> > > > > > >> > > > to do with the consumer protocol guarantees not the repo
> > the
> > > > > code
> > > > > > >>is
> > > > > > >> > in,
> > > > > > >> > > > right? Not sure that concern makes sense.
> > > > > > >> > > >
> > > > > > >> > > > We could argue for adding pretty much anything and
> > > everything
> > > > in
> > > > > > >>the
> > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that
> would
> > > > make
> > > > > > >>the
> > > > > > >> > > project
> > > > > > >> > > > more agile.
> > > > > > >> > > >
> > > > > > >> > > > -Jay
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > > > >> manikumar.reddy@gmail.com
> > > > > > >> > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi Kafka Devs,
> > > > > > >> > > > >
> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > > > Repository.
> > > > > > >> > > > >
> > > > > > >> > > > > There are already open-source alternatives are
> > available.
> > > > But
> > > > > > >>we
> > > > > > >> > would
> > > > > > >> > > > > like to add REST server that
> > > > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> > > Infra
> > > > > > >>tools
> > > > > > >> > comes
> > > > > > >> > > > up
> > > > > > >> > > > > with Rest Interface.
> > > > > > >> > > > > It is useful to have inbuilt Rest API support for
> > Produce,
> > > > > > >>Consume
> > > > > > >> > > > messages
> > > > > > >> > > > > and admin interface for
> > > > > > >> > > > > integrating with external management and provisioning
> > > > > tools.This
> > > > > > >> will
> > > > > > >> > > > also
> > > > > > >> > > > > allow the maintenance of
> > > > > > >> > > > > REST server and adding new features makes it easy
> > because
> > > > > apache
> > > > > > >> > > > community.
> > > > > > >> > > > >
> > > > > > >> > > > > The KIP wiki is the following:
> > > > > > >> > > > > https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-
> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > > > >> > > > >
> > > > > > >> > > > > Your comments and feedback are welcome.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Manikumar
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > --
> > > > > > >> > Thanks,
> > > > > > >> > Neha
> > > > > > >> >
> > > > > > >>
> > > > > > The information contained in this email is strictly confidential
> > and
> > > > for
> > > > > > the use of the addressee only, unless otherwise indicated. If you
> > are
> > > > not
> > > > > > the intended recipient, please do not read, copy, use or disclose
> > to
> > > > > others
> > > > > > this message or any attachment. Please also notify the sender by
> > > > replying
> > > > > > to this email or by telephone (+44(020 7896 0011) and then delete
> > the
> > > > > email
> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > > to
> > > > > the
> > > > > > official business of this company shall be understood as neither
> > > given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > > > > > registered in England and Wales, company number 04008957) and IG
> > > Index
> > > > > > Limited (a company registered in England and Wales, company
> number
> > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and
> > > > IG
> > > > > > Index Limited (register number 114059) are authorised and
> regulated
> > > by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
>
>
> --
> Nacho (Ignacio) Solis
> Kafka
> nsolis@linkedin.com
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Nacho Solis <ns...@linkedin.com.INVALID>.
Are you saying Kafka REST is subjective but Kafka Streams and Kafka Connect
are not subjective?
> "there are likely places that can live without a rest proxy"
There are also places that can live without Kafka Streams and Kafka Connect.
Nacho
On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <ju...@confluent.io> wrote:
> At the high level, I think ideally it makes sense to add a component to
> Apache Kafka if (1) it's widely needed and (2) it needs tight integration
> with Kafka core. For Kafka Stream, we do expect stream processing will be
> used widely in the future. Implementation wise, Kafka Stream only supports
> getting data from Kafka and leverages quite a few of the core
> functionalities in Kafka core. For example, it uses customized rebalance
> callback in the consumer and uses the compacted topic heavily. So, having
> Kafka Stream in the same repo makes it easier for testing when those core
> functionalities evolve over time. Kafka Connect is in the same situation.
>
> For rest proxy, whether it's widely used or not is going to be a bit
> subjective. However, there are likely places that can live without a rest
> proxy. The rest proxy is just a proxy for the regular clients and doesn't
> need to be tightly integrated with Kafka core. So, the case for including
> rest proxy in Apache Kafka is probably not as strong as Kafka Stream and
> Kafka Connect.
>
> Thanks,
>
> Jun
>
> On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > So from my reading essentially the first question needs to answered/and
> > voted on is:
> >
> > Is Apache Kafka Community only about the Core or does the apache
> community
> > also support some subprojects (and just we need some better way to manage
> > this)
> >
> > If vote for Core only wins, then the following should be removed:
> > Kafka Connect
> > Kafka Stream
> >
> > If vote for Core only loses (aka we will support subprojects) then:
> > We should look to add Kafka Rest
> >
> > And we should look to see how we can manage better govern and manage
> > submodules.
> >
> > A good example which id propose here is how some other communities in
> > Apache do this.
> >
> > Each Module has a Module Management Committee(MMC), this is like almost
> > the PMC but at a per module basis.
> >
> > This MMC should essentially hold the binding votes for that module.
> > The MMC should be made up of a single representative from each
> > organisation (so no single organisation can fully veto the community it
> > has to a genuine consenus)
> > The MMC requires at least 3 members (so there cant be a tied vote on 2)
> > For a new Module to be added a MMC committee should be sought
> > A new Module is only capable of being added if the above requirements can
> > be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> > only actively support modules would be added
> >
> > The PMC reviews each module every 6months or Year. If MMC is inactive, a
> > vote/call to find replacements if raised, if none are forthcoming
> dropping
> > the MMC to less than 3 then the module moves to "the attic" (very much
> like
> > apache attic but a little more aggressively)
> >
> > This way the PMC does not need to micro manage every module
> > We only add modules where some amount of active support and maintenance
> > and use is provided by the community
> > We have an automatic way to retire old or inactive projects.
> >
> > Thoughts?
> > Mike
> >
> >
> > ________________________________________
> > From: Harsha Ch <ha...@gmail.com>
> > Sent: Thursday, October 20, 2016 10:26 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >
> > Jay,
> > REST API is something every user is in need of. If the argument is
> to
> > clone and write your API, this will do a disservice to the users as they
> > now have to choose one vs. others instead of keeping one API that is
> > supported in Kafka community.
> >
> > "Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be done
> > and we have an existing code base for HTTP/Kafka access that is heavily
> > used in production seems quite silly."
> >
> > Exactly our point. Why can't we develop this in Apache Kafka
> > community? Instead of us open sourcing another GitHub project and
> creating
> > a divide in users and another version of API. Let's build this in Kafka
> > Community and use the governance model that is proven to provide vendor
> > free user driven consensus features. The argument that is adding this
> REST
> > server to Kafka will affect the agility of the project doesn't mak sense.
> >
> > It looks like your argument is either we develop all these small tools or
> > none at all. We as a community need to look at supporting critical
> > tools/API. Instead of dividing this project into individual external
> > communities. We should build this as part of Kafka which best serves the
> > needs of users.
> > The Streams and Connect projects that were pushed into Kafka
> could
> > have been left in their own Github projects based on your arguments. What
> > about the REST API is so different that such that it should stay out of
> the
> > Kafka project? From my experience, more users are asking for the REST
> API.
> >
> > Thanks,
> > Harsha
> >
> >
> >
> >
> >
> > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
> >
> > > I think the questions around governance make sense, I think we should
> > > really clarify that to make the process more clear so it can be fully
> > > inclusive.
> > >
> > > The idea that we should not collaborate on what is there now, though,
> > > because in the future we might disagree about direction does not really
> > > make sense to me. If in the future we disagree, that is the beauty of
> > open
> > > source, you can always fork off a copy of the code and start an
> > independent
> > > project either in Apache or elsewhere. Pre-emptively re-creating
> another
> > > REST layer when it seems like we all quite agree on what needs to be
> done
> > > and we have an existing code base for HTTP/kafka access that is heavily
> > > used in production seems quite silly.
> > >
> > > Let me give some background on how I at least think about these things.
> > > I've participated in open source projects out of LinkedIn via github as
> > > well as via the ASF. I don't think there is a "right" answer to how to
> do
> > > these but rather some tradeoffs. We thought about this quite a lot in
> the
> > > context of Kafka based on the experience with the Hadoop ecosystem as
> > well
> > > as from other open source communities.
> > >
> > > There is a rich ecosystem around Kafka. Many of the projects are quite
> > > small--single clients or tools that do a single thing well--and almost
> > none
> > > of them are top level apache projects. I don't think trying to force
> each
> > > of these to turn into independent Apache projects is necessarily the
> best
> > > thing for the ecosystem.
> > >
> > > My observation of how this can go wrong is really what I think has
> > happened
> > > to the Hadoop ecosystem. There you see quite a zoo of projects which
> all
> > > drift apart and don't quite work together well. Coordinating even
> simple
> > > changes and standardization across these is exceptionally difficult.
> The
> > > result is a bit of a mess for users--the pieces just don't really come
> > > together very well. This makes sense for independent infrastructure
> > systems
> > > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > > little tool or helper library has lead to a desirable state. I think
> the
> > > mode of operating where the Hadoop vendors spawn off a few new Apache
> > > projects for each new product initiative, especially since often that
> > > project is only valued by that vendor (and the other vendor has a
> > different
> > > competing Apache project) doesn't necessarily do a better job at
> > producing
> > > high quality communities or high quality software.
> > >
> > > These tools/connects/clients/proxies and other integration pieces can
> > take
> > > many forms, but my take of what makes one of these things good is that
> it
> > > remains simple, does its one thing well, and cleaves as closely as
> > possible
> > > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > > monitoring, configuring, etc. For the tools we've contributed we've
> tried
> > > really hard to make them consistent with Kafka as well as with each
> other
> > > in how testing, configuration, monitoring, etc works.
> > >
> > > I think what Apache does superbly well is create a community for
> > managing a
> > > large infrastructure layer like Kafka in a vendor independent way.
> What I
> > > think is less successful is attempting to form full and independent
> > apache
> > > communities around very simple single purpose tools, especially if you
> > hope
> > > for these to come together into a cohesive toolset across multiple such
> > > tools. Much of what Apache does--create a collective decision making
> > > process for resolving disagreement, help to trademark and protect the
> > marks
> > > of the project, etc just isn't that relevant for simple single-purpose
> > > tools.
> > >
> > > So my take is there are a couple of options:
> > >
> > > 1. We can try to put all the small tools into the Apache Project. I
> > > think this is not the right approach as there is simply too many of
> > > them,
> > > many in different languages, serving different protocols,
> integrating
> > > with
> > > particular systems, and a single community can't effectively
> maintain
> > > them
> > > all. Doing this would significantly slow the progress of the Kafka
> > > project.
> > > As a protocol for messaging, I don't really see a case for including
> > > REST
> > > but not MQTT or AMQP which are technically much better suited to
> > > messaging
> > > and both are widely used for that.
> > > 2. We can treat ecosystem projects that aren't top level Apache
> > projects
> > > as invalid and try to recreate them all as Apache projects.
> Honestly,
> > > though, if you go to the Kafka ecosystem page virtually none of the
> > most
> > > popular add-ons to Kafka are Apache projects. The most successful
> > > things in
> > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> > other
> > > clients, as well as the existing REST layer have succeeded at
> > developing
> > > communities that actively contribute and use these pieces and I
> don't
> > > know
> > > that that is a bad thing unless that community proves to be
> > uninclusive,
> > > unresponsive, or goes in a bad technical direction--and those are
> > > failure
> > > modes that all open source efforts face.
> > > 3. We can do what I think makes the most sense and try to work with
> > the
> > > projects that exist in the ecosystem and if the project doesn't
> have a
> > > responsive community or wants to go in a different direction fork or
> > > recreate that work.
> > >
> > > Of course any person can choose whatever of these options they want.
> But
> > > from my point of view, option (3) has been the path of the community so
> > far
> > > and I think it has been quite successful.
> > >
> > > -Jay
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <kafka@harsha.io
> >
> > > wrote:
> > >
> > > > Neha,
> > > > "But I haven't seen any community emails or patches being submitted
> by
> > > you
> > > > guys, so I'm wondering why you are concerned about whether the
> > community
> > > is
> > > > open to accepting patches or not."
> > > >
> > > > I think you are talking about contributing patches to this repository
> > > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > > the guidelines/governance model is not clear on the project and I
> guess
> > > its
> > > > driven by opening a github issue request. Its the repository owned
> by
> > > > confluent and as much I appreciate that the features we mentioned are
> > in
> > > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > > gurantee what we want to add in the furture will be in your roadmap.
> > > >
> > > > Hence the reason having it part of Kafka community will help a lot as
> > > other
> > > > users can participate in the discussions. We are happy to drive any
> > > > feature additions through KIPs which gives everyone a chance to
> > > participate
> > > > and add to the discussions.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > I agree on the governance comments whole heartedly.
> > > > >
> > > > > Also i agree about the contribution comments made earlier in the
> > > thread,
> > > > i
> > > > > personally am less likely to spend any of mine, or give project
> time
> > > > within
> > > > > my internal projects to developers contributing to another
> commercial
> > > > > companies project even so technically open source, as then there is
> > > that
> > > > > commercial companies interest will always prevail and essentially
> can
> > > > > always have a final vote where disagreement. Im sure they never
> > intend
> > > > to,
> > > > > but there is that true reality. This is why we have community open
> > > source
> > > > > projects.
> > > > >
> > > > > I can find many different implementations now of a rest endpoint on
> > > > > GitHub, BitBucket etc. Each one has their benefits and
> disadvantages
> > in
> > > > > their implementation. By making / providing one this would bring
> > > together
> > > > > these solutions, unifying those developers and also bringing the
> best
> > > of
> > > > > all.
> > > > >
> > > > > I understand the concern on the community burden adding/supporting
> > more
> > > > > surface area for every client. But something like REST is universal
> > and
> > > > > worthy to be owned by the community.
> > > > >
> > > > > Mike
> > > > >
> > > > >
> > > > > ________________________________________
> > > > > From: Andrew Schofield <an...@outlook.com>
> > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >
> > > > > There's a massive difference between the governance of Kafka and
> the
> > > > > governance of the REST proxy.
> > > > >
> > > > > In Kafka, there is a broad community of people contributing their
> > > > opinions
> > > > > about future enhancements in the form of KIPs. There's some really
> > deep
> > > > > consideration that goes into some of the trickier KIPs. There are
> > > people
> > > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > > reputations to become committers. I get the impression that the
> > roadmap
> > > > of
> > > > > Kafka is not really community-owned (what's the big feature for
> Kafka
> > > > 0.11,
> > > > > for example), but the conveyor belt of smaller features in the form
> > of
> > > > KIPs
> > > > > works nicely. It's a good example of open-source working well.
> > > > >
> > > > > The equivalent for the REST proxy is basically issues on GitHub.
> The
> > > > > roadmap is less clear. There's not really a community properly
> > engaged
> > > in
> > > > > the way that there is with Kafka. So, you could say that it's clear
> > > that
> > > > > fewer people are interested, but I think the whole governance
> thing
> > > is a
> > > > > big barrier to engagement. And it's looking like it's getting out
> of
> > > > date.
> > > > >
> > > > > In technical terms, I can think of two big improvements to the REST
> > > > proxy.
> > > > > First, it needs to use the new consumer API so that it's possible
> to
> > > > secure
> > > > > connections between the REST proxy and Kafka. I don't care too much
> > > which
> > > > > method calls it uses actually uses to consume messages, but I do
> > care
> > > > that
> > > > > I cannot secure connections because of network security rules.
> > Second,
> > > > > there's an affinity between a consumer and the instance of the REST
> > > proxy
> > > > > to which it first connected. Kafka itself avoids this kind of
> > affinity
> > > > for
> > > > > good reason, and in the name of availability the REST proxy should
> > too.
> > > > > These are natural KIPs.
> > > > >
> > > > > I think it would be good to have the code for the REST proxy
> > > contributed
> > > > > to Apache Kafka so that it would be able to be developed in the
> same
> > > way.
> > > > >
> > > > > Andrew Schofield
> > > > >
> > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > > Sent: 07 October 2016 22:41:52
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >
> > > > > ASF already gives us a clear framework and governance model for
> > > community
> > > > > development. This is already understood by the people contributing
> to
> > > > > Apache Kafka project, and they are the same people who want to
> > > contribute
> > > > > to the REST server capability as well. Everyone is in agreement on
> > the
> > > > > need for collaborating on this effort. So why not contribute the
> code
> > > to
> > > > > Apache Kafka. This will help avoid duplication of effort and forks
> > that
> > > > > may crop up, hugely benefitting the user community. This will also
> > > avoid
> > > > > having to define a process similar to ASF on a GitHub project and
> > > instead
> > > > > there is a single community with clear understanding community
> > process
> > > as
> > > > > defined in ASF.
> > > > >
> > > > > As others have said, this is an important capability for Apache
> > Kafka.
> > > It
> > > > > is worth maintaining this as a part of the project.
> > > > >
> > > > > Regards,
> > > > > Suresh
> > > > >
> > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > > >
> > > > > >I personally think it would be quite wasteful to re-implement the
> > REST
> > > > > >gateway just because that an actively-maintained piece of
> > > > Apache-licensed
> > > > > >software is not governed directly by the Apache Kafka community...
> > > While
> > > > > >kafka-rest repo is owned by Confluent, the contributors including
> > the
> > > > main
> > > > > >one are also part of the Apache Kafka community, so there is a
> > chance
> > > > to
> > > > > >work this out.
> > > > > >
> > > > > >However, there are two valid concerns here that could be
> addressed,
> > > > around
> > > > > >community and accessibility:
> > > > > >>> What we are worried about is a project
> > > > > >>> that's not maintained in a community. So the process of
> accepting
> > > > > >>>patches
> > > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > > >>>community.
> > > > > >>> Not only that, existing REST API project doesn't support new
> > client
> > > > API
> > > > > >and
> > > > > >>> hence there is no security support either.
> > > > > >
> > > > > >This might be easy to fix. Maybe Confluent / kafka-rest community
> > can
> > > > > >clarify that - what is their contribution policy, dev style,
> roadmap
> > > > etc.
> > > > > >If they want, they can make an effort to encourage participation
> > from
> > > > > >people outside Confluent (easily accept contributions, invite
> > external
> > > > > >commiters or have open dev process similar to Apache Kafka etc),
> as
> > > > there
> > > > > >is definitely seems to be some interest on the list. That might
> > clear
> > > > the
> > > > > >community concern and help kafka-rest project (but that is a
> > > calculation
> > > > > >Confluent will have to make).
> > > > > >
> > > > > >The other, independent, concern is that REST is something that is
> > > > expected
> > > > > >to be available out of the box with Kafka. I personally don't feel
> > > > > >strongly
> > > > > >about it (better use proper, efficient APIs from day one), though
> it
> > > is
> > > > > >definitely way smaller than adding a stream processing engine to
> the
> > > > > >project :)
> > > > > >Again,the kafka-rest "community" could take steps to make it even
> > > easier
> > > > > >to
> > > > > >install, configure and run kafka-rest for new users on vanilla
> > Apache
> > > > > >Kafka
> > > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > > >contributions to that end), but that is up to them.
> > > > > >Finally, if after the above steps were taken there would still a
> > > strong
> > > > > >desire to include a great rest gateway with Apache Kafka, I assume
> > the
> > > > > >community could hypothetically fork the existing kafka-rest into
> an
> > > > Apache
> > > > > >Kafka subproject and maintain it "within Apache" instead of
> > > implementing
> > > > > >it
> > > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine
> it
> > > > > >happen
> > > > > >without Confluent blessing, and I think that is likely much less
> > > optimal
> > > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > > having
> > > > a
> > > > > >separate external community around kafka-rest.
> > > > > >
> > > > > >
> > > > > >Just my two cents,
> > > > > >
> > > > > >
> > > > > >Ofir Manor
> > > > > >
> > > > > >Co-Founder & CTO | Equalum
> > > > > >
> > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> <+972%2054-780-1286> |
> > > Email:
> > > > > ofir.manor@equalum.io
> > > > > >
> > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > kafka@harsha.io
> > > >
> > > > > >wrote:
> > > > > >
> > > > > >> Neha & Jay,
> > > > > >> We did look at the open source alternatives.
> Our
> > > > > >>concern
> > > > > >> is what's the patch acceptance and adding features/ bug-fixes to
> > the
> > > > > >> existing project under a Github (although it's licensed under
> > Apache
> > > > > >>2.0).
> > > > > >> It would be great if that project made available under Apache
> and
> > > > > >>driven by
> > > > > >> the community. Adding to the above, not all Kafka users are
> > > > interested
> > > > > >>in
> > > > > >> using the Java client API, they would like to have simple REST
> API
> > > > where
> > > > > >> they can code against using any language. I do believe this adds
> > > value
> > > > > >>to
> > > > > >> Apache Kafka in itself.
> > > > > >>
> > > > > >> "For 1, I don't think there is value in giving in to the NIH
> > > syndrome
> > > > > >>and
> > > > > >> reinventing the wheel. What I'm looking for is a detailed
> > comparison
> > > > of
> > > > > >>the
> > > > > >> gaps and why those can't be improved in the REST proxy that
> > already
> > > > > >>exists
> > > > > >> and is actively maintained."
> > > > > >>
> > > > > >> We are not looking at this as NIH. What we are worried about
> is a
> > > > > >>project
> > > > > >> that's not maintained in a community. So the process of
> accepting
> > > > > >>patches
> > > > > >> and priorities is not clear, and it's not developed in Apache
> > > > community.
> > > > > >> Not only that, existing REST API project doesn't support new
> > client
> > > > API
> > > > > >>and
> > > > > >> hence there is no security support either.
> > > > > >> We don't know the timeline when that's made available. We would
> > like
> > > > to
> > > > > >>add
> > > > > >> admin functionality into the REST API. So the Roadmap of that
> > > project
> > > > is
> > > > > >> not driven by Apache.
> > > > > >>
> > > > > >>
> > > > > >> "This doesn't materially have an impact on expanding the
> usability
> > > of
> > > > > >> Kafka. In my experience, REST proxy + Java clients only cover
> > > ~50%
> > > > of
> > > > > >> all
> > > > > >> Kafka users, and maybe 10% of those are the ones who will use
> > the
> > > > > >>REST
> > > > > >> proxy. The remaining 50% are non-java client users (C,
> python,
> > > go,
> > > > > >>node
> > > > > >> etc)."
> > > > > >>
> > > > > >> REST API is most often asked feature in my interactions with
> Kafka
> > > > > >>users.
> > > > > >> In an organization, There will be independent teams who will
> write
> > > > their
> > > > > >> Kafka clients using different language libraries available
> today,
> > > and
> > > > > >> there is no way to standardize this. Instead of supporting
> several
> > > > > >> different client libraries users will be interested in using a
> > REST
> > > > API
> > > > > >> server. The need for a REST API will only increase as more and
> > more
> > > > > >>users
> > > > > >> start using Kafka.
> > > > > >>
> > > > > >> "More surface area means more work to keep things consistent.
> > > Failure
> > > > > >> to do that has, in fact, hurt the user experience."
> > > > > >> Having myriad Kafka client GitHub projects that support
> different
> > > > > >>languages
> > > > > >> hurts the user experience and pushes burden to maintain these
> > > > libraries.
> > > > > >> REST API is a simple code base that uses existing java client
> > > > libraries
> > > > > >>to
> > > > > >> make life easier for the users.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Harsha
> > > > > >>
> > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> neha@confluent.io>
> > > > > wrote:
> > > > > >>
> > > > > >> > Manikumar,
> > > > > >> >
> > > > > >> > Thanks for sharing the proposal. I think there are 2 parts to
> > this
> > > > > >> > discussion -
> > > > > >> >
> > > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > > >>feature-complete,
> > > > > >> > open-source and actively maintained REST proxy in the
> community?
> > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile
> > and
> > > > > >> maintain
> > > > > >> > the high-quality experience that Kafka users have today?
> > > > > >> >
> > > > > >> > For 1, I don't think there is value in giving in to the NIH
> > > syndrome
> > > > > >>and
> > > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > > comparison
> > > > > >>of
> > > > > >> the
> > > > > >> > gaps and why those can't be improved in the REST proxy that
> > > already
> > > > > >> exists
> > > > > >> > and is actively maintained. For example, we depend on zkClient
> > and
> > > > > >>have
> > > > > >> > found as well as fixed several bugs by working closely with
> the
> > > > people
> > > > > >> who
> > > > > >> > maintain zkClient. This should be possible for REST proxy as
> > well,
> > > > > >>right?
> > > > > >> >
> > > > > >> > For 2, I'd like us to review our history of expanding the
> > surface
> > > > > >>area to
> > > > > >> > add more clients in the past. Here is a summary -
> > > > > >> >
> > > > > >> > - This doesn't materially have an impact on expanding the
> > > > > >>usability of
> > > > > >> > Kafka. In my experience, REST proxy + Java clients only
> cover
> > > > ~50%
> > > > > >>of
> > > > > >> > all
> > > > > >> > Kafka users, and maybe 10% of those are the ones who will
> use
> > > the
> > > > > >>REST
> > > > > >> > proxy. The remaining 50% are non-java client users (C,
> > python,
> > > > go,
> > > > > >> node
> > > > > >> > etc).
> > > > > >> > - People are a lot more excited about promising to
> contribute
> > > > while
> > > > > >> > adding the surface area but not on an ongoing basis down
> the
> > > > line.
> > > > > >> > - More surface area means more work to keep things
> > consistent.
> > > > > >>Failure
> > > > > >> > to do that has, in fact, hurt the user experience.
> > > > > >> > - More surface area hurts agility. We want to do a few
> things
> > > > > >>really
> > > > > >> > well as well as be agile to be able to build on our core
> > > > > >>competency.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > manikumar.reddy@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi Jay,
> > > > > >> > >
> > > > > >> > > Thanks for your reply.
> > > > > >> > >
> > > > > >> > > I agree that we can not add all the clients/tools available
> in
> > > > > >> ecosystem
> > > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > > different
> > > > > >>from
> > > > > >> > > other clients/tools. Since any language that can work with
> > HTTP
> > > > can
> > > > > >> > > easily integrate with this interface, Having an "official"
> > REST
> > > > > >> > > interface helps user community. This also helps us to
> > integrate
> > > > well
> > > > > >> > > with external management and provisioning tools. Apache
> Kafka
> > > > > >>release
> > > > > >> > > with Java clients + REST interface is sufficient for most of
> > the
> > > > > >>user
> > > > > >> > > deployments/requirements. This helps users to deal with less
> > > > number
> > > > > >> > > of distributions/builds.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Manikumar
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <jay@confluent.io
> >
> > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hey guys,
> > > > > >> > > >
> > > > > >> > > > There's already a REST interface maintained as a separate
> > > > > >> project--it's
> > > > > >> > > > open source and apache licensed and actively maintained (
> > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is
> wrong
> > > with
> > > > >
> > > > >
> > > > >
> > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > > github.com
> > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
> cluster.
> > > It
> > > > > makes it easy to produce and consume messages, view the state of
> the
> > > > > cluster, and ...
> > > > >
> > > > > >> that?
> > > > > >> > > You
> > > > > >> > > > mentioned that there was some compatibility concern, but
> > > > > >> compatibility
> > > > > >> > > has
> > > > > >> > > > to do with the consumer protocol guarantees not the repo
> the
> > > > code
> > > > > >>is
> > > > > >> > in,
> > > > > >> > > > right? Not sure that concern makes sense.
> > > > > >> > > >
> > > > > >> > > > We could argue for adding pretty much anything and
> > everything
> > > in
> > > > > >>the
> > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> > > make
> > > > > >>the
> > > > > >> > > project
> > > > > >> > > > more agile.
> > > > > >> > > >
> > > > > >> > > > -Jay
> > > > > >> > > >
> > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > > >> manikumar.reddy@gmail.com
> > > > > >> > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Kafka Devs,
> > > > > >> > > > >
> > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > > Repository.
> > > > > >> > > > >
> > > > > >> > > > > There are already open-source alternatives are
> available.
> > > But
> > > > > >>we
> > > > > >> > would
> > > > > >> > > > > like to add REST server that
> > > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> > Infra
> > > > > >>tools
> > > > > >> > comes
> > > > > >> > > > up
> > > > > >> > > > > with Rest Interface.
> > > > > >> > > > > It is useful to have inbuilt Rest API support for
> Produce,
> > > > > >>Consume
> > > > > >> > > > messages
> > > > > >> > > > > and admin interface for
> > > > > >> > > > > integrating with external management and provisioning
> > > > tools.This
> > > > > >> will
> > > > > >> > > > also
> > > > > >> > > > > allow the maintenance of
> > > > > >> > > > > REST server and adding new features makes it easy
> because
> > > > apache
> > > > > >> > > > community.
> > > > > >> > > > >
> > > > > >> > > > > The KIP wiki is the following:
> > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > > >> > > > >
> > > > > >> > > > > Your comments and feedback are welcome.
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Manikumar
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > --
> > > > > >> > Thanks,
> > > > > >> > Neha
> > > > > >> >
> > > > > >>
> > > > > The information contained in this email is strictly confidential
> and
> > > for
> > > > > the use of the addressee only, unless otherwise indicated. If you
> are
> > > not
> > > > > the intended recipient, please do not read, copy, use or disclose
> to
> > > > others
> > > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to
> > > > the
> > > > > official business of this company shall be understood as neither
> > given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > > registered in England and Wales, company number 04008957) and IG
> > Index
> > > > > Limited (a company registered in England and Wales, company number
> > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> > > IG
> > > > > Index Limited (register number 114059) are authorised and regulated
> > by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
--
Nacho (Ignacio) Solis
Kafka
nsolis@linkedin.com
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Jun Rao <ju...@confluent.io>.
At the high level, I think ideally it makes sense to add a component to
Apache Kafka if (1) it's widely needed and (2) it needs tight integration
with Kafka core. For Kafka Stream, we do expect stream processing will be
used widely in the future. Implementation wise, Kafka Stream only supports
getting data from Kafka and leverages quite a few of the core
functionalities in Kafka core. For example, it uses customized rebalance
callback in the consumer and uses the compacted topic heavily. So, having
Kafka Stream in the same repo makes it easier for testing when those core
functionalities evolve over time. Kafka Connect is in the same situation.
For rest proxy, whether it's widely used or not is going to be a bit
subjective. However, there are likely places that can live without a rest
proxy. The rest proxy is just a proxy for the regular clients and doesn't
need to be tightly integrated with Kafka core. So, the case for including
rest proxy in Apache Kafka is probably not as strong as Kafka Stream and
Kafka Connect.
Thanks,
Jun
On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce <Mi...@ig.com>
wrote:
> So from my reading essentially the first question needs to answered/and
> voted on is:
>
> Is Apache Kafka Community only about the Core or does the apache community
> also support some subprojects (and just we need some better way to manage
> this)
>
> If vote for Core only wins, then the following should be removed:
> Kafka Connect
> Kafka Stream
>
> If vote for Core only loses (aka we will support subprojects) then:
> We should look to add Kafka Rest
>
> And we should look to see how we can manage better govern and manage
> submodules.
>
> A good example which id propose here is how some other communities in
> Apache do this.
>
> Each Module has a Module Management Committee(MMC), this is like almost
> the PMC but at a per module basis.
>
> This MMC should essentially hold the binding votes for that module.
> The MMC should be made up of a single representative from each
> organisation (so no single organisation can fully veto the community it
> has to a genuine consenus)
> The MMC requires at least 3 members (so there cant be a tied vote on 2)
> For a new Module to be added a MMC committee should be sought
> A new Module is only capable of being added if the above requirements can
> be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> only actively support modules would be added
>
> The PMC reviews each module every 6months or Year. If MMC is inactive, a
> vote/call to find replacements if raised, if none are forthcoming dropping
> the MMC to less than 3 then the module moves to "the attic" (very much like
> apache attic but a little more aggressively)
>
> This way the PMC does not need to micro manage every module
> We only add modules where some amount of active support and maintenance
> and use is provided by the community
> We have an automatic way to retire old or inactive projects.
>
> Thoughts?
> Mike
>
>
> ________________________________________
> From: Harsha Ch <ha...@gmail.com>
> Sent: Thursday, October 20, 2016 10:26 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> Jay,
> REST API is something every user is in need of. If the argument is to
> clone and write your API, this will do a disservice to the users as they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
>
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
>
> Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this REST
> server to Kafka will affect the agility of the project doesn't mak sense.
>
> It looks like your argument is either we develop all these small tools or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka could
> have been left in their own Github projects based on your arguments. What
> about the REST API is so different that such that it should stay out of the
> Kafka project? From my experience, more users are asking for the REST API.
>
> Thanks,
> Harsha
>
>
>
>
>
> On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
>
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not really
> > make sense to me. If in the future we disagree, that is the beauty of
> open
> > source, you can always fork off a copy of the code and start an
> independent
> > project either in Apache or elsewhere. Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be done
> > and we have an existing code base for HTTP/kafka access that is heavily
> > used in production seems quite silly.
> >
> > Let me give some background on how I at least think about these things.
> > I've participated in open source projects out of LinkedIn via github as
> > well as via the ASF. I don't think there is a "right" answer to how to do
> > these but rather some tradeoffs. We thought about this quite a lot in the
> > context of Kafka based on the experience with the Hadoop ecosystem as
> well
> > as from other open source communities.
> >
> > There is a rich ecosystem around Kafka. Many of the projects are quite
> > small--single clients or tools that do a single thing well--and almost
> none
> > of them are top level apache projects. I don't think trying to force each
> > of these to turn into independent Apache projects is necessarily the best
> > thing for the ecosystem.
> >
> > My observation of how this can go wrong is really what I think has
> happened
> > to the Hadoop ecosystem. There you see quite a zoo of projects which all
> > drift apart and don't quite work together well. Coordinating even simple
> > changes and standardization across these is exceptionally difficult. The
> > result is a bit of a mess for users--the pieces just don't really come
> > together very well. This makes sense for independent infrastructure
> systems
> > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > little tool or helper library has lead to a desirable state. I think the
> > mode of operating where the Hadoop vendors spawn off a few new Apache
> > projects for each new product initiative, especially since often that
> > project is only valued by that vendor (and the other vendor has a
> different
> > competing Apache project) doesn't necessarily do a better job at
> producing
> > high quality communities or high quality software.
> >
> > These tools/connects/clients/proxies and other integration pieces can
> take
> > many forms, but my take of what makes one of these things good is that it
> > remains simple, does its one thing well, and cleaves as closely as
> possible
> > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > monitoring, configuring, etc. For the tools we've contributed we've tried
> > really hard to make them consistent with Kafka as well as with each other
> > in how testing, configuration, monitoring, etc works.
> >
> > I think what Apache does superbly well is create a community for
> managing a
> > large infrastructure layer like Kafka in a vendor independent way. What I
> > think is less successful is attempting to form full and independent
> apache
> > communities around very simple single purpose tools, especially if you
> hope
> > for these to come together into a cohesive toolset across multiple such
> > tools. Much of what Apache does--create a collective decision making
> > process for resolving disagreement, help to trademark and protect the
> marks
> > of the project, etc just isn't that relevant for simple single-purpose
> > tools.
> >
> > So my take is there are a couple of options:
> >
> > 1. We can try to put all the small tools into the Apache Project. I
> > think this is not the right approach as there is simply too many of
> > them,
> > many in different languages, serving different protocols, integrating
> > with
> > particular systems, and a single community can't effectively maintain
> > them
> > all. Doing this would significantly slow the progress of the Kafka
> > project.
> > As a protocol for messaging, I don't really see a case for including
> > REST
> > but not MQTT or AMQP which are technically much better suited to
> > messaging
> > and both are widely used for that.
> > 2. We can treat ecosystem projects that aren't top level Apache
> projects
> > as invalid and try to recreate them all as Apache projects. Honestly,
> > though, if you go to the Kafka ecosystem page virtually none of the
> most
> > popular add-ons to Kafka are Apache projects. The most successful
> > things in
> > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> other
> > clients, as well as the existing REST layer have succeeded at
> developing
> > communities that actively contribute and use these pieces and I don't
> > know
> > that that is a bad thing unless that community proves to be
> uninclusive,
> > unresponsive, or goes in a bad technical direction--and those are
> > failure
> > modes that all open source efforts face.
> > 3. We can do what I think makes the most sense and try to work with
> the
> > projects that exist in the ecosystem and if the project doesn't have a
> > responsive community or wants to go in a different direction fork or
> > recreate that work.
> >
> > Of course any person can choose whatever of these options they want. But
> > from my point of view, option (3) has been the path of the community so
> far
> > and I think it has been quite successful.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Neha,
> > > "But I haven't seen any community emails or patches being submitted by
> > you
> > > guys, so I'm wondering why you are concerned about whether the
> community
> > is
> > > open to accepting patches or not."
> > >
> > > I think you are talking about contributing patches to this repository
> > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > the guidelines/governance model is not clear on the project and I guess
> > its
> > > driven by opening a github issue request. Its the repository owned by
> > > confluent and as much I appreciate that the features we mentioned are
> in
> > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > gurantee what we want to add in the furture will be in your roadmap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot as
> > other
> > > users can participate in the discussions. We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project time
> > > within
> > > > my internal projects to developers contributing to another commercial
> > > > companies project even so technically open source, as then there is
> > that
> > > > commercial companies interest will always prevail and essentially can
> > > > always have a final vote where disagreement. Im sure they never
> intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint on
> > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages
> in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
> more
> > > > surface area for every client. But something like REST is universal
> and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > ________________________________________
> > > > From: Andrew Schofield <an...@outlook.com>
> > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > There's a massive difference between the governance of Kafka and the
> > > > governance of the REST proxy.
> > > >
> > > > In Kafka, there is a broad community of people contributing their
> > > opinions
> > > > about future enhancements in the form of KIPs. There's some really
> deep
> > > > consideration that goes into some of the trickier KIPs. There are
> > people
> > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > reputations to become committers. I get the impression that the
> roadmap
> > > of
> > > > Kafka is not really community-owned (what's the big feature for Kafka
> > > 0.11,
> > > > for example), but the conveyor belt of smaller features in the form
> of
> > > KIPs
> > > > works nicely. It's a good example of open-source working well.
> > > >
> > > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > > roadmap is less clear. There's not really a community properly
> engaged
> > in
> > > > the way that there is with Kafka. So, you could say that it's clear
> > that
> > > > fewer people are interested, but I think the whole governance thing
> > is a
> > > > big barrier to engagement. And it's looking like it's getting out of
> > > date.
> > > >
> > > > In technical terms, I can think of two big improvements to the REST
> > > proxy.
> > > > First, it needs to use the new consumer API so that it's possible to
> > > secure
> > > > connections between the REST proxy and Kafka. I don't care too much
> > which
> > > > method calls it uses actually uses to consume messages, but I do
> care
> > > that
> > > > I cannot secure connections because of network security rules.
> Second,
> > > > there's an affinity between a consumer and the instance of the REST
> > proxy
> > > > to which it first connected. Kafka itself avoids this kind of
> affinity
> > > for
> > > > good reason, and in the name of availability the REST proxy should
> too.
> > > > These are natural KIPs.
> > > >
> > > > I think it would be good to have the code for the REST proxy
> > contributed
> > > > to Apache Kafka so that it would be able to be developed in the same
> > way.
> > > >
> > > > Andrew Schofield
> > > >
> > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > Sent: 07 October 2016 22:41:52
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > ASF already gives us a clear framework and governance model for
> > community
> > > > development. This is already understood by the people contributing to
> > > > Apache Kafka project, and they are the same people who want to
> > contribute
> > > > to the REST server capability as well. Everyone is in agreement on
> the
> > > > need for collaborating on this effort. So why not contribute the code
> > to
> > > > Apache Kafka. This will help avoid duplication of effort and forks
> that
> > > > may crop up, hugely benefitting the user community. This will also
> > avoid
> > > > having to define a process similar to ASF on a GitHub project and
> > instead
> > > > there is a single community with clear understanding community
> process
> > as
> > > > defined in ASF.
> > > >
> > > > As others have said, this is an important capability for Apache
> Kafka.
> > It
> > > > is worth maintaining this as a part of the project.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > >
> > > > >I personally think it would be quite wasteful to re-implement the
> REST
> > > > >gateway just because that an actively-maintained piece of
> > > Apache-licensed
> > > > >software is not governed directly by the Apache Kafka community...
> > While
> > > > >kafka-rest repo is owned by Confluent, the contributors including
> the
> > > main
> > > > >one are also part of the Apache Kafka community, so there is a
> chance
> > > to
> > > > >work this out.
> > > > >
> > > > >However, there are two valid concerns here that could be addressed,
> > > around
> > > > >community and accessibility:
> > > > >>> What we are worried about is a project
> > > > >>> that's not maintained in a community. So the process of accepting
> > > > >>>patches
> > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > >>>community.
> > > > >>> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >and
> > > > >>> hence there is no security support either.
> > > > >
> > > > >This might be easy to fix. Maybe Confluent / kafka-rest community
> can
> > > > >clarify that - what is their contribution policy, dev style, roadmap
> > > etc.
> > > > >If they want, they can make an effort to encourage participation
> from
> > > > >people outside Confluent (easily accept contributions, invite
> external
> > > > >commiters or have open dev process similar to Apache Kafka etc), as
> > > there
> > > > >is definitely seems to be some interest on the list. That might
> clear
> > > the
> > > > >community concern and help kafka-rest project (but that is a
> > calculation
> > > > >Confluent will have to make).
> > > > >
> > > > >The other, independent, concern is that REST is something that is
> > > expected
> > > > >to be available out of the box with Kafka. I personally don't feel
> > > > >strongly
> > > > >about it (better use proper, efficient APIs from day one), though it
> > is
> > > > >definitely way smaller than adding a stream processing engine to the
> > > > >project :)
> > > > >Again,the kafka-rest "community" could take steps to make it even
> > easier
> > > > >to
> > > > >install, configure and run kafka-rest for new users on vanilla
> Apache
> > > > >Kafka
> > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > >contributions to that end), but that is up to them.
> > > > >Finally, if after the above steps were taken there would still a
> > strong
> > > > >desire to include a great rest gateway with Apache Kafka, I assume
> the
> > > > >community could hypothetically fork the existing kafka-rest into an
> > > Apache
> > > > >Kafka subproject and maintain it "within Apache" instead of
> > implementing
> > > > >it
> > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> > > > >happen
> > > > >without Confluent blessing, and I think that is likely much less
> > optimal
> > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > having
> > > a
> > > > >separate external community around kafka-rest.
> > > > >
> > > > >
> > > > >Just my two cents,
> > > > >
> > > > >
> > > > >Ofir Manor
> > > > >
> > > > >Co-Founder & CTO | Equalum
> > > > >
> > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> <+972%2054-780-1286> |
> > Email:
> > > > ofir.manor@equalum.io
> > > > >
> > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > > > >wrote:
> > > > >
> > > > >> Neha & Jay,
> > > > >> We did look at the open source alternatives. Our
> > > > >>concern
> > > > >> is what's the patch acceptance and adding features/ bug-fixes to
> the
> > > > >> existing project under a Github (although it's licensed under
> Apache
> > > > >>2.0).
> > > > >> It would be great if that project made available under Apache and
> > > > >>driven by
> > > > >> the community. Adding to the above, not all Kafka users are
> > > interested
> > > > >>in
> > > > >> using the Java client API, they would like to have simple REST API
> > > where
> > > > >> they can code against using any language. I do believe this adds
> > value
> > > > >>to
> > > > >> Apache Kafka in itself.
> > > > >>
> > > > >> "For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> reinventing the wheel. What I'm looking for is a detailed
> comparison
> > > of
> > > > >>the
> > > > >> gaps and why those can't be improved in the REST proxy that
> already
> > > > >>exists
> > > > >> and is actively maintained."
> > > > >>
> > > > >> We are not looking at this as NIH. What we are worried about is a
> > > > >>project
> > > > >> that's not maintained in a community. So the process of accepting
> > > > >>patches
> > > > >> and priorities is not clear, and it's not developed in Apache
> > > community.
> > > > >> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >>and
> > > > >> hence there is no security support either.
> > > > >> We don't know the timeline when that's made available. We would
> like
> > > to
> > > > >>add
> > > > >> admin functionality into the REST API. So the Roadmap of that
> > project
> > > is
> > > > >> not driven by Apache.
> > > > >>
> > > > >>
> > > > >> "This doesn't materially have an impact on expanding the usability
> > of
> > > > >> Kafka. In my experience, REST proxy + Java clients only cover
> > ~50%
> > > of
> > > > >> all
> > > > >> Kafka users, and maybe 10% of those are the ones who will use
> the
> > > > >>REST
> > > > >> proxy. The remaining 50% are non-java client users (C, python,
> > go,
> > > > >>node
> > > > >> etc)."
> > > > >>
> > > > >> REST API is most often asked feature in my interactions with Kafka
> > > > >>users.
> > > > >> In an organization, There will be independent teams who will write
> > > their
> > > > >> Kafka clients using different language libraries available today,
> > and
> > > > >> there is no way to standardize this. Instead of supporting several
> > > > >> different client libraries users will be interested in using a
> REST
> > > API
> > > > >> server. The need for a REST API will only increase as more and
> more
> > > > >>users
> > > > >> start using Kafka.
> > > > >>
> > > > >> "More surface area means more work to keep things consistent.
> > Failure
> > > > >> to do that has, in fact, hurt the user experience."
> > > > >> Having myriad Kafka client GitHub projects that support different
> > > > >>languages
> > > > >> hurts the user experience and pushes burden to maintain these
> > > libraries.
> > > > >> REST API is a simple code base that uses existing java client
> > > libraries
> > > > >>to
> > > > >> make life easier for the users.
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > > > wrote:
> > > > >>
> > > > >> > Manikumar,
> > > > >> >
> > > > >> > Thanks for sharing the proposal. I think there are 2 parts to
> this
> > > > >> > discussion -
> > > > >> >
> > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > >>feature-complete,
> > > > >> > open-source and actively maintained REST proxy in the community?
> > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile
> and
> > > > >> maintain
> > > > >> > the high-quality experience that Kafka users have today?
> > > > >> >
> > > > >> > For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > comparison
> > > > >>of
> > > > >> the
> > > > >> > gaps and why those can't be improved in the REST proxy that
> > already
> > > > >> exists
> > > > >> > and is actively maintained. For example, we depend on zkClient
> and
> > > > >>have
> > > > >> > found as well as fixed several bugs by working closely with the
> > > people
> > > > >> who
> > > > >> > maintain zkClient. This should be possible for REST proxy as
> well,
> > > > >>right?
> > > > >> >
> > > > >> > For 2, I'd like us to review our history of expanding the
> surface
> > > > >>area to
> > > > >> > add more clients in the past. Here is a summary -
> > > > >> >
> > > > >> > - This doesn't materially have an impact on expanding the
> > > > >>usability of
> > > > >> > Kafka. In my experience, REST proxy + Java clients only cover
> > > ~50%
> > > > >>of
> > > > >> > all
> > > > >> > Kafka users, and maybe 10% of those are the ones who will use
> > the
> > > > >>REST
> > > > >> > proxy. The remaining 50% are non-java client users (C,
> python,
> > > go,
> > > > >> node
> > > > >> > etc).
> > > > >> > - People are a lot more excited about promising to contribute
> > > while
> > > > >> > adding the surface area but not on an ongoing basis down the
> > > line.
> > > > >> > - More surface area means more work to keep things
> consistent.
> > > > >>Failure
> > > > >> > to do that has, in fact, hurt the user experience.
> > > > >> > - More surface area hurts agility. We want to do a few things
> > > > >>really
> > > > >> > well as well as be agile to be able to build on our core
> > > > >>competency.
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Jay,
> > > > >> > >
> > > > >> > > Thanks for your reply.
> > > > >> > >
> > > > >> > > I agree that we can not add all the clients/tools available in
> > > > >> ecosystem
> > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > different
> > > > >>from
> > > > >> > > other clients/tools. Since any language that can work with
> HTTP
> > > can
> > > > >> > > easily integrate with this interface, Having an "official"
> REST
> > > > >> > > interface helps user community. This also helps us to
> integrate
> > > well
> > > > >> > > with external management and provisioning tools. Apache Kafka
> > > > >>release
> > > > >> > > with Java clients + REST interface is sufficient for most of
> the
> > > > >>user
> > > > >> > > deployments/requirements. This helps users to deal with less
> > > number
> > > > >> > > of distributions/builds.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Manikumar
> > > > >> > >
> > > > >> > >
> > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> > > wrote:
> > > > >> > >
> > > > >> > > > Hey guys,
> > > > >> > > >
> > > > >> > > > There's already a REST interface maintained as a separate
> > > > >> project--it's
> > > > >> > > > open source and apache licensed and actively maintained (
> > > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong
> > with
> > > >
> > > >
> > > >
> > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > github.com
> > > > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
> > It
> > > > makes it easy to produce and consume messages, view the state of the
> > > > cluster, and ...
> > > >
> > > > >> that?
> > > > >> > > You
> > > > >> > > > mentioned that there was some compatibility concern, but
> > > > >> compatibility
> > > > >> > > has
> > > > >> > > > to do with the consumer protocol guarantees not the repo the
> > > code
> > > > >>is
> > > > >> > in,
> > > > >> > > > right? Not sure that concern makes sense.
> > > > >> > > >
> > > > >> > > > We could argue for adding pretty much anything and
> everything
> > in
> > > > >>the
> > > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> > make
> > > > >>the
> > > > >> > > project
> > > > >> > > > more agile.
> > > > >> > > >
> > > > >> > > > -Jay
> > > > >> > > >
> > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > >> manikumar.reddy@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Kafka Devs,
> > > > >> > > > >
> > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > Repository.
> > > > >> > > > >
> > > > >> > > > > There are already open-source alternatives are available.
> > But
> > > > >>we
> > > > >> > would
> > > > >> > > > > like to add REST server that
> > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> Infra
> > > > >>tools
> > > > >> > comes
> > > > >> > > > up
> > > > >> > > > > with Rest Interface.
> > > > >> > > > > It is useful to have inbuilt Rest API support for Produce,
> > > > >>Consume
> > > > >> > > > messages
> > > > >> > > > > and admin interface for
> > > > >> > > > > integrating with external management and provisioning
> > > tools.This
> > > > >> will
> > > > >> > > > also
> > > > >> > > > > allow the maintenance of
> > > > >> > > > > REST server and adding new features makes it easy because
> > > apache
> > > > >> > > > community.
> > > > >> > > > >
> > > > >> > > > > The KIP wiki is the following:
> > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > >> > > > >
> > > > >> > > > > Your comments and feedback are welcome.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Manikumar
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > --
> > > > >> > Thanks,
> > > > >> > Neha
> > > > >> >
> > > > >>
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Michael Pearce <Mi...@ig.com>.
Here is Ignite’s - essentially idea is having different owners/maintainers per module/area that keep a check on that area.
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
On 10/21/16, 3:44 PM, "ismaelj@gmail.com on behalf of Ismael Juma" <ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
Hi Michael,
Can you please share which Apache projects have a MMC? I couldn't find
anything after a quick google.
Ismael
On Fri, Oct 21, 2016 at 7:28 AM, Michael Pearce <Mi...@ig.com>
wrote:
> So from my reading essentially the first question needs to answered/and
> voted on is:
>
> Is Apache Kafka Community only about the Core or does the apache community
> also support some subprojects (and just we need some better way to manage
> this)
>
> If vote for Core only wins, then the following should be removed:
> Kafka Connect
> Kafka Stream
>
> If vote for Core only loses (aka we will support subprojects) then:
> We should look to add Kafka Rest
>
> And we should look to see how we can manage better govern and manage
> submodules.
>
> A good example which id propose here is how some other communities in
> Apache do this.
>
> Each Module has a Module Management Committee(MMC), this is like almost
> the PMC but at a per module basis.
>
> This MMC should essentially hold the binding votes for that module.
> The MMC should be made up of a single representative from each
> organisation (so no single organisation can fully veto the community it
> has to a genuine consenus)
> The MMC requires at least 3 members (so there cant be a tied vote on 2)
> For a new Module to be added a MMC committee should be sought
> A new Module is only capable of being added if the above requirements can
> be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> only actively support modules would be added
>
> The PMC reviews each module every 6months or Year. If MMC is inactive, a
> vote/call to find replacements if raised, if none are forthcoming dropping
> the MMC to less than 3 then the module moves to "the attic" (very much like
> apache attic but a little more aggressively)
>
> This way the PMC does not need to micro manage every module
> We only add modules where some amount of active support and maintenance
> and use is provided by the community
> We have an automatic way to retire old or inactive projects.
>
> Thoughts?
> Mike
>
>
> ________________________________________
> From: Harsha Ch <ha...@gmail.com>
> Sent: Thursday, October 20, 2016 10:26 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> Jay,
> REST API is something every user is in need of. If the argument is to
> clone and write your API, this will do a disservice to the users as they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
>
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
>
> Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this REST
> server to Kafka will affect the agility of the project doesn't mak sense.
>
> It looks like your argument is either we develop all these small tools or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka could
> have been left in their own Github projects based on your arguments. What
> about the REST API is so different that such that it should stay out of the
> Kafka project? From my experience, more users are asking for the REST API.
>
> Thanks,
> Harsha
>
>
>
>
>
> On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
>
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not really
> > make sense to me. If in the future we disagree, that is the beauty of
> open
> > source, you can always fork off a copy of the code and start an
> independent
> > project either in Apache or elsewhere. Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be done
> > and we have an existing code base for HTTP/kafka access that is heavily
> > used in production seems quite silly.
> >
> > Let me give some background on how I at least think about these things.
> > I've participated in open source projects out of LinkedIn via github as
> > well as via the ASF. I don't think there is a "right" answer to how to do
> > these but rather some tradeoffs. We thought about this quite a lot in the
> > context of Kafka based on the experience with the Hadoop ecosystem as
> well
> > as from other open source communities.
> >
> > There is a rich ecosystem around Kafka. Many of the projects are quite
> > small--single clients or tools that do a single thing well--and almost
> none
> > of them are top level apache projects. I don't think trying to force each
> > of these to turn into independent Apache projects is necessarily the best
> > thing for the ecosystem.
> >
> > My observation of how this can go wrong is really what I think has
> happened
> > to the Hadoop ecosystem. There you see quite a zoo of projects which all
> > drift apart and don't quite work together well. Coordinating even simple
> > changes and standardization across these is exceptionally difficult. The
> > result is a bit of a mess for users--the pieces just don't really come
> > together very well. This makes sense for independent infrastructure
> systems
> > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > little tool or helper library has lead to a desirable state. I think the
> > mode of operating where the Hadoop vendors spawn off a few new Apache
> > projects for each new product initiative, especially since often that
> > project is only valued by that vendor (and the other vendor has a
> different
> > competing Apache project) doesn't necessarily do a better job at
> producing
> > high quality communities or high quality software.
> >
> > These tools/connects/clients/proxies and other integration pieces can
> take
> > many forms, but my take of what makes one of these things good is that it
> > remains simple, does its one thing well, and cleaves as closely as
> possible
> > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > monitoring, configuring, etc. For the tools we've contributed we've tried
> > really hard to make them consistent with Kafka as well as with each other
> > in how testing, configuration, monitoring, etc works.
> >
> > I think what Apache does superbly well is create a community for
> managing a
> > large infrastructure layer like Kafka in a vendor independent way. What I
> > think is less successful is attempting to form full and independent
> apache
> > communities around very simple single purpose tools, especially if you
> hope
> > for these to come together into a cohesive toolset across multiple such
> > tools. Much of what Apache does--create a collective decision making
> > process for resolving disagreement, help to trademark and protect the
> marks
> > of the project, etc just isn't that relevant for simple single-purpose
> > tools.
> >
> > So my take is there are a couple of options:
> >
> > 1. We can try to put all the small tools into the Apache Project. I
> > think this is not the right approach as there is simply too many of
> > them,
> > many in different languages, serving different protocols, integrating
> > with
> > particular systems, and a single community can't effectively maintain
> > them
> > all. Doing this would significantly slow the progress of the Kafka
> > project.
> > As a protocol for messaging, I don't really see a case for including
> > REST
> > but not MQTT or AMQP which are technically much better suited to
> > messaging
> > and both are widely used for that.
> > 2. We can treat ecosystem projects that aren't top level Apache
> projects
> > as invalid and try to recreate them all as Apache projects. Honestly,
> > though, if you go to the Kafka ecosystem page virtually none of the
> most
> > popular add-ons to Kafka are Apache projects. The most successful
> > things in
> > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> other
> > clients, as well as the existing REST layer have succeeded at
> developing
> > communities that actively contribute and use these pieces and I don't
> > know
> > that that is a bad thing unless that community proves to be
> uninclusive,
> > unresponsive, or goes in a bad technical direction--and those are
> > failure
> > modes that all open source efforts face.
> > 3. We can do what I think makes the most sense and try to work with
> the
> > projects that exist in the ecosystem and if the project doesn't have a
> > responsive community or wants to go in a different direction fork or
> > recreate that work.
> >
> > Of course any person can choose whatever of these options they want. But
> > from my point of view, option (3) has been the path of the community so
> far
> > and I think it has been quite successful.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Neha,
> > > "But I haven't seen any community emails or patches being submitted by
> > you
> > > guys, so I'm wondering why you are concerned about whether the
> community
> > is
> > > open to accepting patches or not."
> > >
> > > I think you are talking about contributing patches to this repository
> > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > the guidelines/governance model is not clear on the project and I guess
> > its
> > > driven by opening a github issue request. Its the repository owned by
> > > confluent and as much I appreciate that the features we mentioned are
> in
> > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > gurantee what we want to add in the furture will be in your roadmap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot as
> > other
> > > users can participate in the discussions. We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project time
> > > within
> > > > my internal projects to developers contributing to another commercial
> > > > companies project even so technically open source, as then there is
> > that
> > > > commercial companies interest will always prevail and essentially can
> > > > always have a final vote where disagreement. Im sure they never
> intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint on
> > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages
> in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
> more
> > > > surface area for every client. But something like REST is universal
> and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > ________________________________________
> > > > From: Andrew Schofield <an...@outlook.com>
> > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > There's a massive difference between the governance of Kafka and the
> > > > governance of the REST proxy.
> > > >
> > > > In Kafka, there is a broad community of people contributing their
> > > opinions
> > > > about future enhancements in the form of KIPs. There's some really
> deep
> > > > consideration that goes into some of the trickier KIPs. There are
> > people
> > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > reputations to become committers. I get the impression that the
> roadmap
> > > of
> > > > Kafka is not really community-owned (what's the big feature for Kafka
> > > 0.11,
> > > > for example), but the conveyor belt of smaller features in the form
> of
> > > KIPs
> > > > works nicely. It's a good example of open-source working well.
> > > >
> > > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > > roadmap is less clear. There's not really a community properly
> engaged
> > in
> > > > the way that there is with Kafka. So, you could say that it's clear
> > that
> > > > fewer people are interested, but I think the whole governance thing
> > is a
> > > > big barrier to engagement. And it's looking like it's getting out of
> > > date.
> > > >
> > > > In technical terms, I can think of two big improvements to the REST
> > > proxy.
> > > > First, it needs to use the new consumer API so that it's possible to
> > > secure
> > > > connections between the REST proxy and Kafka. I don't care too much
> > which
> > > > method calls it uses actually uses to consume messages, but I do
> care
> > > that
> > > > I cannot secure connections because of network security rules.
> Second,
> > > > there's an affinity between a consumer and the instance of the REST
> > proxy
> > > > to which it first connected. Kafka itself avoids this kind of
> affinity
> > > for
> > > > good reason, and in the name of availability the REST proxy should
> too.
> > > > These are natural KIPs.
> > > >
> > > > I think it would be good to have the code for the REST proxy
> > contributed
> > > > to Apache Kafka so that it would be able to be developed in the same
> > way.
> > > >
> > > > Andrew Schofield
> > > >
> > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > Sent: 07 October 2016 22:41:52
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > ASF already gives us a clear framework and governance model for
> > community
> > > > development. This is already understood by the people contributing to
> > > > Apache Kafka project, and they are the same people who want to
> > contribute
> > > > to the REST server capability as well. Everyone is in agreement on
> the
> > > > need for collaborating on this effort. So why not contribute the code
> > to
> > > > Apache Kafka. This will help avoid duplication of effort and forks
> that
> > > > may crop up, hugely benefitting the user community. This will also
> > avoid
> > > > having to define a process similar to ASF on a GitHub project and
> > instead
> > > > there is a single community with clear understanding community
> process
> > as
> > > > defined in ASF.
> > > >
> > > > As others have said, this is an important capability for Apache
> Kafka.
> > It
> > > > is worth maintaining this as a part of the project.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > >
> > > > >I personally think it would be quite wasteful to re-implement the
> REST
> > > > >gateway just because that an actively-maintained piece of
> > > Apache-licensed
> > > > >software is not governed directly by the Apache Kafka community...
> > While
> > > > >kafka-rest repo is owned by Confluent, the contributors including
> the
> > > main
> > > > >one are also part of the Apache Kafka community, so there is a
> chance
> > > to
> > > > >work this out.
> > > > >
> > > > >However, there are two valid concerns here that could be addressed,
> > > around
> > > > >community and accessibility:
> > > > >>> What we are worried about is a project
> > > > >>> that's not maintained in a community. So the process of accepting
> > > > >>>patches
> > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > >>>community.
> > > > >>> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >and
> > > > >>> hence there is no security support either.
> > > > >
> > > > >This might be easy to fix. Maybe Confluent / kafka-rest community
> can
> > > > >clarify that - what is their contribution policy, dev style, roadmap
> > > etc.
> > > > >If they want, they can make an effort to encourage participation
> from
> > > > >people outside Confluent (easily accept contributions, invite
> external
> > > > >commiters or have open dev process similar to Apache Kafka etc), as
> > > there
> > > > >is definitely seems to be some interest on the list. That might
> clear
> > > the
> > > > >community concern and help kafka-rest project (but that is a
> > calculation
> > > > >Confluent will have to make).
> > > > >
> > > > >The other, independent, concern is that REST is something that is
> > > expected
> > > > >to be available out of the box with Kafka. I personally don't feel
> > > > >strongly
> > > > >about it (better use proper, efficient APIs from day one), though it
> > is
> > > > >definitely way smaller than adding a stream processing engine to the
> > > > >project :)
> > > > >Again,the kafka-rest "community" could take steps to make it even
> > easier
> > > > >to
> > > > >install, configure and run kafka-rest for new users on vanilla
> Apache
> > > > >Kafka
> > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > >contributions to that end), but that is up to them.
> > > > >Finally, if after the above steps were taken there would still a
> > strong
> > > > >desire to include a great rest gateway with Apache Kafka, I assume
> the
> > > > >community could hypothetically fork the existing kafka-rest into an
> > > Apache
> > > > >Kafka subproject and maintain it "within Apache" instead of
> > implementing
> > > > >it
> > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> > > > >happen
> > > > >without Confluent blessing, and I think that is likely much less
> > optimal
> > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > having
> > > a
> > > > >separate external community around kafka-rest.
> > > > >
> > > > >
> > > > >Just my two cents,
> > > > >
> > > > >
> > > > >Ofir Manor
> > > > >
> > > > >Co-Founder & CTO | Equalum
> > > > >
> > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> <+972%2054-780-1286> |
> > Email:
> > > > ofir.manor@equalum.io
> > > > >
> > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > > > >wrote:
> > > > >
> > > > >> Neha & Jay,
> > > > >> We did look at the open source alternatives. Our
> > > > >>concern
> > > > >> is what's the patch acceptance and adding features/ bug-fixes to
> the
> > > > >> existing project under a Github (although it's licensed under
> Apache
> > > > >>2.0).
> > > > >> It would be great if that project made available under Apache and
> > > > >>driven by
> > > > >> the community. Adding to the above, not all Kafka users are
> > > interested
> > > > >>in
> > > > >> using the Java client API, they would like to have simple REST API
> > > where
> > > > >> they can code against using any language. I do believe this adds
> > value
> > > > >>to
> > > > >> Apache Kafka in itself.
> > > > >>
> > > > >> "For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> reinventing the wheel. What I'm looking for is a detailed
> comparison
> > > of
> > > > >>the
> > > > >> gaps and why those can't be improved in the REST proxy that
> already
> > > > >>exists
> > > > >> and is actively maintained."
> > > > >>
> > > > >> We are not looking at this as NIH. What we are worried about is a
> > > > >>project
> > > > >> that's not maintained in a community. So the process of accepting
> > > > >>patches
> > > > >> and priorities is not clear, and it's not developed in Apache
> > > community.
> > > > >> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >>and
> > > > >> hence there is no security support either.
> > > > >> We don't know the timeline when that's made available. We would
> like
> > > to
> > > > >>add
> > > > >> admin functionality into the REST API. So the Roadmap of that
> > project
> > > is
> > > > >> not driven by Apache.
> > > > >>
> > > > >>
> > > > >> "This doesn't materially have an impact on expanding the usability
> > of
> > > > >> Kafka. In my experience, REST proxy + Java clients only cover
> > ~50%
> > > of
> > > > >> all
> > > > >> Kafka users, and maybe 10% of those are the ones who will use
> the
> > > > >>REST
> > > > >> proxy. The remaining 50% are non-java client users (C, python,
> > go,
> > > > >>node
> > > > >> etc)."
> > > > >>
> > > > >> REST API is most often asked feature in my interactions with Kafka
> > > > >>users.
> > > > >> In an organization, There will be independent teams who will write
> > > their
> > > > >> Kafka clients using different language libraries available today,
> > and
> > > > >> there is no way to standardize this. Instead of supporting several
> > > > >> different client libraries users will be interested in using a
> REST
> > > API
> > > > >> server. The need for a REST API will only increase as more and
> more
> > > > >>users
> > > > >> start using Kafka.
> > > > >>
> > > > >> "More surface area means more work to keep things consistent.
> > Failure
> > > > >> to do that has, in fact, hurt the user experience."
> > > > >> Having myriad Kafka client GitHub projects that support different
> > > > >>languages
> > > > >> hurts the user experience and pushes burden to maintain these
> > > libraries.
> > > > >> REST API is a simple code base that uses existing java client
> > > libraries
> > > > >>to
> > > > >> make life easier for the users.
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > > > wrote:
> > > > >>
> > > > >> > Manikumar,
> > > > >> >
> > > > >> > Thanks for sharing the proposal. I think there are 2 parts to
> this
> > > > >> > discussion -
> > > > >> >
> > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > >>feature-complete,
> > > > >> > open-source and actively maintained REST proxy in the community?
> > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile
> and
> > > > >> maintain
> > > > >> > the high-quality experience that Kafka users have today?
> > > > >> >
> > > > >> > For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > comparison
> > > > >>of
> > > > >> the
> > > > >> > gaps and why those can't be improved in the REST proxy that
> > already
> > > > >> exists
> > > > >> > and is actively maintained. For example, we depend on zkClient
> and
> > > > >>have
> > > > >> > found as well as fixed several bugs by working closely with the
> > > people
> > > > >> who
> > > > >> > maintain zkClient. This should be possible for REST proxy as
> well,
> > > > >>right?
> > > > >> >
> > > > >> > For 2, I'd like us to review our history of expanding the
> surface
> > > > >>area to
> > > > >> > add more clients in the past. Here is a summary -
> > > > >> >
> > > > >> > - This doesn't materially have an impact on expanding the
> > > > >>usability of
> > > > >> > Kafka. In my experience, REST proxy + Java clients only cover
> > > ~50%
> > > > >>of
> > > > >> > all
> > > > >> > Kafka users, and maybe 10% of those are the ones who will use
> > the
> > > > >>REST
> > > > >> > proxy. The remaining 50% are non-java client users (C,
> python,
> > > go,
> > > > >> node
> > > > >> > etc).
> > > > >> > - People are a lot more excited about promising to contribute
> > > while
> > > > >> > adding the surface area but not on an ongoing basis down the
> > > line.
> > > > >> > - More surface area means more work to keep things
> consistent.
> > > > >>Failure
> > > > >> > to do that has, in fact, hurt the user experience.
> > > > >> > - More surface area hurts agility. We want to do a few things
> > > > >>really
> > > > >> > well as well as be agile to be able to build on our core
> > > > >>competency.
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Jay,
> > > > >> > >
> > > > >> > > Thanks for your reply.
> > > > >> > >
> > > > >> > > I agree that we can not add all the clients/tools available in
> > > > >> ecosystem
> > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > different
> > > > >>from
> > > > >> > > other clients/tools. Since any language that can work with
> HTTP
> > > can
> > > > >> > > easily integrate with this interface, Having an "official"
> REST
> > > > >> > > interface helps user community. This also helps us to
> integrate
> > > well
> > > > >> > > with external management and provisioning tools. Apache Kafka
> > > > >>release
> > > > >> > > with Java clients + REST interface is sufficient for most of
> the
> > > > >>user
> > > > >> > > deployments/requirements. This helps users to deal with less
> > > number
> > > > >> > > of distributions/builds.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Manikumar
> > > > >> > >
> > > > >> > >
> > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> > > wrote:
> > > > >> > >
> > > > >> > > > Hey guys,
> > > > >> > > >
> > > > >> > > > There's already a REST interface maintained as a separate
> > > > >> project--it's
> > > > >> > > > open source and apache licensed and actively maintained (
> > > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong
> > with
> > > >
> > > >
> > > >
> > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > github.com
> > > > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
> > It
> > > > makes it easy to produce and consume messages, view the state of the
> > > > cluster, and ...
> > > >
> > > > >> that?
> > > > >> > > You
> > > > >> > > > mentioned that there was some compatibility concern, but
> > > > >> compatibility
> > > > >> > > has
> > > > >> > > > to do with the consumer protocol guarantees not the repo the
> > > code
> > > > >>is
> > > > >> > in,
> > > > >> > > > right? Not sure that concern makes sense.
> > > > >> > > >
> > > > >> > > > We could argue for adding pretty much anything and
> everything
> > in
> > > > >>the
> > > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> > make
> > > > >>the
> > > > >> > > project
> > > > >> > > > more agile.
> > > > >> > > >
> > > > >> > > > -Jay
> > > > >> > > >
> > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > >> manikumar.reddy@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Kafka Devs,
> > > > >> > > > >
> > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > Repository.
> > > > >> > > > >
> > > > >> > > > > There are already open-source alternatives are available.
> > But
> > > > >>we
> > > > >> > would
> > > > >> > > > > like to add REST server that
> > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> Infra
> > > > >>tools
> > > > >> > comes
> > > > >> > > > up
> > > > >> > > > > with Rest Interface.
> > > > >> > > > > It is useful to have inbuilt Rest API support for Produce,
> > > > >>Consume
> > > > >> > > > messages
> > > > >> > > > > and admin interface for
> > > > >> > > > > integrating with external management and provisioning
> > > tools.This
> > > > >> will
> > > > >> > > > also
> > > > >> > > > > allow the maintenance of
> > > > >> > > > > REST server and adding new features makes it easy because
> > > apache
> > > > >> > > > community.
> > > > >> > > > >
> > > > >> > > > > The KIP wiki is the following:
> > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > >> > > > >
> > > > >> > > > > Your comments and feedback are welcome.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Manikumar
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > --
> > > > >> > Thanks,
> > > > >> > Neha
> > > > >> >
> > > > >>
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Michael,
Can you please share which Apache projects have a MMC? I couldn't find
anything after a quick google.
Ismael
On Fri, Oct 21, 2016 at 7:28 AM, Michael Pearce <Mi...@ig.com>
wrote:
> So from my reading essentially the first question needs to answered/and
> voted on is:
>
> Is Apache Kafka Community only about the Core or does the apache community
> also support some subprojects (and just we need some better way to manage
> this)
>
> If vote for Core only wins, then the following should be removed:
> Kafka Connect
> Kafka Stream
>
> If vote for Core only loses (aka we will support subprojects) then:
> We should look to add Kafka Rest
>
> And we should look to see how we can manage better govern and manage
> submodules.
>
> A good example which id propose here is how some other communities in
> Apache do this.
>
> Each Module has a Module Management Committee(MMC), this is like almost
> the PMC but at a per module basis.
>
> This MMC should essentially hold the binding votes for that module.
> The MMC should be made up of a single representative from each
> organisation (so no single organisation can fully veto the community it
> has to a genuine consenus)
> The MMC requires at least 3 members (so there cant be a tied vote on 2)
> For a new Module to be added a MMC committee should be sought
> A new Module is only capable of being added if the above requirements can
> be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> only actively support modules would be added
>
> The PMC reviews each module every 6months or Year. If MMC is inactive, a
> vote/call to find replacements if raised, if none are forthcoming dropping
> the MMC to less than 3 then the module moves to "the attic" (very much like
> apache attic but a little more aggressively)
>
> This way the PMC does not need to micro manage every module
> We only add modules where some amount of active support and maintenance
> and use is provided by the community
> We have an automatic way to retire old or inactive projects.
>
> Thoughts?
> Mike
>
>
> ________________________________________
> From: Harsha Ch <ha...@gmail.com>
> Sent: Thursday, October 20, 2016 10:26 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> Jay,
> REST API is something every user is in need of. If the argument is to
> clone and write your API, this will do a disservice to the users as they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
>
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
>
> Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this REST
> server to Kafka will affect the agility of the project doesn't mak sense.
>
> It looks like your argument is either we develop all these small tools or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka could
> have been left in their own Github projects based on your arguments. What
> about the REST API is so different that such that it should stay out of the
> Kafka project? From my experience, more users are asking for the REST API.
>
> Thanks,
> Harsha
>
>
>
>
>
> On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
>
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not really
> > make sense to me. If in the future we disagree, that is the beauty of
> open
> > source, you can always fork off a copy of the code and start an
> independent
> > project either in Apache or elsewhere. Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be done
> > and we have an existing code base for HTTP/kafka access that is heavily
> > used in production seems quite silly.
> >
> > Let me give some background on how I at least think about these things.
> > I've participated in open source projects out of LinkedIn via github as
> > well as via the ASF. I don't think there is a "right" answer to how to do
> > these but rather some tradeoffs. We thought about this quite a lot in the
> > context of Kafka based on the experience with the Hadoop ecosystem as
> well
> > as from other open source communities.
> >
> > There is a rich ecosystem around Kafka. Many of the projects are quite
> > small--single clients or tools that do a single thing well--and almost
> none
> > of them are top level apache projects. I don't think trying to force each
> > of these to turn into independent Apache projects is necessarily the best
> > thing for the ecosystem.
> >
> > My observation of how this can go wrong is really what I think has
> happened
> > to the Hadoop ecosystem. There you see quite a zoo of projects which all
> > drift apart and don't quite work together well. Coordinating even simple
> > changes and standardization across these is exceptionally difficult. The
> > result is a bit of a mess for users--the pieces just don't really come
> > together very well. This makes sense for independent infrastructure
> systems
> > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > little tool or helper library has lead to a desirable state. I think the
> > mode of operating where the Hadoop vendors spawn off a few new Apache
> > projects for each new product initiative, especially since often that
> > project is only valued by that vendor (and the other vendor has a
> different
> > competing Apache project) doesn't necessarily do a better job at
> producing
> > high quality communities or high quality software.
> >
> > These tools/connects/clients/proxies and other integration pieces can
> take
> > many forms, but my take of what makes one of these things good is that it
> > remains simple, does its one thing well, and cleaves as closely as
> possible
> > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > monitoring, configuring, etc. For the tools we've contributed we've tried
> > really hard to make them consistent with Kafka as well as with each other
> > in how testing, configuration, monitoring, etc works.
> >
> > I think what Apache does superbly well is create a community for
> managing a
> > large infrastructure layer like Kafka in a vendor independent way. What I
> > think is less successful is attempting to form full and independent
> apache
> > communities around very simple single purpose tools, especially if you
> hope
> > for these to come together into a cohesive toolset across multiple such
> > tools. Much of what Apache does--create a collective decision making
> > process for resolving disagreement, help to trademark and protect the
> marks
> > of the project, etc just isn't that relevant for simple single-purpose
> > tools.
> >
> > So my take is there are a couple of options:
> >
> > 1. We can try to put all the small tools into the Apache Project. I
> > think this is not the right approach as there is simply too many of
> > them,
> > many in different languages, serving different protocols, integrating
> > with
> > particular systems, and a single community can't effectively maintain
> > them
> > all. Doing this would significantly slow the progress of the Kafka
> > project.
> > As a protocol for messaging, I don't really see a case for including
> > REST
> > but not MQTT or AMQP which are technically much better suited to
> > messaging
> > and both are widely used for that.
> > 2. We can treat ecosystem projects that aren't top level Apache
> projects
> > as invalid and try to recreate them all as Apache projects. Honestly,
> > though, if you go to the Kafka ecosystem page virtually none of the
> most
> > popular add-ons to Kafka are Apache projects. The most successful
> > things in
> > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> other
> > clients, as well as the existing REST layer have succeeded at
> developing
> > communities that actively contribute and use these pieces and I don't
> > know
> > that that is a bad thing unless that community proves to be
> uninclusive,
> > unresponsive, or goes in a bad technical direction--and those are
> > failure
> > modes that all open source efforts face.
> > 3. We can do what I think makes the most sense and try to work with
> the
> > projects that exist in the ecosystem and if the project doesn't have a
> > responsive community or wants to go in a different direction fork or
> > recreate that work.
> >
> > Of course any person can choose whatever of these options they want. But
> > from my point of view, option (3) has been the path of the community so
> far
> > and I think it has been quite successful.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Neha,
> > > "But I haven't seen any community emails or patches being submitted by
> > you
> > > guys, so I'm wondering why you are concerned about whether the
> community
> > is
> > > open to accepting patches or not."
> > >
> > > I think you are talking about contributing patches to this repository
> > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > the guidelines/governance model is not clear on the project and I guess
> > its
> > > driven by opening a github issue request. Its the repository owned by
> > > confluent and as much I appreciate that the features we mentioned are
> in
> > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > gurantee what we want to add in the furture will be in your roadmap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot as
> > other
> > > users can participate in the discussions. We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project time
> > > within
> > > > my internal projects to developers contributing to another commercial
> > > > companies project even so technically open source, as then there is
> > that
> > > > commercial companies interest will always prevail and essentially can
> > > > always have a final vote where disagreement. Im sure they never
> intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint on
> > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages
> in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
> more
> > > > surface area for every client. But something like REST is universal
> and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > ________________________________________
> > > > From: Andrew Schofield <an...@outlook.com>
> > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > There's a massive difference between the governance of Kafka and the
> > > > governance of the REST proxy.
> > > >
> > > > In Kafka, there is a broad community of people contributing their
> > > opinions
> > > > about future enhancements in the form of KIPs. There's some really
> deep
> > > > consideration that goes into some of the trickier KIPs. There are
> > people
> > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > reputations to become committers. I get the impression that the
> roadmap
> > > of
> > > > Kafka is not really community-owned (what's the big feature for Kafka
> > > 0.11,
> > > > for example), but the conveyor belt of smaller features in the form
> of
> > > KIPs
> > > > works nicely. It's a good example of open-source working well.
> > > >
> > > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > > roadmap is less clear. There's not really a community properly
> engaged
> > in
> > > > the way that there is with Kafka. So, you could say that it's clear
> > that
> > > > fewer people are interested, but I think the whole governance thing
> > is a
> > > > big barrier to engagement. And it's looking like it's getting out of
> > > date.
> > > >
> > > > In technical terms, I can think of two big improvements to the REST
> > > proxy.
> > > > First, it needs to use the new consumer API so that it's possible to
> > > secure
> > > > connections between the REST proxy and Kafka. I don't care too much
> > which
> > > > method calls it uses actually uses to consume messages, but I do
> care
> > > that
> > > > I cannot secure connections because of network security rules.
> Second,
> > > > there's an affinity between a consumer and the instance of the REST
> > proxy
> > > > to which it first connected. Kafka itself avoids this kind of
> affinity
> > > for
> > > > good reason, and in the name of availability the REST proxy should
> too.
> > > > These are natural KIPs.
> > > >
> > > > I think it would be good to have the code for the REST proxy
> > contributed
> > > > to Apache Kafka so that it would be able to be developed in the same
> > way.
> > > >
> > > > Andrew Schofield
> > > >
> > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > Sent: 07 October 2016 22:41:52
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > ASF already gives us a clear framework and governance model for
> > community
> > > > development. This is already understood by the people contributing to
> > > > Apache Kafka project, and they are the same people who want to
> > contribute
> > > > to the REST server capability as well. Everyone is in agreement on
> the
> > > > need for collaborating on this effort. So why not contribute the code
> > to
> > > > Apache Kafka. This will help avoid duplication of effort and forks
> that
> > > > may crop up, hugely benefitting the user community. This will also
> > avoid
> > > > having to define a process similar to ASF on a GitHub project and
> > instead
> > > > there is a single community with clear understanding community
> process
> > as
> > > > defined in ASF.
> > > >
> > > > As others have said, this is an important capability for Apache
> Kafka.
> > It
> > > > is worth maintaining this as a part of the project.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > >
> > > > >I personally think it would be quite wasteful to re-implement the
> REST
> > > > >gateway just because that an actively-maintained piece of
> > > Apache-licensed
> > > > >software is not governed directly by the Apache Kafka community...
> > While
> > > > >kafka-rest repo is owned by Confluent, the contributors including
> the
> > > main
> > > > >one are also part of the Apache Kafka community, so there is a
> chance
> > > to
> > > > >work this out.
> > > > >
> > > > >However, there are two valid concerns here that could be addressed,
> > > around
> > > > >community and accessibility:
> > > > >>> What we are worried about is a project
> > > > >>> that's not maintained in a community. So the process of accepting
> > > > >>>patches
> > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > >>>community.
> > > > >>> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >and
> > > > >>> hence there is no security support either.
> > > > >
> > > > >This might be easy to fix. Maybe Confluent / kafka-rest community
> can
> > > > >clarify that - what is their contribution policy, dev style, roadmap
> > > etc.
> > > > >If they want, they can make an effort to encourage participation
> from
> > > > >people outside Confluent (easily accept contributions, invite
> external
> > > > >commiters or have open dev process similar to Apache Kafka etc), as
> > > there
> > > > >is definitely seems to be some interest on the list. That might
> clear
> > > the
> > > > >community concern and help kafka-rest project (but that is a
> > calculation
> > > > >Confluent will have to make).
> > > > >
> > > > >The other, independent, concern is that REST is something that is
> > > expected
> > > > >to be available out of the box with Kafka. I personally don't feel
> > > > >strongly
> > > > >about it (better use proper, efficient APIs from day one), though it
> > is
> > > > >definitely way smaller than adding a stream processing engine to the
> > > > >project :)
> > > > >Again,the kafka-rest "community" could take steps to make it even
> > easier
> > > > >to
> > > > >install, configure and run kafka-rest for new users on vanilla
> Apache
> > > > >Kafka
> > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > >contributions to that end), but that is up to them.
> > > > >Finally, if after the above steps were taken there would still a
> > strong
> > > > >desire to include a great rest gateway with Apache Kafka, I assume
> the
> > > > >community could hypothetically fork the existing kafka-rest into an
> > > Apache
> > > > >Kafka subproject and maintain it "within Apache" instead of
> > implementing
> > > > >it
> > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> > > > >happen
> > > > >without Confluent blessing, and I think that is likely much less
> > optimal
> > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > having
> > > a
> > > > >separate external community around kafka-rest.
> > > > >
> > > > >
> > > > >Just my two cents,
> > > > >
> > > > >
> > > > >Ofir Manor
> > > > >
> > > > >Co-Founder & CTO | Equalum
> > > > >
> > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> <+972%2054-780-1286> |
> > Email:
> > > > ofir.manor@equalum.io
> > > > >
> > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > > > >wrote:
> > > > >
> > > > >> Neha & Jay,
> > > > >> We did look at the open source alternatives. Our
> > > > >>concern
> > > > >> is what's the patch acceptance and adding features/ bug-fixes to
> the
> > > > >> existing project under a Github (although it's licensed under
> Apache
> > > > >>2.0).
> > > > >> It would be great if that project made available under Apache and
> > > > >>driven by
> > > > >> the community. Adding to the above, not all Kafka users are
> > > interested
> > > > >>in
> > > > >> using the Java client API, they would like to have simple REST API
> > > where
> > > > >> they can code against using any language. I do believe this adds
> > value
> > > > >>to
> > > > >> Apache Kafka in itself.
> > > > >>
> > > > >> "For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> reinventing the wheel. What I'm looking for is a detailed
> comparison
> > > of
> > > > >>the
> > > > >> gaps and why those can't be improved in the REST proxy that
> already
> > > > >>exists
> > > > >> and is actively maintained."
> > > > >>
> > > > >> We are not looking at this as NIH. What we are worried about is a
> > > > >>project
> > > > >> that's not maintained in a community. So the process of accepting
> > > > >>patches
> > > > >> and priorities is not clear, and it's not developed in Apache
> > > community.
> > > > >> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >>and
> > > > >> hence there is no security support either.
> > > > >> We don't know the timeline when that's made available. We would
> like
> > > to
> > > > >>add
> > > > >> admin functionality into the REST API. So the Roadmap of that
> > project
> > > is
> > > > >> not driven by Apache.
> > > > >>
> > > > >>
> > > > >> "This doesn't materially have an impact on expanding the usability
> > of
> > > > >> Kafka. In my experience, REST proxy + Java clients only cover
> > ~50%
> > > of
> > > > >> all
> > > > >> Kafka users, and maybe 10% of those are the ones who will use
> the
> > > > >>REST
> > > > >> proxy. The remaining 50% are non-java client users (C, python,
> > go,
> > > > >>node
> > > > >> etc)."
> > > > >>
> > > > >> REST API is most often asked feature in my interactions with Kafka
> > > > >>users.
> > > > >> In an organization, There will be independent teams who will write
> > > their
> > > > >> Kafka clients using different language libraries available today,
> > and
> > > > >> there is no way to standardize this. Instead of supporting several
> > > > >> different client libraries users will be interested in using a
> REST
> > > API
> > > > >> server. The need for a REST API will only increase as more and
> more
> > > > >>users
> > > > >> start using Kafka.
> > > > >>
> > > > >> "More surface area means more work to keep things consistent.
> > Failure
> > > > >> to do that has, in fact, hurt the user experience."
> > > > >> Having myriad Kafka client GitHub projects that support different
> > > > >>languages
> > > > >> hurts the user experience and pushes burden to maintain these
> > > libraries.
> > > > >> REST API is a simple code base that uses existing java client
> > > libraries
> > > > >>to
> > > > >> make life easier for the users.
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > > > wrote:
> > > > >>
> > > > >> > Manikumar,
> > > > >> >
> > > > >> > Thanks for sharing the proposal. I think there are 2 parts to
> this
> > > > >> > discussion -
> > > > >> >
> > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > >>feature-complete,
> > > > >> > open-source and actively maintained REST proxy in the community?
> > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile
> and
> > > > >> maintain
> > > > >> > the high-quality experience that Kafka users have today?
> > > > >> >
> > > > >> > For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > comparison
> > > > >>of
> > > > >> the
> > > > >> > gaps and why those can't be improved in the REST proxy that
> > already
> > > > >> exists
> > > > >> > and is actively maintained. For example, we depend on zkClient
> and
> > > > >>have
> > > > >> > found as well as fixed several bugs by working closely with the
> > > people
> > > > >> who
> > > > >> > maintain zkClient. This should be possible for REST proxy as
> well,
> > > > >>right?
> > > > >> >
> > > > >> > For 2, I'd like us to review our history of expanding the
> surface
> > > > >>area to
> > > > >> > add more clients in the past. Here is a summary -
> > > > >> >
> > > > >> > - This doesn't materially have an impact on expanding the
> > > > >>usability of
> > > > >> > Kafka. In my experience, REST proxy + Java clients only cover
> > > ~50%
> > > > >>of
> > > > >> > all
> > > > >> > Kafka users, and maybe 10% of those are the ones who will use
> > the
> > > > >>REST
> > > > >> > proxy. The remaining 50% are non-java client users (C,
> python,
> > > go,
> > > > >> node
> > > > >> > etc).
> > > > >> > - People are a lot more excited about promising to contribute
> > > while
> > > > >> > adding the surface area but not on an ongoing basis down the
> > > line.
> > > > >> > - More surface area means more work to keep things
> consistent.
> > > > >>Failure
> > > > >> > to do that has, in fact, hurt the user experience.
> > > > >> > - More surface area hurts agility. We want to do a few things
> > > > >>really
> > > > >> > well as well as be agile to be able to build on our core
> > > > >>competency.
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Jay,
> > > > >> > >
> > > > >> > > Thanks for your reply.
> > > > >> > >
> > > > >> > > I agree that we can not add all the clients/tools available in
> > > > >> ecosystem
> > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > different
> > > > >>from
> > > > >> > > other clients/tools. Since any language that can work with
> HTTP
> > > can
> > > > >> > > easily integrate with this interface, Having an "official"
> REST
> > > > >> > > interface helps user community. This also helps us to
> integrate
> > > well
> > > > >> > > with external management and provisioning tools. Apache Kafka
> > > > >>release
> > > > >> > > with Java clients + REST interface is sufficient for most of
> the
> > > > >>user
> > > > >> > > deployments/requirements. This helps users to deal with less
> > > number
> > > > >> > > of distributions/builds.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Manikumar
> > > > >> > >
> > > > >> > >
> > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> > > wrote:
> > > > >> > >
> > > > >> > > > Hey guys,
> > > > >> > > >
> > > > >> > > > There's already a REST interface maintained as a separate
> > > > >> project--it's
> > > > >> > > > open source and apache licensed and actively maintained (
> > > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong
> > with
> > > >
> > > >
> > > >
> > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > github.com
> > > > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
> > It
> > > > makes it easy to produce and consume messages, view the state of the
> > > > cluster, and ...
> > > >
> > > > >> that?
> > > > >> > > You
> > > > >> > > > mentioned that there was some compatibility concern, but
> > > > >> compatibility
> > > > >> > > has
> > > > >> > > > to do with the consumer protocol guarantees not the repo the
> > > code
> > > > >>is
> > > > >> > in,
> > > > >> > > > right? Not sure that concern makes sense.
> > > > >> > > >
> > > > >> > > > We could argue for adding pretty much anything and
> everything
> > in
> > > > >>the
> > > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> > make
> > > > >>the
> > > > >> > > project
> > > > >> > > > more agile.
> > > > >> > > >
> > > > >> > > > -Jay
> > > > >> > > >
> > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > >> manikumar.reddy@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Kafka Devs,
> > > > >> > > > >
> > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > Repository.
> > > > >> > > > >
> > > > >> > > > > There are already open-source alternatives are available.
> > But
> > > > >>we
> > > > >> > would
> > > > >> > > > > like to add REST server that
> > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> Infra
> > > > >>tools
> > > > >> > comes
> > > > >> > > > up
> > > > >> > > > > with Rest Interface.
> > > > >> > > > > It is useful to have inbuilt Rest API support for Produce,
> > > > >>Consume
> > > > >> > > > messages
> > > > >> > > > > and admin interface for
> > > > >> > > > > integrating with external management and provisioning
> > > tools.This
> > > > >> will
> > > > >> > > > also
> > > > >> > > > > allow the maintenance of
> > > > >> > > > > REST server and adding new features makes it easy because
> > > apache
> > > > >> > > > community.
> > > > >> > > > >
> > > > >> > > > > The KIP wiki is the following:
> > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > >> > > > >
> > > > >> > > > > Your comments and feedback are welcome.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Manikumar
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > --
> > > > >> > Thanks,
> > > > >> > Neha
> > > > >> >
> > > > >>
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Michael Pearce <Mi...@ig.com>.
So from my reading essentially the first question needs to answered/and voted on is:
Is Apache Kafka Community only about the Core or does the apache community also support some subprojects (and just we need some better way to manage this)
If vote for Core only wins, then the following should be removed:
Kafka Connect
Kafka Stream
If vote for Core only loses (aka we will support subprojects) then:
We should look to add Kafka Rest
And we should look to see how we can manage better govern and manage submodules.
A good example which id propose here is how some other communities in Apache do this.
Each Module has a Module Management Committee(MMC), this is like almost the PMC but at a per module basis.
This MMC should essentially hold the binding votes for that module.
The MMC should be made up of a single representative from each organisation (so no single organisation can fully veto the community it has to a genuine consenus)
The MMC requires at least 3 members (so there cant be a tied vote on 2)
For a new Module to be added a MMC committee should be sought
A new Module is only capable of being added if the above requirements can be met (e.g. 3 people wishing to step up, from 3 organisations) so that only actively support modules would be added
The PMC reviews each module every 6months or Year. If MMC is inactive, a vote/call to find replacements if raised, if none are forthcoming dropping the MMC to less than 3 then the module moves to "the attic" (very much like apache attic but a little more aggressively)
This way the PMC does not need to micro manage every module
We only add modules where some amount of active support and maintenance and use is provided by the community
We have an automatic way to retire old or inactive projects.
Thoughts?
Mike
________________________________________
From: Harsha Ch <ha...@gmail.com>
Sent: Thursday, October 20, 2016 10:26 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
Jay,
REST API is something every user is in need of. If the argument is to
clone and write your API, this will do a disservice to the users as they
now have to choose one vs. others instead of keeping one API that is
supported in Kafka community.
"Pre-emptively re-creating another
REST layer when it seems like we all quite agree on what needs to be done
and we have an existing code base for HTTP/Kafka access that is heavily
used in production seems quite silly."
Exactly our point. Why can't we develop this in Apache Kafka
community? Instead of us open sourcing another GitHub project and creating
a divide in users and another version of API. Let's build this in Kafka
Community and use the governance model that is proven to provide vendor
free user driven consensus features. The argument that is adding this REST
server to Kafka will affect the agility of the project doesn't mak sense.
It looks like your argument is either we develop all these small tools or
none at all. We as a community need to look at supporting critical
tools/API. Instead of dividing this project into individual external
communities. We should build this as part of Kafka which best serves the
needs of users.
The Streams and Connect projects that were pushed into Kafka could
have been left in their own Github projects based on your arguments. What
about the REST API is so different that such that it should stay out of the
Kafka project? From my experience, more users are asking for the REST API.
Thanks,
Harsha
On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
> I think the questions around governance make sense, I think we should
> really clarify that to make the process more clear so it can be fully
> inclusive.
>
> The idea that we should not collaborate on what is there now, though,
> because in the future we might disagree about direction does not really
> make sense to me. If in the future we disagree, that is the beauty of open
> source, you can always fork off a copy of the code and start an independent
> project either in Apache or elsewhere. Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/kafka access that is heavily
> used in production seems quite silly.
>
> Let me give some background on how I at least think about these things.
> I've participated in open source projects out of LinkedIn via github as
> well as via the ASF. I don't think there is a "right" answer to how to do
> these but rather some tradeoffs. We thought about this quite a lot in the
> context of Kafka based on the experience with the Hadoop ecosystem as well
> as from other open source communities.
>
> There is a rich ecosystem around Kafka. Many of the projects are quite
> small--single clients or tools that do a single thing well--and almost none
> of them are top level apache projects. I don't think trying to force each
> of these to turn into independent Apache projects is necessarily the best
> thing for the ecosystem.
>
> My observation of how this can go wrong is really what I think has happened
> to the Hadoop ecosystem. There you see quite a zoo of projects which all
> drift apart and don't quite work together well. Coordinating even simple
> changes and standardization across these is exceptionally difficult. The
> result is a bit of a mess for users--the pieces just don't really come
> together very well. This makes sense for independent infrastructure systems
> (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> little tool or helper library has lead to a desirable state. I think the
> mode of operating where the Hadoop vendors spawn off a few new Apache
> projects for each new product initiative, especially since often that
> project is only valued by that vendor (and the other vendor has a different
> competing Apache project) doesn't necessarily do a better job at producing
> high quality communities or high quality software.
>
> These tools/connects/clients/proxies and other integration pieces can take
> many forms, but my take of what makes one of these things good is that it
> remains simple, does its one thing well, and cleaves as closely as possible
> to the conventions for Kafka itself--i.e. doesn't invent new ways of
> monitoring, configuring, etc. For the tools we've contributed we've tried
> really hard to make them consistent with Kafka as well as with each other
> in how testing, configuration, monitoring, etc works.
>
> I think what Apache does superbly well is create a community for managing a
> large infrastructure layer like Kafka in a vendor independent way. What I
> think is less successful is attempting to form full and independent apache
> communities around very simple single purpose tools, especially if you hope
> for these to come together into a cohesive toolset across multiple such
> tools. Much of what Apache does--create a collective decision making
> process for resolving disagreement, help to trademark and protect the marks
> of the project, etc just isn't that relevant for simple single-purpose
> tools.
>
> So my take is there are a couple of options:
>
> 1. We can try to put all the small tools into the Apache Project. I
> think this is not the right approach as there is simply too many of
> them,
> many in different languages, serving different protocols, integrating
> with
> particular systems, and a single community can't effectively maintain
> them
> all. Doing this would significantly slow the progress of the Kafka
> project.
> As a protocol for messaging, I don't really see a case for including
> REST
> but not MQTT or AMQP which are technically much better suited to
> messaging
> and both are widely used for that.
> 2. We can treat ecosystem projects that aren't top level Apache projects
> as invalid and try to recreate them all as Apache projects. Honestly,
> though, if you go to the Kafka ecosystem page virtually none of the most
> popular add-ons to Kafka are Apache projects. The most successful
> things in
> the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of other
> clients, as well as the existing REST layer have succeeded at developing
> communities that actively contribute and use these pieces and I don't
> know
> that that is a bad thing unless that community proves to be uninclusive,
> unresponsive, or goes in a bad technical direction--and those are
> failure
> modes that all open source efforts face.
> 3. We can do what I think makes the most sense and try to work with the
> projects that exist in the ecosystem and if the project doesn't have a
> responsive community or wants to go in a different direction fork or
> recreate that work.
>
> Of course any person can choose whatever of these options they want. But
> from my point of view, option (3) has been the path of the community so far
> and I think it has been quite successful.
>
> -Jay
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Neha,
> > "But I haven't seen any community emails or patches being submitted by
> you
> > guys, so I'm wondering why you are concerned about whether the community
> is
> > open to accepting patches or not."
> >
> > I think you are talking about contributing patches to this repository
> > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > the guidelines/governance model is not clear on the project and I guess
> its
> > driven by opening a github issue request. Its the repository owned by
> > confluent and as much I appreciate that the features we mentioned are in
> > the roadmap and welcoming us to contribute to the project. It doesn't
> > gurantee what we want to add in the furture will be in your roadmap.
> >
> > Hence the reason having it part of Kafka community will help a lot as
> other
> > users can participate in the discussions. We are happy to drive any
> > feature additions through KIPs which gives everyone a chance to
> participate
> > and add to the discussions.
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > +1
> > >
> > > I agree on the governance comments whole heartedly.
> > >
> > > Also i agree about the contribution comments made earlier in the
> thread,
> > i
> > > personally am less likely to spend any of mine, or give project time
> > within
> > > my internal projects to developers contributing to another commercial
> > > companies project even so technically open source, as then there is
> that
> > > commercial companies interest will always prevail and essentially can
> > > always have a final vote where disagreement. Im sure they never intend
> > to,
> > > but there is that true reality. This is why we have community open
> source
> > > projects.
> > >
> > > I can find many different implementations now of a rest endpoint on
> > > GitHub, BitBucket etc. Each one has their benefits and disadvantages in
> > > their implementation. By making / providing one this would bring
> together
> > > these solutions, unifying those developers and also bringing the best
> of
> > > all.
> > >
> > > I understand the concern on the community burden adding/supporting more
> > > surface area for every client. But something like REST is universal and
> > > worthy to be owned by the community.
> > >
> > > Mike
> > >
> > >
> > > ________________________________________
> > > From: Andrew Schofield <an...@outlook.com>
> > > Sent: Saturday, October 8, 2016 1:19 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > There's a massive difference between the governance of Kafka and the
> > > governance of the REST proxy.
> > >
> > > In Kafka, there is a broad community of people contributing their
> > opinions
> > > about future enhancements in the form of KIPs. There's some really deep
> > > consideration that goes into some of the trickier KIPs. There are
> people
> > > outside Confluent deeply knowledgeable in Kafka and building the
> > > reputations to become committers. I get the impression that the roadmap
> > of
> > > Kafka is not really community-owned (what's the big feature for Kafka
> > 0.11,
> > > for example), but the conveyor belt of smaller features in the form of
> > KIPs
> > > works nicely. It's a good example of open-source working well.
> > >
> > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > roadmap is less clear. There's not really a community properly engaged
> in
> > > the way that there is with Kafka. So, you could say that it's clear
> that
> > > fewer people are interested, but I think the whole governance thing
> is a
> > > big barrier to engagement. And it's looking like it's getting out of
> > date.
> > >
> > > In technical terms, I can think of two big improvements to the REST
> > proxy.
> > > First, it needs to use the new consumer API so that it's possible to
> > secure
> > > connections between the REST proxy and Kafka. I don't care too much
> which
> > > method calls it uses actually uses to consume messages, but I do care
> > that
> > > I cannot secure connections because of network security rules. Second,
> > > there's an affinity between a consumer and the instance of the REST
> proxy
> > > to which it first connected. Kafka itself avoids this kind of affinity
> > for
> > > good reason, and in the name of availability the REST proxy should too.
> > > These are natural KIPs.
> > >
> > > I think it would be good to have the code for the REST proxy
> contributed
> > > to Apache Kafka so that it would be able to be developed in the same
> way.
> > >
> > > Andrew Schofield
> > >
> > > From: Suresh Srinivas <su...@hortonworks.com>
> > > Sent: 07 October 2016 22:41:52
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > ASF already gives us a clear framework and governance model for
> community
> > > development. This is already understood by the people contributing to
> > > Apache Kafka project, and they are the same people who want to
> contribute
> > > to the REST server capability as well. Everyone is in agreement on the
> > > need for collaborating on this effort. So why not contribute the code
> to
> > > Apache Kafka. This will help avoid duplication of effort and forks that
> > > may crop up, hugely benefitting the user community. This will also
> avoid
> > > having to define a process similar to ASF on a GitHub project and
> instead
> > > there is a single community with clear understanding community process
> as
> > > defined in ASF.
> > >
> > > As others have said, this is an important capability for Apache Kafka.
> It
> > > is worth maintaining this as a part of the project.
> > >
> > > Regards,
> > > Suresh
> > >
> > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > >
> > > >I personally think it would be quite wasteful to re-implement the REST
> > > >gateway just because that an actively-maintained piece of
> > Apache-licensed
> > > >software is not governed directly by the Apache Kafka community...
> While
> > > >kafka-rest repo is owned by Confluent, the contributors including the
> > main
> > > >one are also part of the Apache Kafka community, so there is a chance
> > to
> > > >work this out.
> > > >
> > > >However, there are two valid concerns here that could be addressed,
> > around
> > > >community and accessibility:
> > > >>> What we are worried about is a project
> > > >>> that's not maintained in a community. So the process of accepting
> > > >>>patches
> > > >>> and priorities is not clear, and it's not developed in Apache
> > > >>>community.
> > > >>> Not only that, existing REST API project doesn't support new client
> > API
> > > >and
> > > >>> hence there is no security support either.
> > > >
> > > >This might be easy to fix. Maybe Confluent / kafka-rest community can
> > > >clarify that - what is their contribution policy, dev style, roadmap
> > etc.
> > > >If they want, they can make an effort to encourage participation from
> > > >people outside Confluent (easily accept contributions, invite external
> > > >commiters or have open dev process similar to Apache Kafka etc), as
> > there
> > > >is definitely seems to be some interest on the list. That might clear
> > the
> > > >community concern and help kafka-rest project (but that is a
> calculation
> > > >Confluent will have to make).
> > > >
> > > >The other, independent, concern is that REST is something that is
> > expected
> > > >to be available out of the box with Kafka. I personally don't feel
> > > >strongly
> > > >about it (better use proper, efficient APIs from day one), though it
> is
> > > >definitely way smaller than adding a stream processing engine to the
> > > >project :)
> > > >Again,the kafka-rest "community" could take steps to make it even
> easier
> > > >to
> > > >install, configure and run kafka-rest for new users on vanilla Apache
> > > >Kafka
> > > >(outside the Confluent platform), if they wish that (or welcome
> > > >contributions to that end), but that is up to them.
> > > >Finally, if after the above steps were taken there would still a
> strong
> > > >desire to include a great rest gateway with Apache Kafka, I assume the
> > > >community could hypothetically fork the existing kafka-rest into an
> > Apache
> > > >Kafka subproject and maintain it "within Apache" instead of
> implementing
> > > >it
> > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> > > >happen
> > > >without Confluent blessing, and I think that is likely much less
> optimal
> > > >(pulling in other Confluent / Apache licensed dependencies) than
> having
> > a
> > > >separate external community around kafka-rest.
> > > >
> > > >
> > > >Just my two cents,
> > > >
> > > >
> > > >Ofir Manor
> > > >
> > > >Co-Founder & CTO | Equalum
> > > >
> > > >Mobile: +972-54-7801286 <+972%2054-780-1286> <+972%2054-780-1286> |
> Email:
> > > ofir.manor@equalum.io
> > > >
> > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <kafka@harsha.io
> >
> > > >wrote:
> > > >
> > > >> Neha & Jay,
> > > >> We did look at the open source alternatives. Our
> > > >>concern
> > > >> is what's the patch acceptance and adding features/ bug-fixes to the
> > > >> existing project under a Github (although it's licensed under Apache
> > > >>2.0).
> > > >> It would be great if that project made available under Apache and
> > > >>driven by
> > > >> the community. Adding to the above, not all Kafka users are
> > interested
> > > >>in
> > > >> using the Java client API, they would like to have simple REST API
> > where
> > > >> they can code against using any language. I do believe this adds
> value
> > > >>to
> > > >> Apache Kafka in itself.
> > > >>
> > > >> "For 1, I don't think there is value in giving in to the NIH
> syndrome
> > > >>and
> > > >> reinventing the wheel. What I'm looking for is a detailed comparison
> > of
> > > >>the
> > > >> gaps and why those can't be improved in the REST proxy that already
> > > >>exists
> > > >> and is actively maintained."
> > > >>
> > > >> We are not looking at this as NIH. What we are worried about is a
> > > >>project
> > > >> that's not maintained in a community. So the process of accepting
> > > >>patches
> > > >> and priorities is not clear, and it's not developed in Apache
> > community.
> > > >> Not only that, existing REST API project doesn't support new client
> > API
> > > >>and
> > > >> hence there is no security support either.
> > > >> We don't know the timeline when that's made available. We would like
> > to
> > > >>add
> > > >> admin functionality into the REST API. So the Roadmap of that
> project
> > is
> > > >> not driven by Apache.
> > > >>
> > > >>
> > > >> "This doesn't materially have an impact on expanding the usability
> of
> > > >> Kafka. In my experience, REST proxy + Java clients only cover
> ~50%
> > of
> > > >> all
> > > >> Kafka users, and maybe 10% of those are the ones who will use the
> > > >>REST
> > > >> proxy. The remaining 50% are non-java client users (C, python,
> go,
> > > >>node
> > > >> etc)."
> > > >>
> > > >> REST API is most often asked feature in my interactions with Kafka
> > > >>users.
> > > >> In an organization, There will be independent teams who will write
> > their
> > > >> Kafka clients using different language libraries available today,
> and
> > > >> there is no way to standardize this. Instead of supporting several
> > > >> different client libraries users will be interested in using a REST
> > API
> > > >> server. The need for a REST API will only increase as more and more
> > > >>users
> > > >> start using Kafka.
> > > >>
> > > >> "More surface area means more work to keep things consistent.
> Failure
> > > >> to do that has, in fact, hurt the user experience."
> > > >> Having myriad Kafka client GitHub projects that support different
> > > >>languages
> > > >> hurts the user experience and pushes burden to maintain these
> > libraries.
> > > >> REST API is a simple code base that uses existing java client
> > libraries
> > > >>to
> > > >> make life easier for the users.
> > > >>
> > > >> Thanks,
> > > >> Harsha
> > > >>
> > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > > wrote:
> > > >>
> > > >> > Manikumar,
> > > >> >
> > > >> > Thanks for sharing the proposal. I think there are 2 parts to this
> > > >> > discussion -
> > > >> >
> > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > >>feature-complete,
> > > >> > open-source and actively maintained REST proxy in the community?
> > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> > > >> maintain
> > > >> > the high-quality experience that Kafka users have today?
> > > >> >
> > > >> > For 1, I don't think there is value in giving in to the NIH
> syndrome
> > > >>and
> > > >> > reinventing the wheel. What I'm looking for is a detailed
> comparison
> > > >>of
> > > >> the
> > > >> > gaps and why those can't be improved in the REST proxy that
> already
> > > >> exists
> > > >> > and is actively maintained. For example, we depend on zkClient and
> > > >>have
> > > >> > found as well as fixed several bugs by working closely with the
> > people
> > > >> who
> > > >> > maintain zkClient. This should be possible for REST proxy as well,
> > > >>right?
> > > >> >
> > > >> > For 2, I'd like us to review our history of expanding the surface
> > > >>area to
> > > >> > add more clients in the past. Here is a summary -
> > > >> >
> > > >> > - This doesn't materially have an impact on expanding the
> > > >>usability of
> > > >> > Kafka. In my experience, REST proxy + Java clients only cover
> > ~50%
> > > >>of
> > > >> > all
> > > >> > Kafka users, and maybe 10% of those are the ones who will use
> the
> > > >>REST
> > > >> > proxy. The remaining 50% are non-java client users (C, python,
> > go,
> > > >> node
> > > >> > etc).
> > > >> > - People are a lot more excited about promising to contribute
> > while
> > > >> > adding the surface area but not on an ongoing basis down the
> > line.
> > > >> > - More surface area means more work to keep things consistent.
> > > >>Failure
> > > >> > to do that has, in fact, hurt the user experience.
> > > >> > - More surface area hurts agility. We want to do a few things
> > > >>really
> > > >> > well as well as be agile to be able to build on our core
> > > >>competency.
> > > >> >
> > > >> >
> > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> manikumar.reddy@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Jay,
> > > >> > >
> > > >> > > Thanks for your reply.
> > > >> > >
> > > >> > > I agree that we can not add all the clients/tools available in
> > > >> ecosystem
> > > >> > > page to Kafka repo itself. But we feel REST Interface is
> different
> > > >>from
> > > >> > > other clients/tools. Since any language that can work with HTTP
> > can
> > > >> > > easily integrate with this interface, Having an "official" REST
> > > >> > > interface helps user community. This also helps us to integrate
> > well
> > > >> > > with external management and provisioning tools. Apache Kafka
> > > >>release
> > > >> > > with Java clients + REST interface is sufficient for most of the
> > > >>user
> > > >> > > deployments/requirements. This helps users to deal with less
> > number
> > > >> > > of distributions/builds.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Manikumar
> > > >> > >
> > > >> > >
> > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> > wrote:
> > > >> > >
> > > >> > > > Hey guys,
> > > >> > > >
> > > >> > > > There's already a REST interface maintained as a separate
> > > >> project--it's
> > > >> > > > open source and apache licensed and actively maintained (
> > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong
> with
> > >
> > >
> > >
> > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > github.com
> > > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
> It
> > > makes it easy to produce and consume messages, view the state of the
> > > cluster, and ...
> > >
> > > >> that?
> > > >> > > You
> > > >> > > > mentioned that there was some compatibility concern, but
> > > >> compatibility
> > > >> > > has
> > > >> > > > to do with the consumer protocol guarantees not the repo the
> > code
> > > >>is
> > > >> > in,
> > > >> > > > right? Not sure that concern makes sense.
> > > >> > > >
> > > >> > > > We could argue for adding pretty much anything and everything
> in
> > > >>the
> > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> make
> > > >>the
> > > >> > > project
> > > >> > > > more agile.
> > > >> > > >
> > > >> > > > -Jay
> > > >> > > >
> > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > >> manikumar.reddy@gmail.com
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Kafka Devs,
> > > >> > > > >
> > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> Repository.
> > > >> > > > >
> > > >> > > > > There are already open-source alternatives are available.
> But
> > > >>we
> > > >> > would
> > > >> > > > > like to add REST server that
> > > >> > > > > many users ask for under Apache Kafka repo. Many data Infra
> > > >>tools
> > > >> > comes
> > > >> > > > up
> > > >> > > > > with Rest Interface.
> > > >> > > > > It is useful to have inbuilt Rest API support for Produce,
> > > >>Consume
> > > >> > > > messages
> > > >> > > > > and admin interface for
> > > >> > > > > integrating with external management and provisioning
> > tools.This
> > > >> will
> > > >> > > > also
> > > >> > > > > allow the maintenance of
> > > >> > > > > REST server and adding new features makes it easy because
> > apache
> > > >> > > > community.
> > > >> > > > >
> > > >> > > > > The KIP wiki is the following:
> > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> > > > > 80%3A+Kafka+Rest+Server
> > > >> > > > >
> > > >> > > > > Your comments and feedback are welcome.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Manikumar
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> > --
> > > >> > Thanks,
> > > >> > Neha
> > > >> >
> > > >>
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Harsha Ch <ha...@gmail.com>.
Jay,
REST API is something every user is in need of. If the argument is to
clone and write your API, this will do a disservice to the users as they
now have to choose one vs. others instead of keeping one API that is
supported in Kafka community.
"Pre-emptively re-creating another
REST layer when it seems like we all quite agree on what needs to be done
and we have an existing code base for HTTP/Kafka access that is heavily
used in production seems quite silly."
Exactly our point. Why can't we develop this in Apache Kafka
community? Instead of us open sourcing another GitHub project and creating
a divide in users and another version of API. Let's build this in Kafka
Community and use the governance model that is proven to provide vendor
free user driven consensus features. The argument that is adding this REST
server to Kafka will affect the agility of the project doesn't mak sense.
It looks like your argument is either we develop all these small tools or
none at all. We as a community need to look at supporting critical
tools/API. Instead of dividing this project into individual external
communities. We should build this as part of Kafka which best serves the
needs of users.
The Streams and Connect projects that were pushed into Kafka could
have been left in their own Github projects based on your arguments. What
about the REST API is so different that such that it should stay out of the
Kafka project? From my experience, more users are asking for the REST API.
Thanks,
Harsha
On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <ja...@confluent.io> wrote:
> I think the questions around governance make sense, I think we should
> really clarify that to make the process more clear so it can be fully
> inclusive.
>
> The idea that we should not collaborate on what is there now, though,
> because in the future we might disagree about direction does not really
> make sense to me. If in the future we disagree, that is the beauty of open
> source, you can always fork off a copy of the code and start an independent
> project either in Apache or elsewhere. Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/kafka access that is heavily
> used in production seems quite silly.
>
> Let me give some background on how I at least think about these things.
> I've participated in open source projects out of LinkedIn via github as
> well as via the ASF. I don't think there is a "right" answer to how to do
> these but rather some tradeoffs. We thought about this quite a lot in the
> context of Kafka based on the experience with the Hadoop ecosystem as well
> as from other open source communities.
>
> There is a rich ecosystem around Kafka. Many of the projects are quite
> small--single clients or tools that do a single thing well--and almost none
> of them are top level apache projects. I don't think trying to force each
> of these to turn into independent Apache projects is necessarily the best
> thing for the ecosystem.
>
> My observation of how this can go wrong is really what I think has happened
> to the Hadoop ecosystem. There you see quite a zoo of projects which all
> drift apart and don't quite work together well. Coordinating even simple
> changes and standardization across these is exceptionally difficult. The
> result is a bit of a mess for users--the pieces just don't really come
> together very well. This makes sense for independent infrastructure systems
> (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> little tool or helper library has lead to a desirable state. I think the
> mode of operating where the Hadoop vendors spawn off a few new Apache
> projects for each new product initiative, especially since often that
> project is only valued by that vendor (and the other vendor has a different
> competing Apache project) doesn't necessarily do a better job at producing
> high quality communities or high quality software.
>
> These tools/connects/clients/proxies and other integration pieces can take
> many forms, but my take of what makes one of these things good is that it
> remains simple, does its one thing well, and cleaves as closely as possible
> to the conventions for Kafka itself--i.e. doesn't invent new ways of
> monitoring, configuring, etc. For the tools we've contributed we've tried
> really hard to make them consistent with Kafka as well as with each other
> in how testing, configuration, monitoring, etc works.
>
> I think what Apache does superbly well is create a community for managing a
> large infrastructure layer like Kafka in a vendor independent way. What I
> think is less successful is attempting to form full and independent apache
> communities around very simple single purpose tools, especially if you hope
> for these to come together into a cohesive toolset across multiple such
> tools. Much of what Apache does--create a collective decision making
> process for resolving disagreement, help to trademark and protect the marks
> of the project, etc just isn't that relevant for simple single-purpose
> tools.
>
> So my take is there are a couple of options:
>
> 1. We can try to put all the small tools into the Apache Project. I
> think this is not the right approach as there is simply too many of
> them,
> many in different languages, serving different protocols, integrating
> with
> particular systems, and a single community can't effectively maintain
> them
> all. Doing this would significantly slow the progress of the Kafka
> project.
> As a protocol for messaging, I don't really see a case for including
> REST
> but not MQTT or AMQP which are technically much better suited to
> messaging
> and both are widely used for that.
> 2. We can treat ecosystem projects that aren't top level Apache projects
> as invalid and try to recreate them all as Apache projects. Honestly,
> though, if you go to the Kafka ecosystem page virtually none of the most
> popular add-ons to Kafka are Apache projects. The most successful
> things in
> the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of other
> clients, as well as the existing REST layer have succeeded at developing
> communities that actively contribute and use these pieces and I don't
> know
> that that is a bad thing unless that community proves to be uninclusive,
> unresponsive, or goes in a bad technical direction--and those are
> failure
> modes that all open source efforts face.
> 3. We can do what I think makes the most sense and try to work with the
> projects that exist in the ecosystem and if the project doesn't have a
> responsive community or wants to go in a different direction fork or
> recreate that work.
>
> Of course any person can choose whatever of these options they want. But
> from my point of view, option (3) has been the path of the community so far
> and I think it has been quite successful.
>
> -Jay
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Neha,
> > "But I haven't seen any community emails or patches being submitted by
> you
> > guys, so I'm wondering why you are concerned about whether the community
> is
> > open to accepting patches or not."
> >
> > I think you are talking about contributing patches to this repository
> > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > the guidelines/governance model is not clear on the project and I guess
> its
> > driven by opening a github issue request. Its the repository owned by
> > confluent and as much I appreciate that the features we mentioned are in
> > the roadmap and welcoming us to contribute to the project. It doesn't
> > gurantee what we want to add in the furture will be in your roadmap.
> >
> > Hence the reason having it part of Kafka community will help a lot as
> other
> > users can participate in the discussions. We are happy to drive any
> > feature additions through KIPs which gives everyone a chance to
> participate
> > and add to the discussions.
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > +1
> > >
> > > I agree on the governance comments whole heartedly.
> > >
> > > Also i agree about the contribution comments made earlier in the
> thread,
> > i
> > > personally am less likely to spend any of mine, or give project time
> > within
> > > my internal projects to developers contributing to another commercial
> > > companies project even so technically open source, as then there is
> that
> > > commercial companies interest will always prevail and essentially can
> > > always have a final vote where disagreement. Im sure they never intend
> > to,
> > > but there is that true reality. This is why we have community open
> source
> > > projects.
> > >
> > > I can find many different implementations now of a rest endpoint on
> > > GitHub, BitBucket etc. Each one has their benefits and disadvantages in
> > > their implementation. By making / providing one this would bring
> together
> > > these solutions, unifying those developers and also bringing the best
> of
> > > all.
> > >
> > > I understand the concern on the community burden adding/supporting more
> > > surface area for every client. But something like REST is universal and
> > > worthy to be owned by the community.
> > >
> > > Mike
> > >
> > >
> > > ________________________________________
> > > From: Andrew Schofield <an...@outlook.com>
> > > Sent: Saturday, October 8, 2016 1:19 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > There's a massive difference between the governance of Kafka and the
> > > governance of the REST proxy.
> > >
> > > In Kafka, there is a broad community of people contributing their
> > opinions
> > > about future enhancements in the form of KIPs. There's some really deep
> > > consideration that goes into some of the trickier KIPs. There are
> people
> > > outside Confluent deeply knowledgeable in Kafka and building the
> > > reputations to become committers. I get the impression that the roadmap
> > of
> > > Kafka is not really community-owned (what's the big feature for Kafka
> > 0.11,
> > > for example), but the conveyor belt of smaller features in the form of
> > KIPs
> > > works nicely. It's a good example of open-source working well.
> > >
> > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > roadmap is less clear. There's not really a community properly engaged
> in
> > > the way that there is with Kafka. So, you could say that it's clear
> that
> > > fewer people are interested, but I think the whole governance thing
> is a
> > > big barrier to engagement. And it's looking like it's getting out of
> > date.
> > >
> > > In technical terms, I can think of two big improvements to the REST
> > proxy.
> > > First, it needs to use the new consumer API so that it's possible to
> > secure
> > > connections between the REST proxy and Kafka. I don't care too much
> which
> > > method calls it uses actually uses to consume messages, but I do care
> > that
> > > I cannot secure connections because of network security rules. Second,
> > > there's an affinity between a consumer and the instance of the REST
> proxy
> > > to which it first connected. Kafka itself avoids this kind of affinity
> > for
> > > good reason, and in the name of availability the REST proxy should too.
> > > These are natural KIPs.
> > >
> > > I think it would be good to have the code for the REST proxy
> contributed
> > > to Apache Kafka so that it would be able to be developed in the same
> way.
> > >
> > > Andrew Schofield
> > >
> > > From: Suresh Srinivas <su...@hortonworks.com>
> > > Sent: 07 October 2016 22:41:52
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > ASF already gives us a clear framework and governance model for
> community
> > > development. This is already understood by the people contributing to
> > > Apache Kafka project, and they are the same people who want to
> contribute
> > > to the REST server capability as well. Everyone is in agreement on the
> > > need for collaborating on this effort. So why not contribute the code
> to
> > > Apache Kafka. This will help avoid duplication of effort and forks that
> > > may crop up, hugely benefitting the user community. This will also
> avoid
> > > having to define a process similar to ASF on a GitHub project and
> instead
> > > there is a single community with clear understanding community process
> as
> > > defined in ASF.
> > >
> > > As others have said, this is an important capability for Apache Kafka.
> It
> > > is worth maintaining this as a part of the project.
> > >
> > > Regards,
> > > Suresh
> > >
> > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > >
> > > >I personally think it would be quite wasteful to re-implement the REST
> > > >gateway just because that an actively-maintained piece of
> > Apache-licensed
> > > >software is not governed directly by the Apache Kafka community...
> While
> > > >kafka-rest repo is owned by Confluent, the contributors including the
> > main
> > > >one are also part of the Apache Kafka community, so there is a chance
> > to
> > > >work this out.
> > > >
> > > >However, there are two valid concerns here that could be addressed,
> > around
> > > >community and accessibility:
> > > >>> What we are worried about is a project
> > > >>> that's not maintained in a community. So the process of accepting
> > > >>>patches
> > > >>> and priorities is not clear, and it's not developed in Apache
> > > >>>community.
> > > >>> Not only that, existing REST API project doesn't support new client
> > API
> > > >and
> > > >>> hence there is no security support either.
> > > >
> > > >This might be easy to fix. Maybe Confluent / kafka-rest community can
> > > >clarify that - what is their contribution policy, dev style, roadmap
> > etc.
> > > >If they want, they can make an effort to encourage participation from
> > > >people outside Confluent (easily accept contributions, invite external
> > > >commiters or have open dev process similar to Apache Kafka etc), as
> > there
> > > >is definitely seems to be some interest on the list. That might clear
> > the
> > > >community concern and help kafka-rest project (but that is a
> calculation
> > > >Confluent will have to make).
> > > >
> > > >The other, independent, concern is that REST is something that is
> > expected
> > > >to be available out of the box with Kafka. I personally don't feel
> > > >strongly
> > > >about it (better use proper, efficient APIs from day one), though it
> is
> > > >definitely way smaller than adding a stream processing engine to the
> > > >project :)
> > > >Again,the kafka-rest "community" could take steps to make it even
> easier
> > > >to
> > > >install, configure and run kafka-rest for new users on vanilla Apache
> > > >Kafka
> > > >(outside the Confluent platform), if they wish that (or welcome
> > > >contributions to that end), but that is up to them.
> > > >Finally, if after the above steps were taken there would still a
> strong
> > > >desire to include a great rest gateway with Apache Kafka, I assume the
> > > >community could hypothetically fork the existing kafka-rest into an
> > Apache
> > > >Kafka subproject and maintain it "within Apache" instead of
> implementing
> > > >it
> > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> > > >happen
> > > >without Confluent blessing, and I think that is likely much less
> optimal
> > > >(pulling in other Confluent / Apache licensed dependencies) than
> having
> > a
> > > >separate external community around kafka-rest.
> > > >
> > > >
> > > >Just my two cents,
> > > >
> > > >
> > > >Ofir Manor
> > > >
> > > >Co-Founder & CTO | Equalum
> > > >
> > > >Mobile: +972-54-7801286 <+972%2054-780-1286> <+972%2054-780-1286> |
> Email:
> > > ofir.manor@equalum.io
> > > >
> > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <kafka@harsha.io
> >
> > > >wrote:
> > > >
> > > >> Neha & Jay,
> > > >> We did look at the open source alternatives. Our
> > > >>concern
> > > >> is what's the patch acceptance and adding features/ bug-fixes to the
> > > >> existing project under a Github (although it's licensed under Apache
> > > >>2.0).
> > > >> It would be great if that project made available under Apache and
> > > >>driven by
> > > >> the community. Adding to the above, not all Kafka users are
> > interested
> > > >>in
> > > >> using the Java client API, they would like to have simple REST API
> > where
> > > >> they can code against using any language. I do believe this adds
> value
> > > >>to
> > > >> Apache Kafka in itself.
> > > >>
> > > >> "For 1, I don't think there is value in giving in to the NIH
> syndrome
> > > >>and
> > > >> reinventing the wheel. What I'm looking for is a detailed comparison
> > of
> > > >>the
> > > >> gaps and why those can't be improved in the REST proxy that already
> > > >>exists
> > > >> and is actively maintained."
> > > >>
> > > >> We are not looking at this as NIH. What we are worried about is a
> > > >>project
> > > >> that's not maintained in a community. So the process of accepting
> > > >>patches
> > > >> and priorities is not clear, and it's not developed in Apache
> > community.
> > > >> Not only that, existing REST API project doesn't support new client
> > API
> > > >>and
> > > >> hence there is no security support either.
> > > >> We don't know the timeline when that's made available. We would like
> > to
> > > >>add
> > > >> admin functionality into the REST API. So the Roadmap of that
> project
> > is
> > > >> not driven by Apache.
> > > >>
> > > >>
> > > >> "This doesn't materially have an impact on expanding the usability
> of
> > > >> Kafka. In my experience, REST proxy + Java clients only cover
> ~50%
> > of
> > > >> all
> > > >> Kafka users, and maybe 10% of those are the ones who will use the
> > > >>REST
> > > >> proxy. The remaining 50% are non-java client users (C, python,
> go,
> > > >>node
> > > >> etc)."
> > > >>
> > > >> REST API is most often asked feature in my interactions with Kafka
> > > >>users.
> > > >> In an organization, There will be independent teams who will write
> > their
> > > >> Kafka clients using different language libraries available today,
> and
> > > >> there is no way to standardize this. Instead of supporting several
> > > >> different client libraries users will be interested in using a REST
> > API
> > > >> server. The need for a REST API will only increase as more and more
> > > >>users
> > > >> start using Kafka.
> > > >>
> > > >> "More surface area means more work to keep things consistent.
> Failure
> > > >> to do that has, in fact, hurt the user experience."
> > > >> Having myriad Kafka client GitHub projects that support different
> > > >>languages
> > > >> hurts the user experience and pushes burden to maintain these
> > libraries.
> > > >> REST API is a simple code base that uses existing java client
> > libraries
> > > >>to
> > > >> make life easier for the users.
> > > >>
> > > >> Thanks,
> > > >> Harsha
> > > >>
> > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > > wrote:
> > > >>
> > > >> > Manikumar,
> > > >> >
> > > >> > Thanks for sharing the proposal. I think there are 2 parts to this
> > > >> > discussion -
> > > >> >
> > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > >>feature-complete,
> > > >> > open-source and actively maintained REST proxy in the community?
> > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> > > >> maintain
> > > >> > the high-quality experience that Kafka users have today?
> > > >> >
> > > >> > For 1, I don't think there is value in giving in to the NIH
> syndrome
> > > >>and
> > > >> > reinventing the wheel. What I'm looking for is a detailed
> comparison
> > > >>of
> > > >> the
> > > >> > gaps and why those can't be improved in the REST proxy that
> already
> > > >> exists
> > > >> > and is actively maintained. For example, we depend on zkClient and
> > > >>have
> > > >> > found as well as fixed several bugs by working closely with the
> > people
> > > >> who
> > > >> > maintain zkClient. This should be possible for REST proxy as well,
> > > >>right?
> > > >> >
> > > >> > For 2, I'd like us to review our history of expanding the surface
> > > >>area to
> > > >> > add more clients in the past. Here is a summary -
> > > >> >
> > > >> > - This doesn't materially have an impact on expanding the
> > > >>usability of
> > > >> > Kafka. In my experience, REST proxy + Java clients only cover
> > ~50%
> > > >>of
> > > >> > all
> > > >> > Kafka users, and maybe 10% of those are the ones who will use
> the
> > > >>REST
> > > >> > proxy. The remaining 50% are non-java client users (C, python,
> > go,
> > > >> node
> > > >> > etc).
> > > >> > - People are a lot more excited about promising to contribute
> > while
> > > >> > adding the surface area but not on an ongoing basis down the
> > line.
> > > >> > - More surface area means more work to keep things consistent.
> > > >>Failure
> > > >> > to do that has, in fact, hurt the user experience.
> > > >> > - More surface area hurts agility. We want to do a few things
> > > >>really
> > > >> > well as well as be agile to be able to build on our core
> > > >>competency.
> > > >> >
> > > >> >
> > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> manikumar.reddy@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Jay,
> > > >> > >
> > > >> > > Thanks for your reply.
> > > >> > >
> > > >> > > I agree that we can not add all the clients/tools available in
> > > >> ecosystem
> > > >> > > page to Kafka repo itself. But we feel REST Interface is
> different
> > > >>from
> > > >> > > other clients/tools. Since any language that can work with HTTP
> > can
> > > >> > > easily integrate with this interface, Having an "official" REST
> > > >> > > interface helps user community. This also helps us to integrate
> > well
> > > >> > > with external management and provisioning tools. Apache Kafka
> > > >>release
> > > >> > > with Java clients + REST interface is sufficient for most of the
> > > >>user
> > > >> > > deployments/requirements. This helps users to deal with less
> > number
> > > >> > > of distributions/builds.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Manikumar
> > > >> > >
> > > >> > >
> > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> > wrote:
> > > >> > >
> > > >> > > > Hey guys,
> > > >> > > >
> > > >> > > > There's already a REST interface maintained as a separate
> > > >> project--it's
> > > >> > > > open source and apache licensed and actively maintained (
> > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong
> with
> > >
> > >
> > >
> > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > github.com
> > > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
> It
> > > makes it easy to produce and consume messages, view the state of the
> > > cluster, and ...
> > >
> > > >> that?
> > > >> > > You
> > > >> > > > mentioned that there was some compatibility concern, but
> > > >> compatibility
> > > >> > > has
> > > >> > > > to do with the consumer protocol guarantees not the repo the
> > code
> > > >>is
> > > >> > in,
> > > >> > > > right? Not sure that concern makes sense.
> > > >> > > >
> > > >> > > > We could argue for adding pretty much anything and everything
> in
> > > >>the
> > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> make
> > > >>the
> > > >> > > project
> > > >> > > > more agile.
> > > >> > > >
> > > >> > > > -Jay
> > > >> > > >
> > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > >> manikumar.reddy@gmail.com
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Kafka Devs,
> > > >> > > > >
> > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> Repository.
> > > >> > > > >
> > > >> > > > > There are already open-source alternatives are available.
> But
> > > >>we
> > > >> > would
> > > >> > > > > like to add REST server that
> > > >> > > > > many users ask for under Apache Kafka repo. Many data Infra
> > > >>tools
> > > >> > comes
> > > >> > > > up
> > > >> > > > > with Rest Interface.
> > > >> > > > > It is useful to have inbuilt Rest API support for Produce,
> > > >>Consume
> > > >> > > > messages
> > > >> > > > > and admin interface for
> > > >> > > > > integrating with external management and provisioning
> > tools.This
> > > >> will
> > > >> > > > also
> > > >> > > > > allow the maintenance of
> > > >> > > > > REST server and adding new features makes it easy because
> > apache
> > > >> > > > community.
> > > >> > > > >
> > > >> > > > > The KIP wiki is the following:
> > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> > > > > 80%3A+Kafka+Rest+Server
> > > >> > > > >
> > > >> > > > > Your comments and feedback are welcome.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Manikumar
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> > --
> > > >> > Thanks,
> > > >> > Neha
> > > >> >
> > > >>
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Jay Kreps <ja...@confluent.io>.
Harsha,
You seem to be saying that your only two options are to fork or duplicate
the existing REST project or add it to Kafka to be able to contribute. I
don't think those are the only two options. The other option is to
contribute to the existing successful project--which is Apache licensed and
getting active contribution today. You always have the power to
fork/duplicate later if you don't like the governance/community/direction.
Saying that you have to do this proactively doesn't really make sense.
-Jay
On Thu, Oct 20, 2016 at 2:27 PM, Harsha Chintalapani <ka...@harsha.io>
wrote:
> Jay,
> REST API is something every user is in need of. If the argument is to
> clone and write your API, this will do a disservice to the users as they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
>
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
> Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this REST
> server to Kafka will affect the agility of the project doesn't mak sense.
>
> It looks like your argument is either we develop all these small tools or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka could
> have been left in their own Github projects based on your arguments. What
> about the REST API is so different that such that it should stay out of the
> Kafka project? From my experience, more users are asking for the REST API.
>
> Thanks,
> Harsha
>
> On Sun, Oct 16, 2016 at 5:19 PM Jungtaek Lim <ka...@gmail.com> wrote:
>
> > I guess no one doubts its power on REST server or even UI. I understand
> the
> > difficulty to add a module to project, but it's maximized when there is
> > less support expected hence maintenance issue is likely to rise, and IMHO
> > this seems to be not the case.
> >
> > There're also pain points when project doesn't maintain features and
> > delegates to ecosystem. Based on some points (last commit date, pull
> > request open and closed, and contributor graph), kafka-manager seems to
> > have similar activity to kafka-rest, but it doesn't show any responses
> for
> > pull request supporting Kafka 0.10.0 even though numerous users leave
> > comments wish to support. What Kafka community can do for that project to
> > follow up? Nothing but just persuading by leaving comments hoping that
> will
> > be merged. (or finally come up another implementation) Kafka project
> keeps
> > agile but in point of whole ecosystem it can be less agile.
> >
> > Yes decisions and roadmap of the project are driven by PMCs and I think
> > it's valid right. But we also imagine ASF projects as driven by community
> > aspect, though it's alike to ideal world. KIP makes innovation on
> adopting
> > new feature transparently, which makes many developers inspiring and
> > adopting it to their projects. Hopes that Kafka community continuously
> > drives the transparency model among the ASF projects, and beyond.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2016년 10월 17일 (월) 오전 7:56, Jay Kreps <ja...@confluent.io>님이 작성:
> >
> > Hey Nacho,
> >
> > Yeah, I think it is definitely a call we have to make case by case. We
> have
> > some experience with this: originally we attempted to maintain things
> like
> > non-java clients, a hadoop connector, etc all in the main project. The
> > difficulty of that lead us to the current federated approach. In terms of
> > what is included now, yes, I agree you could potentially have even less
> > included.
> >
> > -Jay
> >
> > On Wed, Oct 12, 2016 at 11:37 AM, Nacho Solis
> <nsolis@linkedin.com.invalid
> > >
> > wrote:
> >
> > > What is the criteria for keeping things in and out of Kafka, what code
> > goes
> > > in or out and what is part of the architecture or not?
> > >
> > > The discussion of what goes into a project and what stays out is an
> > always
> > > evolving question. Different projects treat this in different ways.
> > >
> > > Let me paint 2 extremes. On one side, you have a single monolithic
> > project
> > > that brings everything in one tent. On the other side you have the
> many
> > > modules approach. From what I've learned, Kafka falls in the middle.
> > > Because of this, the question is bound to come up with respect to the
> > > criteria used to bring something into the fold.
> > >
> > > I'll be the first to point out that the distinction between modules,
> > > architecture, software, repositories, governance and community are
> > blurry.
> > > Not to mention that many things are how they are for historical
> reasons.
> > >
> > > I, personally, can't understand why we would not have REST as part of
> the
> > > main Kafka project given that a lot of people use it and we include
> many
> > > things with the current distribution. What many things you may ask?
> > Well,
> > > if we took the modular approach Kafka is a mixture of components,
> here's
> > > the first 4 that come to mind:
> > > 1. The Kafka protocol
> > > 2. The Kafka java libraries
> > > 3. The Kafka broker
> > > 4. The Kafka stream framework
> > > 5. Kafka Connect
> > > 6. MirrorMaker
> > >
> > > All of these could be separate products. You should be able to evolve
> > each
> > > one independently. Even if they have dependencies on each other, you
> > could
> > > potentially replace one part.
> > >
> > > The choice of keeping them all in a single repository, with a single
> > > distribution, under the same governance and community, brings a number
> of
> > > trade offs. It's easy to keep things coherent for example. There is
> > less
> > > of a need to rely on inherent versioning and compatibility (which we
> end
> > up
> > > providing anyway because of the way people usually deploy kafka). We
> all
> > > focus our efforts on a single code base.
> > >
> > > The downside is that it's harder to remove modules that are old or
> > unused.
> > > Modules that are only used by a small subset of the community will have
> > an
> > > impact on the rest of the community. It mixes incentives of what
> people
> > > want to work on and what holds them back. We also need to decide what
> > > belongs in the blessed bundle and what doesnt.
> > >
> > > So, my question boils down to, what criteria is used for bringing stuff
> > in.
> > >
> > > If we have Streams and MirrorMaker and Connect in there, why not have
> > REST?
> > > Specially if there is more than one person/group willing to work on it?
> > > Alternatively, if REST is not included because it's not used by all,
> then
> > > why not remove Streams, Connect and MirrorMaker since they're
> definitely
> > > not used by all? I realize I say this even though at LinkedIn we have a
> > > REST setup of our own, just speaking from a community perspective.
> > >
> > > Nacho
> > >
> > >
> > > (I'm relatively new and I haven't read all of the mail archive, so I'm
> > sure
> > > this has been brought up before, but I decided to chime in anyway)
> > >
> > > On Wed, Oct 12, 2016 at 8:03 AM, Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > I think the questions around governance make sense, I think we should
> > > > really clarify that to make the process more clear so it can be fully
> > > > inclusive.
> > > >
> > > > The idea that we should not collaborate on what is there now, though,
> > > > because in the future we might disagree about direction does not
> really
> > > > make sense to me. If in the future we disagree, that is the beauty of
> > > open
> > > > source, you can always fork off a copy of the code and start an
> > > independent
> > > > project either in Apache or elsewhere. Pre-emptively re-creating
> > another
> > > > REST layer when it seems like we all quite agree on what needs to be
> > done
> > > > and we have an existing code base for HTTP/kafka access that is
> heavily
> > > > used in production seems quite silly.
> > > >
> > > > Let me give some background on how I at least think about these
> things.
> > > > I've participated in open source projects out of LinkedIn via github
> as
> > > > well as via the ASF. I don't think there is a "right" answer to how
> to
> > do
> > > > these but rather some tradeoffs. We thought about this quite a lot in
> > the
> > > > context of Kafka based on the experience with the Hadoop ecosystem as
> > > well
> > > > as from other open source communities.
> > > >
> > > > There is a rich ecosystem around Kafka. Many of the projects are
> quite
> > > > small--single clients or tools that do a single thing well--and
> almost
> > > none
> > > > of them are top level apache projects. I don't think trying to force
> > each
> > > > of these to turn into independent Apache projects is necessarily the
> > best
> > > > thing for the ecosystem.
> > > >
> > > > My observation of how this can go wrong is really what I think has
> > > happened
> > > > to the Hadoop ecosystem. There you see quite a zoo of projects which
> > all
> > > > drift apart and don't quite work together well. Coordinating even
> > simple
> > > > changes and standardization across these is exceptionally difficult.
> > The
> > > > result is a bit of a mess for users--the pieces just don't really
> come
> > > > together very well. This makes sense for independent infrastructure
> > > systems
> > > > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > > > little tool or helper library has lead to a desirable state. I think
> > the
> > > > mode of operating where the Hadoop vendors spawn off a few new Apache
> > > > projects for each new product initiative, especially since often that
> > > > project is only valued by that vendor (and the other vendor has a
> > > different
> > > > competing Apache project) doesn't necessarily do a better job at
> > > producing
> > > > high quality communities or high quality software.
> > > >
> > > > These tools/connects/clients/proxies and other integration pieces can
> > > take
> > > > many forms, but my take of what makes one of these things good is
> that
> > it
> > > > remains simple, does its one thing well, and cleaves as closely as
> > > possible
> > > > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > > > monitoring, configuring, etc. For the tools we've contributed we've
> > tried
> > > > really hard to make them consistent with Kafka as well as with each
> > other
> > > > in how testing, configuration, monitoring, etc works.
> > > >
> > > > I think what Apache does superbly well is create a community for
> > > managing a
> > > > large infrastructure layer like Kafka in a vendor independent way.
> What
> > I
> > > > think is less successful is attempting to form full and independent
> > > apache
> > > > communities around very simple single purpose tools, especially if
> you
> > > hope
> > > > for these to come together into a cohesive toolset across multiple
> such
> > > > tools. Much of what Apache does--create a collective decision making
> > > > process for resolving disagreement, help to trademark and protect the
> > > marks
> > > > of the project, etc just isn't that relevant for simple
> single-purpose
> > > > tools.
> > > >
> > > > So my take is there are a couple of options:
> > > >
> > > > 1. We can try to put all the small tools into the Apache Project.
> I
> > > > think this is not the right approach as there is simply too many
> of
> > > > them,
> > > > many in different languages, serving different protocols,
> > integrating
> > > > with
> > > > particular systems, and a single community can't effectively
> > maintain
> > > > them
> > > > all. Doing this would significantly slow the progress of the Kafka
> > > > project.
> > > > As a protocol for messaging, I don't really see a case for
> including
> > > > REST
> > > > but not MQTT or AMQP which are technically much better suited to
> > > > messaging
> > > > and both are widely used for that.
> > > > 2. We can treat ecosystem projects that aren't top level Apache
> > > projects
> > > > as invalid and try to recreate them all as Apache projects.
> > Honestly,
> > > > though, if you go to the Kafka ecosystem page virtually none of
> the
> > > most
> > > > popular add-ons to Kafka are Apache projects. The most successful
> > > > things in
> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> > > other
> > > > clients, as well as the existing REST layer have succeeded at
> > > developing
> > > > communities that actively contribute and use these pieces and I
> > don't
> > > > know
> > > > that that is a bad thing unless that community proves to be
> > > uninclusive,
> > > > unresponsive, or goes in a bad technical direction--and those are
> > > > failure
> > > > modes that all open source efforts face.
> > > > 3. We can do what I think makes the most sense and try to work
> with
> > > the
> > > > projects that exist in the ecosystem and if the project doesn't
> have
> > a
> > > > responsive community or wants to go in a different direction fork
> or
> > > > recreate that work.
> > > >
> > > > Of course any person can choose whatever of these options they want.
> > But
> > > > from my point of view, option (3) has been the path of the community
> so
> > > far
> > > > and I think it has been quite successful.
> > > >
> > > > -Jay
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > > > wrote:
> > > >
> > > > > Neha,
> > > > > "But I haven't seen any community emails or patches being submitted
> > by
> > > > you
> > > > > guys, so I'm wondering why you are concerned about whether the
> > > community
> > > > is
> > > > > open to accepting patches or not."
> > > > >
> > > > > I think you are talking about contributing patches to this
> repository
> > > > > right? https://github.com/confluentinc/kafka-rest . All I am
> saying
> > > > > the guidelines/governance model is not clear on the project and I
> > guess
> > > > its
> > > > > driven by opening a github issue request. Its the repository owned
> > by
> > > > > confluent and as much I appreciate that the features we mentioned
> are
> > > in
> > > > > the roadmap and welcoming us to contribute to the project. It
> doesn't
> > > > > gurantee what we want to add in the furture will be in your
> roadmap.
> > > > >
> > > > > Hence the reason having it part of Kafka community will help a lot
> as
> > > > other
> > > > > users can participate in the discussions. We are happy to drive
> any
> > > > > feature additions through KIPs which gives everyone a chance to
> > > > participate
> > > > > and add to the discussions.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > I agree on the governance comments whole heartedly.
> > > > > >
> > > > > > Also i agree about the contribution comments made earlier in the
> > > > thread,
> > > > > i
> > > > > > personally am less likely to spend any of mine, or give project
> > time
> > > > > within
> > > > > > my internal projects to developers contributing to another
> > commercial
> > > > > > companies project even so technically open source, as then there
> is
> > > > that
> > > > > > commercial companies interest will always prevail and essentially
> > can
> > > > > > always have a final vote where disagreement. Im sure they never
> > > intend
> > > > > to,
> > > > > > but there is that true reality. This is why we have community
> open
> > > > source
> > > > > > projects.
> > > > > >
> > > > > > I can find many different implementations now of a rest endpoint
> on
> > > > > > GitHub, BitBucket etc. Each one has their benefits and
> > disadvantages
> > > in
> > > > > > their implementation. By making / providing one this would bring
> > > > together
> > > > > > these solutions, unifying those developers and also bringing the
> > best
> > > > of
> > > > > > all.
> > > > > >
> > > > > > I understand the concern on the community burden
> adding/supporting
> > > more
> > > > > > surface area for every client. But something like REST is
> universal
> > > and
> > > > > > worthy to be owned by the community.
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Andrew Schofield <an...@outlook.com>
> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > >
> > > > > > There's a massive difference between the governance of Kafka and
> > the
> > > > > > governance of the REST proxy.
> > > > > >
> > > > > > In Kafka, there is a broad community of people contributing their
> > > > > opinions
> > > > > > about future enhancements in the form of KIPs. There's some
> really
> > > deep
> > > > > > consideration that goes into some of the trickier KIPs. There are
> > > > people
> > > > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > > > reputations to become committers. I get the impression that the
> > > roadmap
> > > > > of
> > > > > > Kafka is not really community-owned (what's the big feature for
> > Kafka
> > > > > 0.11,
> > > > > > for example), but the conveyor belt of smaller features in the
> form
> > > of
> > > > > KIPs
> > > > > > works nicely. It's a good example of open-source working well.
> > > > > >
> > > > > > The equivalent for the REST proxy is basically issues on GitHub.
> > The
> > > > > > roadmap is less clear. There's not really a community properly
> > > engaged
> > > > in
> > > > > > the way that there is with Kafka. So, you could say that it's
> clear
> > > > that
> > > > > > fewer people are interested, but I think the whole governance
> > thing
> > > > is a
> > > > > > big barrier to engagement. And it's looking like it's getting out
> > of
> > > > > date.
> > > > > >
> > > > > > In technical terms, I can think of two big improvements to the
> REST
> > > > > proxy.
> > > > > > First, it needs to use the new consumer API so that it's possible
> > to
> > > > > secure
> > > > > > connections between the REST proxy and Kafka. I don't care too
> much
> > > > which
> > > > > > method calls it uses actually uses to consume messages, but I do
> > > care
> > > > > that
> > > > > > I cannot secure connections because of network security rules.
> > > Second,
> > > > > > there's an affinity between a consumer and the instance of the
> REST
> > > > proxy
> > > > > > to which it first connected. Kafka itself avoids this kind of
> > > affinity
> > > > > for
> > > > > > good reason, and in the name of availability the REST proxy
> should
> > > too.
> > > > > > These are natural KIPs.
> > > > > >
> > > > > > I think it would be good to have the code for the REST proxy
> > > > contributed
> > > > > > to Apache Kafka so that it would be able to be developed in the
> > same
> > > > way.
> > > > > >
> > > > > > Andrew Schofield
> > > > > >
> > > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > > > Sent: 07 October 2016 22:41:52
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > >
> > > > > > ASF already gives us a clear framework and governance model for
> > > > community
> > > > > > development. This is already understood by the people
> contributing
> > to
> > > > > > Apache Kafka project, and they are the same people who want to
> > > > contribute
> > > > > > to the REST server capability as well. Everyone is in agreement
> on
> > > the
> > > > > > need for collaborating on this effort. So why not contribute the
> > code
> > > > to
> > > > > > Apache Kafka. This will help avoid duplication of effort and
> forks
> > > that
> > > > > > may crop up, hugely benefitting the user community. This will
> also
> > > > avoid
> > > > > > having to define a process similar to ASF on a GitHub project and
> > > > instead
> > > > > > there is a single community with clear understanding community
> > > process
> > > > as
> > > > > > defined in ASF.
> > > > > >
> > > > > > As others have said, this is an important capability for Apache
> > > Kafka.
> > > > It
> > > > > > is worth maintaining this as a part of the project.
> > > > > >
> > > > > > Regards,
> > > > > > Suresh
> > > > > >
> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > > > >
> > > > > > >I personally think it would be quite wasteful to re-implement
> the
> > > REST
> > > > > > >gateway just because that an actively-maintained piece of
> > > > > Apache-licensed
> > > > > > >software is not governed directly by the Apache Kafka
> community...
> > > > While
> > > > > > >kafka-rest repo is owned by Confluent, the contributors
> including
> > > the
> > > > > main
> > > > > > >one are also part of the Apache Kafka community, so there is a
> > > chance
> > > > > to
> > > > > > >work this out.
> > > > > > >
> > > > > > >However, there are two valid concerns here that could be
> > addressed,
> > > > > around
> > > > > > >community and accessibility:
> > > > > > >>> What we are worried about is a project
> > > > > > >>> that's not maintained in a community. So the process of
> > accepting
> > > > > > >>>patches
> > > > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > > > >>>community.
> > > > > > >>> Not only that, existing REST API project doesn't support new
> > > client
> > > > > API
> > > > > > >and
> > > > > > >>> hence there is no security support either.
> > > > > > >
> > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
> community
> > > can
> > > > > > >clarify that - what is their contribution policy, dev style,
> > roadmap
> > > > > etc.
> > > > > > >If they want, they can make an effort to encourage participation
> > > from
> > > > > > >people outside Confluent (easily accept contributions, invite
> > > external
> > > > > > >commiters or have open dev process similar to Apache Kafka etc),
> > as
> > > > > there
> > > > > > >is definitely seems to be some interest on the list. That might
> > > clear
> > > > > the
> > > > > > >community concern and help kafka-rest project (but that is a
> > > > calculation
> > > > > > >Confluent will have to make).
> > > > > > >
> > > > > > >The other, independent, concern is that REST is something that
> is
> > > > > expected
> > > > > > >to be available out of the box with Kafka. I personally don't
> feel
> > > > > > >strongly
> > > > > > >about it (better use proper, efficient APIs from day one),
> though
> > it
> > > > is
> > > > > > >definitely way smaller than adding a stream processing engine to
> > the
> > > > > > >project :)
> > > > > > >Again,the kafka-rest "community" could take steps to make it
> even
> > > > easier
> > > > > > >to
> > > > > > >install, configure and run kafka-rest for new users on vanilla
> > > Apache
> > > > > > >Kafka
> > > > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > > > >contributions to that end), but that is up to them.
> > > > > > >Finally, if after the above steps were taken there would still a
> > > > strong
> > > > > > >desire to include a great rest gateway with Apache Kafka, I
> assume
> > > the
> > > > > > >community could hypothetically fork the existing kafka-rest into
> > an
> > > > > Apache
> > > > > > >Kafka subproject and maintain it "within Apache" instead of
> > > > implementing
> > > > > > >it
> > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot
> imagine
> > it
> > > > > > >happen
> > > > > > >without Confluent blessing, and I think that is likely much less
> > > > optimal
> > > > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > > > having
> > > > > a
> > > > > > >separate external community around kafka-rest.
> > > > > > >
> > > > > > >
> > > > > > >Just my two cents,
> > > > > > >
> > > > > > >
> > > > > > >Ofir Manor
> > > > > > >
> > > > > > >Co-Founder & CTO | Equalum
> > > > > > >
> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > <+972%2054-780-1286> <+972%2054-780-1286>
> > | Email:
> > > > > > ofir.manor@equalum.io
> > > > > > >
> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > > kafka@harsha.io
> > > > >
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Neha & Jay,
> > > > > > >> We did look at the open source alternatives.
> > Our
> > > > > > >>concern
> > > > > > >> is what's the patch acceptance and adding features/ bug-fixes
> to
> > > the
> > > > > > >> existing project under a Github (although it's licensed under
> > > Apache
> > > > > > >>2.0).
> > > > > > >> It would be great if that project made available under Apache
> > and
> > > > > > >>driven by
> > > > > > >> the community. Adding to the above, not all Kafka users are
> > > > > interested
> > > > > > >>in
> > > > > > >> using the Java client API, they would like to have simple REST
> > API
> > > > > where
> > > > > > >> they can code against using any language. I do believe this
> adds
> > > > value
> > > > > > >>to
> > > > > > >> Apache Kafka in itself.
> > > > > > >>
> > > > > > >> "For 1, I don't think there is value in giving in to the NIH
> > > > syndrome
> > > > > > >>and
> > > > > > >> reinventing the wheel. What I'm looking for is a detailed
> > > comparison
> > > > > of
> > > > > > >>the
> > > > > > >> gaps and why those can't be improved in the REST proxy that
> > > already
> > > > > > >>exists
> > > > > > >> and is actively maintained."
> > > > > > >>
> > > > > > >> We are not looking at this as NIH. What we are worried about
> is
> > a
> > > > > > >>project
> > > > > > >> that's not maintained in a community. So the process of
> > accepting
> > > > > > >>patches
> > > > > > >> and priorities is not clear, and it's not developed in Apache
> > > > > community.
> > > > > > >> Not only that, existing REST API project doesn't support new
> > > client
> > > > > API
> > > > > > >>and
> > > > > > >> hence there is no security support either.
> > > > > > >> We don't know the timeline when that's made available. We
> would
> > > like
> > > > > to
> > > > > > >>add
> > > > > > >> admin functionality into the REST API. So the Roadmap of that
> > > > project
> > > > > is
> > > > > > >> not driven by Apache.
> > > > > > >>
> > > > > > >>
> > > > > > >> "This doesn't materially have an impact on expanding the
> > usability
> > > > of
> > > > > > >> Kafka. In my experience, REST proxy + Java clients only
> cover
> > > > ~50%
> > > > > of
> > > > > > >> all
> > > > > > >> Kafka users, and maybe 10% of those are the ones who will
> use
> > > the
> > > > > > >>REST
> > > > > > >> proxy. The remaining 50% are non-java client users (C,
> > python,
> > > > go,
> > > > > > >>node
> > > > > > >> etc)."
> > > > > > >>
> > > > > > >> REST API is most often asked feature in my interactions with
> > Kafka
> > > > > > >>users.
> > > > > > >> In an organization, There will be independent teams who will
> > write
> > > > > their
> > > > > > >> Kafka clients using different language libraries available
> > today,
> > > > and
> > > > > > >> there is no way to standardize this. Instead of supporting
> > several
> > > > > > >> different client libraries users will be interested in using a
> > > REST
> > > > > API
> > > > > > >> server. The need for a REST API will only increase as more and
> > > more
> > > > > > >>users
> > > > > > >> start using Kafka.
> > > > > > >>
> > > > > > >> "More surface area means more work to keep things consistent.
> > > > Failure
> > > > > > >> to do that has, in fact, hurt the user experience."
> > > > > > >> Having myriad Kafka client GitHub projects that support
> > different
> > > > > > >>languages
> > > > > > >> hurts the user experience and pushes burden to maintain these
> > > > > libraries.
> > > > > > >> REST API is a simple code base that uses existing java client
> > > > > libraries
> > > > > > >>to
> > > > > > >> make life easier for the users.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Harsha
> > > > > > >>
> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > neha@confluent.io>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Manikumar,
> > > > > > >> >
> > > > > > >> > Thanks for sharing the proposal. I think there are 2 parts
> to
> > > this
> > > > > > >> > discussion -
> > > > > > >> >
> > > > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > > > >>feature-complete,
> > > > > > >> > open-source and actively maintained REST proxy in the
> > community?
> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more
> agile
> > > and
> > > > > > >> maintain
> > > > > > >> > the high-quality experience that Kafka users have today?
> > > > > > >> >
> > > > > > >> > For 1, I don't think there is value in giving in to the NIH
> > > > syndrome
> > > > > > >>and
> > > > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > > > comparison
> > > > > > >>of
> > > > > > >> the
> > > > > > >> > gaps and why those can't be improved in the REST proxy that
> > > > already
> > > > > > >> exists
> > > > > > >> > and is actively maintained. For example, we depend on
> zkClient
> > > and
> > > > > > >>have
> > > > > > >> > found as well as fixed several bugs by working closely with
> > the
> > > > > people
> > > > > > >> who
> > > > > > >> > maintain zkClient. This should be possible for REST proxy as
> > > well,
> > > > > > >>right?
> > > > > > >> >
> > > > > > >> > For 2, I'd like us to review our history of expanding the
> > > surface
> > > > > > >>area to
> > > > > > >> > add more clients in the past. Here is a summary -
> > > > > > >> >
> > > > > > >> > - This doesn't materially have an impact on expanding the
> > > > > > >>usability of
> > > > > > >> > Kafka. In my experience, REST proxy + Java clients only
> > cover
> > > > > ~50%
> > > > > > >>of
> > > > > > >> > all
> > > > > > >> > Kafka users, and maybe 10% of those are the ones who will
> > use
> > > > the
> > > > > > >>REST
> > > > > > >> > proxy. The remaining 50% are non-java client users (C,
> > > python,
> > > > > go,
> > > > > > >> node
> > > > > > >> > etc).
> > > > > > >> > - People are a lot more excited about promising to
> > contribute
> > > > > while
> > > > > > >> > adding the surface area but not on an ongoing basis down
> > the
> > > > > line.
> > > > > > >> > - More surface area means more work to keep things
> > > consistent.
> > > > > > >>Failure
> > > > > > >> > to do that has, in fact, hurt the user experience.
> > > > > > >> > - More surface area hurts agility. We want to do a few
> > things
> > > > > > >>really
> > > > > > >> > well as well as be agile to be able to build on our core
> > > > > > >>competency.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > > manikumar.reddy@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Jay,
> > > > > > >> > >
> > > > > > >> > > Thanks for your reply.
> > > > > > >> > >
> > > > > > >> > > I agree that we can not add all the clients/tools
> available
> > in
> > > > > > >> ecosystem
> > > > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > > > different
> > > > > > >>from
> > > > > > >> > > other clients/tools. Since any language that can work with
> > > HTTP
> > > > > can
> > > > > > >> > > easily integrate with this interface, Having an "official"
> > > REST
> > > > > > >> > > interface helps user community. This also helps us to
> > > integrate
> > > > > well
> > > > > > >> > > with external management and provisioning tools. Apache
> > Kafka
> > > > > > >>release
> > > > > > >> > > with Java clients + REST interface is sufficient for most
> of
> > > the
> > > > > > >>user
> > > > > > >> > > deployments/requirements. This helps users to deal with
> less
> > > > > number
> > > > > > >> > > of distributions/builds.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Manikumar
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> jay@confluent.io
> > >
> > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hey guys,
> > > > > > >> > > >
> > > > > > >> > > > There's already a REST interface maintained as a
> separate
> > > > > > >> project--it's
> > > > > > >> > > > open source and apache licensed and actively maintained
> (
> > > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is
> > wrong
> > > > with
> > > > > >
> > > > > >
> > > > > >
> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > > > github.com
> > > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
> > cluster.
> > > > It
> > > > > > makes it easy to produce and consume messages, view the state of
> > the
> > > > > > cluster, and ...
> > > > > >
> > > > > > >> that?
> > > > > > >> > > You
> > > > > > >> > > > mentioned that there was some compatibility concern, but
> > > > > > >> compatibility
> > > > > > >> > > has
> > > > > > >> > > > to do with the consumer protocol guarantees not the repo
> > the
> > > > > code
> > > > > > >>is
> > > > > > >> > in,
> > > > > > >> > > > right? Not sure that concern makes sense.
> > > > > > >> > > >
> > > > > > >> > > > We could argue for adding pretty much anything and
> > > everything
> > > > in
> > > > > > >>the
> > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that
> would
> > > > make
> > > > > > >>the
> > > > > > >> > > project
> > > > > > >> > > > more agile.
> > > > > > >> > > >
> > > > > > >> > > > -Jay
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > > > >> manikumar.reddy@gmail.com
> > > > > > >> > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi Kafka Devs,
> > > > > > >> > > > >
> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > > > Repository.
> > > > > > >> > > > >
> > > > > > >> > > > > There are already open-source alternatives are
> > available.
> > > > But
> > > > > > >>we
> > > > > > >> > would
> > > > > > >> > > > > like to add REST server that
> > > > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> > > Infra
> > > > > > >>tools
> > > > > > >> > comes
> > > > > > >> > > > up
> > > > > > >> > > > > with Rest Interface.
> > > > > > >> > > > > It is useful to have inbuilt Rest API support for
> > Produce,
> > > > > > >>Consume
> > > > > > >> > > > messages
> > > > > > >> > > > > and admin interface for
> > > > > > >> > > > > integrating with external management and provisioning
> > > > > tools.This
> > > > > > >> will
> > > > > > >> > > > also
> > > > > > >> > > > > allow the maintenance of
> > > > > > >> > > > > REST server and adding new features makes it easy
> > because
> > > > > apache
> > > > > > >> > > > community.
> > > > > > >> > > > >
> > > > > > >> > > > > The KIP wiki is the following:
> > > > > > >> > > > > https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-
> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > > > >> > > > >
> > > > > > >> > > > > Your comments and feedback are welcome.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Manikumar
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > --
> > > > > > >> > Thanks,
> > > > > > >> > Neha
> > > > > > >> >
> > > > > > >>
> > > > > > The information contained in this email is strictly confidential
> > and
> > > > for
> > > > > > the use of the addressee only, unless otherwise indicated. If you
> > are
> > > > not
> > > > > > the intended recipient, please do not read, copy, use or disclose
> > to
> > > > > others
> > > > > > this message or any attachment. Please also notify the sender by
> > > > replying
> > > > > > to this email or by telephone (+44(020 7896 0011) and then delete
> > the
> > > > > email
> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > > to
> > > > > the
> > > > > > official business of this company shall be understood as neither
> > > given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > > > > > registered in England and Wales, company number 04008957) and IG
> > > Index
> > > > > > Limited (a company registered in England and Wales, company
> number
> > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and
> > > > IG
> > > > > > Index Limited (register number 114059) are authorised and
> regulated
> > > by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Nacho (Ignacio) Solis
> > > Kafka
> > > nsolis@linkedin.com
> > >
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Harsha Chintalapani <ka...@harsha.io>.
Jay,
REST API is something every user is in need of. If the argument is to
clone and write your API, this will do a disservice to the users as they
now have to choose one vs. others instead of keeping one API that is
supported in Kafka community.
"Pre-emptively re-creating another
REST layer when it seems like we all quite agree on what needs to be done
and we have an existing code base for HTTP/Kafka access that is heavily
used in production seems quite silly."
Exactly our point. Why can't we develop this in Apache Kafka
community? Instead of us open sourcing another GitHub project and creating
a divide in users and another version of API. Let's build this in Kafka
Community and use the governance model that is proven to provide vendor
free user driven consensus features. The argument that is adding this REST
server to Kafka will affect the agility of the project doesn't mak sense.
It looks like your argument is either we develop all these small tools or
none at all. We as a community need to look at supporting critical
tools/API. Instead of dividing this project into individual external
communities. We should build this as part of Kafka which best serves the
needs of users.
The Streams and Connect projects that were pushed into Kafka could
have been left in their own Github projects based on your arguments. What
about the REST API is so different that such that it should stay out of the
Kafka project? From my experience, more users are asking for the REST API.
Thanks,
Harsha
On Sun, Oct 16, 2016 at 5:19 PM Jungtaek Lim <ka...@gmail.com> wrote:
> I guess no one doubts its power on REST server or even UI. I understand the
> difficulty to add a module to project, but it's maximized when there is
> less support expected hence maintenance issue is likely to rise, and IMHO
> this seems to be not the case.
>
> There're also pain points when project doesn't maintain features and
> delegates to ecosystem. Based on some points (last commit date, pull
> request open and closed, and contributor graph), kafka-manager seems to
> have similar activity to kafka-rest, but it doesn't show any responses for
> pull request supporting Kafka 0.10.0 even though numerous users leave
> comments wish to support. What Kafka community can do for that project to
> follow up? Nothing but just persuading by leaving comments hoping that will
> be merged. (or finally come up another implementation) Kafka project keeps
> agile but in point of whole ecosystem it can be less agile.
>
> Yes decisions and roadmap of the project are driven by PMCs and I think
> it's valid right. But we also imagine ASF projects as driven by community
> aspect, though it's alike to ideal world. KIP makes innovation on adopting
> new feature transparently, which makes many developers inspiring and
> adopting it to their projects. Hopes that Kafka community continuously
> drives the transparency model among the ASF projects, and beyond.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2016년 10월 17일 (월) 오전 7:56, Jay Kreps <ja...@confluent.io>님이 작성:
>
> Hey Nacho,
>
> Yeah, I think it is definitely a call we have to make case by case. We have
> some experience with this: originally we attempted to maintain things like
> non-java clients, a hadoop connector, etc all in the main project. The
> difficulty of that lead us to the current federated approach. In terms of
> what is included now, yes, I agree you could potentially have even less
> included.
>
> -Jay
>
> On Wed, Oct 12, 2016 at 11:37 AM, Nacho Solis <nsolis@linkedin.com.invalid
> >
> wrote:
>
> > What is the criteria for keeping things in and out of Kafka, what code
> goes
> > in or out and what is part of the architecture or not?
> >
> > The discussion of what goes into a project and what stays out is an
> always
> > evolving question. Different projects treat this in different ways.
> >
> > Let me paint 2 extremes. On one side, you have a single monolithic
> project
> > that brings everything in one tent. On the other side you have the many
> > modules approach. From what I've learned, Kafka falls in the middle.
> > Because of this, the question is bound to come up with respect to the
> > criteria used to bring something into the fold.
> >
> > I'll be the first to point out that the distinction between modules,
> > architecture, software, repositories, governance and community are
> blurry.
> > Not to mention that many things are how they are for historical reasons.
> >
> > I, personally, can't understand why we would not have REST as part of the
> > main Kafka project given that a lot of people use it and we include many
> > things with the current distribution. What many things you may ask?
> Well,
> > if we took the modular approach Kafka is a mixture of components, here's
> > the first 4 that come to mind:
> > 1. The Kafka protocol
> > 2. The Kafka java libraries
> > 3. The Kafka broker
> > 4. The Kafka stream framework
> > 5. Kafka Connect
> > 6. MirrorMaker
> >
> > All of these could be separate products. You should be able to evolve
> each
> > one independently. Even if they have dependencies on each other, you
> could
> > potentially replace one part.
> >
> > The choice of keeping them all in a single repository, with a single
> > distribution, under the same governance and community, brings a number of
> > trade offs. It's easy to keep things coherent for example. There is
> less
> > of a need to rely on inherent versioning and compatibility (which we end
> up
> > providing anyway because of the way people usually deploy kafka). We all
> > focus our efforts on a single code base.
> >
> > The downside is that it's harder to remove modules that are old or
> unused.
> > Modules that are only used by a small subset of the community will have
> an
> > impact on the rest of the community. It mixes incentives of what people
> > want to work on and what holds them back. We also need to decide what
> > belongs in the blessed bundle and what doesnt.
> >
> > So, my question boils down to, what criteria is used for bringing stuff
> in.
> >
> > If we have Streams and MirrorMaker and Connect in there, why not have
> REST?
> > Specially if there is more than one person/group willing to work on it?
> > Alternatively, if REST is not included because it's not used by all, then
> > why not remove Streams, Connect and MirrorMaker since they're definitely
> > not used by all? I realize I say this even though at LinkedIn we have a
> > REST setup of our own, just speaking from a community perspective.
> >
> > Nacho
> >
> >
> > (I'm relatively new and I haven't read all of the mail archive, so I'm
> sure
> > this has been brought up before, but I decided to chime in anyway)
> >
> > On Wed, Oct 12, 2016 at 8:03 AM, Jay Kreps <ja...@confluent.io> wrote:
> >
> > > I think the questions around governance make sense, I think we should
> > > really clarify that to make the process more clear so it can be fully
> > > inclusive.
> > >
> > > The idea that we should not collaborate on what is there now, though,
> > > because in the future we might disagree about direction does not really
> > > make sense to me. If in the future we disagree, that is the beauty of
> > open
> > > source, you can always fork off a copy of the code and start an
> > independent
> > > project either in Apache or elsewhere. Pre-emptively re-creating
> another
> > > REST layer when it seems like we all quite agree on what needs to be
> done
> > > and we have an existing code base for HTTP/kafka access that is heavily
> > > used in production seems quite silly.
> > >
> > > Let me give some background on how I at least think about these things.
> > > I've participated in open source projects out of LinkedIn via github as
> > > well as via the ASF. I don't think there is a "right" answer to how to
> do
> > > these but rather some tradeoffs. We thought about this quite a lot in
> the
> > > context of Kafka based on the experience with the Hadoop ecosystem as
> > well
> > > as from other open source communities.
> > >
> > > There is a rich ecosystem around Kafka. Many of the projects are quite
> > > small--single clients or tools that do a single thing well--and almost
> > none
> > > of them are top level apache projects. I don't think trying to force
> each
> > > of these to turn into independent Apache projects is necessarily the
> best
> > > thing for the ecosystem.
> > >
> > > My observation of how this can go wrong is really what I think has
> > happened
> > > to the Hadoop ecosystem. There you see quite a zoo of projects which
> all
> > > drift apart and don't quite work together well. Coordinating even
> simple
> > > changes and standardization across these is exceptionally difficult.
> The
> > > result is a bit of a mess for users--the pieces just don't really come
> > > together very well. This makes sense for independent infrastructure
> > systems
> > > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > > little tool or helper library has lead to a desirable state. I think
> the
> > > mode of operating where the Hadoop vendors spawn off a few new Apache
> > > projects for each new product initiative, especially since often that
> > > project is only valued by that vendor (and the other vendor has a
> > different
> > > competing Apache project) doesn't necessarily do a better job at
> > producing
> > > high quality communities or high quality software.
> > >
> > > These tools/connects/clients/proxies and other integration pieces can
> > take
> > > many forms, but my take of what makes one of these things good is that
> it
> > > remains simple, does its one thing well, and cleaves as closely as
> > possible
> > > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > > monitoring, configuring, etc. For the tools we've contributed we've
> tried
> > > really hard to make them consistent with Kafka as well as with each
> other
> > > in how testing, configuration, monitoring, etc works.
> > >
> > > I think what Apache does superbly well is create a community for
> > managing a
> > > large infrastructure layer like Kafka in a vendor independent way. What
> I
> > > think is less successful is attempting to form full and independent
> > apache
> > > communities around very simple single purpose tools, especially if you
> > hope
> > > for these to come together into a cohesive toolset across multiple such
> > > tools. Much of what Apache does--create a collective decision making
> > > process for resolving disagreement, help to trademark and protect the
> > marks
> > > of the project, etc just isn't that relevant for simple single-purpose
> > > tools.
> > >
> > > So my take is there are a couple of options:
> > >
> > > 1. We can try to put all the small tools into the Apache Project. I
> > > think this is not the right approach as there is simply too many of
> > > them,
> > > many in different languages, serving different protocols,
> integrating
> > > with
> > > particular systems, and a single community can't effectively
> maintain
> > > them
> > > all. Doing this would significantly slow the progress of the Kafka
> > > project.
> > > As a protocol for messaging, I don't really see a case for including
> > > REST
> > > but not MQTT or AMQP which are technically much better suited to
> > > messaging
> > > and both are widely used for that.
> > > 2. We can treat ecosystem projects that aren't top level Apache
> > projects
> > > as invalid and try to recreate them all as Apache projects.
> Honestly,
> > > though, if you go to the Kafka ecosystem page virtually none of the
> > most
> > > popular add-ons to Kafka are Apache projects. The most successful
> > > things in
> > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> > other
> > > clients, as well as the existing REST layer have succeeded at
> > developing
> > > communities that actively contribute and use these pieces and I
> don't
> > > know
> > > that that is a bad thing unless that community proves to be
> > uninclusive,
> > > unresponsive, or goes in a bad technical direction--and those are
> > > failure
> > > modes that all open source efforts face.
> > > 3. We can do what I think makes the most sense and try to work with
> > the
> > > projects that exist in the ecosystem and if the project doesn't have
> a
> > > responsive community or wants to go in a different direction fork or
> > > recreate that work.
> > >
> > > Of course any person can choose whatever of these options they want.
> But
> > > from my point of view, option (3) has been the path of the community so
> > far
> > > and I think it has been quite successful.
> > >
> > > -Jay
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <kafka@harsha.io
> >
> > > wrote:
> > >
> > > > Neha,
> > > > "But I haven't seen any community emails or patches being submitted
> by
> > > you
> > > > guys, so I'm wondering why you are concerned about whether the
> > community
> > > is
> > > > open to accepting patches or not."
> > > >
> > > > I think you are talking about contributing patches to this repository
> > > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > > the guidelines/governance model is not clear on the project and I
> guess
> > > its
> > > > driven by opening a github issue request. Its the repository owned
> by
> > > > confluent and as much I appreciate that the features we mentioned are
> > in
> > > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > > gurantee what we want to add in the furture will be in your roadmap.
> > > >
> > > > Hence the reason having it part of Kafka community will help a lot as
> > > other
> > > > users can participate in the discussions. We are happy to drive any
> > > > feature additions through KIPs which gives everyone a chance to
> > > participate
> > > > and add to the discussions.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > I agree on the governance comments whole heartedly.
> > > > >
> > > > > Also i agree about the contribution comments made earlier in the
> > > thread,
> > > > i
> > > > > personally am less likely to spend any of mine, or give project
> time
> > > > within
> > > > > my internal projects to developers contributing to another
> commercial
> > > > > companies project even so technically open source, as then there is
> > > that
> > > > > commercial companies interest will always prevail and essentially
> can
> > > > > always have a final vote where disagreement. Im sure they never
> > intend
> > > > to,
> > > > > but there is that true reality. This is why we have community open
> > > source
> > > > > projects.
> > > > >
> > > > > I can find many different implementations now of a rest endpoint on
> > > > > GitHub, BitBucket etc. Each one has their benefits and
> disadvantages
> > in
> > > > > their implementation. By making / providing one this would bring
> > > together
> > > > > these solutions, unifying those developers and also bringing the
> best
> > > of
> > > > > all.
> > > > >
> > > > > I understand the concern on the community burden adding/supporting
> > more
> > > > > surface area for every client. But something like REST is universal
> > and
> > > > > worthy to be owned by the community.
> > > > >
> > > > > Mike
> > > > >
> > > > >
> > > > > ________________________________________
> > > > > From: Andrew Schofield <an...@outlook.com>
> > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >
> > > > > There's a massive difference between the governance of Kafka and
> the
> > > > > governance of the REST proxy.
> > > > >
> > > > > In Kafka, there is a broad community of people contributing their
> > > > opinions
> > > > > about future enhancements in the form of KIPs. There's some really
> > deep
> > > > > consideration that goes into some of the trickier KIPs. There are
> > > people
> > > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > > reputations to become committers. I get the impression that the
> > roadmap
> > > > of
> > > > > Kafka is not really community-owned (what's the big feature for
> Kafka
> > > > 0.11,
> > > > > for example), but the conveyor belt of smaller features in the form
> > of
> > > > KIPs
> > > > > works nicely. It's a good example of open-source working well.
> > > > >
> > > > > The equivalent for the REST proxy is basically issues on GitHub.
> The
> > > > > roadmap is less clear. There's not really a community properly
> > engaged
> > > in
> > > > > the way that there is with Kafka. So, you could say that it's clear
> > > that
> > > > > fewer people are interested, but I think the whole governance
> thing
> > > is a
> > > > > big barrier to engagement. And it's looking like it's getting out
> of
> > > > date.
> > > > >
> > > > > In technical terms, I can think of two big improvements to the REST
> > > > proxy.
> > > > > First, it needs to use the new consumer API so that it's possible
> to
> > > > secure
> > > > > connections between the REST proxy and Kafka. I don't care too much
> > > which
> > > > > method calls it uses actually uses to consume messages, but I do
> > care
> > > > that
> > > > > I cannot secure connections because of network security rules.
> > Second,
> > > > > there's an affinity between a consumer and the instance of the REST
> > > proxy
> > > > > to which it first connected. Kafka itself avoids this kind of
> > affinity
> > > > for
> > > > > good reason, and in the name of availability the REST proxy should
> > too.
> > > > > These are natural KIPs.
> > > > >
> > > > > I think it would be good to have the code for the REST proxy
> > > contributed
> > > > > to Apache Kafka so that it would be able to be developed in the
> same
> > > way.
> > > > >
> > > > > Andrew Schofield
> > > > >
> > > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > > Sent: 07 October 2016 22:41:52
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >
> > > > > ASF already gives us a clear framework and governance model for
> > > community
> > > > > development. This is already understood by the people contributing
> to
> > > > > Apache Kafka project, and they are the same people who want to
> > > contribute
> > > > > to the REST server capability as well. Everyone is in agreement on
> > the
> > > > > need for collaborating on this effort. So why not contribute the
> code
> > > to
> > > > > Apache Kafka. This will help avoid duplication of effort and forks
> > that
> > > > > may crop up, hugely benefitting the user community. This will also
> > > avoid
> > > > > having to define a process similar to ASF on a GitHub project and
> > > instead
> > > > > there is a single community with clear understanding community
> > process
> > > as
> > > > > defined in ASF.
> > > > >
> > > > > As others have said, this is an important capability for Apache
> > Kafka.
> > > It
> > > > > is worth maintaining this as a part of the project.
> > > > >
> > > > > Regards,
> > > > > Suresh
> > > > >
> > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > > >
> > > > > >I personally think it would be quite wasteful to re-implement the
> > REST
> > > > > >gateway just because that an actively-maintained piece of
> > > > Apache-licensed
> > > > > >software is not governed directly by the Apache Kafka community...
> > > While
> > > > > >kafka-rest repo is owned by Confluent, the contributors including
> > the
> > > > main
> > > > > >one are also part of the Apache Kafka community, so there is a
> > chance
> > > > to
> > > > > >work this out.
> > > > > >
> > > > > >However, there are two valid concerns here that could be
> addressed,
> > > > around
> > > > > >community and accessibility:
> > > > > >>> What we are worried about is a project
> > > > > >>> that's not maintained in a community. So the process of
> accepting
> > > > > >>>patches
> > > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > > >>>community.
> > > > > >>> Not only that, existing REST API project doesn't support new
> > client
> > > > API
> > > > > >and
> > > > > >>> hence there is no security support either.
> > > > > >
> > > > > >This might be easy to fix. Maybe Confluent / kafka-rest community
> > can
> > > > > >clarify that - what is their contribution policy, dev style,
> roadmap
> > > > etc.
> > > > > >If they want, they can make an effort to encourage participation
> > from
> > > > > >people outside Confluent (easily accept contributions, invite
> > external
> > > > > >commiters or have open dev process similar to Apache Kafka etc),
> as
> > > > there
> > > > > >is definitely seems to be some interest on the list. That might
> > clear
> > > > the
> > > > > >community concern and help kafka-rest project (but that is a
> > > calculation
> > > > > >Confluent will have to make).
> > > > > >
> > > > > >The other, independent, concern is that REST is something that is
> > > > expected
> > > > > >to be available out of the box with Kafka. I personally don't feel
> > > > > >strongly
> > > > > >about it (better use proper, efficient APIs from day one), though
> it
> > > is
> > > > > >definitely way smaller than adding a stream processing engine to
> the
> > > > > >project :)
> > > > > >Again,the kafka-rest "community" could take steps to make it even
> > > easier
> > > > > >to
> > > > > >install, configure and run kafka-rest for new users on vanilla
> > Apache
> > > > > >Kafka
> > > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > > >contributions to that end), but that is up to them.
> > > > > >Finally, if after the above steps were taken there would still a
> > > strong
> > > > > >desire to include a great rest gateway with Apache Kafka, I assume
> > the
> > > > > >community could hypothetically fork the existing kafka-rest into
> an
> > > > Apache
> > > > > >Kafka subproject and maintain it "within Apache" instead of
> > > implementing
> > > > > >it
> > > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine
> it
> > > > > >happen
> > > > > >without Confluent blessing, and I think that is likely much less
> > > optimal
> > > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > > having
> > > > a
> > > > > >separate external community around kafka-rest.
> > > > > >
> > > > > >
> > > > > >Just my two cents,
> > > > > >
> > > > > >
> > > > > >Ofir Manor
> > > > > >
> > > > > >Co-Founder & CTO | Equalum
> > > > > >
> > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> <+972%2054-780-1286> <+972%2054-780-1286>
> | Email:
> > > > > ofir.manor@equalum.io
> > > > > >
> > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > kafka@harsha.io
> > > >
> > > > > >wrote:
> > > > > >
> > > > > >> Neha & Jay,
> > > > > >> We did look at the open source alternatives.
> Our
> > > > > >>concern
> > > > > >> is what's the patch acceptance and adding features/ bug-fixes to
> > the
> > > > > >> existing project under a Github (although it's licensed under
> > Apache
> > > > > >>2.0).
> > > > > >> It would be great if that project made available under Apache
> and
> > > > > >>driven by
> > > > > >> the community. Adding to the above, not all Kafka users are
> > > > interested
> > > > > >>in
> > > > > >> using the Java client API, they would like to have simple REST
> API
> > > > where
> > > > > >> they can code against using any language. I do believe this adds
> > > value
> > > > > >>to
> > > > > >> Apache Kafka in itself.
> > > > > >>
> > > > > >> "For 1, I don't think there is value in giving in to the NIH
> > > syndrome
> > > > > >>and
> > > > > >> reinventing the wheel. What I'm looking for is a detailed
> > comparison
> > > > of
> > > > > >>the
> > > > > >> gaps and why those can't be improved in the REST proxy that
> > already
> > > > > >>exists
> > > > > >> and is actively maintained."
> > > > > >>
> > > > > >> We are not looking at this as NIH. What we are worried about is
> a
> > > > > >>project
> > > > > >> that's not maintained in a community. So the process of
> accepting
> > > > > >>patches
> > > > > >> and priorities is not clear, and it's not developed in Apache
> > > > community.
> > > > > >> Not only that, existing REST API project doesn't support new
> > client
> > > > API
> > > > > >>and
> > > > > >> hence there is no security support either.
> > > > > >> We don't know the timeline when that's made available. We would
> > like
> > > > to
> > > > > >>add
> > > > > >> admin functionality into the REST API. So the Roadmap of that
> > > project
> > > > is
> > > > > >> not driven by Apache.
> > > > > >>
> > > > > >>
> > > > > >> "This doesn't materially have an impact on expanding the
> usability
> > > of
> > > > > >> Kafka. In my experience, REST proxy + Java clients only cover
> > > ~50%
> > > > of
> > > > > >> all
> > > > > >> Kafka users, and maybe 10% of those are the ones who will use
> > the
> > > > > >>REST
> > > > > >> proxy. The remaining 50% are non-java client users (C,
> python,
> > > go,
> > > > > >>node
> > > > > >> etc)."
> > > > > >>
> > > > > >> REST API is most often asked feature in my interactions with
> Kafka
> > > > > >>users.
> > > > > >> In an organization, There will be independent teams who will
> write
> > > > their
> > > > > >> Kafka clients using different language libraries available
> today,
> > > and
> > > > > >> there is no way to standardize this. Instead of supporting
> several
> > > > > >> different client libraries users will be interested in using a
> > REST
> > > > API
> > > > > >> server. The need for a REST API will only increase as more and
> > more
> > > > > >>users
> > > > > >> start using Kafka.
> > > > > >>
> > > > > >> "More surface area means more work to keep things consistent.
> > > Failure
> > > > > >> to do that has, in fact, hurt the user experience."
> > > > > >> Having myriad Kafka client GitHub projects that support
> different
> > > > > >>languages
> > > > > >> hurts the user experience and pushes burden to maintain these
> > > > libraries.
> > > > > >> REST API is a simple code base that uses existing java client
> > > > libraries
> > > > > >>to
> > > > > >> make life easier for the users.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Harsha
> > > > > >>
> > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> neha@confluent.io>
> > > > > wrote:
> > > > > >>
> > > > > >> > Manikumar,
> > > > > >> >
> > > > > >> > Thanks for sharing the proposal. I think there are 2 parts to
> > this
> > > > > >> > discussion -
> > > > > >> >
> > > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > > >>feature-complete,
> > > > > >> > open-source and actively maintained REST proxy in the
> community?
> > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile
> > and
> > > > > >> maintain
> > > > > >> > the high-quality experience that Kafka users have today?
> > > > > >> >
> > > > > >> > For 1, I don't think there is value in giving in to the NIH
> > > syndrome
> > > > > >>and
> > > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > > comparison
> > > > > >>of
> > > > > >> the
> > > > > >> > gaps and why those can't be improved in the REST proxy that
> > > already
> > > > > >> exists
> > > > > >> > and is actively maintained. For example, we depend on zkClient
> > and
> > > > > >>have
> > > > > >> > found as well as fixed several bugs by working closely with
> the
> > > > people
> > > > > >> who
> > > > > >> > maintain zkClient. This should be possible for REST proxy as
> > well,
> > > > > >>right?
> > > > > >> >
> > > > > >> > For 2, I'd like us to review our history of expanding the
> > surface
> > > > > >>area to
> > > > > >> > add more clients in the past. Here is a summary -
> > > > > >> >
> > > > > >> > - This doesn't materially have an impact on expanding the
> > > > > >>usability of
> > > > > >> > Kafka. In my experience, REST proxy + Java clients only
> cover
> > > > ~50%
> > > > > >>of
> > > > > >> > all
> > > > > >> > Kafka users, and maybe 10% of those are the ones who will
> use
> > > the
> > > > > >>REST
> > > > > >> > proxy. The remaining 50% are non-java client users (C,
> > python,
> > > > go,
> > > > > >> node
> > > > > >> > etc).
> > > > > >> > - People are a lot more excited about promising to
> contribute
> > > > while
> > > > > >> > adding the surface area but not on an ongoing basis down
> the
> > > > line.
> > > > > >> > - More surface area means more work to keep things
> > consistent.
> > > > > >>Failure
> > > > > >> > to do that has, in fact, hurt the user experience.
> > > > > >> > - More surface area hurts agility. We want to do a few
> things
> > > > > >>really
> > > > > >> > well as well as be agile to be able to build on our core
> > > > > >>competency.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > manikumar.reddy@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi Jay,
> > > > > >> > >
> > > > > >> > > Thanks for your reply.
> > > > > >> > >
> > > > > >> > > I agree that we can not add all the clients/tools available
> in
> > > > > >> ecosystem
> > > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > > different
> > > > > >>from
> > > > > >> > > other clients/tools. Since any language that can work with
> > HTTP
> > > > can
> > > > > >> > > easily integrate with this interface, Having an "official"
> > REST
> > > > > >> > > interface helps user community. This also helps us to
> > integrate
> > > > well
> > > > > >> > > with external management and provisioning tools. Apache
> Kafka
> > > > > >>release
> > > > > >> > > with Java clients + REST interface is sufficient for most of
> > the
> > > > > >>user
> > > > > >> > > deployments/requirements. This helps users to deal with less
> > > > number
> > > > > >> > > of distributions/builds.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Manikumar
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <jay@confluent.io
> >
> > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hey guys,
> > > > > >> > > >
> > > > > >> > > > There's already a REST interface maintained as a separate
> > > > > >> project--it's
> > > > > >> > > > open source and apache licensed and actively maintained (
> > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is
> wrong
> > > with
> > > > >
> > > > >
> > > > >
> > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > > github.com
> > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
> cluster.
> > > It
> > > > > makes it easy to produce and consume messages, view the state of
> the
> > > > > cluster, and ...
> > > > >
> > > > > >> that?
> > > > > >> > > You
> > > > > >> > > > mentioned that there was some compatibility concern, but
> > > > > >> compatibility
> > > > > >> > > has
> > > > > >> > > > to do with the consumer protocol guarantees not the repo
> the
> > > > code
> > > > > >>is
> > > > > >> > in,
> > > > > >> > > > right? Not sure that concern makes sense.
> > > > > >> > > >
> > > > > >> > > > We could argue for adding pretty much anything and
> > everything
> > > in
> > > > > >>the
> > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> > > make
> > > > > >>the
> > > > > >> > > project
> > > > > >> > > > more agile.
> > > > > >> > > >
> > > > > >> > > > -Jay
> > > > > >> > > >
> > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > > >> manikumar.reddy@gmail.com
> > > > > >> > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Kafka Devs,
> > > > > >> > > > >
> > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > > Repository.
> > > > > >> > > > >
> > > > > >> > > > > There are already open-source alternatives are
> available.
> > > But
> > > > > >>we
> > > > > >> > would
> > > > > >> > > > > like to add REST server that
> > > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> > Infra
> > > > > >>tools
> > > > > >> > comes
> > > > > >> > > > up
> > > > > >> > > > > with Rest Interface.
> > > > > >> > > > > It is useful to have inbuilt Rest API support for
> Produce,
> > > > > >>Consume
> > > > > >> > > > messages
> > > > > >> > > > > and admin interface for
> > > > > >> > > > > integrating with external management and provisioning
> > > > tools.This
> > > > > >> will
> > > > > >> > > > also
> > > > > >> > > > > allow the maintenance of
> > > > > >> > > > > REST server and adding new features makes it easy
> because
> > > > apache
> > > > > >> > > > community.
> > > > > >> > > > >
> > > > > >> > > > > The KIP wiki is the following:
> > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > > >> > > > >
> > > > > >> > > > > Your comments and feedback are welcome.
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Manikumar
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > --
> > > > > >> > Thanks,
> > > > > >> > Neha
> > > > > >> >
> > > > > >>
> > > > > The information contained in this email is strictly confidential
> and
> > > for
> > > > > the use of the addressee only, unless otherwise indicated. If you
> are
> > > not
> > > > > the intended recipient, please do not read, copy, use or disclose
> to
> > > > others
> > > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to
> > > > the
> > > > > official business of this company shall be understood as neither
> > given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > > registered in England and Wales, company number 04008957) and IG
> > Index
> > > > > Limited (a company registered in England and Wales, company number
> > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> > > IG
> > > > > Index Limited (register number 114059) are authorised and regulated
> > by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Nacho (Ignacio) Solis
> > Kafka
> > nsolis@linkedin.com
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Jungtaek Lim <ka...@gmail.com>.
I guess no one doubts its power on REST server or even UI. I understand the
difficulty to add a module to project, but it's maximized when there is
less support expected hence maintenance issue is likely to rise, and IMHO
this seems to be not the case.
There're also pain points when project doesn't maintain features and
delegates to ecosystem. Based on some points (last commit date, pull
request open and closed, and contributor graph), kafka-manager seems to
have similar activity to kafka-rest, but it doesn't show any responses for
pull request supporting Kafka 0.10.0 even though numerous users leave
comments wish to support. What Kafka community can do for that project to
follow up? Nothing but just persuading by leaving comments hoping that will
be merged. (or finally come up another implementation) Kafka project keeps
agile but in point of whole ecosystem it can be less agile.
Yes decisions and roadmap of the project are driven by PMCs and I think
it's valid right. But we also imagine ASF projects as driven by community
aspect, though it's alike to ideal world. KIP makes innovation on adopting
new feature transparently, which makes many developers inspiring and
adopting it to their projects. Hopes that Kafka community continuously
drives the transparency model among the ASF projects, and beyond.
- Jungtaek Lim (HeartSaVioR)
2016년 10월 17일 (월) 오전 7:56, Jay Kreps <ja...@confluent.io>님이 작성:
Hey Nacho,
Yeah, I think it is definitely a call we have to make case by case. We have
some experience with this: originally we attempted to maintain things like
non-java clients, a hadoop connector, etc all in the main project. The
difficulty of that lead us to the current federated approach. In terms of
what is included now, yes, I agree you could potentially have even less
included.
-Jay
On Wed, Oct 12, 2016 at 11:37 AM, Nacho Solis <ns...@linkedin.com.invalid>
wrote:
> What is the criteria for keeping things in and out of Kafka, what code
goes
> in or out and what is part of the architecture or not?
>
> The discussion of what goes into a project and what stays out is an always
> evolving question. Different projects treat this in different ways.
>
> Let me paint 2 extremes. On one side, you have a single monolithic
project
> that brings everything in one tent. On the other side you have the many
> modules approach. From what I've learned, Kafka falls in the middle.
> Because of this, the question is bound to come up with respect to the
> criteria used to bring something into the fold.
>
> I'll be the first to point out that the distinction between modules,
> architecture, software, repositories, governance and community are blurry.
> Not to mention that many things are how they are for historical reasons.
>
> I, personally, can't understand why we would not have REST as part of the
> main Kafka project given that a lot of people use it and we include many
> things with the current distribution. What many things you may ask?
Well,
> if we took the modular approach Kafka is a mixture of components, here's
> the first 4 that come to mind:
> 1. The Kafka protocol
> 2. The Kafka java libraries
> 3. The Kafka broker
> 4. The Kafka stream framework
> 5. Kafka Connect
> 6. MirrorMaker
>
> All of these could be separate products. You should be able to evolve each
> one independently. Even if they have dependencies on each other, you
could
> potentially replace one part.
>
> The choice of keeping them all in a single repository, with a single
> distribution, under the same governance and community, brings a number of
> trade offs. It's easy to keep things coherent for example. There is less
> of a need to rely on inherent versioning and compatibility (which we end
up
> providing anyway because of the way people usually deploy kafka). We all
> focus our efforts on a single code base.
>
> The downside is that it's harder to remove modules that are old or unused.
> Modules that are only used by a small subset of the community will have an
> impact on the rest of the community. It mixes incentives of what people
> want to work on and what holds them back. We also need to decide what
> belongs in the blessed bundle and what doesnt.
>
> So, my question boils down to, what criteria is used for bringing stuff
in.
>
> If we have Streams and MirrorMaker and Connect in there, why not have
REST?
> Specially if there is more than one person/group willing to work on it?
> Alternatively, if REST is not included because it's not used by all, then
> why not remove Streams, Connect and MirrorMaker since they're definitely
> not used by all? I realize I say this even though at LinkedIn we have a
> REST setup of our own, just speaking from a community perspective.
>
> Nacho
>
>
> (I'm relatively new and I haven't read all of the mail archive, so I'm
sure
> this has been brought up before, but I decided to chime in anyway)
>
> On Wed, Oct 12, 2016 at 8:03 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not really
> > make sense to me. If in the future we disagree, that is the beauty of
> open
> > source, you can always fork off a copy of the code and start an
> independent
> > project either in Apache or elsewhere. Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be
done
> > and we have an existing code base for HTTP/kafka access that is heavily
> > used in production seems quite silly.
> >
> > Let me give some background on how I at least think about these things.
> > I've participated in open source projects out of LinkedIn via github as
> > well as via the ASF. I don't think there is a "right" answer to how to
do
> > these but rather some tradeoffs. We thought about this quite a lot in
the
> > context of Kafka based on the experience with the Hadoop ecosystem as
> well
> > as from other open source communities.
> >
> > There is a rich ecosystem around Kafka. Many of the projects are quite
> > small--single clients or tools that do a single thing well--and almost
> none
> > of them are top level apache projects. I don't think trying to force
each
> > of these to turn into independent Apache projects is necessarily the
best
> > thing for the ecosystem.
> >
> > My observation of how this can go wrong is really what I think has
> happened
> > to the Hadoop ecosystem. There you see quite a zoo of projects which all
> > drift apart and don't quite work together well. Coordinating even simple
> > changes and standardization across these is exceptionally difficult. The
> > result is a bit of a mess for users--the pieces just don't really come
> > together very well. This makes sense for independent infrastructure
> systems
> > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > little tool or helper library has lead to a desirable state. I think the
> > mode of operating where the Hadoop vendors spawn off a few new Apache
> > projects for each new product initiative, especially since often that
> > project is only valued by that vendor (and the other vendor has a
> different
> > competing Apache project) doesn't necessarily do a better job at
> producing
> > high quality communities or high quality software.
> >
> > These tools/connects/clients/proxies and other integration pieces can
> take
> > many forms, but my take of what makes one of these things good is that
it
> > remains simple, does its one thing well, and cleaves as closely as
> possible
> > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > monitoring, configuring, etc. For the tools we've contributed we've
tried
> > really hard to make them consistent with Kafka as well as with each
other
> > in how testing, configuration, monitoring, etc works.
> >
> > I think what Apache does superbly well is create a community for
> managing a
> > large infrastructure layer like Kafka in a vendor independent way. What
I
> > think is less successful is attempting to form full and independent
> apache
> > communities around very simple single purpose tools, especially if you
> hope
> > for these to come together into a cohesive toolset across multiple such
> > tools. Much of what Apache does--create a collective decision making
> > process for resolving disagreement, help to trademark and protect the
> marks
> > of the project, etc just isn't that relevant for simple single-purpose
> > tools.
> >
> > So my take is there are a couple of options:
> >
> > 1. We can try to put all the small tools into the Apache Project. I
> > think this is not the right approach as there is simply too many of
> > them,
> > many in different languages, serving different protocols, integrating
> > with
> > particular systems, and a single community can't effectively maintain
> > them
> > all. Doing this would significantly slow the progress of the Kafka
> > project.
> > As a protocol for messaging, I don't really see a case for including
> > REST
> > but not MQTT or AMQP which are technically much better suited to
> > messaging
> > and both are widely used for that.
> > 2. We can treat ecosystem projects that aren't top level Apache
> projects
> > as invalid and try to recreate them all as Apache projects. Honestly,
> > though, if you go to the Kafka ecosystem page virtually none of the
> most
> > popular add-ons to Kafka are Apache projects. The most successful
> > things in
> > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> other
> > clients, as well as the existing REST layer have succeeded at
> developing
> > communities that actively contribute and use these pieces and I don't
> > know
> > that that is a bad thing unless that community proves to be
> uninclusive,
> > unresponsive, or goes in a bad technical direction--and those are
> > failure
> > modes that all open source efforts face.
> > 3. We can do what I think makes the most sense and try to work with
> the
> > projects that exist in the ecosystem and if the project doesn't have
a
> > responsive community or wants to go in a different direction fork or
> > recreate that work.
> >
> > Of course any person can choose whatever of these options they want. But
> > from my point of view, option (3) has been the path of the community so
> far
> > and I think it has been quite successful.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Neha,
> > > "But I haven't seen any community emails or patches being submitted by
> > you
> > > guys, so I'm wondering why you are concerned about whether the
> community
> > is
> > > open to accepting patches or not."
> > >
> > > I think you are talking about contributing patches to this repository
> > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > the guidelines/governance model is not clear on the project and I
guess
> > its
> > > driven by opening a github issue request. Its the repository owned by
> > > confluent and as much I appreciate that the features we mentioned are
> in
> > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > gurantee what we want to add in the furture will be in your roadmap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot as
> > other
> > > users can participate in the discussions. We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project time
> > > within
> > > > my internal projects to developers contributing to another
commercial
> > > > companies project even so technically open source, as then there is
> > that
> > > > commercial companies interest will always prevail and essentially
can
> > > > always have a final vote where disagreement. Im sure they never
> intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint on
> > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages
> in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the
best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
> more
> > > > surface area for every client. But something like REST is universal
> and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > ________________________________________
> > > > From: Andrew Schofield <an...@outlook.com>
> > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > There's a massive difference between the governance of Kafka and the
> > > > governance of the REST proxy.
> > > >
> > > > In Kafka, there is a broad community of people contributing their
> > > opinions
> > > > about future enhancements in the form of KIPs. There's some really
> deep
> > > > consideration that goes into some of the trickier KIPs. There are
> > people
> > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > reputations to become committers. I get the impression that the
> roadmap
> > > of
> > > > Kafka is not really community-owned (what's the big feature for
Kafka
> > > 0.11,
> > > > for example), but the conveyor belt of smaller features in the form
> of
> > > KIPs
> > > > works nicely. It's a good example of open-source working well.
> > > >
> > > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > > roadmap is less clear. There's not really a community properly
> engaged
> > in
> > > > the way that there is with Kafka. So, you could say that it's clear
> > that
> > > > fewer people are interested, but I think the whole governance thing
> > is a
> > > > big barrier to engagement. And it's looking like it's getting out of
> > > date.
> > > >
> > > > In technical terms, I can think of two big improvements to the REST
> > > proxy.
> > > > First, it needs to use the new consumer API so that it's possible to
> > > secure
> > > > connections between the REST proxy and Kafka. I don't care too much
> > which
> > > > method calls it uses actually uses to consume messages, but I do
> care
> > > that
> > > > I cannot secure connections because of network security rules.
> Second,
> > > > there's an affinity between a consumer and the instance of the REST
> > proxy
> > > > to which it first connected. Kafka itself avoids this kind of
> affinity
> > > for
> > > > good reason, and in the name of availability the REST proxy should
> too.
> > > > These are natural KIPs.
> > > >
> > > > I think it would be good to have the code for the REST proxy
> > contributed
> > > > to Apache Kafka so that it would be able to be developed in the same
> > way.
> > > >
> > > > Andrew Schofield
> > > >
> > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > Sent: 07 October 2016 22:41:52
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > ASF already gives us a clear framework and governance model for
> > community
> > > > development. This is already understood by the people contributing
to
> > > > Apache Kafka project, and they are the same people who want to
> > contribute
> > > > to the REST server capability as well. Everyone is in agreement on
> the
> > > > need for collaborating on this effort. So why not contribute the
code
> > to
> > > > Apache Kafka. This will help avoid duplication of effort and forks
> that
> > > > may crop up, hugely benefitting the user community. This will also
> > avoid
> > > > having to define a process similar to ASF on a GitHub project and
> > instead
> > > > there is a single community with clear understanding community
> process
> > as
> > > > defined in ASF.
> > > >
> > > > As others have said, this is an important capability for Apache
> Kafka.
> > It
> > > > is worth maintaining this as a part of the project.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > >
> > > > >I personally think it would be quite wasteful to re-implement the
> REST
> > > > >gateway just because that an actively-maintained piece of
> > > Apache-licensed
> > > > >software is not governed directly by the Apache Kafka community...
> > While
> > > > >kafka-rest repo is owned by Confluent, the contributors including
> the
> > > main
> > > > >one are also part of the Apache Kafka community, so there is a
> chance
> > > to
> > > > >work this out.
> > > > >
> > > > >However, there are two valid concerns here that could be addressed,
> > > around
> > > > >community and accessibility:
> > > > >>> What we are worried about is a project
> > > > >>> that's not maintained in a community. So the process of
accepting
> > > > >>>patches
> > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > >>>community.
> > > > >>> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >and
> > > > >>> hence there is no security support either.
> > > > >
> > > > >This might be easy to fix. Maybe Confluent / kafka-rest community
> can
> > > > >clarify that - what is their contribution policy, dev style,
roadmap
> > > etc.
> > > > >If they want, they can make an effort to encourage participation
> from
> > > > >people outside Confluent (easily accept contributions, invite
> external
> > > > >commiters or have open dev process similar to Apache Kafka etc), as
> > > there
> > > > >is definitely seems to be some interest on the list. That might
> clear
> > > the
> > > > >community concern and help kafka-rest project (but that is a
> > calculation
> > > > >Confluent will have to make).
> > > > >
> > > > >The other, independent, concern is that REST is something that is
> > > expected
> > > > >to be available out of the box with Kafka. I personally don't feel
> > > > >strongly
> > > > >about it (better use proper, efficient APIs from day one), though
it
> > is
> > > > >definitely way smaller than adding a stream processing engine to
the
> > > > >project :)
> > > > >Again,the kafka-rest "community" could take steps to make it even
> > easier
> > > > >to
> > > > >install, configure and run kafka-rest for new users on vanilla
> Apache
> > > > >Kafka
> > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > >contributions to that end), but that is up to them.
> > > > >Finally, if after the above steps were taken there would still a
> > strong
> > > > >desire to include a great rest gateway with Apache Kafka, I assume
> the
> > > > >community could hypothetically fork the existing kafka-rest into an
> > > Apache
> > > > >Kafka subproject and maintain it "within Apache" instead of
> > implementing
> > > > >it
> > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine
it
> > > > >happen
> > > > >without Confluent blessing, and I think that is likely much less
> > optimal
> > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > having
> > > a
> > > > >separate external community around kafka-rest.
> > > > >
> > > > >
> > > > >Just my two cents,
> > > > >
> > > > >
> > > > >Ofir Manor
> > > > >
> > > > >Co-Founder & CTO | Equalum
> > > > >
> > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> <+972%2054-780-1286>
| Email:
> > > > ofir.manor@equalum.io
> > > > >
> > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > > > >wrote:
> > > > >
> > > > >> Neha & Jay,
> > > > >> We did look at the open source alternatives. Our
> > > > >>concern
> > > > >> is what's the patch acceptance and adding features/ bug-fixes to
> the
> > > > >> existing project under a Github (although it's licensed under
> Apache
> > > > >>2.0).
> > > > >> It would be great if that project made available under Apache and
> > > > >>driven by
> > > > >> the community. Adding to the above, not all Kafka users are
> > > interested
> > > > >>in
> > > > >> using the Java client API, they would like to have simple REST
API
> > > where
> > > > >> they can code against using any language. I do believe this adds
> > value
> > > > >>to
> > > > >> Apache Kafka in itself.
> > > > >>
> > > > >> "For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> reinventing the wheel. What I'm looking for is a detailed
> comparison
> > > of
> > > > >>the
> > > > >> gaps and why those can't be improved in the REST proxy that
> already
> > > > >>exists
> > > > >> and is actively maintained."
> > > > >>
> > > > >> We are not looking at this as NIH. What we are worried about is
a
> > > > >>project
> > > > >> that's not maintained in a community. So the process of accepting
> > > > >>patches
> > > > >> and priorities is not clear, and it's not developed in Apache
> > > community.
> > > > >> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >>and
> > > > >> hence there is no security support either.
> > > > >> We don't know the timeline when that's made available. We would
> like
> > > to
> > > > >>add
> > > > >> admin functionality into the REST API. So the Roadmap of that
> > project
> > > is
> > > > >> not driven by Apache.
> > > > >>
> > > > >>
> > > > >> "This doesn't materially have an impact on expanding the
usability
> > of
> > > > >> Kafka. In my experience, REST proxy + Java clients only cover
> > ~50%
> > > of
> > > > >> all
> > > > >> Kafka users, and maybe 10% of those are the ones who will use
> the
> > > > >>REST
> > > > >> proxy. The remaining 50% are non-java client users (C, python,
> > go,
> > > > >>node
> > > > >> etc)."
> > > > >>
> > > > >> REST API is most often asked feature in my interactions with
Kafka
> > > > >>users.
> > > > >> In an organization, There will be independent teams who will
write
> > > their
> > > > >> Kafka clients using different language libraries available
today,
> > and
> > > > >> there is no way to standardize this. Instead of supporting
several
> > > > >> different client libraries users will be interested in using a
> REST
> > > API
> > > > >> server. The need for a REST API will only increase as more and
> more
> > > > >>users
> > > > >> start using Kafka.
> > > > >>
> > > > >> "More surface area means more work to keep things consistent.
> > Failure
> > > > >> to do that has, in fact, hurt the user experience."
> > > > >> Having myriad Kafka client GitHub projects that support different
> > > > >>languages
> > > > >> hurts the user experience and pushes burden to maintain these
> > > libraries.
> > > > >> REST API is a simple code base that uses existing java client
> > > libraries
> > > > >>to
> > > > >> make life easier for the users.
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > > > wrote:
> > > > >>
> > > > >> > Manikumar,
> > > > >> >
> > > > >> > Thanks for sharing the proposal. I think there are 2 parts to
> this
> > > > >> > discussion -
> > > > >> >
> > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > >>feature-complete,
> > > > >> > open-source and actively maintained REST proxy in the
community?
> > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile
> and
> > > > >> maintain
> > > > >> > the high-quality experience that Kafka users have today?
> > > > >> >
> > > > >> > For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > comparison
> > > > >>of
> > > > >> the
> > > > >> > gaps and why those can't be improved in the REST proxy that
> > already
> > > > >> exists
> > > > >> > and is actively maintained. For example, we depend on zkClient
> and
> > > > >>have
> > > > >> > found as well as fixed several bugs by working closely with the
> > > people
> > > > >> who
> > > > >> > maintain zkClient. This should be possible for REST proxy as
> well,
> > > > >>right?
> > > > >> >
> > > > >> > For 2, I'd like us to review our history of expanding the
> surface
> > > > >>area to
> > > > >> > add more clients in the past. Here is a summary -
> > > > >> >
> > > > >> > - This doesn't materially have an impact on expanding the
> > > > >>usability of
> > > > >> > Kafka. In my experience, REST proxy + Java clients only
cover
> > > ~50%
> > > > >>of
> > > > >> > all
> > > > >> > Kafka users, and maybe 10% of those are the ones who will
use
> > the
> > > > >>REST
> > > > >> > proxy. The remaining 50% are non-java client users (C,
> python,
> > > go,
> > > > >> node
> > > > >> > etc).
> > > > >> > - People are a lot more excited about promising to
contribute
> > > while
> > > > >> > adding the surface area but not on an ongoing basis down the
> > > line.
> > > > >> > - More surface area means more work to keep things
> consistent.
> > > > >>Failure
> > > > >> > to do that has, in fact, hurt the user experience.
> > > > >> > - More surface area hurts agility. We want to do a few
things
> > > > >>really
> > > > >> > well as well as be agile to be able to build on our core
> > > > >>competency.
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Jay,
> > > > >> > >
> > > > >> > > Thanks for your reply.
> > > > >> > >
> > > > >> > > I agree that we can not add all the clients/tools available
in
> > > > >> ecosystem
> > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > different
> > > > >>from
> > > > >> > > other clients/tools. Since any language that can work with
> HTTP
> > > can
> > > > >> > > easily integrate with this interface, Having an "official"
> REST
> > > > >> > > interface helps user community. This also helps us to
> integrate
> > > well
> > > > >> > > with external management and provisioning tools. Apache
Kafka
> > > > >>release
> > > > >> > > with Java clients + REST interface is sufficient for most of
> the
> > > > >>user
> > > > >> > > deployments/requirements. This helps users to deal with less
> > > number
> > > > >> > > of distributions/builds.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Manikumar
> > > > >> > >
> > > > >> > >
> > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> > > wrote:
> > > > >> > >
> > > > >> > > > Hey guys,
> > > > >> > > >
> > > > >> > > > There's already a REST interface maintained as a separate
> > > > >> project--it's
> > > > >> > > > open source and apache licensed and actively maintained (
> > > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong
> > with
> > > >
> > > >
> > > >
> > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > github.com
> > > > The Kafka REST Proxy provides a RESTful interface to a Kafka
cluster.
> > It
> > > > makes it easy to produce and consume messages, view the state of the
> > > > cluster, and ...
> > > >
> > > > >> that?
> > > > >> > > You
> > > > >> > > > mentioned that there was some compatibility concern, but
> > > > >> compatibility
> > > > >> > > has
> > > > >> > > > to do with the consumer protocol guarantees not the repo
the
> > > code
> > > > >>is
> > > > >> > in,
> > > > >> > > > right? Not sure that concern makes sense.
> > > > >> > > >
> > > > >> > > > We could argue for adding pretty much anything and
> everything
> > in
> > > > >>the
> > > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> > make
> > > > >>the
> > > > >> > > project
> > > > >> > > > more agile.
> > > > >> > > >
> > > > >> > > > -Jay
> > > > >> > > >
> > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > >> manikumar.reddy@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Kafka Devs,
> > > > >> > > > >
> > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > Repository.
> > > > >> > > > >
> > > > >> > > > > There are already open-source alternatives are available.
> > But
> > > > >>we
> > > > >> > would
> > > > >> > > > > like to add REST server that
> > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> Infra
> > > > >>tools
> > > > >> > comes
> > > > >> > > > up
> > > > >> > > > > with Rest Interface.
> > > > >> > > > > It is useful to have inbuilt Rest API support for
Produce,
> > > > >>Consume
> > > > >> > > > messages
> > > > >> > > > > and admin interface for
> > > > >> > > > > integrating with external management and provisioning
> > > tools.This
> > > > >> will
> > > > >> > > > also
> > > > >> > > > > allow the maintenance of
> > > > >> > > > > REST server and adding new features makes it easy because
> > > apache
> > > > >> > > > community.
> > > > >> > > > >
> > > > >> > > > > The KIP wiki is the following:
> > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > >> > > > >
> > > > >> > > > > Your comments and feedback are welcome.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Manikumar
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > --
> > > > >> > Thanks,
> > > > >> > Neha
> > > > >> >
> > > > >>
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you
are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete
the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
>
>
>
> --
> Nacho (Ignacio) Solis
> Kafka
> nsolis@linkedin.com
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Jay Kreps <ja...@confluent.io>.
Hey Nacho,
Yeah, I think it is definitely a call we have to make case by case. We have
some experience with this: originally we attempted to maintain things like
non-java clients, a hadoop connector, etc all in the main project. The
difficulty of that lead us to the current federated approach. In terms of
what is included now, yes, I agree you could potentially have even less
included.
-Jay
On Wed, Oct 12, 2016 at 11:37 AM, Nacho Solis <ns...@linkedin.com.invalid>
wrote:
> What is the criteria for keeping things in and out of Kafka, what code goes
> in or out and what is part of the architecture or not?
>
> The discussion of what goes into a project and what stays out is an always
> evolving question. Different projects treat this in different ways.
>
> Let me paint 2 extremes. On one side, you have a single monolithic project
> that brings everything in one tent. On the other side you have the many
> modules approach. From what I've learned, Kafka falls in the middle.
> Because of this, the question is bound to come up with respect to the
> criteria used to bring something into the fold.
>
> I'll be the first to point out that the distinction between modules,
> architecture, software, repositories, governance and community are blurry.
> Not to mention that many things are how they are for historical reasons.
>
> I, personally, can't understand why we would not have REST as part of the
> main Kafka project given that a lot of people use it and we include many
> things with the current distribution. What many things you may ask? Well,
> if we took the modular approach Kafka is a mixture of components, here's
> the first 4 that come to mind:
> 1. The Kafka protocol
> 2. The Kafka java libraries
> 3. The Kafka broker
> 4. The Kafka stream framework
> 5. Kafka Connect
> 6. MirrorMaker
>
> All of these could be separate products. You should be able to evolve each
> one independently. Even if they have dependencies on each other, you could
> potentially replace one part.
>
> The choice of keeping them all in a single repository, with a single
> distribution, under the same governance and community, brings a number of
> trade offs. It's easy to keep things coherent for example. There is less
> of a need to rely on inherent versioning and compatibility (which we end up
> providing anyway because of the way people usually deploy kafka). We all
> focus our efforts on a single code base.
>
> The downside is that it's harder to remove modules that are old or unused.
> Modules that are only used by a small subset of the community will have an
> impact on the rest of the community. It mixes incentives of what people
> want to work on and what holds them back. We also need to decide what
> belongs in the blessed bundle and what doesnt.
>
> So, my question boils down to, what criteria is used for bringing stuff in.
>
> If we have Streams and MirrorMaker and Connect in there, why not have REST?
> Specially if there is more than one person/group willing to work on it?
> Alternatively, if REST is not included because it's not used by all, then
> why not remove Streams, Connect and MirrorMaker since they're definitely
> not used by all? I realize I say this even though at LinkedIn we have a
> REST setup of our own, just speaking from a community perspective.
>
> Nacho
>
>
> (I'm relatively new and I haven't read all of the mail archive, so I'm sure
> this has been brought up before, but I decided to chime in anyway)
>
> On Wed, Oct 12, 2016 at 8:03 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not really
> > make sense to me. If in the future we disagree, that is the beauty of
> open
> > source, you can always fork off a copy of the code and start an
> independent
> > project either in Apache or elsewhere. Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be done
> > and we have an existing code base for HTTP/kafka access that is heavily
> > used in production seems quite silly.
> >
> > Let me give some background on how I at least think about these things.
> > I've participated in open source projects out of LinkedIn via github as
> > well as via the ASF. I don't think there is a "right" answer to how to do
> > these but rather some tradeoffs. We thought about this quite a lot in the
> > context of Kafka based on the experience with the Hadoop ecosystem as
> well
> > as from other open source communities.
> >
> > There is a rich ecosystem around Kafka. Many of the projects are quite
> > small--single clients or tools that do a single thing well--and almost
> none
> > of them are top level apache projects. I don't think trying to force each
> > of these to turn into independent Apache projects is necessarily the best
> > thing for the ecosystem.
> >
> > My observation of how this can go wrong is really what I think has
> happened
> > to the Hadoop ecosystem. There you see quite a zoo of projects which all
> > drift apart and don't quite work together well. Coordinating even simple
> > changes and standardization across these is exceptionally difficult. The
> > result is a bit of a mess for users--the pieces just don't really come
> > together very well. This makes sense for independent infrastructure
> systems
> > (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> > little tool or helper library has lead to a desirable state. I think the
> > mode of operating where the Hadoop vendors spawn off a few new Apache
> > projects for each new product initiative, especially since often that
> > project is only valued by that vendor (and the other vendor has a
> different
> > competing Apache project) doesn't necessarily do a better job at
> producing
> > high quality communities or high quality software.
> >
> > These tools/connects/clients/proxies and other integration pieces can
> take
> > many forms, but my take of what makes one of these things good is that it
> > remains simple, does its one thing well, and cleaves as closely as
> possible
> > to the conventions for Kafka itself--i.e. doesn't invent new ways of
> > monitoring, configuring, etc. For the tools we've contributed we've tried
> > really hard to make them consistent with Kafka as well as with each other
> > in how testing, configuration, monitoring, etc works.
> >
> > I think what Apache does superbly well is create a community for
> managing a
> > large infrastructure layer like Kafka in a vendor independent way. What I
> > think is less successful is attempting to form full and independent
> apache
> > communities around very simple single purpose tools, especially if you
> hope
> > for these to come together into a cohesive toolset across multiple such
> > tools. Much of what Apache does--create a collective decision making
> > process for resolving disagreement, help to trademark and protect the
> marks
> > of the project, etc just isn't that relevant for simple single-purpose
> > tools.
> >
> > So my take is there are a couple of options:
> >
> > 1. We can try to put all the small tools into the Apache Project. I
> > think this is not the right approach as there is simply too many of
> > them,
> > many in different languages, serving different protocols, integrating
> > with
> > particular systems, and a single community can't effectively maintain
> > them
> > all. Doing this would significantly slow the progress of the Kafka
> > project.
> > As a protocol for messaging, I don't really see a case for including
> > REST
> > but not MQTT or AMQP which are technically much better suited to
> > messaging
> > and both are widely used for that.
> > 2. We can treat ecosystem projects that aren't top level Apache
> projects
> > as invalid and try to recreate them all as Apache projects. Honestly,
> > though, if you go to the Kafka ecosystem page virtually none of the
> most
> > popular add-ons to Kafka are Apache projects. The most successful
> > things in
> > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> other
> > clients, as well as the existing REST layer have succeeded at
> developing
> > communities that actively contribute and use these pieces and I don't
> > know
> > that that is a bad thing unless that community proves to be
> uninclusive,
> > unresponsive, or goes in a bad technical direction--and those are
> > failure
> > modes that all open source efforts face.
> > 3. We can do what I think makes the most sense and try to work with
> the
> > projects that exist in the ecosystem and if the project doesn't have a
> > responsive community or wants to go in a different direction fork or
> > recreate that work.
> >
> > Of course any person can choose whatever of these options they want. But
> > from my point of view, option (3) has been the path of the community so
> far
> > and I think it has been quite successful.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Neha,
> > > "But I haven't seen any community emails or patches being submitted by
> > you
> > > guys, so I'm wondering why you are concerned about whether the
> community
> > is
> > > open to accepting patches or not."
> > >
> > > I think you are talking about contributing patches to this repository
> > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > the guidelines/governance model is not clear on the project and I guess
> > its
> > > driven by opening a github issue request. Its the repository owned by
> > > confluent and as much I appreciate that the features we mentioned are
> in
> > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > gurantee what we want to add in the furture will be in your roadmap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot as
> > other
> > > users can participate in the discussions. We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project time
> > > within
> > > > my internal projects to developers contributing to another commercial
> > > > companies project even so technically open source, as then there is
> > that
> > > > commercial companies interest will always prevail and essentially can
> > > > always have a final vote where disagreement. Im sure they never
> intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint on
> > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages
> in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
> more
> > > > surface area for every client. But something like REST is universal
> and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > ________________________________________
> > > > From: Andrew Schofield <an...@outlook.com>
> > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > There's a massive difference between the governance of Kafka and the
> > > > governance of the REST proxy.
> > > >
> > > > In Kafka, there is a broad community of people contributing their
> > > opinions
> > > > about future enhancements in the form of KIPs. There's some really
> deep
> > > > consideration that goes into some of the trickier KIPs. There are
> > people
> > > > outside Confluent deeply knowledgeable in Kafka and building the
> > > > reputations to become committers. I get the impression that the
> roadmap
> > > of
> > > > Kafka is not really community-owned (what's the big feature for Kafka
> > > 0.11,
> > > > for example), but the conveyor belt of smaller features in the form
> of
> > > KIPs
> > > > works nicely. It's a good example of open-source working well.
> > > >
> > > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > > roadmap is less clear. There's not really a community properly
> engaged
> > in
> > > > the way that there is with Kafka. So, you could say that it's clear
> > that
> > > > fewer people are interested, but I think the whole governance thing
> > is a
> > > > big barrier to engagement. And it's looking like it's getting out of
> > > date.
> > > >
> > > > In technical terms, I can think of two big improvements to the REST
> > > proxy.
> > > > First, it needs to use the new consumer API so that it's possible to
> > > secure
> > > > connections between the REST proxy and Kafka. I don't care too much
> > which
> > > > method calls it uses actually uses to consume messages, but I do
> care
> > > that
> > > > I cannot secure connections because of network security rules.
> Second,
> > > > there's an affinity between a consumer and the instance of the REST
> > proxy
> > > > to which it first connected. Kafka itself avoids this kind of
> affinity
> > > for
> > > > good reason, and in the name of availability the REST proxy should
> too.
> > > > These are natural KIPs.
> > > >
> > > > I think it would be good to have the code for the REST proxy
> > contributed
> > > > to Apache Kafka so that it would be able to be developed in the same
> > way.
> > > >
> > > > Andrew Schofield
> > > >
> > > > From: Suresh Srinivas <su...@hortonworks.com>
> > > > Sent: 07 October 2016 22:41:52
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > ASF already gives us a clear framework and governance model for
> > community
> > > > development. This is already understood by the people contributing to
> > > > Apache Kafka project, and they are the same people who want to
> > contribute
> > > > to the REST server capability as well. Everyone is in agreement on
> the
> > > > need for collaborating on this effort. So why not contribute the code
> > to
> > > > Apache Kafka. This will help avoid duplication of effort and forks
> that
> > > > may crop up, hugely benefitting the user community. This will also
> > avoid
> > > > having to define a process similar to ASF on a GitHub project and
> > instead
> > > > there is a single community with clear understanding community
> process
> > as
> > > > defined in ASF.
> > > >
> > > > As others have said, this is an important capability for Apache
> Kafka.
> > It
> > > > is worth maintaining this as a part of the project.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > > >
> > > > >I personally think it would be quite wasteful to re-implement the
> REST
> > > > >gateway just because that an actively-maintained piece of
> > > Apache-licensed
> > > > >software is not governed directly by the Apache Kafka community...
> > While
> > > > >kafka-rest repo is owned by Confluent, the contributors including
> the
> > > main
> > > > >one are also part of the Apache Kafka community, so there is a
> chance
> > > to
> > > > >work this out.
> > > > >
> > > > >However, there are two valid concerns here that could be addressed,
> > > around
> > > > >community and accessibility:
> > > > >>> What we are worried about is a project
> > > > >>> that's not maintained in a community. So the process of accepting
> > > > >>>patches
> > > > >>> and priorities is not clear, and it's not developed in Apache
> > > > >>>community.
> > > > >>> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >and
> > > > >>> hence there is no security support either.
> > > > >
> > > > >This might be easy to fix. Maybe Confluent / kafka-rest community
> can
> > > > >clarify that - what is their contribution policy, dev style, roadmap
> > > etc.
> > > > >If they want, they can make an effort to encourage participation
> from
> > > > >people outside Confluent (easily accept contributions, invite
> external
> > > > >commiters or have open dev process similar to Apache Kafka etc), as
> > > there
> > > > >is definitely seems to be some interest on the list. That might
> clear
> > > the
> > > > >community concern and help kafka-rest project (but that is a
> > calculation
> > > > >Confluent will have to make).
> > > > >
> > > > >The other, independent, concern is that REST is something that is
> > > expected
> > > > >to be available out of the box with Kafka. I personally don't feel
> > > > >strongly
> > > > >about it (better use proper, efficient APIs from day one), though it
> > is
> > > > >definitely way smaller than adding a stream processing engine to the
> > > > >project :)
> > > > >Again,the kafka-rest "community" could take steps to make it even
> > easier
> > > > >to
> > > > >install, configure and run kafka-rest for new users on vanilla
> Apache
> > > > >Kafka
> > > > >(outside the Confluent platform), if they wish that (or welcome
> > > > >contributions to that end), but that is up to them.
> > > > >Finally, if after the above steps were taken there would still a
> > strong
> > > > >desire to include a great rest gateway with Apache Kafka, I assume
> the
> > > > >community could hypothetically fork the existing kafka-rest into an
> > > Apache
> > > > >Kafka subproject and maintain it "within Apache" instead of
> > implementing
> > > > >it
> > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> > > > >happen
> > > > >without Confluent blessing, and I think that is likely much less
> > optimal
> > > > >(pulling in other Confluent / Apache licensed dependencies) than
> > having
> > > a
> > > > >separate external community around kafka-rest.
> > > > >
> > > > >
> > > > >Just my two cents,
> > > > >
> > > > >
> > > > >Ofir Manor
> > > > >
> > > > >Co-Founder & CTO | Equalum
> > > > >
> > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> | Email:
> > > > ofir.manor@equalum.io
> > > > >
> > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > > > >wrote:
> > > > >
> > > > >> Neha & Jay,
> > > > >> We did look at the open source alternatives. Our
> > > > >>concern
> > > > >> is what's the patch acceptance and adding features/ bug-fixes to
> the
> > > > >> existing project under a Github (although it's licensed under
> Apache
> > > > >>2.0).
> > > > >> It would be great if that project made available under Apache and
> > > > >>driven by
> > > > >> the community. Adding to the above, not all Kafka users are
> > > interested
> > > > >>in
> > > > >> using the Java client API, they would like to have simple REST API
> > > where
> > > > >> they can code against using any language. I do believe this adds
> > value
> > > > >>to
> > > > >> Apache Kafka in itself.
> > > > >>
> > > > >> "For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> reinventing the wheel. What I'm looking for is a detailed
> comparison
> > > of
> > > > >>the
> > > > >> gaps and why those can't be improved in the REST proxy that
> already
> > > > >>exists
> > > > >> and is actively maintained."
> > > > >>
> > > > >> We are not looking at this as NIH. What we are worried about is a
> > > > >>project
> > > > >> that's not maintained in a community. So the process of accepting
> > > > >>patches
> > > > >> and priorities is not clear, and it's not developed in Apache
> > > community.
> > > > >> Not only that, existing REST API project doesn't support new
> client
> > > API
> > > > >>and
> > > > >> hence there is no security support either.
> > > > >> We don't know the timeline when that's made available. We would
> like
> > > to
> > > > >>add
> > > > >> admin functionality into the REST API. So the Roadmap of that
> > project
> > > is
> > > > >> not driven by Apache.
> > > > >>
> > > > >>
> > > > >> "This doesn't materially have an impact on expanding the usability
> > of
> > > > >> Kafka. In my experience, REST proxy + Java clients only cover
> > ~50%
> > > of
> > > > >> all
> > > > >> Kafka users, and maybe 10% of those are the ones who will use
> the
> > > > >>REST
> > > > >> proxy. The remaining 50% are non-java client users (C, python,
> > go,
> > > > >>node
> > > > >> etc)."
> > > > >>
> > > > >> REST API is most often asked feature in my interactions with Kafka
> > > > >>users.
> > > > >> In an organization, There will be independent teams who will write
> > > their
> > > > >> Kafka clients using different language libraries available today,
> > and
> > > > >> there is no way to standardize this. Instead of supporting several
> > > > >> different client libraries users will be interested in using a
> REST
> > > API
> > > > >> server. The need for a REST API will only increase as more and
> more
> > > > >>users
> > > > >> start using Kafka.
> > > > >>
> > > > >> "More surface area means more work to keep things consistent.
> > Failure
> > > > >> to do that has, in fact, hurt the user experience."
> > > > >> Having myriad Kafka client GitHub projects that support different
> > > > >>languages
> > > > >> hurts the user experience and pushes burden to maintain these
> > > libraries.
> > > > >> REST API is a simple code base that uses existing java client
> > > libraries
> > > > >>to
> > > > >> make life easier for the users.
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > > > wrote:
> > > > >>
> > > > >> > Manikumar,
> > > > >> >
> > > > >> > Thanks for sharing the proposal. I think there are 2 parts to
> this
> > > > >> > discussion -
> > > > >> >
> > > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > > >>feature-complete,
> > > > >> > open-source and actively maintained REST proxy in the community?
> > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile
> and
> > > > >> maintain
> > > > >> > the high-quality experience that Kafka users have today?
> > > > >> >
> > > > >> > For 1, I don't think there is value in giving in to the NIH
> > syndrome
> > > > >>and
> > > > >> > reinventing the wheel. What I'm looking for is a detailed
> > comparison
> > > > >>of
> > > > >> the
> > > > >> > gaps and why those can't be improved in the REST proxy that
> > already
> > > > >> exists
> > > > >> > and is actively maintained. For example, we depend on zkClient
> and
> > > > >>have
> > > > >> > found as well as fixed several bugs by working closely with the
> > > people
> > > > >> who
> > > > >> > maintain zkClient. This should be possible for REST proxy as
> well,
> > > > >>right?
> > > > >> >
> > > > >> > For 2, I'd like us to review our history of expanding the
> surface
> > > > >>area to
> > > > >> > add more clients in the past. Here is a summary -
> > > > >> >
> > > > >> > - This doesn't materially have an impact on expanding the
> > > > >>usability of
> > > > >> > Kafka. In my experience, REST proxy + Java clients only cover
> > > ~50%
> > > > >>of
> > > > >> > all
> > > > >> > Kafka users, and maybe 10% of those are the ones who will use
> > the
> > > > >>REST
> > > > >> > proxy. The remaining 50% are non-java client users (C,
> python,
> > > go,
> > > > >> node
> > > > >> > etc).
> > > > >> > - People are a lot more excited about promising to contribute
> > > while
> > > > >> > adding the surface area but not on an ongoing basis down the
> > > line.
> > > > >> > - More surface area means more work to keep things
> consistent.
> > > > >>Failure
> > > > >> > to do that has, in fact, hurt the user experience.
> > > > >> > - More surface area hurts agility. We want to do a few things
> > > > >>really
> > > > >> > well as well as be agile to be able to build on our core
> > > > >>competency.
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Jay,
> > > > >> > >
> > > > >> > > Thanks for your reply.
> > > > >> > >
> > > > >> > > I agree that we can not add all the clients/tools available in
> > > > >> ecosystem
> > > > >> > > page to Kafka repo itself. But we feel REST Interface is
> > different
> > > > >>from
> > > > >> > > other clients/tools. Since any language that can work with
> HTTP
> > > can
> > > > >> > > easily integrate with this interface, Having an "official"
> REST
> > > > >> > > interface helps user community. This also helps us to
> integrate
> > > well
> > > > >> > > with external management and provisioning tools. Apache Kafka
> > > > >>release
> > > > >> > > with Java clients + REST interface is sufficient for most of
> the
> > > > >>user
> > > > >> > > deployments/requirements. This helps users to deal with less
> > > number
> > > > >> > > of distributions/builds.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Manikumar
> > > > >> > >
> > > > >> > >
> > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> > > wrote:
> > > > >> > >
> > > > >> > > > Hey guys,
> > > > >> > > >
> > > > >> > > > There's already a REST interface maintained as a separate
> > > > >> project--it's
> > > > >> > > > open source and apache licensed and actively maintained (
> > > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong
> > with
> > > >
> > > >
> > > >
> > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > github.com
> > > > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
> > It
> > > > makes it easy to produce and consume messages, view the state of the
> > > > cluster, and ...
> > > >
> > > > >> that?
> > > > >> > > You
> > > > >> > > > mentioned that there was some compatibility concern, but
> > > > >> compatibility
> > > > >> > > has
> > > > >> > > > to do with the consumer protocol guarantees not the repo the
> > > code
> > > > >>is
> > > > >> > in,
> > > > >> > > > right? Not sure that concern makes sense.
> > > > >> > > >
> > > > >> > > > We could argue for adding pretty much anything and
> everything
> > in
> > > > >>the
> > > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> > make
> > > > >>the
> > > > >> > > project
> > > > >> > > > more agile.
> > > > >> > > >
> > > > >> > > > -Jay
> > > > >> > > >
> > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > >> manikumar.reddy@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Kafka Devs,
> > > > >> > > > >
> > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> > Repository.
> > > > >> > > > >
> > > > >> > > > > There are already open-source alternatives are available.
> > But
> > > > >>we
> > > > >> > would
> > > > >> > > > > like to add REST server that
> > > > >> > > > > many users ask for under Apache Kafka repo. Many data
> Infra
> > > > >>tools
> > > > >> > comes
> > > > >> > > > up
> > > > >> > > > > with Rest Interface.
> > > > >> > > > > It is useful to have inbuilt Rest API support for Produce,
> > > > >>Consume
> > > > >> > > > messages
> > > > >> > > > > and admin interface for
> > > > >> > > > > integrating with external management and provisioning
> > > tools.This
> > > > >> will
> > > > >> > > > also
> > > > >> > > > > allow the maintenance of
> > > > >> > > > > REST server and adding new features makes it easy because
> > > apache
> > > > >> > > > community.
> > > > >> > > > >
> > > > >> > > > > The KIP wiki is the following:
> > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > >> > > > >
> > > > >> > > > > Your comments and feedback are welcome.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Manikumar
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > --
> > > > >> > Thanks,
> > > > >> > Neha
> > > > >> >
> > > > >>
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
>
>
>
> --
> Nacho (Ignacio) Solis
> Kafka
> nsolis@linkedin.com
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Nacho Solis <ns...@linkedin.com.INVALID>.
What is the criteria for keeping things in and out of Kafka, what code goes
in or out and what is part of the architecture or not?
The discussion of what goes into a project and what stays out is an always
evolving question. Different projects treat this in different ways.
Let me paint 2 extremes. On one side, you have a single monolithic project
that brings everything in one tent. On the other side you have the many
modules approach. From what I've learned, Kafka falls in the middle.
Because of this, the question is bound to come up with respect to the
criteria used to bring something into the fold.
I'll be the first to point out that the distinction between modules,
architecture, software, repositories, governance and community are blurry.
Not to mention that many things are how they are for historical reasons.
I, personally, can't understand why we would not have REST as part of the
main Kafka project given that a lot of people use it and we include many
things with the current distribution. What many things you may ask? Well,
if we took the modular approach Kafka is a mixture of components, here's
the first 4 that come to mind:
1. The Kafka protocol
2. The Kafka java libraries
3. The Kafka broker
4. The Kafka stream framework
5. Kafka Connect
6. MirrorMaker
All of these could be separate products. You should be able to evolve each
one independently. Even if they have dependencies on each other, you could
potentially replace one part.
The choice of keeping them all in a single repository, with a single
distribution, under the same governance and community, brings a number of
trade offs. It's easy to keep things coherent for example. There is less
of a need to rely on inherent versioning and compatibility (which we end up
providing anyway because of the way people usually deploy kafka). We all
focus our efforts on a single code base.
The downside is that it's harder to remove modules that are old or unused.
Modules that are only used by a small subset of the community will have an
impact on the rest of the community. It mixes incentives of what people
want to work on and what holds them back. We also need to decide what
belongs in the blessed bundle and what doesnt.
So, my question boils down to, what criteria is used for bringing stuff in.
If we have Streams and MirrorMaker and Connect in there, why not have REST?
Specially if there is more than one person/group willing to work on it?
Alternatively, if REST is not included because it's not used by all, then
why not remove Streams, Connect and MirrorMaker since they're definitely
not used by all? I realize I say this even though at LinkedIn we have a
REST setup of our own, just speaking from a community perspective.
Nacho
(I'm relatively new and I haven't read all of the mail archive, so I'm sure
this has been brought up before, but I decided to chime in anyway)
On Wed, Oct 12, 2016 at 8:03 AM, Jay Kreps <ja...@confluent.io> wrote:
> I think the questions around governance make sense, I think we should
> really clarify that to make the process more clear so it can be fully
> inclusive.
>
> The idea that we should not collaborate on what is there now, though,
> because in the future we might disagree about direction does not really
> make sense to me. If in the future we disagree, that is the beauty of open
> source, you can always fork off a copy of the code and start an independent
> project either in Apache or elsewhere. Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/kafka access that is heavily
> used in production seems quite silly.
>
> Let me give some background on how I at least think about these things.
> I've participated in open source projects out of LinkedIn via github as
> well as via the ASF. I don't think there is a "right" answer to how to do
> these but rather some tradeoffs. We thought about this quite a lot in the
> context of Kafka based on the experience with the Hadoop ecosystem as well
> as from other open source communities.
>
> There is a rich ecosystem around Kafka. Many of the projects are quite
> small--single clients or tools that do a single thing well--and almost none
> of them are top level apache projects. I don't think trying to force each
> of these to turn into independent Apache projects is necessarily the best
> thing for the ecosystem.
>
> My observation of how this can go wrong is really what I think has happened
> to the Hadoop ecosystem. There you see quite a zoo of projects which all
> drift apart and don't quite work together well. Coordinating even simple
> changes and standardization across these is exceptionally difficult. The
> result is a bit of a mess for users--the pieces just don't really come
> together very well. This makes sense for independent infrastructure systems
> (Kudu vs HDFS) but I'm not at all convinced that doing this for every
> little tool or helper library has lead to a desirable state. I think the
> mode of operating where the Hadoop vendors spawn off a few new Apache
> projects for each new product initiative, especially since often that
> project is only valued by that vendor (and the other vendor has a different
> competing Apache project) doesn't necessarily do a better job at producing
> high quality communities or high quality software.
>
> These tools/connects/clients/proxies and other integration pieces can take
> many forms, but my take of what makes one of these things good is that it
> remains simple, does its one thing well, and cleaves as closely as possible
> to the conventions for Kafka itself--i.e. doesn't invent new ways of
> monitoring, configuring, etc. For the tools we've contributed we've tried
> really hard to make them consistent with Kafka as well as with each other
> in how testing, configuration, monitoring, etc works.
>
> I think what Apache does superbly well is create a community for managing a
> large infrastructure layer like Kafka in a vendor independent way. What I
> think is less successful is attempting to form full and independent apache
> communities around very simple single purpose tools, especially if you hope
> for these to come together into a cohesive toolset across multiple such
> tools. Much of what Apache does--create a collective decision making
> process for resolving disagreement, help to trademark and protect the marks
> of the project, etc just isn't that relevant for simple single-purpose
> tools.
>
> So my take is there are a couple of options:
>
> 1. We can try to put all the small tools into the Apache Project. I
> think this is not the right approach as there is simply too many of
> them,
> many in different languages, serving different protocols, integrating
> with
> particular systems, and a single community can't effectively maintain
> them
> all. Doing this would significantly slow the progress of the Kafka
> project.
> As a protocol for messaging, I don't really see a case for including
> REST
> but not MQTT or AMQP which are technically much better suited to
> messaging
> and both are widely used for that.
> 2. We can treat ecosystem projects that aren't top level Apache projects
> as invalid and try to recreate them all as Apache projects. Honestly,
> though, if you go to the Kafka ecosystem page virtually none of the most
> popular add-ons to Kafka are Apache projects. The most successful
> things in
> the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of other
> clients, as well as the existing REST layer have succeeded at developing
> communities that actively contribute and use these pieces and I don't
> know
> that that is a bad thing unless that community proves to be uninclusive,
> unresponsive, or goes in a bad technical direction--and those are
> failure
> modes that all open source efforts face.
> 3. We can do what I think makes the most sense and try to work with the
> projects that exist in the ecosystem and if the project doesn't have a
> responsive community or wants to go in a different direction fork or
> recreate that work.
>
> Of course any person can choose whatever of these options they want. But
> from my point of view, option (3) has been the path of the community so far
> and I think it has been quite successful.
>
> -Jay
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Neha,
> > "But I haven't seen any community emails or patches being submitted by
> you
> > guys, so I'm wondering why you are concerned about whether the community
> is
> > open to accepting patches or not."
> >
> > I think you are talking about contributing patches to this repository
> > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > the guidelines/governance model is not clear on the project and I guess
> its
> > driven by opening a github issue request. Its the repository owned by
> > confluent and as much I appreciate that the features we mentioned are in
> > the roadmap and welcoming us to contribute to the project. It doesn't
> > gurantee what we want to add in the furture will be in your roadmap.
> >
> > Hence the reason having it part of Kafka community will help a lot as
> other
> > users can participate in the discussions. We are happy to drive any
> > feature additions through KIPs which gives everyone a chance to
> participate
> > and add to the discussions.
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > +1
> > >
> > > I agree on the governance comments whole heartedly.
> > >
> > > Also i agree about the contribution comments made earlier in the
> thread,
> > i
> > > personally am less likely to spend any of mine, or give project time
> > within
> > > my internal projects to developers contributing to another commercial
> > > companies project even so technically open source, as then there is
> that
> > > commercial companies interest will always prevail and essentially can
> > > always have a final vote where disagreement. Im sure they never intend
> > to,
> > > but there is that true reality. This is why we have community open
> source
> > > projects.
> > >
> > > I can find many different implementations now of a rest endpoint on
> > > GitHub, BitBucket etc. Each one has their benefits and disadvantages in
> > > their implementation. By making / providing one this would bring
> together
> > > these solutions, unifying those developers and also bringing the best
> of
> > > all.
> > >
> > > I understand the concern on the community burden adding/supporting more
> > > surface area for every client. But something like REST is universal and
> > > worthy to be owned by the community.
> > >
> > > Mike
> > >
> > >
> > > ________________________________________
> > > From: Andrew Schofield <an...@outlook.com>
> > > Sent: Saturday, October 8, 2016 1:19 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > There's a massive difference between the governance of Kafka and the
> > > governance of the REST proxy.
> > >
> > > In Kafka, there is a broad community of people contributing their
> > opinions
> > > about future enhancements in the form of KIPs. There's some really deep
> > > consideration that goes into some of the trickier KIPs. There are
> people
> > > outside Confluent deeply knowledgeable in Kafka and building the
> > > reputations to become committers. I get the impression that the roadmap
> > of
> > > Kafka is not really community-owned (what's the big feature for Kafka
> > 0.11,
> > > for example), but the conveyor belt of smaller features in the form of
> > KIPs
> > > works nicely. It's a good example of open-source working well.
> > >
> > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > roadmap is less clear. There's not really a community properly engaged
> in
> > > the way that there is with Kafka. So, you could say that it's clear
> that
> > > fewer people are interested, but I think the whole governance thing
> is a
> > > big barrier to engagement. And it's looking like it's getting out of
> > date.
> > >
> > > In technical terms, I can think of two big improvements to the REST
> > proxy.
> > > First, it needs to use the new consumer API so that it's possible to
> > secure
> > > connections between the REST proxy and Kafka. I don't care too much
> which
> > > method calls it uses actually uses to consume messages, but I do care
> > that
> > > I cannot secure connections because of network security rules. Second,
> > > there's an affinity between a consumer and the instance of the REST
> proxy
> > > to which it first connected. Kafka itself avoids this kind of affinity
> > for
> > > good reason, and in the name of availability the REST proxy should too.
> > > These are natural KIPs.
> > >
> > > I think it would be good to have the code for the REST proxy
> contributed
> > > to Apache Kafka so that it would be able to be developed in the same
> way.
> > >
> > > Andrew Schofield
> > >
> > > From: Suresh Srinivas <su...@hortonworks.com>
> > > Sent: 07 October 2016 22:41:52
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > ASF already gives us a clear framework and governance model for
> community
> > > development. This is already understood by the people contributing to
> > > Apache Kafka project, and they are the same people who want to
> contribute
> > > to the REST server capability as well. Everyone is in agreement on the
> > > need for collaborating on this effort. So why not contribute the code
> to
> > > Apache Kafka. This will help avoid duplication of effort and forks that
> > > may crop up, hugely benefitting the user community. This will also
> avoid
> > > having to define a process similar to ASF on a GitHub project and
> instead
> > > there is a single community with clear understanding community process
> as
> > > defined in ASF.
> > >
> > > As others have said, this is an important capability for Apache Kafka.
> It
> > > is worth maintaining this as a part of the project.
> > >
> > > Regards,
> > > Suresh
> > >
> > > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> > >
> > > >I personally think it would be quite wasteful to re-implement the REST
> > > >gateway just because that an actively-maintained piece of
> > Apache-licensed
> > > >software is not governed directly by the Apache Kafka community...
> While
> > > >kafka-rest repo is owned by Confluent, the contributors including the
> > main
> > > >one are also part of the Apache Kafka community, so there is a chance
> > to
> > > >work this out.
> > > >
> > > >However, there are two valid concerns here that could be addressed,
> > around
> > > >community and accessibility:
> > > >>> What we are worried about is a project
> > > >>> that's not maintained in a community. So the process of accepting
> > > >>>patches
> > > >>> and priorities is not clear, and it's not developed in Apache
> > > >>>community.
> > > >>> Not only that, existing REST API project doesn't support new client
> > API
> > > >and
> > > >>> hence there is no security support either.
> > > >
> > > >This might be easy to fix. Maybe Confluent / kafka-rest community can
> > > >clarify that - what is their contribution policy, dev style, roadmap
> > etc.
> > > >If they want, they can make an effort to encourage participation from
> > > >people outside Confluent (easily accept contributions, invite external
> > > >commiters or have open dev process similar to Apache Kafka etc), as
> > there
> > > >is definitely seems to be some interest on the list. That might clear
> > the
> > > >community concern and help kafka-rest project (but that is a
> calculation
> > > >Confluent will have to make).
> > > >
> > > >The other, independent, concern is that REST is something that is
> > expected
> > > >to be available out of the box with Kafka. I personally don't feel
> > > >strongly
> > > >about it (better use proper, efficient APIs from day one), though it
> is
> > > >definitely way smaller than adding a stream processing engine to the
> > > >project :)
> > > >Again,the kafka-rest "community" could take steps to make it even
> easier
> > > >to
> > > >install, configure and run kafka-rest for new users on vanilla Apache
> > > >Kafka
> > > >(outside the Confluent platform), if they wish that (or welcome
> > > >contributions to that end), but that is up to them.
> > > >Finally, if after the above steps were taken there would still a
> strong
> > > >desire to include a great rest gateway with Apache Kafka, I assume the
> > > >community could hypothetically fork the existing kafka-rest into an
> > Apache
> > > >Kafka subproject and maintain it "within Apache" instead of
> implementing
> > > >it
> > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> > > >happen
> > > >without Confluent blessing, and I think that is likely much less
> optimal
> > > >(pulling in other Confluent / Apache licensed dependencies) than
> having
> > a
> > > >separate external community around kafka-rest.
> > > >
> > > >
> > > >Just my two cents,
> > > >
> > > >
> > > >Ofir Manor
> > > >
> > > >Co-Founder & CTO | Equalum
> > > >
> > > >Mobile: +972-54-7801286 <+972%2054-780-1286> | Email:
> > > ofir.manor@equalum.io
> > > >
> > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <kafka@harsha.io
> >
> > > >wrote:
> > > >
> > > >> Neha & Jay,
> > > >> We did look at the open source alternatives. Our
> > > >>concern
> > > >> is what's the patch acceptance and adding features/ bug-fixes to the
> > > >> existing project under a Github (although it's licensed under Apache
> > > >>2.0).
> > > >> It would be great if that project made available under Apache and
> > > >>driven by
> > > >> the community. Adding to the above, not all Kafka users are
> > interested
> > > >>in
> > > >> using the Java client API, they would like to have simple REST API
> > where
> > > >> they can code against using any language. I do believe this adds
> value
> > > >>to
> > > >> Apache Kafka in itself.
> > > >>
> > > >> "For 1, I don't think there is value in giving in to the NIH
> syndrome
> > > >>and
> > > >> reinventing the wheel. What I'm looking for is a detailed comparison
> > of
> > > >>the
> > > >> gaps and why those can't be improved in the REST proxy that already
> > > >>exists
> > > >> and is actively maintained."
> > > >>
> > > >> We are not looking at this as NIH. What we are worried about is a
> > > >>project
> > > >> that's not maintained in a community. So the process of accepting
> > > >>patches
> > > >> and priorities is not clear, and it's not developed in Apache
> > community.
> > > >> Not only that, existing REST API project doesn't support new client
> > API
> > > >>and
> > > >> hence there is no security support either.
> > > >> We don't know the timeline when that's made available. We would like
> > to
> > > >>add
> > > >> admin functionality into the REST API. So the Roadmap of that
> project
> > is
> > > >> not driven by Apache.
> > > >>
> > > >>
> > > >> "This doesn't materially have an impact on expanding the usability
> of
> > > >> Kafka. In my experience, REST proxy + Java clients only cover
> ~50%
> > of
> > > >> all
> > > >> Kafka users, and maybe 10% of those are the ones who will use the
> > > >>REST
> > > >> proxy. The remaining 50% are non-java client users (C, python,
> go,
> > > >>node
> > > >> etc)."
> > > >>
> > > >> REST API is most often asked feature in my interactions with Kafka
> > > >>users.
> > > >> In an organization, There will be independent teams who will write
> > their
> > > >> Kafka clients using different language libraries available today,
> and
> > > >> there is no way to standardize this. Instead of supporting several
> > > >> different client libraries users will be interested in using a REST
> > API
> > > >> server. The need for a REST API will only increase as more and more
> > > >>users
> > > >> start using Kafka.
> > > >>
> > > >> "More surface area means more work to keep things consistent.
> Failure
> > > >> to do that has, in fact, hurt the user experience."
> > > >> Having myriad Kafka client GitHub projects that support different
> > > >>languages
> > > >> hurts the user experience and pushes burden to maintain these
> > libraries.
> > > >> REST API is a simple code base that uses existing java client
> > libraries
> > > >>to
> > > >> make life easier for the users.
> > > >>
> > > >> Thanks,
> > > >> Harsha
> > > >>
> > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > > wrote:
> > > >>
> > > >> > Manikumar,
> > > >> >
> > > >> > Thanks for sharing the proposal. I think there are 2 parts to this
> > > >> > discussion -
> > > >> >
> > > >> > 1. Should we rewrite a REST proxy given that there is a
> > > >>feature-complete,
> > > >> > open-source and actively maintained REST proxy in the community?
> > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> > > >> maintain
> > > >> > the high-quality experience that Kafka users have today?
> > > >> >
> > > >> > For 1, I don't think there is value in giving in to the NIH
> syndrome
> > > >>and
> > > >> > reinventing the wheel. What I'm looking for is a detailed
> comparison
> > > >>of
> > > >> the
> > > >> > gaps and why those can't be improved in the REST proxy that
> already
> > > >> exists
> > > >> > and is actively maintained. For example, we depend on zkClient and
> > > >>have
> > > >> > found as well as fixed several bugs by working closely with the
> > people
> > > >> who
> > > >> > maintain zkClient. This should be possible for REST proxy as well,
> > > >>right?
> > > >> >
> > > >> > For 2, I'd like us to review our history of expanding the surface
> > > >>area to
> > > >> > add more clients in the past. Here is a summary -
> > > >> >
> > > >> > - This doesn't materially have an impact on expanding the
> > > >>usability of
> > > >> > Kafka. In my experience, REST proxy + Java clients only cover
> > ~50%
> > > >>of
> > > >> > all
> > > >> > Kafka users, and maybe 10% of those are the ones who will use
> the
> > > >>REST
> > > >> > proxy. The remaining 50% are non-java client users (C, python,
> > go,
> > > >> node
> > > >> > etc).
> > > >> > - People are a lot more excited about promising to contribute
> > while
> > > >> > adding the surface area but not on an ongoing basis down the
> > line.
> > > >> > - More surface area means more work to keep things consistent.
> > > >>Failure
> > > >> > to do that has, in fact, hurt the user experience.
> > > >> > - More surface area hurts agility. We want to do a few things
> > > >>really
> > > >> > well as well as be agile to be able to build on our core
> > > >>competency.
> > > >> >
> > > >> >
> > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> manikumar.reddy@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Jay,
> > > >> > >
> > > >> > > Thanks for your reply.
> > > >> > >
> > > >> > > I agree that we can not add all the clients/tools available in
> > > >> ecosystem
> > > >> > > page to Kafka repo itself. But we feel REST Interface is
> different
> > > >>from
> > > >> > > other clients/tools. Since any language that can work with HTTP
> > can
> > > >> > > easily integrate with this interface, Having an "official" REST
> > > >> > > interface helps user community. This also helps us to integrate
> > well
> > > >> > > with external management and provisioning tools. Apache Kafka
> > > >>release
> > > >> > > with Java clients + REST interface is sufficient for most of the
> > > >>user
> > > >> > > deployments/requirements. This helps users to deal with less
> > number
> > > >> > > of distributions/builds.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Manikumar
> > > >> > >
> > > >> > >
> > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> > wrote:
> > > >> > >
> > > >> > > > Hey guys,
> > > >> > > >
> > > >> > > > There's already a REST interface maintained as a separate
> > > >> project--it's
> > > >> > > > open source and apache licensed and actively maintained (
> > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong
> with
> > >
> > >
> > >
> > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > github.com
> > > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
> It
> > > makes it easy to produce and consume messages, view the state of the
> > > cluster, and ...
> > >
> > > >> that?
> > > >> > > You
> > > >> > > > mentioned that there was some compatibility concern, but
> > > >> compatibility
> > > >> > > has
> > > >> > > > to do with the consumer protocol guarantees not the repo the
> > code
> > > >>is
> > > >> > in,
> > > >> > > > right? Not sure that concern makes sense.
> > > >> > > >
> > > >> > > > We could argue for adding pretty much anything and everything
> in
> > > >>the
> > > >> > > > ecosystem page in Kafka itself but I'm not sure that would
> make
> > > >>the
> > > >> > > project
> > > >> > > > more agile.
> > > >> > > >
> > > >> > > > -Jay
> > > >> > > >
> > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > >> manikumar.reddy@gmail.com
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Kafka Devs,
> > > >> > > > >
> > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka
> Repository.
> > > >> > > > >
> > > >> > > > > There are already open-source alternatives are available.
> But
> > > >>we
> > > >> > would
> > > >> > > > > like to add REST server that
> > > >> > > > > many users ask for under Apache Kafka repo. Many data Infra
> > > >>tools
> > > >> > comes
> > > >> > > > up
> > > >> > > > > with Rest Interface.
> > > >> > > > > It is useful to have inbuilt Rest API support for Produce,
> > > >>Consume
> > > >> > > > messages
> > > >> > > > > and admin interface for
> > > >> > > > > integrating with external management and provisioning
> > tools.This
> > > >> will
> > > >> > > > also
> > > >> > > > > allow the maintenance of
> > > >> > > > > REST server and adding new features makes it easy because
> > apache
> > > >> > > > community.
> > > >> > > > >
> > > >> > > > > The KIP wiki is the following:
> > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> > > > > 80%3A+Kafka+Rest+Server
> > > >> > > > >
> > > >> > > > > Your comments and feedback are welcome.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Manikumar
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> > --
> > > >> > Thanks,
> > > >> > Neha
> > > >> >
> > > >>
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
--
Nacho (Ignacio) Solis
Kafka
nsolis@linkedin.com
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Jay Kreps <ja...@confluent.io>.
I think the questions around governance make sense, I think we should
really clarify that to make the process more clear so it can be fully
inclusive.
The idea that we should not collaborate on what is there now, though,
because in the future we might disagree about direction does not really
make sense to me. If in the future we disagree, that is the beauty of open
source, you can always fork off a copy of the code and start an independent
project either in Apache or elsewhere. Pre-emptively re-creating another
REST layer when it seems like we all quite agree on what needs to be done
and we have an existing code base for HTTP/kafka access that is heavily
used in production seems quite silly.
Let me give some background on how I at least think about these things.
I've participated in open source projects out of LinkedIn via github as
well as via the ASF. I don't think there is a "right" answer to how to do
these but rather some tradeoffs. We thought about this quite a lot in the
context of Kafka based on the experience with the Hadoop ecosystem as well
as from other open source communities.
There is a rich ecosystem around Kafka. Many of the projects are quite
small--single clients or tools that do a single thing well--and almost none
of them are top level apache projects. I don't think trying to force each
of these to turn into independent Apache projects is necessarily the best
thing for the ecosystem.
My observation of how this can go wrong is really what I think has happened
to the Hadoop ecosystem. There you see quite a zoo of projects which all
drift apart and don't quite work together well. Coordinating even simple
changes and standardization across these is exceptionally difficult. The
result is a bit of a mess for users--the pieces just don't really come
together very well. This makes sense for independent infrastructure systems
(Kudu vs HDFS) but I'm not at all convinced that doing this for every
little tool or helper library has lead to a desirable state. I think the
mode of operating where the Hadoop vendors spawn off a few new Apache
projects for each new product initiative, especially since often that
project is only valued by that vendor (and the other vendor has a different
competing Apache project) doesn't necessarily do a better job at producing
high quality communities or high quality software.
These tools/connects/clients/proxies and other integration pieces can take
many forms, but my take of what makes one of these things good is that it
remains simple, does its one thing well, and cleaves as closely as possible
to the conventions for Kafka itself--i.e. doesn't invent new ways of
monitoring, configuring, etc. For the tools we've contributed we've tried
really hard to make them consistent with Kafka as well as with each other
in how testing, configuration, monitoring, etc works.
I think what Apache does superbly well is create a community for managing a
large infrastructure layer like Kafka in a vendor independent way. What I
think is less successful is attempting to form full and independent apache
communities around very simple single purpose tools, especially if you hope
for these to come together into a cohesive toolset across multiple such
tools. Much of what Apache does--create a collective decision making
process for resolving disagreement, help to trademark and protect the marks
of the project, etc just isn't that relevant for simple single-purpose
tools.
So my take is there are a couple of options:
1. We can try to put all the small tools into the Apache Project. I
think this is not the right approach as there is simply too many of them,
many in different languages, serving different protocols, integrating with
particular systems, and a single community can't effectively maintain them
all. Doing this would significantly slow the progress of the Kafka project.
As a protocol for messaging, I don't really see a case for including REST
but not MQTT or AMQP which are technically much better suited to messaging
and both are widely used for that.
2. We can treat ecosystem projects that aren't top level Apache projects
as invalid and try to recreate them all as Apache projects. Honestly,
though, if you go to the Kafka ecosystem page virtually none of the most
popular add-ons to Kafka are Apache projects. The most successful things in
the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of other
clients, as well as the existing REST layer have succeeded at developing
communities that actively contribute and use these pieces and I don't know
that that is a bad thing unless that community proves to be uninclusive,
unresponsive, or goes in a bad technical direction--and those are failure
modes that all open source efforts face.
3. We can do what I think makes the most sense and try to work with the
projects that exist in the ecosystem and if the project doesn't have a
responsive community or wants to go in a different direction fork or
recreate that work.
Of course any person can choose whatever of these options they want. But
from my point of view, option (3) has been the path of the community so far
and I think it has been quite successful.
-Jay
On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
wrote:
> Neha,
> "But I haven't seen any community emails or patches being submitted by you
> guys, so I'm wondering why you are concerned about whether the community is
> open to accepting patches or not."
>
> I think you are talking about contributing patches to this repository
> right? https://github.com/confluentinc/kafka-rest . All I am saying
> the guidelines/governance model is not clear on the project and I guess its
> driven by opening a github issue request. Its the repository owned by
> confluent and as much I appreciate that the features we mentioned are in
> the roadmap and welcoming us to contribute to the project. It doesn't
> gurantee what we want to add in the furture will be in your roadmap.
>
> Hence the reason having it part of Kafka community will help a lot as other
> users can participate in the discussions. We are happy to drive any
> feature additions through KIPs which gives everyone a chance to participate
> and add to the discussions.
>
> Thanks,
> Harsha
>
> On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
> wrote:
>
> > +1
> >
> > I agree on the governance comments whole heartedly.
> >
> > Also i agree about the contribution comments made earlier in the thread,
> i
> > personally am less likely to spend any of mine, or give project time
> within
> > my internal projects to developers contributing to another commercial
> > companies project even so technically open source, as then there is that
> > commercial companies interest will always prevail and essentially can
> > always have a final vote where disagreement. Im sure they never intend
> to,
> > but there is that true reality. This is why we have community open source
> > projects.
> >
> > I can find many different implementations now of a rest endpoint on
> > GitHub, BitBucket etc. Each one has their benefits and disadvantages in
> > their implementation. By making / providing one this would bring together
> > these solutions, unifying those developers and also bringing the best of
> > all.
> >
> > I understand the concern on the community burden adding/supporting more
> > surface area for every client. But something like REST is universal and
> > worthy to be owned by the community.
> >
> > Mike
> >
> >
> > ________________________________________
> > From: Andrew Schofield <an...@outlook.com>
> > Sent: Saturday, October 8, 2016 1:19 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >
> > There's a massive difference between the governance of Kafka and the
> > governance of the REST proxy.
> >
> > In Kafka, there is a broad community of people contributing their
> opinions
> > about future enhancements in the form of KIPs. There's some really deep
> > consideration that goes into some of the trickier KIPs. There are people
> > outside Confluent deeply knowledgeable in Kafka and building the
> > reputations to become committers. I get the impression that the roadmap
> of
> > Kafka is not really community-owned (what's the big feature for Kafka
> 0.11,
> > for example), but the conveyor belt of smaller features in the form of
> KIPs
> > works nicely. It's a good example of open-source working well.
> >
> > The equivalent for the REST proxy is basically issues on GitHub. The
> > roadmap is less clear. There's not really a community properly engaged in
> > the way that there is with Kafka. So, you could say that it's clear that
> > fewer people are interested, but I think the whole governance thing is a
> > big barrier to engagement. And it's looking like it's getting out of
> date.
> >
> > In technical terms, I can think of two big improvements to the REST
> proxy.
> > First, it needs to use the new consumer API so that it's possible to
> secure
> > connections between the REST proxy and Kafka. I don't care too much which
> > method calls it uses actually uses to consume messages, but I do care
> that
> > I cannot secure connections because of network security rules. Second,
> > there's an affinity between a consumer and the instance of the REST proxy
> > to which it first connected. Kafka itself avoids this kind of affinity
> for
> > good reason, and in the name of availability the REST proxy should too.
> > These are natural KIPs.
> >
> > I think it would be good to have the code for the REST proxy contributed
> > to Apache Kafka so that it would be able to be developed in the same way.
> >
> > Andrew Schofield
> >
> > From: Suresh Srinivas <su...@hortonworks.com>
> > Sent: 07 October 2016 22:41:52
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >
> > ASF already gives us a clear framework and governance model for community
> > development. This is already understood by the people contributing to
> > Apache Kafka project, and they are the same people who want to contribute
> > to the REST server capability as well. Everyone is in agreement on the
> > need for collaborating on this effort. So why not contribute the code to
> > Apache Kafka. This will help avoid duplication of effort and forks that
> > may crop up, hugely benefitting the user community. This will also avoid
> > having to define a process similar to ASF on a GitHub project and instead
> > there is a single community with clear understanding community process as
> > defined in ASF.
> >
> > As others have said, this is an important capability for Apache Kafka. It
> > is worth maintaining this as a part of the project.
> >
> > Regards,
> > Suresh
> >
> > On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
> >
> > >I personally think it would be quite wasteful to re-implement the REST
> > >gateway just because that an actively-maintained piece of
> Apache-licensed
> > >software is not governed directly by the Apache Kafka community... While
> > >kafka-rest repo is owned by Confluent, the contributors including the
> main
> > >one are also part of the Apache Kafka community, so there is a chance
> to
> > >work this out.
> > >
> > >However, there are two valid concerns here that could be addressed,
> around
> > >community and accessibility:
> > >>> What we are worried about is a project
> > >>> that's not maintained in a community. So the process of accepting
> > >>>patches
> > >>> and priorities is not clear, and it's not developed in Apache
> > >>>community.
> > >>> Not only that, existing REST API project doesn't support new client
> API
> > >and
> > >>> hence there is no security support either.
> > >
> > >This might be easy to fix. Maybe Confluent / kafka-rest community can
> > >clarify that - what is their contribution policy, dev style, roadmap
> etc.
> > >If they want, they can make an effort to encourage participation from
> > >people outside Confluent (easily accept contributions, invite external
> > >commiters or have open dev process similar to Apache Kafka etc), as
> there
> > >is definitely seems to be some interest on the list. That might clear
> the
> > >community concern and help kafka-rest project (but that is a calculation
> > >Confluent will have to make).
> > >
> > >The other, independent, concern is that REST is something that is
> expected
> > >to be available out of the box with Kafka. I personally don't feel
> > >strongly
> > >about it (better use proper, efficient APIs from day one), though it is
> > >definitely way smaller than adding a stream processing engine to the
> > >project :)
> > >Again,the kafka-rest "community" could take steps to make it even easier
> > >to
> > >install, configure and run kafka-rest for new users on vanilla Apache
> > >Kafka
> > >(outside the Confluent platform), if they wish that (or welcome
> > >contributions to that end), but that is up to them.
> > >Finally, if after the above steps were taken there would still a strong
> > >desire to include a great rest gateway with Apache Kafka, I assume the
> > >community could hypothetically fork the existing kafka-rest into an
> Apache
> > >Kafka subproject and maintain it "within Apache" instead of implementing
> > >it
> > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> > >happen
> > >without Confluent blessing, and I think that is likely much less optimal
> > >(pulling in other Confluent / Apache licensed dependencies) than having
> a
> > >separate external community around kafka-rest.
> > >
> > >
> > >Just my two cents,
> > >
> > >
> > >Ofir Manor
> > >
> > >Co-Founder & CTO | Equalum
> > >
> > >Mobile: +972-54-7801286 <+972%2054-780-1286> | Email:
> > ofir.manor@equalum.io
> > >
> > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <ka...@harsha.io>
> > >wrote:
> > >
> > >> Neha & Jay,
> > >> We did look at the open source alternatives. Our
> > >>concern
> > >> is what's the patch acceptance and adding features/ bug-fixes to the
> > >> existing project under a Github (although it's licensed under Apache
> > >>2.0).
> > >> It would be great if that project made available under Apache and
> > >>driven by
> > >> the community. Adding to the above, not all Kafka users are
> interested
> > >>in
> > >> using the Java client API, they would like to have simple REST API
> where
> > >> they can code against using any language. I do believe this adds value
> > >>to
> > >> Apache Kafka in itself.
> > >>
> > >> "For 1, I don't think there is value in giving in to the NIH syndrome
> > >>and
> > >> reinventing the wheel. What I'm looking for is a detailed comparison
> of
> > >>the
> > >> gaps and why those can't be improved in the REST proxy that already
> > >>exists
> > >> and is actively maintained."
> > >>
> > >> We are not looking at this as NIH. What we are worried about is a
> > >>project
> > >> that's not maintained in a community. So the process of accepting
> > >>patches
> > >> and priorities is not clear, and it's not developed in Apache
> community.
> > >> Not only that, existing REST API project doesn't support new client
> API
> > >>and
> > >> hence there is no security support either.
> > >> We don't know the timeline when that's made available. We would like
> to
> > >>add
> > >> admin functionality into the REST API. So the Roadmap of that project
> is
> > >> not driven by Apache.
> > >>
> > >>
> > >> "This doesn't materially have an impact on expanding the usability of
> > >> Kafka. In my experience, REST proxy + Java clients only cover ~50%
> of
> > >> all
> > >> Kafka users, and maybe 10% of those are the ones who will use the
> > >>REST
> > >> proxy. The remaining 50% are non-java client users (C, python, go,
> > >>node
> > >> etc)."
> > >>
> > >> REST API is most often asked feature in my interactions with Kafka
> > >>users.
> > >> In an organization, There will be independent teams who will write
> their
> > >> Kafka clients using different language libraries available today, and
> > >> there is no way to standardize this. Instead of supporting several
> > >> different client libraries users will be interested in using a REST
> API
> > >> server. The need for a REST API will only increase as more and more
> > >>users
> > >> start using Kafka.
> > >>
> > >> "More surface area means more work to keep things consistent. Failure
> > >> to do that has, in fact, hurt the user experience."
> > >> Having myriad Kafka client GitHub projects that support different
> > >>languages
> > >> hurts the user experience and pushes burden to maintain these
> libraries.
> > >> REST API is a simple code base that uses existing java client
> libraries
> > >>to
> > >> make life easier for the users.
> > >>
> > >> Thanks,
> > >> Harsha
> > >>
> > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> > wrote:
> > >>
> > >> > Manikumar,
> > >> >
> > >> > Thanks for sharing the proposal. I think there are 2 parts to this
> > >> > discussion -
> > >> >
> > >> > 1. Should we rewrite a REST proxy given that there is a
> > >>feature-complete,
> > >> > open-source and actively maintained REST proxy in the community?
> > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> > >> maintain
> > >> > the high-quality experience that Kafka users have today?
> > >> >
> > >> > For 1, I don't think there is value in giving in to the NIH syndrome
> > >>and
> > >> > reinventing the wheel. What I'm looking for is a detailed comparison
> > >>of
> > >> the
> > >> > gaps and why those can't be improved in the REST proxy that already
> > >> exists
> > >> > and is actively maintained. For example, we depend on zkClient and
> > >>have
> > >> > found as well as fixed several bugs by working closely with the
> people
> > >> who
> > >> > maintain zkClient. This should be possible for REST proxy as well,
> > >>right?
> > >> >
> > >> > For 2, I'd like us to review our history of expanding the surface
> > >>area to
> > >> > add more clients in the past. Here is a summary -
> > >> >
> > >> > - This doesn't materially have an impact on expanding the
> > >>usability of
> > >> > Kafka. In my experience, REST proxy + Java clients only cover
> ~50%
> > >>of
> > >> > all
> > >> > Kafka users, and maybe 10% of those are the ones who will use the
> > >>REST
> > >> > proxy. The remaining 50% are non-java client users (C, python,
> go,
> > >> node
> > >> > etc).
> > >> > - People are a lot more excited about promising to contribute
> while
> > >> > adding the surface area but not on an ongoing basis down the
> line.
> > >> > - More surface area means more work to keep things consistent.
> > >>Failure
> > >> > to do that has, in fact, hurt the user experience.
> > >> > - More surface area hurts agility. We want to do a few things
> > >>really
> > >> > well as well as be agile to be able to build on our core
> > >>competency.
> > >> >
> > >> >
> > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <manikumar.reddy@gmail.com
> >
> > >> > wrote:
> > >> >
> > >> > > Hi Jay,
> > >> > >
> > >> > > Thanks for your reply.
> > >> > >
> > >> > > I agree that we can not add all the clients/tools available in
> > >> ecosystem
> > >> > > page to Kafka repo itself. But we feel REST Interface is different
> > >>from
> > >> > > other clients/tools. Since any language that can work with HTTP
> can
> > >> > > easily integrate with this interface, Having an "official" REST
> > >> > > interface helps user community. This also helps us to integrate
> well
> > >> > > with external management and provisioning tools. Apache Kafka
> > >>release
> > >> > > with Java clients + REST interface is sufficient for most of the
> > >>user
> > >> > > deployments/requirements. This helps users to deal with less
> number
> > >> > > of distributions/builds.
> > >> > >
> > >> > > Thanks,
> > >> > > Manikumar
> > >> > >
> > >> > >
> > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io>
> wrote:
> > >> > >
> > >> > > > Hey guys,
> > >> > > >
> > >> > > > There's already a REST interface maintained as a separate
> > >> project--it's
> > >> > > > open source and apache licensed and actively maintained (
> > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
> >
> >
> >
> > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > github.com
> > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It
> > makes it easy to produce and consume messages, view the state of the
> > cluster, and ...
> >
> > >> that?
> > >> > > You
> > >> > > > mentioned that there was some compatibility concern, but
> > >> compatibility
> > >> > > has
> > >> > > > to do with the consumer protocol guarantees not the repo the
> code
> > >>is
> > >> > in,
> > >> > > > right? Not sure that concern makes sense.
> > >> > > >
> > >> > > > We could argue for adding pretty much anything and everything in
> > >>the
> > >> > > > ecosystem page in Kafka itself but I'm not sure that would make
> > >>the
> > >> > > project
> > >> > > > more agile.
> > >> > > >
> > >> > > > -Jay
> > >> > > >
> > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > >> manikumar.reddy@gmail.com
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Kafka Devs,
> > >> > > > >
> > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > >> > > > >
> > >> > > > > There are already open-source alternatives are available. But
> > >>we
> > >> > would
> > >> > > > > like to add REST server that
> > >> > > > > many users ask for under Apache Kafka repo. Many data Infra
> > >>tools
> > >> > comes
> > >> > > > up
> > >> > > > > with Rest Interface.
> > >> > > > > It is useful to have inbuilt Rest API support for Produce,
> > >>Consume
> > >> > > > messages
> > >> > > > > and admin interface for
> > >> > > > > integrating with external management and provisioning
> tools.This
> > >> will
> > >> > > > also
> > >> > > > > allow the maintenance of
> > >> > > > > REST server and adding new features makes it easy because
> apache
> > >> > > > community.
> > >> > > > >
> > >> > > > > The KIP wiki is the following:
> > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> > > > > 80%3A+Kafka+Rest+Server
> > >> > > > >
> > >> > > > > Your comments and feedback are welcome.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Manikumar
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > --
> > >> > Thanks,
> > >> > Neha
> > >> >
> > >>
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Harsha Chintalapani <ka...@harsha.io>.
Neha,
"But I haven't seen any community emails or patches being submitted by you
guys, so I'm wondering why you are concerned about whether the community is
open to accepting patches or not."
I think you are talking about contributing patches to this repository
right? https://github.com/confluentinc/kafka-rest . All I am saying
the guidelines/governance model is not clear on the project and I guess its
driven by opening a github issue request. Its the repository owned by
confluent and as much I appreciate that the features we mentioned are in
the roadmap and welcoming us to contribute to the project. It doesn't
gurantee what we want to add in the furture will be in your roadmap.
Hence the reason having it part of Kafka community will help a lot as other
users can participate in the discussions. We are happy to drive any
feature additions through KIPs which gives everyone a chance to participate
and add to the discussions.
Thanks,
Harsha
On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <Mi...@ig.com>
wrote:
> +1
>
> I agree on the governance comments whole heartedly.
>
> Also i agree about the contribution comments made earlier in the thread, i
> personally am less likely to spend any of mine, or give project time within
> my internal projects to developers contributing to another commercial
> companies project even so technically open source, as then there is that
> commercial companies interest will always prevail and essentially can
> always have a final vote where disagreement. Im sure they never intend to,
> but there is that true reality. This is why we have community open source
> projects.
>
> I can find many different implementations now of a rest endpoint on
> GitHub, BitBucket etc. Each one has their benefits and disadvantages in
> their implementation. By making / providing one this would bring together
> these solutions, unifying those developers and also bringing the best of
> all.
>
> I understand the concern on the community burden adding/supporting more
> surface area for every client. But something like REST is universal and
> worthy to be owned by the community.
>
> Mike
>
>
> ________________________________________
> From: Andrew Schofield <an...@outlook.com>
> Sent: Saturday, October 8, 2016 1:19 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> There's a massive difference between the governance of Kafka and the
> governance of the REST proxy.
>
> In Kafka, there is a broad community of people contributing their opinions
> about future enhancements in the form of KIPs. There's some really deep
> consideration that goes into some of the trickier KIPs. There are people
> outside Confluent deeply knowledgeable in Kafka and building the
> reputations to become committers. I get the impression that the roadmap of
> Kafka is not really community-owned (what's the big feature for Kafka 0.11,
> for example), but the conveyor belt of smaller features in the form of KIPs
> works nicely. It's a good example of open-source working well.
>
> The equivalent for the REST proxy is basically issues on GitHub. The
> roadmap is less clear. There's not really a community properly engaged in
> the way that there is with Kafka. So, you could say that it's clear that
> fewer people are interested, but I think the whole governance thing is a
> big barrier to engagement. And it's looking like it's getting out of date.
>
> In technical terms, I can think of two big improvements to the REST proxy.
> First, it needs to use the new consumer API so that it's possible to secure
> connections between the REST proxy and Kafka. I don't care too much which
> method calls it uses actually uses to consume messages, but I do care that
> I cannot secure connections because of network security rules. Second,
> there's an affinity between a consumer and the instance of the REST proxy
> to which it first connected. Kafka itself avoids this kind of affinity for
> good reason, and in the name of availability the REST proxy should too.
> These are natural KIPs.
>
> I think it would be good to have the code for the REST proxy contributed
> to Apache Kafka so that it would be able to be developed in the same way.
>
> Andrew Schofield
>
> From: Suresh Srinivas <su...@hortonworks.com>
> Sent: 07 October 2016 22:41:52
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> ASF already gives us a clear framework and governance model for community
> development. This is already understood by the people contributing to
> Apache Kafka project, and they are the same people who want to contribute
> to the REST server capability as well. Everyone is in agreement on the
> need for collaborating on this effort. So why not contribute the code to
> Apache Kafka. This will help avoid duplication of effort and forks that
> may crop up, hugely benefitting the user community. This will also avoid
> having to define a process similar to ASF on a GitHub project and instead
> there is a single community with clear understanding community process as
> defined in ASF.
>
> As others have said, this is an important capability for Apache Kafka. It
> is worth maintaining this as a part of the project.
>
> Regards,
> Suresh
>
> On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
>
> >I personally think it would be quite wasteful to re-implement the REST
> >gateway just because that an actively-maintained piece of Apache-licensed
> >software is not governed directly by the Apache Kafka community... While
> >kafka-rest repo is owned by Confluent, the contributors including the main
> >one are also part of the Apache Kafka community, so there is a chance to
> >work this out.
> >
> >However, there are two valid concerns here that could be addressed, around
> >community and accessibility:
> >>> What we are worried about is a project
> >>> that's not maintained in a community. So the process of accepting
> >>>patches
> >>> and priorities is not clear, and it's not developed in Apache
> >>>community.
> >>> Not only that, existing REST API project doesn't support new client API
> >and
> >>> hence there is no security support either.
> >
> >This might be easy to fix. Maybe Confluent / kafka-rest community can
> >clarify that - what is their contribution policy, dev style, roadmap etc.
> >If they want, they can make an effort to encourage participation from
> >people outside Confluent (easily accept contributions, invite external
> >commiters or have open dev process similar to Apache Kafka etc), as there
> >is definitely seems to be some interest on the list. That might clear the
> >community concern and help kafka-rest project (but that is a calculation
> >Confluent will have to make).
> >
> >The other, independent, concern is that REST is something that is expected
> >to be available out of the box with Kafka. I personally don't feel
> >strongly
> >about it (better use proper, efficient APIs from day one), though it is
> >definitely way smaller than adding a stream processing engine to the
> >project :)
> >Again,the kafka-rest "community" could take steps to make it even easier
> >to
> >install, configure and run kafka-rest for new users on vanilla Apache
> >Kafka
> >(outside the Confluent platform), if they wish that (or welcome
> >contributions to that end), but that is up to them.
> >Finally, if after the above steps were taken there would still a strong
> >desire to include a great rest gateway with Apache Kafka, I assume the
> >community could hypothetically fork the existing kafka-rest into an Apache
> >Kafka subproject and maintain it "within Apache" instead of implementing
> >it
> >from scratch (though I'm not a lawyer etc) - but I cannot imagine it
> >happen
> >without Confluent blessing, and I think that is likely much less optimal
> >(pulling in other Confluent / Apache licensed dependencies) than having a
> >separate external community around kafka-rest.
> >
> >
> >Just my two cents,
> >
> >
> >Ofir Manor
> >
> >Co-Founder & CTO | Equalum
> >
> >Mobile: +972-54-7801286 <+972%2054-780-1286> | Email:
> ofir.manor@equalum.io
> >
> >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <ka...@harsha.io>
> >wrote:
> >
> >> Neha & Jay,
> >> We did look at the open source alternatives. Our
> >>concern
> >> is what's the patch acceptance and adding features/ bug-fixes to the
> >> existing project under a Github (although it's licensed under Apache
> >>2.0).
> >> It would be great if that project made available under Apache and
> >>driven by
> >> the community. Adding to the above, not all Kafka users are interested
> >>in
> >> using the Java client API, they would like to have simple REST API where
> >> they can code against using any language. I do believe this adds value
> >>to
> >> Apache Kafka in itself.
> >>
> >> "For 1, I don't think there is value in giving in to the NIH syndrome
> >>and
> >> reinventing the wheel. What I'm looking for is a detailed comparison of
> >>the
> >> gaps and why those can't be improved in the REST proxy that already
> >>exists
> >> and is actively maintained."
> >>
> >> We are not looking at this as NIH. What we are worried about is a
> >>project
> >> that's not maintained in a community. So the process of accepting
> >>patches
> >> and priorities is not clear, and it's not developed in Apache community.
> >> Not only that, existing REST API project doesn't support new client API
> >>and
> >> hence there is no security support either.
> >> We don't know the timeline when that's made available. We would like to
> >>add
> >> admin functionality into the REST API. So the Roadmap of that project is
> >> not driven by Apache.
> >>
> >>
> >> "This doesn't materially have an impact on expanding the usability of
> >> Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> >> all
> >> Kafka users, and maybe 10% of those are the ones who will use the
> >>REST
> >> proxy. The remaining 50% are non-java client users (C, python, go,
> >>node
> >> etc)."
> >>
> >> REST API is most often asked feature in my interactions with Kafka
> >>users.
> >> In an organization, There will be independent teams who will write their
> >> Kafka clients using different language libraries available today, and
> >> there is no way to standardize this. Instead of supporting several
> >> different client libraries users will be interested in using a REST API
> >> server. The need for a REST API will only increase as more and more
> >>users
> >> start using Kafka.
> >>
> >> "More surface area means more work to keep things consistent. Failure
> >> to do that has, in fact, hurt the user experience."
> >> Having myriad Kafka client GitHub projects that support different
> >>languages
> >> hurts the user experience and pushes burden to maintain these libraries.
> >> REST API is a simple code base that uses existing java client libraries
> >>to
> >> make life easier for the users.
> >>
> >> Thanks,
> >> Harsha
> >>
> >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io>
> wrote:
> >>
> >> > Manikumar,
> >> >
> >> > Thanks for sharing the proposal. I think there are 2 parts to this
> >> > discussion -
> >> >
> >> > 1. Should we rewrite a REST proxy given that there is a
> >>feature-complete,
> >> > open-source and actively maintained REST proxy in the community?
> >> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> >> maintain
> >> > the high-quality experience that Kafka users have today?
> >> >
> >> > For 1, I don't think there is value in giving in to the NIH syndrome
> >>and
> >> > reinventing the wheel. What I'm looking for is a detailed comparison
> >>of
> >> the
> >> > gaps and why those can't be improved in the REST proxy that already
> >> exists
> >> > and is actively maintained. For example, we depend on zkClient and
> >>have
> >> > found as well as fixed several bugs by working closely with the people
> >> who
> >> > maintain zkClient. This should be possible for REST proxy as well,
> >>right?
> >> >
> >> > For 2, I'd like us to review our history of expanding the surface
> >>area to
> >> > add more clients in the past. Here is a summary -
> >> >
> >> > - This doesn't materially have an impact on expanding the
> >>usability of
> >> > Kafka. In my experience, REST proxy + Java clients only cover ~50%
> >>of
> >> > all
> >> > Kafka users, and maybe 10% of those are the ones who will use the
> >>REST
> >> > proxy. The remaining 50% are non-java client users (C, python, go,
> >> node
> >> > etc).
> >> > - People are a lot more excited about promising to contribute while
> >> > adding the surface area but not on an ongoing basis down the line.
> >> > - More surface area means more work to keep things consistent.
> >>Failure
> >> > to do that has, in fact, hurt the user experience.
> >> > - More surface area hurts agility. We want to do a few things
> >>really
> >> > well as well as be agile to be able to build on our core
> >>competency.
> >> >
> >> >
> >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi Jay,
> >> > >
> >> > > Thanks for your reply.
> >> > >
> >> > > I agree that we can not add all the clients/tools available in
> >> ecosystem
> >> > > page to Kafka repo itself. But we feel REST Interface is different
> >>from
> >> > > other clients/tools. Since any language that can work with HTTP can
> >> > > easily integrate with this interface, Having an "official" REST
> >> > > interface helps user community. This also helps us to integrate well
> >> > > with external management and provisioning tools. Apache Kafka
> >>release
> >> > > with Java clients + REST interface is sufficient for most of the
> >>user
> >> > > deployments/requirements. This helps users to deal with less number
> >> > > of distributions/builds.
> >> > >
> >> > > Thanks,
> >> > > Manikumar
> >> > >
> >> > >
> >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> >> > >
> >> > > > Hey guys,
> >> > > >
> >> > > > There's already a REST interface maintained as a separate
> >> project--it's
> >> > > > open source and apache licensed and actively maintained (
> >> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
>
>
>
> GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> github.com
> The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It
> makes it easy to produce and consume messages, view the state of the
> cluster, and ...
>
> >> that?
> >> > > You
> >> > > > mentioned that there was some compatibility concern, but
> >> compatibility
> >> > > has
> >> > > > to do with the consumer protocol guarantees not the repo the code
> >>is
> >> > in,
> >> > > > right? Not sure that concern makes sense.
> >> > > >
> >> > > > We could argue for adding pretty much anything and everything in
> >>the
> >> > > > ecosystem page in Kafka itself but I'm not sure that would make
> >>the
> >> > > project
> >> > > > more agile.
> >> > > >
> >> > > > -Jay
> >> > > >
> >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> >> manikumar.reddy@gmail.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Kafka Devs,
> >> > > > >
> >> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> >> > > > >
> >> > > > > There are already open-source alternatives are available. But
> >>we
> >> > would
> >> > > > > like to add REST server that
> >> > > > > many users ask for under Apache Kafka repo. Many data Infra
> >>tools
> >> > comes
> >> > > > up
> >> > > > > with Rest Interface.
> >> > > > > It is useful to have inbuilt Rest API support for Produce,
> >>Consume
> >> > > > messages
> >> > > > > and admin interface for
> >> > > > > integrating with external management and provisioning tools.This
> >> will
> >> > > > also
> >> > > > > allow the maintenance of
> >> > > > > REST server and adding new features makes it easy because apache
> >> > > > community.
> >> > > > >
> >> > > > > The KIP wiki is the following:
> >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > > > 80%3A+Kafka+Rest+Server
> >> > > > >
> >> > > > > Your comments and feedback are welcome.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Manikumar
> >> > > > >
> >> > > >
> >> > >
> >> > --
> >> > Thanks,
> >> > Neha
> >> >
> >>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Michael Pearce <Mi...@ig.com>.
+1
I agree on the governance comments whole heartedly.
Also i agree about the contribution comments made earlier in the thread, i personally am less likely to spend any of mine, or give project time within my internal projects to developers contributing to another commercial companies project even so technically open source, as then there is that commercial companies interest will always prevail and essentially can always have a final vote where disagreement. Im sure they never intend to, but there is that true reality. This is why we have community open source projects.
I can find many different implementations now of a rest endpoint on GitHub, BitBucket etc. Each one has their benefits and disadvantages in their implementation. By making / providing one this would bring together these solutions, unifying those developers and also bringing the best of all.
I understand the concern on the community burden adding/supporting more surface area for every client. But something like REST is universal and worthy to be owned by the community.
Mike
________________________________________
From: Andrew Schofield <an...@outlook.com>
Sent: Saturday, October 8, 2016 1:19 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
There's a massive difference between the governance of Kafka and the governance of the REST proxy.
In Kafka, there is a broad community of people contributing their opinions about future enhancements in the form of KIPs. There's some really deep consideration that goes into some of the trickier KIPs. There are people outside Confluent deeply knowledgeable in Kafka and building the reputations to become committers. I get the impression that the roadmap of Kafka is not really community-owned (what's the big feature for Kafka 0.11, for example), but the conveyor belt of smaller features in the form of KIPs works nicely. It's a good example of open-source working well.
The equivalent for the REST proxy is basically issues on GitHub. The roadmap is less clear. There's not really a community properly engaged in the way that there is with Kafka. So, you could say that it's clear that fewer people are interested, but I think the whole governance thing is a big barrier to engagement. And it's looking like it's getting out of date.
In technical terms, I can think of two big improvements to the REST proxy. First, it needs to use the new consumer API so that it's possible to secure connections between the REST proxy and Kafka. I don't care too much which method calls it uses actually uses to consume messages, but I do care that I cannot secure connections because of network security rules. Second, there's an affinity between a consumer and the instance of the REST proxy to which it first connected. Kafka itself avoids this kind of affinity for good reason, and in the name of availability the REST proxy should too. These are natural KIPs.
I think it would be good to have the code for the REST proxy contributed to Apache Kafka so that it would be able to be developed in the same way.
Andrew Schofield
From: Suresh Srinivas <su...@hortonworks.com>
Sent: 07 October 2016 22:41:52
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
ASF already gives us a clear framework and governance model for community
development. This is already understood by the people contributing to
Apache Kafka project, and they are the same people who want to contribute
to the REST server capability as well. Everyone is in agreement on the
need for collaborating on this effort. So why not contribute the code to
Apache Kafka. This will help avoid duplication of effort and forks that
may crop up, hugely benefitting the user community. This will also avoid
having to define a process similar to ASF on a GitHub project and instead
there is a single community with clear understanding community process as
defined in ASF.
As others have said, this is an important capability for Apache Kafka. It
is worth maintaining this as a part of the project.
Regards,
Suresh
On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
>I personally think it would be quite wasteful to re-implement the REST
>gateway just because that an actively-maintained piece of Apache-licensed
>software is not governed directly by the Apache Kafka community... While
>kafka-rest repo is owned by Confluent, the contributors including the main
>one are also part of the Apache Kafka community, so there is a chance to
>work this out.
>
>However, there are two valid concerns here that could be addressed, around
>community and accessibility:
>>> What we are worried about is a project
>>> that's not maintained in a community. So the process of accepting
>>>patches
>>> and priorities is not clear, and it's not developed in Apache
>>>community.
>>> Not only that, existing REST API project doesn't support new client API
>and
>>> hence there is no security support either.
>
>This might be easy to fix. Maybe Confluent / kafka-rest community can
>clarify that - what is their contribution policy, dev style, roadmap etc.
>If they want, they can make an effort to encourage participation from
>people outside Confluent (easily accept contributions, invite external
>commiters or have open dev process similar to Apache Kafka etc), as there
>is definitely seems to be some interest on the list. That might clear the
>community concern and help kafka-rest project (but that is a calculation
>Confluent will have to make).
>
>The other, independent, concern is that REST is something that is expected
>to be available out of the box with Kafka. I personally don't feel
>strongly
>about it (better use proper, efficient APIs from day one), though it is
>definitely way smaller than adding a stream processing engine to the
>project :)
>Again,the kafka-rest "community" could take steps to make it even easier
>to
>install, configure and run kafka-rest for new users on vanilla Apache
>Kafka
>(outside the Confluent platform), if they wish that (or welcome
>contributions to that end), but that is up to them.
>Finally, if after the above steps were taken there would still a strong
>desire to include a great rest gateway with Apache Kafka, I assume the
>community could hypothetically fork the existing kafka-rest into an Apache
>Kafka subproject and maintain it "within Apache" instead of implementing
>it
>from scratch (though I'm not a lawyer etc) - but I cannot imagine it
>happen
>without Confluent blessing, and I think that is likely much less optimal
>(pulling in other Confluent / Apache licensed dependencies) than having a
>separate external community around kafka-rest.
>
>
>Just my two cents,
>
>
>Ofir Manor
>
>Co-Founder & CTO | Equalum
>
>Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io
>
>On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <ka...@harsha.io>
>wrote:
>
>> Neha & Jay,
>> We did look at the open source alternatives. Our
>>concern
>> is what's the patch acceptance and adding features/ bug-fixes to the
>> existing project under a Github (although it's licensed under Apache
>>2.0).
>> It would be great if that project made available under Apache and
>>driven by
>> the community. Adding to the above, not all Kafka users are interested
>>in
>> using the Java client API, they would like to have simple REST API where
>> they can code against using any language. I do believe this adds value
>>to
>> Apache Kafka in itself.
>>
>> "For 1, I don't think there is value in giving in to the NIH syndrome
>>and
>> reinventing the wheel. What I'm looking for is a detailed comparison of
>>the
>> gaps and why those can't be improved in the REST proxy that already
>>exists
>> and is actively maintained."
>>
>> We are not looking at this as NIH. What we are worried about is a
>>project
>> that's not maintained in a community. So the process of accepting
>>patches
>> and priorities is not clear, and it's not developed in Apache community.
>> Not only that, existing REST API project doesn't support new client API
>>and
>> hence there is no security support either.
>> We don't know the timeline when that's made available. We would like to
>>add
>> admin functionality into the REST API. So the Roadmap of that project is
>> not driven by Apache.
>>
>>
>> "This doesn't materially have an impact on expanding the usability of
>> Kafka. In my experience, REST proxy + Java clients only cover ~50% of
>> all
>> Kafka users, and maybe 10% of those are the ones who will use the
>>REST
>> proxy. The remaining 50% are non-java client users (C, python, go,
>>node
>> etc)."
>>
>> REST API is most often asked feature in my interactions with Kafka
>>users.
>> In an organization, There will be independent teams who will write their
>> Kafka clients using different language libraries available today, and
>> there is no way to standardize this. Instead of supporting several
>> different client libraries users will be interested in using a REST API
>> server. The need for a REST API will only increase as more and more
>>users
>> start using Kafka.
>>
>> "More surface area means more work to keep things consistent. Failure
>> to do that has, in fact, hurt the user experience."
>> Having myriad Kafka client GitHub projects that support different
>>languages
>> hurts the user experience and pushes burden to maintain these libraries.
>> REST API is a simple code base that uses existing java client libraries
>>to
>> make life easier for the users.
>>
>> Thanks,
>> Harsha
>>
>> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io> wrote:
>>
>> > Manikumar,
>> >
>> > Thanks for sharing the proposal. I think there are 2 parts to this
>> > discussion -
>> >
>> > 1. Should we rewrite a REST proxy given that there is a
>>feature-complete,
>> > open-source and actively maintained REST proxy in the community?
>> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
>> maintain
>> > the high-quality experience that Kafka users have today?
>> >
>> > For 1, I don't think there is value in giving in to the NIH syndrome
>>and
>> > reinventing the wheel. What I'm looking for is a detailed comparison
>>of
>> the
>> > gaps and why those can't be improved in the REST proxy that already
>> exists
>> > and is actively maintained. For example, we depend on zkClient and
>>have
>> > found as well as fixed several bugs by working closely with the people
>> who
>> > maintain zkClient. This should be possible for REST proxy as well,
>>right?
>> >
>> > For 2, I'd like us to review our history of expanding the surface
>>area to
>> > add more clients in the past. Here is a summary -
>> >
>> > - This doesn't materially have an impact on expanding the
>>usability of
>> > Kafka. In my experience, REST proxy + Java clients only cover ~50%
>>of
>> > all
>> > Kafka users, and maybe 10% of those are the ones who will use the
>>REST
>> > proxy. The remaining 50% are non-java client users (C, python, go,
>> node
>> > etc).
>> > - People are a lot more excited about promising to contribute while
>> > adding the surface area but not on an ongoing basis down the line.
>> > - More surface area means more work to keep things consistent.
>>Failure
>> > to do that has, in fact, hurt the user experience.
>> > - More surface area hurts agility. We want to do a few things
>>really
>> > well as well as be agile to be able to build on our core
>>competency.
>> >
>> >
>> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com>
>> > wrote:
>> >
>> > > Hi Jay,
>> > >
>> > > Thanks for your reply.
>> > >
>> > > I agree that we can not add all the clients/tools available in
>> ecosystem
>> > > page to Kafka repo itself. But we feel REST Interface is different
>>from
>> > > other clients/tools. Since any language that can work with HTTP can
>> > > easily integrate with this interface, Having an "official" REST
>> > > interface helps user community. This also helps us to integrate well
>> > > with external management and provisioning tools. Apache Kafka
>>release
>> > > with Java clients + REST interface is sufficient for most of the
>>user
>> > > deployments/requirements. This helps users to deal with less number
>> > > of distributions/builds.
>> > >
>> > > Thanks,
>> > > Manikumar
>> > >
>> > >
>> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
>> > >
>> > > > Hey guys,
>> > > >
>> > > > There's already a REST interface maintained as a separate
>> project--it's
>> > > > open source and apache licensed and actively maintained (
>> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
github.com
The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It makes it easy to produce and consume messages, view the state of the cluster, and ...
>> that?
>> > > You
>> > > > mentioned that there was some compatibility concern, but
>> compatibility
>> > > has
>> > > > to do with the consumer protocol guarantees not the repo the code
>>is
>> > in,
>> > > > right? Not sure that concern makes sense.
>> > > >
>> > > > We could argue for adding pretty much anything and everything in
>>the
>> > > > ecosystem page in Kafka itself but I'm not sure that would make
>>the
>> > > project
>> > > > more agile.
>> > > >
>> > > > -Jay
>> > > >
>> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
>> manikumar.reddy@gmail.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Hi Kafka Devs,
>> > > > >
>> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
>> > > > >
>> > > > > There are already open-source alternatives are available. But
>>we
>> > would
>> > > > > like to add REST server that
>> > > > > many users ask for under Apache Kafka repo. Many data Infra
>>tools
>> > comes
>> > > > up
>> > > > > with Rest Interface.
>> > > > > It is useful to have inbuilt Rest API support for Produce,
>>Consume
>> > > > messages
>> > > > > and admin interface for
>> > > > > integrating with external management and provisioning tools.This
>> will
>> > > > also
>> > > > > allow the maintenance of
>> > > > > REST server and adding new features makes it easy because apache
>> > > > community.
>> > > > >
>> > > > > The KIP wiki is the following:
>> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > 80%3A+Kafka+Rest+Server
>> > > > >
>> > > > > Your comments and feedback are welcome.
>> > > > >
>> > > > > Thanks,
>> > > > > Manikumar
>> > > > >
>> > > >
>> > >
>> > --
>> > Thanks,
>> > Neha
>> >
>>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Andrew Schofield <an...@outlook.com>.
There's a massive difference between the governance of Kafka and the governance of the REST proxy.
In Kafka, there is a broad community of people contributing their opinions about future enhancements in the form of KIPs. There's some really deep consideration that goes into some of the trickier KIPs. There are people outside Confluent deeply knowledgeable in Kafka and building the reputations to become committers. I get the impression that the roadmap of Kafka is not really community-owned (what's the big feature for Kafka 0.11, for example), but the conveyor belt of smaller features in the form of KIPs works nicely. It's a good example of open-source working well.
The equivalent for the REST proxy is basically issues on GitHub. The roadmap is less clear. There's not really a community properly engaged in the way that there is with Kafka. So, you could say that it's clear that fewer people are interested, but I think the whole governance thing is a big barrier to engagement. And it's looking like it's getting out of date.
In technical terms, I can think of two big improvements to the REST proxy. First, it needs to use the new consumer API so that it's possible to secure connections between the REST proxy and Kafka. I don't care too much which method calls it uses actually uses to consume messages, but I do care that I cannot secure connections because of network security rules. Second, there's an affinity between a consumer and the instance of the REST proxy to which it first connected. Kafka itself avoids this kind of affinity for good reason, and in the name of availability the REST proxy should too. These are natural KIPs.
I think it would be good to have the code for the REST proxy contributed to Apache Kafka so that it would be able to be developed in the same way.
Andrew Schofield
From: Suresh Srinivas <su...@hortonworks.com>
Sent: 07 October 2016 22:41:52
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
ASF already gives us a clear framework and governance model for community
development. This is already understood by the people contributing to
Apache Kafka project, and they are the same people who want to contribute
to the REST server capability as well. Everyone is in agreement on the
need for collaborating on this effort. So why not contribute the code to
Apache Kafka. This will help avoid duplication of effort and forks that
may crop up, hugely benefitting the user community. This will also avoid
having to define a process similar to ASF on a GitHub project and instead
there is a single community with clear understanding community process as
defined in ASF.
As others have said, this is an important capability for Apache Kafka. It
is worth maintaining this as a part of the project.
Regards,
Suresh
On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
>I personally think it would be quite wasteful to re-implement the REST
>gateway just because that an actively-maintained piece of Apache-licensed
>software is not governed directly by the Apache Kafka community... While
>kafka-rest repo is owned by Confluent, the contributors including the main
>one are also part of the Apache Kafka community, so there is a chance to
>work this out.
>
>However, there are two valid concerns here that could be addressed, around
>community and accessibility:
>>> What we are worried about is a project
>>> that's not maintained in a community. So the process of accepting
>>>patches
>>> and priorities is not clear, and it's not developed in Apache
>>>community.
>>> Not only that, existing REST API project doesn't support new client API
>and
>>> hence there is no security support either.
>
>This might be easy to fix. Maybe Confluent / kafka-rest community can
>clarify that - what is their contribution policy, dev style, roadmap etc.
>If they want, they can make an effort to encourage participation from
>people outside Confluent (easily accept contributions, invite external
>commiters or have open dev process similar to Apache Kafka etc), as there
>is definitely seems to be some interest on the list. That might clear the
>community concern and help kafka-rest project (but that is a calculation
>Confluent will have to make).
>
>The other, independent, concern is that REST is something that is expected
>to be available out of the box with Kafka. I personally don't feel
>strongly
>about it (better use proper, efficient APIs from day one), though it is
>definitely way smaller than adding a stream processing engine to the
>project :)
>Again,the kafka-rest "community" could take steps to make it even easier
>to
>install, configure and run kafka-rest for new users on vanilla Apache
>Kafka
>(outside the Confluent platform), if they wish that (or welcome
>contributions to that end), but that is up to them.
>Finally, if after the above steps were taken there would still a strong
>desire to include a great rest gateway with Apache Kafka, I assume the
>community could hypothetically fork the existing kafka-rest into an Apache
>Kafka subproject and maintain it "within Apache" instead of implementing
>it
>from scratch (though I'm not a lawyer etc) - but I cannot imagine it
>happen
>without Confluent blessing, and I think that is likely much less optimal
>(pulling in other Confluent / Apache licensed dependencies) than having a
>separate external community around kafka-rest.
>
>
>Just my two cents,
>
>
>Ofir Manor
>
>Co-Founder & CTO | Equalum
>
>Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io
>
>On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <ka...@harsha.io>
>wrote:
>
>> Neha & Jay,
>> We did look at the open source alternatives. Our
>>concern
>> is what's the patch acceptance and adding features/ bug-fixes to the
>> existing project under a Github (although it's licensed under Apache
>>2.0).
>> It would be great if that project made available under Apache and
>>driven by
>> the community. Adding to the above, not all Kafka users are interested
>>in
>> using the Java client API, they would like to have simple REST API where
>> they can code against using any language. I do believe this adds value
>>to
>> Apache Kafka in itself.
>>
>> "For 1, I don't think there is value in giving in to the NIH syndrome
>>and
>> reinventing the wheel. What I'm looking for is a detailed comparison of
>>the
>> gaps and why those can't be improved in the REST proxy that already
>>exists
>> and is actively maintained."
>>
>> We are not looking at this as NIH. What we are worried about is a
>>project
>> that's not maintained in a community. So the process of accepting
>>patches
>> and priorities is not clear, and it's not developed in Apache community.
>> Not only that, existing REST API project doesn't support new client API
>>and
>> hence there is no security support either.
>> We don't know the timeline when that's made available. We would like to
>>add
>> admin functionality into the REST API. So the Roadmap of that project is
>> not driven by Apache.
>>
>>
>> "This doesn't materially have an impact on expanding the usability of
>> Kafka. In my experience, REST proxy + Java clients only cover ~50% of
>> all
>> Kafka users, and maybe 10% of those are the ones who will use the
>>REST
>> proxy. The remaining 50% are non-java client users (C, python, go,
>>node
>> etc)."
>>
>> REST API is most often asked feature in my interactions with Kafka
>>users.
>> In an organization, There will be independent teams who will write their
>> Kafka clients using different language libraries available today, and
>> there is no way to standardize this. Instead of supporting several
>> different client libraries users will be interested in using a REST API
>> server. The need for a REST API will only increase as more and more
>>users
>> start using Kafka.
>>
>> "More surface area means more work to keep things consistent. Failure
>> to do that has, in fact, hurt the user experience."
>> Having myriad Kafka client GitHub projects that support different
>>languages
>> hurts the user experience and pushes burden to maintain these libraries.
>> REST API is a simple code base that uses existing java client libraries
>>to
>> make life easier for the users.
>>
>> Thanks,
>> Harsha
>>
>> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io> wrote:
>>
>> > Manikumar,
>> >
>> > Thanks for sharing the proposal. I think there are 2 parts to this
>> > discussion -
>> >
>> > 1. Should we rewrite a REST proxy given that there is a
>>feature-complete,
>> > open-source and actively maintained REST proxy in the community?
>> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
>> maintain
>> > the high-quality experience that Kafka users have today?
>> >
>> > For 1, I don't think there is value in giving in to the NIH syndrome
>>and
>> > reinventing the wheel. What I'm looking for is a detailed comparison
>>of
>> the
>> > gaps and why those can't be improved in the REST proxy that already
>> exists
>> > and is actively maintained. For example, we depend on zkClient and
>>have
>> > found as well as fixed several bugs by working closely with the people
>> who
>> > maintain zkClient. This should be possible for REST proxy as well,
>>right?
>> >
>> > For 2, I'd like us to review our history of expanding the surface
>>area to
>> > add more clients in the past. Here is a summary -
>> >
>> > - This doesn't materially have an impact on expanding the
>>usability of
>> > Kafka. In my experience, REST proxy + Java clients only cover ~50%
>>of
>> > all
>> > Kafka users, and maybe 10% of those are the ones who will use the
>>REST
>> > proxy. The remaining 50% are non-java client users (C, python, go,
>> node
>> > etc).
>> > - People are a lot more excited about promising to contribute while
>> > adding the surface area but not on an ongoing basis down the line.
>> > - More surface area means more work to keep things consistent.
>>Failure
>> > to do that has, in fact, hurt the user experience.
>> > - More surface area hurts agility. We want to do a few things
>>really
>> > well as well as be agile to be able to build on our core
>>competency.
>> >
>> >
>> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com>
>> > wrote:
>> >
>> > > Hi Jay,
>> > >
>> > > Thanks for your reply.
>> > >
>> > > I agree that we can not add all the clients/tools available in
>> ecosystem
>> > > page to Kafka repo itself. But we feel REST Interface is different
>>from
>> > > other clients/tools. Since any language that can work with HTTP can
>> > > easily integrate with this interface, Having an "official" REST
>> > > interface helps user community. This also helps us to integrate well
>> > > with external management and provisioning tools. Apache Kafka
>>release
>> > > with Java clients + REST interface is sufficient for most of the
>>user
>> > > deployments/requirements. This helps users to deal with less number
>> > > of distributions/builds.
>> > >
>> > > Thanks,
>> > > Manikumar
>> > >
>> > >
>> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
>> > >
>> > > > Hey guys,
>> > > >
>> > > > There's already a REST interface maintained as a separate
>> project--it's
>> > > > open source and apache licensed and actively maintained (
>> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
github.com
The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It makes it easy to produce and consume messages, view the state of the cluster, and ...
>> that?
>> > > You
>> > > > mentioned that there was some compatibility concern, but
>> compatibility
>> > > has
>> > > > to do with the consumer protocol guarantees not the repo the code
>>is
>> > in,
>> > > > right? Not sure that concern makes sense.
>> > > >
>> > > > We could argue for adding pretty much anything and everything in
>>the
>> > > > ecosystem page in Kafka itself but I'm not sure that would make
>>the
>> > > project
>> > > > more agile.
>> > > >
>> > > > -Jay
>> > > >
>> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
>> manikumar.reddy@gmail.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Hi Kafka Devs,
>> > > > >
>> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
>> > > > >
>> > > > > There are already open-source alternatives are available. But
>>we
>> > would
>> > > > > like to add REST server that
>> > > > > many users ask for under Apache Kafka repo. Many data Infra
>>tools
>> > comes
>> > > > up
>> > > > > with Rest Interface.
>> > > > > It is useful to have inbuilt Rest API support for Produce,
>>Consume
>> > > > messages
>> > > > > and admin interface for
>> > > > > integrating with external management and provisioning tools.This
>> will
>> > > > also
>> > > > > allow the maintenance of
>> > > > > REST server and adding new features makes it easy because apache
>> > > > community.
>> > > > >
>> > > > > The KIP wiki is the following:
>> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > 80%3A+Kafka+Rest+Server
>> > > > >
>> > > > > Your comments and feedback are welcome.
>> > > > >
>> > > > > Thanks,
>> > > > > Manikumar
>> > > > >
>> > > >
>> > >
>> > --
>> > Thanks,
>> > Neha
>> >
>>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Suresh Srinivas <su...@hortonworks.com>.
ASF already gives us a clear framework and governance model for community
development. This is already understood by the people contributing to
Apache Kafka project, and they are the same people who want to contribute
to the REST server capability as well. Everyone is in agreement on the
need for collaborating on this effort. So why not contribute the code to
Apache Kafka. This will help avoid duplication of effort and forks that
may crop up, hugely benefitting the user community. This will also avoid
having to define a process similar to ASF on a GitHub project and instead
there is a single community with clear understanding community process as
defined in ASF.
As others have said, this is an important capability for Apache Kafka. It
is worth maintaining this as a part of the project.
Regards,
Suresh
On 10/6/16, 8:32 AM, "Ofir Manor" <of...@equalum.io> wrote:
>I personally think it would be quite wasteful to re-implement the REST
>gateway just because that an actively-maintained piece of Apache-licensed
>software is not governed directly by the Apache Kafka community... While
>kafka-rest repo is owned by Confluent, the contributors including the main
>one are also part of the Apache Kafka community, so there is a chance to
>work this out.
>
>However, there are two valid concerns here that could be addressed, around
>community and accessibility:
>>> What we are worried about is a project
>>> that's not maintained in a community. So the process of accepting
>>>patches
>>> and priorities is not clear, and it's not developed in Apache
>>>community.
>>> Not only that, existing REST API project doesn't support new client API
>and
>>> hence there is no security support either.
>
>This might be easy to fix. Maybe Confluent / kafka-rest community can
>clarify that - what is their contribution policy, dev style, roadmap etc.
>If they want, they can make an effort to encourage participation from
>people outside Confluent (easily accept contributions, invite external
>commiters or have open dev process similar to Apache Kafka etc), as there
>is definitely seems to be some interest on the list. That might clear the
>community concern and help kafka-rest project (but that is a calculation
>Confluent will have to make).
>
>The other, independent, concern is that REST is something that is expected
>to be available out of the box with Kafka. I personally don't feel
>strongly
>about it (better use proper, efficient APIs from day one), though it is
>definitely way smaller than adding a stream processing engine to the
>project :)
>Again,the kafka-rest "community" could take steps to make it even easier
>to
>install, configure and run kafka-rest for new users on vanilla Apache
>Kafka
>(outside the Confluent platform), if they wish that (or welcome
>contributions to that end), but that is up to them.
>Finally, if after the above steps were taken there would still a strong
>desire to include a great rest gateway with Apache Kafka, I assume the
>community could hypothetically fork the existing kafka-rest into an Apache
>Kafka subproject and maintain it "within Apache" instead of implementing
>it
>from scratch (though I'm not a lawyer etc) - but I cannot imagine it
>happen
>without Confluent blessing, and I think that is likely much less optimal
>(pulling in other Confluent / Apache licensed dependencies) than having a
>separate external community around kafka-rest.
>
>
>Just my two cents,
>
>
>Ofir Manor
>
>Co-Founder & CTO | Equalum
>
>Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io
>
>On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <ka...@harsha.io>
>wrote:
>
>> Neha & Jay,
>> We did look at the open source alternatives. Our
>>concern
>> is what's the patch acceptance and adding features/ bug-fixes to the
>> existing project under a Github (although it's licensed under Apache
>>2.0).
>> It would be great if that project made available under Apache and
>>driven by
>> the community. Adding to the above, not all Kafka users are interested
>>in
>> using the Java client API, they would like to have simple REST API where
>> they can code against using any language. I do believe this adds value
>>to
>> Apache Kafka in itself.
>>
>> "For 1, I don't think there is value in giving in to the NIH syndrome
>>and
>> reinventing the wheel. What I'm looking for is a detailed comparison of
>>the
>> gaps and why those can't be improved in the REST proxy that already
>>exists
>> and is actively maintained."
>>
>> We are not looking at this as NIH. What we are worried about is a
>>project
>> that's not maintained in a community. So the process of accepting
>>patches
>> and priorities is not clear, and it's not developed in Apache community.
>> Not only that, existing REST API project doesn't support new client API
>>and
>> hence there is no security support either.
>> We don't know the timeline when that's made available. We would like to
>>add
>> admin functionality into the REST API. So the Roadmap of that project is
>> not driven by Apache.
>>
>>
>> "This doesn't materially have an impact on expanding the usability of
>> Kafka. In my experience, REST proxy + Java clients only cover ~50% of
>> all
>> Kafka users, and maybe 10% of those are the ones who will use the
>>REST
>> proxy. The remaining 50% are non-java client users (C, python, go,
>>node
>> etc)."
>>
>> REST API is most often asked feature in my interactions with Kafka
>>users.
>> In an organization, There will be independent teams who will write their
>> Kafka clients using different language libraries available today, and
>> there is no way to standardize this. Instead of supporting several
>> different client libraries users will be interested in using a REST API
>> server. The need for a REST API will only increase as more and more
>>users
>> start using Kafka.
>>
>> "More surface area means more work to keep things consistent. Failure
>> to do that has, in fact, hurt the user experience."
>> Having myriad Kafka client GitHub projects that support different
>>languages
>> hurts the user experience and pushes burden to maintain these libraries.
>> REST API is a simple code base that uses existing java client libraries
>>to
>> make life easier for the users.
>>
>> Thanks,
>> Harsha
>>
>> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io> wrote:
>>
>> > Manikumar,
>> >
>> > Thanks for sharing the proposal. I think there are 2 parts to this
>> > discussion -
>> >
>> > 1. Should we rewrite a REST proxy given that there is a
>>feature-complete,
>> > open-source and actively maintained REST proxy in the community?
>> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
>> maintain
>> > the high-quality experience that Kafka users have today?
>> >
>> > For 1, I don't think there is value in giving in to the NIH syndrome
>>and
>> > reinventing the wheel. What I'm looking for is a detailed comparison
>>of
>> the
>> > gaps and why those can't be improved in the REST proxy that already
>> exists
>> > and is actively maintained. For example, we depend on zkClient and
>>have
>> > found as well as fixed several bugs by working closely with the people
>> who
>> > maintain zkClient. This should be possible for REST proxy as well,
>>right?
>> >
>> > For 2, I'd like us to review our history of expanding the surface
>>area to
>> > add more clients in the past. Here is a summary -
>> >
>> > - This doesn't materially have an impact on expanding the
>>usability of
>> > Kafka. In my experience, REST proxy + Java clients only cover ~50%
>>of
>> > all
>> > Kafka users, and maybe 10% of those are the ones who will use the
>>REST
>> > proxy. The remaining 50% are non-java client users (C, python, go,
>> node
>> > etc).
>> > - People are a lot more excited about promising to contribute while
>> > adding the surface area but not on an ongoing basis down the line.
>> > - More surface area means more work to keep things consistent.
>>Failure
>> > to do that has, in fact, hurt the user experience.
>> > - More surface area hurts agility. We want to do a few things
>>really
>> > well as well as be agile to be able to build on our core
>>competency.
>> >
>> >
>> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com>
>> > wrote:
>> >
>> > > Hi Jay,
>> > >
>> > > Thanks for your reply.
>> > >
>> > > I agree that we can not add all the clients/tools available in
>> ecosystem
>> > > page to Kafka repo itself. But we feel REST Interface is different
>>from
>> > > other clients/tools. Since any language that can work with HTTP can
>> > > easily integrate with this interface, Having an "official" REST
>> > > interface helps user community. This also helps us to integrate well
>> > > with external management and provisioning tools. Apache Kafka
>>release
>> > > with Java clients + REST interface is sufficient for most of the
>>user
>> > > deployments/requirements. This helps users to deal with less number
>> > > of distributions/builds.
>> > >
>> > > Thanks,
>> > > Manikumar
>> > >
>> > >
>> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
>> > >
>> > > > Hey guys,
>> > > >
>> > > > There's already a REST interface maintained as a separate
>> project--it's
>> > > > open source and apache licensed and actively maintained (
>> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
>> that?
>> > > You
>> > > > mentioned that there was some compatibility concern, but
>> compatibility
>> > > has
>> > > > to do with the consumer protocol guarantees not the repo the code
>>is
>> > in,
>> > > > right? Not sure that concern makes sense.
>> > > >
>> > > > We could argue for adding pretty much anything and everything in
>>the
>> > > > ecosystem page in Kafka itself but I'm not sure that would make
>>the
>> > > project
>> > > > more agile.
>> > > >
>> > > > -Jay
>> > > >
>> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
>> manikumar.reddy@gmail.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Hi Kafka Devs,
>> > > > >
>> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
>> > > > >
>> > > > > There are already open-source alternatives are available. But
>>we
>> > would
>> > > > > like to add REST server that
>> > > > > many users ask for under Apache Kafka repo. Many data Infra
>>tools
>> > comes
>> > > > up
>> > > > > with Rest Interface.
>> > > > > It is useful to have inbuilt Rest API support for Produce,
>>Consume
>> > > > messages
>> > > > > and admin interface for
>> > > > > integrating with external management and provisioning tools.This
>> will
>> > > > also
>> > > > > allow the maintenance of
>> > > > > REST server and adding new features makes it easy because apache
>> > > > community.
>> > > > >
>> > > > > The KIP wiki is the following:
>> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > 80%3A+Kafka+Rest+Server
>> > > > >
>> > > > > Your comments and feedback are welcome.
>> > > > >
>> > > > > Thanks,
>> > > > > Manikumar
>> > > > >
>> > > >
>> > >
>> > --
>> > Thanks,
>> > Neha
>> >
>>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Ofir Manor <of...@equalum.io>.
I personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka community, so there is a chance to
work this out.
However, there are two valid concerns here that could be addressed, around
community and accessibility:
>> What we are worried about is a project
>> that's not maintained in a community. So the process of accepting patches
>> and priorities is not clear, and it's not developed in Apache community.
>> Not only that, existing REST API project doesn't support new client API
and
>> hence there is no security support either.
This might be easy to fix. Maybe Confluent / kafka-rest community can
clarify that - what is their contribution policy, dev style, roadmap etc.
If they want, they can make an effort to encourage participation from
people outside Confluent (easily accept contributions, invite external
commiters or have open dev process similar to Apache Kafka etc), as there
is definitely seems to be some interest on the list. That might clear the
community concern and help kafka-rest project (but that is a calculation
Confluent will have to make).
The other, independent, concern is that REST is something that is expected
to be available out of the box with Kafka. I personally don't feel strongly
about it (better use proper, efficient APIs from day one), though it is
definitely way smaller than adding a stream processing engine to the
project :)
Again,the kafka-rest "community" could take steps to make it even easier to
install, configure and run kafka-rest for new users on vanilla Apache Kafka
(outside the Confluent platform), if they wish that (or welcome
contributions to that end), but that is up to them.
Finally, if after the above steps were taken there would still a strong
desire to include a great rest gateway with Apache Kafka, I assume the
community could hypothetically fork the existing kafka-rest into an Apache
Kafka subproject and maintain it "within Apache" instead of implementing it
from scratch (though I'm not a lawyer etc) - but I cannot imagine it happen
without Confluent blessing, and I think that is likely much less optimal
(pulling in other Confluent / Apache licensed dependencies) than having a
separate external community around kafka-rest.
Just my two cents,
Ofir Manor
Co-Founder & CTO | Equalum
Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io
On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <ka...@harsha.io>
wrote:
> Neha & Jay,
> We did look at the open source alternatives. Our concern
> is what's the patch acceptance and adding features/ bug-fixes to the
> existing project under a Github (although it's licensed under Apache 2.0).
> It would be great if that project made available under Apache and driven by
> the community. Adding to the above, not all Kafka users are interested in
> using the Java client API, they would like to have simple REST API where
> they can code against using any language. I do believe this adds value to
> Apache Kafka in itself.
>
> "For 1, I don't think there is value in giving in to the NIH syndrome and
> reinventing the wheel. What I'm looking for is a detailed comparison of the
> gaps and why those can't be improved in the REST proxy that already exists
> and is actively maintained."
>
> We are not looking at this as NIH. What we are worried about is a project
> that's not maintained in a community. So the process of accepting patches
> and priorities is not clear, and it's not developed in Apache community.
> Not only that, existing REST API project doesn't support new client API and
> hence there is no security support either.
> We don't know the timeline when that's made available. We would like to add
> admin functionality into the REST API. So the Roadmap of that project is
> not driven by Apache.
>
>
> "This doesn't materially have an impact on expanding the usability of
> Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
> Kafka users, and maybe 10% of those are the ones who will use the REST
> proxy. The remaining 50% are non-java client users (C, python, go, node
> etc)."
>
> REST API is most often asked feature in my interactions with Kafka users.
> In an organization, There will be independent teams who will write their
> Kafka clients using different language libraries available today, and
> there is no way to standardize this. Instead of supporting several
> different client libraries users will be interested in using a REST API
> server. The need for a REST API will only increase as more and more users
> start using Kafka.
>
> "More surface area means more work to keep things consistent. Failure
> to do that has, in fact, hurt the user experience."
> Having myriad Kafka client GitHub projects that support different languages
> hurts the user experience and pushes burden to maintain these libraries.
> REST API is a simple code base that uses existing java client libraries to
> make life easier for the users.
>
> Thanks,
> Harsha
>
> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io> wrote:
>
> > Manikumar,
> >
> > Thanks for sharing the proposal. I think there are 2 parts to this
> > discussion -
> >
> > 1. Should we rewrite a REST proxy given that there is a feature-complete,
> > open-source and actively maintained REST proxy in the community?
> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> maintain
> > the high-quality experience that Kafka users have today?
> >
> > For 1, I don't think there is value in giving in to the NIH syndrome and
> > reinventing the wheel. What I'm looking for is a detailed comparison of
> the
> > gaps and why those can't be improved in the REST proxy that already
> exists
> > and is actively maintained. For example, we depend on zkClient and have
> > found as well as fixed several bugs by working closely with the people
> who
> > maintain zkClient. This should be possible for REST proxy as well, right?
> >
> > For 2, I'd like us to review our history of expanding the surface area to
> > add more clients in the past. Here is a summary -
> >
> > - This doesn't materially have an impact on expanding the usability of
> > Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> > all
> > Kafka users, and maybe 10% of those are the ones who will use the REST
> > proxy. The remaining 50% are non-java client users (C, python, go,
> node
> > etc).
> > - People are a lot more excited about promising to contribute while
> > adding the surface area but not on an ongoing basis down the line.
> > - More surface area means more work to keep things consistent. Failure
> > to do that has, in fact, hurt the user experience.
> > - More surface area hurts agility. We want to do a few things really
> > well as well as be agile to be able to build on our core competency.
> >
> >
> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Thanks for your reply.
> > >
> > > I agree that we can not add all the clients/tools available in
> ecosystem
> > > page to Kafka repo itself. But we feel REST Interface is different from
> > > other clients/tools. Since any language that can work with HTTP can
> > > easily integrate with this interface, Having an "official" REST
> > > interface helps user community. This also helps us to integrate well
> > > with external management and provisioning tools. Apache Kafka release
> > > with Java clients + REST interface is sufficient for most of the user
> > > deployments/requirements. This helps users to deal with less number
> > > of distributions/builds.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > >
> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > Hey guys,
> > > >
> > > > There's already a REST interface maintained as a separate
> project--it's
> > > > open source and apache licensed and actively maintained (
> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
> that?
> > > You
> > > > mentioned that there was some compatibility concern, but
> compatibility
> > > has
> > > > to do with the consumer protocol guarantees not the repo the code is
> > in,
> > > > right? Not sure that concern makes sense.
> > > >
> > > > We could argue for adding pretty much anything and everything in the
> > > > ecosystem page in Kafka itself but I'm not sure that would make the
> > > project
> > > > more agile.
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> manikumar.reddy@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Kafka Devs,
> > > > >
> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > > >
> > > > > There are already open-source alternatives are available. But we
> > would
> > > > > like to add REST server that
> > > > > many users ask for under Apache Kafka repo. Many data Infra tools
> > comes
> > > > up
> > > > > with Rest Interface.
> > > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > > messages
> > > > > and admin interface for
> > > > > integrating with external management and provisioning tools.This
> will
> > > > also
> > > > > allow the maintenance of
> > > > > REST server and adding new features makes it easy because apache
> > > > community.
> > > > >
> > > > > The KIP wiki is the following:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 80%3A+Kafka+Rest+Server
> > > > >
> > > > > Your comments and feedback are welcome.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> > --
> > Thanks,
> > Neha
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Ben Davison <be...@7digital.com>.
I also think it would be good to have a REST client that can interact with
the new management API's coming down the pipe.
On Tue, Oct 4, 2016 at 10:35 PM, Edoardo Comar <EC...@uk.ibm.com> wrote:
> Harsha
> thanks for opening the discussion on this KIP.
>
> While I understand he founding members' stand that the Kafka project can
> not expand its surface to a large number of clients,
> I strongly agree with your well explained points below and support your
> KIP.
>
> A REST API is not just on the same level as any client, it's a basic
> building block of open web technologies.
> It's the API that most of our first time user want to try out (or that
> would be users ask for and expect to be there).
>
> A REST API for Kafka and a robust server implementation
> under the open governance of the Apache community would be most welcome.
>
> +1
>
> Edo
> --------------------------------------------------
> Edoardo Comar
> IBM MessageHub
>
> IBM United Kingdom Limited Registered in England and Wales with number
> 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
>
> Harsha Chintalapani <ka...@harsha.io> wrote on 02/10/2016 21:23:15:
>
> > From: Harsha Chintalapani <ka...@harsha.io>
> > To: dev@kafka.apache.org
> > Date: 02/10/2016 21:23
> > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >
> > Neha & Jay,
> > We did look at the open source alternatives. Our
> concern
> > is what's the patch acceptance and adding features/ bug-fixes to the
> > existing project under a Github (although it's licensed under Apache
> 2.0).
> > It would be great if that project made available under Apache and driven
> by
> > the community. Adding to the above, not all Kafka users are interested
> in
> > using the Java client API, they would like to have simple REST API where
> > they can code against using any language. I do believe this adds value
> to
> > Apache Kafka in itself.
> >
> > "For 1, I don't think there is value in giving in to the NIH syndrome
> and
> > reinventing the wheel. What I'm looking for is a detailed comparison of
> the
> > gaps and why those can't be improved in the REST proxy that already
> exists
> > and is actively maintained."
> >
> > We are not looking at this as NIH. What we are worried about is a
> project
> > that's not maintained in a community. So the process of accepting
> patches
> > and priorities is not clear, and it's not developed in Apache community.
> > Not only that, existing REST API project doesn't support new client API
> and
> > hence there is no security support either.
> > We don't know the timeline when that's made available. We would like to
> add
> > admin functionality into the REST API. So the Roadmap of that project is
> > not driven by Apache.
> >
> >
> > "This doesn't materially have an impact on expanding the usability of
> > Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
> > Kafka users, and maybe 10% of those are the ones who will use the
> REST
> > proxy. The remaining 50% are non-java client users (C, python, go,
> node
> > etc)."
> >
> > REST API is most often asked feature in my interactions with Kafka
> users.
> > In an organization, There will be independent teams who will write their
> > Kafka clients using different language libraries available today, and
> > there is no way to standardize this. Instead of supporting several
> > different client libraries users will be interested in using a REST API
> > server. The need for a REST API will only increase as more and more
> users
> > start using Kafka.
> >
> > "More surface area means more work to keep things consistent. Failure
> > to do that has, in fact, hurt the user experience."
> > Having myriad Kafka client GitHub projects that support different
> languages
> > hurts the user experience and pushes burden to maintain these libraries.
> > REST API is a simple code base that uses existing java client libraries
> to
> > make life easier for the users.
> >
> > Thanks,
> > Harsha
> >
> > On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io> wrote:
> >
> > > Manikumar,
> > >
> > > Thanks for sharing the proposal. I think there are 2 parts to this
> > > discussion -
> > >
> > > 1. Should we rewrite a REST proxy given that there is a
> feature-complete,
> > > open-source and actively maintained REST proxy in the community?
> > > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> maintain
> > > the high-quality experience that Kafka users have today?
> > >
> > > For 1, I don't think there is value in giving in to the NIH syndrome
> and
> > > reinventing the wheel. What I'm looking for is a detailed comparison
> of the
> > > gaps and why those can't be improved in the REST proxy that already
> exists
> > > and is actively maintained. For example, we depend on zkClient and
> have
> > > found as well as fixed several bugs by working closely with the people
> who
> > > maintain zkClient. This should be possible for REST proxy as well,
> right?
> > >
> > > For 2, I'd like us to review our history of expanding the surface area
> to
> > > add more clients in the past. Here is a summary -
> > >
> > > - This doesn't materially have an impact on expanding the usability
> of
> > > Kafka. In my experience, REST proxy + Java clients only cover ~50%
> of
> > > all
> > > Kafka users, and maybe 10% of those are the ones who will use the
> REST
> > > proxy. The remaining 50% are non-java client users (C, python, go,
> node
> > > etc).
> > > - People are a lot more excited about promising to contribute while
> > > adding the surface area but not on an ongoing basis down the line.
> > > - More surface area means more work to keep things consistent.
> Failure
> > > to do that has, in fact, hurt the user experience.
> > > - More surface area hurts agility. We want to do a few things
> really
> > > well as well as be agile to be able to build on our core
> competency.
> > >
> > >
> > > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi Jay,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > I agree that we can not add all the clients/tools available in
> ecosystem
> > > > page to Kafka repo itself. But we feel REST Interface is different
> from
> > > > other clients/tools. Since any language that can work with HTTP can
> > > > easily integrate with this interface, Having an "official" REST
> > > > interface helps user community. This also helps us to integrate well
> > > > with external management and provisioning tools. Apache Kafka
> release
> > > > with Java clients + REST interface is sufficient for most of the
> user
> > > > deployments/requirements. This helps users to deal with less number
> > > > of distributions/builds.
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > > >
> > > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> > > >
> > > > > Hey guys,
> > > > >
> > > > > There's already a REST interface maintained as a separate
> project--it's
> > > > > open source and apache licensed and actively maintained (
> > > > > https://github.com/confluentinc/kafka-rest). What is wrong with
> that?
> > > > You
> > > > > mentioned that there was some compatibility concern, but
> compatibility
> > > > has
> > > > > to do with the consumer protocol guarantees not the repo the code
> is
> > > in,
> > > > > right? Not sure that concern makes sense.
> > > > >
> > > > > We could argue for adding pretty much anything and everything in
> the
> > > > > ecosystem page in Kafka itself but I'm not sure that would make
> the
> > > > project
> > > > > more agile.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar
> <manikumar.reddy@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Kafka Devs,
> > > > > >
> > > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > > > >
> > > > > > There are already open-source alternatives are available. But
> we
> > > would
> > > > > > like to add REST server that
> > > > > > many users ask for under Apache Kafka repo. Many data Infra
> tools
> > > comes
> > > > > up
> > > > > > with Rest Interface.
> > > > > > It is useful to have inbuilt Rest API support for Produce,
> Consume
> > > > > messages
> > > > > > and admin interface for
> > > > > > integrating with external management and provisioning tools.This
> will
> > > > > also
> > > > > > allow the maintenance of
> > > > > > REST server and adding new features makes it easy because apache
> > > > > community.
> > > > > >
> > > > > > The KIP wiki is the following:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 80%3A+Kafka+Rest+Server
> > > > > >
> > > > > > Your comments and feedback are welcome.
> > > > > >
> > > > > > Thanks,
> > > > > > Manikumar
> > > > > >
> > > > >
> > > >
> > > --
> > > Thanks,
> > > Neha
> > >
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Edoardo Comar <EC...@uk.ibm.com>.
Harsha
thanks for opening the discussion on this KIP.
While I understand he founding members' stand that the Kafka project can
not expand its surface to a large number of clients,
I strongly agree with your well explained points below and support your
KIP.
A REST API is not just on the same level as any client, it's a basic
building block of open web technologies.
It's the API that most of our first time user want to try out (or that
would be users ask for and expect to be there).
A REST API for Kafka and a robust server implementation
under the open governance of the Apache community would be most welcome.
+1
Edo
--------------------------------------------------
Edoardo Comar
IBM MessageHub
IBM United Kingdom Limited Registered in England and Wales with number
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
3AU
Harsha Chintalapani <ka...@harsha.io> wrote on 02/10/2016 21:23:15:
> From: Harsha Chintalapani <ka...@harsha.io>
> To: dev@kafka.apache.org
> Date: 02/10/2016 21:23
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> Neha & Jay,
> We did look at the open source alternatives. Our
concern
> is what's the patch acceptance and adding features/ bug-fixes to the
> existing project under a Github (although it's licensed under Apache
2.0).
> It would be great if that project made available under Apache and driven
by
> the community. Adding to the above, not all Kafka users are interested
in
> using the Java client API, they would like to have simple REST API where
> they can code against using any language. I do believe this adds value
to
> Apache Kafka in itself.
>
> "For 1, I don't think there is value in giving in to the NIH syndrome
and
> reinventing the wheel. What I'm looking for is a detailed comparison of
the
> gaps and why those can't be improved in the REST proxy that already
exists
> and is actively maintained."
>
> We are not looking at this as NIH. What we are worried about is a
project
> that's not maintained in a community. So the process of accepting
patches
> and priorities is not clear, and it's not developed in Apache community.
> Not only that, existing REST API project doesn't support new client API
and
> hence there is no security support either.
> We don't know the timeline when that's made available. We would like to
add
> admin functionality into the REST API. So the Roadmap of that project is
> not driven by Apache.
>
>
> "This doesn't materially have an impact on expanding the usability of
> Kafka. In my experience, REST proxy + Java clients only cover ~50% of
all
> Kafka users, and maybe 10% of those are the ones who will use the
REST
> proxy. The remaining 50% are non-java client users (C, python, go,
node
> etc)."
>
> REST API is most often asked feature in my interactions with Kafka
users.
> In an organization, There will be independent teams who will write their
> Kafka clients using different language libraries available today, and
> there is no way to standardize this. Instead of supporting several
> different client libraries users will be interested in using a REST API
> server. The need for a REST API will only increase as more and more
users
> start using Kafka.
>
> "More surface area means more work to keep things consistent. Failure
> to do that has, in fact, hurt the user experience."
> Having myriad Kafka client GitHub projects that support different
languages
> hurts the user experience and pushes burden to maintain these libraries.
> REST API is a simple code base that uses existing java client libraries
to
> make life easier for the users.
>
> Thanks,
> Harsha
>
> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io> wrote:
>
> > Manikumar,
> >
> > Thanks for sharing the proposal. I think there are 2 parts to this
> > discussion -
> >
> > 1. Should we rewrite a REST proxy given that there is a
feature-complete,
> > open-source and actively maintained REST proxy in the community?
> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
maintain
> > the high-quality experience that Kafka users have today?
> >
> > For 1, I don't think there is value in giving in to the NIH syndrome
and
> > reinventing the wheel. What I'm looking for is a detailed comparison
of the
> > gaps and why those can't be improved in the REST proxy that already
exists
> > and is actively maintained. For example, we depend on zkClient and
have
> > found as well as fixed several bugs by working closely with the people
who
> > maintain zkClient. This should be possible for REST proxy as well,
right?
> >
> > For 2, I'd like us to review our history of expanding the surface area
to
> > add more clients in the past. Here is a summary -
> >
> > - This doesn't materially have an impact on expanding the usability
of
> > Kafka. In my experience, REST proxy + Java clients only cover ~50%
of
> > all
> > Kafka users, and maybe 10% of those are the ones who will use the
REST
> > proxy. The remaining 50% are non-java client users (C, python, go,
node
> > etc).
> > - People are a lot more excited about promising to contribute while
> > adding the surface area but not on an ongoing basis down the line.
> > - More surface area means more work to keep things consistent.
Failure
> > to do that has, in fact, hurt the user experience.
> > - More surface area hurts agility. We want to do a few things
really
> > well as well as be agile to be able to build on our core
competency.
> >
> >
> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Thanks for your reply.
> > >
> > > I agree that we can not add all the clients/tools available in
ecosystem
> > > page to Kafka repo itself. But we feel REST Interface is different
from
> > > other clients/tools. Since any language that can work with HTTP can
> > > easily integrate with this interface, Having an "official" REST
> > > interface helps user community. This also helps us to integrate well
> > > with external management and provisioning tools. Apache Kafka
release
> > > with Java clients + REST interface is sufficient for most of the
user
> > > deployments/requirements. This helps users to deal with less number
> > > of distributions/builds.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > >
> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > Hey guys,
> > > >
> > > > There's already a REST interface maintained as a separate
project--it's
> > > > open source and apache licensed and actively maintained (
> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
that?
> > > You
> > > > mentioned that there was some compatibility concern, but
compatibility
> > > has
> > > > to do with the consumer protocol guarantees not the repo the code
is
> > in,
> > > > right? Not sure that concern makes sense.
> > > >
> > > > We could argue for adding pretty much anything and everything in
the
> > > > ecosystem page in Kafka itself but I'm not sure that would make
the
> > > project
> > > > more agile.
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar
<manikumar.reddy@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Kafka Devs,
> > > > >
> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > > >
> > > > > There are already open-source alternatives are available. But
we
> > would
> > > > > like to add REST server that
> > > > > many users ask for under Apache Kafka repo. Many data Infra
tools
> > comes
> > > > up
> > > > > with Rest Interface.
> > > > > It is useful to have inbuilt Rest API support for Produce,
Consume
> > > > messages
> > > > > and admin interface for
> > > > > integrating with external management and provisioning tools.This
will
> > > > also
> > > > > allow the maintenance of
> > > > > REST server and adding new features makes it easy because apache
> > > > community.
> > > > >
> > > > > The KIP wiki is the following:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 80%3A+Kafka+Rest+Server
> > > > >
> > > > > Your comments and feedback are welcome.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> > --
> > Thanks,
> > Neha
> >
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Harsha Chintalapani <ka...@harsha.io>.
Neha & Jay,
We did look at the open source alternatives. Our concern
is what's the patch acceptance and adding features/ bug-fixes to the
existing project under a Github (although it's licensed under Apache 2.0).
It would be great if that project made available under Apache and driven by
the community. Adding to the above, not all Kafka users are interested in
using the Java client API, they would like to have simple REST API where
they can code against using any language. I do believe this adds value to
Apache Kafka in itself.
"For 1, I don't think there is value in giving in to the NIH syndrome and
reinventing the wheel. What I'm looking for is a detailed comparison of the
gaps and why those can't be improved in the REST proxy that already exists
and is actively maintained."
We are not looking at this as NIH. What we are worried about is a project
that's not maintained in a community. So the process of accepting patches
and priorities is not clear, and it's not developed in Apache community.
Not only that, existing REST API project doesn't support new client API and
hence there is no security support either.
We don't know the timeline when that's made available. We would like to add
admin functionality into the REST API. So the Roadmap of that project is
not driven by Apache.
"This doesn't materially have an impact on expanding the usability of
Kafka. In my experience, REST proxy + Java clients only cover ~50% of all
Kafka users, and maybe 10% of those are the ones who will use the REST
proxy. The remaining 50% are non-java client users (C, python, go, node
etc)."
REST API is most often asked feature in my interactions with Kafka users.
In an organization, There will be independent teams who will write their
Kafka clients using different language libraries available today, and
there is no way to standardize this. Instead of supporting several
different client libraries users will be interested in using a REST API
server. The need for a REST API will only increase as more and more users
start using Kafka.
"More surface area means more work to keep things consistent. Failure
to do that has, in fact, hurt the user experience."
Having myriad Kafka client GitHub projects that support different languages
hurts the user experience and pushes burden to maintain these libraries.
REST API is a simple code base that uses existing java client libraries to
make life easier for the users.
Thanks,
Harsha
On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <ne...@confluent.io> wrote:
> Manikumar,
>
> Thanks for sharing the proposal. I think there are 2 parts to this
> discussion -
>
> 1. Should we rewrite a REST proxy given that there is a feature-complete,
> open-source and actively maintained REST proxy in the community?
> 2. Does adding a REST proxy to Apache Kafka make us more agile and maintain
> the high-quality experience that Kafka users have today?
>
> For 1, I don't think there is value in giving in to the NIH syndrome and
> reinventing the wheel. What I'm looking for is a detailed comparison of the
> gaps and why those can't be improved in the REST proxy that already exists
> and is actively maintained. For example, we depend on zkClient and have
> found as well as fixed several bugs by working closely with the people who
> maintain zkClient. This should be possible for REST proxy as well, right?
>
> For 2, I'd like us to review our history of expanding the surface area to
> add more clients in the past. Here is a summary -
>
> - This doesn't materially have an impact on expanding the usability of
> Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
> Kafka users, and maybe 10% of those are the ones who will use the REST
> proxy. The remaining 50% are non-java client users (C, python, go, node
> etc).
> - People are a lot more excited about promising to contribute while
> adding the surface area but not on an ongoing basis down the line.
> - More surface area means more work to keep things consistent. Failure
> to do that has, in fact, hurt the user experience.
> - More surface area hurts agility. We want to do a few things really
> well as well as be agile to be able to build on our core competency.
>
>
> On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi Jay,
> >
> > Thanks for your reply.
> >
> > I agree that we can not add all the clients/tools available in ecosystem
> > page to Kafka repo itself. But we feel REST Interface is different from
> > other clients/tools. Since any language that can work with HTTP can
> > easily integrate with this interface, Having an "official" REST
> > interface helps user community. This also helps us to integrate well
> > with external management and provisioning tools. Apache Kafka release
> > with Java clients + REST interface is sufficient for most of the user
> > deployments/requirements. This helps users to deal with less number
> > of distributions/builds.
> >
> > Thanks,
> > Manikumar
> >
> >
> > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> >
> > > Hey guys,
> > >
> > > There's already a REST interface maintained as a separate project--it's
> > > open source and apache licensed and actively maintained (
> > > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> > You
> > > mentioned that there was some compatibility concern, but compatibility
> > has
> > > to do with the consumer protocol guarantees not the repo the code is
> in,
> > > right? Not sure that concern makes sense.
> > >
> > > We could argue for adding pretty much anything and everything in the
> > > ecosystem page in Kafka itself but I'm not sure that would make the
> > project
> > > more agile.
> > >
> > > -Jay
> > >
> > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <manikumar.reddy@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Kafka Devs,
> > > >
> > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > >
> > > > There are already open-source alternatives are available. But we
> would
> > > > like to add REST server that
> > > > many users ask for under Apache Kafka repo. Many data Infra tools
> comes
> > > up
> > > > with Rest Interface.
> > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > messages
> > > > and admin interface for
> > > > integrating with external management and provisioning tools.This will
> > > also
> > > > allow the maintenance of
> > > > REST server and adding new features makes it easy because apache
> > > community.
> > > >
> > > > The KIP wiki is the following:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 80%3A+Kafka+Rest+Server
> > > >
> > > > Your comments and feedback are welcome.
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > >
> >
> --
> Thanks,
> Neha
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Neha Narkhede <ne...@confluent.io>.
Manikumar,
Thanks for sharing the proposal. I think there are 2 parts to this
discussion -
1. Should we rewrite a REST proxy given that there is a feature-complete,
open-source and actively maintained REST proxy in the community?
2. Does adding a REST proxy to Apache Kafka make us more agile and maintain
the high-quality experience that Kafka users have today?
For 1, I don't think there is value in giving in to the NIH syndrome and
reinventing the wheel. What I'm looking for is a detailed comparison of the
gaps and why those can't be improved in the REST proxy that already exists
and is actively maintained. For example, we depend on zkClient and have
found as well as fixed several bugs by working closely with the people who
maintain zkClient. This should be possible for REST proxy as well, right?
For 2, I'd like us to review our history of expanding the surface area to
add more clients in the past. Here is a summary -
- This doesn't materially have an impact on expanding the usability of
Kafka. In my experience, REST proxy + Java clients only cover ~50% of all
Kafka users, and maybe 10% of those are the ones who will use the REST
proxy. The remaining 50% are non-java client users (C, python, go, node
etc).
- People are a lot more excited about promising to contribute while
adding the surface area but not on an ongoing basis down the line.
- More surface area means more work to keep things consistent. Failure
to do that has, in fact, hurt the user experience.
- More surface area hurts agility. We want to do a few things really
well as well as be agile to be able to build on our core competency.
On Sat, Oct 1, 2016 at 5:38 AM Manikumar <ma...@gmail.com> wrote:
> Hi Jay,
>
> Thanks for your reply.
>
> I agree that we can not add all the clients/tools available in ecosystem
> page to Kafka repo itself. But we feel REST Interface is different from
> other clients/tools. Since any language that can work with HTTP can
> easily integrate with this interface, Having an "official" REST
> interface helps user community. This also helps us to integrate well
> with external management and provisioning tools. Apache Kafka release
> with Java clients + REST interface is sufficient for most of the user
> deployments/requirements. This helps users to deal with less number
> of distributions/builds.
>
> Thanks,
> Manikumar
>
>
> On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > Hey guys,
> >
> > There's already a REST interface maintained as a separate project--it's
> > open source and apache licensed and actively maintained (
> > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> You
> > mentioned that there was some compatibility concern, but compatibility
> has
> > to do with the consumer protocol guarantees not the repo the code is in,
> > right? Not sure that concern makes sense.
> >
> > We could argue for adding pretty much anything and everything in the
> > ecosystem page in Kafka itself but I'm not sure that would make the
> project
> > more agile.
> >
> > -Jay
> >
> > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi Kafka Devs,
> > >
> > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > >
> > > There are already open-source alternatives are available. But we would
> > > like to add REST server that
> > > many users ask for under Apache Kafka repo. Many data Infra tools comes
> > up
> > > with Rest Interface.
> > > It is useful to have inbuilt Rest API support for Produce, Consume
> > messages
> > > and admin interface for
> > > integrating with external management and provisioning tools.This will
> > also
> > > allow the maintenance of
> > > REST server and adding new features makes it easy because apache
> > community.
> > >
> > > The KIP wiki is the following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 80%3A+Kafka+Rest+Server
> > >
> > > Your comments and feedback are welcome.
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>
--
Thanks,
Neha
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Manikumar <ma...@gmail.com>.
Hi Jay,
Thanks for your reply.
I agree that we can not add all the clients/tools available in ecosystem
page to Kafka repo itself. But we feel REST Interface is different from
other clients/tools. Since any language that can work with HTTP can
easily integrate with this interface, Having an "official" REST
interface helps user community. This also helps us to integrate well
with external management and provisioning tools. Apache Kafka release
with Java clients + REST interface is sufficient for most of the user
deployments/requirements. This helps users to deal with less number
of distributions/builds.
Thanks,
Manikumar
On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> Hey guys,
>
> There's already a REST interface maintained as a separate project--it's
> open source and apache licensed and actively maintained (
> https://github.com/confluentinc/kafka-rest). What is wrong with that? You
> mentioned that there was some compatibility concern, but compatibility has
> to do with the consumer protocol guarantees not the repo the code is in,
> right? Not sure that concern makes sense.
>
> We could argue for adding pretty much anything and everything in the
> ecosystem page in Kafka itself but I'm not sure that would make the project
> more agile.
>
> -Jay
>
> On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi Kafka Devs,
> >
> > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> >
> > There are already open-source alternatives are available. But we would
> > like to add REST server that
> > many users ask for under Apache Kafka repo. Many data Infra tools comes
> up
> > with Rest Interface.
> > It is useful to have inbuilt Rest API support for Produce, Consume
> messages
> > and admin interface for
> > integrating with external management and provisioning tools.This will
> also
> > allow the maintenance of
> > REST server and adding new features makes it easy because apache
> community.
> >
> > The KIP wiki is the following:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 80%3A+Kafka+Rest+Server
> >
> > Your comments and feedback are welcome.
> >
> > Thanks,
> > Manikumar
> >
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Jay Kreps <ja...@confluent.io>.
Hey guys,
There's already a REST interface maintained as a separate project--it's
open source and apache licensed and actively maintained (
https://github.com/confluentinc/kafka-rest). What is wrong with that? You
mentioned that there was some compatibility concern, but compatibility has
to do with the consumer protocol guarantees not the repo the code is in,
right? Not sure that concern makes sense.
We could argue for adding pretty much anything and everything in the
ecosystem page in Kafka itself but I'm not sure that would make the project
more agile.
-Jay
On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <ma...@gmail.com>
wrote:
> Hi Kafka Devs,
>
> I created KIP-80 to add Kafka REST Server to Kafka Repository.
>
> There are already open-source alternatives are available. But we would
> like to add REST server that
> many users ask for under Apache Kafka repo. Many data Infra tools comes up
> with Rest Interface.
> It is useful to have inbuilt Rest API support for Produce, Consume messages
> and admin interface for
> integrating with external management and provisioning tools.This will also
> allow the maintenance of
> REST server and adding new features makes it easy because apache community.
>
> The KIP wiki is the following:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 80%3A+Kafka+Rest+Server
>
> Your comments and feedback are welcome.
>
> Thanks,
> Manikumar
>
Re: [DISCUSS] KIP-80: Kafka REST Server
Posted by Harsha Chintalapani <ka...@harsha.io>.
Thanks Mani for the KIP. I'll go over it and add my thoughts on this thread.
On Wed, Sep 28, 2016 at 12:04 AM Manikumar <ma...@gmail.com>
wrote:
> Hi Kafka Devs,
>
> I created KIP-80 to add Kafka REST Server to Kafka Repository.
>
> There are already open-source alternatives are available. But we would
> like to add REST server that
> many users ask for under Apache Kafka repo. Many data Infra tools comes up
> with Rest Interface.
> It is useful to have inbuilt Rest API support for Produce, Consume messages
> and admin interface for
> integrating with external management and provisioning tools.This will also
> allow the maintenance of
> REST server and adding new features makes it easy because apache community.
>
> The KIP wiki is the following:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 80%3A+Kafka+Rest+Server
>
> Your comments and feedback are welcome.
>
> Thanks,
> Manikumar
>