REST API

[Yogita Bijani <yo...@gmail.com>]

Hi,
I am a newbie, I was just checking if there has been any updates on the REST proxy.
I have tried to come up with a draft of REST API and can post it for review and discussion if it is still an open topic.

Thanks
Yogita

Sent from my iPad

Re: REST API

[Jay Kreps <ja...@gmail.com>]

Makes sense.

-Jay


On Fri, Jul 19, 2013 at 5:25 PM, David Arthur <mu...@gmail.com> wrote:

> I think there would be many advantages to having a REST API in Kafka, as a
> core component. Not just for data services (producing/consuming messages),
> but also for admin. I could imagine a Kafka web console that showed various
> stats, allowed for maintenance operations, showed pretty viz of the
> cluster, etc.
>
> The REST prototype I put together using Jetty was really a simplistic POC.
> In reality, more modern web techniques like chunked transfer encoding and
> streaming should be used for high throughput (in, and out). Netty would
> probably be a good choice for a networking library.
>
> I'd be willing to bet that a sufficiently high performance and easy to use
> REST API could put a lot of the 3rd party client libraries out of business,
> so-to-speak.
>
> To more directly answer your question, Jay, I think if something like this
> is done it should be done as a top-level component in Kafka. Is contrib
> even maintained/updated anymore? Hasn't Camus supplanted the hadoop stuff?
>
>
> On 7/19/13 4:24 PM, Jay Kreps wrote:
>
>> I think this is really cool and would be useful for a lot of people.
>> David,
>> do you think it would really benefit from being a contrib package in kafka
>> versus a separately hosted github project?
>>
>> Here was our experience: contrib packages tend to get added but the
>> authors
>> and don't necessarily want involvement in kafka internals or necessarily
>> follow the mailing list to answer questions. When we take then they end up
>> kind of stagnating because there is overhead for the code review process,
>> and the main kafka committers aren't generally qualified to fully maintain
>> the code.
>>
>> What we have seen is that external projects that host add ons do really
>> well. The author tends to get more credit for their work, directly
>> responds
>> to questions from users, etc.
>>
>> On the other hand maybe we feel this is central enough that it should just
>> be a top-level package in kafka. I think this makes sense for things which
>> are closely tied to internals, are general enough that there is only one
>> good way to do it, and will be very common for kafka users to want. If
>> this
>> sounds better for the REST proxy the important thing is just that someone
>> has bandwidth to be able to support it and ensure we we document it, have
>> sufficient tests to keep it in sync, etc.
>>
>> Let me know what you think, if the preference is to add it to kafka then I
>> can help with code reviews and stuff.
>>
>> -Jay
>>
>>
>> On Thu, Jul 18, 2013 at 8:15 PM, David Arthur <mu...@gmail.com> wrote:
>>
>>  I did some work on this a while ago: https://github.com/mumrah/**
>>> kafka/commits/rest <https://github.com/mumrah/**kafka/commits/rest<https://github.com/mumrah/kafka/commits/rest>
>>> >
>>>
>>>
>>> I think people would still be interested. I know I would :)
>>>
>>> -David
>>>
>>>
>>>
>>> On 7/18/13 6:19 PM, Yogita Bijani wrote:
>>>
>>>  Hi,
>>>> I am a newbie, I was just checking if there has been any updates on the
>>>> REST proxy.
>>>> I have tried to come up with a draft of REST API and can post it for
>>>> review and discussion if it is still an open topic.
>>>>
>>>> Thanks
>>>> Yogita
>>>>
>>>> Sent from my iPad
>>>>
>>>>
>>>
>

Re: REST API

[David Arthur <mu...@gmail.com>]

I think there would be many advantages to having a REST API in Kafka, as 
a core component. Not just for data services (producing/consuming 
messages), but also for admin. I could imagine a Kafka web console that 
showed various stats, allowed for maintenance operations, showed pretty 
viz of the cluster, etc.

The REST prototype I put together using Jetty was really a simplistic 
POC. In reality, more modern web techniques like chunked transfer 
encoding and streaming should be used for high throughput (in, and out). 
Netty would probably be a good choice for a networking library.

I'd be willing to bet that a sufficiently high performance and easy to 
use REST API could put a lot of the 3rd party client libraries out of 
business, so-to-speak.

To more directly answer your question, Jay, I think if something like 
this is done it should be done as a top-level component in Kafka. Is 
contrib even maintained/updated anymore? Hasn't Camus supplanted the 
hadoop stuff?

On 7/19/13 4:24 PM, Jay Kreps wrote:
> I think this is really cool and would be useful for a lot of people. David,
> do you think it would really benefit from being a contrib package in kafka
> versus a separately hosted github project?
>
> Here was our experience: contrib packages tend to get added but the authors
> and don't necessarily want involvement in kafka internals or necessarily
> follow the mailing list to answer questions. When we take then they end up
> kind of stagnating because there is overhead for the code review process,
> and the main kafka committers aren't generally qualified to fully maintain
> the code.
>
> What we have seen is that external projects that host add ons do really
> well. The author tends to get more credit for their work, directly responds
> to questions from users, etc.
>
> On the other hand maybe we feel this is central enough that it should just
> be a top-level package in kafka. I think this makes sense for things which
> are closely tied to internals, are general enough that there is only one
> good way to do it, and will be very common for kafka users to want. If this
> sounds better for the REST proxy the important thing is just that someone
> has bandwidth to be able to support it and ensure we we document it, have
> sufficient tests to keep it in sync, etc.
>
> Let me know what you think, if the preference is to add it to kafka then I
> can help with code reviews and stuff.
>
> -Jay
>
>
> On Thu, Jul 18, 2013 at 8:15 PM, David Arthur <mu...@gmail.com> wrote:
>
>> I did some work on this a while ago: https://github.com/mumrah/**
>> kafka/commits/rest <https://github.com/mumrah/kafka/commits/rest>
>>
>> I think people would still be interested. I know I would :)
>>
>> -David
>>
>>
>>
>> On 7/18/13 6:19 PM, Yogita Bijani wrote:
>>
>>> Hi,
>>> I am a newbie, I was just checking if there has been any updates on the
>>> REST proxy.
>>> I have tried to come up with a draft of REST API and can post it for
>>> review and discussion if it is still an open topic.
>>>
>>> Thanks
>>> Yogita
>>>
>>> Sent from my iPad
>>>
>>

Re: REST API

[Jay Kreps <ja...@gmail.com>]

I think this is really cool and would be useful for a lot of people. David,
do you think it would really benefit from being a contrib package in kafka
versus a separately hosted github project?

Here was our experience: contrib packages tend to get added but the authors
and don't necessarily want involvement in kafka internals or necessarily
follow the mailing list to answer questions. When we take then they end up
kind of stagnating because there is overhead for the code review process,
and the main kafka committers aren't generally qualified to fully maintain
the code.

What we have seen is that external projects that host add ons do really
well. The author tends to get more credit for their work, directly responds
to questions from users, etc.

On the other hand maybe we feel this is central enough that it should just
be a top-level package in kafka. I think this makes sense for things which
are closely tied to internals, are general enough that there is only one
good way to do it, and will be very common for kafka users to want. If this
sounds better for the REST proxy the important thing is just that someone
has bandwidth to be able to support it and ensure we we document it, have
sufficient tests to keep it in sync, etc.

Let me know what you think, if the preference is to add it to kafka then I
can help with code reviews and stuff.

-Jay


On Thu, Jul 18, 2013 at 8:15 PM, David Arthur <mu...@gmail.com> wrote:

> I did some work on this a while ago: https://github.com/mumrah/**
> kafka/commits/rest <https://github.com/mumrah/kafka/commits/rest>
>
> I think people would still be interested. I know I would :)
>
> -David
>
>
>
> On 7/18/13 6:19 PM, Yogita Bijani wrote:
>
>> Hi,
>> I am a newbie, I was just checking if there has been any updates on the
>> REST proxy.
>> I have tried to come up with a draft of REST API and can post it for
>> review and discussion if it is still an open topic.
>>
>> Thanks
>> Yogita
>>
>> Sent from my iPad
>>
>
>

Re: REST API

[David Arthur <mu...@gmail.com>]

I did some work on this a while ago: 
https://github.com/mumrah/kafka/commits/rest

I think people would still be interested. I know I would :)

-David


On 7/18/13 6:19 PM, Yogita Bijani wrote:
> Hi,
> I am a newbie, I was just checking if there has been any updates on the REST proxy.
> I have tried to come up with a draft of REST API and can post it for review and discussion if it is still an open topic.
>
> Thanks
> Yogita
>
> Sent from my iPad