You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mark <st...@gmail.com> on 2011/12/08 17:23:29 UTC
Clients
Are consumer/producer clients typically written in Java or Scala? Are
they both full-featured?
Are there any reasons why the kafka-rb gem couldn't be full featured
supporting both queuing and pub-sub models?
Thanks
Re: Clients
Posted by David Ormsbee <da...@datadoghq.com>.
There is some ZooKeeper stuff towards the bottom of:
http://incubator.apache.org/kafka/design.html
It's a good overview of how it works, but it's not totally accurate or
complete. I've been looking through Scala client code and occasionally
bugging this mailing list. I'll try to organize my notes and add it to the
wiki sometime early next week.
Take care.
Dave
On Thu, Dec 8, 2011 at 2:12 PM, Mark <st...@gmail.com> wrote:
> David,
>
> Thanks, this is a great start. Is there any online documentation on how
> the Java/Scala consumers interact with Zookeeper? This would be useful so
> all consumers behavior the same way.
>
>
> On 12/8/11 10:46 AM, David Ormsbee wrote:
>
>> Hi Mark,
>>
>> If you're interested, I've started a wiki entry to help people who are
>> writing Kafka clients:
>>
>> https://cwiki.apache.org/**confluence/display/KAFKA/**
>> Writing+a+Driver+for+Kafka<https://cwiki.apache.org/confluence/display/KAFKA/Writing+a+Driver+for+Kafka>
>>
>> I'm currently working on one for Python. There's nothing on ZooKeeper in
>> there yet, but I'm implementing that functionality now and should be able
>> to add that information to the wiki soon.
>>
>> Thank you.
>>
>> Dave
>>
>> On Thu, Dec 8, 2011 at 12:12 PM, Mark<static.void.dev@gmail.com**>
>> wrote:
>>
>> Zookeeper integration is possible with ruby so there is nothing stopping
>>> someone from writing a full featured client then correct?
>>>
>>>
>>> On 12/8/11 8:57 AM, Jun Rao wrote:
>>>
>>> Mark,
>>>>
>>>> The java/scala clients are complete. The non-java support is kind of
>>>> weak
>>>> at this point. In particular, Kafka has a thick client (lots of
>>>> interaction
>>>> with ZK during rebalance and committing offsets) in the consumer. This
>>>> makes reimplementing the logic in each language hard, At this moment,
>>>> the
>>>> RB client only supports SimpleConsumer, not the high level consumer.
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>> On Thu, Dec 8, 2011 at 8:23 AM, Mark<static.void.dev@gmail.com****>
>>>> wrote:
>>>>
>>>> Are consumer/producer clients typically written in Java or Scala? Are
>>>>
>>>>> they
>>>>> both full-featured?
>>>>>
>>>>> Are there any reasons why the kafka-rb gem couldn't be full featured
>>>>> supporting both queuing and pub-sub models?
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
Re: Clients
Posted by Mark <st...@gmail.com>.
David,
Thanks, this is a great start. Is there any online documentation on how
the Java/Scala consumers interact with Zookeeper? This would be useful
so all consumers behavior the same way.
On 12/8/11 10:46 AM, David Ormsbee wrote:
> Hi Mark,
>
> If you're interested, I've started a wiki entry to help people who are
> writing Kafka clients:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Writing+a+Driver+for+Kafka
>
> I'm currently working on one for Python. There's nothing on ZooKeeper in
> there yet, but I'm implementing that functionality now and should be able
> to add that information to the wiki soon.
>
> Thank you.
>
> Dave
>
> On Thu, Dec 8, 2011 at 12:12 PM, Mark<st...@gmail.com> wrote:
>
>> Zookeeper integration is possible with ruby so there is nothing stopping
>> someone from writing a full featured client then correct?
>>
>>
>> On 12/8/11 8:57 AM, Jun Rao wrote:
>>
>>> Mark,
>>>
>>> The java/scala clients are complete. The non-java support is kind of weak
>>> at this point. In particular, Kafka has a thick client (lots of
>>> interaction
>>> with ZK during rebalance and committing offsets) in the consumer. This
>>> makes reimplementing the logic in each language hard, At this moment, the
>>> RB client only supports SimpleConsumer, not the high level consumer.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Thu, Dec 8, 2011 at 8:23 AM, Mark<static.void.dev@gmail.com**> wrote:
>>>
>>> Are consumer/producer clients typically written in Java or Scala? Are
>>>> they
>>>> both full-featured?
>>>>
>>>> Are there any reasons why the kafka-rb gem couldn't be full featured
>>>> supporting both queuing and pub-sub models?
>>>>
>>>> Thanks
>>>>
>>>>
Re: Clients
Posted by David Ormsbee <da...@datadoghq.com>.
Hi Mark,
If you're interested, I've started a wiki entry to help people who are
writing Kafka clients:
https://cwiki.apache.org/confluence/display/KAFKA/Writing+a+Driver+for+Kafka
I'm currently working on one for Python. There's nothing on ZooKeeper in
there yet, but I'm implementing that functionality now and should be able
to add that information to the wiki soon.
Thank you.
Dave
On Thu, Dec 8, 2011 at 12:12 PM, Mark <st...@gmail.com> wrote:
> Zookeeper integration is possible with ruby so there is nothing stopping
> someone from writing a full featured client then correct?
>
>
> On 12/8/11 8:57 AM, Jun Rao wrote:
>
>> Mark,
>>
>> The java/scala clients are complete. The non-java support is kind of weak
>> at this point. In particular, Kafka has a thick client (lots of
>> interaction
>> with ZK during rebalance and committing offsets) in the consumer. This
>> makes reimplementing the logic in each language hard, At this moment, the
>> RB client only supports SimpleConsumer, not the high level consumer.
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Dec 8, 2011 at 8:23 AM, Mark<static.void.dev@gmail.com**> wrote:
>>
>> Are consumer/producer clients typically written in Java or Scala? Are
>>> they
>>> both full-featured?
>>>
>>> Are there any reasons why the kafka-rb gem couldn't be full featured
>>> supporting both queuing and pub-sub models?
>>>
>>> Thanks
>>>
>>>
Re: Clients
Posted by Jun Rao <ju...@gmail.com>.
No, it's just that nobody has done that yet.
Jun
On Thu, Dec 8, 2011 at 9:12 AM, Mark <st...@gmail.com> wrote:
> Zookeeper integration is possible with ruby so there is nothing stopping
> someone from writing a full featured client then correct?
>
>
> On 12/8/11 8:57 AM, Jun Rao wrote:
>
>> Mark,
>>
>> The java/scala clients are complete. The non-java support is kind of weak
>> at this point. In particular, Kafka has a thick client (lots of
>> interaction
>> with ZK during rebalance and committing offsets) in the consumer. This
>> makes reimplementing the logic in each language hard, At this moment, the
>> RB client only supports SimpleConsumer, not the high level consumer.
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Dec 8, 2011 at 8:23 AM, Mark<static.void.dev@gmail.com**> wrote:
>>
>> Are consumer/producer clients typically written in Java or Scala? Are
>>> they
>>> both full-featured?
>>>
>>> Are there any reasons why the kafka-rb gem couldn't be full featured
>>> supporting both queuing and pub-sub models?
>>>
>>> Thanks
>>>
>>>
Re: Clients
Posted by Mark <st...@gmail.com>.
Zookeeper integration is possible with ruby so there is nothing stopping
someone from writing a full featured client then correct?
On 12/8/11 8:57 AM, Jun Rao wrote:
> Mark,
>
> The java/scala clients are complete. The non-java support is kind of weak
> at this point. In particular, Kafka has a thick client (lots of interaction
> with ZK during rebalance and committing offsets) in the consumer. This
> makes reimplementing the logic in each language hard, At this moment, the
> RB client only supports SimpleConsumer, not the high level consumer.
>
> Thanks,
>
> Jun
>
> On Thu, Dec 8, 2011 at 8:23 AM, Mark<st...@gmail.com> wrote:
>
>> Are consumer/producer clients typically written in Java or Scala? Are they
>> both full-featured?
>>
>> Are there any reasons why the kafka-rb gem couldn't be full featured
>> supporting both queuing and pub-sub models?
>>
>> Thanks
>>
Re: Clients
Posted by Jun Rao <ju...@gmail.com>.
Mark,
The java/scala clients are complete. The non-java support is kind of weak
at this point. In particular, Kafka has a thick client (lots of interaction
with ZK during rebalance and committing offsets) in the consumer. This
makes reimplementing the logic in each language hard, At this moment, the
RB client only supports SimpleConsumer, not the high level consumer.
Thanks,
Jun
On Thu, Dec 8, 2011 at 8:23 AM, Mark <st...@gmail.com> wrote:
> Are consumer/producer clients typically written in Java or Scala? Are they
> both full-featured?
>
> Are there any reasons why the kafka-rb gem couldn't be full featured
> supporting both queuing and pub-sub models?
>
> Thanks
>