You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Marc Labbe <mr...@gmail.com> on 2013/01/31 21:30:32 UTC
Consumer re-design and Python
Hi,
I am fairly new to Kafka and Scala, I am trying to see through the consumer
re-design changes, proposed and implemented for 0.8 and after, which will
affect other languages implementations. There are documentation pages on
the wiki, JIRA issues but I still can't figure out what's already there for
0.8, what will be there in the future and how it affects the consumers
written in other languages (Python in my case).
For instance, I am looking at
https://cwiki.apache.org/KAFKA/consumer-client-re-design.html and the very
well documented
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design
and
I am not sure what part is in the works, done and still a proposal. I feel
there are changes there already in 0.8 but not completely, referring
especially to KAFKA-364 and KAFKA-264.
Is this all accurate and up to date? There are talks of a coordinator as
well but from what I see, this hasn't been implemented so far.
After all, maybe my question is: other than the wire protocol changes, what
changes should I expect to do to SimpleConsumer client written in Python
for v0.8? What should I do next to implement a high level consumer
(ZookeeperConsumerConnector?) which fits with the design proposal?
Has anyone started making changes to their implementation yet (thinking
Brod or Samsa)? I'll post that question on github too.
Thanks and cheers!
marc
Re: Consumer re-design and Python
Posted by Marc Labbe <mr...@gmail.com>.
Thanks for the info David.
That was my understanding was well for what you pointed. Even though I
didn't mention it, I has also seen your kafka-python but I felt it was a
little too low level for my needs. Do you have a branch for 0.8 support on
github (didn't see any)?
cheers,
marc
On Mon, Feb 11, 2013 at 10:31 AM, David Arthur <mu...@gmail.com> wrote:
> On 1/31/13 3:30 PM, Marc Labbe wrote:
>
>> Hi,
>>
>> I am fairly new to Kafka and Scala, I am trying to see through the
>> consumer
>> re-design changes, proposed and implemented for 0.8 and after, which will
>> affect other languages implementations. There are documentation pages on
>> the wiki, JIRA issues but I still can't figure out what's already there
>> for
>> 0.8, what will be there in the future and how it affects the consumers
>> written in other languages (Python in my case).
>>
>> For instance, I am looking at
>> https://cwiki.apache.org/**KAFKA/consumer-client-re-**design.html<https://cwiki.apache.org/KAFKA/consumer-client-re-design.html>and the very
>> well documented
>> https://cwiki.apache.org/**confluence/display/KAFKA/**
>> Kafka+Detailed+Consumer+**Coordinator+Design<https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design>
>> and
>> I am not sure what part is in the works, done and still a proposal. I feel
>> there are changes there already in 0.8 but not completely, referring
>> especially to KAFKA-364 and KAFKA-264.
>>
> https://cwiki.apache.org/**confluence/display/KAFKA/**
> Consumer+Client+Re-Design<https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design>
>
> and
>
> https://cwiki.apache.org/**confluence/display/KAFKA/**
> Kafka+Detailed+Consumer+**Coordinator+Design<https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design>
>
> are the current design docs (as far as I know).
>
>
>> Is this all accurate and up to date? There are talks of a coordinator as
>> well but from what I see, this hasn't been implemented so far.
>>
> From my understanding, the client redesign has not been finalized and it
> still in-progress/todo.
>
>
>> After all, maybe my question is: other than the wire protocol changes,
>> what
>> changes should I expect to do to SimpleConsumer client written in Python
>> for v0.8? What should I do next to implement a high level consumer
>> (ZookeeperConsumerConnector?) which fits with the design proposal?
>>
> With 0.8, you will not need to connect to ZooKeeper from the clients. With
> KAFKA-657, offsets are centrally managed by the broker. Any broker can
> handle these requests.
>
>
>> Has anyone started making changes to their implementation yet (thinking
>> Brod or Samsa)? I'll post that question on github too.
>>
> I am working updating my Python client: https://github.com/mumrah/**
> kafka-python <https://github.com/mumrah/kafka-python>, still a ways to go
> yet. The biggest change (besides centralized offset management) is that
> each topic+partition is owned by a specific broker (the leader). When
> producing messages, you must send them to the correct leader. This requires
> that clients maintain some state of what belongs where which is a pain, but
> such is the cost of replication.
>
>>
>> Thanks and cheers!
>> marc
>>
>> -David
>
>
Re: Consumer re-design and Python
Posted by David Arthur <mu...@gmail.com>.
On 1/31/13 3:30 PM, Marc Labbe wrote:
> Hi,
>
> I am fairly new to Kafka and Scala, I am trying to see through the consumer
> re-design changes, proposed and implemented for 0.8 and after, which will
> affect other languages implementations. There are documentation pages on
> the wiki, JIRA issues but I still can't figure out what's already there for
> 0.8, what will be there in the future and how it affects the consumers
> written in other languages (Python in my case).
>
> For instance, I am looking at
> https://cwiki.apache.org/KAFKA/consumer-client-re-design.html and the very
> well documented
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design
> and
> I am not sure what part is in the works, done and still a proposal. I feel
> there are changes there already in 0.8 but not completely, referring
> especially to KAFKA-364 and KAFKA-264.
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
and
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design
are the current design docs (as far as I know).
>
> Is this all accurate and up to date? There are talks of a coordinator as
> well but from what I see, this hasn't been implemented so far.
From my understanding, the client redesign has not been finalized and
it still in-progress/todo.
>
> After all, maybe my question is: other than the wire protocol changes, what
> changes should I expect to do to SimpleConsumer client written in Python
> for v0.8? What should I do next to implement a high level consumer
> (ZookeeperConsumerConnector?) which fits with the design proposal?
With 0.8, you will not need to connect to ZooKeeper from the clients.
With KAFKA-657, offsets are centrally managed by the broker. Any broker
can handle these requests.
>
> Has anyone started making changes to their implementation yet (thinking
> Brod or Samsa)? I'll post that question on github too.
I am working updating my Python client:
https://github.com/mumrah/kafka-python, still a ways to go yet. The
biggest change (besides centralized offset management) is that each
topic+partition is owned by a specific broker (the leader). When
producing messages, you must send them to the correct leader. This
requires that clients maintain some state of what belongs where which is
a pain, but such is the cost of replication.
>
> Thanks and cheers!
> marc
>
-David