You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chris Curtin <cu...@gmail.com> on 2013/05/01 15:44:23 UTC
Re: Kafka wiki Documentation conventions - looking for feedback
I've tested my examples with the new (4/30) release and they work, so I've
updated the documentation.
Thanks,
Chris
On Mon, Apr 29, 2013 at 6:18 PM, Jun Rao <ju...@gmail.com> wrote:
> Thanks. I also updated your producer example to reflect a recent config
> change (broker.list => metadata.broker.list).
>
> Jun
>
>
> On Mon, Apr 29, 2013 at 11:19 AM, Chris Curtin <curtin.chris@gmail.com
> >wrote:
>
> > Thanks, I missed that the addition of consumers can cause a re-balance.
> > Thought it was only on Leader changes.
> >
> > I've updated the wording in the example.
> >
> > I'll pull down the beta and test my application then change the names on
> > the properties.
> >
> > Thanks,
> >
> > Chris
> >
> >
> > On Mon, Apr 29, 2013 at 11:58 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Basically, every time a consumer joins a group, every consumer in the
> > > groups gets a ZK notification and each of them tries to own a subset of
> > the
> > > total number of partitions. A given partition is only assigned to one
> of
> > > the consumers in the same group. Once the ownership is determined, each
> > > consumer consumes messages coming from its partitions and manages the
> > > offset of those partitions. Since at any given point of time, a
> partition
> > > is only owned by one consumer, there won't be conflicts on updating the
> > > offsets. More details are described in the "consumer rebalancing
> > algorithm"
> > > section of http://kafka.apache.org/07/design.html
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Apr 29, 2013 at 8:16 AM, Chris Curtin <curtin.chris@gmail.com
> > > >wrote:
> > >
> > > > Jun, can you explain this a little better? I thought when using
> > Consumer
> > > > Groups that on startup Kafka connects to ZooKeeper and finds the last
> > > read
> > > > offset for every partition in the topic being requested for the
> group.
> > > That
> > > > is then the starting point for the consumer threads.
> > > >
> > > > If a second process starts while the first one is running with the
> same
> > > > Consumer Group, won't the second one read the last offsets consumed
> by
> > > the
> > > > already running process and start processing from there? Then as the
> > > first
> > > > process syncs consumed offsets, won't the 2nd process's next update
> > > > overwrite them?
> > > >
> > > > Thanks,
> > > >
> > > > Chris
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Apr 29, 2013 at 11:03 AM, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > > > Chris,
> > > > >
> > > > > Thanks for the writeup. Looks great overall. A couple of comments.
> > > > >
> > > > > 1. At the beginning, it sounds like that one can't run multiple
> > > processes
> > > > > of consumers in the same group. This is actually not true. We can
> > > create
> > > > > multiple instances of consumers for the same group in the same JVM
> or
> > > > > different JVMs. The consumers will auto-balance among themselves.
> > > > >
> > > > > 2. We have changed the name of some config properties.
> > > > > auto.commit.interval.ms is correct. However, zk.connect,
> > > > > zk.session.timeout.ms and zk.sync.time.ms are changed to
> > > > > zookeeper.connect,
> > > > > zookeeper.session.timeout.ms, and zookeeper.sync.time.ms,
> > > respectively.
> > > > >
> > > > > I will add a link to your wiki in our website.
> > > > >
> > > > > Thanks again.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Apr 29, 2013 at 5:54 AM, Chris Curtin <
> > curtin.chris@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > I finished and published it this morning:
> > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > >
> > > > > > One question: when documenting the ConsumerConfig parameters I
> > > couldn't
> > > > > > find a description for the 'auto.commit.interval.ms' setting. I
> > > found
> > > > > one
> > > > > > for 'autocommit.interval.ms' (no '.' between auto and commit) in
> > the
> > > > > > Google
> > > > > > Cache only. Which spelling is it? Also is my description of it
> > > correct?
> > > > > >
> > > > > > I'll take a look at custom encoders later this week. Today and
> > > Tuesday
> > > > > are
> > > > > > going to be pretty busy.
> > > > > >
> > > > > > Please let me know if there are changes needed to the High Level
> > > > Consumer
> > > > > > page.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > >
> > > > > > On Mon, Apr 29, 2013 at 12:50 AM, Jun Rao <ju...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Chris,
> > > > > > >
> > > > > > > Any update of the high level consumer example?
> > > > > > >
> > > > > > > Also, in the Producer example, it would be useful to describe
> how
> > > to
> > > > > > write
> > > > > > > a customized encoder. One subtle thing is that the encoder
> needs
> > a
> > > > > > > constructor that takes a a single VerifiableProperties
> argument (
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-869).
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Kafka wiki Documentation conventions - looking for feedback
Posted by Jun Rao <ju...@gmail.com>.
The following are sample encoder/decoder in java.
class StringEncode implements Encoder<String> {
private String encoding = null;
public StringEncoder(VerifiableProperties props) {
if(props == null)
encoding = "UTF8";
else
encoding = props.getString("serializer.encoding", "UTF8");
}
public byte[] def toBytes(String s) {
if(s == null)
return null;
else
return s.getBytes(encoding);
}
}
class StringDecoder implements Decoder<String> {
private String encoding = null;
public StringDecoder(VerifiableProperties props) {
if(props == null)
encoding = "UTF8";
else
encoding = props.getString("serializer.encoding", "UTF8");
}
public String fromBytes(byte bytes[]) {
return new String(bytes, encoding);
}
}
Thanks,
Jun
On Wed, May 1, 2013 at 12:33 PM, Chris Curtin <cu...@gmail.com>wrote:
> Hi Jun
>
> I've added #1 and #2.
>
> I'll need to think about where to put #3, maybe even adding a 'tips and
> tricks' section?
>
> I've not had to do any encoder/decoders. Can anyone else offer some example
> code I can incorporate into an example?
>
> Thanks,
>
> Chris
>
>
> On Wed, May 1, 2013 at 11:45 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Chris,
> >
> > Thanks. This is very helpful. I linked your wiki pages to our website. A
> > few more comments:
> >
> > 1. Producer: The details of the meaning of request.required.acks are
> > described in http://kafka.apache.org/08/configuration.html. It would be
> > great if you can add a link to the description in your wiki.
> >
> > 2. High level consumer: Could you add the proper way of stopping the
> > consumer? One just need to call consumer.shutdown(). After this is
> called,
> > hasNext() call in the Kafka stream iterator will return false.
> >
> > 3. SimpleConsumer: We have the following api that returns the offset of
> the
> > last message exposed to the consumer. The difference btw high watermark
> and
> > the offset of the last consumed message tells you how many messages the
> > consumer is behind the broker.
> > highWatermark(topic: String, partition: Int)
> >
> > Finally, it would be great if you can extend the wiki with customized
> > encoder (Producer) and decoder (Consumer) at some point.
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, May 1, 2013 at 6:44 AM, Chris Curtin <cu...@gmail.com>
> > wrote:
> >
> > > I've tested my examples with the new (4/30) release and they work, so
> > I've
> > > updated the documentation.
> > >
> > > Thanks,
> > >
> > > Chris
> > >
> > >
> > > On Mon, Apr 29, 2013 at 6:18 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Thanks. I also updated your producer example to reflect a recent
> config
> > > > change (broker.list => metadata.broker.list).
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Apr 29, 2013 at 11:19 AM, Chris Curtin <
> curtin.chris@gmail.com
> > > > >wrote:
> > > >
> > > > > Thanks, I missed that the addition of consumers can cause a
> > re-balance.
> > > > > Thought it was only on Leader changes.
> > > > >
> > > > > I've updated the wording in the example.
> > > > >
> > > > > I'll pull down the beta and test my application then change the
> names
> > > on
> > > > > the properties.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Chris
> > > > >
> > > > >
> > > > > On Mon, Apr 29, 2013 at 11:58 AM, Jun Rao <ju...@gmail.com>
> wrote:
> > > > >
> > > > > > Basically, every time a consumer joins a group, every consumer in
> > the
> > > > > > groups gets a ZK notification and each of them tries to own a
> > subset
> > > of
> > > > > the
> > > > > > total number of partitions. A given partition is only assigned to
> > one
> > > > of
> > > > > > the consumers in the same group. Once the ownership is
> determined,
> > > each
> > > > > > consumer consumes messages coming from its partitions and manages
> > the
> > > > > > offset of those partitions. Since at any given point of time, a
> > > > partition
> > > > > > is only owned by one consumer, there won't be conflicts on
> updating
> > > the
> > > > > > offsets. More details are described in the "consumer rebalancing
> > > > > algorithm"
> > > > > > section of http://kafka.apache.org/07/design.html
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Mon, Apr 29, 2013 at 8:16 AM, Chris Curtin <
> > > curtin.chris@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Jun, can you explain this a little better? I thought when using
> > > > > Consumer
> > > > > > > Groups that on startup Kafka connects to ZooKeeper and finds
> the
> > > last
> > > > > > read
> > > > > > > offset for every partition in the topic being requested for the
> > > > group.
> > > > > > That
> > > > > > > is then the starting point for the consumer threads.
> > > > > > >
> > > > > > > If a second process starts while the first one is running with
> > the
> > > > same
> > > > > > > Consumer Group, won't the second one read the last offsets
> > consumed
> > > > by
> > > > > > the
> > > > > > > already running process and start processing from there? Then
> as
> > > the
> > > > > > first
> > > > > > > process syncs consumed offsets, won't the 2nd process's next
> > update
> > > > > > > overwrite them?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Apr 29, 2013 at 11:03 AM, Jun Rao <ju...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Chris,
> > > > > > > >
> > > > > > > > Thanks for the writeup. Looks great overall. A couple of
> > > comments.
> > > > > > > >
> > > > > > > > 1. At the beginning, it sounds like that one can't run
> multiple
> > > > > > processes
> > > > > > > > of consumers in the same group. This is actually not true. We
> > can
> > > > > > create
> > > > > > > > multiple instances of consumers for the same group in the
> same
> > > JVM
> > > > or
> > > > > > > > different JVMs. The consumers will auto-balance among
> > themselves.
> > > > > > > >
> > > > > > > > 2. We have changed the name of some config properties.
> > > > > > > > auto.commit.interval.ms is correct. However, zk.connect,
> > > > > > > > zk.session.timeout.ms and zk.sync.time.ms are changed to
> > > > > > > > zookeeper.connect,
> > > > > > > > zookeeper.session.timeout.ms, and zookeeper.sync.time.ms,
> > > > > > respectively.
> > > > > > > >
> > > > > > > > I will add a link to your wiki in our website.
> > > > > > > >
> > > > > > > > Thanks again.
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Apr 29, 2013 at 5:54 AM, Chris Curtin <
> > > > > curtin.chris@gmail.com
> > > > > > > > >wrote:
> > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > I finished and published it this morning:
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > > > > >
> > > > > > > > > One question: when documenting the ConsumerConfig
> parameters
> > I
> > > > > > couldn't
> > > > > > > > > find a description for the 'auto.commit.interval.ms'
> > setting.
> > > I
> > > > > > found
> > > > > > > > one
> > > > > > > > > for 'autocommit.interval.ms' (no '.' between auto and
> > commit)
> > > in
> > > > > the
> > > > > > > > > Google
> > > > > > > > > Cache only. Which spelling is it? Also is my description of
> > it
> > > > > > correct?
> > > > > > > > >
> > > > > > > > > I'll take a look at custom encoders later this week. Today
> > and
> > > > > > Tuesday
> > > > > > > > are
> > > > > > > > > going to be pretty busy.
> > > > > > > > >
> > > > > > > > > Please let me know if there are changes needed to the High
> > > Level
> > > > > > > Consumer
> > > > > > > > > page.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Apr 29, 2013 at 12:50 AM, Jun Rao <
> junrao@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Chris,
> > > > > > > > > >
> > > > > > > > > > Any update of the high level consumer example?
> > > > > > > > > >
> > > > > > > > > > Also, in the Producer example, it would be useful to
> > describe
> > > > how
> > > > > > to
> > > > > > > > > write
> > > > > > > > > > a customized encoder. One subtle thing is that the
> encoder
> > > > needs
> > > > > a
> > > > > > > > > > constructor that takes a a single VerifiableProperties
> > > > argument (
> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-869).
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Kafka wiki Documentation conventions - looking for feedback
Posted by Chris Curtin <cu...@gmail.com>.
Hi Jun
I've added #1 and #2.
I'll need to think about where to put #3, maybe even adding a 'tips and
tricks' section?
I've not had to do any encoder/decoders. Can anyone else offer some example
code I can incorporate into an example?
Thanks,
Chris
On Wed, May 1, 2013 at 11:45 AM, Jun Rao <ju...@gmail.com> wrote:
> Chris,
>
> Thanks. This is very helpful. I linked your wiki pages to our website. A
> few more comments:
>
> 1. Producer: The details of the meaning of request.required.acks are
> described in http://kafka.apache.org/08/configuration.html. It would be
> great if you can add a link to the description in your wiki.
>
> 2. High level consumer: Could you add the proper way of stopping the
> consumer? One just need to call consumer.shutdown(). After this is called,
> hasNext() call in the Kafka stream iterator will return false.
>
> 3. SimpleConsumer: We have the following api that returns the offset of the
> last message exposed to the consumer. The difference btw high watermark and
> the offset of the last consumed message tells you how many messages the
> consumer is behind the broker.
> highWatermark(topic: String, partition: Int)
>
> Finally, it would be great if you can extend the wiki with customized
> encoder (Producer) and decoder (Consumer) at some point.
> Thanks,
>
> Jun
>
>
> On Wed, May 1, 2013 at 6:44 AM, Chris Curtin <cu...@gmail.com>
> wrote:
>
> > I've tested my examples with the new (4/30) release and they work, so
> I've
> > updated the documentation.
> >
> > Thanks,
> >
> > Chris
> >
> >
> > On Mon, Apr 29, 2013 at 6:18 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Thanks. I also updated your producer example to reflect a recent config
> > > change (broker.list => metadata.broker.list).
> > >
> > > Jun
> > >
> > >
> > > On Mon, Apr 29, 2013 at 11:19 AM, Chris Curtin <curtin.chris@gmail.com
> > > >wrote:
> > >
> > > > Thanks, I missed that the addition of consumers can cause a
> re-balance.
> > > > Thought it was only on Leader changes.
> > > >
> > > > I've updated the wording in the example.
> > > >
> > > > I'll pull down the beta and test my application then change the names
> > on
> > > > the properties.
> > > >
> > > > Thanks,
> > > >
> > > > Chris
> > > >
> > > >
> > > > On Mon, Apr 29, 2013 at 11:58 AM, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > > > Basically, every time a consumer joins a group, every consumer in
> the
> > > > > groups gets a ZK notification and each of them tries to own a
> subset
> > of
> > > > the
> > > > > total number of partitions. A given partition is only assigned to
> one
> > > of
> > > > > the consumers in the same group. Once the ownership is determined,
> > each
> > > > > consumer consumes messages coming from its partitions and manages
> the
> > > > > offset of those partitions. Since at any given point of time, a
> > > partition
> > > > > is only owned by one consumer, there won't be conflicts on updating
> > the
> > > > > offsets. More details are described in the "consumer rebalancing
> > > > algorithm"
> > > > > section of http://kafka.apache.org/07/design.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Apr 29, 2013 at 8:16 AM, Chris Curtin <
> > curtin.chris@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Jun, can you explain this a little better? I thought when using
> > > > Consumer
> > > > > > Groups that on startup Kafka connects to ZooKeeper and finds the
> > last
> > > > > read
> > > > > > offset for every partition in the topic being requested for the
> > > group.
> > > > > That
> > > > > > is then the starting point for the consumer threads.
> > > > > >
> > > > > > If a second process starts while the first one is running with
> the
> > > same
> > > > > > Consumer Group, won't the second one read the last offsets
> consumed
> > > by
> > > > > the
> > > > > > already running process and start processing from there? Then as
> > the
> > > > > first
> > > > > > process syncs consumed offsets, won't the 2nd process's next
> update
> > > > > > overwrite them?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Apr 29, 2013 at 11:03 AM, Jun Rao <ju...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Chris,
> > > > > > >
> > > > > > > Thanks for the writeup. Looks great overall. A couple of
> > comments.
> > > > > > >
> > > > > > > 1. At the beginning, it sounds like that one can't run multiple
> > > > > processes
> > > > > > > of consumers in the same group. This is actually not true. We
> can
> > > > > create
> > > > > > > multiple instances of consumers for the same group in the same
> > JVM
> > > or
> > > > > > > different JVMs. The consumers will auto-balance among
> themselves.
> > > > > > >
> > > > > > > 2. We have changed the name of some config properties.
> > > > > > > auto.commit.interval.ms is correct. However, zk.connect,
> > > > > > > zk.session.timeout.ms and zk.sync.time.ms are changed to
> > > > > > > zookeeper.connect,
> > > > > > > zookeeper.session.timeout.ms, and zookeeper.sync.time.ms,
> > > > > respectively.
> > > > > > >
> > > > > > > I will add a link to your wiki in our website.
> > > > > > >
> > > > > > > Thanks again.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Apr 29, 2013 at 5:54 AM, Chris Curtin <
> > > > curtin.chris@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > I finished and published it this morning:
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > > > >
> > > > > > > > One question: when documenting the ConsumerConfig parameters
> I
> > > > > couldn't
> > > > > > > > find a description for the 'auto.commit.interval.ms'
> setting.
> > I
> > > > > found
> > > > > > > one
> > > > > > > > for 'autocommit.interval.ms' (no '.' between auto and
> commit)
> > in
> > > > the
> > > > > > > > Google
> > > > > > > > Cache only. Which spelling is it? Also is my description of
> it
> > > > > correct?
> > > > > > > >
> > > > > > > > I'll take a look at custom encoders later this week. Today
> and
> > > > > Tuesday
> > > > > > > are
> > > > > > > > going to be pretty busy.
> > > > > > > >
> > > > > > > > Please let me know if there are changes needed to the High
> > Level
> > > > > > Consumer
> > > > > > > > page.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Apr 29, 2013 at 12:50 AM, Jun Rao <ju...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Chris,
> > > > > > > > >
> > > > > > > > > Any update of the high level consumer example?
> > > > > > > > >
> > > > > > > > > Also, in the Producer example, it would be useful to
> describe
> > > how
> > > > > to
> > > > > > > > write
> > > > > > > > > a customized encoder. One subtle thing is that the encoder
> > > needs
> > > > a
> > > > > > > > > constructor that takes a a single VerifiableProperties
> > > argument (
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-869).
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Kafka wiki Documentation conventions - looking for feedback
Posted by Jun Rao <ju...@gmail.com>.
Chris,
Thanks. This is very helpful. I linked your wiki pages to our website. A
few more comments:
1. Producer: The details of the meaning of request.required.acks are
described in http://kafka.apache.org/08/configuration.html. It would be
great if you can add a link to the description in your wiki.
2. High level consumer: Could you add the proper way of stopping the
consumer? One just need to call consumer.shutdown(). After this is called,
hasNext() call in the Kafka stream iterator will return false.
3. SimpleConsumer: We have the following api that returns the offset of the
last message exposed to the consumer. The difference btw high watermark and
the offset of the last consumed message tells you how many messages the
consumer is behind the broker.
highWatermark(topic: String, partition: Int)
Finally, it would be great if you can extend the wiki with customized
encoder (Producer) and decoder (Consumer) at some point.
Thanks,
Jun
On Wed, May 1, 2013 at 6:44 AM, Chris Curtin <cu...@gmail.com> wrote:
> I've tested my examples with the new (4/30) release and they work, so I've
> updated the documentation.
>
> Thanks,
>
> Chris
>
>
> On Mon, Apr 29, 2013 at 6:18 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Thanks. I also updated your producer example to reflect a recent config
> > change (broker.list => metadata.broker.list).
> >
> > Jun
> >
> >
> > On Mon, Apr 29, 2013 at 11:19 AM, Chris Curtin <curtin.chris@gmail.com
> > >wrote:
> >
> > > Thanks, I missed that the addition of consumers can cause a re-balance.
> > > Thought it was only on Leader changes.
> > >
> > > I've updated the wording in the example.
> > >
> > > I'll pull down the beta and test my application then change the names
> on
> > > the properties.
> > >
> > > Thanks,
> > >
> > > Chris
> > >
> > >
> > > On Mon, Apr 29, 2013 at 11:58 AM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Basically, every time a consumer joins a group, every consumer in the
> > > > groups gets a ZK notification and each of them tries to own a subset
> of
> > > the
> > > > total number of partitions. A given partition is only assigned to one
> > of
> > > > the consumers in the same group. Once the ownership is determined,
> each
> > > > consumer consumes messages coming from its partitions and manages the
> > > > offset of those partitions. Since at any given point of time, a
> > partition
> > > > is only owned by one consumer, there won't be conflicts on updating
> the
> > > > offsets. More details are described in the "consumer rebalancing
> > > algorithm"
> > > > section of http://kafka.apache.org/07/design.html
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Apr 29, 2013 at 8:16 AM, Chris Curtin <
> curtin.chris@gmail.com
> > > > >wrote:
> > > >
> > > > > Jun, can you explain this a little better? I thought when using
> > > Consumer
> > > > > Groups that on startup Kafka connects to ZooKeeper and finds the
> last
> > > > read
> > > > > offset for every partition in the topic being requested for the
> > group.
> > > > That
> > > > > is then the starting point for the consumer threads.
> > > > >
> > > > > If a second process starts while the first one is running with the
> > same
> > > > > Consumer Group, won't the second one read the last offsets consumed
> > by
> > > > the
> > > > > already running process and start processing from there? Then as
> the
> > > > first
> > > > > process syncs consumed offsets, won't the 2nd process's next update
> > > > > overwrite them?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Chris
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Apr 29, 2013 at 11:03 AM, Jun Rao <ju...@gmail.com>
> wrote:
> > > > >
> > > > > > Chris,
> > > > > >
> > > > > > Thanks for the writeup. Looks great overall. A couple of
> comments.
> > > > > >
> > > > > > 1. At the beginning, it sounds like that one can't run multiple
> > > > processes
> > > > > > of consumers in the same group. This is actually not true. We can
> > > > create
> > > > > > multiple instances of consumers for the same group in the same
> JVM
> > or
> > > > > > different JVMs. The consumers will auto-balance among themselves.
> > > > > >
> > > > > > 2. We have changed the name of some config properties.
> > > > > > auto.commit.interval.ms is correct. However, zk.connect,
> > > > > > zk.session.timeout.ms and zk.sync.time.ms are changed to
> > > > > > zookeeper.connect,
> > > > > > zookeeper.session.timeout.ms, and zookeeper.sync.time.ms,
> > > > respectively.
> > > > > >
> > > > > > I will add a link to your wiki in our website.
> > > > > >
> > > > > > Thanks again.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Mon, Apr 29, 2013 at 5:54 AM, Chris Curtin <
> > > curtin.chris@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > I finished and published it this morning:
> > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > > >
> > > > > > > One question: when documenting the ConsumerConfig parameters I
> > > > couldn't
> > > > > > > find a description for the 'auto.commit.interval.ms' setting.
> I
> > > > found
> > > > > > one
> > > > > > > for 'autocommit.interval.ms' (no '.' between auto and commit)
> in
> > > the
> > > > > > > Google
> > > > > > > Cache only. Which spelling is it? Also is my description of it
> > > > correct?
> > > > > > >
> > > > > > > I'll take a look at custom encoders later this week. Today and
> > > > Tuesday
> > > > > > are
> > > > > > > going to be pretty busy.
> > > > > > >
> > > > > > > Please let me know if there are changes needed to the High
> Level
> > > > > Consumer
> > > > > > > page.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Apr 29, 2013 at 12:50 AM, Jun Rao <ju...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Chris,
> > > > > > > >
> > > > > > > > Any update of the high level consumer example?
> > > > > > > >
> > > > > > > > Also, in the Producer example, it would be useful to describe
> > how
> > > > to
> > > > > > > write
> > > > > > > > a customized encoder. One subtle thing is that the encoder
> > needs
> > > a
> > > > > > > > constructor that takes a a single VerifiableProperties
> > argument (
> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-869).
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>