You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Bae, Jae Hyeon" <me...@gmail.com> on 2012/11/19 07:35:34 UTC
Suggestion on ZkClient usage in Kafka
I want to suggest kafka should create only one instance of ZkClient
globally because ZkClient is thread safe and it will make many users
easily customize kafka source code for Zookeeper.
In our company's cloud environment, it is not recommended to create
ZkClient from zkConnect string directly because zookeeper cluster can
be dynamically changing. So, I have to create ZkClient using our
company's own platform library. Because of this requirement, I can't
use kafka jar file directly. I can modify and build kafka source code
but I have to repeat this work whenever I update kafka version, pretty
annoying.
So, my suggestion is, let me pass ZkClient outs of Producer, Consumer,
and Broker, as the following example.
Producer<String, String> producer =
ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
ConsumerConnector connector =
Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
ConsumerConfig(consumerProps));
KafkaServer is a little more complicated but I believe without much
effort we can refactor KafkaServer to be customized with ZkClient.
I really appreciate if this suggestion is accepted and merged to 0.8.
If you want me to contribute with this suggestion, please let me know
your opinion. If you are positive with this idea, I will contribute
very happily.
Thank you
Best, Jae
Re: Suggestion on ZkClient usage in Kafka
Posted by Jun Rao <ju...@gmail.com>.
You can try to put all brokers in a vip and expose the vip to the producer.
If there is no vip, it takes the same amount effort as moving a zk cluster
to a new set of hosts.
Thanks,
Jun
On Tue, Nov 20, 2012 at 10:20 AM, David Arthur <mu...@gmail.com> wrote:
> If I understand correctly, the brokers stay informed about one another
> through ZooKeeper and therefor any broker can give info about any other
> broker?
>
> This is an interesting approach. What would happen if your broker list
> changed dramatically over time?
>
> On Nov 20, 2012, at 1:02 PM, Neha Narkhede wrote:
>
> > This is being discussed in another thread -
> > http://markmail.org/message/mypnt7sgkqt55jb2?q=Jason+async+producer
> >
> > Basically, you want zookeeper on the producer to do just one thing -
> > notify the change in the liveness of brokers in Kafka
> > cluster. In 0.8, brokers are not the entity to worry about, what we
> > care about are replicas for the partitions that the producer
> > is sending data to, in particular just the leader replica (since only
> > the leader can accept writes for a partition)
> >
> > The producer keeps a cache of (topic, partition) -> leader-replica.
> > Now, if that cache is either empty or stale (due to changes
> > on the Kafka cluster), the next produce request will get an ACK with
> > an error code NotLeaderForPartition. That's when it
> > fires the getMetadata request that refreshes its cache. Assuming
> > you've configured your producer to rety (producer.num.retries)
> > more than once, it will succeed sending data the next time.
> >
> > In other words, instead of zookeeper 'notifying' us of the changes on
> > the Kafka cluster, we let the producer lazily update its
> > cache by invoking a special API on any of the Kafka brokers. That way,
> > we have much fewer connections to zk, zk upgrades
> > are easier, so are upgrades to the producer and we also achieve the
> > goal of replica discovery.
> >
> > Thanks,
> > Neha
> >
> > On Tue, Nov 20, 2012 at 9:58 AM, Bae, Jae Hyeon <me...@gmail.com>
> wrote:
> >> In the case that producer does not require zk.connect, how can the
> >> producer recognize the new brokers or brokers which went down?
> >>
> >> On Tue, Nov 20, 2012 at 8:31 AM, Jun Rao <ju...@gmail.com> wrote:
> >>> David,
> >>>
> >>> The change in 0.8 is that instead of requiring zk.connect, we require
> >>> broker.list. In both cases, you typically provide a list of hosts and
> >>> ports. Functionality wise, they achieve the same thing, ie, the
> producer is
> >>> able to send the data to the right broker. Are you saying that
> zk.connect
> >>> is more convenient? One benefit of using broker.list is that one can
> >>> provide a vip as the only host. This makes it easy to add/remove
> brokers
> >>> since no producer side config needs to be changed. Changing hosts in
> >>> zk.connect, on the other hand, requires config changes in the client.
> >>> Another reason for removing zkclient in the producer is that if the
> client
> >>> GCs, it can cause churns in the producer and extra load on the zk
> server.
> >>> Since our producer can be embedded in any client, it's hard for us to
> >>> control the GC rate. So, removing zkclient in the producer releases the
> >>> potential pressure from client GC.
> >>>
> >>> We still rely on ZK for failure detection and leader election on the
> broker
> >>> and the consumer though.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Tue, Nov 20, 2012 at 7:54 AM, David Arthur <mu...@gmail.com>
> wrote:
> >>>
> >>>>
> >>>> On Nov 20, 2012, at 12:23 AM, Jun Rao wrote:
> >>>>
> >>>>> Jason,
> >>>>>
> >>>>> In 0.8, producer doesn't use zkclient at all. You just need to set
> >>>>> broker.list.
> >>>>
> >>>> This seems like a regression in functionality. For me, one of the
> benefits
> >>>> of Kafka is only needing to know about ZooKeeper
> >>>>
> >>>>> A number of things have changed In 0.8. First, number of
> >>>>> partitions of a topic is global in a cluster and they don't really
> change
> >>>>> as new brokers are added. Second, a partition is assigned to multiple
> >>>>> brokers for replication and one of the replicas is the leader which
> >>>> serves
> >>>>> writes. When a producer starts up, it first uses the getMetadata api
> to
> >>>>> figure out the replica assignment for the relevant topic/partition.
> It
> >>>> then
> >>>>> issues producer request directly the broker where the leader
> resides. If
> >>>>> the leader broker goes down, the producer gets an exception and it
> will
> >>>>> re-issue the getMetadata api to obtain the information about the new
> >>>> leader.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <jb...@squareup.com>
> >>>> wrote:
> >>>>>
> >>>>>> Well, they do use zk though, to get the initial list of kafka
> nodes, and
> >>>>>> while zk is available, presumably they do use it to keep up with the
> >>>>>> dynamically changing set of kafka brokers, no? You are just saying
> >>>> that if
> >>>>>> zk goes away, 0.8 producers can keep on producing, as long as the
> kafka
> >>>>>> cluster remains stable?
> >>>>>>
> >>>>>> Jason
> >>>>>>
> >>>>>> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <
> >>>> neha.narkhede@gmail.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> In 0.8, producers don't use zk. When producers encounter an error
> >>>>>>> while sending data, they use a special getMetadata request to
> refresh
> >>>>>>> the kafka cluster info from a randomly selected Kafka broker, and
> >>>>>>> retry sending the data.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Neha
> >>>>>>>
> >>>>>>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <
> jbr@squareup.com>
> >>>>>>> wrote:
> >>>>>>>> Are you saying that in 0.8, producers don't use zkclient? Or
> don't
> >>>>>> need
> >>>>>>>> it? How can a producer dynamically respond to a change in the
> kafka
> >>>>>>>> cluster without zk?
> >>>>>>>>
> >>>>>>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>>> Jae,
> >>>>>>>>>
> >>>>>>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses
> a
> >>>> new
> >>>>>>>>> getMetadata api to get topic/partition/leader information from
> the
> >>>>>>> broker.
> >>>>>>>>> Consumers still need ZK client. We plan to redesign the consumer
> post
> >>>>>>> 0.8
> >>>>>>>>> and can keep this in mind.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Jun
> >>>>>>>>>
> >>>>>>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <
> metacret@gmail.com
> >>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I want to suggest kafka should create only one instance of
> ZkClient
> >>>>>>>>>> globally because ZkClient is thread safe and it will make many
> users
> >>>>>>>>>> easily customize kafka source code for Zookeeper.
> >>>>>>>>>>
> >>>>>>>>>> In our company's cloud environment, it is not recommended to
> create
> >>>>>>>>>> ZkClient from zkConnect string directly because zookeeper
> cluster
> >>>>>> can
> >>>>>>>>>> be dynamically changing. So, I have to create ZkClient using our
> >>>>>>>>>> company's own platform library. Because of this requirement, I
> can't
> >>>>>>>>>> use kafka jar file directly. I can modify and build kafka source
> >>>>>> code
> >>>>>>>>>> but I have to repeat this work whenever I update kafka version,
> >>>>>> pretty
> >>>>>>>>>> annoying.
> >>>>>>>>>>
> >>>>>>>>>> So, my suggestion is, let me pass ZkClient outs of Producer,
> >>>>>> Consumer,
> >>>>>>>>>> and Broker, as the following example.
> >>>>>>>>>>
> >>>>>>>>>> Producer<String, String> producer =
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
> >>>>>>>>>>
> >>>>>>>>>> ConsumerConnector connector =
> >>>>>>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
> >>>>>>>>>> ConsumerConfig(consumerProps));
> >>>>>>>>>>
> >>>>>>>>>> KafkaServer is a little more complicated but I believe without
> much
> >>>>>>>>>> effort we can refactor KafkaServer to be customized with
> ZkClient.
> >>>>>>>>>>
> >>>>>>>>>> I really appreciate if this suggestion is accepted and merged to
> >>>>>> 0.8.
> >>>>>>>>>> If you want me to contribute with this suggestion, please let me
> >>>>>> know
> >>>>>>>>>> your opinion. If you are positive with this idea, I will
> contribute
> >>>>>>>>>> very happily.
> >>>>>>>>>>
> >>>>>>>>>> Thank you
> >>>>>>>>>> Best, Jae
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
>
>
Re: Suggestion on ZkClient usage in Kafka
Posted by David Arthur <mu...@gmail.com>.
If I understand correctly, the brokers stay informed about one another through ZooKeeper and therefor any broker can give info about any other broker?
This is an interesting approach. What would happen if your broker list changed dramatically over time?
On Nov 20, 2012, at 1:02 PM, Neha Narkhede wrote:
> This is being discussed in another thread -
> http://markmail.org/message/mypnt7sgkqt55jb2?q=Jason+async+producer
>
> Basically, you want zookeeper on the producer to do just one thing -
> notify the change in the liveness of brokers in Kafka
> cluster. In 0.8, brokers are not the entity to worry about, what we
> care about are replicas for the partitions that the producer
> is sending data to, in particular just the leader replica (since only
> the leader can accept writes for a partition)
>
> The producer keeps a cache of (topic, partition) -> leader-replica.
> Now, if that cache is either empty or stale (due to changes
> on the Kafka cluster), the next produce request will get an ACK with
> an error code NotLeaderForPartition. That's when it
> fires the getMetadata request that refreshes its cache. Assuming
> you've configured your producer to rety (producer.num.retries)
> more than once, it will succeed sending data the next time.
>
> In other words, instead of zookeeper 'notifying' us of the changes on
> the Kafka cluster, we let the producer lazily update its
> cache by invoking a special API on any of the Kafka brokers. That way,
> we have much fewer connections to zk, zk upgrades
> are easier, so are upgrades to the producer and we also achieve the
> goal of replica discovery.
>
> Thanks,
> Neha
>
> On Tue, Nov 20, 2012 at 9:58 AM, Bae, Jae Hyeon <me...@gmail.com> wrote:
>> In the case that producer does not require zk.connect, how can the
>> producer recognize the new brokers or brokers which went down?
>>
>> On Tue, Nov 20, 2012 at 8:31 AM, Jun Rao <ju...@gmail.com> wrote:
>>> David,
>>>
>>> The change in 0.8 is that instead of requiring zk.connect, we require
>>> broker.list. In both cases, you typically provide a list of hosts and
>>> ports. Functionality wise, they achieve the same thing, ie, the producer is
>>> able to send the data to the right broker. Are you saying that zk.connect
>>> is more convenient? One benefit of using broker.list is that one can
>>> provide a vip as the only host. This makes it easy to add/remove brokers
>>> since no producer side config needs to be changed. Changing hosts in
>>> zk.connect, on the other hand, requires config changes in the client.
>>> Another reason for removing zkclient in the producer is that if the client
>>> GCs, it can cause churns in the producer and extra load on the zk server.
>>> Since our producer can be embedded in any client, it's hard for us to
>>> control the GC rate. So, removing zkclient in the producer releases the
>>> potential pressure from client GC.
>>>
>>> We still rely on ZK for failure detection and leader election on the broker
>>> and the consumer though.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Tue, Nov 20, 2012 at 7:54 AM, David Arthur <mu...@gmail.com> wrote:
>>>
>>>>
>>>> On Nov 20, 2012, at 12:23 AM, Jun Rao wrote:
>>>>
>>>>> Jason,
>>>>>
>>>>> In 0.8, producer doesn't use zkclient at all. You just need to set
>>>>> broker.list.
>>>>
>>>> This seems like a regression in functionality. For me, one of the benefits
>>>> of Kafka is only needing to know about ZooKeeper
>>>>
>>>>> A number of things have changed In 0.8. First, number of
>>>>> partitions of a topic is global in a cluster and they don't really change
>>>>> as new brokers are added. Second, a partition is assigned to multiple
>>>>> brokers for replication and one of the replicas is the leader which
>>>> serves
>>>>> writes. When a producer starts up, it first uses the getMetadata api to
>>>>> figure out the replica assignment for the relevant topic/partition. It
>>>> then
>>>>> issues producer request directly the broker where the leader resides. If
>>>>> the leader broker goes down, the producer gets an exception and it will
>>>>> re-issue the getMetadata api to obtain the information about the new
>>>> leader.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <jb...@squareup.com>
>>>> wrote:
>>>>>
>>>>>> Well, they do use zk though, to get the initial list of kafka nodes, and
>>>>>> while zk is available, presumably they do use it to keep up with the
>>>>>> dynamically changing set of kafka brokers, no? You are just saying
>>>> that if
>>>>>> zk goes away, 0.8 producers can keep on producing, as long as the kafka
>>>>>> cluster remains stable?
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <
>>>> neha.narkhede@gmail.com
>>>>>>> wrote:
>>>>>>
>>>>>>> In 0.8, producers don't use zk. When producers encounter an error
>>>>>>> while sending data, they use a special getMetadata request to refresh
>>>>>>> the kafka cluster info from a randomly selected Kafka broker, and
>>>>>>> retry sending the data.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Neha
>>>>>>>
>>>>>>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com>
>>>>>>> wrote:
>>>>>>>> Are you saying that in 0.8, producers don't use zkclient? Or don't
>>>>>> need
>>>>>>>> it? How can a producer dynamically respond to a change in the kafka
>>>>>>>> cluster without zk?
>>>>>>>>
>>>>>>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Jae,
>>>>>>>>>
>>>>>>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses a
>>>> new
>>>>>>>>> getMetadata api to get topic/partition/leader information from the
>>>>>>> broker.
>>>>>>>>> Consumers still need ZK client. We plan to redesign the consumer post
>>>>>>> 0.8
>>>>>>>>> and can keep this in mind.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <metacret@gmail.com
>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I want to suggest kafka should create only one instance of ZkClient
>>>>>>>>>> globally because ZkClient is thread safe and it will make many users
>>>>>>>>>> easily customize kafka source code for Zookeeper.
>>>>>>>>>>
>>>>>>>>>> In our company's cloud environment, it is not recommended to create
>>>>>>>>>> ZkClient from zkConnect string directly because zookeeper cluster
>>>>>> can
>>>>>>>>>> be dynamically changing. So, I have to create ZkClient using our
>>>>>>>>>> company's own platform library. Because of this requirement, I can't
>>>>>>>>>> use kafka jar file directly. I can modify and build kafka source
>>>>>> code
>>>>>>>>>> but I have to repeat this work whenever I update kafka version,
>>>>>> pretty
>>>>>>>>>> annoying.
>>>>>>>>>>
>>>>>>>>>> So, my suggestion is, let me pass ZkClient outs of Producer,
>>>>>> Consumer,
>>>>>>>>>> and Broker, as the following example.
>>>>>>>>>>
>>>>>>>>>> Producer<String, String> producer =
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
>>>>>>>>>>
>>>>>>>>>> ConsumerConnector connector =
>>>>>>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
>>>>>>>>>> ConsumerConfig(consumerProps));
>>>>>>>>>>
>>>>>>>>>> KafkaServer is a little more complicated but I believe without much
>>>>>>>>>> effort we can refactor KafkaServer to be customized with ZkClient.
>>>>>>>>>>
>>>>>>>>>> I really appreciate if this suggestion is accepted and merged to
>>>>>> 0.8.
>>>>>>>>>> If you want me to contribute with this suggestion, please let me
>>>>>> know
>>>>>>>>>> your opinion. If you are positive with this idea, I will contribute
>>>>>>>>>> very happily.
>>>>>>>>>>
>>>>>>>>>> Thank you
>>>>>>>>>> Best, Jae
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
Re: Suggestion on ZkClient usage in Kafka
Posted by Neha Narkhede <ne...@gmail.com>.
This is being discussed in another thread -
http://markmail.org/message/mypnt7sgkqt55jb2?q=Jason+async+producer
Basically, you want zookeeper on the producer to do just one thing -
notify the change in the liveness of brokers in Kafka
cluster. In 0.8, brokers are not the entity to worry about, what we
care about are replicas for the partitions that the producer
is sending data to, in particular just the leader replica (since only
the leader can accept writes for a partition)
The producer keeps a cache of (topic, partition) -> leader-replica.
Now, if that cache is either empty or stale (due to changes
on the Kafka cluster), the next produce request will get an ACK with
an error code NotLeaderForPartition. That's when it
fires the getMetadata request that refreshes its cache. Assuming
you've configured your producer to rety (producer.num.retries)
more than once, it will succeed sending data the next time.
In other words, instead of zookeeper 'notifying' us of the changes on
the Kafka cluster, we let the producer lazily update its
cache by invoking a special API on any of the Kafka brokers. That way,
we have much fewer connections to zk, zk upgrades
are easier, so are upgrades to the producer and we also achieve the
goal of replica discovery.
Thanks,
Neha
On Tue, Nov 20, 2012 at 9:58 AM, Bae, Jae Hyeon <me...@gmail.com> wrote:
> In the case that producer does not require zk.connect, how can the
> producer recognize the new brokers or brokers which went down?
>
> On Tue, Nov 20, 2012 at 8:31 AM, Jun Rao <ju...@gmail.com> wrote:
>> David,
>>
>> The change in 0.8 is that instead of requiring zk.connect, we require
>> broker.list. In both cases, you typically provide a list of hosts and
>> ports. Functionality wise, they achieve the same thing, ie, the producer is
>> able to send the data to the right broker. Are you saying that zk.connect
>> is more convenient? One benefit of using broker.list is that one can
>> provide a vip as the only host. This makes it easy to add/remove brokers
>> since no producer side config needs to be changed. Changing hosts in
>> zk.connect, on the other hand, requires config changes in the client.
>> Another reason for removing zkclient in the producer is that if the client
>> GCs, it can cause churns in the producer and extra load on the zk server.
>> Since our producer can be embedded in any client, it's hard for us to
>> control the GC rate. So, removing zkclient in the producer releases the
>> potential pressure from client GC.
>>
>> We still rely on ZK for failure detection and leader election on the broker
>> and the consumer though.
>>
>> Thanks,
>>
>> Jun
>>
>> On Tue, Nov 20, 2012 at 7:54 AM, David Arthur <mu...@gmail.com> wrote:
>>
>>>
>>> On Nov 20, 2012, at 12:23 AM, Jun Rao wrote:
>>>
>>> > Jason,
>>> >
>>> > In 0.8, producer doesn't use zkclient at all. You just need to set
>>> > broker.list.
>>>
>>> This seems like a regression in functionality. For me, one of the benefits
>>> of Kafka is only needing to know about ZooKeeper
>>>
>>> > A number of things have changed In 0.8. First, number of
>>> > partitions of a topic is global in a cluster and they don't really change
>>> > as new brokers are added. Second, a partition is assigned to multiple
>>> > brokers for replication and one of the replicas is the leader which
>>> serves
>>> > writes. When a producer starts up, it first uses the getMetadata api to
>>> > figure out the replica assignment for the relevant topic/partition. It
>>> then
>>> > issues producer request directly the broker where the leader resides. If
>>> > the leader broker goes down, the producer gets an exception and it will
>>> > re-issue the getMetadata api to obtain the information about the new
>>> leader.
>>> >
>>> > Thanks,
>>> >
>>> > Jun
>>> >
>>> > On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <jb...@squareup.com>
>>> wrote:
>>> >
>>> >> Well, they do use zk though, to get the initial list of kafka nodes, and
>>> >> while zk is available, presumably they do use it to keep up with the
>>> >> dynamically changing set of kafka brokers, no? You are just saying
>>> that if
>>> >> zk goes away, 0.8 producers can keep on producing, as long as the kafka
>>> >> cluster remains stable?
>>> >>
>>> >> Jason
>>> >>
>>> >> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <
>>> neha.narkhede@gmail.com
>>> >>> wrote:
>>> >>
>>> >>> In 0.8, producers don't use zk. When producers encounter an error
>>> >>> while sending data, they use a special getMetadata request to refresh
>>> >>> the kafka cluster info from a randomly selected Kafka broker, and
>>> >>> retry sending the data.
>>> >>>
>>> >>> Thanks,
>>> >>> Neha
>>> >>>
>>> >>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com>
>>> >>> wrote:
>>> >>>> Are you saying that in 0.8, producers don't use zkclient? Or don't
>>> >> need
>>> >>>> it? How can a producer dynamically respond to a change in the kafka
>>> >>>> cluster without zk?
>>> >>>>
>>> >>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
>>> >>>>
>>> >>>>> Jae,
>>> >>>>>
>>> >>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses a
>>> new
>>> >>>>> getMetadata api to get topic/partition/leader information from the
>>> >>> broker.
>>> >>>>> Consumers still need ZK client. We plan to redesign the consumer post
>>> >>> 0.8
>>> >>>>> and can keep this in mind.
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>>
>>> >>>>> Jun
>>> >>>>>
>>> >>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <metacret@gmail.com
>>> >
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> I want to suggest kafka should create only one instance of ZkClient
>>> >>>>>> globally because ZkClient is thread safe and it will make many users
>>> >>>>>> easily customize kafka source code for Zookeeper.
>>> >>>>>>
>>> >>>>>> In our company's cloud environment, it is not recommended to create
>>> >>>>>> ZkClient from zkConnect string directly because zookeeper cluster
>>> >> can
>>> >>>>>> be dynamically changing. So, I have to create ZkClient using our
>>> >>>>>> company's own platform library. Because of this requirement, I can't
>>> >>>>>> use kafka jar file directly. I can modify and build kafka source
>>> >> code
>>> >>>>>> but I have to repeat this work whenever I update kafka version,
>>> >> pretty
>>> >>>>>> annoying.
>>> >>>>>>
>>> >>>>>> So, my suggestion is, let me pass ZkClient outs of Producer,
>>> >> Consumer,
>>> >>>>>> and Broker, as the following example.
>>> >>>>>>
>>> >>>>>> Producer<String, String> producer =
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>
>>> >>
>>> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
>>> >>>>>>
>>> >>>>>> ConsumerConnector connector =
>>> >>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
>>> >>>>>> ConsumerConfig(consumerProps));
>>> >>>>>>
>>> >>>>>> KafkaServer is a little more complicated but I believe without much
>>> >>>>>> effort we can refactor KafkaServer to be customized with ZkClient.
>>> >>>>>>
>>> >>>>>> I really appreciate if this suggestion is accepted and merged to
>>> >> 0.8.
>>> >>>>>> If you want me to contribute with this suggestion, please let me
>>> >> know
>>> >>>>>> your opinion. If you are positive with this idea, I will contribute
>>> >>>>>> very happily.
>>> >>>>>>
>>> >>>>>> Thank you
>>> >>>>>> Best, Jae
>>> >>>>>>
>>> >>>>>
>>> >>>
>>> >>
>>>
>>>
Re: Suggestion on ZkClient usage in Kafka
Posted by "Bae, Jae Hyeon" <me...@gmail.com>.
In the case that producer does not require zk.connect, how can the
producer recognize the new brokers or brokers which went down?
On Tue, Nov 20, 2012 at 8:31 AM, Jun Rao <ju...@gmail.com> wrote:
> David,
>
> The change in 0.8 is that instead of requiring zk.connect, we require
> broker.list. In both cases, you typically provide a list of hosts and
> ports. Functionality wise, they achieve the same thing, ie, the producer is
> able to send the data to the right broker. Are you saying that zk.connect
> is more convenient? One benefit of using broker.list is that one can
> provide a vip as the only host. This makes it easy to add/remove brokers
> since no producer side config needs to be changed. Changing hosts in
> zk.connect, on the other hand, requires config changes in the client.
> Another reason for removing zkclient in the producer is that if the client
> GCs, it can cause churns in the producer and extra load on the zk server.
> Since our producer can be embedded in any client, it's hard for us to
> control the GC rate. So, removing zkclient in the producer releases the
> potential pressure from client GC.
>
> We still rely on ZK for failure detection and leader election on the broker
> and the consumer though.
>
> Thanks,
>
> Jun
>
> On Tue, Nov 20, 2012 at 7:54 AM, David Arthur <mu...@gmail.com> wrote:
>
>>
>> On Nov 20, 2012, at 12:23 AM, Jun Rao wrote:
>>
>> > Jason,
>> >
>> > In 0.8, producer doesn't use zkclient at all. You just need to set
>> > broker.list.
>>
>> This seems like a regression in functionality. For me, one of the benefits
>> of Kafka is only needing to know about ZooKeeper
>>
>> > A number of things have changed In 0.8. First, number of
>> > partitions of a topic is global in a cluster and they don't really change
>> > as new brokers are added. Second, a partition is assigned to multiple
>> > brokers for replication and one of the replicas is the leader which
>> serves
>> > writes. When a producer starts up, it first uses the getMetadata api to
>> > figure out the replica assignment for the relevant topic/partition. It
>> then
>> > issues producer request directly the broker where the leader resides. If
>> > the leader broker goes down, the producer gets an exception and it will
>> > re-issue the getMetadata api to obtain the information about the new
>> leader.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <jb...@squareup.com>
>> wrote:
>> >
>> >> Well, they do use zk though, to get the initial list of kafka nodes, and
>> >> while zk is available, presumably they do use it to keep up with the
>> >> dynamically changing set of kafka brokers, no? You are just saying
>> that if
>> >> zk goes away, 0.8 producers can keep on producing, as long as the kafka
>> >> cluster remains stable?
>> >>
>> >> Jason
>> >>
>> >> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <
>> neha.narkhede@gmail.com
>> >>> wrote:
>> >>
>> >>> In 0.8, producers don't use zk. When producers encounter an error
>> >>> while sending data, they use a special getMetadata request to refresh
>> >>> the kafka cluster info from a randomly selected Kafka broker, and
>> >>> retry sending the data.
>> >>>
>> >>> Thanks,
>> >>> Neha
>> >>>
>> >>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com>
>> >>> wrote:
>> >>>> Are you saying that in 0.8, producers don't use zkclient? Or don't
>> >> need
>> >>>> it? How can a producer dynamically respond to a change in the kafka
>> >>>> cluster without zk?
>> >>>>
>> >>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
>> >>>>
>> >>>>> Jae,
>> >>>>>
>> >>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses a
>> new
>> >>>>> getMetadata api to get topic/partition/leader information from the
>> >>> broker.
>> >>>>> Consumers still need ZK client. We plan to redesign the consumer post
>> >>> 0.8
>> >>>>> and can keep this in mind.
>> >>>>>
>> >>>>> Thanks,
>> >>>>>
>> >>>>> Jun
>> >>>>>
>> >>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <metacret@gmail.com
>> >
>> >>>>> wrote:
>> >>>>>
>> >>>>>> I want to suggest kafka should create only one instance of ZkClient
>> >>>>>> globally because ZkClient is thread safe and it will make many users
>> >>>>>> easily customize kafka source code for Zookeeper.
>> >>>>>>
>> >>>>>> In our company's cloud environment, it is not recommended to create
>> >>>>>> ZkClient from zkConnect string directly because zookeeper cluster
>> >> can
>> >>>>>> be dynamically changing. So, I have to create ZkClient using our
>> >>>>>> company's own platform library. Because of this requirement, I can't
>> >>>>>> use kafka jar file directly. I can modify and build kafka source
>> >> code
>> >>>>>> but I have to repeat this work whenever I update kafka version,
>> >> pretty
>> >>>>>> annoying.
>> >>>>>>
>> >>>>>> So, my suggestion is, let me pass ZkClient outs of Producer,
>> >> Consumer,
>> >>>>>> and Broker, as the following example.
>> >>>>>>
>> >>>>>> Producer<String, String> producer =
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> >>
>> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
>> >>>>>>
>> >>>>>> ConsumerConnector connector =
>> >>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
>> >>>>>> ConsumerConfig(consumerProps));
>> >>>>>>
>> >>>>>> KafkaServer is a little more complicated but I believe without much
>> >>>>>> effort we can refactor KafkaServer to be customized with ZkClient.
>> >>>>>>
>> >>>>>> I really appreciate if this suggestion is accepted and merged to
>> >> 0.8.
>> >>>>>> If you want me to contribute with this suggestion, please let me
>> >> know
>> >>>>>> your opinion. If you are positive with this idea, I will contribute
>> >>>>>> very happily.
>> >>>>>>
>> >>>>>> Thank you
>> >>>>>> Best, Jae
>> >>>>>>
>> >>>>>
>> >>>
>> >>
>>
>>
Re: Suggestion on ZkClient usage in Kafka
Posted by Jun Rao <ju...@gmail.com>.
David,
The change in 0.8 is that instead of requiring zk.connect, we require
broker.list. In both cases, you typically provide a list of hosts and
ports. Functionality wise, they achieve the same thing, ie, the producer is
able to send the data to the right broker. Are you saying that zk.connect
is more convenient? One benefit of using broker.list is that one can
provide a vip as the only host. This makes it easy to add/remove brokers
since no producer side config needs to be changed. Changing hosts in
zk.connect, on the other hand, requires config changes in the client.
Another reason for removing zkclient in the producer is that if the client
GCs, it can cause churns in the producer and extra load on the zk server.
Since our producer can be embedded in any client, it's hard for us to
control the GC rate. So, removing zkclient in the producer releases the
potential pressure from client GC.
We still rely on ZK for failure detection and leader election on the broker
and the consumer though.
Thanks,
Jun
On Tue, Nov 20, 2012 at 7:54 AM, David Arthur <mu...@gmail.com> wrote:
>
> On Nov 20, 2012, at 12:23 AM, Jun Rao wrote:
>
> > Jason,
> >
> > In 0.8, producer doesn't use zkclient at all. You just need to set
> > broker.list.
>
> This seems like a regression in functionality. For me, one of the benefits
> of Kafka is only needing to know about ZooKeeper
>
> > A number of things have changed In 0.8. First, number of
> > partitions of a topic is global in a cluster and they don't really change
> > as new brokers are added. Second, a partition is assigned to multiple
> > brokers for replication and one of the replicas is the leader which
> serves
> > writes. When a producer starts up, it first uses the getMetadata api to
> > figure out the replica assignment for the relevant topic/partition. It
> then
> > issues producer request directly the broker where the leader resides. If
> > the leader broker goes down, the producer gets an exception and it will
> > re-issue the getMetadata api to obtain the information about the new
> leader.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> >
> >> Well, they do use zk though, to get the initial list of kafka nodes, and
> >> while zk is available, presumably they do use it to keep up with the
> >> dynamically changing set of kafka brokers, no? You are just saying
> that if
> >> zk goes away, 0.8 producers can keep on producing, as long as the kafka
> >> cluster remains stable?
> >>
> >> Jason
> >>
> >> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> >>> wrote:
> >>
> >>> In 0.8, producers don't use zk. When producers encounter an error
> >>> while sending data, they use a special getMetadata request to refresh
> >>> the kafka cluster info from a randomly selected Kafka broker, and
> >>> retry sending the data.
> >>>
> >>> Thanks,
> >>> Neha
> >>>
> >>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com>
> >>> wrote:
> >>>> Are you saying that in 0.8, producers don't use zkclient? Or don't
> >> need
> >>>> it? How can a producer dynamically respond to a change in the kafka
> >>>> cluster without zk?
> >>>>
> >>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
> >>>>
> >>>>> Jae,
> >>>>>
> >>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses a
> new
> >>>>> getMetadata api to get topic/partition/leader information from the
> >>> broker.
> >>>>> Consumers still need ZK client. We plan to redesign the consumer post
> >>> 0.8
> >>>>> and can keep this in mind.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <metacret@gmail.com
> >
> >>>>> wrote:
> >>>>>
> >>>>>> I want to suggest kafka should create only one instance of ZkClient
> >>>>>> globally because ZkClient is thread safe and it will make many users
> >>>>>> easily customize kafka source code for Zookeeper.
> >>>>>>
> >>>>>> In our company's cloud environment, it is not recommended to create
> >>>>>> ZkClient from zkConnect string directly because zookeeper cluster
> >> can
> >>>>>> be dynamically changing. So, I have to create ZkClient using our
> >>>>>> company's own platform library. Because of this requirement, I can't
> >>>>>> use kafka jar file directly. I can modify and build kafka source
> >> code
> >>>>>> but I have to repeat this work whenever I update kafka version,
> >> pretty
> >>>>>> annoying.
> >>>>>>
> >>>>>> So, my suggestion is, let me pass ZkClient outs of Producer,
> >> Consumer,
> >>>>>> and Broker, as the following example.
> >>>>>>
> >>>>>> Producer<String, String> producer =
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
> >>>>>>
> >>>>>> ConsumerConnector connector =
> >>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
> >>>>>> ConsumerConfig(consumerProps));
> >>>>>>
> >>>>>> KafkaServer is a little more complicated but I believe without much
> >>>>>> effort we can refactor KafkaServer to be customized with ZkClient.
> >>>>>>
> >>>>>> I really appreciate if this suggestion is accepted and merged to
> >> 0.8.
> >>>>>> If you want me to contribute with this suggestion, please let me
> >> know
> >>>>>> your opinion. If you are positive with this idea, I will contribute
> >>>>>> very happily.
> >>>>>>
> >>>>>> Thank you
> >>>>>> Best, Jae
> >>>>>>
> >>>>>
> >>>
> >>
>
>
Re: Suggestion on ZkClient usage in Kafka
Posted by Jun Rao <ju...@gmail.com>.
In 0.8, both the broker and the consumer still need zkclient. So, a zk
cluster is still needed.
Thanks,
Jun
On Tue, Nov 20, 2012 at 8:04 AM, Jason Rosenberg <jb...@squareup.com> wrote:
> Agreed, I'm not sure I understand the move away from zk. Is it still
> required for consumers, and for the brokers themselves? If so, we still
> need to deploy a zk cluster anyway.
>
> Will kafka now support coordinating 1000's of producer clients?
>
> Jason
>
> On Tue, Nov 20, 2012 at 7:54 AM, David Arthur <mu...@gmail.com> wrote:
>
> >
> > On Nov 20, 2012, at 12:23 AM, Jun Rao wrote:
> >
> > > Jason,
> > >
> > > In 0.8, producer doesn't use zkclient at all. You just need to set
> > > broker.list.
> >
> > This seems like a regression in functionality. For me, one of the
> benefits
> > of Kafka is only needing to know about ZooKeeper
> >
> > > A number of things have changed In 0.8. First, number of
> > > partitions of a topic is global in a cluster and they don't really
> change
> > > as new brokers are added. Second, a partition is assigned to multiple
> > > brokers for replication and one of the replicas is the leader which
> > serves
> > > writes. When a producer starts up, it first uses the getMetadata api to
> > > figure out the replica assignment for the relevant topic/partition. It
> > then
> > > issues producer request directly the broker where the leader resides.
> If
> > > the leader broker goes down, the producer gets an exception and it will
> > > re-issue the getMetadata api to obtain the information about the new
> > leader.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <jb...@squareup.com>
> > wrote:
> > >
> > >> Well, they do use zk though, to get the initial list of kafka nodes,
> and
> > >> while zk is available, presumably they do use it to keep up with the
> > >> dynamically changing set of kafka brokers, no? You are just saying
> > that if
> > >> zk goes away, 0.8 producers can keep on producing, as long as the
> kafka
> > >> cluster remains stable?
> > >>
> > >> Jason
> > >>
> > >> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <
> > neha.narkhede@gmail.com
> > >>> wrote:
> > >>
> > >>> In 0.8, producers don't use zk. When producers encounter an error
> > >>> while sending data, they use a special getMetadata request to refresh
> > >>> the kafka cluster info from a randomly selected Kafka broker, and
> > >>> retry sending the data.
> > >>>
> > >>> Thanks,
> > >>> Neha
> > >>>
> > >>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com>
> > >>> wrote:
> > >>>> Are you saying that in 0.8, producers don't use zkclient? Or don't
> > >> need
> > >>>> it? How can a producer dynamically respond to a change in the kafka
> > >>>> cluster without zk?
> > >>>>
> > >>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
> > >>>>
> > >>>>> Jae,
> > >>>>>
> > >>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses a
> > new
> > >>>>> getMetadata api to get topic/partition/leader information from the
> > >>> broker.
> > >>>>> Consumers still need ZK client. We plan to redesign the consumer
> post
> > >>> 0.8
> > >>>>> and can keep this in mind.
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Jun
> > >>>>>
> > >>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <
> metacret@gmail.com
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> I want to suggest kafka should create only one instance of
> ZkClient
> > >>>>>> globally because ZkClient is thread safe and it will make many
> users
> > >>>>>> easily customize kafka source code for Zookeeper.
> > >>>>>>
> > >>>>>> In our company's cloud environment, it is not recommended to
> create
> > >>>>>> ZkClient from zkConnect string directly because zookeeper cluster
> > >> can
> > >>>>>> be dynamically changing. So, I have to create ZkClient using our
> > >>>>>> company's own platform library. Because of this requirement, I
> can't
> > >>>>>> use kafka jar file directly. I can modify and build kafka source
> > >> code
> > >>>>>> but I have to repeat this work whenever I update kafka version,
> > >> pretty
> > >>>>>> annoying.
> > >>>>>>
> > >>>>>> So, my suggestion is, let me pass ZkClient outs of Producer,
> > >> Consumer,
> > >>>>>> and Broker, as the following example.
> > >>>>>>
> > >>>>>> Producer<String, String> producer =
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> >
> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
> > >>>>>>
> > >>>>>> ConsumerConnector connector =
> > >>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
> > >>>>>> ConsumerConfig(consumerProps));
> > >>>>>>
> > >>>>>> KafkaServer is a little more complicated but I believe without
> much
> > >>>>>> effort we can refactor KafkaServer to be customized with ZkClient.
> > >>>>>>
> > >>>>>> I really appreciate if this suggestion is accepted and merged to
> > >> 0.8.
> > >>>>>> If you want me to contribute with this suggestion, please let me
> > >> know
> > >>>>>> your opinion. If you are positive with this idea, I will
> contribute
> > >>>>>> very happily.
> > >>>>>>
> > >>>>>> Thank you
> > >>>>>> Best, Jae
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> >
> >
>
Re: Suggestion on ZkClient usage in Kafka
Posted by Jason Rosenberg <jb...@squareup.com>.
Agreed, I'm not sure I understand the move away from zk. Is it still
required for consumers, and for the brokers themselves? If so, we still
need to deploy a zk cluster anyway.
Will kafka now support coordinating 1000's of producer clients?
Jason
On Tue, Nov 20, 2012 at 7:54 AM, David Arthur <mu...@gmail.com> wrote:
>
> On Nov 20, 2012, at 12:23 AM, Jun Rao wrote:
>
> > Jason,
> >
> > In 0.8, producer doesn't use zkclient at all. You just need to set
> > broker.list.
>
> This seems like a regression in functionality. For me, one of the benefits
> of Kafka is only needing to know about ZooKeeper
>
> > A number of things have changed In 0.8. First, number of
> > partitions of a topic is global in a cluster and they don't really change
> > as new brokers are added. Second, a partition is assigned to multiple
> > brokers for replication and one of the replicas is the leader which
> serves
> > writes. When a producer starts up, it first uses the getMetadata api to
> > figure out the replica assignment for the relevant topic/partition. It
> then
> > issues producer request directly the broker where the leader resides. If
> > the leader broker goes down, the producer gets an exception and it will
> > re-issue the getMetadata api to obtain the information about the new
> leader.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> >
> >> Well, they do use zk though, to get the initial list of kafka nodes, and
> >> while zk is available, presumably they do use it to keep up with the
> >> dynamically changing set of kafka brokers, no? You are just saying
> that if
> >> zk goes away, 0.8 producers can keep on producing, as long as the kafka
> >> cluster remains stable?
> >>
> >> Jason
> >>
> >> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> >>> wrote:
> >>
> >>> In 0.8, producers don't use zk. When producers encounter an error
> >>> while sending data, they use a special getMetadata request to refresh
> >>> the kafka cluster info from a randomly selected Kafka broker, and
> >>> retry sending the data.
> >>>
> >>> Thanks,
> >>> Neha
> >>>
> >>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com>
> >>> wrote:
> >>>> Are you saying that in 0.8, producers don't use zkclient? Or don't
> >> need
> >>>> it? How can a producer dynamically respond to a change in the kafka
> >>>> cluster without zk?
> >>>>
> >>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
> >>>>
> >>>>> Jae,
> >>>>>
> >>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses a
> new
> >>>>> getMetadata api to get topic/partition/leader information from the
> >>> broker.
> >>>>> Consumers still need ZK client. We plan to redesign the consumer post
> >>> 0.8
> >>>>> and can keep this in mind.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <metacret@gmail.com
> >
> >>>>> wrote:
> >>>>>
> >>>>>> I want to suggest kafka should create only one instance of ZkClient
> >>>>>> globally because ZkClient is thread safe and it will make many users
> >>>>>> easily customize kafka source code for Zookeeper.
> >>>>>>
> >>>>>> In our company's cloud environment, it is not recommended to create
> >>>>>> ZkClient from zkConnect string directly because zookeeper cluster
> >> can
> >>>>>> be dynamically changing. So, I have to create ZkClient using our
> >>>>>> company's own platform library. Because of this requirement, I can't
> >>>>>> use kafka jar file directly. I can modify and build kafka source
> >> code
> >>>>>> but I have to repeat this work whenever I update kafka version,
> >> pretty
> >>>>>> annoying.
> >>>>>>
> >>>>>> So, my suggestion is, let me pass ZkClient outs of Producer,
> >> Consumer,
> >>>>>> and Broker, as the following example.
> >>>>>>
> >>>>>> Producer<String, String> producer =
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
> >>>>>>
> >>>>>> ConsumerConnector connector =
> >>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
> >>>>>> ConsumerConfig(consumerProps));
> >>>>>>
> >>>>>> KafkaServer is a little more complicated but I believe without much
> >>>>>> effort we can refactor KafkaServer to be customized with ZkClient.
> >>>>>>
> >>>>>> I really appreciate if this suggestion is accepted and merged to
> >> 0.8.
> >>>>>> If you want me to contribute with this suggestion, please let me
> >> know
> >>>>>> your opinion. If you are positive with this idea, I will contribute
> >>>>>> very happily.
> >>>>>>
> >>>>>> Thank you
> >>>>>> Best, Jae
> >>>>>>
> >>>>>
> >>>
> >>
>
>
Re: Suggestion on ZkClient usage in Kafka
Posted by David Arthur <mu...@gmail.com>.
On Nov 20, 2012, at 12:23 AM, Jun Rao wrote:
> Jason,
>
> In 0.8, producer doesn't use zkclient at all. You just need to set
> broker.list.
This seems like a regression in functionality. For me, one of the benefits of Kafka is only needing to know about ZooKeeper
> A number of things have changed In 0.8. First, number of
> partitions of a topic is global in a cluster and they don't really change
> as new brokers are added. Second, a partition is assigned to multiple
> brokers for replication and one of the replicas is the leader which serves
> writes. When a producer starts up, it first uses the getMetadata api to
> figure out the replica assignment for the relevant topic/partition. It then
> issues producer request directly the broker where the leader resides. If
> the leader broker goes down, the producer gets an exception and it will
> re-issue the getMetadata api to obtain the information about the new leader.
>
> Thanks,
>
> Jun
>
> On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <jb...@squareup.com> wrote:
>
>> Well, they do use zk though, to get the initial list of kafka nodes, and
>> while zk is available, presumably they do use it to keep up with the
>> dynamically changing set of kafka brokers, no? You are just saying that if
>> zk goes away, 0.8 producers can keep on producing, as long as the kafka
>> cluster remains stable?
>>
>> Jason
>>
>> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <neha.narkhede@gmail.com
>>> wrote:
>>
>>> In 0.8, producers don't use zk. When producers encounter an error
>>> while sending data, they use a special getMetadata request to refresh
>>> the kafka cluster info from a randomly selected Kafka broker, and
>>> retry sending the data.
>>>
>>> Thanks,
>>> Neha
>>>
>>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com>
>>> wrote:
>>>> Are you saying that in 0.8, producers don't use zkclient? Or don't
>> need
>>>> it? How can a producer dynamically respond to a change in the kafka
>>>> cluster without zk?
>>>>
>>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>
>>>>> Jae,
>>>>>
>>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses a new
>>>>> getMetadata api to get topic/partition/leader information from the
>>> broker.
>>>>> Consumers still need ZK client. We plan to redesign the consumer post
>>> 0.8
>>>>> and can keep this in mind.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <me...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I want to suggest kafka should create only one instance of ZkClient
>>>>>> globally because ZkClient is thread safe and it will make many users
>>>>>> easily customize kafka source code for Zookeeper.
>>>>>>
>>>>>> In our company's cloud environment, it is not recommended to create
>>>>>> ZkClient from zkConnect string directly because zookeeper cluster
>> can
>>>>>> be dynamically changing. So, I have to create ZkClient using our
>>>>>> company's own platform library. Because of this requirement, I can't
>>>>>> use kafka jar file directly. I can modify and build kafka source
>> code
>>>>>> but I have to repeat this work whenever I update kafka version,
>> pretty
>>>>>> annoying.
>>>>>>
>>>>>> So, my suggestion is, let me pass ZkClient outs of Producer,
>> Consumer,
>>>>>> and Broker, as the following example.
>>>>>>
>>>>>> Producer<String, String> producer =
>>>>>>
>>>>>>
>>>>>
>>>
>> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
>>>>>>
>>>>>> ConsumerConnector connector =
>>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
>>>>>> ConsumerConfig(consumerProps));
>>>>>>
>>>>>> KafkaServer is a little more complicated but I believe without much
>>>>>> effort we can refactor KafkaServer to be customized with ZkClient.
>>>>>>
>>>>>> I really appreciate if this suggestion is accepted and merged to
>> 0.8.
>>>>>> If you want me to contribute with this suggestion, please let me
>> know
>>>>>> your opinion. If you are positive with this idea, I will contribute
>>>>>> very happily.
>>>>>>
>>>>>> Thank you
>>>>>> Best, Jae
>>>>>>
>>>>>
>>>
>>
Re: Suggestion on ZkClient usage in Kafka
Posted by Jun Rao <ju...@gmail.com>.
Jason,
In 0.8, producer doesn't use zkclient at all. You just need to set
broker.list. A number of things have changed In 0.8. First, number of
partitions of a topic is global in a cluster and they don't really change
as new brokers are added. Second, a partition is assigned to multiple
brokers for replication and one of the replicas is the leader which serves
writes. When a producer starts up, it first uses the getMetadata api to
figure out the replica assignment for the relevant topic/partition. It then
issues producer request directly the broker where the leader resides. If
the leader broker goes down, the producer gets an exception and it will
re-issue the getMetadata api to obtain the information about the new leader.
Thanks,
Jun
On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> Well, they do use zk though, to get the initial list of kafka nodes, and
> while zk is available, presumably they do use it to keep up with the
> dynamically changing set of kafka brokers, no? You are just saying that if
> zk goes away, 0.8 producers can keep on producing, as long as the kafka
> cluster remains stable?
>
> Jason
>
> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > In 0.8, producers don't use zk. When producers encounter an error
> > while sending data, they use a special getMetadata request to refresh
> > the kafka cluster info from a randomly selected Kafka broker, and
> > retry sending the data.
> >
> > Thanks,
> > Neha
> >
> > On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com>
> > wrote:
> > > Are you saying that in 0.8, producers don't use zkclient? Or don't
> need
> > > it? How can a producer dynamically respond to a change in the kafka
> > > cluster without zk?
> > >
> > > On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > >> Jae,
> > >>
> > >> In 0.8, producers don't need ZK client anymore. Instead, it uses a new
> > >> getMetadata api to get topic/partition/leader information from the
> > broker.
> > >> Consumers still need ZK client. We plan to redesign the consumer post
> > 0.8
> > >> and can keep this in mind.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <me...@gmail.com>
> > >> wrote:
> > >>
> > >> > I want to suggest kafka should create only one instance of ZkClient
> > >> > globally because ZkClient is thread safe and it will make many users
> > >> > easily customize kafka source code for Zookeeper.
> > >> >
> > >> > In our company's cloud environment, it is not recommended to create
> > >> > ZkClient from zkConnect string directly because zookeeper cluster
> can
> > >> > be dynamically changing. So, I have to create ZkClient using our
> > >> > company's own platform library. Because of this requirement, I can't
> > >> > use kafka jar file directly. I can modify and build kafka source
> code
> > >> > but I have to repeat this work whenever I update kafka version,
> pretty
> > >> > annoying.
> > >> >
> > >> > So, my suggestion is, let me pass ZkClient outs of Producer,
> Consumer,
> > >> > and Broker, as the following example.
> > >> >
> > >> > Producer<String, String> producer =
> > >> >
> > >> >
> > >>
> >
> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
> > >> >
> > >> > ConsumerConnector connector =
> > >> > Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
> > >> > ConsumerConfig(consumerProps));
> > >> >
> > >> > KafkaServer is a little more complicated but I believe without much
> > >> > effort we can refactor KafkaServer to be customized with ZkClient.
> > >> >
> > >> > I really appreciate if this suggestion is accepted and merged to
> 0.8.
> > >> > If you want me to contribute with this suggestion, please let me
> know
> > >> > your opinion. If you are positive with this idea, I will contribute
> > >> > very happily.
> > >> >
> > >> > Thank you
> > >> > Best, Jae
> > >> >
> > >>
> >
>
Re: Suggestion on ZkClient usage in Kafka
Posted by Jason Rosenberg <jb...@squareup.com>.
Well, they do use zk though, to get the initial list of kafka nodes, and
while zk is available, presumably they do use it to keep up with the
dynamically changing set of kafka brokers, no? You are just saying that if
zk goes away, 0.8 producers can keep on producing, as long as the kafka
cluster remains stable?
Jason
On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <ne...@gmail.com>wrote:
> In 0.8, producers don't use zk. When producers encounter an error
> while sending data, they use a special getMetadata request to refresh
> the kafka cluster info from a randomly selected Kafka broker, and
> retry sending the data.
>
> Thanks,
> Neha
>
> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> > Are you saying that in 0.8, producers don't use zkclient? Or don't need
> > it? How can a producer dynamically respond to a change in the kafka
> > cluster without zk?
> >
> > On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> >> Jae,
> >>
> >> In 0.8, producers don't need ZK client anymore. Instead, it uses a new
> >> getMetadata api to get topic/partition/leader information from the
> broker.
> >> Consumers still need ZK client. We plan to redesign the consumer post
> 0.8
> >> and can keep this in mind.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <me...@gmail.com>
> >> wrote:
> >>
> >> > I want to suggest kafka should create only one instance of ZkClient
> >> > globally because ZkClient is thread safe and it will make many users
> >> > easily customize kafka source code for Zookeeper.
> >> >
> >> > In our company's cloud environment, it is not recommended to create
> >> > ZkClient from zkConnect string directly because zookeeper cluster can
> >> > be dynamically changing. So, I have to create ZkClient using our
> >> > company's own platform library. Because of this requirement, I can't
> >> > use kafka jar file directly. I can modify and build kafka source code
> >> > but I have to repeat this work whenever I update kafka version, pretty
> >> > annoying.
> >> >
> >> > So, my suggestion is, let me pass ZkClient outs of Producer, Consumer,
> >> > and Broker, as the following example.
> >> >
> >> > Producer<String, String> producer =
> >> >
> >> >
> >>
> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
> >> >
> >> > ConsumerConnector connector =
> >> > Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
> >> > ConsumerConfig(consumerProps));
> >> >
> >> > KafkaServer is a little more complicated but I believe without much
> >> > effort we can refactor KafkaServer to be customized with ZkClient.
> >> >
> >> > I really appreciate if this suggestion is accepted and merged to 0.8.
> >> > If you want me to contribute with this suggestion, please let me know
> >> > your opinion. If you are positive with this idea, I will contribute
> >> > very happily.
> >> >
> >> > Thank you
> >> > Best, Jae
> >> >
> >>
>
Re: Suggestion on ZkClient usage in Kafka
Posted by Neha Narkhede <ne...@gmail.com>.
In 0.8, producers don't use zk. When producers encounter an error
while sending data, they use a special getMetadata request to refresh
the kafka cluster info from a randomly selected Kafka broker, and
retry sending the data.
Thanks,
Neha
On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> Are you saying that in 0.8, producers don't use zkclient? Or don't need
> it? How can a producer dynamically respond to a change in the kafka
> cluster without zk?
>
> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
>
>> Jae,
>>
>> In 0.8, producers don't need ZK client anymore. Instead, it uses a new
>> getMetadata api to get topic/partition/leader information from the broker.
>> Consumers still need ZK client. We plan to redesign the consumer post 0.8
>> and can keep this in mind.
>>
>> Thanks,
>>
>> Jun
>>
>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <me...@gmail.com>
>> wrote:
>>
>> > I want to suggest kafka should create only one instance of ZkClient
>> > globally because ZkClient is thread safe and it will make many users
>> > easily customize kafka source code for Zookeeper.
>> >
>> > In our company's cloud environment, it is not recommended to create
>> > ZkClient from zkConnect string directly because zookeeper cluster can
>> > be dynamically changing. So, I have to create ZkClient using our
>> > company's own platform library. Because of this requirement, I can't
>> > use kafka jar file directly. I can modify and build kafka source code
>> > but I have to repeat this work whenever I update kafka version, pretty
>> > annoying.
>> >
>> > So, my suggestion is, let me pass ZkClient outs of Producer, Consumer,
>> > and Broker, as the following example.
>> >
>> > Producer<String, String> producer =
>> >
>> >
>> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
>> >
>> > ConsumerConnector connector =
>> > Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
>> > ConsumerConfig(consumerProps));
>> >
>> > KafkaServer is a little more complicated but I believe without much
>> > effort we can refactor KafkaServer to be customized with ZkClient.
>> >
>> > I really appreciate if this suggestion is accepted and merged to 0.8.
>> > If you want me to contribute with this suggestion, please let me know
>> > your opinion. If you are positive with this idea, I will contribute
>> > very happily.
>> >
>> > Thank you
>> > Best, Jae
>> >
>>
Re: Suggestion on ZkClient usage in Kafka
Posted by Jason Rosenberg <jb...@squareup.com>.
Are you saying that in 0.8, producers don't use zkclient? Or don't need
it? How can a producer dynamically respond to a change in the kafka
cluster without zk?
On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <ju...@gmail.com> wrote:
> Jae,
>
> In 0.8, producers don't need ZK client anymore. Instead, it uses a new
> getMetadata api to get topic/partition/leader information from the broker.
> Consumers still need ZK client. We plan to redesign the consumer post 0.8
> and can keep this in mind.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <me...@gmail.com>
> wrote:
>
> > I want to suggest kafka should create only one instance of ZkClient
> > globally because ZkClient is thread safe and it will make many users
> > easily customize kafka source code for Zookeeper.
> >
> > In our company's cloud environment, it is not recommended to create
> > ZkClient from zkConnect string directly because zookeeper cluster can
> > be dynamically changing. So, I have to create ZkClient using our
> > company's own platform library. Because of this requirement, I can't
> > use kafka jar file directly. I can modify and build kafka source code
> > but I have to repeat this work whenever I update kafka version, pretty
> > annoying.
> >
> > So, my suggestion is, let me pass ZkClient outs of Producer, Consumer,
> > and Broker, as the following example.
> >
> > Producer<String, String> producer =
> >
> >
> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
> >
> > ConsumerConnector connector =
> > Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
> > ConsumerConfig(consumerProps));
> >
> > KafkaServer is a little more complicated but I believe without much
> > effort we can refactor KafkaServer to be customized with ZkClient.
> >
> > I really appreciate if this suggestion is accepted and merged to 0.8.
> > If you want me to contribute with this suggestion, please let me know
> > your opinion. If you are positive with this idea, I will contribute
> > very happily.
> >
> > Thank you
> > Best, Jae
> >
>
Re: Suggestion on ZkClient usage in Kafka
Posted by Jun Rao <ju...@gmail.com>.
Jae,
In 0.8, producers don't need ZK client anymore. Instead, it uses a new
getMetadata api to get topic/partition/leader information from the broker.
Consumers still need ZK client. We plan to redesign the consumer post 0.8
and can keep this in mind.
Thanks,
Jun
On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <me...@gmail.com> wrote:
> I want to suggest kafka should create only one instance of ZkClient
> globally because ZkClient is thread safe and it will make many users
> easily customize kafka source code for Zookeeper.
>
> In our company's cloud environment, it is not recommended to create
> ZkClient from zkConnect string directly because zookeeper cluster can
> be dynamically changing. So, I have to create ZkClient using our
> company's own platform library. Because of this requirement, I can't
> use kafka jar file directly. I can modify and build kafka source code
> but I have to repeat this work whenever I update kafka version, pretty
> annoying.
>
> So, my suggestion is, let me pass ZkClient outs of Producer, Consumer,
> and Broker, as the following example.
>
> Producer<String, String> producer =
>
> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
>
> ConsumerConnector connector =
> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
> ConsumerConfig(consumerProps));
>
> KafkaServer is a little more complicated but I believe without much
> effort we can refactor KafkaServer to be customized with ZkClient.
>
> I really appreciate if this suggestion is accepted and merged to 0.8.
> If you want me to contribute with this suggestion, please let me know
> your opinion. If you are positive with this idea, I will contribute
> very happily.
>
> Thank you
> Best, Jae
>
RE: Suggestion on ZkClient usage in Kafka
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
>From a technical perspective, it's not hard to specify a class to load in Java via a config file and the strategy pattern. I'm doing it on a project I'm working on for several aspects of the system. As long as the class adheres to a standardized interface, you could switch out the default Zookeeper client with whatever you wanted. All you have to do is make sure your class is part of the classpath. It's one of the few things I truly like about Java.
Casey
________________________________________
From: Bae, Jae Hyeon [metacret@gmail.com]
Sent: Monday, November 19, 2012 1:35 AM
To: kafka-users@incubator.apache.org
Subject: Suggestion on ZkClient usage in Kafka
I want to suggest kafka should create only one instance of ZkClient
globally because ZkClient is thread safe and it will make many users
easily customize kafka source code for Zookeeper.
In our company's cloud environment, it is not recommended to create
ZkClient from zkConnect string directly because zookeeper cluster can
be dynamically changing. So, I have to create ZkClient using our
company's own platform library. Because of this requirement, I can't
use kafka jar file directly. I can modify and build kafka source code
but I have to repeat this work whenever I update kafka version, pretty
annoying.
So, my suggestion is, let me pass ZkClient outs of Producer, Consumer,
and Broker, as the following example.
Producer<String, String> producer =
ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
ConsumerConnector connector =
Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
ConsumerConfig(consumerProps));
KafkaServer is a little more complicated but I believe without much
effort we can refactor KafkaServer to be customized with ZkClient.
I really appreciate if this suggestion is accepted and merged to 0.8.
If you want me to contribute with this suggestion, please let me know
your opinion. If you are positive with this idea, I will contribute
very happily.
Thank you
Best, Jae