You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Karl Kirch <kk...@wdtinc.com> on 2013/04/23 17:50:17 UTC
Expected behavior for clustered kafka setup
Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that only one broker is being used to actually send messages through. We're seeing this behavior after a restart and deletion of the kafka data (/tmp/kafka-logs). In the past its used all three and worked fine.
Is this normal? (i.e. it takes some time to spread across the cluster) Or should it distribute stuff immediately?
Karl
Re: Expected behavior for clustered kafka setup
Posted by Jun Rao <ju...@gmail.com>.
It doesn't really matter, but bringing ZK up first is probably better.
Thanks,
Jun
On Tue, Apr 23, 2013 at 12:14 PM, Karl Kirch <kk...@wdtinc.com> wrote:
> Thanks,
>
> I'll try to keep an eye on that and see if that fixes the problem.
> Also, how will that affect things if I have a node completely go away
> (i.e. zookeeper and kafka both go offline for a single node).
> Should I still bring up zookeeper on that node first or does it matter at
> that point?
>
>
> Karl
>
>
> On Apr 23, 2013, at 2:11 PM, Xavier Stevens <xa...@gaikai.com>
> wrote:
>
> > You should bring up your Zookeeper instances first and then the Kafka
> > brokers.
> >
> >
> > On Tue, Apr 23, 2013 at 11:56 AM, Karl Kirch <kk...@wdtinc.com> wrote:
> >
> >> Now to make things even more interesting. I restarted 2 and now it sees
> >> all 3 nodes.
> >> I think I've got some sort of weirdness happening with how I'm bringing
> >> everything online.
> >>
> >> I've tried starting up all the zookeeper instances and then each of the
> >> brokers. I've tried starting up all of the kafka brokers and then the
> >> zookeeper instances. I've also tried bringing up each machine (zookeeper
> >> and kafka) up individually. All seem to give inconsistent results.
> >>
> >> Is there a recommended way to bring nodes online?
> >>
> >> Karl
> >>
> >> On Apr 23, 2013, at 1:36 PM, Karl Kirch <kk...@wdtinc.com> wrote:
> >>
> >>> I've tried doing what was suggested (creating the topic folders and
> >> restarting) and I'm getting slightly different but still bad results.
> >>> Now it'll write to node 1 and 3 but not to 2.
> >>>
> >>> Karl
> >>>
> >>> On Apr 23, 2013, at 1:27 PM, Neha Narkhede <ne...@gmail.com>
> >>> wrote:
> >>>
> >>>> Can you see if you are running to
> >>>> https://issues.apache.org/jira/browse/KAFKA-278 ?
> >>>>
> >>>> Thanks,
> >>>> Neha
> >>>>
> >>>> On Tue, Apr 23, 2013 at 11:24 AM, Karl Kirch <kk...@wdtinc.com>
> wrote:
> >>>>> Zookeeper based.
> >>>>>
> >>>>> Karl
> >>>>>
> >>>>> On Apr 23, 2013, at 1:15 PM, Neha Narkhede <ne...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> What kind of producer do you use ? Zookeeper based or broker list ?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Neha
> >>>>>>
> >>>>>> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch <kk...@wdtinc.com>
> >> wrote:
> >>>>>>> For some extra info, this was a restart of the just zookeeper and
> >> kafka, our clients were all still online.
> >>>>>>>
> >>>>>>> Karl
> >>>>>>>
> >>>>>>> On Apr 23, 2013, at 10:50 AM, Karl Kirch <kk...@wdtinc.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10
> >> partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that
> only
> >> one broker is being used to actually send messages through. We're seeing
> >> this behavior after a restart and deletion of the kafka data
> >> (/tmp/kafka-logs). In the past its used all three and worked fine.
> >>>>>>>>
> >>>>>>>> Is this normal? (i.e. it takes some time to spread across the
> >> cluster) Or should it distribute stuff immediately?
> >>>>>>>>
> >>>>>>>> Karl
> >>>>>>>
> >>>>>
> >>>
> >>
> >>
>
>
Re: Expected behavior for clustered kafka setup
Posted by Karl Kirch <kk...@wdtinc.com>.
Thanks,
I'll try to keep an eye on that and see if that fixes the problem.
Also, how will that affect things if I have a node completely go away (i.e. zookeeper and kafka both go offline for a single node).
Should I still bring up zookeeper on that node first or does it matter at that point?
Karl
On Apr 23, 2013, at 2:11 PM, Xavier Stevens <xa...@gaikai.com>
wrote:
> You should bring up your Zookeeper instances first and then the Kafka
> brokers.
>
>
> On Tue, Apr 23, 2013 at 11:56 AM, Karl Kirch <kk...@wdtinc.com> wrote:
>
>> Now to make things even more interesting. I restarted 2 and now it sees
>> all 3 nodes.
>> I think I've got some sort of weirdness happening with how I'm bringing
>> everything online.
>>
>> I've tried starting up all the zookeeper instances and then each of the
>> brokers. I've tried starting up all of the kafka brokers and then the
>> zookeeper instances. I've also tried bringing up each machine (zookeeper
>> and kafka) up individually. All seem to give inconsistent results.
>>
>> Is there a recommended way to bring nodes online?
>>
>> Karl
>>
>> On Apr 23, 2013, at 1:36 PM, Karl Kirch <kk...@wdtinc.com> wrote:
>>
>>> I've tried doing what was suggested (creating the topic folders and
>> restarting) and I'm getting slightly different but still bad results.
>>> Now it'll write to node 1 and 3 but not to 2.
>>>
>>> Karl
>>>
>>> On Apr 23, 2013, at 1:27 PM, Neha Narkhede <ne...@gmail.com>
>>> wrote:
>>>
>>>> Can you see if you are running to
>>>> https://issues.apache.org/jira/browse/KAFKA-278 ?
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>> On Tue, Apr 23, 2013 at 11:24 AM, Karl Kirch <kk...@wdtinc.com> wrote:
>>>>> Zookeeper based.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Apr 23, 2013, at 1:15 PM, Neha Narkhede <ne...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> What kind of producer do you use ? Zookeeper based or broker list ?
>>>>>>
>>>>>> Thanks,
>>>>>> Neha
>>>>>>
>>>>>> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch <kk...@wdtinc.com>
>> wrote:
>>>>>>> For some extra info, this was a restart of the just zookeeper and
>> kafka, our clients were all still online.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>> On Apr 23, 2013, at 10:50 AM, Karl Kirch <kk...@wdtinc.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10
>> partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that only
>> one broker is being used to actually send messages through. We're seeing
>> this behavior after a restart and deletion of the kafka data
>> (/tmp/kafka-logs). In the past its used all three and worked fine.
>>>>>>>>
>>>>>>>> Is this normal? (i.e. it takes some time to spread across the
>> cluster) Or should it distribute stuff immediately?
>>>>>>>>
>>>>>>>> Karl
>>>>>>>
>>>>>
>>>
>>
>>
Re: Expected behavior for clustered kafka setup
Posted by Xavier Stevens <xa...@gaikai.com>.
You should bring up your Zookeeper instances first and then the Kafka
brokers.
On Tue, Apr 23, 2013 at 11:56 AM, Karl Kirch <kk...@wdtinc.com> wrote:
> Now to make things even more interesting. I restarted 2 and now it sees
> all 3 nodes.
> I think I've got some sort of weirdness happening with how I'm bringing
> everything online.
>
> I've tried starting up all the zookeeper instances and then each of the
> brokers. I've tried starting up all of the kafka brokers and then the
> zookeeper instances. I've also tried bringing up each machine (zookeeper
> and kafka) up individually. All seem to give inconsistent results.
>
> Is there a recommended way to bring nodes online?
>
> Karl
>
> On Apr 23, 2013, at 1:36 PM, Karl Kirch <kk...@wdtinc.com> wrote:
>
> > I've tried doing what was suggested (creating the topic folders and
> restarting) and I'm getting slightly different but still bad results.
> > Now it'll write to node 1 and 3 but not to 2.
> >
> > Karl
> >
> > On Apr 23, 2013, at 1:27 PM, Neha Narkhede <ne...@gmail.com>
> > wrote:
> >
> >> Can you see if you are running to
> >> https://issues.apache.org/jira/browse/KAFKA-278 ?
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Tue, Apr 23, 2013 at 11:24 AM, Karl Kirch <kk...@wdtinc.com> wrote:
> >>> Zookeeper based.
> >>>
> >>> Karl
> >>>
> >>> On Apr 23, 2013, at 1:15 PM, Neha Narkhede <ne...@gmail.com>
> >>> wrote:
> >>>
> >>>> What kind of producer do you use ? Zookeeper based or broker list ?
> >>>>
> >>>> Thanks,
> >>>> Neha
> >>>>
> >>>> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch <kk...@wdtinc.com>
> wrote:
> >>>>> For some extra info, this was a restart of the just zookeeper and
> kafka, our clients were all still online.
> >>>>>
> >>>>> Karl
> >>>>>
> >>>>> On Apr 23, 2013, at 10:50 AM, Karl Kirch <kk...@wdtinc.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10
> partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that only
> one broker is being used to actually send messages through. We're seeing
> this behavior after a restart and deletion of the kafka data
> (/tmp/kafka-logs). In the past its used all three and worked fine.
> >>>>>>
> >>>>>> Is this normal? (i.e. it takes some time to spread across the
> cluster) Or should it distribute stuff immediately?
> >>>>>>
> >>>>>> Karl
> >>>>>
> >>>
> >
>
>
Re: Expected behavior for clustered kafka setup
Posted by Karl Kirch <kk...@wdtinc.com>.
Now to make things even more interesting. I restarted 2 and now it sees all 3 nodes.
I think I've got some sort of weirdness happening with how I'm bringing everything online.
I've tried starting up all the zookeeper instances and then each of the brokers. I've tried starting up all of the kafka brokers and then the zookeeper instances. I've also tried bringing up each machine (zookeeper and kafka) up individually. All seem to give inconsistent results.
Is there a recommended way to bring nodes online?
Karl
On Apr 23, 2013, at 1:36 PM, Karl Kirch <kk...@wdtinc.com> wrote:
> I've tried doing what was suggested (creating the topic folders and restarting) and I'm getting slightly different but still bad results.
> Now it'll write to node 1 and 3 but not to 2.
>
> Karl
>
> On Apr 23, 2013, at 1:27 PM, Neha Narkhede <ne...@gmail.com>
> wrote:
>
>> Can you see if you are running to
>> https://issues.apache.org/jira/browse/KAFKA-278 ?
>>
>> Thanks,
>> Neha
>>
>> On Tue, Apr 23, 2013 at 11:24 AM, Karl Kirch <kk...@wdtinc.com> wrote:
>>> Zookeeper based.
>>>
>>> Karl
>>>
>>> On Apr 23, 2013, at 1:15 PM, Neha Narkhede <ne...@gmail.com>
>>> wrote:
>>>
>>>> What kind of producer do you use ? Zookeeper based or broker list ?
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch <kk...@wdtinc.com> wrote:
>>>>> For some extra info, this was a restart of the just zookeeper and kafka, our clients were all still online.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Apr 23, 2013, at 10:50 AM, Karl Kirch <kk...@wdtinc.com>
>>>>> wrote:
>>>>>
>>>>>> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that only one broker is being used to actually send messages through. We're seeing this behavior after a restart and deletion of the kafka data (/tmp/kafka-logs). In the past its used all three and worked fine.
>>>>>>
>>>>>> Is this normal? (i.e. it takes some time to spread across the cluster) Or should it distribute stuff immediately?
>>>>>>
>>>>>> Karl
>>>>>
>>>
>
Re: Expected behavior for clustered kafka setup
Posted by Karl Kirch <kk...@wdtinc.com>.
I've tried doing what was suggested (creating the topic folders and restarting) and I'm getting slightly different but still bad results.
Now it'll write to node 1 and 3 but not to 2.
Karl
On Apr 23, 2013, at 1:27 PM, Neha Narkhede <ne...@gmail.com>
wrote:
> Can you see if you are running to
> https://issues.apache.org/jira/browse/KAFKA-278 ?
>
> Thanks,
> Neha
>
> On Tue, Apr 23, 2013 at 11:24 AM, Karl Kirch <kk...@wdtinc.com> wrote:
>> Zookeeper based.
>>
>> Karl
>>
>> On Apr 23, 2013, at 1:15 PM, Neha Narkhede <ne...@gmail.com>
>> wrote:
>>
>>> What kind of producer do you use ? Zookeeper based or broker list ?
>>>
>>> Thanks,
>>> Neha
>>>
>>> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch <kk...@wdtinc.com> wrote:
>>>> For some extra info, this was a restart of the just zookeeper and kafka, our clients were all still online.
>>>>
>>>> Karl
>>>>
>>>> On Apr 23, 2013, at 10:50 AM, Karl Kirch <kk...@wdtinc.com>
>>>> wrote:
>>>>
>>>>> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that only one broker is being used to actually send messages through. We're seeing this behavior after a restart and deletion of the kafka data (/tmp/kafka-logs). In the past its used all three and worked fine.
>>>>>
>>>>> Is this normal? (i.e. it takes some time to spread across the cluster) Or should it distribute stuff immediately?
>>>>>
>>>>> Karl
>>>>
>>
Re: Expected behavior for clustered kafka setup
Posted by Neha Narkhede <ne...@gmail.com>.
Can you see if you are running to
https://issues.apache.org/jira/browse/KAFKA-278 ?
Thanks,
Neha
On Tue, Apr 23, 2013 at 11:24 AM, Karl Kirch <kk...@wdtinc.com> wrote:
> Zookeeper based.
>
> Karl
>
> On Apr 23, 2013, at 1:15 PM, Neha Narkhede <ne...@gmail.com>
> wrote:
>
>> What kind of producer do you use ? Zookeeper based or broker list ?
>>
>> Thanks,
>> Neha
>>
>> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch <kk...@wdtinc.com> wrote:
>>> For some extra info, this was a restart of the just zookeeper and kafka, our clients were all still online.
>>>
>>> Karl
>>>
>>> On Apr 23, 2013, at 10:50 AM, Karl Kirch <kk...@wdtinc.com>
>>> wrote:
>>>
>>>> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that only one broker is being used to actually send messages through. We're seeing this behavior after a restart and deletion of the kafka data (/tmp/kafka-logs). In the past its used all three and worked fine.
>>>>
>>>> Is this normal? (i.e. it takes some time to spread across the cluster) Or should it distribute stuff immediately?
>>>>
>>>> Karl
>>>
>
Re: Expected behavior for clustered kafka setup
Posted by Karl Kirch <kk...@wdtinc.com>.
Zookeeper based.
Karl
On Apr 23, 2013, at 1:15 PM, Neha Narkhede <ne...@gmail.com>
wrote:
> What kind of producer do you use ? Zookeeper based or broker list ?
>
> Thanks,
> Neha
>
> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch <kk...@wdtinc.com> wrote:
>> For some extra info, this was a restart of the just zookeeper and kafka, our clients were all still online.
>>
>> Karl
>>
>> On Apr 23, 2013, at 10:50 AM, Karl Kirch <kk...@wdtinc.com>
>> wrote:
>>
>>> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that only one broker is being used to actually send messages through. We're seeing this behavior after a restart and deletion of the kafka data (/tmp/kafka-logs). In the past its used all three and worked fine.
>>>
>>> Is this normal? (i.e. it takes some time to spread across the cluster) Or should it distribute stuff immediately?
>>>
>>> Karl
>>
Re: Expected behavior for clustered kafka setup
Posted by Neha Narkhede <ne...@gmail.com>.
What kind of producer do you use ? Zookeeper based or broker list ?
Thanks,
Neha
On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch <kk...@wdtinc.com> wrote:
> For some extra info, this was a restart of the just zookeeper and kafka, our clients were all still online.
>
> Karl
>
> On Apr 23, 2013, at 10:50 AM, Karl Kirch <kk...@wdtinc.com>
> wrote:
>
>> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that only one broker is being used to actually send messages through. We're seeing this behavior after a restart and deletion of the kafka data (/tmp/kafka-logs). In the past its used all three and worked fine.
>>
>> Is this normal? (i.e. it takes some time to spread across the cluster) Or should it distribute stuff immediately?
>>
>> Karl
>
Re: Expected behavior for clustered kafka setup
Posted by Karl Kirch <kk...@wdtinc.com>.
For some extra info, this was a restart of the just zookeeper and kafka, our clients were all still online.
Karl
On Apr 23, 2013, at 10:50 AM, Karl Kirch <kk...@wdtinc.com>
wrote:
> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that only one broker is being used to actually send messages through. We're seeing this behavior after a restart and deletion of the kafka data (/tmp/kafka-logs). In the past its used all three and worked fine.
>
> Is this normal? (i.e. it takes some time to spread across the cluster) Or should it distribute stuff immediately?
>
> Karl