You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Javier Iseruk <ja...@mercadolibre.com> on 2013/05/09 17:51:02 UTC
Problem with consumers in a consumers group
Hi.
I`m sending nginx logs to kafka. Each nginx goes to one exclusive partition.
I guarantee this by asigning to each nginx a number (1 to 22), and I use
that number for a custom partitoner's partition method as the key parameter:
return Integer.parseInt(key) % partitions
I have 22 nginxs, so I have 22 partitions with data.
The cluster has 5 brokers with ten partitions, so the total partitions for
the topic is 50.
In teory I could be able to start up to 22 consumers and each one should be
able to consume from one partition.
But what is happening is that I only can start three consumers:
The first one consumes the ten partitions of the first broker.
The second one consumes the ten partitions of the second broker.
The third one consumes the remaining 2 partitions of the third broker.
Any other consumer that I start in the same consumer group, doesn`t
consumes anything, and the three consumers that are consuming are not
balanced.
I am using Kafka 0.7.1, with zookeeper and the java consumer connector,
with the zk.connect property.
What could it be happening?
Thanks for the help!
--
Javier Iseruk
Architecture Team
Phone: +5411-4640-8063
Re: Problem with consumers in a consumers group
Posted by Dennis Haller <dh...@talenttech.com>.
I'm not sure if this relates directly to your problem, but you are using
non-standard topic names because of the @ character. Topic names should
only include alphanumeric plus hyphen and underscore. Have you checked your
logs for any errors because of processing these topic names?
Regards,
Dennis
On Thu, May 9, 2013 at 1:18 PM, Javier Iseruk <
javier.iseruk@mercadolibre.com> wrote:
> Hi again!
> I have more info to share.
> What I'm seeing in zookeeper is that the consumers are owning almost the
> same partitions quantity, no matter if that partition has data or not. So
> for example:
> Consumer 1: 17 partitions (all with data)
> Consumer 2: 17 partitions (only five with data)
> Consumer 3: 16 partitions (no data at all)
>
> That is is the photo I'm seeing now.
>
> I would expect that the consumers own the same not empty partitions
> quantity. I don`t care about the empty partitions, but apparently, Kafka
> divides the partitions no matter if it`s an empty partition or not.
> So I would like to see something like this:
> Consumer 1: 17 partitions (7 with data)
> Consumer 2: 17 partitions (7 with data)
> Consumer 3: 16 partitions (8 data at all)
>
> Or if I add another consumer, the result expected would be something like
> this:
> Consumer 1: 13 partitions (5 with data)
> Consumer 2: 13 partitions (5 with data)
> Consumer 3: 12 partitions (6 data at all)
> Consumer 4: 12 partitions (6 data at all)
>
> Am I right or wrong?
>
> Thanks for the help!!! (And sorry for my english)
>
>
>
>
>
> On Thu, May 9, 2013 at 3:40 PM, Javier Iseruk <
> javier.iseruk@mercadolibre.com> wrote:
>
> > Thanks for the answer. I send you the info:
> > I am ommiting all the files in each directory, just leaves one for
> showing
> > the structure.
> > There are two topics: access_log@nginx-nginx-api-master and
> > access_log@nginx-nginx-api-v1.
> > The consumer uses a whitelist with the regex access_log@nginx-nginx-api-
> .*,
> > so both topis are consumed
> > I don't see any problem with the topics names.
> >
> > Broker 0:
> > /kafka/logs> tree
> > .
> > |-- access_log@nginx-nginx-api-master-0
> > | `-- 00000000104220676901.kafka
> > |-- access_log@nginx-nginx-api-master-1
> > | `-- 00000001848656795194.kafka
> > |-- access_log@nginx-nginx-api-master-2
> > | `-- 00000000205422337816.kafka
> > |-- access_log@nginx-nginx-api-master-3
> > | `-- 00000000065714271675.kafka
> > |-- access_log@nginx-nginx-api-master-4
> > | `-- 00000002042994150834.kafka
> > |-- access_log@nginx-nginx-api-master-5
> > | `-- 00000003554704806448.kafka
> > |-- access_log@nginx-nginx-api-master-6
> > | `-- 00000005445803344687.kafka
> > |-- access_log@nginx-nginx-api-master-7
> > | `-- 00000003606342657110.kafka
> > |-- access_log@nginx-nginx-api-master-8
> > | `-- 00000002693751681000.kafka
> > |-- access_log@nginx-nginx-api-master-9
> > | `-- 00000001393405947605.kafka
> > |-- access_log@nginx-nginx-api-v1-0
> > | `-- 00000000132598986468.kafka
> > |-- access_log@nginx-nginx-api-v1-1
> > | `-- 00000000137269970386.kafka
> > |-- access_log@nginx-nginx-api-v1-2
> > | `-- 00000000136972308161.kafka
> > |-- access_log@nginx-nginx-api-v1-3
> > | `-- 00000000146486674802.kafka
> > |-- access_log@nginx-nginx-api-v1-6
> > | `-- 00000000009125337012.kafka
> > |-- access_log@nginx-nginx-api-v1-8
> > | `-- 00000000009010149647.kafka
> >
> > broker 1:
> > /kafka/logs> tree
> > .
> > |-- access_log@nginx-nginx-api-master-0
> > | `-- 00000002313153902757.kafka
> > |-- access_log@nginx-nginx-api-master-1
> > | `-- 00000003097259627500.kafka
> > |-- access_log@nginx-nginx-api-master-2
> > | `-- 00000002959536779137.kafka
> > |-- access_log@nginx-nginx-api-master-3
> > | `-- 00000003053863102518.kafka
> > |-- access_log@nginx-nginx-api-master-4
> > | `-- 00000002187535473763.kafka
> > |-- access_log@nginx-nginx-api-master-5
> > | `-- 00000004427181341253.kafka
> > |-- access_log@nginx-nginx-api-master-6
> > | `-- 00000002356845031168.kafka
> > |-- access_log@nginx-nginx-api-master-7
> > | `-- 00000002286128315593.kafka
> > |-- access_log@nginx-nginx-api-master-8
> > | `-- 00000002240322631141.kafka
> > |-- access_log@nginx-nginx-api-master-9
> > | `-- 00000002298179990598.kafka
> > |-- access_log@nginx-nginx-api-v1-0
> > | `-- 00000000000111649340.kafka
> > |-- access_log@nginx-nginx-api-v1-1
> > | `-- 00000000000112654964.kafka
> > |-- access_log@nginx-nginx-api-v1-2
> > | `-- 00000000000110500958.kafka
> > |-- access_log@nginx-nginx-api-v1-3
> > | `-- 00000000000107970659.kafka
> >
> > broker 2:
> > /kafka/logs> tree
> > .
> > |-- access_log@nginx-nginx-api-master-0
> > | `-- 00000001295587955823.kafka
> > |-- access_log@nginx-nginx-api-master-1
> > | `-- 00000001280489056047.kafka
> > |-- access_log@nginx-nginx-api-master-2
> > | `-- 00000000465978408295.kafka
> > |-- access_log@nginx-nginx-api-master-3
> > | `-- 00000000459780591967.kafka
> > |-- access_log@nginx-nginx-api-master-4
> > | `-- 00000000558193144600.kafka
> > |-- access_log@nginx-nginx-api-master-5
> > | `-- 00000000556797739267.kafka
> > |-- access_log@nginx-nginx-api-master-6
> > | `-- 00000000553188940491.kafka
> > |-- access_log@nginx-nginx-api-master-7
> > | `-- 00000000562912390935.kafka
> > |-- access_log@nginx-nginx-api-master-8
> > | `-- 00000000594913040282.kafka
> > |-- access_log@nginx-nginx-api-master-9
> > | `-- 00000000691253299404.kafka
> > |-- access_log@nginx-nginx-api-v1-0
> > | `-- 00000000005593211143.kafka
> > |-- access_log@nginx-nginx-api-v1-1
> > | `-- 00000000005569669640.kafka
> > |-- access_log@nginx-nginx-api-v1-2
> > | `-- 00000000005572829089.kafka
> > |-- access_log@nginx-nginx-api-v1-3
> > | `-- 00000000005598264785.kafka
> >
> > broker 3:
> > /kafka/logs> tree
> > .
> > |-- access_log@nginx-nginx-api-master-0
> > | `-- 00000000000048203157.kafka
> > |-- access_log@nginx-nginx-api-master-1
> > | `-- 00000000000003173076.kaf|-- access_log@nginx-nginx-api-v1-0
> > | `-- 00000000000000223076.kafka
> > |-- access_log@nginx-nginx-api-v1-2
> > | `-- 00000000000000135488.kafka
> >
> > broker 4:
> > /kafka/logs> tree
> > .
> > |-- access_log@nginx-nginx-api-master-0
> > | `-- 00000000000101790264.kafka
> > |-- access_log@nginx-nginx-api-master-1
> > | `-- 00000000000650193547.kafka
> > |-- access_log@nginx-nginx-api-master-4
> > | `-- 00000000000516227252.kafka
> > |-- access_log@nginx-nginx-api-master-5
> > | `-- 00000000000062572953.kafka
> > |-- access_log@nginx-nginx-api-master-6
> > | `-- 00000000000566421014.kafka
> > |-- access_log@nginx-nginx-api-master-8
> > | `-- 00000000000892503077.kafka
> > |-- access_log@nginx-nginx-api-v1-0
> > | `-- 00000000000011969830.kafka
> > |-- access_log@nginx-nginx-api-v1-2
> > | `-- 00000000000019642155.kafka
> > |-- access_log@nginx-nginx-api-v1-3
> > | `-- 00000000000000636597.kafka
> >
> > Thanks!
> > Javier Iseruk
> >
> >
> > On Thu, May 9, 2013 at 1:18 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
> >
> >> Could you check if the topic name is the same on all 5 brokers ? Can you
> >> send around the contents of the kafka log directory on all 5 brokers ?
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk <
> >> javier.iseruk@mercadolibre.com> wrote:
> >>
> >> > Hi.
> >> > I`m sending nginx logs to kafka. Each nginx goes to one exclusive
> >> > partition.
> >> > I guarantee this by asigning to each nginx a number (1 to 22), and I
> use
> >> > that number for a custom partitoner's partition method as the key
> >> > parameter:
> >> > return Integer.parseInt(key) % partitions
> >> >
> >> > I have 22 nginxs, so I have 22 partitions with data.
> >> > The cluster has 5 brokers with ten partitions, so the total partitions
> >> for
> >> > the topic is 50.
> >> >
> >> > In teory I could be able to start up to 22 consumers and each one
> >> should be
> >> > able to consume from one partition.
> >> > But what is happening is that I only can start three consumers:
> >> > The first one consumes the ten partitions of the first broker.
> >> > The second one consumes the ten partitions of the second broker.
> >> > The third one consumes the remaining 2 partitions of the third broker.
> >> >
> >> > Any other consumer that I start in the same consumer group, doesn`t
> >> > consumes anything, and the three consumers that are consuming are not
> >> > balanced.
> >> >
> >> > I am using Kafka 0.7.1, with zookeeper and the java consumer
> connector,
> >> > with the zk.connect property.
> >> >
> >> > What could it be happening?
> >> >
> >> > Thanks for the help!
> >> >
> >> > --
> >> > Javier Iseruk
> >> > Architecture Team
> >> > Phone: +5411-4640-8063
> >> >
> >>
> >
> >
> >
> > --
> > Javier Iseruk
> > Architecture Team
> > Phone: +5411-4640-8063
> >
> >
> >
>
>
> --
> Javier Iseruk
> Architecture Team
> Phone: +5411-4640-8063
>
Re: Problem with consumers in a consumers group
Posted by Javier Iseruk <ja...@mercadolibre.com>.
Hi again!
I have more info to share.
What I'm seeing in zookeeper is that the consumers are owning almost the
same partitions quantity, no matter if that partition has data or not. So
for example:
Consumer 1: 17 partitions (all with data)
Consumer 2: 17 partitions (only five with data)
Consumer 3: 16 partitions (no data at all)
That is is the photo I'm seeing now.
I would expect that the consumers own the same not empty partitions
quantity. I don`t care about the empty partitions, but apparently, Kafka
divides the partitions no matter if it`s an empty partition or not.
So I would like to see something like this:
Consumer 1: 17 partitions (7 with data)
Consumer 2: 17 partitions (7 with data)
Consumer 3: 16 partitions (8 data at all)
Or if I add another consumer, the result expected would be something like
this:
Consumer 1: 13 partitions (5 with data)
Consumer 2: 13 partitions (5 with data)
Consumer 3: 12 partitions (6 data at all)
Consumer 4: 12 partitions (6 data at all)
Am I right or wrong?
Thanks for the help!!! (And sorry for my english)
On Thu, May 9, 2013 at 3:40 PM, Javier Iseruk <
javier.iseruk@mercadolibre.com> wrote:
> Thanks for the answer. I send you the info:
> I am ommiting all the files in each directory, just leaves one for showing
> the structure.
> There are two topics: access_log@nginx-nginx-api-master and
> access_log@nginx-nginx-api-v1.
> The consumer uses a whitelist with the regex access_log@nginx-nginx-api-.*,
> so both topis are consumed
> I don't see any problem with the topics names.
>
> Broker 0:
> /kafka/logs> tree
> .
> |-- access_log@nginx-nginx-api-master-0
> | `-- 00000000104220676901.kafka
> |-- access_log@nginx-nginx-api-master-1
> | `-- 00000001848656795194.kafka
> |-- access_log@nginx-nginx-api-master-2
> | `-- 00000000205422337816.kafka
> |-- access_log@nginx-nginx-api-master-3
> | `-- 00000000065714271675.kafka
> |-- access_log@nginx-nginx-api-master-4
> | `-- 00000002042994150834.kafka
> |-- access_log@nginx-nginx-api-master-5
> | `-- 00000003554704806448.kafka
> |-- access_log@nginx-nginx-api-master-6
> | `-- 00000005445803344687.kafka
> |-- access_log@nginx-nginx-api-master-7
> | `-- 00000003606342657110.kafka
> |-- access_log@nginx-nginx-api-master-8
> | `-- 00000002693751681000.kafka
> |-- access_log@nginx-nginx-api-master-9
> | `-- 00000001393405947605.kafka
> |-- access_log@nginx-nginx-api-v1-0
> | `-- 00000000132598986468.kafka
> |-- access_log@nginx-nginx-api-v1-1
> | `-- 00000000137269970386.kafka
> |-- access_log@nginx-nginx-api-v1-2
> | `-- 00000000136972308161.kafka
> |-- access_log@nginx-nginx-api-v1-3
> | `-- 00000000146486674802.kafka
> |-- access_log@nginx-nginx-api-v1-6
> | `-- 00000000009125337012.kafka
> |-- access_log@nginx-nginx-api-v1-8
> | `-- 00000000009010149647.kafka
>
> broker 1:
> /kafka/logs> tree
> .
> |-- access_log@nginx-nginx-api-master-0
> | `-- 00000002313153902757.kafka
> |-- access_log@nginx-nginx-api-master-1
> | `-- 00000003097259627500.kafka
> |-- access_log@nginx-nginx-api-master-2
> | `-- 00000002959536779137.kafka
> |-- access_log@nginx-nginx-api-master-3
> | `-- 00000003053863102518.kafka
> |-- access_log@nginx-nginx-api-master-4
> | `-- 00000002187535473763.kafka
> |-- access_log@nginx-nginx-api-master-5
> | `-- 00000004427181341253.kafka
> |-- access_log@nginx-nginx-api-master-6
> | `-- 00000002356845031168.kafka
> |-- access_log@nginx-nginx-api-master-7
> | `-- 00000002286128315593.kafka
> |-- access_log@nginx-nginx-api-master-8
> | `-- 00000002240322631141.kafka
> |-- access_log@nginx-nginx-api-master-9
> | `-- 00000002298179990598.kafka
> |-- access_log@nginx-nginx-api-v1-0
> | `-- 00000000000111649340.kafka
> |-- access_log@nginx-nginx-api-v1-1
> | `-- 00000000000112654964.kafka
> |-- access_log@nginx-nginx-api-v1-2
> | `-- 00000000000110500958.kafka
> |-- access_log@nginx-nginx-api-v1-3
> | `-- 00000000000107970659.kafka
>
> broker 2:
> /kafka/logs> tree
> .
> |-- access_log@nginx-nginx-api-master-0
> | `-- 00000001295587955823.kafka
> |-- access_log@nginx-nginx-api-master-1
> | `-- 00000001280489056047.kafka
> |-- access_log@nginx-nginx-api-master-2
> | `-- 00000000465978408295.kafka
> |-- access_log@nginx-nginx-api-master-3
> | `-- 00000000459780591967.kafka
> |-- access_log@nginx-nginx-api-master-4
> | `-- 00000000558193144600.kafka
> |-- access_log@nginx-nginx-api-master-5
> | `-- 00000000556797739267.kafka
> |-- access_log@nginx-nginx-api-master-6
> | `-- 00000000553188940491.kafka
> |-- access_log@nginx-nginx-api-master-7
> | `-- 00000000562912390935.kafka
> |-- access_log@nginx-nginx-api-master-8
> | `-- 00000000594913040282.kafka
> |-- access_log@nginx-nginx-api-master-9
> | `-- 00000000691253299404.kafka
> |-- access_log@nginx-nginx-api-v1-0
> | `-- 00000000005593211143.kafka
> |-- access_log@nginx-nginx-api-v1-1
> | `-- 00000000005569669640.kafka
> |-- access_log@nginx-nginx-api-v1-2
> | `-- 00000000005572829089.kafka
> |-- access_log@nginx-nginx-api-v1-3
> | `-- 00000000005598264785.kafka
>
> broker 3:
> /kafka/logs> tree
> .
> |-- access_log@nginx-nginx-api-master-0
> | `-- 00000000000048203157.kafka
> |-- access_log@nginx-nginx-api-master-1
> | `-- 00000000000003173076.kaf|-- access_log@nginx-nginx-api-v1-0
> | `-- 00000000000000223076.kafka
> |-- access_log@nginx-nginx-api-v1-2
> | `-- 00000000000000135488.kafka
>
> broker 4:
> /kafka/logs> tree
> .
> |-- access_log@nginx-nginx-api-master-0
> | `-- 00000000000101790264.kafka
> |-- access_log@nginx-nginx-api-master-1
> | `-- 00000000000650193547.kafka
> |-- access_log@nginx-nginx-api-master-4
> | `-- 00000000000516227252.kafka
> |-- access_log@nginx-nginx-api-master-5
> | `-- 00000000000062572953.kafka
> |-- access_log@nginx-nginx-api-master-6
> | `-- 00000000000566421014.kafka
> |-- access_log@nginx-nginx-api-master-8
> | `-- 00000000000892503077.kafka
> |-- access_log@nginx-nginx-api-v1-0
> | `-- 00000000000011969830.kafka
> |-- access_log@nginx-nginx-api-v1-2
> | `-- 00000000000019642155.kafka
> |-- access_log@nginx-nginx-api-v1-3
> | `-- 00000000000000636597.kafka
>
> Thanks!
> Javier Iseruk
>
>
> On Thu, May 9, 2013 at 1:18 PM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> Could you check if the topic name is the same on all 5 brokers ? Can you
>> send around the contents of the kafka log directory on all 5 brokers ?
>>
>> Thanks,
>> Neha
>>
>>
>> On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk <
>> javier.iseruk@mercadolibre.com> wrote:
>>
>> > Hi.
>> > I`m sending nginx logs to kafka. Each nginx goes to one exclusive
>> > partition.
>> > I guarantee this by asigning to each nginx a number (1 to 22), and I use
>> > that number for a custom partitoner's partition method as the key
>> > parameter:
>> > return Integer.parseInt(key) % partitions
>> >
>> > I have 22 nginxs, so I have 22 partitions with data.
>> > The cluster has 5 brokers with ten partitions, so the total partitions
>> for
>> > the topic is 50.
>> >
>> > In teory I could be able to start up to 22 consumers and each one
>> should be
>> > able to consume from one partition.
>> > But what is happening is that I only can start three consumers:
>> > The first one consumes the ten partitions of the first broker.
>> > The second one consumes the ten partitions of the second broker.
>> > The third one consumes the remaining 2 partitions of the third broker.
>> >
>> > Any other consumer that I start in the same consumer group, doesn`t
>> > consumes anything, and the three consumers that are consuming are not
>> > balanced.
>> >
>> > I am using Kafka 0.7.1, with zookeeper and the java consumer connector,
>> > with the zk.connect property.
>> >
>> > What could it be happening?
>> >
>> > Thanks for the help!
>> >
>> > --
>> > Javier Iseruk
>> > Architecture Team
>> > Phone: +5411-4640-8063
>> >
>>
>
>
>
> --
> Javier Iseruk
> Architecture Team
> Phone: +5411-4640-8063
>
>
>
--
Javier Iseruk
Architecture Team
Phone: +5411-4640-8063
Re: Problem with consumers in a consumers group
Posted by Javier Iseruk <ja...@mercadolibre.com>.
Thanks for the answer. I send you the info:
I am ommiting all the files in each directory, just leaves one for showing
the structure.
There are two topics: access_log@nginx-nginx-api-master and
access_log@nginx-nginx-api-v1.
The consumer uses a whitelist with the regex access_log@nginx-nginx-api-.*,
so both topis are consumed
I don't see any problem with the topics names.
Broker 0:
/kafka/logs> tree
.
|-- access_log@nginx-nginx-api-master-0
| `-- 00000000104220676901.kafka
|-- access_log@nginx-nginx-api-master-1
| `-- 00000001848656795194.kafka
|-- access_log@nginx-nginx-api-master-2
| `-- 00000000205422337816.kafka
|-- access_log@nginx-nginx-api-master-3
| `-- 00000000065714271675.kafka
|-- access_log@nginx-nginx-api-master-4
| `-- 00000002042994150834.kafka
|-- access_log@nginx-nginx-api-master-5
| `-- 00000003554704806448.kafka
|-- access_log@nginx-nginx-api-master-6
| `-- 00000005445803344687.kafka
|-- access_log@nginx-nginx-api-master-7
| `-- 00000003606342657110.kafka
|-- access_log@nginx-nginx-api-master-8
| `-- 00000002693751681000.kafka
|-- access_log@nginx-nginx-api-master-9
| `-- 00000001393405947605.kafka
|-- access_log@nginx-nginx-api-v1-0
| `-- 00000000132598986468.kafka
|-- access_log@nginx-nginx-api-v1-1
| `-- 00000000137269970386.kafka
|-- access_log@nginx-nginx-api-v1-2
| `-- 00000000136972308161.kafka
|-- access_log@nginx-nginx-api-v1-3
| `-- 00000000146486674802.kafka
|-- access_log@nginx-nginx-api-v1-6
| `-- 00000000009125337012.kafka
|-- access_log@nginx-nginx-api-v1-8
| `-- 00000000009010149647.kafka
broker 1:
/kafka/logs> tree
.
|-- access_log@nginx-nginx-api-master-0
| `-- 00000002313153902757.kafka
|-- access_log@nginx-nginx-api-master-1
| `-- 00000003097259627500.kafka
|-- access_log@nginx-nginx-api-master-2
| `-- 00000002959536779137.kafka
|-- access_log@nginx-nginx-api-master-3
| `-- 00000003053863102518.kafka
|-- access_log@nginx-nginx-api-master-4
| `-- 00000002187535473763.kafka
|-- access_log@nginx-nginx-api-master-5
| `-- 00000004427181341253.kafka
|-- access_log@nginx-nginx-api-master-6
| `-- 00000002356845031168.kafka
|-- access_log@nginx-nginx-api-master-7
| `-- 00000002286128315593.kafka
|-- access_log@nginx-nginx-api-master-8
| `-- 00000002240322631141.kafka
|-- access_log@nginx-nginx-api-master-9
| `-- 00000002298179990598.kafka
|-- access_log@nginx-nginx-api-v1-0
| `-- 00000000000111649340.kafka
|-- access_log@nginx-nginx-api-v1-1
| `-- 00000000000112654964.kafka
|-- access_log@nginx-nginx-api-v1-2
| `-- 00000000000110500958.kafka
|-- access_log@nginx-nginx-api-v1-3
| `-- 00000000000107970659.kafka
broker 2:
/kafka/logs> tree
.
|-- access_log@nginx-nginx-api-master-0
| `-- 00000001295587955823.kafka
|-- access_log@nginx-nginx-api-master-1
| `-- 00000001280489056047.kafka
|-- access_log@nginx-nginx-api-master-2
| `-- 00000000465978408295.kafka
|-- access_log@nginx-nginx-api-master-3
| `-- 00000000459780591967.kafka
|-- access_log@nginx-nginx-api-master-4
| `-- 00000000558193144600.kafka
|-- access_log@nginx-nginx-api-master-5
| `-- 00000000556797739267.kafka
|-- access_log@nginx-nginx-api-master-6
| `-- 00000000553188940491.kafka
|-- access_log@nginx-nginx-api-master-7
| `-- 00000000562912390935.kafka
|-- access_log@nginx-nginx-api-master-8
| `-- 00000000594913040282.kafka
|-- access_log@nginx-nginx-api-master-9
| `-- 00000000691253299404.kafka
|-- access_log@nginx-nginx-api-v1-0
| `-- 00000000005593211143.kafka
|-- access_log@nginx-nginx-api-v1-1
| `-- 00000000005569669640.kafka
|-- access_log@nginx-nginx-api-v1-2
| `-- 00000000005572829089.kafka
|-- access_log@nginx-nginx-api-v1-3
| `-- 00000000005598264785.kafka
broker 3:
/kafka/logs> tree
.
|-- access_log@nginx-nginx-api-master-0
| `-- 00000000000048203157.kafka
|-- access_log@nginx-nginx-api-master-1
| `-- 00000000000003173076.kaf|-- access_log@nginx-nginx-api-v1-0
| `-- 00000000000000223076.kafka
|-- access_log@nginx-nginx-api-v1-2
| `-- 00000000000000135488.kafka
broker 4:
/kafka/logs> tree
.
|-- access_log@nginx-nginx-api-master-0
| `-- 00000000000101790264.kafka
|-- access_log@nginx-nginx-api-master-1
| `-- 00000000000650193547.kafka
|-- access_log@nginx-nginx-api-master-4
| `-- 00000000000516227252.kafka
|-- access_log@nginx-nginx-api-master-5
| `-- 00000000000062572953.kafka
|-- access_log@nginx-nginx-api-master-6
| `-- 00000000000566421014.kafka
|-- access_log@nginx-nginx-api-master-8
| `-- 00000000000892503077.kafka
|-- access_log@nginx-nginx-api-v1-0
| `-- 00000000000011969830.kafka
|-- access_log@nginx-nginx-api-v1-2
| `-- 00000000000019642155.kafka
|-- access_log@nginx-nginx-api-v1-3
| `-- 00000000000000636597.kafka
Thanks!
Javier Iseruk
On Thu, May 9, 2013 at 1:18 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Could you check if the topic name is the same on all 5 brokers ? Can you
> send around the contents of the kafka log directory on all 5 brokers ?
>
> Thanks,
> Neha
>
>
> On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk <
> javier.iseruk@mercadolibre.com> wrote:
>
> > Hi.
> > I`m sending nginx logs to kafka. Each nginx goes to one exclusive
> > partition.
> > I guarantee this by asigning to each nginx a number (1 to 22), and I use
> > that number for a custom partitoner's partition method as the key
> > parameter:
> > return Integer.parseInt(key) % partitions
> >
> > I have 22 nginxs, so I have 22 partitions with data.
> > The cluster has 5 brokers with ten partitions, so the total partitions
> for
> > the topic is 50.
> >
> > In teory I could be able to start up to 22 consumers and each one should
> be
> > able to consume from one partition.
> > But what is happening is that I only can start three consumers:
> > The first one consumes the ten partitions of the first broker.
> > The second one consumes the ten partitions of the second broker.
> > The third one consumes the remaining 2 partitions of the third broker.
> >
> > Any other consumer that I start in the same consumer group, doesn`t
> > consumes anything, and the three consumers that are consuming are not
> > balanced.
> >
> > I am using Kafka 0.7.1, with zookeeper and the java consumer connector,
> > with the zk.connect property.
> >
> > What could it be happening?
> >
> > Thanks for the help!
> >
> > --
> > Javier Iseruk
> > Architecture Team
> > Phone: +5411-4640-8063
> >
>
--
Javier Iseruk
Architecture Team
Phone: +5411-4640-8063
Re: Problem with consumers in a consumers group
Posted by Neha Narkhede <ne...@gmail.com>.
Could you check if the topic name is the same on all 5 brokers ? Can you
send around the contents of the kafka log directory on all 5 brokers ?
Thanks,
Neha
On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk <
javier.iseruk@mercadolibre.com> wrote:
> Hi.
> I`m sending nginx logs to kafka. Each nginx goes to one exclusive
> partition.
> I guarantee this by asigning to each nginx a number (1 to 22), and I use
> that number for a custom partitoner's partition method as the key
> parameter:
> return Integer.parseInt(key) % partitions
>
> I have 22 nginxs, so I have 22 partitions with data.
> The cluster has 5 brokers with ten partitions, so the total partitions for
> the topic is 50.
>
> In teory I could be able to start up to 22 consumers and each one should be
> able to consume from one partition.
> But what is happening is that I only can start three consumers:
> The first one consumes the ten partitions of the first broker.
> The second one consumes the ten partitions of the second broker.
> The third one consumes the remaining 2 partitions of the third broker.
>
> Any other consumer that I start in the same consumer group, doesn`t
> consumes anything, and the three consumers that are consuming are not
> balanced.
>
> I am using Kafka 0.7.1, with zookeeper and the java consumer connector,
> with the zk.connect property.
>
> What could it be happening?
>
> Thanks for the help!
>
> --
> Javier Iseruk
> Architecture Team
> Phone: +5411-4640-8063
>