You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Palur Sandeep <ps...@hawk.iit.edu> on 2014/11/13 23:12:13 UTC
One of two consumers is always Idle though I have 2 partitions
Dear Developers,
I am 2nd year masters student at IIT. I am using Kafka for one of my
research projects.My question is the following:
1. I have a producer, consumer and a broker(that contains 1st partition of
my topic) on node1
2. I have a producer, consumer, zookeeper and a broker(that contains 2nd
partition of my topic) on node2
3. Here comes my problem: though I have two partitions only one consumer
pulls messages and the other one is always idle.
What is that I can do to keep both of my consumer busy?
Thank you
--
Regards,
Sandeep Palur
Data-Intensive Distributed Systems Laboratory, CS/IIT
Department of Computer Science, Illinois Institute of Technology (IIT)
Phone : 312-647-9833
Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
Re: One of two consumers is always Idle though I have 2 partitions
Posted by Sharninder <sh...@gmail.com>.
If you're not using your own partitioning logic, messages are partitioned
randomly. This is the current default behavior I believe.
On Fri, Nov 14, 2014 at 12:01 PM, Palur Sandeep <ps...@hawk.iit.edu>
wrote:
> Thank you Chia-chun,Joe and Jagat.
>
> I am not using any custom partitioner logic. Here is what I observed when I
> ran kafka on 4 nodes with the following structure:
>
> 1. Each node has a producer, consumer and a broker (that contains one
> partition of my topic) and one of the machine has the Zookeeper too.
> 2. Producer in each node sends 10000 messages to my topic.
> 3. I observed that consumer in all 4 nodes gets some messages and some
> times only 2 nodes receive messages and one doesn't and sometimes only one
> node receives messages and 3 doesnt receive any messages.
>
> So according to my observation, producer is sending messages to random
> partition.
>
> Am I correct?
>
> Thank you
> Sandeep
>
>
>
>
>
>
> On Thu, Nov 13, 2014 at 9:34 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Yup, sounds like
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
> > ?
> >
> > This should go away with 0.8.2 with the default partions now being 1 =8^)
> > with auto create topics.
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop
> > ********************************************/
> > On Nov 13, 2014 8:34 PM, "Chia-Chun Shih" <ch...@gmail.com>
> wrote:
> >
> > > Hi Palur,
> > >
> > > When producing messages, did you specify a key in your KeyedMessage? If
> > > not, producer will send all messages to ONE randomly selected partition
> > and
> > > stick to this partition for 10 minutes by default.
> > >
> > > regards,
> > > Chia-Chun
> > >
> > > 2014-11-14 7:19 GMT+08:00 Jagat Singh <ja...@gmail.com>:
> > >
> > > > It would be worth reading once the consumer section from the
> > > documentation.
> > > >
> > > > https://kafka.apache.org/documentation.html
> > > >
> > > >
> > > >
> > > > On Fri, Nov 14, 2014 at 10:09 AM, Palur Sandeep <
> psandeep@hawk.iit.edu
> > >
> > > > wrote:
> > > >
> > > > > Yes, they are on the same consumer group, but I have two
> partitions.
> > > > >
> > > > > On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh <jagatsingh@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Are both of them in same Consumer Group?
> > > > > >
> > > > > > On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep <
> > > psandeep@hawk.iit.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Dear Developers,
> > > > > > >
> > > > > > > I am 2nd year masters student at IIT. I am using Kafka for one
> of
> > > my
> > > > > > > research projects.My question is the following:
> > > > > > >
> > > > > > > 1. I have a producer, consumer and a broker(that contains 1st
> > > > partition
> > > > > > of
> > > > > > > my topic) on node1
> > > > > > > 2. I have a producer, consumer, zookeeper and a broker(that
> > > contains
> > > > > 2nd
> > > > > > > partition of my topic) on node2
> > > > > > > 3. Here comes my problem: though I have two partitions only one
> > > > > consumer
> > > > > > > pulls messages and the other one is always idle.
> > > > > > >
> > > > > > > What is that I can do to keep both of my consumer busy?
> > > > > > >
> > > > > > > Thank you
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Sandeep Palur
> > > > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > > > > > Department of Computer Science, Illinois Institute of
> Technology
> > > > (IIT)
> > > > > > > Phone : 312-647-9833
> > > > > > > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Sandeep Palur
> > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > > > Department of Computer Science, Illinois Institute of Technology
> > (IIT)
> > > > > Phone : 312-647-9833
> > > > > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Regards,
> Sandeep Palur
> Data-Intensive Distributed Systems Laboratory, CS/IIT
> Department of Computer Science, Illinois Institute of Technology (IIT)
> Phone : 312-647-9833
> Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
>
--
--
Sharninder
Re: One of two consumers is always Idle though I have 2 partitions
Posted by Palur Sandeep <ps...@hawk.iit.edu>.
Thank you Chia-chun,Joe and Jagat.
I am not using any custom partitioner logic. Here is what I observed when I
ran kafka on 4 nodes with the following structure:
1. Each node has a producer, consumer and a broker (that contains one
partition of my topic) and one of the machine has the Zookeeper too.
2. Producer in each node sends 10000 messages to my topic.
3. I observed that consumer in all 4 nodes gets some messages and some
times only 2 nodes receive messages and one doesn't and sometimes only one
node receives messages and 3 doesnt receive any messages.
So according to my observation, producer is sending messages to random
partition.
Am I correct?
Thank you
Sandeep
On Thu, Nov 13, 2014 at 9:34 PM, Joe Stein <jo...@stealth.ly> wrote:
> Yup, sounds like
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
> ?
>
> This should go away with 0.8.2 with the default partions now being 1 =8^)
> with auto create topics.
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop
> ********************************************/
> On Nov 13, 2014 8:34 PM, "Chia-Chun Shih" <ch...@gmail.com> wrote:
>
> > Hi Palur,
> >
> > When producing messages, did you specify a key in your KeyedMessage? If
> > not, producer will send all messages to ONE randomly selected partition
> and
> > stick to this partition for 10 minutes by default.
> >
> > regards,
> > Chia-Chun
> >
> > 2014-11-14 7:19 GMT+08:00 Jagat Singh <ja...@gmail.com>:
> >
> > > It would be worth reading once the consumer section from the
> > documentation.
> > >
> > > https://kafka.apache.org/documentation.html
> > >
> > >
> > >
> > > On Fri, Nov 14, 2014 at 10:09 AM, Palur Sandeep <psandeep@hawk.iit.edu
> >
> > > wrote:
> > >
> > > > Yes, they are on the same consumer group, but I have two partitions.
> > > >
> > > > On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh <ja...@gmail.com>
> > > wrote:
> > > >
> > > > > Are both of them in same Consumer Group?
> > > > >
> > > > > On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep <
> > psandeep@hawk.iit.edu>
> > > > > wrote:
> > > > >
> > > > > > Dear Developers,
> > > > > >
> > > > > > I am 2nd year masters student at IIT. I am using Kafka for one of
> > my
> > > > > > research projects.My question is the following:
> > > > > >
> > > > > > 1. I have a producer, consumer and a broker(that contains 1st
> > > partition
> > > > > of
> > > > > > my topic) on node1
> > > > > > 2. I have a producer, consumer, zookeeper and a broker(that
> > contains
> > > > 2nd
> > > > > > partition of my topic) on node2
> > > > > > 3. Here comes my problem: though I have two partitions only one
> > > > consumer
> > > > > > pulls messages and the other one is always idle.
> > > > > >
> > > > > > What is that I can do to keep both of my consumer busy?
> > > > > >
> > > > > > Thank you
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Sandeep Palur
> > > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > > > > Department of Computer Science, Illinois Institute of Technology
> > > (IIT)
> > > > > > Phone : 312-647-9833
> > > > > > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Sandeep Palur
> > > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > > Department of Computer Science, Illinois Institute of Technology
> (IIT)
> > > > Phone : 312-647-9833
> > > > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> > > >
> > >
> >
>
--
Regards,
Sandeep Palur
Data-Intensive Distributed Systems Laboratory, CS/IIT
Department of Computer Science, Illinois Institute of Technology (IIT)
Phone : 312-647-9833
Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
Re: One of two consumers is always Idle though I have 2 partitions
Posted by Joe Stein <jo...@stealth.ly>.
Yup, sounds like
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
?
This should go away with 0.8.2 with the default partions now being 1 =8^)
with auto create topics.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/
On Nov 13, 2014 8:34 PM, "Chia-Chun Shih" <ch...@gmail.com> wrote:
> Hi Palur,
>
> When producing messages, did you specify a key in your KeyedMessage? If
> not, producer will send all messages to ONE randomly selected partition and
> stick to this partition for 10 minutes by default.
>
> regards,
> Chia-Chun
>
> 2014-11-14 7:19 GMT+08:00 Jagat Singh <ja...@gmail.com>:
>
> > It would be worth reading once the consumer section from the
> documentation.
> >
> > https://kafka.apache.org/documentation.html
> >
> >
> >
> > On Fri, Nov 14, 2014 at 10:09 AM, Palur Sandeep <ps...@hawk.iit.edu>
> > wrote:
> >
> > > Yes, they are on the same consumer group, but I have two partitions.
> > >
> > > On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh <ja...@gmail.com>
> > wrote:
> > >
> > > > Are both of them in same Consumer Group?
> > > >
> > > > On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep <
> psandeep@hawk.iit.edu>
> > > > wrote:
> > > >
> > > > > Dear Developers,
> > > > >
> > > > > I am 2nd year masters student at IIT. I am using Kafka for one of
> my
> > > > > research projects.My question is the following:
> > > > >
> > > > > 1. I have a producer, consumer and a broker(that contains 1st
> > partition
> > > > of
> > > > > my topic) on node1
> > > > > 2. I have a producer, consumer, zookeeper and a broker(that
> contains
> > > 2nd
> > > > > partition of my topic) on node2
> > > > > 3. Here comes my problem: though I have two partitions only one
> > > consumer
> > > > > pulls messages and the other one is always idle.
> > > > >
> > > > > What is that I can do to keep both of my consumer busy?
> > > > >
> > > > > Thank you
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Sandeep Palur
> > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > > > Department of Computer Science, Illinois Institute of Technology
> > (IIT)
> > > > > Phone : 312-647-9833
> > > > > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Sandeep Palur
> > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > Department of Computer Science, Illinois Institute of Technology (IIT)
> > > Phone : 312-647-9833
> > > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> > >
> >
>
Re: One of two consumers is always Idle though I have 2 partitions
Posted by Chia-Chun Shih <ch...@gmail.com>.
Hi Palur,
When producing messages, did you specify a key in your KeyedMessage? If
not, producer will send all messages to ONE randomly selected partition and
stick to this partition for 10 minutes by default.
regards,
Chia-Chun
2014-11-14 7:19 GMT+08:00 Jagat Singh <ja...@gmail.com>:
> It would be worth reading once the consumer section from the documentation.
>
> https://kafka.apache.org/documentation.html
>
>
>
> On Fri, Nov 14, 2014 at 10:09 AM, Palur Sandeep <ps...@hawk.iit.edu>
> wrote:
>
> > Yes, they are on the same consumer group, but I have two partitions.
> >
> > On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh <ja...@gmail.com>
> wrote:
> >
> > > Are both of them in same Consumer Group?
> > >
> > > On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep <ps...@hawk.iit.edu>
> > > wrote:
> > >
> > > > Dear Developers,
> > > >
> > > > I am 2nd year masters student at IIT. I am using Kafka for one of my
> > > > research projects.My question is the following:
> > > >
> > > > 1. I have a producer, consumer and a broker(that contains 1st
> partition
> > > of
> > > > my topic) on node1
> > > > 2. I have a producer, consumer, zookeeper and a broker(that contains
> > 2nd
> > > > partition of my topic) on node2
> > > > 3. Here comes my problem: though I have two partitions only one
> > consumer
> > > > pulls messages and the other one is always idle.
> > > >
> > > > What is that I can do to keep both of my consumer busy?
> > > >
> > > > Thank you
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Sandeep Palur
> > > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > > Department of Computer Science, Illinois Institute of Technology
> (IIT)
> > > > Phone : 312-647-9833
> > > > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Sandeep Palur
> > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > Department of Computer Science, Illinois Institute of Technology (IIT)
> > Phone : 312-647-9833
> > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> >
>
Re: One of two consumers is always Idle though I have 2 partitions
Posted by Jagat Singh <ja...@gmail.com>.
It would be worth reading once the consumer section from the documentation.
https://kafka.apache.org/documentation.html
On Fri, Nov 14, 2014 at 10:09 AM, Palur Sandeep <ps...@hawk.iit.edu>
wrote:
> Yes, they are on the same consumer group, but I have two partitions.
>
> On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh <ja...@gmail.com> wrote:
>
> > Are both of them in same Consumer Group?
> >
> > On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep <ps...@hawk.iit.edu>
> > wrote:
> >
> > > Dear Developers,
> > >
> > > I am 2nd year masters student at IIT. I am using Kafka for one of my
> > > research projects.My question is the following:
> > >
> > > 1. I have a producer, consumer and a broker(that contains 1st partition
> > of
> > > my topic) on node1
> > > 2. I have a producer, consumer, zookeeper and a broker(that contains
> 2nd
> > > partition of my topic) on node2
> > > 3. Here comes my problem: though I have two partitions only one
> consumer
> > > pulls messages and the other one is always idle.
> > >
> > > What is that I can do to keep both of my consumer busy?
> > >
> > > Thank you
> > >
> > >
> > > --
> > > Regards,
> > > Sandeep Palur
> > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > Department of Computer Science, Illinois Institute of Technology (IIT)
> > > Phone : 312-647-9833
> > > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> > >
> >
>
>
>
> --
> Regards,
> Sandeep Palur
> Data-Intensive Distributed Systems Laboratory, CS/IIT
> Department of Computer Science, Illinois Institute of Technology (IIT)
> Phone : 312-647-9833
> Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
>
Re: One of two consumers is always Idle though I have 2 partitions
Posted by Palur Sandeep <ps...@hawk.iit.edu>.
Yes, they are on the same consumer group, but I have two partitions.
On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh <ja...@gmail.com> wrote:
> Are both of them in same Consumer Group?
>
> On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep <ps...@hawk.iit.edu>
> wrote:
>
> > Dear Developers,
> >
> > I am 2nd year masters student at IIT. I am using Kafka for one of my
> > research projects.My question is the following:
> >
> > 1. I have a producer, consumer and a broker(that contains 1st partition
> of
> > my topic) on node1
> > 2. I have a producer, consumer, zookeeper and a broker(that contains 2nd
> > partition of my topic) on node2
> > 3. Here comes my problem: though I have two partitions only one consumer
> > pulls messages and the other one is always idle.
> >
> > What is that I can do to keep both of my consumer busy?
> >
> > Thank you
> >
> >
> > --
> > Regards,
> > Sandeep Palur
> > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > Department of Computer Science, Illinois Institute of Technology (IIT)
> > Phone : 312-647-9833
> > Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
> >
>
--
Regards,
Sandeep Palur
Data-Intensive Distributed Systems Laboratory, CS/IIT
Department of Computer Science, Illinois Institute of Technology (IIT)
Phone : 312-647-9833
Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
Re: One of two consumers is always Idle though I have 2 partitions
Posted by Jagat Singh <ja...@gmail.com>.
Are both of them in same Consumer Group?
On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep <ps...@hawk.iit.edu>
wrote:
> Dear Developers,
>
> I am 2nd year masters student at IIT. I am using Kafka for one of my
> research projects.My question is the following:
>
> 1. I have a producer, consumer and a broker(that contains 1st partition of
> my topic) on node1
> 2. I have a producer, consumer, zookeeper and a broker(that contains 2nd
> partition of my topic) on node2
> 3. Here comes my problem: though I have two partitions only one consumer
> pulls messages and the other one is always idle.
>
> What is that I can do to keep both of my consumer busy?
>
> Thank you
>
>
> --
> Regards,
> Sandeep Palur
> Data-Intensive Distributed Systems Laboratory, CS/IIT
> Department of Computer Science, Illinois Institute of Technology (IIT)
> Phone : 312-647-9833
> Email : psandeep@hawk.iit.edu <sr...@hawk.iit.edu>
>