You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Todd Gatts <tg...@us.ibm.com> on 2014/03/19 17:47:23 UTC
Can a producer detect when a topic has no consumers?
I'd like to avoid creating messages for a topic is the topic currently has
no consumers. Is there a way a producer can subscribe to the number of
consumers of a topic? Or if that's not possible, can a producer request
the current number of consumers of a topic?
The searchable hadoop kafka topic didn't return anything for "no consumer"
or "zero consumers", and a find on "zero" or "count" on the documentation
html page found lots of matches, but none about knowing the number of
consumers.
I'm certain I can code a solution around this problem, but I'd hate to
reinvent the wheel for such a straightforward problem.
Todd Gatts
Re: Can a producer detect when a topic has no consumers?
Posted by Ian Friedman <ia...@flurry.com>.
What Guozhang said is essentially what I was saying. I realized my code is not directly applicable to your problem as it only checks the partition offsets and not whether a consumer is actually consuming the partitions.
--Ian
On Mar 20, 2014, at 5:38 PM, Todd Gatts <tg...@us.ibm.com> wrote:
> Guozhang:
>
> Nice approach. I'll give that a try as well. Thanks.
>
> Todd
>
>
> Guozhang Wang ---2014-03-19 07:02:20 PM---Currently producers cannot detect if a topic is consumed by someone or not themselves. If you want t
>
> From: Guozhang Wang <wa...@gmail.com>
> To: "users@kafka.apache.org" <us...@kafka.apache.org>,
> Date: 2014-03-19 07:02 PM
> Subject: Re: Can a producer detect when a topic has no consumers?
>
>
>
> Currently producers cannot detect if a topic is consumed by someone or not
> themselves. If you want to know globally if a specific topic is consumed by
> anyone or not, you can query the ZK for all the consumer groups, and search
> if they contain the topic name.
>
> Guozhang
>
>
> On Wed, Mar 19, 2014 at 1:01 PM, Ian Friedman <ia...@flurry.com> wrote:
>
> > Hey Todd,
> >
> > For my purposes, I wrote a Partitioner that reads all of the
> > ConsumerOffsets for a particular group from Zookeeper, and then sends its
> > message to the one with least lag (with some probabilistic weighting thrown
> > in to prevent it from getting overwhelmed). I basically read and
> > reimplemented the code from ConsumerOffsetChecker.scala to do this. Does
> > that help at all? Would it help to see my code?
> >
> > --Ian
> >
> > On Mar 19, 2014, at 12:47 PM, Todd Gatts <tg...@us.ibm.com> wrote:
> >
> > >
> > >
> > > I'd like to avoid creating messages for a topic is the topic currently
> > has
> > > no consumers. Is there a way a producer can subscribe to the number of
> > > consumers of a topic? Or if that's not possible, can a producer request
> > > the current number of consumers of a topic?
> > >
> > > The searchable hadoop kafka topic didn't return anything for "no
> > consumer"
> > > or "zero consumers", and a find on "zero" or "count" on the documentation
> > > html page found lots of matches, but none about knowing the number of
> > > consumers.
> > >
> > > I'm certain I can code a solution around this problem, but I'd hate to
> > > reinvent the wheel for such a straightforward problem.
> > >
> > > Todd Gatts
> >
> >
>
>
> --
> -- Guozhang
>
Re: Can a producer detect when a topic has no consumers?
Posted by Todd Gatts <tg...@us.ibm.com>.
Guozhang:
Nice approach. I'll give that a try as well. Thanks.
Todd
From: Guozhang Wang <wa...@gmail.com>
To: "users@kafka.apache.org" <us...@kafka.apache.org>,
Date: 2014-03-19 07:02 PM
Subject: Re: Can a producer detect when a topic has no consumers?
Currently producers cannot detect if a topic is consumed by someone or not
themselves. If you want to know globally if a specific topic is consumed by
anyone or not, you can query the ZK for all the consumer groups, and search
if they contain the topic name.
Guozhang
On Wed, Mar 19, 2014 at 1:01 PM, Ian Friedman <ia...@flurry.com> wrote:
> Hey Todd,
>
> For my purposes, I wrote a Partitioner that reads all of the
> ConsumerOffsets for a particular group from Zookeeper, and then sends its
> message to the one with least lag (with some probabilistic weighting
thrown
> in to prevent it from getting overwhelmed). I basically read and
> reimplemented the code from ConsumerOffsetChecker.scala to do this. Does
> that help at all? Would it help to see my code?
>
> --Ian
>
> On Mar 19, 2014, at 12:47 PM, Todd Gatts <tg...@us.ibm.com> wrote:
>
> >
> >
> > I'd like to avoid creating messages for a topic is the topic currently
> has
> > no consumers. Is there a way a producer can subscribe to the number of
> > consumers of a topic? Or if that's not possible, can a producer
request
> > the current number of consumers of a topic?
> >
> > The searchable hadoop kafka topic didn't return anything for "no
> consumer"
> > or "zero consumers", and a find on "zero" or "count" on the
documentation
> > html page found lots of matches, but none about knowing the number of
> > consumers.
> >
> > I'm certain I can code a solution around this problem, but I'd hate to
> > reinvent the wheel for such a straightforward problem.
> >
> > Todd Gatts
>
>
--
-- Guozhang
Re: Can a producer detect when a topic has no consumers?
Posted by Guozhang Wang <wa...@gmail.com>.
Currently producers cannot detect if a topic is consumed by someone or not
themselves. If you want to know globally if a specific topic is consumed by
anyone or not, you can query the ZK for all the consumer groups, and search
if they contain the topic name.
Guozhang
On Wed, Mar 19, 2014 at 1:01 PM, Ian Friedman <ia...@flurry.com> wrote:
> Hey Todd,
>
> For my purposes, I wrote a Partitioner that reads all of the
> ConsumerOffsets for a particular group from Zookeeper, and then sends its
> message to the one with least lag (with some probabilistic weighting thrown
> in to prevent it from getting overwhelmed). I basically read and
> reimplemented the code from ConsumerOffsetChecker.scala to do this. Does
> that help at all? Would it help to see my code?
>
> --Ian
>
> On Mar 19, 2014, at 12:47 PM, Todd Gatts <tg...@us.ibm.com> wrote:
>
> >
> >
> > I'd like to avoid creating messages for a topic is the topic currently
> has
> > no consumers. Is there a way a producer can subscribe to the number of
> > consumers of a topic? Or if that's not possible, can a producer request
> > the current number of consumers of a topic?
> >
> > The searchable hadoop kafka topic didn't return anything for "no
> consumer"
> > or "zero consumers", and a find on "zero" or "count" on the documentation
> > html page found lots of matches, but none about knowing the number of
> > consumers.
> >
> > I'm certain I can code a solution around this problem, but I'd hate to
> > reinvent the wheel for such a straightforward problem.
> >
> > Todd Gatts
>
>
--
-- Guozhang
Re: Can a producer detect when a topic has no consumers?
Posted by Todd Gatts <tg...@us.ibm.com>.
Ian:
That's a nice way to look at the problem. Yes, I'd would welcome looking
at a snippet of your code.
Todd
From: Ian Friedman <ia...@flurry.com>
To: users@kafka.apache.org, Todd Gatts/Raleigh/IBM@IBMUS,
Date: 2014-03-19 04:02 PM
Subject: Re: Can a producer detect when a topic has no consumers?
Hey Todd,
For my purposes, I wrote a Partitioner that reads all of the
ConsumerOffsets for a particular group from Zookeeper, and then sends its
message to the one with least lag (with some probabilistic weighting thrown
in to prevent it from getting overwhelmed). I basically read and
reimplemented the code from ConsumerOffsetChecker.scala to do this. Does
that help at all? Would it help to see my code?
--Ian
On Mar 19, 2014, at 12:47 PM, Todd Gatts <tg...@us.ibm.com> wrote:
>
>
> I'd like to avoid creating messages for a topic is the topic currently
has
> no consumers. Is there a way a producer can subscribe to the number of
> consumers of a topic? Or if that's not possible, can a producer request
> the current number of consumers of a topic?
>
> The searchable hadoop kafka topic didn't return anything for "no
consumer"
> or "zero consumers", and a find on "zero" or "count" on the documentation
> html page found lots of matches, but none about knowing the number of
> consumers.
>
> I'm certain I can code a solution around this problem, but I'd hate to
> reinvent the wheel for such a straightforward problem.
>
> Todd Gatts
Re: Can a producer detect when a topic has no consumers?
Posted by Ian Friedman <ia...@flurry.com>.
Hey Todd,
For my purposes, I wrote a Partitioner that reads all of the ConsumerOffsets for a particular group from Zookeeper, and then sends its message to the one with least lag (with some probabilistic weighting thrown in to prevent it from getting overwhelmed). I basically read and reimplemented the code from ConsumerOffsetChecker.scala to do this. Does that help at all? Would it help to see my code?
--Ian
On Mar 19, 2014, at 12:47 PM, Todd Gatts <tg...@us.ibm.com> wrote:
>
>
> I'd like to avoid creating messages for a topic is the topic currently has
> no consumers. Is there a way a producer can subscribe to the number of
> consumers of a topic? Or if that's not possible, can a producer request
> the current number of consumers of a topic?
>
> The searchable hadoop kafka topic didn't return anything for "no consumer"
> or "zero consumers", and a find on "zero" or "count" on the documentation
> html page found lots of matches, but none about knowing the number of
> consumers.
>
> I'm certain I can code a solution around this problem, but I'd hate to
> reinvent the wheel for such a straightforward problem.
>
> Todd Gatts