You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Joris Peeters <j....@wintoncapital.com> on 2015/09/10 13:09:23 UTC
automatically consume from all topics
Hello,
Is there a simple way to set up a consumer that automatically picks up all the topics for all the partitions, dynamically extending its range as new topics get created?
The underlying idea is that we want to have a few over-arching consumers (I'm aware that's not great for the scalability, but that's not such a concern at present), to
- Gather various statistics, metrics, system pressure, ... and dispatch to the appropriate monitoring systems,
- Apply some end-to-end business-logic testing, to continuously assert certain expectations (e.g. "if this-sort-of message arrived, then we expect that-sort-of-message to be received within this time" etc).
I'm sure I can piece something together that does this, but perhaps it comes out of the box. (Couldn't find it, though).
We're using the Java client and Kafka 8.2.1.
Joris Peeters
Developer
Research and Data Technology
T:
+44 (0) 20 8576 5800
Winton
Grove House
27 Hammersmith Grove
London W6 0NE
wintoncapital.com<http://www.wintoncapital.com/>
[cid:image002.jpg@01D0EBC1.87503320]<http://www.wintoncapital.com/>
Winton Capital Management Limited ("Winton") is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States.
This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email.
This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results.
Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.
Re: automatically consume from all topics
Posted by Alexis Midon <al...@airbnb.com.INVALID>.
Thanks Tao.
for reference, here is the corresponding listener
https://github.com/apache/kafka/blob/9dbeb71ab258955e04b46991c1baf880b07633f4/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L1000-L1002
On Sat, Sep 12, 2015 at 5:42 AM, tao xiao <xi...@gmail.com> wrote:
> It is handled by the implementation of the high level consumer
> automatically. High level consumer internal registers an listener on ZK
> topic znode. Any changes to it including topic creation high level consumer
> will be notify and and then rebalanced topics if the new topic matches the
> regex. You can check the source code high level for reference
> On Sat, Sep 12, 2015 at 5:18 AM Alexis Midon
> <al...@airbnb.com.invalid> wrote:
>
> > When a new topic is created, I agree that the regex would remain
> unchanged
> > but how would an existing consumer be notified of the topic creation?
> > afaik there's no such notification mechanism in the High level consumer.
> >
> >
> >
> > On Thu, Sep 10, 2015 at 8:43 AM, tao xiao <xi...@gmail.com> wrote:
> >
> > > You can create message streams using regex that includes all topics.
> The
> > > beauty of regex is that any new topic created will be automatically
> > > consumed as long as the name of the topic matches the regex
> > >
> > > You check the method createMessageStreamsByFilter in high level API
> > >
> > > On Thu, Sep 10, 2015 at 11:03 PM Gerrit Jansen van Vuuren <
> > > gerritjvv@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm not sure about the high level consumer but I maintain a kafka
> > > consumer
> > > > that can add and remove topics dynamically.
> > > >
> > > > https://github.com/gerritjvv/kafka-fast
> > > > see
> > > >
> > > >
> > >
> >
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java
> > > > if you're using java/scala
> > > >
> > > >
> > > >
> > > > On Thu, Sep 10, 2015 at 3:48 PM, Helleren, Erik <
> > > > Erik.Helleren@cmegroup.com>
> > > > wrote:
> > > >
> > > > > So, there are several ways to do this. Lets assume the goal is to
> > add
> > > > > more topics to the application at runtime. And that this app is
> > > > currently
> > > > > written to be distributed via the magic of consumer groups.
> Sadly, I
> > > > > don¹t think the High level consumer is well designed for this
> > > particular
> > > > > use case. The app would have to poll using something like the
> topic
> > > list
> > > > > script (bin/kafka-topics.sh --list Š), close the existing high
> level
> > > > > consumer on a change, and start a new one. And then do this on all
> > the
> > > > > nodes of your application (Should be easier than doing it on just
> one
> > > > > actually). This would result in a huge latency spike and a problem
> > > when
> > > > > it comes to migrating the state involved in your example
> expectation.
> > > > >
> > > > > The next option still requires polling, but it needs a custom FT
> and
> > > > > distribution scheme. There might need a leader so some things only
> > > > happen
> > > > > once. Just use the simple consumer API, and have one thread per
> > > > > partition. The leader would have to tell a follower something like
> > > > ³Start
> > > > > listening to topic X, partition Y², which is risky and difficult to
> > do.
> > > > >
> > > > > The simplest option, assuming that each topic is independent when
> it
> > > > comes
> > > > > to expectations, is don¹t go with a cluster. Just have a
> > > script/watcher
> > > > > app that does the polling and then, when it detects new topics, for
> > > each
> > > > > new topic: start a new instance of your app on a new box that
> listens
> > > to
> > > > > that single topic. It might take a few seconds to startup, but its
> > > easy
> > > > > to code, easy to maintain, and easy to understand. Which makes
> for a
> > > > more
> > > > > resilient application.
> > > > > -Erik
> > > > >
> > > > >
> > > > >
> > > > > From: Joris Peeters <j....@wintoncapital.com>
> > > > > Reply-To: "users@kafka.apache.org" <us...@kafka.apache.org>
> > > > > Date: Thursday, September 10, 2015 at 6:09 AM
> > > > > To: "users@kafka.apache.org" <us...@kafka.apache.org>
> > > > > Subject: automatically consume from all topics
> > > > >
> > > > >
> > > > > Hello,
> > > > >
> > > > > Is there a simple way to set up a consumer that automatically picks
> > up
> > > > all
> > > > > the topics for all the partitions, dynamically extending its range
> as
> > > new
> > > > > topics get created?
> > > > >
> > > > > The underlying idea is that we want to have a few over-arching
> > > consumers
> > > > > (I¹m aware that¹s not great for the scalability, but that¹s not
> such
> > a
> > > > > concern at present), to
> > > > > -
> > > > > Gather various statistics, metrics, system pressure, Š and dispatch
> > to
> > > > the
> > > > > appropriate monitoring systems,
> > > > > -
> > > > > Apply some end-to-end business-logic testing, to continuously
> assert
> > > > > certain expectations (e.g. ³if this-sort-of message arrived, then
> we
> > > > > expect that-sort-of-message to be received within this time² etc).
> > > > >
> > > > >
> > > > > I¹m sure I can piece something together that does this, but perhaps
> > it
> > > > > comes out of the box. (Couldn¹t find it, though).
> > > > > We¹re using the Java client and Kafka 8.2.1.
> > > > >
> > > > > Joris Peeters
> > > > > Developer
> > > > >
> > > > > Research and Data Technology
> > > > > T:
> > > > > +44 (0) 20 8576 5800
> > > > >
> > > > > Winton
> > > > > Grove House
> > > > > 27 Hammersmith Grove
> > > > > London W6 0NE
> > > > >
> > > > > wintoncapital.com <http://www.wintoncapital.com/>
> > > > >
> > > > > <http://www.wintoncapital.com/>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Winton Capital Management Limited (³Winton²) is a limited company
> > > > > registered in England and Wales with its registered offices at 16
> Old
> > > > > Bailey, London, EC4M 7EG (Registered Company No. 3311531).
> > > > > Winton is authorised and regulated by the Financial Conduct
> > Authority
> > > in
> > > > > the United Kingdom, registered as an investment adviser with the US
> > > > > Securities and Exchange Commission, registered with the US
> Commodity
> > > > > Futures Trading Commission and a member of the
> > > > > National Futures Association in the United States.
> > > > > This communication, including any attachments, is confidential and
> > may
> > > be
> > > > > privileged. This email is for use by the intended recipient only.
> If
> > > you
> > > > > receive it in error, please notify the sender and
> > > > > delete it. You should not copy or disclose all or any part of this
> > > > email.
> > > > > This email does not constitute an offer or solicitation and nothing
> > > > > contained in this email constitutes, and should not be construed
> as,
> > > > > investment advice. Prospective investors should request offering
> > > > > materials and consult their own advisers with respect to
> investment
> > > > > decisions and inform themselves as to applicable legal
> requirements,
> > > > > exchange control regulations and taxes in the countries of their
> > > > > citizenship, residence or domicile. Past performance is
> > > > > not indicative of future results.
> > > > > Winton takes reasonable steps to ensure the accuracy and integrity
> of
> > > its
> > > > > communications, including emails. However Winton accepts no
> liability
> > > for
> > > > > any materials transmitted. Emails are not secure
> > > > > and cannot be guaranteed to be error free.
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: automatically consume from all topics
Posted by tao xiao <xi...@gmail.com>.
It is handled by the implementation of the high level consumer
automatically. High level consumer internal registers an listener on ZK
topic znode. Any changes to it including topic creation high level consumer
will be notify and and then rebalanced topics if the new topic matches the
regex. You can check the source code high level for reference
On Sat, Sep 12, 2015 at 5:18 AM Alexis Midon
<al...@airbnb.com.invalid> wrote:
> When a new topic is created, I agree that the regex would remain unchanged
> but how would an existing consumer be notified of the topic creation?
> afaik there's no such notification mechanism in the High level consumer.
>
>
>
> On Thu, Sep 10, 2015 at 8:43 AM, tao xiao <xi...@gmail.com> wrote:
>
> > You can create message streams using regex that includes all topics. The
> > beauty of regex is that any new topic created will be automatically
> > consumed as long as the name of the topic matches the regex
> >
> > You check the method createMessageStreamsByFilter in high level API
> >
> > On Thu, Sep 10, 2015 at 11:03 PM Gerrit Jansen van Vuuren <
> > gerritjvv@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I'm not sure about the high level consumer but I maintain a kafka
> > consumer
> > > that can add and remove topics dynamically.
> > >
> > > https://github.com/gerritjvv/kafka-fast
> > > see
> > >
> > >
> >
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java
> > > if you're using java/scala
> > >
> > >
> > >
> > > On Thu, Sep 10, 2015 at 3:48 PM, Helleren, Erik <
> > > Erik.Helleren@cmegroup.com>
> > > wrote:
> > >
> > > > So, there are several ways to do this. Lets assume the goal is to
> add
> > > > more topics to the application at runtime. And that this app is
> > > currently
> > > > written to be distributed via the magic of consumer groups. Sadly, I
> > > > don¹t think the High level consumer is well designed for this
> > particular
> > > > use case. The app would have to poll using something like the topic
> > list
> > > > script (bin/kafka-topics.sh --list Š), close the existing high level
> > > > consumer on a change, and start a new one. And then do this on all
> the
> > > > nodes of your application (Should be easier than doing it on just one
> > > > actually). This would result in a huge latency spike and a problem
> > when
> > > > it comes to migrating the state involved in your example expectation.
> > > >
> > > > The next option still requires polling, but it needs a custom FT and
> > > > distribution scheme. There might need a leader so some things only
> > > happen
> > > > once. Just use the simple consumer API, and have one thread per
> > > > partition. The leader would have to tell a follower something like
> > > ³Start
> > > > listening to topic X, partition Y², which is risky and difficult to
> do.
> > > >
> > > > The simplest option, assuming that each topic is independent when it
> > > comes
> > > > to expectations, is don¹t go with a cluster. Just have a
> > script/watcher
> > > > app that does the polling and then, when it detects new topics, for
> > each
> > > > new topic: start a new instance of your app on a new box that listens
> > to
> > > > that single topic. It might take a few seconds to startup, but its
> > easy
> > > > to code, easy to maintain, and easy to understand. Which makes for a
> > > more
> > > > resilient application.
> > > > -Erik
> > > >
> > > >
> > > >
> > > > From: Joris Peeters <j....@wintoncapital.com>
> > > > Reply-To: "users@kafka.apache.org" <us...@kafka.apache.org>
> > > > Date: Thursday, September 10, 2015 at 6:09 AM
> > > > To: "users@kafka.apache.org" <us...@kafka.apache.org>
> > > > Subject: automatically consume from all topics
> > > >
> > > >
> > > > Hello,
> > > >
> > > > Is there a simple way to set up a consumer that automatically picks
> up
> > > all
> > > > the topics for all the partitions, dynamically extending its range as
> > new
> > > > topics get created?
> > > >
> > > > The underlying idea is that we want to have a few over-arching
> > consumers
> > > > (I¹m aware that¹s not great for the scalability, but that¹s not such
> a
> > > > concern at present), to
> > > > -
> > > > Gather various statistics, metrics, system pressure, Š and dispatch
> to
> > > the
> > > > appropriate monitoring systems,
> > > > -
> > > > Apply some end-to-end business-logic testing, to continuously assert
> > > > certain expectations (e.g. ³if this-sort-of message arrived, then we
> > > > expect that-sort-of-message to be received within this time² etc).
> > > >
> > > >
> > > > I¹m sure I can piece something together that does this, but perhaps
> it
> > > > comes out of the box. (Couldn¹t find it, though).
> > > > We¹re using the Java client and Kafka 8.2.1.
> > > >
> > > > Joris Peeters
> > > > Developer
> > > >
> > > > Research and Data Technology
> > > > T:
> > > > +44 (0) 20 8576 5800
> > > >
> > > > Winton
> > > > Grove House
> > > > 27 Hammersmith Grove
> > > > London W6 0NE
> > > >
> > > > wintoncapital.com <http://www.wintoncapital.com/>
> > > >
> > > > <http://www.wintoncapital.com/>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Winton Capital Management Limited (³Winton²) is a limited company
> > > > registered in England and Wales with its registered offices at 16 Old
> > > > Bailey, London, EC4M 7EG (Registered Company No. 3311531).
> > > > Winton is authorised and regulated by the Financial Conduct
> Authority
> > in
> > > > the United Kingdom, registered as an investment adviser with the US
> > > > Securities and Exchange Commission, registered with the US Commodity
> > > > Futures Trading Commission and a member of the
> > > > National Futures Association in the United States.
> > > > This communication, including any attachments, is confidential and
> may
> > be
> > > > privileged. This email is for use by the intended recipient only. If
> > you
> > > > receive it in error, please notify the sender and
> > > > delete it. You should not copy or disclose all or any part of this
> > > email.
> > > > This email does not constitute an offer or solicitation and nothing
> > > > contained in this email constitutes, and should not be construed as,
> > > > investment advice. Prospective investors should request offering
> > > > materials and consult their own advisers with respect to investment
> > > > decisions and inform themselves as to applicable legal requirements,
> > > > exchange control regulations and taxes in the countries of their
> > > > citizenship, residence or domicile. Past performance is
> > > > not indicative of future results.
> > > > Winton takes reasonable steps to ensure the accuracy and integrity of
> > its
> > > > communications, including emails. However Winton accepts no liability
> > for
> > > > any materials transmitted. Emails are not secure
> > > > and cannot be guaranteed to be error free.
> > > >
> > > >
> > >
> >
>
Re: automatically consume from all topics
Posted by Alexis Midon <al...@airbnb.com.INVALID>.
When a new topic is created, I agree that the regex would remain unchanged
but how would an existing consumer be notified of the topic creation?
afaik there's no such notification mechanism in the High level consumer.
On Thu, Sep 10, 2015 at 8:43 AM, tao xiao <xi...@gmail.com> wrote:
> You can create message streams using regex that includes all topics. The
> beauty of regex is that any new topic created will be automatically
> consumed as long as the name of the topic matches the regex
>
> You check the method createMessageStreamsByFilter in high level API
>
> On Thu, Sep 10, 2015 at 11:03 PM Gerrit Jansen van Vuuren <
> gerritjvv@gmail.com> wrote:
>
> > Hi,
> >
> > I'm not sure about the high level consumer but I maintain a kafka
> consumer
> > that can add and remove topics dynamically.
> >
> > https://github.com/gerritjvv/kafka-fast
> > see
> >
> >
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java
> > if you're using java/scala
> >
> >
> >
> > On Thu, Sep 10, 2015 at 3:48 PM, Helleren, Erik <
> > Erik.Helleren@cmegroup.com>
> > wrote:
> >
> > > So, there are several ways to do this. Lets assume the goal is to add
> > > more topics to the application at runtime. And that this app is
> > currently
> > > written to be distributed via the magic of consumer groups. Sadly, I
> > > don¹t think the High level consumer is well designed for this
> particular
> > > use case. The app would have to poll using something like the topic
> list
> > > script (bin/kafka-topics.sh --list Š), close the existing high level
> > > consumer on a change, and start a new one. And then do this on all the
> > > nodes of your application (Should be easier than doing it on just one
> > > actually). This would result in a huge latency spike and a problem
> when
> > > it comes to migrating the state involved in your example expectation.
> > >
> > > The next option still requires polling, but it needs a custom FT and
> > > distribution scheme. There might need a leader so some things only
> > happen
> > > once. Just use the simple consumer API, and have one thread per
> > > partition. The leader would have to tell a follower something like
> > ³Start
> > > listening to topic X, partition Y², which is risky and difficult to do.
> > >
> > > The simplest option, assuming that each topic is independent when it
> > comes
> > > to expectations, is don¹t go with a cluster. Just have a
> script/watcher
> > > app that does the polling and then, when it detects new topics, for
> each
> > > new topic: start a new instance of your app on a new box that listens
> to
> > > that single topic. It might take a few seconds to startup, but its
> easy
> > > to code, easy to maintain, and easy to understand. Which makes for a
> > more
> > > resilient application.
> > > -Erik
> > >
> > >
> > >
> > > From: Joris Peeters <j....@wintoncapital.com>
> > > Reply-To: "users@kafka.apache.org" <us...@kafka.apache.org>
> > > Date: Thursday, September 10, 2015 at 6:09 AM
> > > To: "users@kafka.apache.org" <us...@kafka.apache.org>
> > > Subject: automatically consume from all topics
> > >
> > >
> > > Hello,
> > >
> > > Is there a simple way to set up a consumer that automatically picks up
> > all
> > > the topics for all the partitions, dynamically extending its range as
> new
> > > topics get created?
> > >
> > > The underlying idea is that we want to have a few over-arching
> consumers
> > > (I¹m aware that¹s not great for the scalability, but that¹s not such a
> > > concern at present), to
> > > -
> > > Gather various statistics, metrics, system pressure, Š and dispatch to
> > the
> > > appropriate monitoring systems,
> > > -
> > > Apply some end-to-end business-logic testing, to continuously assert
> > > certain expectations (e.g. ³if this-sort-of message arrived, then we
> > > expect that-sort-of-message to be received within this time² etc).
> > >
> > >
> > > I¹m sure I can piece something together that does this, but perhaps it
> > > comes out of the box. (Couldn¹t find it, though).
> > > We¹re using the Java client and Kafka 8.2.1.
> > >
> > > Joris Peeters
> > > Developer
> > >
> > > Research and Data Technology
> > > T:
> > > +44 (0) 20 8576 5800
> > >
> > > Winton
> > > Grove House
> > > 27 Hammersmith Grove
> > > London W6 0NE
> > >
> > > wintoncapital.com <http://www.wintoncapital.com/>
> > >
> > > <http://www.wintoncapital.com/>
> > >
> > >
> > >
> > >
> > >
> > > Winton Capital Management Limited (³Winton²) is a limited company
> > > registered in England and Wales with its registered offices at 16 Old
> > > Bailey, London, EC4M 7EG (Registered Company No. 3311531).
> > > Winton is authorised and regulated by the Financial Conduct Authority
> in
> > > the United Kingdom, registered as an investment adviser with the US
> > > Securities and Exchange Commission, registered with the US Commodity
> > > Futures Trading Commission and a member of the
> > > National Futures Association in the United States.
> > > This communication, including any attachments, is confidential and may
> be
> > > privileged. This email is for use by the intended recipient only. If
> you
> > > receive it in error, please notify the sender and
> > > delete it. You should not copy or disclose all or any part of this
> > email.
> > > This email does not constitute an offer or solicitation and nothing
> > > contained in this email constitutes, and should not be construed as,
> > > investment advice. Prospective investors should request offering
> > > materials and consult their own advisers with respect to investment
> > > decisions and inform themselves as to applicable legal requirements,
> > > exchange control regulations and taxes in the countries of their
> > > citizenship, residence or domicile. Past performance is
> > > not indicative of future results.
> > > Winton takes reasonable steps to ensure the accuracy and integrity of
> its
> > > communications, including emails. However Winton accepts no liability
> for
> > > any materials transmitted. Emails are not secure
> > > and cannot be guaranteed to be error free.
> > >
> > >
> >
>
Re: automatically consume from all topics
Posted by tao xiao <xi...@gmail.com>.
You can create message streams using regex that includes all topics. The
beauty of regex is that any new topic created will be automatically
consumed as long as the name of the topic matches the regex
You check the method createMessageStreamsByFilter in high level API
On Thu, Sep 10, 2015 at 11:03 PM Gerrit Jansen van Vuuren <
gerritjvv@gmail.com> wrote:
> Hi,
>
> I'm not sure about the high level consumer but I maintain a kafka consumer
> that can add and remove topics dynamically.
>
> https://github.com/gerritjvv/kafka-fast
> see
>
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java
> if you're using java/scala
>
>
>
> On Thu, Sep 10, 2015 at 3:48 PM, Helleren, Erik <
> Erik.Helleren@cmegroup.com>
> wrote:
>
> > So, there are several ways to do this. Lets assume the goal is to add
> > more topics to the application at runtime. And that this app is
> currently
> > written to be distributed via the magic of consumer groups. Sadly, I
> > don¹t think the High level consumer is well designed for this particular
> > use case. The app would have to poll using something like the topic list
> > script (bin/kafka-topics.sh --list Š), close the existing high level
> > consumer on a change, and start a new one. And then do this on all the
> > nodes of your application (Should be easier than doing it on just one
> > actually). This would result in a huge latency spike and a problem when
> > it comes to migrating the state involved in your example expectation.
> >
> > The next option still requires polling, but it needs a custom FT and
> > distribution scheme. There might need a leader so some things only
> happen
> > once. Just use the simple consumer API, and have one thread per
> > partition. The leader would have to tell a follower something like
> ³Start
> > listening to topic X, partition Y², which is risky and difficult to do.
> >
> > The simplest option, assuming that each topic is independent when it
> comes
> > to expectations, is don¹t go with a cluster. Just have a script/watcher
> > app that does the polling and then, when it detects new topics, for each
> > new topic: start a new instance of your app on a new box that listens to
> > that single topic. It might take a few seconds to startup, but its easy
> > to code, easy to maintain, and easy to understand. Which makes for a
> more
> > resilient application.
> > -Erik
> >
> >
> >
> > From: Joris Peeters <j....@wintoncapital.com>
> > Reply-To: "users@kafka.apache.org" <us...@kafka.apache.org>
> > Date: Thursday, September 10, 2015 at 6:09 AM
> > To: "users@kafka.apache.org" <us...@kafka.apache.org>
> > Subject: automatically consume from all topics
> >
> >
> > Hello,
> >
> > Is there a simple way to set up a consumer that automatically picks up
> all
> > the topics for all the partitions, dynamically extending its range as new
> > topics get created?
> >
> > The underlying idea is that we want to have a few over-arching consumers
> > (I¹m aware that¹s not great for the scalability, but that¹s not such a
> > concern at present), to
> > -
> > Gather various statistics, metrics, system pressure, Š and dispatch to
> the
> > appropriate monitoring systems,
> > -
> > Apply some end-to-end business-logic testing, to continuously assert
> > certain expectations (e.g. ³if this-sort-of message arrived, then we
> > expect that-sort-of-message to be received within this time² etc).
> >
> >
> > I¹m sure I can piece something together that does this, but perhaps it
> > comes out of the box. (Couldn¹t find it, though).
> > We¹re using the Java client and Kafka 8.2.1.
> >
> > Joris Peeters
> > Developer
> >
> > Research and Data Technology
> > T:
> > +44 (0) 20 8576 5800
> >
> > Winton
> > Grove House
> > 27 Hammersmith Grove
> > London W6 0NE
> >
> > wintoncapital.com <http://www.wintoncapital.com/>
> >
> > <http://www.wintoncapital.com/>
> >
> >
> >
> >
> >
> > Winton Capital Management Limited (³Winton²) is a limited company
> > registered in England and Wales with its registered offices at 16 Old
> > Bailey, London, EC4M 7EG (Registered Company No. 3311531).
> > Winton is authorised and regulated by the Financial Conduct Authority in
> > the United Kingdom, registered as an investment adviser with the US
> > Securities and Exchange Commission, registered with the US Commodity
> > Futures Trading Commission and a member of the
> > National Futures Association in the United States.
> > This communication, including any attachments, is confidential and may be
> > privileged. This email is for use by the intended recipient only. If you
> > receive it in error, please notify the sender and
> > delete it. You should not copy or disclose all or any part of this
> email.
> > This email does not constitute an offer or solicitation and nothing
> > contained in this email constitutes, and should not be construed as,
> > investment advice. Prospective investors should request offering
> > materials and consult their own advisers with respect to investment
> > decisions and inform themselves as to applicable legal requirements,
> > exchange control regulations and taxes in the countries of their
> > citizenship, residence or domicile. Past performance is
> > not indicative of future results.
> > Winton takes reasonable steps to ensure the accuracy and integrity of its
> > communications, including emails. However Winton accepts no liability for
> > any materials transmitted. Emails are not secure
> > and cannot be guaranteed to be error free.
> >
> >
>
Re: automatically consume from all topics
Posted by Gerrit Jansen van Vuuren <ge...@gmail.com>.
Hi,
I'm not sure about the high level consumer but I maintain a kafka consumer
that can add and remove topics dynamically.
https://github.com/gerritjvv/kafka-fast
see
https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java
if you're using java/scala
On Thu, Sep 10, 2015 at 3:48 PM, Helleren, Erik <Er...@cmegroup.com>
wrote:
> So, there are several ways to do this. Lets assume the goal is to add
> more topics to the application at runtime. And that this app is currently
> written to be distributed via the magic of consumer groups. Sadly, I
> don¹t think the High level consumer is well designed for this particular
> use case. The app would have to poll using something like the topic list
> script (bin/kafka-topics.sh --list Š), close the existing high level
> consumer on a change, and start a new one. And then do this on all the
> nodes of your application (Should be easier than doing it on just one
> actually). This would result in a huge latency spike and a problem when
> it comes to migrating the state involved in your example expectation.
>
> The next option still requires polling, but it needs a custom FT and
> distribution scheme. There might need a leader so some things only happen
> once. Just use the simple consumer API, and have one thread per
> partition. The leader would have to tell a follower something like ³Start
> listening to topic X, partition Y², which is risky and difficult to do.
>
> The simplest option, assuming that each topic is independent when it comes
> to expectations, is don¹t go with a cluster. Just have a script/watcher
> app that does the polling and then, when it detects new topics, for each
> new topic: start a new instance of your app on a new box that listens to
> that single topic. It might take a few seconds to startup, but its easy
> to code, easy to maintain, and easy to understand. Which makes for a more
> resilient application.
> -Erik
>
>
>
> From: Joris Peeters <j....@wintoncapital.com>
> Reply-To: "users@kafka.apache.org" <us...@kafka.apache.org>
> Date: Thursday, September 10, 2015 at 6:09 AM
> To: "users@kafka.apache.org" <us...@kafka.apache.org>
> Subject: automatically consume from all topics
>
>
> Hello,
>
> Is there a simple way to set up a consumer that automatically picks up all
> the topics for all the partitions, dynamically extending its range as new
> topics get created?
>
> The underlying idea is that we want to have a few over-arching consumers
> (I¹m aware that¹s not great for the scalability, but that¹s not such a
> concern at present), to
> -
> Gather various statistics, metrics, system pressure, Š and dispatch to the
> appropriate monitoring systems,
> -
> Apply some end-to-end business-logic testing, to continuously assert
> certain expectations (e.g. ³if this-sort-of message arrived, then we
> expect that-sort-of-message to be received within this time² etc).
>
>
> I¹m sure I can piece something together that does this, but perhaps it
> comes out of the box. (Couldn¹t find it, though).
> We¹re using the Java client and Kafka 8.2.1.
>
> Joris Peeters
> Developer
>
> Research and Data Technology
> T:
> +44 (0) 20 8576 5800
>
> Winton
> Grove House
> 27 Hammersmith Grove
> London W6 0NE
>
> wintoncapital.com <http://www.wintoncapital.com/>
>
> <http://www.wintoncapital.com/>
>
>
>
>
>
> Winton Capital Management Limited (³Winton²) is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531).
> Winton is authorised and regulated by the Financial Conduct Authority in
> the United Kingdom, registered as an investment adviser with the US
> Securities and Exchange Commission, registered with the US Commodity
> Futures Trading Commission and a member of the
> National Futures Association in the United States.
> This communication, including any attachments, is confidential and may be
> privileged. This email is for use by the intended recipient only. If you
> receive it in error, please notify the sender and
> delete it. You should not copy or disclose all or any part of this email.
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering
> materials and consult their own advisers with respect to investment
> decisions and inform themselves as to applicable legal requirements,
> exchange control regulations and taxes in the countries of their
> citizenship, residence or domicile. Past performance is
> not indicative of future results.
> Winton takes reasonable steps to ensure the accuracy and integrity of its
> communications, including emails. However Winton accepts no liability for
> any materials transmitted. Emails are not secure
> and cannot be guaranteed to be error free.
>
>
Re: automatically consume from all topics
Posted by "Helleren, Erik" <Er...@cmegroup.com>.
So, there are several ways to do this. Lets assume the goal is to add
more topics to the application at runtime. And that this app is currently
written to be distributed via the magic of consumer groups. Sadly, I
don¹t think the High level consumer is well designed for this particular
use case. The app would have to poll using something like the topic list
script (bin/kafka-topics.sh --list Š), close the existing high level
consumer on a change, and start a new one. And then do this on all the
nodes of your application (Should be easier than doing it on just one
actually). This would result in a huge latency spike and a problem when
it comes to migrating the state involved in your example expectation.
The next option still requires polling, but it needs a custom FT and
distribution scheme. There might need a leader so some things only happen
once. Just use the simple consumer API, and have one thread per
partition. The leader would have to tell a follower something like ³Start
listening to topic X, partition Y², which is risky and difficult to do.
The simplest option, assuming that each topic is independent when it comes
to expectations, is don¹t go with a cluster. Just have a script/watcher
app that does the polling and then, when it detects new topics, for each
new topic: start a new instance of your app on a new box that listens to
that single topic. It might take a few seconds to startup, but its easy
to code, easy to maintain, and easy to understand. Which makes for a more
resilient application.
-Erik
From: Joris Peeters <j....@wintoncapital.com>
Reply-To: "users@kafka.apache.org" <us...@kafka.apache.org>
Date: Thursday, September 10, 2015 at 6:09 AM
To: "users@kafka.apache.org" <us...@kafka.apache.org>
Subject: automatically consume from all topics
Hello,
Is there a simple way to set up a consumer that automatically picks up all
the topics for all the partitions, dynamically extending its range as new
topics get created?
The underlying idea is that we want to have a few over-arching consumers
(I¹m aware that¹s not great for the scalability, but that¹s not such a
concern at present), to
-
Gather various statistics, metrics, system pressure, Š and dispatch to the
appropriate monitoring systems,
-
Apply some end-to-end business-logic testing, to continuously assert
certain expectations (e.g. ³if this-sort-of message arrived, then we
expect that-sort-of-message to be received within this time² etc).
I¹m sure I can piece something together that does this, but perhaps it
comes out of the box. (Couldn¹t find it, though).
We¹re using the Java client and Kafka 8.2.1.
Joris Peeters
Developer
Research and Data Technology
T:
+44 (0) 20 8576 5800
Winton
Grove House
27 Hammersmith Grove
London W6 0NE
wintoncapital.com <http://www.wintoncapital.com/>
<http://www.wintoncapital.com/>
Winton Capital Management Limited (³Winton²) is a limited company
registered in England and Wales with its registered offices at 16 Old
Bailey, London, EC4M 7EG (Registered Company No. 3311531).
Winton is authorised and regulated by the Financial Conduct Authority in
the United Kingdom, registered as an investment adviser with the US
Securities and Exchange Commission, registered with the US Commodity
Futures Trading Commission and a member of the
National Futures Association in the United States.
This communication, including any attachments, is confidential and may be
privileged. This email is for use by the intended recipient only. If you
receive it in error, please notify the sender and
delete it. You should not copy or disclose all or any part of this email.
This email does not constitute an offer or solicitation and nothing
contained in this email constitutes, and should not be construed as,
investment advice. Prospective investors should request offering
materials and consult their own advisers with respect to investment
decisions and inform themselves as to applicable legal requirements,
exchange control regulations and taxes in the countries of their
citizenship, residence or domicile. Past performance is
not indicative of future results.
Winton takes reasonable steps to ensure the accuracy and integrity of its
communications, including emails. However Winton accepts no liability for
any materials transmitted. Emails are not secure
and cannot be guaranteed to be error free.