You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Joris Peeters <j....@wintoncapital.com> on 2015/09/16 17:05:20 UTC
topics, partitions, consumergroups
I intend to write some bespoke monitoring for our internal kafka system. Amongst others, it should give us warning if some topics are lagging behind too much (i.e. production much faster than consumption).
What would currently be the neatest way to get a list of all topics, partitions and consumergroups? It's the latter that I'm most unsure about.
I imagine I could actually consume the "__consumer_offsets" topic (we're using the offsets-are-committed-in-kafka approach), but perhaps there is a neater api exposed. I couldn't immediately find it, though.
(I am working in Java/Scala, with the latest official java kafka client).
As always, any help would be appreciated!
-J
Winton Capital Management Limited ("Winton") is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States.
This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email.
This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results.
Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.
RE: topics, partitions, consumergroups
Posted by Joris Peeters <j....@wintoncapital.com>.
Oh great! Thanks for that tip - that looks exactly like what I described. Surprised/embarrassed I didn't find that myself whilst searching.
-----Original Message-----
From: tao xiao [mailto:xiaotao183@gmail.com]
Sent: 17 September 2015 02:20
To: users@kafka.apache.org
Subject: Re: topics, partitions, consumergroups
Joris,
You checkout Burrow https://github.com/linkedin/Burrow which gives you monitoring and alerting capability for offset stored in Kafka
On Wed, 16 Sep 2015 at 23:05 Joris Peeters <j....@wintoncapital.com>
wrote:
> I intend to write some bespoke monitoring for our internal kafka system.
> Amongst others, it should give us warning if some topics are lagging
> behind too much (i.e. production much faster than consumption).
>
> What would currently be the neatest way to get a list of all topics,
> partitions and consumergroups? It's the latter that I'm most unsure about.
> I imagine I could actually consume the "__consumer_offsets" topic
> (we're using the offsets-are-committed-in-kafka approach), but perhaps
> there is a neater api exposed. I couldn't immediately find it, though.
>
> (I am working in Java/Scala, with the latest official java kafka client).
>
> As always, any help would be appreciated!
> -J
>
>
>
> Winton Capital Management Limited ("Winton") is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the
> United Kingdom, registered as an investment adviser with the US
> Securities and Exchange Commission, registered with the US Commodity
> Futures Trading Commission and a member of the National Futures
> Association in the United States.
>
> This communication, including any attachments, is confidential and may
> be privileged. This email is for use by the intended recipient only.
> If you receive it in error, please notify the sender and delete it.
> You should not copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering
> materials and consult their own advisers with respect to investment
> decisions and inform themselves as to applicable legal requirements,
> exchange control regulations and taxes in the countries of their
> citizenship, residence or domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of
> its communications, including emails. However Winton accepts no
> liability for any materials transmitted. Emails are not secure and
> cannot be guaranteed to be error free.
>
Winton Capital Management Limited (“Winton”) is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States.
This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email.
This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results.
Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.
Re: topics, partitions, consumergroups
Posted by tao xiao <xi...@gmail.com>.
Joris,
You checkout Burrow https://github.com/linkedin/Burrow which gives you
monitoring and alerting capability for offset stored in Kafka
On Wed, 16 Sep 2015 at 23:05 Joris Peeters <j....@wintoncapital.com>
wrote:
> I intend to write some bespoke monitoring for our internal kafka system.
> Amongst others, it should give us warning if some topics are lagging behind
> too much (i.e. production much faster than consumption).
>
> What would currently be the neatest way to get a list of all topics,
> partitions and consumergroups? It's the latter that I'm most unsure about.
> I imagine I could actually consume the "__consumer_offsets" topic (we're
> using the offsets-are-committed-in-kafka approach), but perhaps there is a
> neater api exposed. I couldn't immediately find it, though.
>
> (I am working in Java/Scala, with the latest official java kafka client).
>
> As always, any help would be appreciated!
> -J
>
>
>
> Winton Capital Management Limited ("Winton") is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the United
> Kingdom, registered as an investment adviser with the US Securities and
> Exchange Commission, registered with the US Commodity Futures Trading
> Commission and a member of the National Futures Association in the United
> States.
>
> This communication, including any attachments, is confidential and may be
> privileged. This email is for use by the intended recipient only. If you
> receive it in error, please notify the sender and delete it. You should not
> copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering materials
> and consult their own advisers with respect to investment decisions and
> inform themselves as to applicable legal requirements, exchange control
> regulations and taxes in the countries of their citizenship, residence or
> domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of its
> communications, including emails. However Winton accepts no liability for
> any materials transmitted. Emails are not secure and cannot be guaranteed
> to be error free.
>