You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Joris Peeters <j....@wintoncapital.com> on 2015/09/23 10:17:16 UTC
high level consumer timeout?
I have a topic with three partitions, to which I send 5 messages (very rapidly after one another) and they get partitioned well ([1,2,2] as it happens).
I'm launching three identical high level java customers (kafka 8.2.1), single-threaded, to consume those messages. Once a message is received, the consumers have to do a fair amount of work, which typically takes a few minutes. Right now I'm committing the offset after every message has been completed (so I have auto.commit.enable to false *).
What I am seeing, with this setup, is that the three consumers each work through their first message and then just hang (in the subsequent "it.hasNext()"). The remaining two messages do not arrive. When, on the other hand, I set auto.commit.enable to true, everything goes well and the three consumers churn through the five messages without trouble.
This makes me suspect there is some sort of heartbeat issue going on, that is triggered by the auto.commit? We don't want to auto-commit in this case, though, so does anyone know of a different solution that could be tried?
My situation sounds somewhat similar to http://grokbase.com/t/kafka/users/14ck70w38q/kafka-consumer-session-timeouts - but I am not sure if anyone got to the bottom of that.
As always, any help would be much appreciated!
-Joris.
(*) These are the consumer properties I have set explicitly:
props.put("auto.commit.interval.ms", "1000");
props.put("auto.commit.enable", "false"); // or true.
props.put("offsets.storage", "kafka");
Winton Capital Management Limited ("Winton") is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States.
This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email.
This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results.
Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.
Re: high level consumer timeout?
Posted by noah <ia...@gmail.com>.
I've found that (especially in tests) I can get a ConsumerIterator, but
that the consumer is not actually connected and settled onto its partitions
yet. Unfortunately, I haven't found a good way to establish that aside from
sending pilot messages through first. smallest is easier :)
On Wed, Sep 23, 2015 at 4:26 AM Joris Peeters <j....@wintoncapital.com>
wrote:
> Looks like you are right, although I am not sure I understand why. :)
>
> The outlined scenario is indeed part of a test suite, which goes through
> the following stages (with some temporal separation between each),
>
> 1/ Create a new topic (with some guid in the name), with three partitions
> 2/ Launch the three consumers (all part of the same consumer group, which
> is fixed at compile time).
> 3/ Fire the five messages
>
> Adding 'props.put("auto.offset.reset", "smallest");' to my list of
> settings seems to resolve the hanging issue I mentioned.
>
> The line
> >>
> If this is set to largest, the consumer may lose some messages when the
> number of partitions, for the topics it subscribes to, changes on the
> broker. To prevent data loss during partition addition, set
> auto.offset.reset to smallest"
> <<
>
> seems to suggest that problems would only occur if the number of
> partitions changes on the broker, but that is set to (and stays at) 3.
> Whilst I'm quite interested in finding out what is going on here, I'm in
> any case quite thrilled that at least it seems to work now. :) Thanks!
> -J
>
> -----Original Message-----
> From: noah [mailto:iamnoah@gmail.com]
> Sent: 23 September 2015 09:44
> To: users@kafka.apache.org
> Subject: Re: high level consumer timeout?
>
> Assuming this is a test case with a new topic/consumer groups for each
> run, do you have auto.offset.reset=smallest? This happens to me constantly
> in tests because my consumers end up missing the first message since the
> default is largest (in which case auto commit is a red herring.)
>
> On Wed, Sep 23, 2015 at 3:17 AM Joris Peeters <j.peeters@wintoncapital.com
> >
> wrote:
>
> > I have a topic with three partitions, to which I send 5 messages (very
> > rapidly after one another) and they get partitioned well ([1,2,2] as
> > it happens).
> > I'm launching three identical high level java customers (kafka 8.2.1),
> > single-threaded, to consume those messages. Once a message is
> > received, the consumers have to do a fair amount of work, which
> > typically takes a few minutes. Right now I'm committing the offset
> > after every message has been completed (so I have auto.commit.enable to
> false *).
> >
> > What I am seeing, with this setup, is that the three consumers each
> > work through their first message and then just hang (in the subsequent
> > "it.hasNext()"). The remaining two messages do not arrive. When, on
> > the other hand, I set auto.commit.enable to true, everything goes well
> > and the three consumers churn through the five messages without trouble.
> >
> > This makes me suspect there is some sort of heartbeat issue going on,
> > that is triggered by the auto.commit? We don't want to auto-commit in
> > this case, though, so does anyone know of a different solution that
> could be tried?
> >
> > My situation sounds somewhat similar to
> > http://grokbase.com/t/kafka/users/14ck70w38q/kafka-consumer-session-ti
> > meouts
> > - but I am not sure if anyone got to the bottom of that.
> >
> > As always, any help would be much appreciated!
> > -Joris.
> >
> >
> > (*) These are the consumer properties I have set explicitly:
> >
> > props.put("auto.commit.interval.ms", "1000");
> > props.put("auto.commit.enable", "false"); // or true.
> > props.put("offsets.storage", "kafka");
> >
> >
> >
> >
> > Winton Capital Management Limited ("Winton") is a limited company
> > registered in England and Wales with its registered offices at 16 Old
> > Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> > authorised and regulated by the Financial Conduct Authority in the
> > United Kingdom, registered as an investment adviser with the US
> > Securities and Exchange Commission, registered with the US Commodity
> > Futures Trading Commission and a member of the National Futures
> > Association in the United States.
> >
> > This communication, including any attachments, is confidential and may
> > be privileged. This email is for use by the intended recipient only.
> > If you receive it in error, please notify the sender and delete it.
> > You should not copy or disclose all or any part of this email.
> >
> > This email does not constitute an offer or solicitation and nothing
> > contained in this email constitutes, and should not be construed as,
> > investment advice. Prospective investors should request offering
> > materials and consult their own advisers with respect to investment
> > decisions and inform themselves as to applicable legal requirements,
> > exchange control regulations and taxes in the countries of their
> > citizenship, residence or domicile. Past performance is not indicative
> of future results.
> >
> > Winton takes reasonable steps to ensure the accuracy and integrity of
> > its communications, including emails. However Winton accepts no
> > liability for any materials transmitted. Emails are not secure and
> > cannot be guaranteed to be error free.
> >
>
>
> Winton Capital Management Limited (“Winton”) is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the United
> Kingdom, registered as an investment adviser with the US Securities and
> Exchange Commission, registered with the US Commodity Futures Trading
> Commission and a member of the National Futures Association in the United
> States.
>
> This communication, including any attachments, is confidential and may be
> privileged. This email is for use by the intended recipient only. If you
> receive it in error, please notify the sender and delete it. You should not
> copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering materials
> and consult their own advisers with respect to investment decisions and
> inform themselves as to applicable legal requirements, exchange control
> regulations and taxes in the countries of their citizenship, residence or
> domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of its
> communications, including emails. However Winton accepts no liability for
> any materials transmitted. Emails are not secure and cannot be guaranteed
> to be error free.
>
RE: high level consumer timeout?
Posted by Joris Peeters <j....@wintoncapital.com>.
Looks like you are right, although I am not sure I understand why. :)
The outlined scenario is indeed part of a test suite, which goes through the following stages (with some temporal separation between each),
1/ Create a new topic (with some guid in the name), with three partitions
2/ Launch the three consumers (all part of the same consumer group, which is fixed at compile time).
3/ Fire the five messages
Adding 'props.put("auto.offset.reset", "smallest");' to my list of settings seems to resolve the hanging issue I mentioned.
The line
>>
If this is set to largest, the consumer may lose some messages when the number of partitions, for the topics it subscribes to, changes on the broker. To prevent data loss during partition addition, set auto.offset.reset to smallest"
<<
seems to suggest that problems would only occur if the number of partitions changes on the broker, but that is set to (and stays at) 3.
Whilst I'm quite interested in finding out what is going on here, I'm in any case quite thrilled that at least it seems to work now. :) Thanks!
-J
-----Original Message-----
From: noah [mailto:iamnoah@gmail.com]
Sent: 23 September 2015 09:44
To: users@kafka.apache.org
Subject: Re: high level consumer timeout?
Assuming this is a test case with a new topic/consumer groups for each run, do you have auto.offset.reset=smallest? This happens to me constantly in tests because my consumers end up missing the first message since the default is largest (in which case auto commit is a red herring.)
On Wed, Sep 23, 2015 at 3:17 AM Joris Peeters <j....@wintoncapital.com>
wrote:
> I have a topic with three partitions, to which I send 5 messages (very
> rapidly after one another) and they get partitioned well ([1,2,2] as
> it happens).
> I'm launching three identical high level java customers (kafka 8.2.1),
> single-threaded, to consume those messages. Once a message is
> received, the consumers have to do a fair amount of work, which
> typically takes a few minutes. Right now I'm committing the offset
> after every message has been completed (so I have auto.commit.enable to false *).
>
> What I am seeing, with this setup, is that the three consumers each
> work through their first message and then just hang (in the subsequent
> "it.hasNext()"). The remaining two messages do not arrive. When, on
> the other hand, I set auto.commit.enable to true, everything goes well
> and the three consumers churn through the five messages without trouble.
>
> This makes me suspect there is some sort of heartbeat issue going on,
> that is triggered by the auto.commit? We don't want to auto-commit in
> this case, though, so does anyone know of a different solution that could be tried?
>
> My situation sounds somewhat similar to
> http://grokbase.com/t/kafka/users/14ck70w38q/kafka-consumer-session-ti
> meouts
> - but I am not sure if anyone got to the bottom of that.
>
> As always, any help would be much appreciated!
> -Joris.
>
>
> (*) These are the consumer properties I have set explicitly:
>
> props.put("auto.commit.interval.ms", "1000");
> props.put("auto.commit.enable", "false"); // or true.
> props.put("offsets.storage", "kafka");
>
>
>
>
> Winton Capital Management Limited ("Winton") is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the
> United Kingdom, registered as an investment adviser with the US
> Securities and Exchange Commission, registered with the US Commodity
> Futures Trading Commission and a member of the National Futures
> Association in the United States.
>
> This communication, including any attachments, is confidential and may
> be privileged. This email is for use by the intended recipient only.
> If you receive it in error, please notify the sender and delete it.
> You should not copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering
> materials and consult their own advisers with respect to investment
> decisions and inform themselves as to applicable legal requirements,
> exchange control regulations and taxes in the countries of their
> citizenship, residence or domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of
> its communications, including emails. However Winton accepts no
> liability for any materials transmitted. Emails are not secure and
> cannot be guaranteed to be error free.
>
Winton Capital Management Limited (“Winton”) is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States.
This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email.
This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results.
Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.
Re: high level consumer timeout?
Posted by noah <ia...@gmail.com>.
Assuming this is a test case with a new topic/consumer groups for each run,
do you have auto.offset.reset=smallest? This happens to me constantly in
tests because my consumers end up missing the first message since the
default is largest (in which case auto commit is a red herring.)
On Wed, Sep 23, 2015 at 3:17 AM Joris Peeters <j....@wintoncapital.com>
wrote:
> I have a topic with three partitions, to which I send 5 messages (very
> rapidly after one another) and they get partitioned well ([1,2,2] as it
> happens).
> I'm launching three identical high level java customers (kafka 8.2.1),
> single-threaded, to consume those messages. Once a message is received, the
> consumers have to do a fair amount of work, which typically takes a few
> minutes. Right now I'm committing the offset after every message has been
> completed (so I have auto.commit.enable to false *).
>
> What I am seeing, with this setup, is that the three consumers each work
> through their first message and then just hang (in the subsequent
> "it.hasNext()"). The remaining two messages do not arrive. When, on the
> other hand, I set auto.commit.enable to true, everything goes well and the
> three consumers churn through the five messages without trouble.
>
> This makes me suspect there is some sort of heartbeat issue going on, that
> is triggered by the auto.commit? We don't want to auto-commit in this case,
> though, so does anyone know of a different solution that could be tried?
>
> My situation sounds somewhat similar to
> http://grokbase.com/t/kafka/users/14ck70w38q/kafka-consumer-session-timeouts
> - but I am not sure if anyone got to the bottom of that.
>
> As always, any help would be much appreciated!
> -Joris.
>
>
> (*) These are the consumer properties I have set explicitly:
>
> props.put("auto.commit.interval.ms", "1000");
> props.put("auto.commit.enable", "false"); // or true.
> props.put("offsets.storage", "kafka");
>
>
>
>
> Winton Capital Management Limited ("Winton") is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the United
> Kingdom, registered as an investment adviser with the US Securities and
> Exchange Commission, registered with the US Commodity Futures Trading
> Commission and a member of the National Futures Association in the United
> States.
>
> This communication, including any attachments, is confidential and may be
> privileged. This email is for use by the intended recipient only. If you
> receive it in error, please notify the sender and delete it. You should not
> copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering materials
> and consult their own advisers with respect to investment decisions and
> inform themselves as to applicable legal requirements, exchange control
> regulations and taxes in the countries of their citizenship, residence or
> domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of its
> communications, including emails. However Winton accepts no liability for
> any materials transmitted. Emails are not secure and cannot be guaranteed
> to be error free.
>