You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vanessa Gligor <va...@gmail.com> on 2016/03/02 15:11:00 UTC
session.timeout.ms limit - Kafka Consumer
Hello,
I am using Kafka higher consumer 0.9.0. I am not using the auto commit for
the offsets, so after I consume the messaged (poll from Kafka) I will have
to commit the offsets manually.
The issue that I have is actually that the processing of the messages takes
longer than 30s (and I cannot call poll again, before these messages are
processed) and when I try to commit the offset a exception is thrown:
ERROR o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred
while committing offsets for group MetadataConsumerSpout.
(I have found on stackoverflow this explanation: so if you wait for longer
that the timeout request then the coordinator for the topic will kickout
the consumer because it will think is dead and it will rebalance the group)
In order to get rid of this I have thought about a couple of solutions:
1. The configuration session.timeout.ms has a maximum value, so if I try to
set it to 60 seconds, also I get an exception, because this value is not in
the valid interval.
2. I have tried to find a solution to get a paginated request when the
polling method is called - no success.
3. I have tried to send a heart beat from the outside of the poll (because
this method sends the heartbeats) - no success.
Thank you.
Re: session.timeout.ms limit - Kafka Consumer
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
In fact, KIP-41 has been implemented in trunk -- see
https://issues.apache.org/jira/browse/KAFKA-3007. Testing against a version
including that change would be greatly appreciated to ensure it fully
addresses the problems you're seeing.
-Ewen
On Wed, Mar 2, 2016 at 7:00 AM, Olson,Andrew <AO...@cerner.com> wrote:
> This topic is currently being discussed at
> https://issues.apache.org/jira/browse/KAFKA-2986 and
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records
>
>
> On 3/2/16, 8:11 AM, "Vanessa Gligor" <va...@gmail.com> wrote:
>
> >Hello,
> >
> >I am using Kafka higher consumer 0.9.0. I am not using the auto commit for
> >the offsets, so after I consume the messaged (poll from Kafka) I will have
> >to commit the offsets manually.
> >
> >The issue that I have is actually that the processing of the messages
> takes
> >longer than 30s (and I cannot call poll again, before these messages are
> >processed) and when I try to commit the offset a exception is thrown:
> >ERROR o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred
> >while committing offsets for group MetadataConsumerSpout.
> >(I have found on stackoverflow this explanation: so if you wait for longer
> >that the timeout request then the coordinator for the topic will kickout
> >the consumer because it will think is dead and it will rebalance the
> group)
> >
> >In order to get rid of this I have thought about a couple of solutions:
> >
> >1. The configuration session.timeout.ms has a maximum value, so if I try
> to
> >set it to 60 seconds, also I get an exception, because this value is not
> in
> >the valid interval.
> >
> >2. I have tried to find a solution to get a paginated request when the
> >polling method is called - no success.
> >
> >3. I have tried to send a heart beat from the outside of the poll (because
> >this method sends the heartbeats) - no success.
> >
> >
> >Thank you.
>
> CONFIDENTIALITY NOTICE This message and any included attachments are from
> Cerner Corporation and are intended only for the addressee. The information
> contained in this message is confidential and may constitute inside or
> non-public information under international, federal, or state securities
> laws. Unauthorized forwarding, printing, copying, distribution, or use of
> such information is strictly prohibited and may be unlawful. If you are not
> the addressee, please promptly delete this message and notify the sender of
> the delivery error by e-mail or you may call Cerner's corporate offices in
> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
--
Thanks,
Ewen
Re: session.timeout.ms limit - Kafka Consumer
Posted by "Olson,Andrew" <AO...@CERNER.COM>.
This topic is currently being discussed at https://issues.apache.org/jira/browse/KAFKA-2986 and https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records
On 3/2/16, 8:11 AM, "Vanessa Gligor" <va...@gmail.com> wrote:
>Hello,
>
>I am using Kafka higher consumer 0.9.0. I am not using the auto commit for
>the offsets, so after I consume the messaged (poll from Kafka) I will have
>to commit the offsets manually.
>
>The issue that I have is actually that the processing of the messages takes
>longer than 30s (and I cannot call poll again, before these messages are
>processed) and when I try to commit the offset a exception is thrown:
>ERROR o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred
>while committing offsets for group MetadataConsumerSpout.
>(I have found on stackoverflow this explanation: so if you wait for longer
>that the timeout request then the coordinator for the topic will kickout
>the consumer because it will think is dead and it will rebalance the group)
>
>In order to get rid of this I have thought about a couple of solutions:
>
>1. The configuration session.timeout.ms has a maximum value, so if I try to
>set it to 60 seconds, also I get an exception, because this value is not in
>the valid interval.
>
>2. I have tried to find a solution to get a paginated request when the
>polling method is called - no success.
>
>3. I have tried to send a heart beat from the outside of the poll (because
>this method sends the heartbeats) - no success.
>
>
>Thank you.
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.