You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Daniccan VP <da...@iqsystech.com> on 2016/10/14 10:32:38 UTC

Offset out of range error in Kafka Spout

Hi,

I am currently working on a project which involves Storm and Kafka interaction. I am using Kafka Spout which would consume the messages from the Kafka Brokers followed by the processing using the ATDM Bolt.

Now, the issue that I am facing is, sometimes when I run my Storm topology after a delay, I get a error, "Got fetch request with offset out of range" and this run in a indefinite loop. This is probably because the Log Retention period in Kafka has expired. But, is there a way to change the offset dynamically to the next available offset if this error occurs.

Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : daniccan@iqsystech.com
******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Re: Offset out of range error in Kafka Spout

Posted by Amber Kulkarni <am...@gmail.com>.
You can check spout config properties like
startOffsetTime
useStartOffsetTimeIfOffsetOutOfRange, etc.

https://community.hortonworks.com/questions/2737/kafka-spout-not-reading-from-beginning-even-if-the.html
this may help

On Wed, Nov 2, 2016 at 9:04 AM, Daniccan VP <da...@iqsystech.com> wrote:
> Storm version is 1.0.2 and Kafka version is 0.10.1.0.
>
> Thanks and Regards,
> Daniccan VP | Junior Software Engineer
> Email : daniccan@iqsystech.com
>
> ________________________________________
> From: Zhechao Ma [mazhechaomaillist@gmail.com]
> Sent: Wednesday, November 02, 2016 8:51 AM
> To: user@storm.apache.org
> Subject: Re: Offset out of range error in Kafka Spout
>
> which version of storm-kafka do you use?
>
> 2016-11-02 1:48 GMT+08:00 Daniccan VP <da...@iqsystech.com>>:
> Hi,
>
> I am currently working on a project which involves Storm and Kafka interaction. I am using Kafka Spout which would consume the messages from the Kafka Brokers followed by the processing using the ATDM Bolt.
>
> Now, the issue that I am facing is, sometimes when I run my Storm topology after a delay, I get a error, "Got fetch request with offset out of range" and this run in a indefinite loop. This is probably because the Log Retention period in Kafka has expired. But, is there a way to change the offset dynamically to the next available offset if this error occurs.
>
> Thanks and Regards,
> Daniccan VP | Junior Software Engineer
> Email : daniccan@iqsystech.com<http://iqsystech.com>
> ******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.
> ******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.
>
>
>
> --
> Thanks
> Zhechao Ma
> ******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.



-- 
Regards,
Amber Kulkarni

Re: Offset out of range error in Kafka Spout

Posted by Zhechao Ma <ma...@gmail.com>.
Not exactly, I mean the storm-kafka dependency. Maybe this issue can help
you.

https://issues.apache.org/jira/browse/STORM-643

2016-11-02 11:34 GMT+08:00 Daniccan VP <da...@iqsystech.com>:

> Storm version is 1.0.2 and Kafka version is 0.10.1.0.
>
> Thanks and Regards,
> Daniccan VP | Junior Software Engineer
> Email : daniccan@iqsystech.com
>
> ________________________________________
> From: Zhechao Ma [mazhechaomaillist@gmail.com]
> Sent: Wednesday, November 02, 2016 8:51 AM
> To: user@storm.apache.org
> Subject: Re: Offset out of range error in Kafka Spout
>
> which version of storm-kafka do you use?
>
> 2016-11-02 1:48 GMT+08:00 Daniccan VP <daniccan@iqsystech.com<mailto:
> daniccan@iqsystech.com>>:
> Hi,
>
> I am currently working on a project which involves Storm and Kafka
> interaction. I am using Kafka Spout which would consume the messages from
> the Kafka Brokers followed by the processing using the ATDM Bolt.
>
> Now, the issue that I am facing is, sometimes when I run my Storm topology
> after a delay, I get a error, "Got fetch request with offset out of range"
> and this run in a indefinite loop. This is probably because the Log
> Retention period in Kafka has expired. But, is there a way to change the
> offset dynamically to the next available offset if this error occurs.
>
> Thanks and Regards,
> Daniccan VP | Junior Software Engineer
> Email : daniccan@iqsystech.com<http://iqsystech.com>
> ************************************************************
> ************************************************************
> ******************************************* This email and any files
> transmitted with it are confidential and intended solely for the use of the
> individual or entity to whom they have been addressed. If you are not the
> intended recipient, you are notified that disclosing, copying, distributing
> or taking any action in reliance on the contents of this information is
> strictly prohibited. Please notify the sender immediately by e-mail if you
> have received this e-mail by mistake and delete this e-mail from your
> system.
> ************************************************************
> ************************************************************
> ******************************************* This email and any files
> transmitted with it are confidential and intended solely for the use of the
> individual or entity to whom they have been addressed. If you are not the
> intended recipient, you are notified that disclosing, copying, distributing
> or taking any action in reliance on the contents of this information is
> strictly prohibited. Please notify the sender immediately by e-mail if you
> have received this e-mail by mistake and delete this e-mail from your
> system.
>
>
>
> --
> Thanks
> Zhechao Ma
> ************************************************************
> ************************************************************
> ******************************************* This email and any files
> transmitted with it are confidential and intended solely for the use of the
> individual or entity to whom they have been addressed. If you are not the
> intended recipient, you are notified that disclosing, copying, distributing
> or taking any action in reliance on the contents of this information is
> strictly prohibited. Please notify the sender immediately by e-mail if you
> have received this e-mail by mistake and delete this e-mail from your
> system.
>



-- 
Thanks
Zhechao Ma

RE: Offset out of range error in Kafka Spout

Posted by Daniccan VP <da...@iqsystech.com>.
Storm version is 1.0.2 and Kafka version is 0.10.1.0.

Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : daniccan@iqsystech.com

________________________________________
From: Zhechao Ma [mazhechaomaillist@gmail.com]
Sent: Wednesday, November 02, 2016 8:51 AM
To: user@storm.apache.org
Subject: Re: Offset out of range error in Kafka Spout

which version of storm-kafka do you use?

2016-11-02 1:48 GMT+08:00 Daniccan VP <da...@iqsystech.com>>:
Hi,

I am currently working on a project which involves Storm and Kafka interaction. I am using Kafka Spout which would consume the messages from the Kafka Brokers followed by the processing using the ATDM Bolt.

Now, the issue that I am facing is, sometimes when I run my Storm topology after a delay, I get a error, "Got fetch request with offset out of range" and this run in a indefinite loop. This is probably because the Log Retention period in Kafka has expired. But, is there a way to change the offset dynamically to the next available offset if this error occurs.

Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : daniccan@iqsystech.com<http://iqsystech.com>
******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.
******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.



--
Thanks
Zhechao Ma
******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Re: Offset out of range error in Kafka Spout

Posted by Zhechao Ma <ma...@gmail.com>.
which version of storm-kafka do you use?

2016-11-02 1:48 GMT+08:00 Daniccan VP <da...@iqsystech.com>:

> Hi,
>
> I am currently working on a project which involves Storm and Kafka
> interaction. I am using Kafka Spout which would consume the messages from
> the Kafka Brokers followed by the processing using the ATDM Bolt.
>
> Now, the issue that I am facing is, sometimes when I run my Storm topology
> after a delay, I get a error, *"Got fetch request with offset out of
> range"* and this run in a indefinite loop. This is probably because the
> Log Retention period in Kafka has expired. But, is there a way to change
> the offset dynamically to the next available offset if this error occurs.
>
> Thanks and Regards,
> *Daniccan VP **|* *Junior Software Engineer*
> Email : daniccan@iqsystech.com
> ************************************************************
> ************************************************************
> ******************************************* This email and any files
> transmitted with it are confidential and intended solely for the use of the
> individual or entity to whom they have been addressed. If you are not the
> intended recipient, you are notified that disclosing, copying, distributing
> or taking any action in reliance on the contents of this information is
> strictly prohibited. Please notify the sender immediately by e-mail if you
> have received this e-mail by mistake and delete this e-mail from your
> system.
> ************************************************************
> ************************************************************
> ******************************************* This email and any files
> transmitted with it are confidential and intended solely for the use of the
> individual or entity to whom they have been addressed. If you are not the
> intended recipient, you are notified that disclosing, copying, distributing
> or taking any action in reliance on the contents of this information is
> strictly prohibited. Please notify the sender immediately by e-mail if you
> have received this e-mail by mistake and delete this e-mail from your
> system.
>



-- 
Thanks
Zhechao Ma

RE: Offset out of range error in Kafka Spout

Posted by Daniccan VP <da...@iqsystech.com>.
Hi,

I am currently working on a project which involves Storm and Kafka interaction. I am using Kafka Spout which would consume the messages from the Kafka Brokers followed by the processing using the ATDM Bolt.

Now, the issue that I am facing is, sometimes when I run my Storm topology after a delay, I get a error, "Got fetch request with offset out of range" and this run in a indefinite loop. This is probably because the Log Retention period in Kafka has expired. But, is there a way to change the offset dynamically to the next available offset if this error occurs.

Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : daniccan@iqsystech.com
******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.
******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.