You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Mohit Goyal <it...@gmail.com> on 2022/07/21 07:16:54 UTC
Notify spout thread while sleep in case of empty nextTuple
Hi All,
We are using apache storm to read msgs from kafka queues. Traffic usually
is high in these kafka queues so processing is almost real time. But I'm
facing one problem when load is relatively low and spout doesn't have msgs
to read. In this case, the spout will go into sleep for sometime (~5
seconds) depending upon my sleep/backoff strategy. During this spout sleep
interval, If there is a msg available in kafka queues, my spout will not
read it. Spout will only be able to process the msg once the sleep/backoff
time is done. Hence there is a delay or around 3-4 sec in processing of
that msg. Is there any way present in Apache Storm to notify the sleeping
spout so that it can start processing again?
Any leads are highly appreciated.
*Thanks & Regards*
*Mohit Goyal*
Re: Notify spout thread while sleep in case of empty nextTuple
Posted by Rui Abreu <ru...@gmail.com>.
Can you elaborate on the sleep/backoff strategy part?
On Fri, Jul 29, 2022, 06:31 Mohit Goyal <it...@gmail.com> wrote:
> Hi Rui,
>
> I'm using storm 1.2.1. I've written my spouts on top of BaseRichSpout.
> Passing default values.
>
> *Thanks & Regards*
> *Mohit Goyal*
>
>
> On Fri, Jul 22, 2022 at 7:45 PM Rui Abreu <ru...@gmail.com> wrote:
>
>> Hi Mohit,
>>
>> Which version of Storm are you using?
>> What are the values you are passing to the Spout on KafkaSpoutConfig ?
>>
>> On Thu, 21 Jul 2022 at 08:17, Mohit Goyal <it...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> We are using apache storm to read msgs from kafka queues. Traffic
>>> usually is high in these kafka queues so processing is almost real time.
>>> But I'm facing one problem when load is relatively low and spout doesn't
>>> have msgs to read. In this case, the spout will go into sleep for sometime
>>> (~5 seconds) depending upon my sleep/backoff strategy. During this spout
>>> sleep interval, If there is a msg available in kafka queues, my spout will
>>> not read it. Spout will only be able to process the msg once the
>>> sleep/backoff time is done. Hence there is a delay or around 3-4 sec in
>>> processing of that msg. Is there any way present in Apache Storm to
>>> notify the sleeping spout so that it can start processing again?
>>>
>>> Any leads are highly appreciated.
>>>
>>> *Thanks & Regards*
>>> *Mohit Goyal*
>>>
>>
Re: Notify spout thread while sleep in case of empty nextTuple
Posted by Mohit Goyal <it...@gmail.com>.
Hi Rui,
I'm using storm 1.2.1. I've written my spouts on top of BaseRichSpout.
Passing default values.
*Thanks & Regards*
*Mohit Goyal*
On Fri, Jul 22, 2022 at 7:45 PM Rui Abreu <ru...@gmail.com> wrote:
> Hi Mohit,
>
> Which version of Storm are you using?
> What are the values you are passing to the Spout on KafkaSpoutConfig ?
>
> On Thu, 21 Jul 2022 at 08:17, Mohit Goyal <it...@gmail.com> wrote:
>
>> Hi All,
>>
>> We are using apache storm to read msgs from kafka queues. Traffic usually
>> is high in these kafka queues so processing is almost real time. But I'm
>> facing one problem when load is relatively low and spout doesn't have msgs
>> to read. In this case, the spout will go into sleep for sometime (~5
>> seconds) depending upon my sleep/backoff strategy. During this spout sleep
>> interval, If there is a msg available in kafka queues, my spout will not
>> read it. Spout will only be able to process the msg once the sleep/backoff
>> time is done. Hence there is a delay or around 3-4 sec in processing of
>> that msg. Is there any way present in Apache Storm to notify the sleeping
>> spout so that it can start processing again?
>>
>> Any leads are highly appreciated.
>>
>> *Thanks & Regards*
>> *Mohit Goyal*
>>
>
Re: Notify spout thread while sleep in case of empty nextTuple
Posted by Rui Abreu <ru...@gmail.com>.
Hi Mohit,
Which version of Storm are you using?
What are the values you are passing to the Spout on KafkaSpoutConfig ?
On Thu, 21 Jul 2022 at 08:17, Mohit Goyal <it...@gmail.com> wrote:
> Hi All,
>
> We are using apache storm to read msgs from kafka queues. Traffic usually
> is high in these kafka queues so processing is almost real time. But I'm
> facing one problem when load is relatively low and spout doesn't have msgs
> to read. In this case, the spout will go into sleep for sometime (~5
> seconds) depending upon my sleep/backoff strategy. During this spout sleep
> interval, If there is a msg available in kafka queues, my spout will not
> read it. Spout will only be able to process the msg once the sleep/backoff
> time is done. Hence there is a delay or around 3-4 sec in processing of
> that msg. Is there any way present in Apache Storm to notify the sleeping
> spout so that it can start processing again?
>
> Any leads are highly appreciated.
>
> *Thanks & Regards*
> *Mohit Goyal*
>