You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Aniket Bhatnagar <an...@gmail.com> on 2014/09/24 09:29:38 UTC
[Streaming] Non-blocking recommendation in custom receiver
documentation and KinesisReceiver's worker.run blocking calll
Hi all
Reading through Spark streaming's custom receiver documentation, it is
recommended that onStart and onStop methods should not block indefinitely.
However, looking at the source code of KinesisReceiver, the onStart method
calls worker.run that blocks until worker is shutdown (via a call to
onStop).
So, my question is what are the ramifications of making a blocking call in
onStart and whether this is something that should be addressed
in KinesisReceiver implementation.
Thanks,
Aniket
Re: [Streaming] Non-blocking recommendation in custom receiver
documentation and KinesisReceiver's worker.run blocking calll
Posted by Aniket Bhatnagar <an...@gmail.com>.
Hi TD
Unfortunately, I am off for a week so I won't be able to test this until
next week. Will keep you posted.
Aniket
On Sat, May 23, 2015, 6:16 AM Tathagata Das <td...@databricks.com> wrote:
> Hey Aniket, I just checked in the fix in Spark master and branch-1.4.
> Could you download Spark and test it out?
>
>
>
> On Thu, May 21, 2015 at 1:43 AM, Tathagata Das <td...@databricks.com>
> wrote:
>
>> Thanks for the JIRA. I will look into this issue.
>>
>> TD
>>
>> On Thu, May 21, 2015 at 1:31 AM, Aniket Bhatnagar <
>> aniket.bhatnagar@gmail.com> wrote:
>>
>>> I ran into one of the issues that are potentially caused because of this
>>> and have logged a JIRA bug -
>>> https://issues.apache.org/jira/browse/SPARK-7788
>>>
>>> Thanks,
>>> Aniket
>>>
>>> On Wed, Sep 24, 2014 at 12:59 PM Aniket Bhatnagar <
>>> aniket.bhatnagar@gmail.com> wrote:
>>>
>>>> Hi all
>>>>
>>>> Reading through Spark streaming's custom receiver documentation, it is
>>>> recommended that onStart and onStop methods should not block indefinitely.
>>>> However, looking at the source code of KinesisReceiver, the onStart method
>>>> calls worker.run that blocks until worker is shutdown (via a call to
>>>> onStop).
>>>>
>>>> So, my question is what are the ramifications of making a blocking call
>>>> in onStart and whether this is something that should be addressed
>>>> in KinesisReceiver implementation.
>>>>
>>>> Thanks,
>>>> Aniket
>>>>
>>>
>>
>
Re: [Streaming] Non-blocking recommendation in custom receiver
documentation and KinesisReceiver's worker.run blocking calll
Posted by Tathagata Das <td...@databricks.com>.
Hey Aniket, I just checked in the fix in Spark master and branch-1.4. Could
you download Spark and test it out?
On Thu, May 21, 2015 at 1:43 AM, Tathagata Das <td...@databricks.com> wrote:
> Thanks for the JIRA. I will look into this issue.
>
> TD
>
> On Thu, May 21, 2015 at 1:31 AM, Aniket Bhatnagar <
> aniket.bhatnagar@gmail.com> wrote:
>
>> I ran into one of the issues that are potentially caused because of this
>> and have logged a JIRA bug -
>> https://issues.apache.org/jira/browse/SPARK-7788
>>
>> Thanks,
>> Aniket
>>
>> On Wed, Sep 24, 2014 at 12:59 PM Aniket Bhatnagar <
>> aniket.bhatnagar@gmail.com> wrote:
>>
>>> Hi all
>>>
>>> Reading through Spark streaming's custom receiver documentation, it is
>>> recommended that onStart and onStop methods should not block indefinitely.
>>> However, looking at the source code of KinesisReceiver, the onStart method
>>> calls worker.run that blocks until worker is shutdown (via a call to
>>> onStop).
>>>
>>> So, my question is what are the ramifications of making a blocking call
>>> in onStart and whether this is something that should be addressed
>>> in KinesisReceiver implementation.
>>>
>>> Thanks,
>>> Aniket
>>>
>>
>
Re: [Streaming] Non-blocking recommendation in custom receiver
documentation and KinesisReceiver's worker.run blocking calll
Posted by Tathagata Das <td...@databricks.com>.
Thanks for the JIRA. I will look into this issue.
TD
On Thu, May 21, 2015 at 1:31 AM, Aniket Bhatnagar <
aniket.bhatnagar@gmail.com> wrote:
> I ran into one of the issues that are potentially caused because of this
> and have logged a JIRA bug -
> https://issues.apache.org/jira/browse/SPARK-7788
>
> Thanks,
> Aniket
>
> On Wed, Sep 24, 2014 at 12:59 PM Aniket Bhatnagar <
> aniket.bhatnagar@gmail.com> wrote:
>
>> Hi all
>>
>> Reading through Spark streaming's custom receiver documentation, it is
>> recommended that onStart and onStop methods should not block indefinitely.
>> However, looking at the source code of KinesisReceiver, the onStart method
>> calls worker.run that blocks until worker is shutdown (via a call to
>> onStop).
>>
>> So, my question is what are the ramifications of making a blocking call
>> in onStart and whether this is something that should be addressed
>> in KinesisReceiver implementation.
>>
>> Thanks,
>> Aniket
>>
>
Re: [Streaming] Non-blocking recommendation in custom receiver
documentation and KinesisReceiver's worker.run blocking calll
Posted by Aniket Bhatnagar <an...@gmail.com>.
I ran into one of the issues that are potentially caused because of this
and have logged a JIRA bug -
https://issues.apache.org/jira/browse/SPARK-7788
Thanks,
Aniket
On Wed, Sep 24, 2014 at 12:59 PM Aniket Bhatnagar <
aniket.bhatnagar@gmail.com> wrote:
> Hi all
>
> Reading through Spark streaming's custom receiver documentation, it is
> recommended that onStart and onStop methods should not block indefinitely.
> However, looking at the source code of KinesisReceiver, the onStart method
> calls worker.run that blocks until worker is shutdown (via a call to
> onStop).
>
> So, my question is what are the ramifications of making a blocking call in
> onStart and whether this is something that should be addressed
> in KinesisReceiver implementation.
>
> Thanks,
> Aniket
>