You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Mike Fang <ch...@gmail.com> on 2015/06/18 03:59:20 UTC

understanding on the "waiting batches" and "scheduling delay" in Streaming UI

Hi,



I have a spark streaming program running for ~ 25hrs. When I check the
Streaming UI tab. I found the “Waiting batches” is 144. But the “scheduling
delay” is 0. I am a bit confused.

If the “waiting batches” is 144, that means many batches are waiting in the
queue to be processed? If this is the case, the scheduling delay should be
high rather than 0. Am I missing anything?



Thanks,

Mike

Re: understanding on the "waiting batches" and "scheduling delay" in Streaming UI

Posted by Mike Fang <ch...@gmail.com>.

Hi TD/Das,

Attached the screenshot.

Still running on 1.3 now. Is under upgrade to 1.4.
Will update again when ready for 1.4.

Thanks,
Mike

On Fri, Jun 19, 2015 at 9:48 AM, Tathagata Das <td...@databricks.com> wrote:

> Also, could you give a screenshot of the streaming UI. Even better, could
> you run it on Spark 1.4 which has a new streaming UI and then use that for
> debugging/screenshot?
>
> TD
>
> On Thu, Jun 18, 2015 at 3:05 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Which version of spark? and what is your data source? For some reason,
>> your processing delay is exceeding the batch duration. And its strange that
>> you are not seeing any scheduling delay.
>>
>> Thanks
>> Best Regards
>>
>> On Thu, Jun 18, 2015 at 7:29 AM, Mike Fang <ch...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I have a spark streaming program running for ~ 25hrs. When I check the
>>> Streaming UI tab. I found the “Waiting batches” is 144. But the “scheduling
>>> delay” is 0. I am a bit confused.
>>>
>>> If the “waiting batches” is 144, that means many batches are waiting in
>>> the queue to be processed? If this is the case, the scheduling delay should
>>> be high rather than 0. Am I missing anything?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>
>>
>

Re: understanding on the "waiting batches" and "scheduling delay" in Streaming UI

Posted by Tathagata Das <td...@databricks.com>.

Also, could you give a screenshot of the streaming UI. Even better, could
you run it on Spark 1.4 which has a new streaming UI and then use that for
debugging/screenshot?

TD

On Thu, Jun 18, 2015 at 3:05 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:

> Which version of spark? and what is your data source? For some reason,
> your processing delay is exceeding the batch duration. And its strange that
> you are not seeing any scheduling delay.
>
> Thanks
> Best Regards
>
> On Thu, Jun 18, 2015 at 7:29 AM, Mike Fang <ch...@gmail.com> wrote:
>
>> Hi,
>>
>>
>>
>> I have a spark streaming program running for ~ 25hrs. When I check the
>> Streaming UI tab. I found the “Waiting batches” is 144. But the “scheduling
>> delay” is 0. I am a bit confused.
>>
>> If the “waiting batches” is 144, that means many batches are waiting in
>> the queue to be processed? If this is the case, the scheduling delay should
>> be high rather than 0. Am I missing anything?
>>
>>
>>
>> Thanks,
>>
>> Mike
>>
>
>

Re: understanding on the "waiting batches" and "scheduling delay" in Streaming UI

Posted by "Fang, Mike" <ch...@paypal.com.INVALID>.

Hi Das,

Thanks for your reply. Somehow I missed it..
I am using Spark 1.3. The data source is from kafka.
Yeah, not sure why the delay is 0. I'll run against 1.4 and give a screenshot.

Thanks,
Mike

From: Akhil Das <ak...@sigmoidanalytics.com>>
Date: Thursday, June 18, 2015 at 6:05 PM
To: Mike Fang <ch...@gmail.com>>
Cc: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Re: understanding on the "waiting batches" and "scheduling delay" in Streaming UI

Which version of spark? and what is your data source? For some reason, your processing delay is exceeding the batch duration. And its strange that you are not seeing any scheduling delay.

Thanks
Best Regards

On Thu, Jun 18, 2015 at 7:29 AM, Mike Fang <ch...@gmail.com>> wrote:
Hi,

I have a spark streaming program running for ~ 25hrs. When I check the Streaming UI tab. I found the "Waiting batches" is 144. But the "scheduling delay" is 0. I am a bit confused.
If the "waiting batches" is 144, that means many batches are waiting in the queue to be processed? If this is the case, the scheduling delay should be high rather than 0. Am I missing anything?

Thanks,
Mike

Re: understanding on the "waiting batches" and "scheduling delay" in Streaming UI

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

Which version of spark? and what is your data source? For some reason, your
processing delay is exceeding the batch duration. And its strange that you
are not seeing any scheduling delay.

Thanks
Best Regards

On Thu, Jun 18, 2015 at 7:29 AM, Mike Fang <ch...@gmail.com> wrote:

> Hi,
>
>
>
> I have a spark streaming program running for ~ 25hrs. When I check the
> Streaming UI tab. I found the “Waiting batches” is 144. But the “scheduling
> delay” is 0. I am a bit confused.
>
> If the “waiting batches” is 144, that means many batches are waiting in
> the queue to be processed? If this is the case, the scheduling delay should
> be high rather than 0. Am I missing anything?
>
>
>
> Thanks,
>
> Mike
>