You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "bit1129@163.com" <bi...@163.com> on 2015/04/03 09:59:28 UTC

About Waiting batches on the spark streaming UI

I copied the following from the spark streaming UI, I don't know why the Waiting batches is 1, my understanding is that it should be 72.
Following  is my understanding: 
1. Total time is 1minute 35 seconds=95 seconds
2. Batch interval is 1 second, so, 95 batches are generated in 95 seconds.
3. Processed batches are 23(Correct, because in my processing code, it does nothing but sleep 4 seconds)
4. Then the waiting batches should be 95-23=72


Started at: Fri Apr 03 15:17:47 CST 2015 
Time since start: 1 minute 35 seconds 
Network receivers: 1 
Batch interval: 1 second 
Processed batches: 23 
Waiting batches: 1 
Received records: 0 
Processed records: 0       



bit1129@163.com

Re: Re: About Waiting batches on the spark streaming UI

Posted by "bit1129@163.com" <bi...@163.com>.
Thanks Tathagata for the explanation!



bit1129@163.com
 
From: Tathagata Das
Date: 2015-04-04 01:28
To: Ted Yu
CC: bit1129; user
Subject: Re: About Waiting batches on the spark streaming UI
Maybe that should be marked as waiting as well. Will keep that in mind. We plan to update the ui soon, so will keep that in mind.
On Apr 3, 2015 10:12 AM, "Ted Yu" <yu...@gmail.com> wrote:
Maybe add another stat for batches waiting in the job queue ?

Cheers

On Fri, Apr 3, 2015 at 10:01 AM, Tathagata Das <td...@databricks.com> wrote:
Very good question! This is because the current code is written such that the ui considers a batch as waiting only when it has actually started being processed. Thats batched waiting in the job queue is not considered in the calculation. It is arguable that it may be more intuitive to count that in the waiting as well.
On Apr 3, 2015 12:59 AM, "bit1129@163.com" <bi...@163.com> wrote:

I copied the following from the spark streaming UI, I don't know why the Waiting batches is 1, my understanding is that it should be 72.
Following  is my understanding: 
1. Total time is 1minute 35 seconds=95 seconds
2. Batch interval is 1 second, so, 95 batches are generated in 95 seconds.
3. Processed batches are 23(Correct, because in my processing code, it does nothing but sleep 4 seconds)
4. Then the waiting batches should be 95-23=72


Started at: Fri Apr 03 15:17:47 CST 2015 
Time since start: 1 minute 35 seconds 
Network receivers: 1 
Batch interval: 1 second 
Processed batches: 23 
Waiting batches: 1 
Received records: 0 
Processed records: 0       



bit1129@163.com


Re: About Waiting batches on the spark streaming UI

Posted by Tathagata Das <td...@databricks.com>.
Maybe that should be marked as waiting as well. Will keep that in mind. We
plan to update the ui soon, so will keep that in mind.
On Apr 3, 2015 10:12 AM, "Ted Yu" <yu...@gmail.com> wrote:

> Maybe add another stat for batches waiting in the job queue ?
>
> Cheers
>
> On Fri, Apr 3, 2015 at 10:01 AM, Tathagata Das <td...@databricks.com>
> wrote:
>
>> Very good question! This is because the current code is written such that
>> the ui considers a batch as waiting only when it has actually started being
>> processed. Thats batched waiting in the job queue is not considered in the
>> calculation. It is arguable that it may be more intuitive to count that in
>> the waiting as well.
>> On Apr 3, 2015 12:59 AM, "bit1129@163.com" <bi...@163.com> wrote:
>>
>>>
>>> I copied the following from the spark streaming UI, I don't know why the
>>> Waiting batches is 1, my understanding is that it should be 72.
>>> Following  is my understanding:
>>> 1. Total time is 1minute 35 seconds=95 seconds
>>> 2. Batch interval is 1 second, so, 95 batches are generated in 95
>>> seconds.
>>> 3. Processed batches are 23(Correct, because in my processing code, it
>>> does nothing but sleep 4 seconds)
>>> 4. Then the waiting batches should be 95-23=72
>>>
>>>
>>>
>>>    - *Started at: * Fri Apr 03 15:17:47 CST 2015
>>>    - *Time since start: *1 minute 35 seconds
>>>    - *Network receivers: *1
>>>    - *Batch interval: *1 second
>>>    - *Processed batches: *23
>>>    - *Waiting batches: *1
>>>    - *Received records: *0
>>>    - *Processed records: *0
>>>
>>>
>>> ------------------------------
>>> bit1129@163.com
>>>
>>
>

Re: About Waiting batches on the spark streaming UI

Posted by Ted Yu <yu...@gmail.com>.
Maybe add another stat for batches waiting in the job queue ?

Cheers

On Fri, Apr 3, 2015 at 10:01 AM, Tathagata Das <td...@databricks.com> wrote:

> Very good question! This is because the current code is written such that
> the ui considers a batch as waiting only when it has actually started being
> processed. Thats batched waiting in the job queue is not considered in the
> calculation. It is arguable that it may be more intuitive to count that in
> the waiting as well.
> On Apr 3, 2015 12:59 AM, "bit1129@163.com" <bi...@163.com> wrote:
>
>>
>> I copied the following from the spark streaming UI, I don't know why the
>> Waiting batches is 1, my understanding is that it should be 72.
>> Following  is my understanding:
>> 1. Total time is 1minute 35 seconds=95 seconds
>> 2. Batch interval is 1 second, so, 95 batches are generated in 95 seconds.
>> 3. Processed batches are 23(Correct, because in my processing code, it
>> does nothing but sleep 4 seconds)
>> 4. Then the waiting batches should be 95-23=72
>>
>>
>>
>>    - *Started at: * Fri Apr 03 15:17:47 CST 2015
>>    - *Time since start: *1 minute 35 seconds
>>    - *Network receivers: *1
>>    - *Batch interval: *1 second
>>    - *Processed batches: *23
>>    - *Waiting batches: *1
>>    - *Received records: *0
>>    - *Processed records: *0
>>
>>
>> ------------------------------
>> bit1129@163.com
>>
>

Re: About Waiting batches on the spark streaming UI

Posted by Tathagata Das <td...@databricks.com>.
Very good question! This is because the current code is written such that
the ui considers a batch as waiting only when it has actually started being
processed. Thats batched waiting in the job queue is not considered in the
calculation. It is arguable that it may be more intuitive to count that in
the waiting as well.
On Apr 3, 2015 12:59 AM, "bit1129@163.com" <bi...@163.com> wrote:

>
> I copied the following from the spark streaming UI, I don't know why the
> Waiting batches is 1, my understanding is that it should be 72.
> Following  is my understanding:
> 1. Total time is 1minute 35 seconds=95 seconds
> 2. Batch interval is 1 second, so, 95 batches are generated in 95 seconds.
> 3. Processed batches are 23(Correct, because in my processing code, it
> does nothing but sleep 4 seconds)
> 4. Then the waiting batches should be 95-23=72
>
>
>
>    - *Started at: * Fri Apr 03 15:17:47 CST 2015
>    - *Time since start: *1 minute 35 seconds
>    - *Network receivers: *1
>    - *Batch interval: *1 second
>    - *Processed batches: *23
>    - *Waiting batches: *1
>    - *Received records: *0
>    - *Processed records: *0
>
>
> ------------------------------
> bit1129@163.com
>