You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Prabhu Joseph <pr...@gmail.com> on 2016/03/16 01:14:02 UTC

Spark UI Completed Jobs

Hi All,


Spark UI Completed Jobs section shows below information, what is the
skipped value shown for Stages and Tasks below.

Job_ID    Description    Submitted                    Duration
Stages (Succeeded/Total)    Tasks (for all stages): Succeeded/Total

11             count          2016/03/14 15:35:32      1.4 min
164/164 * (163 skipped)   *            19841/19788
*(41405 skipped)*
Thanks,
Prabhu Joseph

Re: Spark UI Completed Jobs

Posted by Prabhu Joseph <pr...@gmail.com>.
Thanks Mark and Jeff

On Wed, Mar 16, 2016 at 7:11 AM, Mark Hamstra <ma...@clearstorydata.com>
wrote:

> Looks to me like the one remaining Stage would execute 19788 Task if all
> of those Tasks succeeded on the first try; but because of retries, 19841
> Tasks were actually executed.  Meanwhile, there were 41405 Tasks in the the
> 163 Stages that were skipped.
>
> I think -- but the Spark UI's accounting may not be 100% accurate and bug
> free.
>
> On Tue, Mar 15, 2016 at 6:34 PM, Prabhu Joseph <prabhujose.gates@gmail.com
> > wrote:
>
>> Okay, so out of 164 stages, is 163 are skipped. And how 41405 tasks are
>> skipped if the total is only 19788.
>>
>> On Wed, Mar 16, 2016 at 6:31 AM, Mark Hamstra <ma...@clearstorydata.com>
>> wrote:
>>
>>> It's not just if the RDD is explicitly cached, but also if the map
>>> outputs for stages have been materialized into shuffle files and are still
>>> accessible through the map output tracker.  Because of that, explicitly
>>> caching RDD actions often gains you little or nothing, since even without a
>>> call to cache() or persist() the prior computation will largely be reused
>>> and stages will show up as skipped -- i.e. no need to recompute that stage.
>>>
>>> On Tue, Mar 15, 2016 at 5:50 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>> If RDD is cached, this RDD is only computed once and the stages for
>>>> computing this RDD in the following jobs are skipped.
>>>>
>>>>
>>>> On Wed, Mar 16, 2016 at 8:14 AM, Prabhu Joseph <
>>>> prabhujose.gates@gmail.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>>
>>>>> Spark UI Completed Jobs section shows below information, what is the
>>>>> skipped value shown for Stages and Tasks below.
>>>>>
>>>>> Job_ID    Description    Submitted
>>>>> Duration           Stages (Succeeded/Total)    Tasks (for all stages):
>>>>> Succeeded/Total
>>>>>
>>>>> 11             count          2016/03/14 15:35:32      1.4
>>>>> min             164/164 * (163 skipped)   *            19841/19788
>>>>> *(41405 skipped)*
>>>>> Thanks,
>>>>> Prabhu Joseph
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>>
>>>> Jeff Zhang
>>>>
>>>
>>>
>>
>

Re: Spark UI Completed Jobs

Posted by Mark Hamstra <ma...@clearstorydata.com>.
Looks to me like the one remaining Stage would execute 19788 Task if all of
those Tasks succeeded on the first try; but because of retries, 19841 Tasks
were actually executed.  Meanwhile, there were 41405 Tasks in the the 163
Stages that were skipped.

I think -- but the Spark UI's accounting may not be 100% accurate and bug
free.

On Tue, Mar 15, 2016 at 6:34 PM, Prabhu Joseph <pr...@gmail.com>
wrote:

> Okay, so out of 164 stages, is 163 are skipped. And how 41405 tasks are
> skipped if the total is only 19788.
>
> On Wed, Mar 16, 2016 at 6:31 AM, Mark Hamstra <ma...@clearstorydata.com>
> wrote:
>
>> It's not just if the RDD is explicitly cached, but also if the map
>> outputs for stages have been materialized into shuffle files and are still
>> accessible through the map output tracker.  Because of that, explicitly
>> caching RDD actions often gains you little or nothing, since even without a
>> call to cache() or persist() the prior computation will largely be reused
>> and stages will show up as skipped -- i.e. no need to recompute that stage.
>>
>> On Tue, Mar 15, 2016 at 5:50 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>>> If RDD is cached, this RDD is only computed once and the stages for
>>> computing this RDD in the following jobs are skipped.
>>>
>>>
>>> On Wed, Mar 16, 2016 at 8:14 AM, Prabhu Joseph <
>>> prabhujose.gates@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>>
>>>> Spark UI Completed Jobs section shows below information, what is the
>>>> skipped value shown for Stages and Tasks below.
>>>>
>>>> Job_ID    Description    Submitted
>>>> Duration           Stages (Succeeded/Total)    Tasks (for all stages):
>>>> Succeeded/Total
>>>>
>>>> 11             count          2016/03/14 15:35:32      1.4
>>>> min             164/164 * (163 skipped)   *            19841/19788
>>>> *(41405 skipped)*
>>>> Thanks,
>>>> Prabhu Joseph
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>>
>

Re: Spark UI Completed Jobs

Posted by Prabhu Joseph <pr...@gmail.com>.
Okay, so out of 164 stages, is 163 are skipped. And how 41405 tasks are
skipped if the total is only 19788.

On Wed, Mar 16, 2016 at 6:31 AM, Mark Hamstra <ma...@clearstorydata.com>
wrote:

> It's not just if the RDD is explicitly cached, but also if the map outputs
> for stages have been materialized into shuffle files and are still
> accessible through the map output tracker.  Because of that, explicitly
> caching RDD actions often gains you little or nothing, since even without a
> call to cache() or persist() the prior computation will largely be reused
> and stages will show up as skipped -- i.e. no need to recompute that stage.
>
> On Tue, Mar 15, 2016 at 5:50 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>> If RDD is cached, this RDD is only computed once and the stages for
>> computing this RDD in the following jobs are skipped.
>>
>>
>> On Wed, Mar 16, 2016 at 8:14 AM, Prabhu Joseph <
>> prabhujose.gates@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>>
>>> Spark UI Completed Jobs section shows below information, what is the
>>> skipped value shown for Stages and Tasks below.
>>>
>>> Job_ID    Description    Submitted                    Duration
>>> Stages (Succeeded/Total)    Tasks (for all stages): Succeeded/Total
>>>
>>> 11             count          2016/03/14 15:35:32      1.4
>>> min             164/164 * (163 skipped)   *            19841/19788
>>> *(41405 skipped)*
>>> Thanks,
>>> Prabhu Joseph
>>>
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>

Re: Spark UI Completed Jobs

Posted by Mark Hamstra <ma...@clearstorydata.com>.
It's not just if the RDD is explicitly cached, but also if the map outputs
for stages have been materialized into shuffle files and are still
accessible through the map output tracker.  Because of that, explicitly
caching RDD actions often gains you little or nothing, since even without a
call to cache() or persist() the prior computation will largely be reused
and stages will show up as skipped -- i.e. no need to recompute that stage.

On Tue, Mar 15, 2016 at 5:50 PM, Jeff Zhang <zj...@gmail.com> wrote:

> If RDD is cached, this RDD is only computed once and the stages for
> computing this RDD in the following jobs are skipped.
>
>
> On Wed, Mar 16, 2016 at 8:14 AM, Prabhu Joseph <prabhujose.gates@gmail.com
> > wrote:
>
>> Hi All,
>>
>>
>> Spark UI Completed Jobs section shows below information, what is the
>> skipped value shown for Stages and Tasks below.
>>
>> Job_ID    Description    Submitted                    Duration
>> Stages (Succeeded/Total)    Tasks (for all stages): Succeeded/Total
>>
>> 11             count          2016/03/14 15:35:32      1.4
>> min             164/164 * (163 skipped)   *            19841/19788
>> *(41405 skipped)*
>> Thanks,
>> Prabhu Joseph
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Spark UI Completed Jobs

Posted by Jeff Zhang <zj...@gmail.com>.
If RDD is cached, this RDD is only computed once and the stages for
computing this RDD in the following jobs are skipped.


On Wed, Mar 16, 2016 at 8:14 AM, Prabhu Joseph <pr...@gmail.com>
wrote:

> Hi All,
>
>
> Spark UI Completed Jobs section shows below information, what is the
> skipped value shown for Stages and Tasks below.
>
> Job_ID    Description    Submitted                    Duration
> Stages (Succeeded/Total)    Tasks (for all stages): Succeeded/Total
>
> 11             count          2016/03/14 15:35:32      1.4 min
> 164/164 * (163 skipped)   *            19841/19788
> *(41405 skipped)*
> Thanks,
> Prabhu Joseph
>



-- 
Best Regards

Jeff Zhang