You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "S. Zhou" <my...@yahoo.com.INVALID> on 2014/08/13 01:39:34 UTC
how to access workers from spark context
I tried to access worker info from spark context but it seems spark context does no expose such API. The reason of doing that is: it seems spark context itself does not have logic to detect if its workers are in dead status. So I like to add such logic by myself.
BTW, it seems spark web UI has some logic of detecting dead workers. But all relevant classes are declared as private for the spark package scope.
Please let me know how to solve this issue (or if there is an alternative way to achieve the same purpose)
Thanks
Re: how to access workers from spark context
Posted by "S. Zhou" <my...@yahoo.com.INVALID>.
actually if you search the spark mail archives you will find many similar topics. At this time, I just want to manage it by myself.
On Tuesday, August 12, 2014 8:46 PM, Stanley Shi <ss...@pivotal.io> wrote:
This seems a bug, right? It's not the user's responsibility to manage the workers.
On Wed, Aug 13, 2014 at 11:28 AM, S. Zhou <my...@yahoo.com> wrote:
Sometimes workers are dead but spark context does not know it and still send jobs.
>
>
>
>On Tuesday, August 12, 2014 7:14 PM, Stanley Shi <ss...@pivotal.io> wrote:
>
>
>
>Why do you need to detect the worker status in the application? you application generally don't need to know where it is executed.
>
>
>
>On Wed, Aug 13, 2014 at 7:39 AM, S. Zhou <my...@yahoo.com.invalid> wrote:
>
>I tried to access worker info from spark context but it seems spark context does no expose such API. The reason of doing that is: it seems spark context itself does not have logic to detect if its workers are in dead status. So I like to add such logic by myself.
>>
>>
>>BTW, it seems spark web UI has some logic of detecting dead workers. But all relevant classes are declared as private for the spark package scope.
>>
>>
>>Please let me know how to solve this issue (or if there is an alternative way to achieve the same purpose)
>>
>>
>>Thanks
>>
>>
>
>
>
>--
>
>Regards,
>Stanley Shi,
>
>
>
--
Regards,
Stanley Shi,
Re: how to access workers from spark context
Posted by Stanley Shi <ss...@pivotal.io>.
This seems a bug, right? It's not the user's responsibility to manage the
workers.
On Wed, Aug 13, 2014 at 11:28 AM, S. Zhou <my...@yahoo.com> wrote:
> Sometimes workers are dead but spark context does not know it and still
> send jobs.
>
>
> On Tuesday, August 12, 2014 7:14 PM, Stanley Shi <ss...@pivotal.io>
> wrote:
>
>
> Why do you need to detect the worker status in the application? you
> application generally don't need to know where it is executed.
>
>
> On Wed, Aug 13, 2014 at 7:39 AM, S. Zhou <my...@yahoo.com.invalid> wrote:
>
> I tried to access worker info from spark context but it seems spark
> context does no expose such API. The reason of doing that is: it seems
> spark context itself does not have logic to detect if its workers are in
> dead status. So I like to add such logic by myself.
>
> BTW, it seems spark web UI has some logic of detecting dead workers. But
> all relevant classes are declared as private for the spark package scope.
>
> Please let me know how to solve this issue (or if there is an alternative
> way to achieve the same purpose)
>
> Thanks
>
>
>
>
> --
> Regards,
> *Stanley Shi,*
>
>
>
>
--
Regards,
*Stanley Shi,*
Re: how to access workers from spark context
Posted by "S. Zhou" <my...@yahoo.com.INVALID>.
Sometimes workers are dead but spark context does not know it and still send jobs.
On Tuesday, August 12, 2014 7:14 PM, Stanley Shi <ss...@pivotal.io> wrote:
Why do you need to detect the worker status in the application? you application generally don't need to know where it is executed.
On Wed, Aug 13, 2014 at 7:39 AM, S. Zhou <my...@yahoo.com.invalid> wrote:
I tried to access worker info from spark context but it seems spark context does no expose such API. The reason of doing that is: it seems spark context itself does not have logic to detect if its workers are in dead status. So I like to add such logic by myself.
>
>
>BTW, it seems spark web UI has some logic of detecting dead workers. But all relevant classes are declared as private for the spark package scope.
>
>
>Please let me know how to solve this issue (or if there is an alternative way to achieve the same purpose)
>
>
>Thanks
>
>
--
Regards,
Stanley Shi,