You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Utkarsh Sengar <ut...@gmail.com> on 2016/02/02 00:24:25 UTC
Using accumulator to push custom logs to driver
I am trying to debug code executed in executors by logging. Even when I add
log4j's LOG.info(..) inside .map() I don't see it in mesos task logs in the
corresponding slaves.
Its anyway inefficient to keep checking multiple slaves for logs.
One way to deal with this is to push logs to a central location.
Another way (for debugging purposes) is to use accumulators . Is it
advisable to use accumulators to push string from executors to driver?
It will simplify things when I am debugging datasets, bugs which is hard to
reproduce locally etc.
Suggestions/comments?
-Utkarsh
Re: Using accumulator to push custom logs to driver
Posted by Holden Karau <ho...@pigscanfly.ca>.
Ah if its manual ad-hoc logging of the 100 to 200 lines then thats probably
OK.
On Mon, Feb 1, 2016 at 3:48 PM, Utkarsh Sengar <ut...@gmail.com>
wrote:
> Not alot of string messages, I need it mostly for debugging purposed which
> I will use on an ahdoc basis - manually add debug statements which returns
> info about the dataset etc.
> I would assume the strings will vary from 100-200lines max, that would be
> about 50-100KB if they are really long lines.
>
> -Utkarsh
>
> On Mon, Feb 1, 2016 at 3:40 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> I wouldn't use accumulators for things which could get large, they can
>> become kind of a bottle neck. Do you have a lot of string messages you want
>> to bring back or only a few?
>>
>> On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <ut...@gmail.com>
>> wrote:
>>
>>> I am trying to debug code executed in executors by logging. Even when I
>>> add log4j's LOG.info(..) inside .map() I don't see it in mesos task logs in
>>> the corresponding slaves.
>>> Its anyway inefficient to keep checking multiple slaves for logs.
>>>
>>> One way to deal with this is to push logs to a central location.
>>>
>>> Another way (for debugging purposes) is to use accumulators . Is it
>>> advisable to use accumulators to push string from executors to driver?
>>> It will simplify things when I am debugging datasets, bugs which is hard
>>> to reproduce locally etc.
>>>
>>> Suggestions/comments?
>>>
>>> -Utkarsh
>>>
>>
>>
>>
>> --
>> Cell : 425-233-8271
>> Twitter: https://twitter.com/holdenkarau
>>
>
>
>
> --
> Thanks,
> -Utkarsh
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Re: Using accumulator to push custom logs to driver
Posted by Utkarsh Sengar <ut...@gmail.com>.
Not alot of string messages, I need it mostly for debugging purposed which
I will use on an ahdoc basis - manually add debug statements which returns
info about the dataset etc.
I would assume the strings will vary from 100-200lines max, that would be
about 50-100KB if they are really long lines.
-Utkarsh
On Mon, Feb 1, 2016 at 3:40 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> I wouldn't use accumulators for things which could get large, they can
> become kind of a bottle neck. Do you have a lot of string messages you want
> to bring back or only a few?
>
> On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <ut...@gmail.com>
> wrote:
>
>> I am trying to debug code executed in executors by logging. Even when I
>> add log4j's LOG.info(..) inside .map() I don't see it in mesos task logs in
>> the corresponding slaves.
>> Its anyway inefficient to keep checking multiple slaves for logs.
>>
>> One way to deal with this is to push logs to a central location.
>>
>> Another way (for debugging purposes) is to use accumulators . Is it
>> advisable to use accumulators to push string from executors to driver?
>> It will simplify things when I am debugging datasets, bugs which is hard
>> to reproduce locally etc.
>>
>> Suggestions/comments?
>>
>> -Utkarsh
>>
>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
>
--
Thanks,
-Utkarsh
Re: Using accumulator to push custom logs to driver
Posted by Holden Karau <ho...@pigscanfly.ca>.
I wouldn't use accumulators for things which could get large, they can
become kind of a bottle neck. Do you have a lot of string messages you want
to bring back or only a few?
On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <ut...@gmail.com>
wrote:
> I am trying to debug code executed in executors by logging. Even when I
> add log4j's LOG.info(..) inside .map() I don't see it in mesos task logs in
> the corresponding slaves.
> Its anyway inefficient to keep checking multiple slaves for logs.
>
> One way to deal with this is to push logs to a central location.
>
> Another way (for debugging purposes) is to use accumulators . Is it
> advisable to use accumulators to push string from executors to driver?
> It will simplify things when I am debugging datasets, bugs which is hard
> to reproduce locally etc.
>
> Suggestions/comments?
>
> -Utkarsh
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau