You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sourav Chandra <so...@livestream.com> on 2014/03/24 08:20:39 UTC
spark executor/driver log files management
Hi,
I have few questions regarding log file management in spark:
1. Currently I did not find any way to modify the lof file name for
executor/drivers). Its hardcoded as stdout and stderr. Also there is no log
rotation.
In case of streaming application this will grow forever and become
unmanageable. Is there any way to overcome this?
Thanks,
--
Sourav Chandra
Senior Software Engineer
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
sourav.chandra@livestream.com
o: +91 80 4121 8723
m: +91 988 699 3746
skype: sourav.chandra
Livestream
"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,
Bangalore 560034
www.livestream.com
Re: spark executor/driver log files management
Posted by Tathagata Das <ta...@gmail.com>.
The logs from the executor are redirected to stdout only because there is a
default log4j.properties that is configured to do so. If you put your
log4j.properties with rolling file appender in the classpath (refer to
Spark docs for that), all the logs will get redirected to a separate files
that will get rolled over. So even though there will be a stdout, nothing
much would get printed in that. Though the side effect of this modification
is that you wont be able to see the logs from the Spark web UI as that only
shows stdout.
On Mon, Mar 24, 2014 at 10:48 PM, Sourav Chandra <
sourav.chandra@livestream.com> wrote:
> Hi TD,
>
> I thought about that but was not sure whether this will have any impact in
> spark UI/ Executor runner as it redirects stream to stderr/stdout. But
> ideally it should not as it will fetch the log record from stderr file
> (which is latest)..
>
> Is my understanding correct?
>
> Thanks,
> Sourav
>
>
> On Tue, Mar 25, 2014 at 3:26 AM, Tathagata Das <
> tathagata.das1565@gmail.com> wrote:
>
>> You can use RollingFileAppenders in log4j.properties.
>>
>> http://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html
>>
>> You can have other scripts delete old logs.
>>
>> TD
>>
>>
>> On Mon, Mar 24, 2014 at 12:20 AM, Sourav Chandra <
>> sourav.chandra@livestream.com> wrote:
>>
>>> Hi,
>>>
>>> I have few questions regarding log file management in spark:
>>>
>>> 1. Currently I did not find any way to modify the lof file name for
>>> executor/drivers). Its hardcoded as stdout and stderr. Also there is no log
>>> rotation.
>>>
>>> In case of streaming application this will grow forever and become
>>> unmanageable. Is there any way to overcome this?
>>>
>>> Thanks,
>>> --
>>>
>>> Sourav Chandra
>>>
>>> Senior Software Engineer
>>>
>>> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>>>
>>> sourav.chandra@livestream.com
>>>
>>> o: +91 80 4121 8723
>>>
>>> m: +91 988 699 3746
>>>
>>> skype: sourav.chandra
>>>
>>> Livestream
>>>
>>> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
>>> Block, Koramangala Industrial Area,
>>>
>>> Bangalore 560034
>>>
>>> www.livestream.com
>>>
>>
>>
>
>
> --
>
> Sourav Chandra
>
> Senior Software Engineer
>
> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>
> sourav.chandra@livestream.com
>
> o: +91 80 4121 8723
>
> m: +91 988 699 3746
>
> skype: sourav.chandra
>
> Livestream
>
> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
> Block, Koramangala Industrial Area,
>
> Bangalore 560034
>
> www.livestream.com
>
Re: spark executor/driver log files management
Posted by Sourav Chandra <so...@livestream.com>.
Hi TD,
I thought about that but was not sure whether this will have any impact in
spark UI/ Executor runner as it redirects stream to stderr/stdout. But
ideally it should not as it will fetch the log record from stderr file
(which is latest)..
Is my understanding correct?
Thanks,
Sourav
On Tue, Mar 25, 2014 at 3:26 AM, Tathagata Das
<ta...@gmail.com>wrote:
> You can use RollingFileAppenders in log4j.properties.
>
> http://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html
>
> You can have other scripts delete old logs.
>
> TD
>
>
> On Mon, Mar 24, 2014 at 12:20 AM, Sourav Chandra <
> sourav.chandra@livestream.com> wrote:
>
>> Hi,
>>
>> I have few questions regarding log file management in spark:
>>
>> 1. Currently I did not find any way to modify the lof file name for
>> executor/drivers). Its hardcoded as stdout and stderr. Also there is no log
>> rotation.
>>
>> In case of streaming application this will grow forever and become
>> unmanageable. Is there any way to overcome this?
>>
>> Thanks,
>> --
>>
>> Sourav Chandra
>>
>> Senior Software Engineer
>>
>> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>>
>> sourav.chandra@livestream.com
>>
>> o: +91 80 4121 8723
>>
>> m: +91 988 699 3746
>>
>> skype: sourav.chandra
>>
>> Livestream
>>
>> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
>> Block, Koramangala Industrial Area,
>>
>> Bangalore 560034
>>
>> www.livestream.com
>>
>
>
--
Sourav Chandra
Senior Software Engineer
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
sourav.chandra@livestream.com
o: +91 80 4121 8723
m: +91 988 699 3746
skype: sourav.chandra
Livestream
"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,
Bangalore 560034
www.livestream.com
Re: spark executor/driver log files management
Posted by Tathagata Das <ta...@gmail.com>.
You can use RollingFileAppenders in log4j.properties.
http://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html
You can have other scripts delete old logs.
TD
On Mon, Mar 24, 2014 at 12:20 AM, Sourav Chandra <
sourav.chandra@livestream.com> wrote:
> Hi,
>
> I have few questions regarding log file management in spark:
>
> 1. Currently I did not find any way to modify the lof file name for
> executor/drivers). Its hardcoded as stdout and stderr. Also there is no log
> rotation.
>
> In case of streaming application this will grow forever and become
> unmanageable. Is there any way to overcome this?
>
> Thanks,
> --
>
> Sourav Chandra
>
> Senior Software Engineer
>
> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>
> sourav.chandra@livestream.com
>
> o: +91 80 4121 8723
>
> m: +91 988 699 3746
>
> skype: sourav.chandra
>
> Livestream
>
> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
> Block, Koramangala Industrial Area,
>
> Bangalore 560034
>
> www.livestream.com
>