You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by al...@gmail.com,
al...@gmail.com on 2018/06/19 07:52:06 UTC
Stdout both to file and console
Good morning,
I would like to have stdout of notebooks both printed out to console and file. How can I achieve that? I tried to play around with log4j but without any success; it seems it requires a custom appender implementation. Any other simpler idea?
Re: Stdout both to file and console
Posted by Alessandro Liparoti <al...@gmail.com>.
Thanks. I will give it a try!
*Alessandro Liparoti*
2018-06-20 21:21 GMT+02:00 Ruslan Dautkhanov <da...@gmail.com>:
>
> Something like this for log4j should do
>
> log4j.rootLogger = INFO, dailyfile
>> log4j.appender.stdout = org.apache.log4j.ConsoleAppender
>> log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
>> log4j.appender.stdout.layout.ConversionPattern=%5p [%d] ({%t} %F[%M]:%L)
>> - %m%n
>> log4j.appender.dailyfile.DatePattern=.yyyy-MM-dd
>> log4j.appender.dailyfile.Threshold = DEBUG
>> log4j.appender.dailyfile = org.apache.log4j.DailyRollingFileAppender
>> log4j.appender.dailyfile.File = ${zeppelin.log.file}
>> log4j.appender.dailyfile.layout = org.apache.log4j.PatternLayout
>> log4j.appender.dailyfile.layout.ConversionPattern=%5p [%d] ({%t}
>> %F[%M]:%L) - %m%n
>>
>> log4j.logger.org.apache.zeppelin.interpreter.InterpreterFactory=DEBUG
>> log4j.logger.org.apache.zeppelin.notebook.Paragraph=DEBUG
>> log4j.logger.org.apache.zeppelin.scheduler=DEBUG
>> log4j.logger.org.apache.zeppelin.spark=DEBUG
>> log4j.logger.org.apache.zeppelin.python=DEBUG
>> log4j.logger.org.apache.zeppelin.interpreter.util=DEBUG
>> log4j.logger.org.apache.zeppelin.interpreter.remote=DEBUG
>> log4j.logger.org.apache.zeppelin.interpreter.remote.
>> RemoteInterpreterServer=DEBUG
>> log4j.logger.org.glassfish.jersey.internal.inject.Providers=SEVERE
>
>
>
>
> --
> Ruslan Dautkhanov
>
>
> On Wed, Jun 20, 2018 at 3:01 AM Alessandro Liparoti <
> alessandro.lipa@gmail.com> wrote:
>
>> Hi,
>> yes spark UI is a tool I already use for it but as Rusian mentioned would
>> be good to have this functionality.
>> Rusian: which verbose level allows me to have stdout in log files? Is
>> there an attribute to add to the appender for this?
>>
>> *Alessandro Liparoti*
>>
>> 2018-06-19 19:52 GMT+02:00 Ruslan Dautkhanov <da...@gmail.com>:
>>
>>> If you set pretty verbose level in log4j then you can see output in log
>>> files. I've seen it there.
>>> Then you can use regexps to strip out paragraph outputs from rest of
>>> debugging messages.
>>> May work as a one off effort. Might be a good idea to file an
>>> enhancement request - this can be also useful
>>> for scheduled notebook runs - would be great to go back and review each
>>> scheduled note executions etc.
>>>
>>>
>>>
>>> On Tue, Jun 19, 2018 at 2:56 AM Alessandro Liparoti <
>>> alessandro.lipa@gmail.com> wrote:
>>>
>>>> I am comparing performances between different implementations of a
>>>> spark job and I am testing a chunk of code which prints partial results and
>>>> info to sdtout. I can surely replace all the prints with logger calls and
>>>> collect them. I just wanted to know if there was a way to avoid this or if
>>>> this functionality was of easier implementation.
>>>>
>>>> *Alessandro Liparoti*
>>>>
>>>> 2018-06-19 10:52 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>>>>
>>>>>
>>>>> Not sure what kind of analysis you want to do, is the logging info in
>>>>> the interpreter log file enough for you ? (You can update the log level in
>>>>> log4j.properties to get more logs)
>>>>>
>>>>> Alessandro Liparoti <al...@gmail.com>于2018年6月19日周二 下午4:47写道：
>>>>>
>>>>>> I would like to post-analyze the output of verbose jobs in the
>>>>>> notebook and save them, avoiding to relaunch the jobs again. It would be
>>>>>> also good to have the stderr logged to file.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> *Alessandro Liparoti*
>>>>>>
>>>>>> 2018-06-19 10:43 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>>>>>>
>>>>>>>
>>>>>>> I am not afraid it is not possible now. The stdout of notebooks is
>>>>>>> not based on log4j. If you want it output to file as well, you might need
>>>>>>> to change the code of the interpreter itself.
>>>>>>> Usually it is not necessary to log it to log file as well, could you
>>>>>>> tell why you want that ? Thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> alessandro.lipa@gmail.com <al...@gmail.com>于2018年6月19日周二
>>>>>>> 下午3:52写道：
>>>>>>>
>>>>>>>> Good morning,
>>>>>>>> I would like to have stdout of notebooks both printed out to
>>>>>>>> console and file. How can I achieve that? I tried to play around with log4j
>>>>>>>> but without any success; it seems it requires a custom appender
>>>>>>>> implementation. Any other simpler idea?
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
Re: Stdout both to file and console
Posted by Ruslan Dautkhanov <da...@gmail.com>.
Something like this for log4j should do
log4j.rootLogger = INFO, dailyfile
> log4j.appender.stdout = org.apache.log4j.ConsoleAppender
> log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
> log4j.appender.stdout.layout.ConversionPattern=%5p [%d] ({%t} %F[%M]:%L) -
> %m%n
> log4j.appender.dailyfile.DatePattern=.yyyy-MM-dd
> log4j.appender.dailyfile.Threshold = DEBUG
> log4j.appender.dailyfile = org.apache.log4j.DailyRollingFileAppender
> log4j.appender.dailyfile.File = ${zeppelin.log.file}
> log4j.appender.dailyfile.layout = org.apache.log4j.PatternLayout
> log4j.appender.dailyfile.layout.ConversionPattern=%5p [%d] ({%t}
> %F[%M]:%L) - %m%n
>
> log4j.logger.org.apache.zeppelin.interpreter.InterpreterFactory=DEBUG
> log4j.logger.org.apache.zeppelin.notebook.Paragraph=DEBUG
> log4j.logger.org.apache.zeppelin.scheduler=DEBUG
> log4j.logger.org.apache.zeppelin.spark=DEBUG
> log4j.logger.org.apache.zeppelin.python=DEBUG
> log4j.logger.org.apache.zeppelin.interpreter.util=DEBUG
> log4j.logger.org.apache.zeppelin.interpreter.remote=DEBUG
>
> log4j.logger.org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer=DEBUG
> log4j.logger.org.glassfish.jersey.internal.inject.Providers=SEVERE
--
Ruslan Dautkhanov
On Wed, Jun 20, 2018 at 3:01 AM Alessandro Liparoti <
alessandro.lipa@gmail.com> wrote:
> Hi,
> yes spark UI is a tool I already use for it but as Rusian mentioned would
> be good to have this functionality.
> Rusian: which verbose level allows me to have stdout in log files? Is
> there an attribute to add to the appender for this?
>
> *Alessandro Liparoti*
>
> 2018-06-19 19:52 GMT+02:00 Ruslan Dautkhanov <da...@gmail.com>:
>
>> If you set pretty verbose level in log4j then you can see output in log
>> files. I've seen it there.
>> Then you can use regexps to strip out paragraph outputs from rest of
>> debugging messages.
>> May work as a one off effort. Might be a good idea to file an enhancement
>> request - this can be also useful
>> for scheduled notebook runs - would be great to go back and review each
>> scheduled note executions etc.
>>
>>
>>
>> On Tue, Jun 19, 2018 at 2:56 AM Alessandro Liparoti <
>> alessandro.lipa@gmail.com> wrote:
>>
>>> I am comparing performances between different implementations of a spark
>>> job and I am testing a chunk of code which prints partial results and info
>>> to sdtout. I can surely replace all the prints with logger calls and
>>> collect them. I just wanted to know if there was a way to avoid this or if
>>> this functionality was of easier implementation.
>>>
>>> *Alessandro Liparoti*
>>>
>>> 2018-06-19 10:52 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>>>
>>>>
>>>> Not sure what kind of analysis you want to do, is the logging info in
>>>> the interpreter log file enough for you ? (You can update the log level in
>>>> log4j.properties to get more logs)
>>>>
>>>> Alessandro Liparoti <al...@gmail.com>于2018年6月19日周二 下午4:47写道：
>>>>
>>>>> I would like to post-analyze the output of verbose jobs in the
>>>>> notebook and save them, avoiding to relaunch the jobs again. It would be
>>>>> also good to have the stderr logged to file.
>>>>>
>>>>> Thanks
>>>>>
>>>>> *Alessandro Liparoti*
>>>>>
>>>>> 2018-06-19 10:43 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>>>>>
>>>>>>
>>>>>> I am not afraid it is not possible now. The stdout of notebooks is
>>>>>> not based on log4j. If you want it output to file as well, you might need
>>>>>> to change the code of the interpreter itself.
>>>>>> Usually it is not necessary to log it to log file as well, could you
>>>>>> tell why you want that ? Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>> alessandro.lipa@gmail.com <al...@gmail.com>于2018年6月19日周二
>>>>>> 下午3:52写道：
>>>>>>
>>>>>>> Good morning,
>>>>>>> I would like to have stdout of notebooks both printed out to console
>>>>>>> and file. How can I achieve that? I tried to play around with log4j but
>>>>>>> without any success; it seems it requires a custom appender implementation.
>>>>>>> Any other simpler idea?
>>>>>>>
>>>>>>
>>>>>
>>>
>
Re: Stdout both to file and console
Posted by Alessandro Liparoti <al...@gmail.com>.
Hi,
yes spark UI is a tool I already use for it but as Rusian mentioned would
be good to have this functionality.
Rusian: which verbose level allows me to have stdout in log files? Is there
an attribute to add to the appender for this?
*Alessandro Liparoti*
2018-06-19 19:52 GMT+02:00 Ruslan Dautkhanov <da...@gmail.com>:
> If you set pretty verbose level in log4j then you can see output in log
> files. I've seen it there.
> Then you can use regexps to strip out paragraph outputs from rest of
> debugging messages.
> May work as a one off effort. Might be a good idea to file an enhancement
> request - this can be also useful
> for scheduled notebook runs - would be great to go back and review each
> scheduled note executions etc.
>
>
>
> On Tue, Jun 19, 2018 at 2:56 AM Alessandro Liparoti <
> alessandro.lipa@gmail.com> wrote:
>
>> I am comparing performances between different implementations of a spark
>> job and I am testing a chunk of code which prints partial results and info
>> to sdtout. I can surely replace all the prints with logger calls and
>> collect them. I just wanted to know if there was a way to avoid this or if
>> this functionality was of easier implementation.
>>
>> *Alessandro Liparoti*
>>
>> 2018-06-19 10:52 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>>
>>>
>>> Not sure what kind of analysis you want to do, is the logging info in
>>> the interpreter log file enough for you ? (You can update the log level in
>>> log4j.properties to get more logs)
>>>
>>> Alessandro Liparoti <al...@gmail.com>于2018年6月19日周二 下午4:47写道：
>>>
>>>> I would like to post-analyze the output of verbose jobs in the notebook
>>>> and save them, avoiding to relaunch the jobs again. It would be also good
>>>> to have the stderr logged to file.
>>>>
>>>> Thanks
>>>>
>>>> *Alessandro Liparoti*
>>>>
>>>> 2018-06-19 10:43 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>>>>
>>>>>
>>>>> I am not afraid it is not possible now. The stdout of notebooks is not
>>>>> based on log4j. If you want it output to file as well, you might need to
>>>>> change the code of the interpreter itself.
>>>>> Usually it is not necessary to log it to log file as well, could you
>>>>> tell why you want that ? Thanks
>>>>>
>>>>>
>>>>>
>>>>> alessandro.lipa@gmail.com <al...@gmail.com>于2018年6月19日周二
>>>>> 下午3:52写道：
>>>>>
>>>>>> Good morning,
>>>>>> I would like to have stdout of notebooks both printed out to console
>>>>>> and file. How can I achieve that? I tried to play around with log4j but
>>>>>> without any success; it seems it requires a custom appender implementation.
>>>>>> Any other simpler idea?
>>>>>>
>>>>>
>>>>
>>
Re: Stdout both to file and console
Posted by Ruslan Dautkhanov <da...@gmail.com>.
If you set pretty verbose level in log4j then you can see output in log
files. I've seen it there.
Then you can use regexps to strip out paragraph outputs from rest of
debugging messages.
May work as a one off effort. Might be a good idea to file an enhancement
request - this can be also useful
for scheduled notebook runs - would be great to go back and review each
scheduled note executions etc.
On Tue, Jun 19, 2018 at 2:56 AM Alessandro Liparoti <
alessandro.lipa@gmail.com> wrote:
> I am comparing performances between different implementations of a spark
> job and I am testing a chunk of code which prints partial results and info
> to sdtout. I can surely replace all the prints with logger calls and
> collect them. I just wanted to know if there was a way to avoid this or if
> this functionality was of easier implementation.
>
> *Alessandro Liparoti*
>
> 2018-06-19 10:52 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>
>>
>> Not sure what kind of analysis you want to do, is the logging info in the
>> interpreter log file enough for you ? (You can update the log level in
>> log4j.properties to get more logs)
>>
>> Alessandro Liparoti <al...@gmail.com>于2018年6月19日周二 下午4:47写道：
>>
>>> I would like to post-analyze the output of verbose jobs in the notebook
>>> and save them, avoiding to relaunch the jobs again. It would be also good
>>> to have the stderr logged to file.
>>>
>>> Thanks
>>>
>>> *Alessandro Liparoti*
>>>
>>> 2018-06-19 10:43 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>>>
>>>>
>>>> I am not afraid it is not possible now. The stdout of notebooks is not
>>>> based on log4j. If you want it output to file as well, you might need to
>>>> change the code of the interpreter itself.
>>>> Usually it is not necessary to log it to log file as well, could you
>>>> tell why you want that ? Thanks
>>>>
>>>>
>>>>
>>>> alessandro.lipa@gmail.com <al...@gmail.com>于2018年6月19日周二
>>>> 下午3:52写道：
>>>>
>>>>> Good morning,
>>>>> I would like to have stdout of notebooks both printed out to console
>>>>> and file. How can I achieve that? I tried to play around with log4j but
>>>>> without any success; it seems it requires a custom appender implementation.
>>>>> Any other simpler idea?
>>>>>
>>>>
>>>
>
Re: Stdout both to file and console
Posted by Jeff Zhang <zj...@gmail.com>.
Why not use spark web ui to compare the performance ? It seems easier for
me.
Alessandro Liparoti <al...@gmail.com>于2018年6月19日周二 下午4:56写道：
> I am comparing performances between different implementations of a spark
> job and I am testing a chunk of code which prints partial results and info
> to sdtout. I can surely replace all the prints with logger calls and
> collect them. I just wanted to know if there was a way to avoid this or if
> this functionality was of easier implementation.
>
> *Alessandro Liparoti*
>
> 2018-06-19 10:52 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>
>>
>> Not sure what kind of analysis you want to do, is the logging info in the
>> interpreter log file enough for you ? (You can update the log level in
>> log4j.properties to get more logs)
>>
>> Alessandro Liparoti <al...@gmail.com>于2018年6月19日周二 下午4:47写道：
>>
>>> I would like to post-analyze the output of verbose jobs in the notebook
>>> and save them, avoiding to relaunch the jobs again. It would be also good
>>> to have the stderr logged to file.
>>>
>>> Thanks
>>>
>>> *Alessandro Liparoti*
>>>
>>> 2018-06-19 10:43 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>>>
>>>>
>>>> I am not afraid it is not possible now. The stdout of notebooks is not
>>>> based on log4j. If you want it output to file as well, you might need to
>>>> change the code of the interpreter itself.
>>>> Usually it is not necessary to log it to log file as well, could you
>>>> tell why you want that ? Thanks
>>>>
>>>>
>>>>
>>>> alessandro.lipa@gmail.com <al...@gmail.com>于2018年6月19日周二
>>>> 下午3:52写道：
>>>>
>>>>> Good morning,
>>>>> I would like to have stdout of notebooks both printed out to console
>>>>> and file. How can I achieve that? I tried to play around with log4j but
>>>>> without any success; it seems it requires a custom appender implementation.
>>>>> Any other simpler idea?
>>>>>
>>>>
>>>
>
Re: Stdout both to file and console
Posted by Alessandro Liparoti <al...@gmail.com>.
I am comparing performances between different implementations of a spark
job and I am testing a chunk of code which prints partial results and info
to sdtout. I can surely replace all the prints with logger calls and
collect them. I just wanted to know if there was a way to avoid this or if
this functionality was of easier implementation.
*Alessandro Liparoti*
2018-06-19 10:52 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>
> Not sure what kind of analysis you want to do, is the logging info in the
> interpreter log file enough for you ? (You can update the log level in
> log4j.properties to get more logs)
>
> Alessandro Liparoti <al...@gmail.com>于2018年6月19日周二 下午4:47写道：
>
>> I would like to post-analyze the output of verbose jobs in the notebook
>> and save them, avoiding to relaunch the jobs again. It would be also good
>> to have the stderr logged to file.
>>
>> Thanks
>>
>> *Alessandro Liparoti*
>>
>> 2018-06-19 10:43 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>>
>>>
>>> I am not afraid it is not possible now. The stdout of notebooks is not
>>> based on log4j. If you want it output to file as well, you might need to
>>> change the code of the interpreter itself.
>>> Usually it is not necessary to log it to log file as well, could you
>>> tell why you want that ? Thanks
>>>
>>>
>>>
>>> alessandro.lipa@gmail.com <al...@gmail.com>于2018年6月19日周二
>>> 下午3:52写道：
>>>
>>>> Good morning,
>>>> I would like to have stdout of notebooks both printed out to console
>>>> and file. How can I achieve that? I tried to play around with log4j but
>>>> without any success; it seems it requires a custom appender implementation.
>>>> Any other simpler idea?
>>>>
>>>
>>
Re: Stdout both to file and console
Posted by Jeff Zhang <zj...@gmail.com>.
Not sure what kind of analysis you want to do, is the logging info in the
interpreter log file enough for you ? (You can update the log level in
log4j.properties to get more logs)
Alessandro Liparoti <al...@gmail.com>于2018年6月19日周二 下午4:47写道：
> I would like to post-analyze the output of verbose jobs in the notebook
> and save them, avoiding to relaunch the jobs again. It would be also good
> to have the stderr logged to file.
>
> Thanks
>
> *Alessandro Liparoti*
>
> 2018-06-19 10:43 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>
>>
>> I am not afraid it is not possible now. The stdout of notebooks is not
>> based on log4j. If you want it output to file as well, you might need to
>> change the code of the interpreter itself.
>> Usually it is not necessary to log it to log file as well, could you tell
>> why you want that ? Thanks
>>
>>
>>
>> alessandro.lipa@gmail.com <al...@gmail.com>于2018年6月19日周二
>> 下午3:52写道：
>>
>>> Good morning,
>>> I would like to have stdout of notebooks both printed out to console and
>>> file. How can I achieve that? I tried to play around with log4j but without
>>> any success; it seems it requires a custom appender implementation. Any
>>> other simpler idea?
>>>
>>
>
Re: Stdout both to file and console
Posted by Alessandro Liparoti <al...@gmail.com>.
I would like to post-analyze the output of verbose jobs in the notebook and
save them, avoiding to relaunch the jobs again. It would be also good to
have the stderr logged to file.
Thanks
*Alessandro Liparoti*
2018-06-19 10:43 GMT+02:00 Jeff Zhang <zj...@gmail.com>:
>
> I am not afraid it is not possible now. The stdout of notebooks is not
> based on log4j. If you want it output to file as well, you might need to
> change the code of the interpreter itself.
> Usually it is not necessary to log it to log file as well, could you tell
> why you want that ? Thanks
>
>
>
> alessandro.lipa@gmail.com <al...@gmail.com>于2018年6月19日周二
> 下午3:52写道：
>
>> Good morning,
>> I would like to have stdout of notebooks both printed out to console and
>> file. How can I achieve that? I tried to play around with log4j but without
>> any success; it seems it requires a custom appender implementation. Any
>> other simpler idea?
>>
>
Re: Stdout both to file and console
Posted by Jeff Zhang <zj...@gmail.com>.
I am not afraid it is not possible now. The stdout of notebooks is not
based on log4j. If you want it output to file as well, you might need to
change the code of the interpreter itself.
Usually it is not necessary to log it to log file as well, could you tell
why you want that ? Thanks
alessandro.lipa@gmail.com <al...@gmail.com>于2018年6月19日周二 下午3:52写道：
> Good morning,
> I would like to have stdout of notebooks both printed out to console and
> file. How can I achieve that? I tried to play around with log4j but without
> any success; it seems it requires a custom appender implementation. Any
> other simpler idea?
>