You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by zhao selery <se...@gmail.com> on 2015/09/02 05:14:06 UTC
spark worker log stderr
I run scala code from zeppelin notebook, when I look at spark log from
spark/work/app-20150901193944-0015/0/stderr
Log file looks like:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/zhaoxq/Downloads/zeppelin2/incubator-zeppelin/interpreter/spark/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/zhaoxq/Downloads/zeppelin2/incubator-zeppelin/zeppelin-interpreter/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/zhaoxq/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
But if I run same code from spark-shell, the log file containts correct
spark log info:
15/09/01 19:31:58 INFO CoarseGrainedExecutorBackend: Registered signal
handlers for [TERM, HUP, INT]
15/09/01 19:31:58 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/09/01 19:31:59 INFO SecurityManager: Changing view acls to: zhaoxq
15/09/01 19:31:59 INFO SecurityManager: Changing modify acls to: zhaoxq
15/09/01 19:31:59 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(zhaoxq); users
with modify permissions: Set(zhaoxq)
15/09/01 19:31:59 INFO Slf4jLogger: Slf4jLogger started
.................
................
Also, if I rebuild zeppelin with remote false mode, which is:
change setting.getOption().setRemote(false); in
org.apache.zeppelin.interpreter.InterpreterFactory,
spark worker log is same as running from spark-shell, which is what I want.
So, my question is how can I get useful spark worker log with zeppelin
remote mode? Thanks.
Re: spark worker log stderr
Posted by zhao selery <se...@gmail.com>.
Thanks a lot for you reply, I copied my zeppelin distribution to another
machine and connect to another spark master, I can see spark log without
any problem, not sure what's the problem of my previous environment.
2015-09-13 17:11 GMT-07:00 Alexander Bezzubov <ab...@nflabs.com>:
> Hi,
>
> I think at this point it might be worth creating an issue to track this
> and bring it to attention dev@
>
> Thanks!
>
> On Wed, Sep 2, 2015 at 12:14 PM, zhao selery <se...@gmail.com> wrote:
>
>> I run scala code from zeppelin notebook, when I look at spark log from
>> spark/work/app-20150901193944-0015/0/stderr
>> Log file looks like:
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in
>> [jar:file:/home/zhaoxq/Downloads/zeppelin2/incubator-zeppelin/interpreter/spark/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/home/zhaoxq/Downloads/zeppelin2/incubator-zeppelin/zeppelin-interpreter/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/home/zhaoxq/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> explanation.
>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>
>>
>> But if I run same code from spark-shell, the log file containts correct
>> spark log info:
>> 15/09/01 19:31:58 INFO CoarseGrainedExecutorBackend: Registered signal
>> handlers for [TERM, HUP, INT]
>> 15/09/01 19:31:58 WARN NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>> 15/09/01 19:31:59 INFO SecurityManager: Changing view acls to: zhaoxq
>> 15/09/01 19:31:59 INFO SecurityManager: Changing modify acls to: zhaoxq
>> 15/09/01 19:31:59 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users with view permissions: Set(zhaoxq); users
>> with modify permissions: Set(zhaoxq)
>> 15/09/01 19:31:59 INFO Slf4jLogger: Slf4jLogger started
>> .................
>> ................
>>
>>
>> Also, if I rebuild zeppelin with remote false mode, which is:
>> change setting.getOption().setRemote(false); in
>> org.apache.zeppelin.interpreter.InterpreterFactory,
>> spark worker log is same as running from spark-shell, which is what I
>> want.
>>
>>
>> So, my question is how can I get useful spark worker log with zeppelin
>> remote mode? Thanks.
>>
>
>
>
> --
> --
> Kind regards,
> Alexander.
>
>
Re: spark worker log stderr
Posted by Alexander Bezzubov <ab...@nflabs.com>.
Hi,
I think at this point it might be worth creating an issue to track this and
bring it to attention dev@
Thanks!
On Wed, Sep 2, 2015 at 12:14 PM, zhao selery <se...@gmail.com> wrote:
> I run scala code from zeppelin notebook, when I look at spark log from
> spark/work/app-20150901193944-0015/0/stderr
> Log file looks like:
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/zhaoxq/Downloads/zeppelin2/incubator-zeppelin/interpreter/spark/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/zhaoxq/Downloads/zeppelin2/incubator-zeppelin/zeppelin-interpreter/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/zhaoxq/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
>
> But if I run same code from spark-shell, the log file containts correct
> spark log info:
> 15/09/01 19:31:58 INFO CoarseGrainedExecutorBackend: Registered signal
> handlers for [TERM, HUP, INT]
> 15/09/01 19:31:58 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/09/01 19:31:59 INFO SecurityManager: Changing view acls to: zhaoxq
> 15/09/01 19:31:59 INFO SecurityManager: Changing modify acls to: zhaoxq
> 15/09/01 19:31:59 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(zhaoxq); users
> with modify permissions: Set(zhaoxq)
> 15/09/01 19:31:59 INFO Slf4jLogger: Slf4jLogger started
> .................
> ................
>
>
> Also, if I rebuild zeppelin with remote false mode, which is:
> change setting.getOption().setRemote(false); in
> org.apache.zeppelin.interpreter.InterpreterFactory,
> spark worker log is same as running from spark-shell, which is what I want.
>
>
> So, my question is how can I get useful spark worker log with zeppelin
> remote mode? Thanks.
>
--
--
Kind regards,
Alexander.