You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Adrian Mocanu <am...@verticalscope.com> on 2014/05/12 16:51:39 UTC
missing method in my slf4j after excluding Spark ZK log4j
Hey guys,
I've asked before, in Spark 0.9 - I now use 0.9.1, about removing log4j dependency and was told that it was gone. However I still find it part of zookeeper imports. This is fine since I exclude it myself in the sbt file, but another issue arises.
I wonder if anyone else has run into this.
Spark uses log4j v1.2.17 and slf4j-log4j12:1.7.2
I use slf4j 1.7.5, logback 1.0.13, and log4joverslf4j v 1.7.5
I think my slf4j 1.7.5 doesn't agree with what zookeeper expects in its log4j v 1.2.17 because I get missing method error:
java.lang.NoSuchMethodError: org.apache.log4j.Logger.setLevel(Lorg/apache/log4j/Level;)V
at org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(AkkaUtils.scala:58)
at org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(AkkaUtils.scala:58)
at scala.Option.map(Option.scala:145)
at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:58)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:500)
at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:76)
...
Is there a way to find out what versions of slf4j I need to make it work with log4j 1.2.17?
-Adrian
Re: missing method in my slf4j after excluding Spark ZK log4j
Posted by Tathagata Das <ta...@gmail.com>.
This gives dependency tree in SBT (spark uses this).
https://github.com/jrudolph/sbt-dependency-graph
TD
On Mon, May 12, 2014 at 4:55 PM, Sean Owen <so...@cloudera.com> wrote:
> It sounds like you are doing everything right.
>
> NoSuchMethodError suggests it's finding log4j, just not the right
> version. That method is definitely in 1.2; it might have been removed
> in 2.x? (http://logging.apache.org/log4j/2.x/manual/migration.html)
>
> So I wonder if something is sneaking in log4j 2.x in your app? that's
> a first guess.
>
> I'd say consult mvn dependency:tree, but you're on sbt and I don't
> know the equivalent.
>
> On Mon, May 12, 2014 at 3:51 PM, Adrian Mocanu
> <am...@verticalscope.com> wrote:
> > Hey guys,
> >
> > I've asked before, in Spark 0.9 - I now use 0.9.1, about removing log4j
> > dependency and was told that it was gone. However I still find it part of
> > zookeeper imports. This is fine since I exclude it myself in the sbt
> file,
> > but another issue arises.
> >
> > I wonder if anyone else has run into this.
> >
> >
> >
> > Spark uses log4j v1.2.17 and slf4j-log4j12:1.7.2
> >
> > I use slf4j 1.7.5, logback 1.0.13, and log4joverslf4j v 1.7.5
> >
> >
> >
> > I think my slf4j 1.7.5 doesn't agree with what zookeeper expects in its
> > log4j v 1.2.17 because I get missing method error:
> >
> > java.lang.NoSuchMethodError:
> > org.apache.log4j.Logger.setLevel(Lorg/apache/log4j/Level;)V
> >
> > at
> >
> org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(AkkaUtils.scala:58)
> >
> > at
> >
> org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(AkkaUtils.scala:58)
> >
> > at scala.Option.map(Option.scala:145)
> >
> > at
> > org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:58)
> >
> > at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
> >
> > at
> > org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
> >
> > at
> >
> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:500)
> >
> > at
> >
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:76)
> >
> > ...
> >
> >
> >
> > Is there a way to find out what versions of slf4j I need to make it work
> > with log4j 1.2.17?
> >
> >
> >
> > -Adrian
> >
> >
>
RE: missing method in my slf4j after excluding Spark ZK log4j
Posted by Adrian Mocanu <am...@verticalscope.com>.
Yea, had to change the versions of slf4j and log4joversslf4j to1.7.6
-----Original Message-----
From: Sean Owen [mailto:sowen@cloudera.com]
Sent: May-12-14 7:55 PM
To: user@spark.apache.org
Subject: Re: missing method in my slf4j after excluding Spark ZK log4j
It sounds like you are doing everything right.
NoSuchMethodError suggests it's finding log4j, just not the right version. That method is definitely in 1.2; it might have been removed in 2.x? (http://logging.apache.org/log4j/2.x/manual/migration.html)
So I wonder if something is sneaking in log4j 2.x in your app? that's a first guess.
I'd say consult mvn dependency:tree, but you're on sbt and I don't know the equivalent.
On Mon, May 12, 2014 at 3:51 PM, Adrian Mocanu <am...@verticalscope.com> wrote:
> Hey guys,
>
> I've asked before, in Spark 0.9 - I now use 0.9.1, about removing
> log4j dependency and was told that it was gone. However I still find
> it part of zookeeper imports. This is fine since I exclude it myself
> in the sbt file, but another issue arises.
>
> I wonder if anyone else has run into this.
>
>
>
> Spark uses log4j v1.2.17 and slf4j-log4j12:1.7.2
>
> I use slf4j 1.7.5, logback 1.0.13, and log4joverslf4j v 1.7.5
>
>
>
> I think my slf4j 1.7.5 doesn't agree with what zookeeper expects in
> its log4j v 1.2.17 because I get missing method error:
>
> java.lang.NoSuchMethodError:
> org.apache.log4j.Logger.setLevel(Lorg/apache/log4j/Level;)V
>
> at
> org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(Akk
> aUtils.scala:58)
>
> at
> org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(Akk
> aUtils.scala:58)
>
> at scala.Option.map(Option.scala:145)
>
> at
> org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:58)
>
> at
> org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
>
> at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
>
> at
> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(Str
> eamingContext.scala:500)
>
> at
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.sc
> ala:76)
>
> ...
>
>
>
> Is there a way to find out what versions of slf4j I need to make it
> work with log4j 1.2.17?
>
>
>
> -Adrian
>
>
Re: missing method in my slf4j after excluding Spark ZK log4j
Posted by Sean Owen <so...@cloudera.com>.
It sounds like you are doing everything right.
NoSuchMethodError suggests it's finding log4j, just not the right
version. That method is definitely in 1.2; it might have been removed
in 2.x? (http://logging.apache.org/log4j/2.x/manual/migration.html)
So I wonder if something is sneaking in log4j 2.x in your app? that's
a first guess.
I'd say consult mvn dependency:tree, but you're on sbt and I don't
know the equivalent.
On Mon, May 12, 2014 at 3:51 PM, Adrian Mocanu
<am...@verticalscope.com> wrote:
> Hey guys,
>
> I've asked before, in Spark 0.9 - I now use 0.9.1, about removing log4j
> dependency and was told that it was gone. However I still find it part of
> zookeeper imports. This is fine since I exclude it myself in the sbt file,
> but another issue arises.
>
> I wonder if anyone else has run into this.
>
>
>
> Spark uses log4j v1.2.17 and slf4j-log4j12:1.7.2
>
> I use slf4j 1.7.5, logback 1.0.13, and log4joverslf4j v 1.7.5
>
>
>
> I think my slf4j 1.7.5 doesn't agree with what zookeeper expects in its
> log4j v 1.2.17 because I get missing method error:
>
> java.lang.NoSuchMethodError:
> org.apache.log4j.Logger.setLevel(Lorg/apache/log4j/Level;)V
>
> at
> org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(AkkaUtils.scala:58)
>
> at
> org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(AkkaUtils.scala:58)
>
> at scala.Option.map(Option.scala:145)
>
> at
> org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:58)
>
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
>
> at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
>
> at
> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:500)
>
> at
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:76)
>
> ...
>
>
>
> Is there a way to find out what versions of slf4j I need to make it work
> with log4j 1.2.17?
>
>
>
> -Adrian
>
>
Re: missing method in my slf4j after excluding Spark ZK log4j
Posted by Paul Brown <pr...@mult.ifario.us>.
Hi, Adrian --
If my memory serves, you need 1.7.7 of the various slf4j modules to avoid
that issue.
Best.
-- Paul
—
prb@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/
On Mon, May 12, 2014 at 7:51 AM, Adrian Mocanu <am...@verticalscope.com>wrote:
> Hey guys,
>
> I've asked before, in Spark 0.9 - I now use 0.9.1, about removing log4j
> dependency and was told that it was gone. However I still find it part of
> zookeeper imports. This is fine since I exclude it myself in the sbt file,
> but another issue arises.
>
> I wonder if anyone else has run into this.
>
>
>
> Spark uses log4j v1.2.17 and slf4j-log4j12:1.7.2
>
> I use slf4j 1.7.5, logback 1.0.13, and log4joverslf4j v 1.7.5
>
>
>
> I think my slf4j 1.7.5 doesn't agree with what zookeeper expects in its
> log4j v 1.2.17 because I get missing method error:
>
> java.lang.NoSuchMethodError:
> org.apache.log4j.Logger.setLevel(Lorg/apache/log4j/Level;)V
>
> at
> org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(AkkaUtils.scala:58)
>
> at
> org.apache.spark.util.AkkaUtils$$anonfun$createActorSystem$1.apply(AkkaUtils.scala:58)
>
> at scala.Option.map(Option.scala:145)
>
> at
> org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:58)
>
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
>
> at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
>
> at
> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:500)
>
> at
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:76)
>
> ...
>
>
>
> Is there a way to find out what versions of slf4j I need to make it work
> with log4j 1.2.17?
>
>
>
> -Adrian
>
>
>