You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ryan Compton (JIRA)" <ji...@apache.org> on 2014/05/29 20:23:01 UTC
[jira] [Comment Edited] (SPARK-1952) slf4j version conflicts with
pig
[ https://issues.apache.org/jira/browse/SPARK-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012650#comment-14012650 ]
Ryan Compton edited comment on SPARK-1952 at 5/29/14 6:22 PM:
--------------------------------------------------------------
Thanks so much!
Here's the fix I had to make (removed jul-to-slf4j and jcl-over-slf4j)
{code}
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 29dcd86..a113ead 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -336,8 +336,8 @@ object SparkBuild extends Build {
"log4j" % "log4j" % "1.2.17",
"org.slf4j" % "slf4j-api" % slf4jVersion,
"org.slf4j" % "slf4j-log4j12" % slf4jVersion,
- "org.slf4j" % "jul-to-slf4j" % slf4jVersion,
- "org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
+// "org.slf4j" % "jul-to-slf4j" % slf4jVersion,
+// "org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
"commons-daemon" % "commons-daemon" % "1.0.10", // workaround for bug HADOOP-9407
"com.ning" % "compress-lzf" % "1.0.0",
"org.xerial.snappy" % "snappy-java" % "1.0.5",
{code}
This led to several of these during compilation:
{code}
[warn] Class org.apache.commons.logging.Log not found - continuing with a stub.
{code}
But it did successfully remove the "SLF4JLocationAwareLog.class"
{code}
[rfcompton@node19 spark-1.0.0]$ jar tvf assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop0.20.2-cdh3u4.jar | grep -i slf4j | grep Location
455 Mon Mar 25 21:49:22 PDT 2013 org/slf4j/spi/LocationAwareLogger.class
479 Fri Dec 13 16:44:40 PST 2013 parquet/org/slf4j/spi/LocationAwareLogger.class
{code}
For reference, there was no "\*-\*-slf4j" in 0.9.1 :
{code}
[rfcompton@node19 spark-0.9.1-bin-hadoop1]$ cat project/SparkBuild.scala | grep slf4
val slf4jVersion = "1.7.2"
"org.slf4j" % "slf4j-api" % slf4jVersion,
"org.slf4j" % "slf4j-log4j12" % slf4jVersion,
"org.spark-project.akka" %% "akka-slf4j" % "2.2.3-shaded-protobuf" excludeAll(excludeNetty),
{code}
I'm the only guy in my lab using Spark so I have no idea if it's common to keep Pig UDFs in the same jar as Spark code. I'm going to assume it is and submit a pull request now.
was (Author: rcompton):
Thanks so much!
Here's the fix I had to make (removed jul-to-slf4j and jcl-over-slf4j)
{code}
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 29dcd86..a113ead 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -336,8 +336,8 @@ object SparkBuild extends Build {
"log4j" % "log4j" % "1.2.17",
"org.slf4j" % "slf4j-api" % slf4jVersion,
"org.slf4j" % "slf4j-log4j12" % slf4jVersion,
- "org.slf4j" % "jul-to-slf4j" % slf4jVersion,
- "org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
+// "org.slf4j" % "jul-to-slf4j" % slf4jVersion,
+// "org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
"commons-daemon" % "commons-daemon" % "1.0.10", // workaround for bug HADOOP-9407
"com.ning" % "compress-lzf" % "1.0.0",
"org.xerial.snappy" % "snappy-java" % "1.0.5",
{code}
This led to several of these during compilation:
{code}
[warn] Class org.apache.commons.logging.Log not found - continuing with a stub.
{code}
But it did successfully remove the "SLF4JLocationAwareLog.class"
{code}
[rfcompton@node19 spark-1.0.0]$ jar tvf assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop0.20.2-cdh3u4.jar | grep -i slf4j | grep Location
455 Mon Mar 25 21:49:22 PDT 2013 org/slf4j/spi/LocationAwareLogger.class
479 Fri Dec 13 16:44:40 PST 2013 parquet/org/slf4j/spi/LocationAwareLogger.class
{code}
For reference, there was no "*-*-slf4j" in 0.9.1 :
{code}
[rfcompton@node19 spark-0.9.1-bin-hadoop1]$ cat project/SparkBuild.scala | grep slf4
val slf4jVersion = "1.7.2"
"org.slf4j" % "slf4j-api" % slf4jVersion,
"org.slf4j" % "slf4j-log4j12" % slf4jVersion,
"org.spark-project.akka" %% "akka-slf4j" % "2.2.3-shaded-protobuf" excludeAll(excludeNetty),
{code}
I'm the only guy in my lab using Spark so I have no idea if it's common to keep Pig UDFs in the same jar as Spark code. I'm going to assume it is and submit a pull request now.
> slf4j version conflicts with pig
> --------------------------------
>
> Key: SPARK-1952
> URL: https://issues.apache.org/jira/browse/SPARK-1952
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Environment: pig 12.1 on Cloudera Hadoop, CDH3
> Reporter: Ryan Compton
> Labels: pig, slf4j
>
> Upgrading from Spark-0.9.1 to Spark-1.0.0 causes all Pig scripts to fail when they "register" a jar containing Spark. The error appears to be related to org.slf4j.spi.LocationAwareLogger.log.
> {code}
> Caused by: java.lang.RuntimeException: Could not resolve error that
> occured when launching map reduce job: java.lang.NoSuchMethodError:
> org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$JobControlThreadExceptionHandler.uncaughtException(MapReduceLauncher.java:598)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1874)
> {code}
> To reproduce: compile Spark via $ SPARK_HADOOP_VERSION=0.20.2-cdh3u4 sbt/sbt assembly and register the resulting jar into a pig script. E.g.
> {code}
> REGISTER /usr/share/spark-1.0.0/assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop0.20.2-cdh3u4.jar;
> data0 = LOAD 'data' USING PigStorage();
> ttt = LIMIT data0 10;
> DUMP ttt;
> {code}
> The Spark-1.0 jar includes some slf4j dependencies that were not present in 0.9.1
> {code}
> rfcompton@node19 /u/s/o/s/a/t/scala-2.10> jar tvf spark-assembly-1.0.0-SNAPSHOT-hadoop0.20.2-cdh3u4.jar | grep -i "slf" | grep LocationAware
> 3259 Mon Mar 25 21:49:34 PDT 2013 org/apache/commons/logging/impl/SLF4JLocationAwareLog.class
> 455 Mon Mar 25 21:49:22 PDT 2013 org/slf4j/spi/LocationAwareLogger.class
> 479 Fri Dec 13 16:44:40 PST 2013 parquet/org/slf4j/spi/LocationAwareLogger.class
> {code}
> vs.
> {code}
> rfcompton@node19 /u/s/o/s/a/t/scala-2.10> jar tvf spark-assembly-0.9.1-hadoop0.20.2-cdh3u3.jar | grep -i "slf" | grep LocationAware
> 455 Mon Mar 25 21:49:22 PDT 2013 org/slf4j/spi/LocationAwareLogger.class
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)