You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/07/01 22:54:05 UTC
[jira] [Commented] (SPARK-8768) SparkSubmitSuite fails on Hadoop
1.x builds due to java.lang.VerifyError in Akka Protobuf
[ https://issues.apache.org/jira/browse/SPARK-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14610988#comment-14610988 ]
Josh Rosen commented on SPARK-8768:
-----------------------------------
We didn't notice this earlier because the Master Maven Pre-YARN build was misconfigured and was building against the wrong Hadoop versions.
> SparkSubmitSuite fails on Hadoop 1.x builds due to java.lang.VerifyError in Akka Protobuf
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-8768
> URL: https://issues.apache.org/jira/browse/SPARK-8768
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 1.5.0
> Reporter: Josh Rosen
> Priority: Blocker
>
> The end-to-end SparkSubmitSuite tests ("launch simple application with spark-submit", "include jars passed in through --jars", and "include jars passed in through --packages") are currently failing for the pre-YARN Hadoop builds.
> I managed to reproduce one of the Jenkins failures locally:
> {code}
> build/mvn -Phadoop-1 -Dhadoop.version=1.2.1 -Phive -Phive-thriftserver -Pkinesis-asl test -DwildcardSuites=org.apache.spark.deploy.SparkSubmitSuite -Dtest=none
> {code}
> Here's the output from unit-tests.log:
> {code}
> ===== TEST OUTPUT FOR o.a.s.deploy.SparkSubmitSuite: 'launch simple application with spark-submit' =====
> 15/07/01 13:39:58.964 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: Class path contains multiple SLF4J bindings.
> 15/07/01 13:39:58.964 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: Found binding in [jar:file:/Users/joshrosen/Documents/spark-2/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop1.2.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 15/07/01 13:39:58.965 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: Found binding in [jar:file:/Users/joshrosen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 15/07/01 13:39:58.965 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 15/07/01 13:39:58.965 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 15/07/01 13:39:58.966 redirect stderr for command ./bin/spark-submit INFO Utils: 15/07/01 13:39:58 INFO SparkContext: Running Spark version 1.5.0-SNAPSHOT
> 15/07/01 13:39:59.334 redirect stderr for command ./bin/spark-submit INFO Utils: 15/07/01 13:39:59 INFO SecurityManager: Changing view acls to: joshrosen
> 15/07/01 13:39:59.335 redirect stderr for command ./bin/spark-submit INFO Utils: 15/07/01 13:39:59 INFO SecurityManager: Changing modify acls to: joshrosen
> 15/07/01 13:39:59.335 redirect stderr for command ./bin/spark-submit INFO Utils: 15/07/01 13:39:59 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(joshrosen); users with modify permissions: Set(joshrosen)
> 15/07/01 13:39:59.898 redirect stderr for command ./bin/spark-submit INFO Utils: 15/07/01 13:39:59 INFO Slf4jLogger: Slf4jLogger started
> 15/07/01 13:39:59.934 redirect stderr for command ./bin/spark-submit INFO Utils: 15/07/01 13:39:59 INFO Remoting: Starting remoting
> 15/07/01 13:40:00.009 redirect stderr for command ./bin/spark-submit INFO Utils: 15/07/01 13:40:00 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.remote.default-remote-dispatcher-5] shutting down ActorSystem [sparkDriver]
> 15/07/01 13:40:00.009 redirect stderr for command ./bin/spark-submit INFO Utils: java.lang.VerifyError: class akka.remote.WireFormats$AkkaControlMessage overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
> 15/07/01 13:40:00.009 redirect stderr for command ./bin/spark-submit INFO Utils: at java.lang.ClassLoader.defineClass1(Native Method)
> 15/07/01 13:40:00.009 redirect stderr for command ./bin/spark-submit INFO Utils: at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> 15/07/01 13:40:00.009 redirect stderr for command ./bin/spark-submit INFO Utils: at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at java.security.AccessController.doPrivileged(Native Method)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at akka.remote.transport.AkkaPduProtobufCodec$.constructControlMessagePdu(AkkaPduCodec.scala:231)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at akka.remote.transport.AkkaPduProtobufCodec$.<init>(AkkaPduCodec.scala:153)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at akka.remote.transport.AkkaPduProtobufCodec$.<clinit>(AkkaPduCodec.scala)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:733)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:703)
> 15/07/01 13:40:00.010 redirect stderr for command ./bin/spark-submit INFO Utils: at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
> 15/07/01 13:40:00.011 redirect stderr for command ./bin/spark-submit INFO Utils: at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> 15/07/01 13:40:00.011 redirect stderr for command ./bin/spark-submit INFO Utils: at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> 15/07/01 13:40:00.011 redirect stderr for command ./bin/spark-submit INFO Utils: at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> {code}
> A Google search reveals that we've run into this problem before:
> https://mail-archives.apache.org/mod_mbox/spark-reviews/201407.mbox/%3Cgit-pr-1685-spark@git.apache.org%3E
> This may be caused by Protobuf version conflicts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org