You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Emre Sevinc <em...@gmail.com> on 2014/12/05 12:52:54 UTC
How can I compile only the core and streaming (so that I can get test
utilities of streaming)?
Hello,
I'm currently developing a Spark Streaming application and trying to write
my first unit test. I've used Java for this application, and I also need
use Java (and JUnit) for writing unit tests.
I could not find any documentation that focuses on Spark Streaming unit
testing, all I could find was the Java based unit tests in Spark Streaming
source code:
https://github.com/apache/spark/blob/branch-1.1/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
<https://webmail.cronos.be/owa/redir.aspx?C=Jqwj6T-dSk63ezaf0lH8P7NV7ZXa4NEIco3EZ9VQhfNEvS3bxfKu6ZKfDqYOFlFaCAvyOEVymdw.&URL=https%3a%2f%2fgithub.com%2fapache%2fspark%2fblob%2fbranch-1.1%2fstreaming%2fsrc%2ftest%2fjava%2forg%2fapache%2fspark%2fstreaming%2fJavaAPISuite.java>
that depends on a Scala file:
https://github.com/apache/spark/blob/branch-1.1/streaming/src/test/java/org/apache/spark/streaming/JavaTestUtils.scala
<https://webmail.cronos.be/owa/redir.aspx?C=Jqwj6T-dSk63ezaf0lH8P7NV7ZXa4NEIco3EZ9VQhfNEvS3bxfKu6ZKfDqYOFlFaCAvyOEVymdw.&URL=https%3a%2f%2fgithub.com%2fapache%2fspark%2fblob%2fbranch-1.1%2fstreaming%2fsrc%2ftest%2fjava%2forg%2fapache%2fspark%2fstreaming%2fJavaTestUtils.scala>
which, in turn, depends on the Scala test files in
https://github.com/apache/spark/tree/branch-1.1/streaming/src/test/scala/org/apache/spark/streaming
<https://webmail.cronos.be/owa/redir.aspx?C=Jqwj6T-dSk63ezaf0lH8P7NV7ZXa4NEIco3EZ9VQhfNEvS3bxfKu6ZKfDqYOFlFaCAvyOEVymdw.&URL=https%3a%2f%2fgithub.com%2fapache%2fspark%2ftree%2fbranch-1.1%2fstreaming%2fsrc%2ftest%2fscala%2forg%2fapache%2fspark%2fstreaming>
So I thought that I could grab the Spark source code, switch to branch-1.1
branch and then only compile 'core' and 'streaming' modules, hopefully
ending up with the compiled classes (or jar files) of the Streaming test
utilities, so that I can import them in my Java based Spark Streaming
application.
However, trying to build it via the following command line failed:
mvn -pl core,streaming package
You can see the full output at the end of this message.
Any ideas how to progress?
Full output of the build:
emre@emre-ubuntu:~/code/spark$ mvn -pl core,streaming package
[INFO] Scanning for projects...
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Core
[INFO] Spark Project Streaming
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Building Spark Project Core 1.1.2-SNAPSHOT
[INFO]
------------------------------------------------------------------------
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.pom
(5 KB at 5.8 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar
(31 KB at 200.4 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.3/commons-math3-3.3.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.3/commons-math3-3.3.pom
(24 KB at 178.9 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/spark-project/akka/akka-testkit_2.10/2.2.3-shaded-protobuf/akka-testkit_2.10-2.2.3-shaded-protobuf.pom
Downloaded:
https://repo1.maven.org/maven2/org/spark-project/akka/akka-testkit_2.10/2.2.3-shaded-protobuf/akka-testkit_2.10-2.2.3-shaded-protobuf.pom
(3 KB at 22.5 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/scala-lang/scalap/2.10.4/scalap-2.10.4.pom
Downloaded:
https://repo1.maven.org/maven2/org/scala-lang/scalap/2.10.4/scalap-2.10.4.pom
(2 KB at 19.2 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/derby/derby/10.4.2.0/derby-10.4.2.0.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/derby/derby/10.4.2.0/derby-10.4.2.0.pom
(2 KB at 14.9 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.pom
Downloaded:
https://repo1.maven.org/maven2/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.pom
(1010 B at 4.1 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.pom
Downloaded:
https://repo1.maven.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.pom
(5 KB at 42.9 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/easymock/easymock-parent/3.1/easymock-parent-3.1.pom
Downloaded:
https://repo1.maven.org/maven2/org/easymock/easymock-parent/3.1/easymock-parent-3.1.pom
(13 KB at 133.8 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/easymock/easymock/3.1/easymock-3.1.pom
Downloaded:
https://repo1.maven.org/maven2/org/easymock/easymock/3.1/easymock-3.1.pom
(6 KB at 38.8 KB/sec)
Downloading:
https://repo1.maven.org/maven2/cglib/cglib-nodep/2.2.2/cglib-nodep-2.2.2.pom
Downloaded:
https://repo1.maven.org/maven2/cglib/cglib-nodep/2.2.2/cglib-nodep-2.2.2.pom
(2 KB at 9.9 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.3/commons-math3-3.3.jar
Downloading:
https://repo1.maven.org/maven2/org/spark-project/akka/akka-testkit_2.10/2.2.3-shaded-protobuf/akka-testkit_2.10-2.2.3-shaded-protobuf.jar
Downloading:
https://repo1.maven.org/maven2/org/scala-lang/scalap/2.10.4/scalap-2.10.4.jar
Downloading:
https://repo1.maven.org/maven2/org/apache/derby/derby/10.4.2.0/derby-10.4.2.0.jar
Downloading:
https://repo1.maven.org/maven2/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.jar
Downloaded:
https://repo1.maven.org/maven2/org/scala-lang/scalap/2.10.4/scalap-2.10.4.jar
(834 KB at 146.7 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.jar
Downloaded:
https://repo1.maven.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.jar
(11 KB at 65.8 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/easymock/easymock/3.1/easymock-3.1.jar
Downloaded:
https://repo1.maven.org/maven2/org/easymock/easymock/3.1/easymock-3.1.jar
(110 KB at 125.5 KB/sec)
Downloading:
https://repo1.maven.org/maven2/cglib/cglib-nodep/2.2.2/cglib-nodep-2.2.2.jar
Downloaded:
https://repo1.maven.org/maven2/cglib/cglib-nodep/2.2.2/cglib-nodep-2.2.2.jar
(320 KB at 205.4 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/spark-project/akka/akka-testkit_2.10/2.2.3-shaded-protobuf/akka-testkit_2.10-2.2.3-shaded-protobuf.jar
(233 KB at 23.9 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.3/commons-math3-3.3.jar
(1907 KB at 192.1 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.jar
(1461 KB at 113.2 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/apache/derby/derby/10.4.2.0/derby-10.4.2.0.jar
(2390 KB at 131.1 KB/sec)
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-versions) @
spark-core_2.10 ---
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-api/1.3.1/enforcer-api-1.3.1.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-api/1.3.1/enforcer-api-1.3.1.pom
(3 KB at 34.0 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-rules/1.3.1/enforcer-rules-1.3.1.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-rules/1.3.1/enforcer-rules-1.3.1.pom
(4 KB at 66.6 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-api/1.3.1/enforcer-api-1.3.1.jar
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-rules/1.3.1/enforcer-rules-1.3.1.jar
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-api/1.3.1/enforcer-api-1.3.1.jar
(10 KB at 75.7 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-rules/1.3.1/enforcer-rules-1.3.1.jar
(86 KB at 292.3 KB/sec)
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-scala-sources) @
spark-core_2.10 ---
[INFO] Source directory: /home/emre/code/spark/core/src/main/scala added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
spark-core_2.10 ---
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/shared/maven-artifact-resolver/1.0/maven-artifact-resolver-1.0.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/shared/maven-artifact-resolver/1.0/maven-artifact-resolver-1.0.pom
(6 KB at 65.8 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-resources/1.0-alpha-7/plexus-resources-1.0-alpha-7.pom
Downloaded:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-resources/1.0-alpha-7/plexus-resources-1.0-alpha-7.pom
(2 KB at 21.4 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.pom
(11 KB at 143.3 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/shared/maven-artifact-resolver/1.0/maven-artifact-resolver-1.0.jar
Downloading:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.12/plexus-interpolation-1.12.jar
Downloading:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-resources/1.0-alpha-7/plexus-resources-1.0-alpha-7.jar
Downloading:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/1.5.15/plexus-utils-1.5.15.jar
Downloading:
https://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.jar
Dec 05, 2014 12:30:22 PM
org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec
execute
INFO: I/O exception
(org.apache.maven.wagon.providers.http.httpclient.NoHttpResponseException)
caught when processing request: The target server failed to respond
Dec 05, 2014 12:30:22 PM
org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec
execute
INFO: Retrying request
Downloaded:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-resources/1.0-alpha-7/plexus-resources-1.0-alpha-7.jar
(23 KB at 45.6 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/shared/maven-artifact-resolver/1.0/maven-artifact-resolver-1.0.jar
(14 KB at 13.3 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/1.5.15/plexus-utils-1.5.15.jar
(223 KB at 122.9 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.12/plexus-interpolation-1.12.jar
(60 KB at 29.6 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.jar
(439 KB at 159.9 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/apache-jar-resource-bundle/1.4/apache-jar-resource-bundle-1.4.jar
Downloaded:
https://repo1.maven.org/maven2/org/apache/apache-jar-resource-bundle/1.4/apache-jar-resource-bundle-1.4.jar
(13 KB at 98.6 KB/sec)
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (default) @ spark-core_2.10 ---
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0.11/maven-plugin-api-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0.11/maven-plugin-api-2.0.11.pom
(2 KB at 19.6 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven/2.0.11/maven-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven/2.0.11/maven-2.0.11.pom
(22 KB at 187.1 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-parent/12/maven-parent-12.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-parent/12/maven-parent-12.pom
(22 KB at 58.0 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-project/2.0.11/maven-project-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-project/2.0.11/maven-project-2.0.11.pom
(3 KB at 46.8 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-settings/2.0.11/maven-settings-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-settings/2.0.11/maven-settings-2.0.11.pom
(3 KB at 40.2 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-model/2.0.11/maven-model-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-model/2.0.11/maven-model-2.0.11.pom
(4 KB at 59.8 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.pom
Downloaded:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.pom
(2 KB at 26.5 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-profile/2.0.11/maven-profile-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-profile/2.0.11/maven-profile-2.0.11.pom
(2 KB at 31.7 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-artifact-manager/2.0.11/maven-artifact-manager-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-artifact-manager/2.0.11/maven-artifact-manager-2.0.11.pom
(3 KB at 10.9 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-repository-metadata/2.0.11/maven-repository-metadata-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-repository-metadata/2.0.11/maven-repository-metadata-2.0.11.pom
(3 KB at 27.2 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-artifact/2.0.11/maven-artifact-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-artifact/2.0.11/maven-artifact-2.0.11.pom
(2 KB at 22.8 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/maven-plugin-registry/2.0.11/maven-plugin-registry-2.0.11.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/maven-plugin-registry/2.0.11/maven-plugin-registry-2.0.11.pom
(2 KB at 30.9 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/ant/ant/1.8.2/ant-1.8.2.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/ant/ant/1.8.2/ant-1.8.2.pom (10
KB at 100.5 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/ant/ant-parent/1.8.2/ant-parent-1.8.2.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/ant/ant-parent/1.8.2/ant-parent-1.8.2.pom
(5 KB at 84.8 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.pom
(3 KB at 43.0 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/ant/ant/1.8.2/ant-1.8.2.jar
Downloading:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.jar
Downloading:
https://repo1.maven.org/maven2/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.jar
Downloaded:
https://repo1.maven.org/maven2/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.jar
(13 KB at 138.3 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.jar
(35 KB at 256.7 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/apache/ant/ant/1.8.2/ant-1.8.2.jar (1889
KB at 416.0 KB/sec)
[WARNING] Parameter tasks is deprecated, use target instead
[INFO] Executing tasks
main:
[unzip] Expanding:
/home/emre/code/spark/python/lib/py4j-0.8.2.1-src.zip into
/home/emre/code/spark/python/build
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
spark-core_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 9 resources
[INFO] Copying 21 resources
[INFO] Copying 7 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
spark-core_2.10 ---
[WARNING] Zinc server is not available at port 3030 - reverting to normal
incremental compile
[INFO] Using incremental compilation
[INFO] 'compiler-interface' not yet compiled for Scala 2.10.4. Compiling...
[INFO] Compilation completed in 10.873 s
[INFO] compiler plugin:
BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
Downloading:
https://repo1.maven.org/maven2/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar
Downloaded:
https://repo1.maven.org/maven2/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar
(1855 KB at 485.7 KB/sec)
[INFO] Compiling 390 Scala sources and 28 Java sources to
/home/emre/code/spark/core/target/scala-2.10/classes...
[WARNING] warning: [options] bootstrap class path not set in conjunction
with -source 1.6
[WARNING] 1 warning
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
spark-core_2.10 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 28 source files to
/home/emre/code/spark/core/target/scala-2.10/classes
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-test-source
(add-scala-test-sources) @ spark-core_2.10 ---
[INFO] Test Source directory: /home/emre/code/spark/core/src/test/scala
added.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources)
@ spark-core_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile-first)
@ spark-core_2.10 ---
[WARNING] Zinc server is not available at port 3030 - reverting to normal
incremental compile
[INFO] Using incremental compilation
[INFO] compiler plugin:
BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
[INFO] Compiling 111 Scala sources and 3 Java sources to
/home/emre/code/spark/core/target/scala-2.10/test-classes...
[WARNING] warning: [options] bootstrap class path not set in conjunction
with -source 1.6
[WARNING] Note:
/home/emre/code/spark/core/src/test/java/org/apache/spark/JavaAPISuite.java
uses unchecked or unsafe operations.
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[WARNING] 1 warning
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
spark-core_2.10 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 3 source files to
/home/emre/code/spark/core/target/scala-2.10/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ spark-core_2.10
---
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.pom
(6 KB at 14.4 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.pom
(3 KB at 13.3 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.pom
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.pom
(3 KB at 40.1 KB/sec)
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.jar
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.jar
Downloading:
https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.jar
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.jar
(39 KB at 184.9 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.jar
(144 KB at 141.3 KB/sec)
Downloaded:
https://repo1.maven.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.jar
(260 KB at 166.2 KB/sec)
[INFO] Tests are skipped.
[INFO]
[INFO] --- scalatest-maven-plugin:1.0-RC2:test (test) @ spark-core_2.10 ---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
MaxPermSize=512m; support was removed in 8.0
Discovery starting.
Discovery completed in 7 seconds, 167 milliseconds.
Run starting. Expected test count is: 809
ExternalSorterSuite:
- empty data stream
- few elements per partition
- empty partitions with spilling
- empty partitions with spilling, bypass merge-sort
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
- spilling in local cluster *** FAILED ***
org.apache.spark.SparkException: Job aborted due to stage failure: Master
removed our application: FAILED
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
...
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
- spilling in local cluster with many reduce tasks *** FAILED ***
org.apache.spark.SparkException: Job aborted due to stage failure: Master
removed our application: FAILED
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
...
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
- cleanup of intermediate files in sorter
- cleanup of intermediate files in sorter, bypass merge-sort
- cleanup of intermediate files in sorter if there are errors
- cleanup of intermediate files in sorter if there are errors, bypass
merge-sort
- cleanup of intermediate files in shuffle
- cleanup of intermediate files in shuffle with errors
- no partial aggregation or sorting
- partial aggregation without spill
- partial aggregation with spill, no ordering
- partial aggregation with spill, with ordering
- sorting without aggregation, no spill
- sorting without aggregation, with spill
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such
file or directory
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Core ................................. FAILURE [03:36
min]
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 03:37 min
[INFO] Finished at: 2014-12-05T12:33:36+01:00
[INFO] Final Memory: 37M/597M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.scalatest:scalatest-maven-plugin:1.0-RC2:test (test) on project
spark-core_2.10: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
--
Emre Sevinç
Re: How can I compile only the core and streaming (so that I can get
test utilities of streaming)?
Posted by Ted Yu <yu...@gmail.com>.
I don't think 'sbt assembly' would touch local maven repo for Spark.
Looking at dependency:tree output:
[INFO] org.apache.spark:spark-streaming_2.10:jar:1.1.0-SNAPSHOT
[INFO] +- org.apache.spark:spark-core_2.10:jar:1.1.0-SNAPSHOT:compile
spark-streaming only depends on spark-core other than third party jars.
That's why the command works.
Cheers
On Fri, Dec 5, 2014 at 7:52 AM, Emre Sevinc <em...@gmail.com> wrote:
> Hello,
>
> Specifying '-DskipTests' on commandline worked, though I can't be sure
> whether first running 'sbt assembly' also contributed to the solution.
> (I've tried 'sbt assembly' because branch-1.1's README says to use sbt).
>
> Thanks for the answer.
>
> Kind regards,
> Emre Sevinç
>
Re: How can I compile only the core and streaming (so that I can get
test utilities of streaming)?
Posted by Emre Sevinc <em...@gmail.com>.
Hello,
Specifying '-DskipTests' on commandline worked, though I can't be sure
whether first running 'sbt assembly' also contributed to the solution.
(I've tried 'sbt assembly' because branch-1.1's README says to use sbt).
Thanks for the answer.
Kind regards,
Emre Sevinç
Re: How can I compile only the core and streaming (so that I can get test utilities of streaming)?
Posted by Ted Yu <yu...@gmail.com>.
Please specify '-DskipTests' on commandline.
Cheers
On Dec 5, 2014, at 3:52 AM, Emre Sevinc <em...@gmail.com> wrote:
> Hello,
>
> I'm currently developing a Spark Streaming application and trying to write my first unit test. I've used Java for this application, and I also need use Java (and JUnit) for writing unit tests.
>
> I could not find any documentation that focuses on Spark Streaming unit testing, all I could find was the Java based unit tests in Spark Streaming source code:
>
> https://github.com/apache/spark/blob/branch-1.1/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
>
> that depends on a Scala file:
>
> https://github.com/apache/spark/blob/branch-1.1/streaming/src/test/java/org/apache/spark/streaming/JavaTestUtils.scala
>
> which, in turn, depends on the Scala test files in
>
> https://github.com/apache/spark/tree/branch-1.1/streaming/src/test/scala/org/apache/spark/streaming
>
> So I thought that I could grab the Spark source code, switch to branch-1.1 branch and then only compile 'core' and 'streaming' modules, hopefully ending up with the compiled classes (or jar files) of the Streaming test utilities, so that I can import them in my Java based Spark Streaming application.
>
> However, trying to build it via the following command line failed:
>
> mvn -pl core,streaming package
>
> You can see the full output at the end of this message.
>
> Any ideas how to progress?
>
> Full output of the build:
>
> emre@emre-ubuntu:~/code/spark$ mvn -pl core,streaming package
> [INFO] Scanning for projects...
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Build Order:
> [INFO]
> [INFO] Spark Project Core
> [INFO] Spark Project Streaming
> [INFO]
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Spark Project Core 1.1.2-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.pom (5 KB at 5.8 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar (31 KB at 200.4 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.3/commons-math3-3.3.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.3/commons-math3-3.3.pom (24 KB at 178.9 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/spark-project/akka/akka-testkit_2.10/2.2.3-shaded-protobuf/akka-testkit_2.10-2.2.3-shaded-protobuf.pom
> Downloaded: https://repo1.maven.org/maven2/org/spark-project/akka/akka-testkit_2.10/2.2.3-shaded-protobuf/akka-testkit_2.10-2.2.3-shaded-protobuf.pom (3 KB at 22.5 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/scala-lang/scalap/2.10.4/scalap-2.10.4.pom
> Downloaded: https://repo1.maven.org/maven2/org/scala-lang/scalap/2.10.4/scalap-2.10.4.pom (2 KB at 19.2 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/derby/derby/10.4.2.0/derby-10.4.2.0.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/derby/derby/10.4.2.0/derby-10.4.2.0.pom (2 KB at 14.9 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.pom
> Downloaded: https://repo1.maven.org/maven2/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.pom (1010 B at 4.1 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.pom
> Downloaded: https://repo1.maven.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.pom (5 KB at 42.9 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/easymock/easymock-parent/3.1/easymock-parent-3.1.pom
> Downloaded: https://repo1.maven.org/maven2/org/easymock/easymock-parent/3.1/easymock-parent-3.1.pom (13 KB at 133.8 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/easymock/easymock/3.1/easymock-3.1.pom
> Downloaded: https://repo1.maven.org/maven2/org/easymock/easymock/3.1/easymock-3.1.pom (6 KB at 38.8 KB/sec)
> Downloading: https://repo1.maven.org/maven2/cglib/cglib-nodep/2.2.2/cglib-nodep-2.2.2.pom
> Downloaded: https://repo1.maven.org/maven2/cglib/cglib-nodep/2.2.2/cglib-nodep-2.2.2.pom (2 KB at 9.9 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.3/commons-math3-3.3.jar
> Downloading: https://repo1.maven.org/maven2/org/spark-project/akka/akka-testkit_2.10/2.2.3-shaded-protobuf/akka-testkit_2.10-2.2.3-shaded-protobuf.jar
> Downloading: https://repo1.maven.org/maven2/org/scala-lang/scalap/2.10.4/scalap-2.10.4.jar
> Downloading: https://repo1.maven.org/maven2/org/apache/derby/derby/10.4.2.0/derby-10.4.2.0.jar
> Downloading: https://repo1.maven.org/maven2/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.jar
> Downloaded: https://repo1.maven.org/maven2/org/scala-lang/scalap/2.10.4/scalap-2.10.4.jar (834 KB at 146.7 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.jar
> Downloaded: https://repo1.maven.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.jar (11 KB at 65.8 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/easymock/easymock/3.1/easymock-3.1.jar
> Downloaded: https://repo1.maven.org/maven2/org/easymock/easymock/3.1/easymock-3.1.jar (110 KB at 125.5 KB/sec)
> Downloading: https://repo1.maven.org/maven2/cglib/cglib-nodep/2.2.2/cglib-nodep-2.2.2.jar
> Downloaded: https://repo1.maven.org/maven2/cglib/cglib-nodep/2.2.2/cglib-nodep-2.2.2.jar (320 KB at 205.4 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/spark-project/akka/akka-testkit_2.10/2.2.3-shaded-protobuf/akka-testkit_2.10-2.2.3-shaded-protobuf.jar (233 KB at 23.9 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.3/commons-math3-3.3.jar (1907 KB at 192.1 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.jar (1461 KB at 113.2 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/apache/derby/derby/10.4.2.0/derby-10.4.2.0.jar (2390 KB at 131.1 KB/sec)
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-versions) @ spark-core_2.10 ---
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-api/1.3.1/enforcer-api-1.3.1.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-api/1.3.1/enforcer-api-1.3.1.pom (3 KB at 34.0 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-rules/1.3.1/enforcer-rules-1.3.1.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-rules/1.3.1/enforcer-rules-1.3.1.pom (4 KB at 66.6 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-api/1.3.1/enforcer-api-1.3.1.jar
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-rules/1.3.1/enforcer-rules-1.3.1.jar
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-api/1.3.1/enforcer-api-1.3.1.jar (10 KB at 75.7 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/enforcer/enforcer-rules/1.3.1/enforcer-rules-1.3.1.jar (86 KB at 292.3 KB/sec)
> [INFO]
> [INFO] --- build-helper-maven-plugin:1.8:add-source (add-scala-sources) @ spark-core_2.10 ---
> [INFO] Source directory: /home/emre/code/spark/core/src/main/scala added.
> [INFO]
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-core_2.10 ---
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/shared/maven-artifact-resolver/1.0/maven-artifact-resolver-1.0.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/shared/maven-artifact-resolver/1.0/maven-artifact-resolver-1.0.pom (6 KB at 65.8 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-resources/1.0-alpha-7/plexus-resources-1.0-alpha-7.pom
> Downloaded: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-resources/1.0-alpha-7/plexus-resources-1.0-alpha-7.pom (2 KB at 21.4 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.pom (11 KB at 143.3 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/shared/maven-artifact-resolver/1.0/maven-artifact-resolver-1.0.jar
> Downloading: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.12/plexus-interpolation-1.12.jar
> Downloading: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-resources/1.0-alpha-7/plexus-resources-1.0-alpha-7.jar
> Downloading: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/1.5.15/plexus-utils-1.5.15.jar
> Downloading: https://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.jar
> Dec 05, 2014 12:30:22 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
> INFO: I/O exception (org.apache.maven.wagon.providers.http.httpclient.NoHttpResponseException) caught when processing request: The target server failed to respond
> Dec 05, 2014 12:30:22 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
> INFO: Retrying request
> Downloaded: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-resources/1.0-alpha-7/plexus-resources-1.0-alpha-7.jar (23 KB at 45.6 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/shared/maven-artifact-resolver/1.0/maven-artifact-resolver-1.0.jar (14 KB at 13.3 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/1.5.15/plexus-utils-1.5.15.jar (223 KB at 122.9 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.12/plexus-interpolation-1.12.jar (60 KB at 29.6 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.jar (439 KB at 159.9 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/apache-jar-resource-bundle/1.4/apache-jar-resource-bundle-1.4.jar
> Downloaded: https://repo1.maven.org/maven2/org/apache/apache-jar-resource-bundle/1.4/apache-jar-resource-bundle-1.4.jar (13 KB at 98.6 KB/sec)
> [INFO]
> [INFO] --- maven-antrun-plugin:1.7:run (default) @ spark-core_2.10 ---
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0.11/maven-plugin-api-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0.11/maven-plugin-api-2.0.11.pom (2 KB at 19.6 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven/2.0.11/maven-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven/2.0.11/maven-2.0.11.pom (22 KB at 187.1 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-parent/12/maven-parent-12.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-parent/12/maven-parent-12.pom (22 KB at 58.0 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-project/2.0.11/maven-project-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-project/2.0.11/maven-project-2.0.11.pom (3 KB at 46.8 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-settings/2.0.11/maven-settings-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-settings/2.0.11/maven-settings-2.0.11.pom (3 KB at 40.2 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-model/2.0.11/maven-model-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-model/2.0.11/maven-model-2.0.11.pom (4 KB at 59.8 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.pom
> Downloaded: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.pom (2 KB at 26.5 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-profile/2.0.11/maven-profile-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-profile/2.0.11/maven-profile-2.0.11.pom (2 KB at 31.7 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-artifact-manager/2.0.11/maven-artifact-manager-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-artifact-manager/2.0.11/maven-artifact-manager-2.0.11.pom (3 KB at 10.9 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-repository-metadata/2.0.11/maven-repository-metadata-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-repository-metadata/2.0.11/maven-repository-metadata-2.0.11.pom (3 KB at 27.2 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-artifact/2.0.11/maven-artifact-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-artifact/2.0.11/maven-artifact-2.0.11.pom (2 KB at 22.8 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/maven-plugin-registry/2.0.11/maven-plugin-registry-2.0.11.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/maven-plugin-registry/2.0.11/maven-plugin-registry-2.0.11.pom (2 KB at 30.9 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/ant/ant/1.8.2/ant-1.8.2.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/ant/ant/1.8.2/ant-1.8.2.pom (10 KB at 100.5 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/ant/ant-parent/1.8.2/ant-parent-1.8.2.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/ant/ant-parent/1.8.2/ant-parent-1.8.2.pom (5 KB at 84.8 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.pom (3 KB at 43.0 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/ant/ant/1.8.2/ant-1.8.2.jar
> Downloading: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.jar
> Downloading: https://repo1.maven.org/maven2/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.jar
> Downloaded: https://repo1.maven.org/maven2/org/apache/ant/ant-launcher/1.8.2/ant-launcher-1.8.2.jar (13 KB at 138.3 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.jar (35 KB at 256.7 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/apache/ant/ant/1.8.2/ant-1.8.2.jar (1889 KB at 416.0 KB/sec)
> [WARNING] Parameter tasks is deprecated, use target instead
> [INFO] Executing tasks
>
> main:
> [unzip] Expanding: /home/emre/code/spark/python/lib/py4j-0.8.2.1-src.zip into /home/emre/code/spark/python/build
> [INFO] Executed tasks
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-core_2.10 ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 9 resources
> [INFO] Copying 21 resources
> [INFO] Copying 7 resources
> [INFO] Copying 3 resources
> [INFO]
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ spark-core_2.10 ---
> [WARNING] Zinc server is not available at port 3030 - reverting to normal incremental compile
> [INFO] Using incremental compilation
> [INFO] 'compiler-interface' not yet compiled for Scala 2.10.4. Compiling...
> [INFO] Compilation completed in 10.873 s
> [INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
> Downloading: https://repo1.maven.org/maven2/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar
> Downloaded: https://repo1.maven.org/maven2/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar (1855 KB at 485.7 KB/sec)
> [INFO] Compiling 390 Scala sources and 28 Java sources to /home/emre/code/spark/core/target/scala-2.10/classes...
> [WARNING] warning: [options] bootstrap class path not set in conjunction with -source 1.6
> [WARNING] 1 warning
> [INFO]
> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ spark-core_2.10 ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 28 source files to /home/emre/code/spark/core/target/scala-2.10/classes
> [INFO]
> [INFO] --- build-helper-maven-plugin:1.8:add-test-source (add-scala-test-sources) @ spark-core_2.10 ---
> [INFO] Test Source directory: /home/emre/code/spark/core/src/test/scala added.
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ spark-core_2.10 ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 4 resources
> [INFO] Copying 3 resources
> [INFO]
> [INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile-first) @ spark-core_2.10 ---
> [WARNING] Zinc server is not available at port 3030 - reverting to normal incremental compile
> [INFO] Using incremental compilation
> [INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
> [INFO] Compiling 111 Scala sources and 3 Java sources to /home/emre/code/spark/core/target/scala-2.10/test-classes...
> [WARNING] warning: [options] bootstrap class path not set in conjunction with -source 1.6
> [WARNING] Note: /home/emre/code/spark/core/src/test/java/org/apache/spark/JavaAPISuite.java uses unchecked or unsafe operations.
> [WARNING] Note: Recompile with -Xlint:unchecked for details.
> [WARNING] 1 warning
> [INFO]
> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ spark-core_2.10 ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 3 source files to /home/emre/code/spark/core/target/scala-2.10/test-classes
> [INFO]
> [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ spark-core_2.10 ---
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.pom (6 KB at 14.4 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.pom (3 KB at 13.3 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.pom
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.pom (3 KB at 40.1 KB/sec)
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.jar
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.jar
> Downloading: https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.jar
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.jar (39 KB at 184.9 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.jar (144 KB at 141.3 KB/sec)
> Downloaded: https://repo1.maven.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.jar (260 KB at 166.2 KB/sec)
> [INFO] Tests are skipped.
> [INFO]
> [INFO] --- scalatest-maven-plugin:1.0-RC2:test (test) @ spark-core_2.10 ---
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
> Discovery starting.
> Discovery completed in 7 seconds, 167 milliseconds.
> Run starting. Expected test count is: 809
> ExternalSorterSuite:
> - empty data stream
> - few elements per partition
> - empty partitions with spilling
> - empty partitions with spilling, bypass merge-sort
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> - spilling in local cluster *** FAILED ***
> org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
> at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
> at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
> at scala.Option.foreach(Option.scala:236)
> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
> ...
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> - spilling in local cluster with many reduce tasks *** FAILED ***
> org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
> at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
> at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
> at scala.Option.foreach(Option.scala:236)
> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
> ...
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> - cleanup of intermediate files in sorter
> - cleanup of intermediate files in sorter, bypass merge-sort
> - cleanup of intermediate files in sorter if there are errors
> - cleanup of intermediate files in sorter if there are errors, bypass merge-sort
> - cleanup of intermediate files in shuffle
> - cleanup of intermediate files in shuffle with errors
> - no partial aggregation or sorting
> - partial aggregation without spill
> - partial aggregation with spill, no ordering
> - partial aggregation with spill, with ordering
> - sorting without aggregation, no spill
> - sorting without aggregation, with spill
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> ls: cannot access /home/emre/code/spark/assembly/target/scala-2.10: No such file or directory
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Spark Project Core ................................. FAILURE [03:36 min]
> [INFO] Spark Project Streaming ............................ SKIPPED
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 03:37 min
> [INFO] Finished at: 2014-12-05T12:33:36+01:00
> [INFO] Final Memory: 37M/597M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0-RC2:test (test) on project spark-core_2.10: There are test failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please read the following articles:
> [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>
> --
> Emre Sevinç