You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Busbey (JIRA)" <ji...@apache.org> on 2014/11/22 22:42:12 UTC
[jira] [Updated] (SPARK-4556) binary distribution assembly can't
run in local mode
[ https://issues.apache.org/jira/browse/SPARK-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Busbey updated SPARK-4556:
-------------------------------
Description:
After building the binary distribution assembly, the resultant tarball can't be used for local mode.
{code}
busbey2-MBA:spark busbey$ mvn -Pbigtop-dist -DskipTests=true package
[INFO] Scanning for projects...
...SNIP...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 32.227 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 31.402 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 8.864 s]
[INFO] Spark Project Core ................................. SUCCESS [15:39 min]
[INFO] Spark Project Bagel ................................ SUCCESS [ 29.470 s]
[INFO] Spark Project GraphX ............................... SUCCESS [05:20 min]
[INFO] Spark Project Streaming ............................ SUCCESS [11:02 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [11:26 min]
[INFO] Spark Project SQL .................................. SUCCESS [11:33 min]
[INFO] Spark Project ML Library ........................... SUCCESS [14:27 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 40.980 s]
[INFO] Spark Project Hive ................................. SUCCESS [11:45 min]
[INFO] Spark Project REPL ................................. SUCCESS [03:15 min]
[INFO] Spark Project Assembly ............................. SUCCESS [04:22 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [ 43.567 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 50.367 s]
[INFO] Spark Project External Flume ....................... SUCCESS [01:41 min]
[INFO] Spark Project External MQTT ........................ SUCCESS [ 40.973 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 54.878 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [01:23 min]
[INFO] Spark Project Examples ............................. SUCCESS [10:19 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:47 h
[INFO] Finished at: 2014-11-22T02:13:51-06:00
[INFO] Final Memory: 79M/2759M
[INFO] ------------------------------------------------------------------------
busbey2-MBA:spark busbey$ cd assembly/target/
busbey2-MBA:target busbey$ mkdir dist-temp
busbey2-MBA:target busbey$ tar -C dist-temp -xzf spark-assembly_2.10-1.3.0-SNAPSHOT-dist.tar.gz
busbey2-MBA:target busbey$ cd dist-temp/
busbey2-MBA:dist-temp busbey$ ./bin/spark-shell
ls: /Users/busbey/projects/spark/assembly/target/dist-temp/assembly/target/scala-2.10: No such file or directory
Failed to find Spark assembly in /Users/busbey/projects/spark/assembly/target/dist-temp/assembly/target/scala-2.10
You need to build Spark before running this program.
{code}
It looks like the classpath calculations in {{bin/compute_classpath.sh}} don't handle it.
If I move all of the spark-*.jar files from the top level into the lib folder and touch the RELEASE file, then the spark shell launches in local mode normally.
was:
After building the binary distribution assembly, the resultant tarball can't be used for local mode.
{code}
busbey2-MBA:spark busbey$ mvn -Pbigtop-dist -DskipTests=true package
[INFO] Scanning for projects...
...SNIP...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 32.227 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 31.402 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 8.864 s]
[INFO] Spark Project Core ................................. SUCCESS [15:39 min]
[INFO] Spark Project Bagel ................................ SUCCESS [ 29.470 s]
[INFO] Spark Project GraphX ............................... SUCCESS [05:20 min]
[INFO] Spark Project Streaming ............................ SUCCESS [11:02 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [11:26 min]
[INFO] Spark Project SQL .................................. SUCCESS [11:33 min]
[INFO] Spark Project ML Library ........................... SUCCESS [14:27 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 40.980 s]
[INFO] Spark Project Hive ................................. SUCCESS [11:45 min]
[INFO] Spark Project REPL ................................. SUCCESS [03:15 min]
[INFO] Spark Project Assembly ............................. SUCCESS [04:22 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [ 43.567 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 50.367 s]
[INFO] Spark Project External Flume ....................... SUCCESS [01:41 min]
[INFO] Spark Project External MQTT ........................ SUCCESS [ 40.973 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 54.878 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [01:23 min]
[INFO] Spark Project Examples ............................. SUCCESS [10:19 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:47 h
[INFO] Finished at: 2014-11-22T02:13:51-06:00
[INFO] Final Memory: 79M/2759M
[INFO] ------------------------------------------------------------------------
{code}
busbey2-MBA:spark busbey$ cd assembly/target/
busbey2-MBA:target busbey$ mkdir dist-temp
busbey2-MBA:target busbey$ tar -C dist-temp -xzf spark-assembly_2.10-1.3.0-SNAPSHOT-dist.tar.gz
busbey2-MBA:target busbey$ cd dist-temp/
busbey2-MBA:dist-temp busbey$ ./bin/spark-shell
ls: /Users/busbey/projects/spark/assembly/target/dist-temp/assembly/target/scala-2.10: No such file or directory
Failed to find Spark assembly in /Users/busbey/projects/spark/assembly/target/dist-temp/assembly/target/scala-2.10
You need to build Spark before running this program.
{code}
It looks like the classpath calculations in {{bin/compute_classpath.sh}} don't handle it.
If I move all of the spark-*.jar files from the top level into the lib folder and touch the RELEASE file, then the spark shell launches in local mode normally.
> binary distribution assembly can't run in local mode
> ----------------------------------------------------
>
> Key: SPARK-4556
> URL: https://issues.apache.org/jira/browse/SPARK-4556
> Project: Spark
> Issue Type: Bug
> Components: Build, Spark Shell
> Reporter: Sean Busbey
>
> After building the binary distribution assembly, the resultant tarball can't be used for local mode.
> {code}
> busbey2-MBA:spark busbey$ mvn -Pbigtop-dist -DskipTests=true package
> [INFO] Scanning for projects...
> ...SNIP...
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [ 32.227 s]
> [INFO] Spark Project Networking ........................... SUCCESS [ 31.402 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 8.864 s]
> [INFO] Spark Project Core ................................. SUCCESS [15:39 min]
> [INFO] Spark Project Bagel ................................ SUCCESS [ 29.470 s]
> [INFO] Spark Project GraphX ............................... SUCCESS [05:20 min]
> [INFO] Spark Project Streaming ............................ SUCCESS [11:02 min]
> [INFO] Spark Project Catalyst ............................. SUCCESS [11:26 min]
> [INFO] Spark Project SQL .................................. SUCCESS [11:33 min]
> [INFO] Spark Project ML Library ........................... SUCCESS [14:27 min]
> [INFO] Spark Project Tools ................................ SUCCESS [ 40.980 s]
> [INFO] Spark Project Hive ................................. SUCCESS [11:45 min]
> [INFO] Spark Project REPL ................................. SUCCESS [03:15 min]
> [INFO] Spark Project Assembly ............................. SUCCESS [04:22 min]
> [INFO] Spark Project External Twitter ..................... SUCCESS [ 43.567 s]
> [INFO] Spark Project External Flume Sink .................. SUCCESS [ 50.367 s]
> [INFO] Spark Project External Flume ....................... SUCCESS [01:41 min]
> [INFO] Spark Project External MQTT ........................ SUCCESS [ 40.973 s]
> [INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 54.878 s]
> [INFO] Spark Project External Kafka ....................... SUCCESS [01:23 min]
> [INFO] Spark Project Examples ............................. SUCCESS [10:19 min]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:47 h
> [INFO] Finished at: 2014-11-22T02:13:51-06:00
> [INFO] Final Memory: 79M/2759M
> [INFO] ------------------------------------------------------------------------
> busbey2-MBA:spark busbey$ cd assembly/target/
> busbey2-MBA:target busbey$ mkdir dist-temp
> busbey2-MBA:target busbey$ tar -C dist-temp -xzf spark-assembly_2.10-1.3.0-SNAPSHOT-dist.tar.gz
> busbey2-MBA:target busbey$ cd dist-temp/
> busbey2-MBA:dist-temp busbey$ ./bin/spark-shell
> ls: /Users/busbey/projects/spark/assembly/target/dist-temp/assembly/target/scala-2.10: No such file or directory
> Failed to find Spark assembly in /Users/busbey/projects/spark/assembly/target/dist-temp/assembly/target/scala-2.10
> You need to build Spark before running this program.
> {code}
> It looks like the classpath calculations in {{bin/compute_classpath.sh}} don't handle it.
> If I move all of the spark-*.jar files from the top level into the lib folder and touch the RELEASE file, then the spark shell launches in local mode normally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org