[jira] [Resolved] (SPARK-19245) Cannot build spark-assembly jar

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-19245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-19245.
-------------------------------
    Resolution: Not A Problem

There aren't assemblies anymore: https://issues.apache.org/jira/browse/SPARK-11157
Questions about how to use it should go to the mailing list.

> Cannot build spark-assembly jar
> -------------------------------
>
>                 Key: SPARK-19245
>                 URL: https://issues.apache.org/jira/browse/SPARK-19245
>             Project: Spark
>          Issue Type: Documentation
>          Components: Build
>    Affects Versions: 2.1.0
>            Reporter: Barry Becker
>            Priority: Minor
>
> This may just be an issue with documentation, but I am not sure.
> In previous versions of spark (spark 2.0.2 and earlier) one could execute a command like 
> build/mvn -Pmesos -Phadoop-2.6 -Dhadoop.version=2.6.5 -Dscala-2.11 -DskipTests clean package
> and have it produce an assembly jar called something like spark-assembly-x.y.z-hadoop2.6.5.jar. This no longer seems possible with spark 2.1.0. 
> There is a -bigtop-dist option (not documented on the main 2.1 build page) that allows you to 
> create a spark-assembly_2.11-2.1.0-dist.tar.gzip in spark-2.1.0/assembly/target.
> If I unpack it, it contains a lot of jars and a README that states:
> {quote}
> This is an assembly module for Spark project.
> It creates a single tar.gz file that includes all needed dependency of the project except for org.apache.hadoop.* jars that are supposed to be available from the deployed Hadoop cluster.
> This module is off by default. To activate it specify the profile in the command line
>   -Pbigtop-dist
> If you need to build an assembly for a different version of Hadoop the
> hadoop-version system property needs to be set as in this example:
>   -Dhadoop.version=2.0.6-alpha
> {quote}
> My question is, how can I produce a single mineset-assembly-x.y.y-hadoopA.B.C.jar? If that it no longer possible, how are we supposed to use the bigtop-dist.tar.gzip file? Do we copy it to the same lib directory and unpack it there? In any case, some clarification on the build page (http://spark.apache.org/docs/latest/building-spark.html) would help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org