[jira] [Created] (SPARK-19245) Cannot build spark-assembly jar

["Barry Becker (JIRA)" <ji...@apache.org>]

Barry Becker created SPARK-19245:
------------------------------------

             Summary: Cannot build spark-assembly jar
                 Key: SPARK-19245
                 URL: https://issues.apache.org/jira/browse/SPARK-19245
             Project: Spark
          Issue Type: Documentation
          Components: Build
    Affects Versions: 2.1.0
            Reporter: Barry Becker
            Priority: Minor


This may just be an issue with documentation, but I am not sure.
In previous versions of spark (spark 2.0.2 and earlier) one could execute a command like 

build/mvn -Pmesos -Phadoop-2.6 -Dhadoop.version=2.6.5 -Dscala-2.11 -DskipTests clean package

and have it produce an assembly jar called something like spark-assembly-x.y.z-hadoop2.6.5.jar. This no longer seems possible with spark 2.1.0. 
There is a -bigtop-dist option (not documented on the main 2.1 build page) that allows you to 
create a spark-assembly_2.11-2.1.0-dist.tar.gzip in spark-2.1.0/assembly/target.
If I unpack it, it contains a lot of jars and a README that states:
{quote}
This is an assembly module for Spark project.
It creates a single tar.gz file that includes all needed dependency of the project except for org.apache.hadoop.* jars that are supposed to be available from the deployed Hadoop cluster.
This module is off by default. To activate it specify the profile in the command line
  -Pbigtop-dist
If you need to build an assembly for a different version of Hadoop the
hadoop-version system property needs to be set as in this example:
  -Dhadoop.version=2.0.6-alpha
{quote}
My question is, how can I produce a single mineset-assembly-x.y.y-hadoopA.B.C.jar? If that it no longer possible, how are we supposed to use the bigtop-dist.tar.gzip file? Do we copy it to the same lib directory and unpack it there? In any case, some clarification on the build page (http://spark.apache.org/docs/latest/building-spark.html) would help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org