[jira] [Updated] (SPARK-11157) Allow Spark to be built without assemblies

["Josh Rosen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Rosen updated SPARK-11157:
-------------------------------
    Target Version/s: 2.0.0

> Allow Spark to be built without assemblies
> ------------------------------------------
>
>                 Key: SPARK-11157
>                 URL: https://issues.apache.org/jira/browse/SPARK-11157
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Build, Spark Core, YARN
>            Reporter: Marcelo Vanzin
>         Attachments: no-assemblies.pdf
>
>
> For reasoning, discussion of pros and cons, and other more detailed information, please see attached doc.
> The idea is to be able to build a Spark distribution that has just a directory full of jars instead of the huge assembly files we currently have.
> Getting there requires changes in a bunch of places, I'll try to list the ones I identified in the document, in the order that I think would be needed to not break things:
> * make streaming backends not be assemblies
> Since people may depend on the current assembly artifacts in their deployments, we can't really remove them; but we can make them be dummy jars and rely on dependency resolution to download all the jars.
> PySpark tests would also need some tweaking here.
> * make examples jar not be an assembly
> Probably requires tweaks to the {{run-example}} script. The location of the examples jar would have to change (it won't be able to live in the same place as the main Spark jars anymore).
> * update YARN backend to handle a directory full of jars when launching apps
> Currently YARN localizes the Spark assembly (depending on the user configuration); it needs to be modified so that it can localize all needed libraries instead of a single jar.
> * Modify launcher library to handle the jars directory
> This should be trivial
> * Modify {{assembly/pom.xml}} to generate assembly or a {{libs}} directory depending on which profile is enabled.
> We should keep the option to build with the assembly on by default, for backwards compatibility, to give people time to prepare.
> Filing this bug as an umbrella; please file sub-tasks if you plan to work on a specific part of the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org