[jira] [Commented] (SPARK-11081) Make spark-core pull in Jersey and javax.ws.rs dependencies separately for easier overriding

["Matt Cheah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-11081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037040#comment-15037040 ] 

Matt Cheah commented on SPARK-11081:
------------------------------------

Quick question as I develop this as I'm fairly new to Maven - how is Spark publishing org.spark-project.akka? I presume that this is a version of Akka that we're somehow modifying and we're publishing it (for Hadoop 1 builds), and it would be helpful to use that as a starting point for publishing org.spark-project.jersey.

> Make spark-core pull in Jersey and javax.ws.rs dependencies separately for easier overriding
> --------------------------------------------------------------------------------------------
>
>                 Key: SPARK-11081
>                 URL: https://issues.apache.org/jira/browse/SPARK-11081
>             Project: Spark
>          Issue Type: Improvement
>          Components: Build, Spark Core
>            Reporter: Mingyu Kim
>
> As seen from this thread (https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vRx-d2uZQ@mail.gmail.com%3E), Spark is incompatible with Jersey 2 especially when Spark is embedded in an application running with Jersey.
> There was an in-depth discussion on options for shading and making it easier for users to be able to use Jersey 2 with Spark applications: https://github.com/apache/spark/pull/9615
> To recap the discussion, Jersey 1 has two issues:
> 1. It has classes listed in META-INF/services/ files that would be loaded even if Jersey 2 was being loaded on the classpath in a higher precedence. This means that Jersey 2 would attempt to use Jersey 1 implementations in some places regardless of user attempts to override the dependency with things like userClassPathFirst.
> 2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is compatible with.
> Also discussed was the fact that plain old shading doesn't work here, since you would need to shade lines in META-INF/services as well, not just classes. Not to mention that shading JAX-RS annotations is tricky as well.
> To recap the discussion as what needs to happen Spark-side, we need to:
> 1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is the Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually shade/namespace the classes that way, just the artifact name)
> 2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact, say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files live in this artifact as well)
> 3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2
> 4. Spark assembly excludes META-INF/services/javax.ws.rs.* - it turns out these files aren't actually necessary for Jersey 1 to function properly in general (we need to test this more however)
> Now a user that wants to depend on Jersey 2, and is depending on Spark maven artifacts, would do the following in their application
> 1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs dependencies
> 2. In my application's dependencies, exclude org.spark-project.javax.ws.rs from spark-core. We keep org.spark-project.jersey because spark-core needs it, but it will use the javax.ws.rs classes that my application is providing.
> 3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new javax.ws.rs jars to the executors



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org