[jira] [Updated] (SPARK-1789) Multiple versions of Netty dependencies cause FlumeStreamSuite failure

["Patrick Wendell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Wendell updated SPARK-1789:
-----------------------------------

    Assignee: Sean Owen

> Multiple versions of Netty dependencies cause FlumeStreamSuite failure
> ----------------------------------------------------------------------
>
>                 Key: SPARK-1789
>                 URL: https://issues.apache.org/jira/browse/SPARK-1789
>             Project: Spark
>          Issue Type: Bug
>          Components: Build
>    Affects Versions: 0.9.1
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>              Labels: flume, netty, test
>             Fix For: 1.0.0
>
>
> TL;DR is there is a bit of JAR hell trouble with Netty, that can be mostly resolved and will resolve a test failure.
> I hit the error described at http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-startup-time-out-td1753.html while running FlumeStreamingSuite, and have for a short while (is it just me?)
> velvia notes:
> "I have found a workaround.  If you add akka 2.2.4 to your dependencies, then everything works, probably because akka 2.2.4 brings in newer version of Jetty." 
> There are at least 3 versions of Netty in play in the build:
> - the new Flume 1.4.0 dependency brings in io.netty:netty:3.4.0.Final, and that is the immediate problem
> - the custom version of akka 2.2.3 depends on io.netty:netty:3.6.6.
> - but, Spark Core directly uses io.netty:netty-all:4.0.17.Final
> The POMs try to exclude other versions of netty, but are excluding org.jboss.netty:netty, when in fact older versions of io.netty:netty (not netty-all) are also an issue.
> The org.jboss.netty:netty excludes are largely unnecessary. I replaced many of them with io.netty:netty exclusions until everything agreed on io.netty:netty-all:4.0.17.Final.
> But this didn't work, since Akka 2.2.3 doesn't work with Netty 4.x. Down-grading to 3.6.6.Final across the board made some Spark code not compile.
> If the build *keeps* io.netty:netty:3.6.6.Final as well, everything seems to work. Part of the reason seems to be that Netty 3.x used the old `org.jboss.netty` packages. This is less than ideal, but is no worse than the current situation. 
> So this PR resolves the issue and improves the JAR hell, even if it leaves the existing theoretical Netty 3-vs-4 conflict:
> - Remove org.jboss.netty excludes where possible, for clarity; they're not needed except with Hadoop artifacts
> - Add io.netty:netty excludes where needed -- except, let akka keep its io.netty:netty
> - Change a bit of test code that actually depended on Netty 3.x, to use 4.x equivalent
> - Update SBT build accordingly
> A better change would be to update Akka far enough such that it agrees on Netty 4.x, but I don't know if that's feasible.



--
This message was sent by Atlassian JIRA
(v6.2#6252)