spark-assembly libraries conflict with application libraries

[Robert James <sr...@gmail.com>]

spark-submit includes a spark-assembly uber jar, which has older
versions of many common libraries.  These conflict with some of the
dependencies we need.  I have been racking my brain trying to find a
solution (including experimenting with ProGuard), but haven't been
able to: when we use spark-submit, we get NoMethodErrors, even though
the code compiles fine, because the runtime classes are different than
the compile time classes!

Can someone recommend a solution? We are using scala, sbt, and
sbt-assembly, but are happy using another tool (please provide
instructions how to).