Spark 3.0.0-preview and s3a

[vincent gromakowski <vi...@gmail.com>]

Hi Spark users,
I am testing the preview of Spark 3 with s3a and hadoop 3.2 but I have got
NoClassDefFoundError and cannot find what is the issue. I suppose there is
some lib conflict. Can someone provide a working configuration?



*Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:816) at
org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:792)*


Here is my SBT file

libraryDependencies ++= {
  Seq(
    "org.apache.spark" %% "spark-core" % "3.0.0-preview" % "provided",
    "org.apache.spark" %% "spark-sql" % "3.0.0-preview" % "provided",
    "org.apache.hadoop" % "hadoop-cloud-storage" % "3.2.1",
    "org.scalactic" %% "scalactic" % "3.1.0",
    "org.scalatest" %% "scalatest" % "3.1.0" % Test
  )
}

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("javax.xml.stream.**" ->
"shaded-javax.xml.stream.@1").inLibrary("javax.xml.stream" %
"stax-api" % "1.0-2"),
  ShadeRule.rename("*" ->
"shaded-@1").inLibrary("com.fasterxml.jackson.core" % "jackson-core" %
"2.10.0"),
  ShadeRule.rename("*" ->
"shaded2-@1").inLibrary("com.fasterxml.jackson.core" %
"jackson-databind" % "2.10.0"),
  ShadeRule.rename("mozilla.**" ->
"shaded-mozilla.@1").inLibrary("com.amazonaws" % "aws-java-sdk-bundle"
% "1.11.375"),
)

assemblyMergeStrategy in assembly := {
  case "mime.types" => MergeStrategy.rename
  case x if x.contains("versions.properties") => MergeStrategy.rename
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}