You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ma...@materna.de on 2017/07/27 14:46:53 UTC

guava not compatible to hadoop version 2.6.5

After upgrading from apache spark 2.1.1 to 2.2.0 our integration test fail with an exception:

java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat
                at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:312)
                at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)

Seems there is an incompatibility with hadoop artifacts and guava, more details are described here:

https://stackoverflow.com/questions/36427291/illegalaccesserror-to-guavas-stopwatch-from-org-apache-hadoop-mapreduce-lib-inp

Advising maven to use more recent versions of affected artifacts solved this issue.

Questions: Is this a reasonable workaround for deployment into production? Or should we still stay on 2.1.1 until these dependencies are fixed in spark? Is there a ticket referencing this issue?