You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ma...@materna.de on 2017/07/27 14:46:53 UTC
guava not compatible to hadoop version 2.6.5
After upgrading from apache spark 2.1.1 to 2.2.0 our integration test fail with an exception:
java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:312)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)
Seems there is an incompatibility with hadoop artifacts and guava, more details are described here:
https://stackoverflow.com/questions/36427291/illegalaccesserror-to-guavas-stopwatch-from-org-apache-hadoop-mapreduce-lib-inp
Advising maven to use more recent versions of affected artifacts solved this issue.
Questions: Is this a reasonable workaround for deployment into production? Or should we still stay on 2.1.1 until these dependencies are fixed in spark? Is there a ticket referencing this issue?