java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT on EMR

[Daniel Zhang <ja...@hotmail.com>]

Hi,

I know the JIRA of this error (https://issues.apache.org/jira/browse/SPARK-18112), and I read all the comments and even PR for it.

But I am facing this issue on AWS EMR, and only in Oozie Spark Action. I am looking for someone can give me a hint or direction,  so I can see if I can overcome this issue on EMR.

I am testing a simple Spark application on EMR-5.12.2, which comes with Hadoop 2.8.3 + HCatalog 2.3.2 + Spark 2.2.1, and using AWS Glue Data Catalog for both Hive + Spark table metadata.

First of all, both Hive and Spark work fine with AWS Glue as metadata catalog. And my spark application works in spark-submit.

[hadoop@ip-172-31-65-232 oozieJobs]$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.1
      /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_171)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.sql("show databases").show
+---------------+
|   databaseName|
+---------------+
|        default|
|googleanalytics|
|       sampledb|
+---------------+


I can access and query the database I created in Glue without any issue on spark-shell or spark-sql.
And as part of later problem, I can see when it works in this case, there is no set of "spark.sql.hive.metastore.version" in spark-shell, as the default value is shown below:

scala> spark.conf.get("spark.sql.hive.metastore.version")
res2: String = 1.2.1


Even though it shows version as "1.2.1", but I knew that by using Glue the hive metastore version will be "2.3.2", I can see "hive-metastore-2.3.2-amzn-1.jar" in the Hive library path.

Now here comes the issue, when I test the Spark code in the Oozie Spark action, and "enableHiveSupport" on the Spark session, it works with spark-submit in the command line, but failed with the following error in the oozie runtime:

ailing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, HIVE_STATS_JDBC_TIMEOUT
java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
        at org.apache.spark.sql.hive.HiveUtils$.hiveClientConfigurations(HiveUtils.scala:200)
        at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:265)
        at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
        at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:195)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:195)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:195)


I know this most likely caused by the Oozie runtime classpath, but I spent days of trying and still cannot find out a solution. We use Spark as our core of ETL engine, and the ability to manage and query the HiveCatalog is critical for us.

Here are what puzzled me:

  *   I know this issue was supposed fixing in Spark 2.2.0, and on this ERM, we are using Spark 2.2.1
  *   There is 1.2.1 version of hive metastore jar under the spark jars on EMR. Does this mean in the successful spark-shell runtime, spark indeed is using 1.2.1 version of hive-metastore?

[hadoop@ip-172-31-65-232 oozieJobs]$ ls /usr/lib/spark/jars/*hive-meta*
/usr/lib/spark/jars/hive-metastore-1.2.1-spark2-amzn-0.jar

  *   There is 2.3.2 version of hive metastore jar under the Hive component on this EMR, which I believe it pointing to the Glue, right?

[hadoop@ip-172-31-65-232 oozieJobs]$ ls /usr/lib/hive/lib/*hive-meta*
/usr/lib/hive/lib/hive-metastore-2.3.2-amzn-1.jar  /usr/lib/hive/lib/hive-metastore.jar

  *   I specified the "oozie.action.sharelib.for.spark=spark,hive" in the oozie, and I can see oozie runtime loads the jars from both spark and hive share libs. There is NO hive-metastore-1.2.1-spark2-amzn-0.jar in the oozie SPARK sharelib, and there is indeed hive-metastore-2.3.2-amzn-1.jar in the oozie HIVE sharelib.
  *   Based on my understanding of (https://issues.apache.org/jira/browse/SPARK-18112), here are what I did so far trying to fix this in oozie runtime, but none of them works
     *   I added hive-metastore-1.2.1-spark2-amzn-0.jar into hdfs of ozzie spark share lib, and run "oozie admin -sharelibupdate".  After that, I confirm this library loaded in the oozie runtime log of my spark action, but I got the same error message.
     *   I added "--conf spark.sql.hive.metastore.version=2.3.2" in the <spark-opts> of my oozie spark action, and confirm this configuration in spark session, but I still got the same error message above.
     *   I added "--conf spark.sql.hive.metastore.version=2.3.2 --conf spark.sql.hive.metastore.jars=maven", but still got the same error message
     *   I added "--conf spark.sql.hive.metastore.version=2.3.2 --conf spark.sql.hive.metastore.jars=/etc/spark/conf/hive-site.xml,/usr/lib/spark/jars/*" in oozie spark action, but got the same error message
     *   I added "--conf spark.sql.hive.metastore.version=2.3.2 --conf hive.metastore.uris=thrift://ip-172-31-65-232.ec2.internal:9083 --conf spark.sql.hive.metastore.jars=/etc/spark/conf/hive-site.xml,/usr/lib/spark/jars/*" in the oozie spark action, but got the same error.

I run out of options to try, and I really have no idea what is missing in the oozie runtime causing this error in the Spark.

Let me know if you have any idea.

Thanks

Yong