You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "thinktothings (JIRA)" <ji...@apache.org> on 2018/12/20 13:45:00 UTC
[jira] [Created] (SPARK-26421) Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea local mode
thinktothings created SPARK-26421:
-------------------------------------
Summary: Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea local mode
Key: SPARK-26421
URL: https://issues.apache.org/jira/browse/SPARK-26421
Project: Spark
Issue Type: Bug
Components: Deploy
Affects Versions: 2.4.0
Environment: ).idea maven project
).jdk 1.8.0_191
).hadoop 3.1.1
).spark 2.4.0
Reporter: thinktothings
).Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea local mode
).idea maven project
).spark.sql connect hive
val spark = SparkSession
.builder()
.master("local")
.appName("Spark Hive Example")
.config("spark.sql.warehouse.dir", warehouseLocation)
.enableHiveSupport()
.getOrCreate()
spark.sql("show databases").show()
).do this error ,local environment not cluser
----------------------
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1117)
at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:866)
at com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run$.main(Run.scala:19)
at com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run.main(Run.scala)
Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.1
at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
... 8 more
Process finished with exit code 1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org