You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shaul Lahav (JIRA)" <ji...@apache.org> on 2018/06/07 09:32:00 UTC
[jira] [Updated] (SPARK-24483) enableHiveSupport doesn't work with
Spark 2.3 on EMR
[ https://issues.apache.org/jira/browse/SPARK-24483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shaul Lahav updated SPARK-24483:
--------------------------------
Description:
I run a spark job on an EMR cluster when using the SparkSession.Builder to create a spark Session.
The job includes querying a Hive table, and when it is executed Spark throws "
org.apache.spark.sql.AnalysisException: Table or view not found".
I printed all the config options in Spark context and noticed that "spark.sql.catalogImplementation" is missing.
If I provide this option as a "-conf" in "spark-submit" then it is added and everything works.
This is the code I use to instantiate the SparkSession:
val spark: SparkSession = SparkSession
.builder
.config("mapreduce.fileoutputcommitter.algorithm.version", "2")
.enableHiveSupport()
.getOrCreate()
Also note that I added the following two debugging statements just before the code that creates the session to see if there was already a session in place, and they both returned "false":
SparkSession.getActiveSession.isDefined
SparkSession.getDefaultSession.isDefined
was:
I run a spark job on an EMR cluster when using the SparkSession.Builder to create a spark Session.
The job includes querying a Hive table, and when it is executed Spark throws "
org.apache.spark.sql.AnalysisException: Table or view not found".
I printed all the config options in Spark context and noticed that "spark.sql.catalogImplementation" is missing.
If I provide this option as a "-conf" in "spark-submit" then it is added and everything works.
This is the code I use to instantiate the SparkSession:
val spark: SparkSession = SparkSession
.builder
.config("mapreduce.fileoutputcommitter.algorithm.version", "2")
.enableHiveSupport()
.getOrCreate()
> enableHiveSupport doesn't work with Spark 2.3 on EMR
> ----------------------------------------------------
>
> Key: SPARK-24483
> URL: https://issues.apache.org/jira/browse/SPARK-24483
> Project: Spark
> Issue Type: Bug
> Components: Project Infra
> Affects Versions: 2.3.0
> Environment: EMR v5.13 (Spark 2.3.0)
> Reporter: Shaul Lahav
> Priority: Major
>
> I run a spark job on an EMR cluster when using the SparkSession.Builder to create a spark Session.
> The job includes querying a Hive table, and when it is executed Spark throws "
> org.apache.spark.sql.AnalysisException: Table or view not found".
> I printed all the config options in Spark context and noticed that "spark.sql.catalogImplementation" is missing.
> If I provide this option as a "-conf" in "spark-submit" then it is added and everything works.
> This is the code I use to instantiate the SparkSession:
> val spark: SparkSession = SparkSession
> .builder
> .config("mapreduce.fileoutputcommitter.algorithm.version", "2")
> .enableHiveSupport()
> .getOrCreate()
> Also note that I added the following two debugging statements just before the code that creates the session to see if there was already a session in place, and they both returned "false":
> SparkSession.getActiveSession.isDefined
> SparkSession.getDefaultSession.isDefined
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org