You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/01/31 08:15:25 UTC

[GitHub] yaooqinn opened a new pull request #23709: [SPARK-26794] SparkSession enableHiveSupport does not point to hive but in-memory while the SparkContext exists

yaooqinn opened a new pull request #23709: [SPARK-26794] SparkSession enableHiveSupport does not point to hive but in-memory while the SparkContext exists
URL: https://github.com/apache/spark/pull/23709
 
 
   ## What changes were proposed in this pull request?
   
   ```java
   
   public class SqlDemo
   {
       public static void main(final String[] args) throws Exception {
        
           SparkConf conf = new SparkConf().setAppName("spark-sql-demo");
           JavaSparkContext sc = new JavaSparkContext(conf);
           SparkSession ss = SparkSession.builder().enableHiveSupport().getOrCreate();
           ss.sql("show databases").show();
           }
           
       }
   }
   ```
   Before https://issues.apache.org/jira/browse/SPARK-20946, the demo above point to the right hive metastore if the hive-site.xml is present. But now it can only point to the default in-memory one.
   
   Catalog is now as a variable shared across SparkSessions, it is instantiated with SparkContext's conf. After https://issues.apache.org/jira/browse/SPARK-20946, Session level configs are not pass to SparkContext's conf anymore, so the enableHiveSupport API takes no affect on the catalog instance.
   
   You can set spark.sql.catalogImplementation=hive application wide to solve the problem, or never create a sc before you call SparkSession.builder().enableHiveSupport().getOrCreate()
   
   
   
   ## How was this patch tested?
   
   (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
   (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
   
   Please review http://spark.apache.org/contributing.html before opening a pull request.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org