[GitHub] [incubator-hudi] sassai commented on issue #1646: [SUPPORT]: Unable to query Hive table through Spark SQL

[GitBox <gi...@apache.org>]

sassai commented on issue #1646:
URL: https://github.com/apache/incubator-hudi/issues/1646#issuecomment-632666310


   Hi @bvaradar,
   
   thanks for the reply. After some digging I found a solution. 
   
   The problem was that the spark did not load the jars specified in `HIVE_AUX_JARS_PATH` and on the Cloudera Data Platform the `spark-defaults.conf` has the following properties.
   
   ```console
   spark.sql.hive.metastore.jars=${env:HADOOP_COMMON_HOME}/../hive/lib/*:${env:HADOOP_COMMON_HOME}/client/*
   ```
   
   To fix this problem I edited the `spark-defaults.conf` via the Cloudera Manager and added the path to the hudi-mr-bundle jars. 
   
   Here is a brief description on how to resolve the issue on CDP:
   
   1. Go to Cloudera Manager > Cluster > Spark > Configuration > search for "safety".
   
   2. Edit the snippet for `spark-conf/spark-defaults.conf` and add
   
   ```console
   spark.sql.hive.metastore.jars=${env:HADOOP_COMMON_HOME}/../hive/lib/*:${env:HADOOP_COMMON_HOME}/client/*:/shared/jars/hive/*
   ```
   
   > Note: /shared/jars/hive/* is the path containing the hudi-mr-bundle (HIVE_AUX_JARS_PATH)
   
   3. Restart Spark


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org