[GitHub] [incubator-hudi] firecast commented on issue #954: org.apache.hudi.org.apache.hadoop_hive.metastore.api.NoSuchObjectException: table not found

[GitBox <gi...@apache.org>]

firecast commented on issue #954:  org.apache.hudi.org.apache.hadoop_hive.metastore.api.NoSuchObjectException: <hivedb.tableName> table not found
URL: https://github.com/apache/incubator-hudi/issues/954#issuecomment-544157071
 
 
   I also faced the same issue although I was using a remote hive instance instead of the AWS Glue Data Catalog. A quick fix I did to fix the issue were the following
   1. https://github.com/apache/incubator-hudi/blob/ed745dfdbf254bfc2ec6d9c7baed8ccbf571abab/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L169 to 
   ```scala
   syncHive(basePath, fs, parameters, sqlContext)
   ```
   2. https://github.com/apache/incubator-hudi/blob/ed745dfdbf254bfc2ec6d9c7baed8ccbf571abab/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L231 to 
   ```scala
   private def syncHive(basePath: Path, fs: FileSystem, parameters: Map[String, String], sqlContext: SQLContext): Boolean = {
   ```
   3. Add the following lines before this line https://github.com/apache/incubator-hudi/blob/ed745dfdbf254bfc2ec6d9c7baed8ccbf571abab/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L235
   ```scala
   val hiveMetastoreURIs = sqlContext.sparkSession.conf.get(ConfVars.METASTOREURIS.varname)
   hiveConf.setVar(ConfVars.METASTOREURIS, hiveMetastoreURIs)
   ```
   
   What this basically does is add the thrift URI you have set while creating the Spark Session to the Hive configuration. A temporary solution if anyone has a similar spark configuration as mine.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services