You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/04 04:28:14 UTC

[GitHub] [iceberg] felixYyu opened a new issue #3669: Spark:can't load HadoopTable by path

felixYyu opened a new issue #3669:
URL: https://github.com/apache/iceberg/issues/3669


   iceberg-spark-3.2_2.12
   
   in IcebergSource's head document:
   [table = "file:/path/to/table" -&gt; loads a HadoopTable at given path]，but under the gived path can't load the hadoopTable. because the catalog has been used 'hive' DEFAULT_CATALOG_NAME.
   `if (path.contains("/")) {
         // contains a path. Return iceberg default catalog and a PathIdentifier
         return new Spark3Util.CatalogAndIdentifier(catalogManager.catalog(DEFAULT_CATALOG_NAME),
             new PathIdentifier(path));
       }`
   
   my test code:
   `val spark = SparkSession
         .builder()
         .master("local[2]")
         .appName("IcebergAPI")
         .config("spark.sql.catalog.hadoop_prod", "org.apache.iceberg.spark.SparkCatalog")
         .config("spark.sql.catalog.hadoop_prod.type", "hadoop")
         .config("spark.sql.catalog.hadoop_prod.warehouse", "file:///D:\\lake-icebergv2\\warehouse")
         .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
         .config("spark.sql.sources.partitionOverwriteMode", "dynamic")
         .config("spark.rapids.sql.enabled", true)//GPU
         .getOrCreate()
   
       val rootPath = spark.conf.get(s"spark.sql.catalog.hadoop_prod.warehouse")
       println(rootPath)
       val table = spark.read.format("iceberg").load(s"$rootPath/$schemaName/$tableName")
       table.show(truncate = false)`
   
   
   error log:
   Cannot initialize Catalog implementation org.apache.iceberg.hive.HiveCatalog: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
   	Missing org.apache.iceberg.hive.HiveCatalog [java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] felixYyu commented on issue #3669: Spark:can't load HadoopTable by path

Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #3669:
URL: https://github.com/apache/iceberg/issues/3669#issuecomment-987535452


   I use hadoop catalog，so hive catalog shoudn‘t be finded。the hadoop table  can't load table by path?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] felixYyu closed issue #3669: Spark:can't load HadoopTable by path

Posted by GitBox <gi...@apache.org>.
felixYyu closed issue #3669:
URL: https://github.com/apache/iceberg/issues/3669


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] felixYyu commented on issue #3669: Spark:can't load HadoopTable by path

Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #3669:
URL: https://github.com/apache/iceberg/issues/3669#issuecomment-989522003


   try test again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue commented on issue #3669: Spark:can't load HadoopTable by path

Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #3669:
URL: https://github.com/apache/iceberg/issues/3669#issuecomment-987415885


   Looks like Hive is missing from your classpath. If you add it, I think it should work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] felixYyu commented on issue #3669: Spark:can't load HadoopTable by path

Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #3669:
URL: https://github.com/apache/iceberg/issues/3669#issuecomment-985966613


   cc @flyrain @szehon-ho @karuppayya @RussellSpitzer @rdblue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org