You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/10/11 09:52:28 UTC

[GitHub] [iceberg] sun-help opened a new issue, #5956: #spark场景下 使用SparkActions 的rewriteDataFiles 报错

sun-help opened a new issue, #5956:
URL: https://github.com/apache/iceberg/issues/5956

   ### Query engine
   
   spark
   
   ### Question
   
   Exception in thread "main" java.lang.IllegalArgumentException: Cannot initialize Catalog implementation org.apache.iceberg.hive.HiveCatalog: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
   	Missing org.apache.iceberg.hive.HiveCatalog [java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException]
   	at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:182)
   	at org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:234)
   	at org.apache.iceberg.spark.SparkCatalog.buildIcebergCatalog(SparkCatalog.java:119)
   	at org.apache.iceberg.spark.SparkCatalog.initialize(SparkCatalog.java:411)
   	at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:60)
   	at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$catalog$1(CatalogManager.scala:52)
   	at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
   	at org.apache.spark.sql.connector.catalog.CatalogManager.catalog(CatalogManager.scala:52)
   	at org.apache.iceberg.spark.source.IcebergSource.catalogAndIdentifier(IcebergSource.java:140)
   	at org.apache.iceberg.spark.source.IcebergSource.extractIdentifier(IcebergSource.java:159)
   	at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:225)
   	at scala.Option.map(Option.scala:230)
   	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
   	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188)
   	at org.apache.iceberg.spark.actions.Spark3BinPackStrategy.rewriteFiles(Spark3BinPackStrategy.java:68)
   	at org.apache.iceberg.spark.actions.BaseRewriteDataFilesSparkAction.lambda$rewriteFiles$2(BaseRewriteDataFilesSparkAction.java:232)
   	at org.apache.iceberg.spark.actions.BaseSparkAction.withJobGroupInfo(BaseSparkAction.java:98)
   	at org.apache.iceberg.spark.actions.BaseRewriteDataFilesSparkAction.rewriteFiles(BaseRewriteDataFilesSparkAction.java:230)
   	at org.apache.iceberg.spark.actions.BaseRewriteDataFilesSparkAction.lambda$doExecute$4(BaseRewriteDataFilesSparkAction.java:269)
   	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
   	at org.apache.iceberg.util.Tasks$Builder.access$300(Tasks.java:70)
   	at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:310)
   	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at java.lang.Thread.run(Thread.java:748)
   Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
   	Missing org.apache.iceberg.hive.HiveCatalog [java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException]
   	at org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:227)
   	at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:180)
   	... 26 more
   求解答


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] sun-help commented on issue #5956: #spark场景下 使用SparkActions 的rewriteDataFiles 报错

Posted by GitBox <gi...@apache.org>.
sun-help commented on issue #5956:
URL: https://github.com/apache/iceberg/issues/5956#issuecomment-1275623986

   > "Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog Missing org.apache.iceberg.hive.HiveCatalog [java.lang.NoClassDefFoundError"
   > 
   > consider your program dependency is contain the right class or class confict?
   
   thinks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] sun-help closed issue #5956: #spark场景下 使用SparkActions 的rewriteDataFiles 报错

Posted by GitBox <gi...@apache.org>.
sun-help closed issue #5956: #spark场景下 使用SparkActions 的rewriteDataFiles 报错
URL: https://github.com/apache/iceberg/issues/5956


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] youngxinler commented on issue #5956: #spark场景下 使用SparkActions 的rewriteDataFiles 报错

Posted by GitBox <gi...@apache.org>.
youngxinler commented on issue #5956:
URL: https://github.com/apache/iceberg/issues/5956#issuecomment-1274593869

   "Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
   Missing org.apache.iceberg.hive.HiveCatalog [java.lang.NoClassDefFoundError"
   
   consider your program dependency is contain the right class or class confict?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org