You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/06 07:12:06 UTC
[GitHub] [hudi] sunke38 opened a new issue, #5515: [SUPPORT]FileNotFoundException for hadoop-common-3.1.3.jar
sunke38 opened a new issue, #5515:
URL: https://github.com/apache/hudi/issues/5515
**Describe the problem you faced**
A clear and concise description of the problem.
I keep get a warning that
Failed to create catalog table in metastore: MetaException(message:java.lang.RuntimeException: java.io.FileNotFoundException: /home/kadm/module/hadoop/share/hadoop/common/hadoop-common-3.1.3.jar (No such file or directory))
My hadoop version is 3.2.2 and I was set version on pom.xml when I compiled hudi. There are hadoop-common-3.3.2.jar in my hadoop folder. I dont not my it keep looking for 3.1.3
**Expected behavior**
I tried to create a table for query by sparkSQL
spark.sql("create table hudi_trips using hudi location 'hdfs://hadoop203:9820/datas/hudi-warehouse/hudi_trips_cow'")
**Environment Description**
* Hudi version : 0.11
* Spark version : 3.2.1
* Hadoop version : 3.2.2
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
**Stacktrace**
```Add the stacktrace of the error.```
`2022-05-06 06:59:39,682 WARN command.CreateHoodieTableCommand: Failed to create catalog table in metastore: MetaException(message:java.lang.RuntimeException: java.io.FileNotFoundException: /home/kadm/module/hadoop/share/hadoop/common/hadoop-common-3.1.3.jar (No such file or directory))
org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table default.hudi_trips not found
at org.apache.spark.sql.errors.QueryCompilationErrors$.noSuchTableError(QueryCompilationErrors.scala:838)
at org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog.loadTable(V2SessionCatalog.scala:68)
at org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension.loadTable(DelegatingCatalogExtension.java:68)
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.loadTable(HoodieCatalog.scala:93)
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.createHoodieTable(HoodieCatalog.scala:264)
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.createTable(HoodieCatalog.scala:112)
at org.apache.spark.sql.execution.datasources.v2.CreateTableExec.run(CreateTableExec.scala:42)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:110)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:106)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:106)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:91)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:219)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
... 47 elided
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JoshuaZhuCN commented on issue #5515: [SUPPORT]FileNotFoundException for hadoop-common-3.1.3.jar
Posted by GitBox <gi...@apache.org>.
JoshuaZhuCN commented on issue #5515:
URL: https://github.com/apache/hudi/issues/5515#issuecomment-1121985462
> I have the same problem。
>
> spark：3.1.3 hudi：0.10.1 hadoop：3.1.0
I solved it. After I deleted some removed machines in the cluster from the hosts, there was no error that hadoop-common-x.x.x.jar could not be found
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #5515: [SUPPORT]FileNotFoundException for hadoop-common-3.1.3.jar
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #5515: [SUPPORT]FileNotFoundException for hadoop-common-3.1.3.jar
URL: https://github.com/apache/hudi/issues/5515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5515: [SUPPORT]FileNotFoundException for hadoop-common-3.1.3.jar
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5515:
URL: https://github.com/apache/hudi/issues/5515#issuecomment-1123176709
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JoshuaZhuCN commented on issue #5515: [SUPPORT]FileNotFoundException for hadoop-common-3.1.3.jar
Posted by GitBox <gi...@apache.org>.
JoshuaZhuCN commented on issue #5515:
URL: https://github.com/apache/hudi/issues/5515#issuecomment-1120149228
I have the same problem。
spark：3.1.3
hudi：0.10.1
hadoop：3.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org