You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "372242283 (via GitHub)" <gi...@apache.org> on 2023/02/06 06:04:48 UTC
[GitHub] [iceberg] 372242283 commented on issue #3466: "Manifest is missing" ValidationException when there have Concurrent applications to rewrite manifests
372242283 commented on issue #3466:
URL: https://github.com/apache/iceberg/issues/3466#issuecomment-1418562275
I have the same problem, which happens occasionally,big guys, help me analyze it Thank you:
version:iceberg1.1.0,spark3.1.3
CALL hive_catalog.system.rewrite_manifests('xx.tableA', false)
ERROR Msg：
23/02/05 21:12:04 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
23/02/05 21:12:04 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
23/02/05 21:12:05 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/02/05 21:12:05 INFO MemoryStore: MemoryStore cleared
23/02/05 21:12:05 INFO BlockManager: BlockManager stopped
23/02/05 21:12:05 INFO BlockManagerMaster: BlockManagerMaster stopped
23/02/05 21:12:05 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/02/05 21:12:05 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.iceberg.exceptions.ValidationException: Manifest is missing: hdfs://xxx:8020/user/hive/warehouse/xx.db/tableA/metadata/d9c19289-fbf1-48a8-90be-a8eecdf088b4-m0.avro
at org.apache.iceberg.BaseRewriteManifests.lambda$validateDeletedManifests$7(BaseRewriteManifests.java:299)
at java.base/java.util.Optional.ifPresent(Unknown Source)
at org.apache.iceberg.BaseRewriteManifests.validateDeletedManifests(BaseRewriteManifests.java:297)
at org.apache.iceberg.BaseRewriteManifests.apply(BaseRewriteManifests.java:195)
at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:223)
at org.apache.iceberg.BaseRewriteManifests.apply(BaseRewriteManifests.java:50)
at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:369)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:402)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:212)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:189)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:367)
at org.apache.iceberg.BaseRewriteManifests.commit(BaseRewriteManifests.java:50)
at org.apache.iceberg.spark.actions.BaseSnapshotUpdateSparkAction.commit(BaseSnapshotUpdateSparkAction.java:43)
at org.apache.iceberg.spark.actions.BaseRewriteManifestsSparkAction.replaceManifests(BaseRewriteManifestsSparkAction.java:312)
at org.apache.iceberg.spark.actions.BaseRewriteManifestsSparkAction.doExecute(BaseRewriteManifestsSparkAction.java:185)
at org.apache.iceberg.spark.actions.BaseSparkAction.withJobGroupInfo(BaseSparkAction.java:97)
at org.apache.iceberg.spark.actions.BaseRewriteManifestsSparkAction.execute(BaseRewriteManifestsSparkAction.java:150)
at org.apache.iceberg.spark.actions.BaseRewriteManifestsSparkAction.execute(BaseRewriteManifestsSparkAction.java:80)
at org.apache.iceberg.spark.procedures.RewriteManifestsProcedure.lambda$call$0(RewriteManifestsProcedure.java:97)
at org.apache.iceberg.spark.procedures.BaseProcedure.execute(BaseProcedure.java:86)
at org.apache.iceberg.spark.procedures.BaseProcedure.modifyIcebergTable(BaseProcedure.java:74)
at org.apache.iceberg.spark.procedures.RewriteManifestsProcedure.call(RewriteManifestsProcedure.java:88)
at org.apache.spark.sql.execution.datasources.v2.CallExec.run(CallExec.scala:33)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:46)
at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3700)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org