You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/02/18 09:40:00 UTC
[GitHub] [iceberg] felixYyu opened a new issue #4161: Spark:CALL [rewrite_manifests] error Manifest is missing
felixYyu opened a new issue #4161:
URL: https://github.com/apache/iceberg/issues/4161
spark 3.2.1
iceberg 0.13.0
How to deal with this case? thanks.
```
try{
val manifestsOutput = spark.sql(
s"""
|CALL $catalogName.system.rewrite_manifests('${metaTable.schema}.${metaTable.tableName}', false)
|""".stripMargin)
manifestsOutput.show(false)
} catch {
case e: ValidationException => println(s"Current ${table.name()} table CALL [rewrite_manifests] error ${e.getMessage}")
}
```
error message:
`Current hadoop_prod.wmsb.wms_do table CALL [rewrite_manifests] error Manifest is missing: hdfs://jybigdata/iceberg/warehouse/wmsb/wms_do/metadata/ab8d63b7-4d8c-450c-8cf5-e89eb257ba71-m0.avro`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #4161: Spark:CALL [rewrite_manifests] error Manifest is missing
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #4161:
URL: https://github.com/apache/iceberg/issues/4161#issuecomment-1066440549
@felixYyu : Please add more information while closing the issue. Like, is the issue fixed? (If yes, which PR has fixed it) and why we are closing the issue. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] felixYyu commented on issue #4161: Spark:CALL [rewrite_manifests] error Manifest is missing
Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #4161:
URL: https://github.com/apache/iceberg/issues/4161#issuecomment-1045997888
cc @racevedoo，@ajantha-bhat
the expire_snapshots->rewrite_data_files->rewrite_manifests->remove_orphan_files of processing in a transaction is it correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #4161: Spark:CALL [rewrite_manifests] error Manifest is missing
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #4161:
URL: https://github.com/apache/iceberg/issues/4161#issuecomment-1046041862
> the expire_snapshots->rewrite_data_files->rewrite_manifests->remove_orphan_files of processing in a transaction is it correct?
regarding the order I guess `expire_snapshots` should be called after `rewrite_data_files` files to clean up old data files which are compacted into the new files.
No need to call `remove_orphan_files` every time (as having orphan files is rare scenario)
And regarding the error on `rewrite_manifests`,
a. Is it a sequential operation or concurrent with other operations here?
b. Is this problem only for table in hadoop catalog ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] felixYyu closed issue #4161: Spark:CALL [rewrite_manifests] error Manifest is missing
Posted by GitBox <gi...@apache.org>.
felixYyu closed issue #4161:
URL: https://github.com/apache/iceberg/issues/4161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] felixYyu commented on issue #4161: Spark:CALL [rewrite_manifests] error Manifest is missing
Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #4161:
URL: https://github.com/apache/iceberg/issues/4161#issuecomment-1044260521
BaseRewriteManifests check method:
```
private void validateDeletedManifests(Set<ManifestFile> currentManifests) {
// directly deleted manifests must be still present in the current snapshot
deletedManifests.stream()
.filter(manifest -> !currentManifests.contains(manifest))
.findAny()
.ifPresent(manifest -> {
throw new ValidationException("Manifest is missing: %s", manifest.path());
});
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] felixYyu commented on issue #4161: Spark:CALL [rewrite_manifests] error Manifest is missing
Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #4161:
URL: https://github.com/apache/iceberg/issues/4161#issuecomment-1046392866
thanks @ajantha-bhat .
a.it is sequential operation, order of expire_snapshots->rewrite_data_files->rewrite_manifests->remove_orphan_files.
b.I only test for table in hadoop catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org