You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sweetpythoncode (via GitHub)" <gi...@apache.org> on 2023/03/04 08:46:19 UTC
[GitHub] [iceberg] sweetpythoncode opened a new issue, #7007: Iceberg failed when file deleted from the s3 bucket
sweetpythoncode opened a new issue, #7007:
URL: https://github.com/apache/iceberg/issues/7007
### Query engine
Spark
### Question
When a file is deleted from the s3 bucket but manifests still point to that file, Spark failed with
`Caused by: java.io.FileNotFoundException: No such file or directory 's3:`
on
`spark.sql("SELECT count(*) FROM dev.test_iceberg_schema.test_table").show()`
any ideas on how to handle cases when files from metadata are deleted not through Iceberg, but inside the bucket(manually or with cli)? Is that possible to expire the snapshot immediately, instead of waiting 1 day?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #7007: Iceberg failed when file deleted from the s3 bucket
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #7007:
URL: https://github.com/apache/iceberg/issues/7007#issuecomment-1455705210
@sweetpythoncode according to https://iceberg.apache.org/docs/latest/maintenance/ you could expire snapshots that are older than X (whatever you configure X to be)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] sweetpythoncode commented on issue #7007: Iceberg failed when file deleted from the s3 bucket
Posted by "sweetpythoncode (via GitHub)" <gi...@apache.org>.
sweetpythoncode commented on issue #7007:
URL: https://github.com/apache/iceberg/issues/7007#issuecomment-1456277295
@nastra Its works only if i will wait 1 day, until that all queries will fail :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #7007: Iceberg failed when file deleted from the s3 bucket
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #7007:
URL: https://github.com/apache/iceberg/issues/7007#issuecomment-1456289541
Why do you have to wait a day? You can just expire all snapshots older than "now()".
That said for repairs like this I would probably use the Table api to delete the files in question from the Iceberg table metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] sweetpythoncode commented on issue #7007: Iceberg failed when file deleted from the s3 bucket
Posted by "sweetpythoncode (via GitHub)" <gi...@apache.org>.
sweetpythoncode commented on issue #7007:
URL: https://github.com/apache/iceberg/issues/7007#issuecomment-1456401873
@RussellSpitzer @nastra Thanks guys, I handle that in this way
- `DELETE FROM table` deleted file from the bucket
- Run expire snapshot older than `now()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] sweetpythoncode closed issue #7007: Iceberg failed when file deleted from the s3 bucket
Posted by "sweetpythoncode (via GitHub)" <gi...@apache.org>.
sweetpythoncode closed issue #7007: Iceberg failed when file deleted from the s3 bucket
URL: https://github.com/apache/iceberg/issues/7007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org