You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ZENOTME (via GitHub)" <gi...@apache.org> on 2024/03/13 12:23:22 UTC
[I] Do we need to clean the file if close return fail in FileWriter [iceberg-rust]
ZENOTME opened a new issue, #260:
URL: https://github.com/apache/iceberg-rust/issues/260
For FileWriter close fail, it may result in a garbage file. I look through how java process it, and seems it provided a interface like `abort` so user can call abort if it close fail. But our writer will consume the writer in `fn close(self)` which means that we can't access it later. So there are three solutions:
1. Clean them automatically.
2. Return them in error. But then all error return by writer should contains this message. I think it's not a good way.
3. Can we modify the interface to `fn close(&mut self)`.🤔 But it may make the implementation be more complicate. E.g. we need to keep track the state in writer.
My concern of first solution is that is this what user expect?
cc @liurenjie1024 @Xuanwo @Fokko @JanKaul
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Do we need to clean the file if close return fail in FileWriter [iceberg-rust]
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on issue #260:
URL: https://github.com/apache/iceberg-rust/issues/260#issuecomment-1998982646
Compared to [remove_orphan_files](https://iceberg.apache.org/docs/latest/spark-procedures/#remove_orphan_files), I think this can be a light weight way to clean the file rather than search the whole metadata file.
But as @Xuanwo explains, for a single file, it's easy, or no need to clean in object storage services. Seems it is not worth enough to do this, let's close it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Do we need to clean the file if close return fail in FileWriter [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #260:
URL: https://github.com/apache/iceberg-rust/issues/260#issuecomment-1994292504
> For FileWriter close fail, it may result in a garbage file.
I assume that `FileWriter` just about writing a single file.
For most object storage services (like s3, gcs and azblob), it should be fine:
- Files via `PutObject` doesn't exist while failed.
- Files via `Multipart` doesn't exist while failed and incomplete muiltpart is easy to cleanup
For file based services (like fs, hdfs), it should also be fine if users enabled opendal's `atomic_write_dir` feature which makes sure that file only been visible after `close` returning success.
> 3. Can we modify the interface to `fn close(&mut self)`
OpenDAL follows this pattern to allow users to abort writing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Do we need to clean the file if close return fail in FileWriter [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #260:
URL: https://github.com/apache/iceberg-rust/issues/260#issuecomment-1994322570
I don't think we should clean up garbage files in `FileWriter` api. Garbage files not only produced in `close` failure, but also in task failure in distribute computing engine, and there are many causes to it. I think we should provide sth like [remove_orphan_files](https://iceberg.apache.org/docs/latest/spark-procedures/#remove_orphan_files) to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Do we need to clean the file if close return fail in FileWriter [iceberg-rust]
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME closed issue #260: Do we need to clean the file if close return fail in FileWriter
URL: https://github.com/apache/iceberg-rust/issues/260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org