You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dentiny (via GitHub)" <gi...@apache.org> on 2025/04/24 20:39:44 UTC
[D] Ideas: add directory support for `FileIO` [iceberg-rust]
GitHub user dentiny created a discussion: Ideas: add directory support for `FileIO`
Hi team, I notice
- `FileIO` is the only abstraction inside of iceberg-rust project to provide IO operation manipulation capabilities;
- Current implementation for `FileIO` focuses only on "files", or in object storage's word, "object".
I'm wondering if it makes sense to add "directory" support as well?
For example, the file hierarchy for iceberg table namespace and table is built upon directories.
The following is the output for [rest catalog docker image](https://github.com/databricks/iceberg-rest-image), support our table has created namespace "default" and table "test_table".
```sh
[/workspaces/pg_moonlink] (hjiang/filesystem-catalog)
vscode@4a9f5d144667$ ls /tmp/iceberg-test/default/test_table/
data metadata
```
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
GitHub user liurenjie1024 added a comment to the discussion: Ideas: add directory support for `FileIO`
cc @Xuanwo is more familiar with storage systems, maybe he could help you here.
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12958351
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny added a comment to the discussion: Ideas: add directory support for `FileIO`
Thank you for the reply!
I agree with your point, "directory" concepts, AFAIK, exists mostly on POSIX system and GCS.
Right now I'm implementing a filesystem based catalog, which definitely requires directory-related access, it's awkward to use file_io` for file-centric operations, while `std::fs` for directory operation.
Take duckdb as an example, it has a filesystem interface, but not all the operations apply to all detailed filesystem instances.
For example, `pipe` only apply to local filesystem: https://github.com/duckdb/duckdb/blob/0069af20abea2660d67849257bc568570388d50c/src/include/duckdb/common/file_system.hpp#L173-L174, and set the default behavior to throw (for other filesystems).
I used as a temporary workaround and leave the discussion thread here, and I completely understand and agree with your argument, would like to hear if you have other suggestions. :(
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12958326
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
GitHub user liurenjie1024 added a comment to the discussion: Ideas: add directory support for `FileIO`
Thanks @dentiny for raising this. `FileIO` is an abstraction over different storage systems, and we should only provide a minimum interface so that most storage systems can fit into it. As with directory operation, I don't think we should provide that for several reasons:
1. Apache iceberg spec has no requirement for file location, the file hierarchy you see is just default implementation, but not required in iceberg spec.
2. Most object stores's support for directory is limited and inefficient.
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12958300
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo edited a comment on the discussion: Ideas: add directory support for `FileIO`
> So we don't actually require list operation involved here?
Yes, implementing compaction doesn't require list operations. I mentioned it here because most services that implement compaction often also need to implement vacuuming to ensure that data is garbage collected on the storage side, which does require list operations. For examples, some data files might been written in the storage but not committed for long.
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12967686
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny added a comment to the discussion: Ideas: add directory support for `FileIO`
That makes sense to me.
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12967800
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny edited a comment on the discussion: Ideas: add directory support for `FileIO`
> We can expose a list operation in FileIO. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well.
Sorry if I mis-understand.
A second thought, iceberg is a location-unaware specification, which means data file could be placed anywhere, it's valid implementation as long as manifest files could track them.
For compaction, my understanding, is to
(1) get data files to compact from manifest files
(2) read data files and deletion files / deletion vector
(3) apply certain compaction strategies (i.e. sorting)
(4) write compacted data files into destination
So we don't actually require list operation involved here?
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12967384
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny edited a comment on the discussion: Ideas: add directory support for `FileIO`
> We can expose a list operation in FileIO. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well.
Sorry if I mis-understand.
A second thought, iceberg is a location-unaware specification, which means data file could be placed anywhere, it's valid implementation as long as manifest files could track them.
For compaction, my understanding, is to
(1) read data files and deletion files / deletion vector
(2) apply certain compaction strategies (i.e. sorting)
(3) write compacted data files into destination
So we don't actually require list operation involved here?
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12967384
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny added a comment to the discussion: Ideas: add directory support for `FileIO`
> We can expose a list operation in FileIO. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well.
Sorry if I mis-understand.
Iceberg is a location-unaware specification, which means data file could be placed anywhere, it's valid implementation as long as manifest files could track them.
For compaction, my understanding, is to
(1) read data files and deletion files / deletion vector
(2) apply certain compaction strategies (i.e. sorting)
(3) rewrite data files into destination
So we don't actually require list operation involved here?
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12967384
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny edited a comment on the discussion: Ideas: add directory support for `FileIO`
More context why I want to have the directory support, I'm implementing a filesystem catalog feature for two purpose:
- Hermetic unit test
+ bringing rest docker image is a little painful, but I have ways to handle
+ `MemoryCatalog` doesn't support transaction features I need
- Local usage for customer initial try out (consider it production environment)
+ `MemoryCatalog` is not intended for production usage
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12939760
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Ideas: add directory support for `FileIO`
> So we don't actually require list operation involved here?
Yes, implementing compaction doesn't require list operations. I mentioned it here because most services that implement compaction often also need to implement vacuuming to ensure that data is garbage collected on the storage side, which does require list operations.
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12967686
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny edited a comment on the discussion: Ideas: add directory support for `FileIO`
Thank you for the reply!
I agree with your point, "directory" concepts, AFAIK, exists mostly on POSIX system and GCS.
Right now I'm implementing a filesystem based catalog, which definitely requires directory-related access, it's awkward to use `file_io` for file-centric operations, while `std::fs` for directory operation.
Take duckdb as an example, it has a filesystem interface, but not all the operations apply to all detailed filesystem instances.
For example, `pipe` only apply to local filesystem: https://github.com/duckdb/duckdb/blob/0069af20abea2660d67849257bc568570388d50c/src/include/duckdb/common/file_system.hpp#L173-L174, and set the default behavior to throw (for other filesystems).
I used as a temporary workaround and leave the discussion thread here, and I completely understand and agree with your argument, would like to hear if you have other suggestions. :(
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12958326
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Ideas: add directory support for `FileIO`
Hi, thank you @dentiny for starting this.
The topic of a file-based catalog has come up multiple times throughout Iceberg's history and has been rejected for similar reasons:
- Iceberg is designed to avoid performing list object operations on storage services. It doesn't depend on storage services to provide listing support, which is one of the reasons it outperforms the Hive table format.
- The filesystem catalog itself offers only very limited features. For example, users cannot rename a table atomicly.
The latest discussion on this topic can be found at https://lists.apache.org/thread/v7x65kxrrozwlvsgstobm7685541lf5w.
Some people are exploring this direction (although not in exactly the same way) at https://olympiaformat.org. You might find it interesting.
---
Let’s return to our current discussion.
I personally agree that there are some interesting use cases for an FS-based catalog. For example, if the layout on S3 is organized in the way you mentioned, users can read from this catalog directly without needing to connect to other services.
(Note: This is an implementation detail regarding how other catalogs organize namespaces and tables. It’s possible for them to store the entire namespace in a Postgres database.)
I think the way forward for us is as follows:
We can expose a `list` operation in `FileIO`. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well.
You could then implement an FS catalog outside of iceberg-rust as a third-party catalog.
What do you think?
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12958750
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny added a comment to the discussion: Ideas: add directory support for `FileIO`
Thank you for the thoughtful reply! I think all points you've made make sense to me.
As a newcomer, one thing I found filesystem catalog attractive is, it provides a way to demonstrate what metadata file and manifest files looks like in a straightforward way; rest catalog with "file" schema also generates a few files (manifest files and manifest list), but metadata files and version hint file seems to stored internally (i.e. in sqlite).
> You could then implement an FS catalog outside of iceberg-rust as a third-party catalog.
Sure! I have an internal implementation for FS catalog, which only implements limited `TableUpdate`s;
I will try to get more updates covered and put on my github for public access at least.
> We can expose a list operation in FileIO. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well.
Nice! I'll definitely rely on the feature.
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12966963
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny edited a comment on the discussion: Ideas: add directory support for `FileIO`
Thank you for the reply!
I agree with your point, "directory" concepts, AFAIK, exists mostly on POSIX system and GCS.
Right now I'm implementing a filesystem based catalog, which definitely requires directory-related access, it's awkward to use `file_io` for file-centric operations, while `std::fs` for directory operation.
Take duckdb as an example, it has a filesystem interface, but not all the operations apply to all detailed filesystem instances.
For example, `pipe` only apply to local filesystem: https://github.com/duckdb/duckdb/blob/0069af20abea2660d67849257bc568570388d50c/src/include/duckdb/common/file_system.hpp#L173-L174, and set the default behavior to throw (for other filesystems).
I used it as a temporary workaround and leave the discussion thread here, and I completely understand and agree with your argument, would like to hear if you have other suggestions. :(
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12958326
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny edited a comment on the discussion: Ideas: add directory support for `FileIO`
> We can expose a list operation in FileIO. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well.
Sorry if I mis-understand.
A second thought, iceberg is a location-unaware specification, which means data file could be placed anywhere, it's valid implementation as long as manifest files could track them.
For compaction, my understanding, is to
(1) read data files and deletion files / deletion vector
(2) apply certain compaction strategies (i.e. sorting)
(3) rewrite data files into destination
So we don't actually require list operation involved here?
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12967384
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [D] Ideas: add directory support for `FileIO` [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
GitHub user dentiny added a comment to the discussion: Ideas: add directory support for `FileIO`
More context why I want to have the directory support:
I'm implementing a filesystem catalog feature for two purpose:
- Hermetic unit test
+ bringing rest docker image is a little painful, but I have ways to handle
+ `MemoryCatalog` doesn't support transaction features I need
- Local usage for customer initial try out (consider it production environment)
+ `MemoryCatalog` is not intended for production usage
GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12939760
----
This is an automatically sent email for issues@iceberg.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@iceberg.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org