You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tustvold (via GitHub)" <gi...@apache.org> on 2025/01/27 10:54:06 UTC
[I] object_store: Directory Support [arrow-rs]
tustvold opened a new issue, #7030:
URL: https://github.com/apache/arrow-rs/issues/7030
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
-->
Currently object_store treats directories as an implementation detail:
* LocalFileSystem automatically creates them on-demand.
* There is no mechanism to unambiguously refer to a directory (#7026)
* Various implementations filter out directories when listing (e.g. [azure](https://github.com/apache/arrow-rs/blob/main/object_store/src/azure/client.rs#L1013))
This ensures portability but has a couple of downsides:
* There is no way to delete directories in LocalFileSystem
* There is no way to create or delete objects with trailing / (#7026) which are sometimes used as pseudo directories
* Some stores may have more efficient mechanisms for deleting a directory and all contents (e.g. LocalFileSystem / ADLS2)
A pattern used by some tools is to create empty objects with a trailing `/` to represent directories on stores that don't natively support directories. However, these pseudo directories do not really behave like normal directories:
* A filename can be created without its parent directory existing
* A directory can be deleted without deleting its children
Additionally, creating such pseudo directories on stores with native directories is at best confusing, if it is even possible.
This inherent inconsistency makes it very hard to devise a coherent API
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
I think we should at least do the following:
* Better document the current situation w.r.t directories
* Skip over pseudo directories when listing (we have no mechanism to create them, so whatever is creating them should manage their lifecycle)
I'd also be interested if people have any ideas of a coherent way to support a notion of directories
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
* https://github.com/apache/arrow-rs/pull/5978 added automatic cleanup of empty directories to LocalFileSystem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Directory Support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7030:
URL: https://github.com/apache/arrow-rs/issues/7030#issuecomment-2636125443
What should happen for systems that natively support directories, and therefore where such files cannot be created?
Perhaps you could articulate what your use-case for such paths is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Directory Support [arrow-rs]
Posted by "stayrascal (via GitHub)" <gi...@apache.org>.
stayrascal commented on issue #7030:
URL: https://github.com/apache/arrow-rs/issues/7030#issuecomment-2636100281
Thanks @tustvold for driving this discussion.
Since supporting directory will bring much complexity on current design, I'm thinking if we can only support head/put/delete/list a object with a trailing `/`, just lifts the path restriction to accept a normal string as the object key, and treats it as a normal object, and shifts the directory/file logic to upstream, e.g. whether allow to delete a non empty dir, create a dir on a existing file, etc. The `object_store` still keep the atomic behavior as the behind object store provided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Directory Support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7030:
URL: https://github.com/apache/arrow-rs/issues/7030#issuecomment-2651569613
> implementation's type usable on its own, without the trait?
I'm not entirely sure I follow, if you're not making use of the type erasure, i.e. you know the concrete store type, you might as well just use std::fs directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Directory Support [arrow-rs]
Posted by "Kinrany (via GitHub)" <gi...@apache.org>.
Kinrany commented on issue #7030:
URL: https://github.com/apache/arrow-rs/issues/7030#issuecomment-2651519464
> There is no way to delete directories in LocalFileSystem
As far as I understand the main issue here is that of needing two ways to access the same thing, right?
The implementations are very different and the trait cannot expose all of their capabilities, so perhaps it makes sense to just make each individual implementation's type usable on its own, without the trait?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Directory Support [arrow-rs]
Posted by "Kinrany (via GitHub)" <gi...@apache.org>.
Kinrany commented on issue #7030:
URL: https://github.com/apache/arrow-rs/issues/7030#issuecomment-2651943149
> I'm not entirely sure I follow, if you're not making use of the type erasure, i.e. you know the concrete store type, you might as well just use std::fs directly?
Yeah. But that means you're using two clients to access the same data store. The local file system is the least interesting example of course.
I'm under the impression here that some things, like directory handling, are just not unified enough, and so if it is something the user is concerned with, they already know the implementations they are actually using.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org