You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paraseba (via GitHub)" <gi...@apache.org> on 2024/09/09 15:34:38 UTC
[I] LocalFileSystem `list` operation returns objects in wrong order [arrow-rs]
paraseba opened a new issue, #6375:
URL: https://github.com/apache/arrow-rs/issues/6375
**Describe the bug**
The `LocalFileSystem` `list` implementation returns objects in what looks like arbitrary order.
S3's `ListObjectsV2` returns objects in lexicographical order of their keys:
> General purpose bucket - For general purpose buckets, ListObjectsV2 returns objects in lexicographical order based on their key names.
**To Reproduce**
The following code fails the assertion:
```rust
use futures::TryStreamExt;
use object_store::{local::LocalFileSystem, ObjectStore, PutPayload};
use std::assert_eq;
use std::fs::create_dir_all;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let dir = "/tmp/reproducer";
create_dir_all(dir)?;
let store = LocalFileSystem::new_with_prefix(dir)?;
for i in 0..=9 {
store.put(&(i.to_string()).into(), PutPayload::from_static(b"42")).await?;
}
let actual = store
.list(None)
.map_ok(|om| om.location.to_string())
.try_collect::<Vec<_>>()
.await?;
let expected = (0..=9).map(|s| s.to_string()).collect::<Vec<_>>();
assert_eq!(actual, expected);
Ok(())
}
```
**Expected behavior**
`LocalFileSystem` should mimic what `ListObjectsV2` does, returning objects in lexicographical order of their keys.
**Additional context**
`InMemory` doesn't fail this test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] LocalFileSystem `list` operation returns objects in wrong order [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6375:
URL: https://github.com/apache/arrow-rs/issues/6375#issuecomment-2338480080
Another potential option is to implement your own wrapper over `LocalFileSystem` that sorted the results by file name
like
```rust
struct SortedLocalFileSystem {
inner: LocalFileSystem
}
impl ObjectStore for SortedLocalFileSystem {
// in the implementation you could intercept the various calls and implement
// whatever behavior you wanted
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] LocalFileSystem `list` operation returns objects in wrong order [arrow-rs]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #6375:
URL: https://github.com/apache/arrow-rs/issues/6375#issuecomment-2338489671
> Would it make sense to have sorting behavior as optional?
Sorry, but it doesn't make sense to me now. If you care about the order, you can implement something like what @alamb mentioned, which should be simple.
> Intended only for small stores that fin in memory?
The most interesting part is this: we don't know how many items there are until we list them all. This inconsistent behavior based on item size might introduce more issues. For example, users might find the object_store behaves one way when there are only 10 items, but differently in production services.
> I'm not sure what level of "compatibility" the different stores implementations try to achieve.
https://github.com/apache/arrow-rs/blob/7a5155c5f0e21559203d4a5363cffd7ec0394817/object_store/src/lib.rs#L721C1-L721C69
No guarantee has been provided so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] LocalFileSystem `list` operation returns objects in wrong order [arrow-rs]
Posted by "paraseba (via GitHub)" <gi...@apache.org>.
paraseba commented on issue #6375:
URL: https://github.com/apache/arrow-rs/issues/6375#issuecomment-2338559652
Thanks for the quick responses. Closing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] LocalFileSystem `list` operation returns objects in wrong order [arrow-rs]
Posted by "paraseba (via GitHub)" <gi...@apache.org>.
paraseba closed issue #6375: LocalFileSystem `list` operation returns objects in wrong order
URL: https://github.com/apache/arrow-rs/issues/6375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] LocalFileSystem `list` operation returns objects in wrong order [arrow-rs]
Posted by "paraseba (via GitHub)" <gi...@apache.org>.
paraseba commented on issue #6375:
URL: https://github.com/apache/arrow-rs/issues/6375#issuecomment-2338473905
@Xuanwo I see. Would it make sense to have sorting behavior as optional? Intended only for small stores that fin in memory? `walkdir` library has the option to do that. Otherwise, is the listing behavior documented somewhere? I'm not sure what level of "compatibility" the different stores implementations try to achieve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] LocalFileSystem `list` operation returns objects in wrong order [arrow-rs]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #6375:
URL: https://github.com/apache/arrow-rs/issues/6375#issuecomment-2338456070
> `LocalFileSystem` should mimic what `ListObjectsV2` does, returning objects in lexicographical order of their keys.
Hi, this behavior is expected. We can't perform sorting over the LocalFileSystem since we don't have such an API, and we can't collect keys and sort them in-memory. The best we can do is return objects in the order that the filesystem provides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org