You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "amunra (via GitHub)" <gi...@apache.org> on 2025/11/19 17:31:26 UTC
[I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
amunra opened a new issue, #6805:
URL: https://github.com/apache/opendal/issues/6805
### Describe the bug
The function documents "This function only reads the immediate children of the specified directory.", but it looks like it's also listing the current directory.
### Steps to Reproduce
```rust
use opendal::{Operator, services::Fs};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Create a temporary directory for our operations
let temp_dir = std::env::temp_dir().join("opendal_example");
std::fs::create_dir_all(&temp_dir)?;
println!("Using temp directory: {}", temp_dir.display());
println!();
// Initialize OpenDAL with filesystem service
let builder = Fs::default();
let builder = builder.root(&temp_dir.to_string_lossy());
let op: Operator = Operator::new(builder)?.finish();
// Create the directory structure and files
println!("Creating files and directories...");
// Create files in a/
op.write("a/a", "content").await?;
// Create files in a/ab/
op.write("a/ab/aba", "content").await?;
op.write("a/ab/abb", "content").await?;
// Create files in b/
op.write("b/ba", "content").await?;
op.write("b/bb", "content").await?;
println!("Files created successfully!");
println!();
// List root directory
println!("Listing '/':");
println!("{}", "=".repeat(50));
let entries = op.list("/").await?;
for entry in entries {
let path = entry.path();
let metadata = entry.metadata();
let entry_type = if metadata.is_dir() { "DIR " } else { "FILE" };
println!(" [{}] {}", entry_type, path);
}
println!();
// List a/ directory
println!("Listing '/a/':");
println!("{}", "=".repeat(50));
let entries = op.list("/a/").await?;
for entry in entries {
let path = entry.path();
let metadata = entry.metadata();
let entry_type = if metadata.is_dir() { "DIR " } else { "FILE" };
println!(" [{}] {}", entry_type, path);
}
println!();
// List a/ab/ directory
println!("Listing '/a/ab/':");
println!("{}", "=".repeat(50));
let entries = op.list("/a/ab/").await?;
for entry in entries {
let path = entry.path();
let metadata = entry.metadata();
let entry_type = if metadata.is_dir() { "DIR " } else { "FILE" };
println!(" [{}] {}", entry_type, path);
}
println!();
println!("Done!");
Ok(())
}
```
Which outputs:
```
Creating files and directories...
Files created successfully!
Listing '/':
==================================================
[DIR ] /
[DIR ] a/
[DIR ] b/
Listing '/a/':
==================================================
[DIR ] a/
[DIR ] a/ab/
[FILE] a/a
Listing '/a/ab/':
==================================================
[DIR ] a/ab/
[FILE] a/ab/abb
[FILE] a/ab/aba
Done!
```
### Expected Behavior
Expecting the path specified to `operator.list(..)` NOT to be included. In other words, the test program above should output:
```
Creating files and directories...
Files created successfully!
Listing '/':
==================================================
[DIR ] a/
[DIR ] b/
Listing '/a/':
==================================================
[DIR ] a/ab/
[FILE] a/a
Listing '/a/ab/':
==================================================
[FILE] a/ab/abb
[FILE] a/ab/aba
Done!
```
### Additional Context
OpenDAL 0.54.1
We were previously running OpenDAL 0.42.0 which did not have this bug: This is a regression.
### Are you willing to submit a PR to fix this bug?
- [ ] Yes, I would like to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "amunra (via GitHub)" <gi...@apache.org>.
amunra commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3557113261
Yes, either that or if the directory you're scanning isn't there, return an `ErrorKind::NotFound`. This latter approach would also work, and would be consistent with `operator.read`.
At the moment it's "stringly-typed programming" and the API breaks the principle of least surprise and it's why I initially genuinely believed this to be a bug.
There's two distinct cases during the iteration process:
* Handling the presence or absence of the directory that you're listing.
* Handling the children of the directory you're listing.
One needs a string comparison to distinguish these. Not ideal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3553898991
<!-- Greeting -->
Hi @amunra! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the opendal team.
<!-- Answer -->
This is a known regression in OpenDAL 0.54.1: `operator.list(path)` in the fs service is incorrectly including the directory itself in the listing, which is not intended and contradicts the documentation. Maintainers have acknowledged the bug and confirmed that the directory should be filtered out during listing. There are plans to add integration tests and a fix for this behavior, but it hasn't landed yet. You can track progress and details in [issue #5394](https://github.com/apache/opendal/issues/5394). No official workaround is available at the moment.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/81eb33fb-ed15-41ec-8e6c-071c984d8244?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/81eb33fb-ed15-41ec-8e6c-071c984d8244?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/81eb33fb-ed15-41ec-8e6c-071c984d8244?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/81eb33fb-ed15-41ec-8e6c-071c984d8244?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/81eb33fb-ed15-41ec-8e6c-071c984d8244?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/81eb33fb-ed15-41ec-8e6c-071c984d8244?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/81eb33fb-ed15-41ec-8e6c-071c984d8244?feedback_type=other)</sup> [](https://app.dosu.dev/997a2c5a-0c54-497b-989d-44651635dd36/ask?utm_source=githu
b) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/opendal/issues/6805)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3553950111
> Is it at least consistent across services?
Yes, all services follow the same behavior: return `abc/` itself if it exists.
> Out of curiosity, what is the rationale here for this behaviour?
To make the list behavior consistent across all those patterns:
- s3 vs fs
- `abc/` might not exist on s3, and there’s no way to tell if `abc/` exists
- `abc` and `abc/` can both exist at the same time on s3
- file vs dir: returning the file but not the dir seems inconsistent
> Is there a Lister option to not do this?
Not for now, but maybe we can have a layer for that (which just ignore the dir itself for you)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3553919061
<!-- Answer -->
You're right—issue #5394 was closed 8 months ago and does not appear to have resolved this bug for the fs service. The bug where `operator.list(path)` includes the directory itself is still present in 0.54.1 and is not currently tracked by any open issue or PR. Maintainers have previously acknowledged that the directory itself should not be listed and suggested filtering it out, but the fix hasn't landed yet and the issue remains unresolved as of the latest release ([source](https://github.com/apache/opendal/issues/5394), [source](https://github.com/apache/opendal/issues/3960)).
If you need this fixed urgently, you may want to consider reopening the issue or monitoring for updates. Let me know if you want help summarizing the problem for maintainers or if you need guidance on a temporary workaround.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/1d26f02c-ab04-4702-a89a-0927b0ae9797?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/1d26f02c-ab04-4702-a89a-0927b0ae9797?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/1d26f02c-ab04-4702-a89a-0927b0ae9797?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/1d26f02c-ab04-4702-a89a-0927b0ae9797?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/1d26f02c-ab04-4702-a89a-0927b0ae9797?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/1d26f02c-ab04-4702-a89a-0927b0ae9797?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/1d26f02c-ab04-4702-a89a-0927b0ae9797?feedback_type=other)</sup> [](https://app.dosu.dev/997a2c5a-0c54-497b-989d-44651635dd36/ask?utm_source=githu
b) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/opendal/issues/6805)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] usability bug: `operator.list(path)` confusingly also lists `path` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #6805: usability bug: `operator.list(path)` confusingly also lists `path`
URL: https://github.com/apache/opendal/issues/6805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "amunra (via GitHub)" <gi...@apache.org>.
amunra commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3553913507
@dosu, are you certain? That PR was closed 8 months ago.
This is quite a critical bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "amunra (via GitHub)" <gi...@apache.org>.
amunra commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3553971399
Honestly the best possible change here at this stsge is clear documentation.
If the docs say what to filter out authoritatively (I assume this is `if entry.path() == path { continue; }`, it's probably good enough.
Longer term a better API would be one that returns None if the path does not exist, and an empty iterator if the path exists, but has no children. This is how I will wrap things up within my own code.
A layer would just add complexity for no reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "amunra (via GitHub)" <gi...@apache.org>.
amunra commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3553933794
..that's an unusual choice. Is it at least consistent across services?
Out of curiosity, what is the rationale here for this behaviour?
Is there a `Lister` option to _not_ do this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3553925588
Hi @amunra, sorry our docs might be outdated. It’s expected that `list("abc/")` will return `abc/` itself if it exists, which is always true on `fs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3556602412
> Honestly the best possible change here at this stage is clear documentation.
Yes, I agree.
> Longer term a better API would be one that returns None if the path does not exist, and an empty iterator if the path exists, but has no children. This is how I will wrap things up within my own code.
list now returns `Vec<Entry>`, can you elaborate more on this design? Are you suggesting that it should return an `Option<Vec<Entry>>`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fs service, `operator.list(path)` incorrectly also lists `path` [opendal]
Posted by "amunra (via GitHub)" <gi...@apache.org>.
amunra commented on issue #6805:
URL: https://github.com/apache/opendal/issues/6805#issuecomment-3557171120
Extra subtlety about the string comparison is that one list in multiple different ways too.
For example, `operator.list("/foo/")` and `operator.list("foo/")` are effectivelty equivalent.
If I currently need to exclude the directory being listed, do I:
* Always exclude the leading `/` slash?
* Always include it?
* Does it depend on how I made the call to `operator.list(..)`?
These kind of details can make correctly handling the case quite tricky, and if I don't filter it correctly my next call to `entry.name()` could end up triggering incorrect logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org