You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opendal.apache.org by "rick-200 (via GitHub)" <gi...@apache.org> on 2025/06/21 05:52:42 UTC
Is there a way to tell if the file size is 0 or if the service doesn't provide size?
GitHub user rick-200 created a discussion: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
The entry metadata provided by the list operation may or may not contain file size information.
However, it seems to be not possible to distinguish between the following two cases: the underlying service does not provide it (e.g. Fs), or the file itself has a size of 0, since `Metadata::content_length()` returns 0 if `content_length` is `None`.
```rust
/// Content length of this entry. It will be `0` if the content length is not set by the storage services.
pub fn content_length(&self) -> u64 {
self.content_length.unwrap_or_default()
}
```
I'm not sure if I missed some API or this was intentional. It really doesn't seem to make sense to me.
GitHub link: https://github.com/apache/opendal/discussions/6323
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Posted by "rick-200 (via GitHub)" <gi...@apache.org>.
GitHub user rick-200 added a comment to the discussion: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Thank you for your reply!
However, when using the **`list`** operation, some services, such as `Fs`, do not provide the file size. For example, the following code `content_length` returns 0 for of all files.
Is `Fs` the only one who get this behavior? I understand this is due to the fact that `readdir` on unix systems only returns the filename, so extra `stat` is needed.
It does feel like there could be some confusion here, if the program relies on the file size returned by the `list` operation, this is unreliable due to the presence of `Fs`, and if the program performs the extra `stat` used to get the size, it's wasteful.
```rust
use opendal::{services, Operator};
#[tokio::main]
async fn main() {
let op = Operator::new(services::Fs::default().root("D:/")).unwrap().finish();
let entries = op.list_with("").await.unwrap();
for entry in entries {
println!("{} {}", entry.path(), entry.metadata().content_length());
}
}
```
GitHub link: https://github.com/apache/opendal/discussions/6323#discussioncomment-13536842
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Posted by "meteorgan (via GitHub)" <gi...@apache.org>.
GitHub user meteorgan added a comment to the discussion: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
In fact, `content_length` is always available. Particularly, when the service can perform `stat` and `write` operations, we have a test to ensure that.
https://github.com/apache/opendal/blob/23f01dd545ebd2a7445f08cc5e7de8a19e70fb25/core/tests/behavior/async_stat.rs#L69-L86
GitHub link: https://github.com/apache/opendal/discussions/6323#discussioncomment-13536676
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Posted by "rick-200 (via GitHub)" <gi...@apache.org>.
GitHub user rick-200 edited a comment on the discussion: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Thank you for your reply!
However, when using the **`list`** operation, some services, such as `Fs`, do not provide the file size. For example, the following code `content_length` returns 0 for of all files.
Is `Fs` the only one who get this behavior? I understand this is due to the fact that `readdir` on unix systems only returns the filename, so extra `stat` is needed.
It does feel like there could be some confusion here, if the program relies on the file size returned by the `list` operation, this is unreliable due to the presence of `Fs` (can't tell if the file size is zero or if the service isn't providing it.), and if the program performs the extra `stat` used to get the size, it's wasteful.
```rust
use opendal::{services, Operator};
#[tokio::main]
async fn main() {
let op = Operator::new(services::Fs::default().root("D:/")).unwrap().finish();
let entries = op.list_with("").await.unwrap();
for entry in entries {
println!("{} {}", entry.path(), entry.metadata().content_length());
}
}
```
GitHub link: https://github.com/apache/opendal/discussions/6323#discussioncomment-13536842
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Yes, we used to call `stat` during listing, but we've changed that behavior. Maybe we should just make `content_length` return `Option<u64>`, although that could be quite annoying since most other services do provide this field.
Another solution is to use different representations for metadata returned by the list. However, this can also make the API inconvenient to use.
GitHub link: https://github.com/apache/opendal/discussions/6323#discussioncomment-13592805
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Posted by "rick-200 (via GitHub)" <gi...@apache.org>.
GitHub user rick-200 edited a comment on the discussion: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Thank you for your reply!
However, when using the **`list`** operation, some services, such as `Fs`, do not provide the file size. For example, the following code `content_length` returns 0 for of all files.
Is `Fs` the only one who get this behavior? I understand this is due to the fact that `readdir` on unix systems only returns the filename, so extra `stat` is needed.
It does feel like there could be some confusion here, if the program relies on the file size returned by the `list` operation, this is unreliable due to the presence of `Fs` (Can't tell if the file size is zero or if the service isn't providing it.), and if the program performs the extra `stat` used to get the size, it's wasteful.
```rust
use opendal::{services, Operator};
#[tokio::main]
async fn main() {
let op = Operator::new(services::Fs::default().root("D:/")).unwrap().finish();
let entries = op.list_with("").await.unwrap();
for entry in entries {
println!("{} {}", entry.path(), entry.metadata().content_length());
}
}
```
GitHub link: https://github.com/apache/opendal/discussions/6323#discussioncomment-13536842
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Posted by "meteorgan (via GitHub)" <gi...@apache.org>.
GitHub user meteorgan added a comment to the discussion: Is there a way to tell if the file size is 0 or if the service doesn't provide size?
Hmmm, the `Fs` case is not the only case. The `content_length` returned `List` operations depends on each service's API, which, as you noted, can be confusing and needs clarification. what's your opinion ? @Xuanwo
The good news is we don't always need to use `stat` to retrieve the size, only when the size is 0.
GitHub link: https://github.com/apache/opendal/discussions/6323#discussioncomment-13538213
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org