You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2026/05/26 10:30:51 UTC
[I] bug(services/hf): avoid extra HEAD probe for bucket reads [opendal]
Xuanwo opened a new issue, #7622:
URL: https://github.com/apache/opendal/issues/7622
### Describe the bug
Users reported that reads from Hugging Face bucket through OpenDAL are slow and can hit rate limits.
For bucket repos, OpenDAL currently still calls `HfCore::maybe_xet_file()` before creating the reader. This sends a HEAD request to the resolve URL on every read to discover `X-Xet-Hash` and size metadata, even though buckets are expected to use XET.
Relevant code:
- `core/services/hf/src/reader.rs`: `HfReader::try_new()` calls `core.maybe_xet_file(path)` before reading.
- `core/services/hf/src/core.rs`: `maybe_xet_file()` issues a HEAD request and parses `X-Xet-Hash`.
- `core/services/hf/src/backend.rs`: bucket `stat()` uses the same probe.
### Steps to Reproduce
Read many objects or ranges from an HF bucket-backed OpenDAL operator.
The issue is easier to observe with workloads that repeatedly call read or range-read, such as parquet/object_store-style readers.
### Expected Behavior
HF bucket reads should avoid an extra per-read HEAD probe when the required XET metadata can be resolved through a bucket-native path, cached metadata, list/stat metadata, or another lower-request-count mechanism.
A normal read path should not double the number of HF-facing requests before data transfer starts.
### Additional Context
This is separate from #7577, which tracks HF dataset behavior-test timeouts on CI. This issue is specifically about HF bucket read request amplification and rate limiting.
Related broader API work: #5872 tracks returning metadata from read operations, which can help integrations avoid extra stat calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(services/hf): avoid extra HEAD probe for bucket reads [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #7622:
URL: https://github.com/apache/opendal/issues/7622#issuecomment-4553700961
> [@Xuanwo](https://github.com/Xuanwo) we need to call the resolve or path-info endpoints to associate a file path with a xet hash identifier. A possible "trick" is to call path-info for a directory of files instead of once per file reducing the number of http requests. Could we do that in opendal?
I'm considering skip the xet path while performing reading operations. Use `resolve` to get the endpoint and send HTTP GET to read the data. So we don't need to get the xet hash for the first place, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(services/hf): avoid extra HEAD probe for bucket reads [opendal]
Posted by "kszucs (via GitHub)" <gi...@apache.org>.
kszucs commented on issue #7622:
URL: https://github.com/apache/opendal/issues/7622#issuecomment-4543727546
@Xuanwo we need to call the resolve or path-info endpoints to associate a file path with a xet hash identifier. A possible "trick" is to call path-info for a directory of files instead of once per file reducing the number of http requests. Could we do that in opendal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(services/hf): avoid extra HEAD probe for bucket reads [opendal]
Posted by "kszucs (via GitHub)" <gi...@apache.org>.
kszucs commented on issue #7622:
URL: https://github.com/apache/opendal/issues/7622#issuecomment-4543456671
We can omit that HTTP call for bucket type repositories since buckets are xet-only, though we will still need that check for other repository types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org