You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "djouallah (via GitHub)" <gi...@apache.org> on 2026/04/22 04:12:29 UTC
[I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
djouallah opened a new issue, #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695
# `MicrosoftAzure::list_with_offset` returns empty on OneLake since 0.13.0 (regression from #623)
## Describe the bug
Against Microsoft Fabric OneLake (`*.dfs.fabric.microsoft.com`), `ObjectStore::list_with_offset(prefix, offset)` returns **zero** entries even when the prefix contains files strictly greater than `offset`. The equivalent `list(prefix)` on the same store returns the correct files, so the data is reachable — only the offset-based listing is broken.
This regressed in `object_store` 0.13.0 via #623, which replaced the default fallback with an Azure-specific implementation that uses the ADLS Gen2 `startFrom` URI parameter. OneLake's REST surface does not handle `startFrom` the same way the standard ADLS Gen2 endpoint does.
## Impact
Every downstream that uses `list_with_offset` against OneLake is broken on `object_store >= 0.13.0`:
- `delta-kernel-rs` (used by DuckDB's delta extension, delta-rs): loading a Delta table with a `_last_checkpoint` hint fails with `Invalid Checkpoint: Had a _last_checkpoint hint but didn't find any checkpoints`. See [delta-io/delta-kernel-rs#2433](https://github.com/delta-io/delta-kernel-rs/issues/2433) and the (now-closed) workaround attempt [#2437](https://github.com/delta-io/delta-kernel-rs/pull/2437).
- `lakehq/sail` (does NOT use delta-kernel-rs; independently hits the same bug): [lakehq/sail#1730](https://github.com/lakehq/sail/issues/1730).
## To Reproduce
Minimal, no-delta-kernel reproducer below. The only thing swapped between the two runs is the `object_store` pin.
`Cargo.toml`:
```toml
[package]
name = "onelake-repro"
version = "0.0.1"
edition = "2021"
[dependencies]
# Swap between "=0.12.5" (works) and "=0.13.2" (broken)
object_store = { version = "=0.13.2", features = ["azure"] }
futures = "0.3"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
url = "2"
anyhow = "1"
```
`src/main.rs`:
```rust
use std::env;
use anyhow::{anyhow, Context, Result};
use futures::stream::StreamExt;
use object_store::azure::{AzureConfigKey, MicrosoftAzureBuilder};
use object_store::path::Path;
use object_store::{ObjectMeta, ObjectStore};
#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
async fn main() -> Result<()> {
let args: Vec<String> = env::args().collect();
if args.len() != 5 {
return Err(anyhow!("usage: onelake-repro <workspace> <lakehouse> <table> <checkpoint_version>"));
}
let workspace = &args[1];
let lakehouse = &args[2];
let table = &args[3];
let ckpt_version: u64 = args[4].parse()?;
let token = env::var("AZURE_STORAGE_TOKEN")
.context("AZURE_STORAGE_TOKEN not set")?;
let url = format!(
"abfss://{workspace}@onelake.dfs.fabric.microsoft.com/{lakehouse}.Lakehouse/Tables/{table}/"
);
let store = MicrosoftAzureBuilder::new()
.with_url(url.as_str())
.with_config(AzureConfigKey::Token, token)
.build()?;
let prefix_str = format!("{lakehouse}.Lakehouse/Tables/{table}/_delta_log");
let prefix = Path::from(prefix_str.as_str());
let offset = Path::from(format!("{prefix_str}/{ckpt_version:020}").as_str());
let a = collect(store.list(Some(&prefix))).await?;
println!("A) list(prefix): {} entries", a.len());
for loc in &a { println!(" {loc}"); }
let b = collect(store.list_with_offset(Some(&prefix), &offset)).await?;
println!("\nB) list_with_offset(prefix, offset): {} entries", b.len());
for loc in &b { println!(" {loc}"); }
Ok(())
}
async fn collect<S>(mut s: S) -> Result<Vec<String>>
where S: futures::Stream<Item = object_store::Result<ObjectMeta>> + Unpin
{
let mut out = vec![];
while let Some(m) = s.next().await { out.push(m?.location.to_string()); }
out.sort();
Ok(out)
}
```
Run:
```bash
export AZURE_STORAGE_TOKEN=$(az account get-access-token --resource https://storage.azure.com/ --query accessToken -o tsv)
cargo run --release -- <workspace> <lakehouse> <table> <checkpoint_version>
```
## Expected behavior
`list_with_offset(prefix, offset)` should return exactly the files in `list(prefix)` whose location is lexicographically greater than `offset`.
## Actual behavior
Against the same OneLake table (a Delta table with `_last_checkpoint` at v10):
**With `object_store = "=0.12.5"`** (works):
```
A) list(prefix): 11 entries
_delta_log/00000000000000000005.json
_delta_log/00000000000000000006.json
_delta_log/00000000000000000007.json
_delta_log/00000000000000000008.json
_delta_log/00000000000000000009.json
_delta_log/00000000000000000010.checkpoint.parquet
_delta_log/00000000000000000010.json
_delta_log/00000000000000000011.json
_delta_log/00000000000000000012.json
_delta_log/00000000000000000013.json
_delta_log/_last_checkpoint
B) list_with_offset(prefix, _delta_log/00000000000000000010): 6 entries
_delta_log/00000000000000000010.checkpoint.parquet
_delta_log/00000000000000000010.json
_delta_log/00000000000000000011.json
_delta_log/00000000000000000012.json
_delta_log/00000000000000000013.json
_delta_log/_last_checkpoint
```
**With `object_store = "=0.13.2"`** (broken):
```
A) list(prefix): 11 entries <-- identical to above
...
B) list_with_offset(prefix, _delta_log/00000000000000000010): 0 entries
```
(Only `list_with_offset` differs between the two runs.)
## Suspected cause
[#623](https://github.com/apache/arrow-rs-object-store/pull/623) added a direct `list_with_offset` implementation for Azure that sends `startFrom=<offset>` per the [ADLS Gen2 list-blobs API](https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs?view=rest-storageservices-datalakestoragegen2-2019-12-12&tabs=microsoft-entra-id#uri-parameters). OneLake's endpoint apparently does not implement `startFrom` compatibly — it returns an empty list regardless of the offset value.
This matches `lonless9`'s analysis on [lakehq/sail#1730](https://github.com/lakehq/sail/issues/1730) and the related [Azurite#2619](https://github.com/Azure/Azurite/issues/2619#issuecomment-3660701055).
## Environment
- `object_store` 0.13.2 (and 0.13.0, 0.13.1 — all contain #623)
- OneLake endpoint `onelake.dfs.fabric.microsoft.com`
- Service-principal / Azure CLI bearer token (same auth in both runs; auth is not the issue)
- Observed on Windows 11 / rustc 1.95.0, but not platform-dependent
---
Repro and report co-drafted with [Claude Code](https://claude.com/claude-code) (Claude Opus 4.7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695#issuecomment-4300225759
> Both IMO are bugs in Onelake, and either would allow us to handle this more gracefully.
Thanks for the ping @tustvold. Smriti and I both work on the OneLake team. Let me take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah closed issue #695: MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623)
URL: https://github.com/apache/arrow-rs-object-store/issues/695
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695#issuecomment-4300609376
Thanks for the repro and pointers. I was able to confirm this locally with a OneLake environment. This appears to be an API parity issue on the OneLake API side. We'll investigate and fix it.
In the meantime, I think we should address the regression for OneLake users. I like @tustvold's suggestion; we can:
1. Add a config option to disable `startFrom`
2. Automatically disable `startFrom` for OneLake endpoints
I'm happy to put up a fix for both. Once `startFrom` is fixed/supported in OneLake, we can revisit and re-enable the optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on issue #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695#issuecomment-4293523636
Follow-up: confirmed the issue is **OneLake-specific**, not a general regression against Azure storage accounts. Tested against a standard ADLS Gen2 account (`*.dfs.core.windows.net`) with the same `object_store = "=0.13.2"` binary.
Test table `_delta_log/` listing (3 files):
- `_delta_log/00000000000000000000.crc`
- `_delta_log/00000000000000000000.json`
- `_delta_log/_autostats/00000000000000000000.0000000000.stats.json`
`list_with_offset` results on **standard ADLS Gen2** (`dataiceberg.dfs.core.windows.net`):
| offset | files > offset (expected) | returned |
|---|---|---|
| `_delta_log/00000000000000000000` | 3 (all files; `.crc`, `.json`, `_autostats` all lex > bare offset) | 3 ✓ |
| `_delta_log/00000000000000000001` | 1 (`_autostats` only; `_` 0x5F > `0` 0x30, while `00...00.crc/json` < `...01`) | 1 ✓ |
| `_delta_log/00000000000000000099` | 1 (`_autostats` only; same reasoning) | 1 ✓ |
So `startFrom` does exactly what #623 intends on standard ADLS Gen2 — a proper lexicographic server-side cursor.
Against OneLake (same binary, same auth flow, a `_delta_log/` with 11 files including `00000000000000000010.checkpoint.parquet`, `.../00000000000000000011.json` etc.), `list_with_offset(prefix, _delta_log/00000000000000000010)` returns **0** files instead of the expected 6. So the OneLake endpoint either:
- Rejects or mis-parses the `startFrom` parameter entirely, OR
- Interprets it as an inclusive prefix boundary rather than an exclusive lex cursor
Either way, the OneLake list-blobs REST surface is not compatible with the `startFrom` contract that #623 relies on. Standard ADLS Gen2 users are unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on issue #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695#issuecomment-4726448070
confirmed issue fixed upstream by onelake team :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "omar (via GitHub)" <gi...@apache.org>.
omar commented on issue #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695#issuecomment-4297149921
Related to @tustvold point, from [Azure docs titled “OneLake API parity with Azure Storage”](https://learn.microsoft.com/en-us/fabric/onelake/onelake-api-parity):
> OneLake supports the same APIs as Azure Data Lake Storage (ADLS) and Azure Blob Storage. This API parity enables users to read, write, and manage their data in OneLake with the tools they already use today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695#issuecomment-4307029659
A couple of updates:
We found that this issue only affects the endpoints using "friendly names". Endpoints using GUIDs works as intended.
I opened #696 to disable the `startFrom` optimization for OneLake endpoints. I also opened a tracking issue (#697) to re-enable the optimization once we (OneLake team) fixes the bug on our side.
I decided not to add a config to disable `startFrom` entirely, as it is more involved. We can fix forward OneLake first and then follow up with a config option to disable `startFrom`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695#issuecomment-4294703296
Does onelake claim to be ADLS Gen2 compatible? If so this is really an Azure issue...
That being said, providing an option to disable offset pushdown makes sense to me, and would provide an escape hatch for scenarios such as this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695#issuecomment-4298426884
As stated before we should add an escape valve to allow disabling startFrom, and set this automatically when parsing a Onelake endpoint, however, I should state:
* Onelake is silently ignoring arguments it doesn't understand
* Onelake is silently ignoring a version it doesn't support
Both IMO are bugs in Onelake, and either would allow us to handle this more gracefully.
Tagging @kevinjqliu and @SmritiAgrawal04 as you've worked on onelake support in this library before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MicrosoftAzure::list_with_offset returns empty on OneLake since 0.13.0 (regression from #623) [arrow-rs-object-store]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #695:
URL: https://github.com/apache/arrow-rs-object-store/issues/695#issuecomment-4663874998
I closed #696 since the underlying issue is fixed and deployed on the OneLake side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org