You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/25 16:39:49 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by e5d0b7575f1b7bfe4b19da7246f2d9733357e18f
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ffde48b7af Publish built docs triggered by e5d0b7575f1b7bfe4b19da7246f2d9733357e18f
ffde48b7af is described below
commit ffde48b7af14588f7d23ae2ea66598c50e8390d8
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Sep 25 16:39:45 2026 +0000
Publish built docs triggered by e5d0b7575f1b7bfe4b19da7246f2d9733357e18f
---
...id-013012468fa27c9e9fdb0a14196d357c619c68a8.svg | 2 +-
_sources/about/versioning_policy.md.txt | 6 ++++-
.../s3-credential-provider-design.md.txt | 17 +++++++++++++
.../latest/s3-credential-providers.md.txt | 29 ++++++++++++++++++++++
about/versioning_policy.html | 6 ++++-
.../s3-credential-provider-design.html | 15 +++++++++++
searchindex.js | 2 +-
user-guide/latest/s3-credential-providers.html | 24 ++++++++++++++++++
8 files changed, 97 insertions(+), 4 deletions(-)
diff --git a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
index 21cdfcca5b..658c71ac4e 100644
--- a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
+++ b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/about/versioning_policy.md.txt b/_sources/about/versioning_policy.md.txt
index 7b6e4974c8..90319ed160 100644
--- a/_sources/about/versioning_policy.md.txt
+++ b/_sources/about/versioning_policy.md.txt
@@ -276,13 +276,17 @@ here, because a vendor jar built against one Comet release is loaded by another.
The SPI consists of:
- `CometS3CredentialProvider`, the interface a vendor implements.
+- `CometS3LocationScopedCredentialProvider`, an optional extension of it for buckets whose credentials
+ differ by location.
- `CometS3Credentials`, the value a provider returns.
- `CometS3CredentialContext` and `CometS3AccessMode`, describing the request being served.
Additive changes are allowed in a minor release, for example a new accessor on
`CometS3CredentialContext`, because a vendor jar compiled against an earlier `1.x` continues to
load and run. Any change that would break such a jar, including adding an abstract method to
-`CometS3CredentialProvider` without a default implementation, requires a major release.
+`CometS3CredentialProvider` or `CometS3LocationScopedCredentialProvider` without a default
+implementation, requires a major release. The same holds for changing the meaning of an existing
+method, such as how `CometS3LocationScopedCredentialProvider` matches a path to a location.
`CometS3CredentialDispatcher` is the JNI entry point Comet uses to reach a provider. It is internal
despite living in the same package, and vendors must not call it.
diff --git a/_sources/contributor-guide/s3-credential-provider-design.md.txt b/_sources/contributor-guide/s3-credential-provider-design.md.txt
index 25d1951eca..8fa449dd55 100644
--- a/_sources/contributor-guide/s3-credential-provider-design.md.txt
+++ b/_sources/contributor-guide/s3-credential-provider-design.md.txt
@@ -73,6 +73,23 @@ Comet's bridge does not maintain a TTL cache, schedule refresh, or broadcast cat
A Comet-side cache would have to either expose a tuning knob (TTL, max size, eviction policy) and grow over time, or be hardcoded and surprise vendors whose policies disagree. The bridge intentionally has neither and forwards every call.
+## Location-scoped credentials on the Parquet path
+
+`object_store::CredentialProvider::get_credential` receives no request path, so one `AmazonS3` store presents one credential, and the process-wide `object_store` cache holds one store per `(scheme://bucket, config_hash, hdfs_backend)`. A base provider therefore gets one credential per bucket, requested with the path of the first file Comet reads. A bucket whose policies differ by location (one for `warehouse/sales`, another for `warehouse/finance`) needs a store per location and somethin [...]
+
+`CometS3LocationScopedCredentialProvider` supplies that. `getPolicyLocations(bucket)` returns every location in the bucket that has its own policy, and `create_store` returns a `LocationScopedObjectStore` for the bucket instead of a plain store:
+
+- It is cached and registered like any other S3 store, one per key, so later scans on the bucket share it and the cache and registry keep one store per identity. As for any store, scans that miss the cache at the same moment each build one, and the last one cached wins.
+- Each request is served by the store of the longest location covering its path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. Routing is per request, so a partition whose files span several locations reads each file with its own location's credential.
+- A location's store is an `AmazonS3` whose bridge is bound to the location itself, so the vendor sees one stable path per location. The bridge is derived from the bucket's bridge, sharing its provider registration, so creating it calls no `ensureInitialized` and loads no classes on the Tokio worker that usually creates it. It is built on first use, usually inside an async read, from an `S3StoreTemplate` that resolved the region when the bucket's store was created, so building never bloc [...]
+- The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, so the store fetches the locations again and retries the request once if its path now routes elsewhere. Every attempt, successful or not, starts a new snapshot generation, so requests routed from the same generation share one attempt and a failed attempt fails them all instead of each calling the provider in turn. The retry budget is per request, with no state that outlives it.
+
+The locations come from asking for the bucket's whole list rather than which prefixes one session covers. Asking per session leaves Comet to discover the other scopes from 403s, which needs mutable per-store state and cannot route a partition that spans scopes. With the whole list up front, routing is a function of the path.
+
+This is consistent with [Why no Comet-side cache](#why-no-comet-side-cache): the store caches no credentials, and every request still calls `getCredentialsForPath`. What it keeps is the provider's location list and a store for each location that has been read, so it grows with the vendor's policy list, not with the paths read. The list is only fetched again after a 403, so a change that causes none is not seen until the executor builds a new store.
+
+The dispatcher returns `null` for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the 403 retry, since Comet only reads through these stores.
+
## Path-specific behavior
`object_store::CredentialProvider` and `reqsign_core::ProvideCredential` differ in what they consume:
diff --git a/_sources/user-guide/latest/s3-credential-providers.md.txt b/_sources/user-guide/latest/s3-credential-providers.md.txt
index 33777c497a..df3d76005b 100644
--- a/_sources/user-guide/latest/s3-credential-providers.md.txt
+++ b/_sources/user-guide/latest/s3-credential-providers.md.txt
@@ -201,6 +201,35 @@ public CometS3Credentials getCredentialsForPath(CometS3CredentialContext ctx) th
}
```
+### Credentials per location
+
+On the Parquet path, a `CometS3CredentialProvider` gets one credential per bucket: Comet requests it with the path of the first file it reads from the bucket and uses it for every file there. If your policies differ by location within a bucket, for example one policy for `warehouse/sales` and another for `warehouse/finance`, implement `CometS3LocationScopedCredentialProvider` and tell Comet where those locations are:
+
+```java
+public final class MyLocationProvider implements CometS3LocationScopedCredentialProvider {
+ @Override
+ public List<String> getPolicyLocations(String bucket) throws Exception {
+ return policyService.locationsWithPolicies(bucket); // ["warehouse/sales", "warehouse/finance"]
+ }
+
+ @Override
+ public CometS3Credentials getCredentialsForPath(CometS3CredentialContext ctx) throws Exception {
+ // ctx.getPath() is a returned location with a leading slash, or "/" for the bucket root.
+ return sessionForLocation(ctx.getBucket(), ctx.getPath(), ctx.getMode());
+ }
+}
+```
+
+Comet serves each request with the credential of the longest location that covers its path. A location covers a path when the path is the location itself or lies below it, compared one `/`-separated segment at a time, so `warehouse/sales` covers `warehouse/sales/part-0.parquet` but not `warehouse/sales_eu/part-0.parquet`. The bucket root covers every path that no returned location covers, and an empty list serves the whole bucket with the root's credential. Write locations the way `Comet [...]
+
+Comet requests a location's credential by calling `getCredentialsForPath` with the location as the path, as you returned it but with a leading slash. Every request under a location shares that credential, so it must authorize every path the location is the longest match for, and your cache can key on the location. Locations apply to Comet's native Parquet reads only; Iceberg reads call `getCredentialsForPath` as they do for any provider.
+
+**When Comet asks.** Comet calls `getPolicyLocations` when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its path now falls under a different location, so a location added while a job runs is picked [...]
+
+**Failures.** If `getPolicyLocations` throws or returns `null`, or returns a location that is `null` or invalid, the read fails. A location is invalid if, once decoded, it is not valid UTF-8 or has a segment that is empty, `.`, `..`, or contains a control character, so a URI such as `s3://bucket/a` is invalid too. Comet does not fall back to a broader credential.
+
+**Backward compatibility.** Providers that implement only `CometS3CredentialProvider` are unaffected. Comet calls nothing new on them and keeps one credential per bucket, as before.
+
### Composing multiple credential backends
A single configured provider class is the dispatcher. If a vendor needs to route across several credential backends (per bucket, per path prefix, per tenant), the dispatch lives inside the vendor's class:
diff --git a/about/versioning_policy.html b/about/versioning_policy.html
index ee407934ad..3453e1bee6 100644
--- a/about/versioning_policy.html
+++ b/about/versioning_policy.html
@@ -732,13 +732,17 @@ here, because a vendor jar built against one Comet release is loaded by another.
<p>The SPI consists of:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">CometS3CredentialProvider</span></code>, the interface a vendor implements.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">CometS3LocationScopedCredentialProvider</span></code>, an optional extension of it for buckets whose credentials
+differ by location.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">CometS3Credentials</span></code>, the value a provider returns.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">CometS3CredentialContext</span></code> and <code class="docutils literal notranslate"><span class="pre">CometS3AccessMode</span></code>, describing the request being served.</p></li>
</ul>
<p>Additive changes are allowed in a minor release, for example a new accessor on
<code class="docutils literal notranslate"><span class="pre">CometS3CredentialContext</span></code>, because a vendor jar compiled against an earlier <code class="docutils literal notranslate"><span class="pre">1.x</span></code> continues to
load and run. Any change that would break such a jar, including adding an abstract method to
-<code class="docutils literal notranslate"><span class="pre">CometS3CredentialProvider</span></code> without a default implementation, requires a major release.</p>
+<code class="docutils literal notranslate"><span class="pre">CometS3CredentialProvider</span></code> or <code class="docutils literal notranslate"><span class="pre">CometS3LocationScopedCredentialProvider</span></code> without a default
+implementation, requires a major release. The same holds for changing the meaning of an existing
+method, such as how <code class="docutils literal notranslate"><span class="pre">CometS3LocationScopedCredentialProvider</span></code> matches a path to a location.</p>
<p><code class="docutils literal notranslate"><span class="pre">CometS3CredentialDispatcher</span></code> is the JNI entry point Comet uses to reach a provider. It is internal
despite living in the same package, and vendors must not call it.</p>
<p>See the <a class="reference internal" href="../user-guide/latest/s3-credential-providers.html"><span class="std std-doc">S3 Credential Providers</span></a> guide for the
diff --git a/contributor-guide/s3-credential-provider-design.html b/contributor-guide/s3-credential-provider-design.html
index c0cd8c88df..c7eeaccb39 100644
--- a/contributor-guide/s3-credential-provider-design.html
+++ b/contributor-guide/s3-credential-provider-design.html
@@ -602,6 +602,20 @@ under the License.
</ul>
<p>A Comet-side cache would have to either expose a tuning knob (TTL, max size, eviction policy) and grow over time, or be hardcoded and surprise vendors whose policies disagree. The bridge intentionally has neither and forwards every call.</p>
</section>
+<section id="location-scoped-credentials-on-the-parquet-path">
+<h2>Location-scoped credentials on the Parquet path<a class="headerlink" href="#location-scoped-credentials-on-the-parquet-path" title="Link to this heading">#</a></h2>
+<p><code class="docutils literal notranslate"><span class="pre">object_store::CredentialProvider::get_credential</span></code> receives no request path, so one <code class="docutils literal notranslate"><span class="pre">AmazonS3</span></code> store presents one credential, and the process-wide <code class="docutils literal notranslate"><span class="pre">object_store</span></code> cache holds one store per <code class="docutils literal notranslate"><span class="pre">(scheme://bucket,</sp [...]
+<p><code class="docutils literal notranslate"><span class="pre">CometS3LocationScopedCredentialProvider</span></code> supplies that. <code class="docutils literal notranslate"><span class="pre">getPolicyLocations(bucket)</span></code> returns every location in the bucket that has its own policy, and <code class="docutils literal notranslate"><span class="pre">create_store</span></code> returns a <code class="docutils literal notranslate"><span class="pre">LocationScopedObjectStore</span> [...]
+<ul class="simple">
+<li><p>It is cached and registered like any other S3 store, one per key, so later scans on the bucket share it and the cache and registry keep one store per identity. As for any store, scans that miss the cache at the same moment each build one, and the last one cached wins.</p></li>
+<li><p>Each request is served by the store of the longest location covering its path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. Routing is per request, so a partition whose files span several locations reads each file with its own location’s credential.</p></li>
+<li><p>A location’s store is an <code class="docutils literal notranslate"><span class="pre">AmazonS3</span></code> whose bridge is bound to the location itself, so the vendor sees one stable path per location. The bridge is derived from the bucket’s bridge, sharing its provider registration, so creating it calls no <code class="docutils literal notranslate"><span class="pre">ensureInitialized</span></code> and loads no classes on the Tokio worker that usually creates it. It is built on [...]
+<li><p>The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, so the store fetches the locations again and retries the request once if its path now routes elsewhere. Every attempt, successful or not, starts a new snapshot generation, so requests routed from the same generation share one attempt and a failed attempt fails them all instead of each calling the provider in turn. The retry budget is per request, with no state that outliv [...]
+</ul>
+<p>The locations come from asking for the bucket’s whole list rather than which prefixes one session covers. Asking per session leaves Comet to discover the other scopes from 403s, which needs mutable per-store state and cannot route a partition that spans scopes. With the whole list up front, routing is a function of the path.</p>
+<p>This is consistent with <a class="reference internal" href="#why-no-comet-side-cache">Why no Comet-side cache</a>: the store caches no credentials, and every request still calls <code class="docutils literal notranslate"><span class="pre">getCredentialsForPath</span></code>. What it keeps is the provider’s location list and a store for each location that has been read, so it grows with the vendor’s policy list, not with the paths read. The list is only fetched again after a 403, so a [...]
+<p>The dispatcher returns <code class="docutils literal notranslate"><span class="pre">null</span></code> for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the 403 retry, since Comet only reads through these stores.</p>
+</section>
<section id="path-specific-behavior">
<h2>Path-specific behavior<a class="headerlink" href="#path-specific-behavior" title="Link to this heading">#</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">object_store::CredentialProvider</span></code> and <code class="docutils literal notranslate"><span class="pre">reqsign_core::ProvideCredential</span></code> differ in what they consume:</p>
@@ -709,6 +723,7 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-fqcn-dispatchkey-catalogproperties-keying">Why <code class="docutils literal notranslate"><span class="pre">(FQCN,</span> <span class="pre">dispatchKey,</span> <span class="pre">catalogProperties)</span></code> keying</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-fresh-construction-in-initialize-not-probing-a-jvm-wide-static">Why fresh construction in <code class="docutils literal notranslate"><span class="pre">initialize</span></code>, not probing a JVM-wide static</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-no-comet-side-cache">Why no Comet-side cache</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#location-scoped-credentials-on-the-parquet-path">Location-scoped credentials on the Parquet path</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#path-specific-behavior">Path-specific behavior</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#property-bag-handling-on-the-iceberg-path">Property-bag handling on the Iceberg path</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#returns-or-throws-not-a-fall-through-value">Returns or throws, not a fall-through value</a></li>
diff --git a/searchindex.js b/searchindex.js
index d3cc8a98d1..af8193e669 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [74, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [74, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/s3-credential-providers.html b/user-guide/latest/s3-credential-providers.html
index cf9bbf9c90..31a123a652 100644
--- a/user-guide/latest/s3-credential-providers.html
+++ b/user-guide/latest/s3-credential-providers.html
@@ -945,6 +945,29 @@ under the License.
</pre></div>
</div>
</section>
+<section id="credentials-per-location">
+<h3>Credentials per location<a class="headerlink" href="#credentials-per-location" title="Link to this heading">#</a></h3>
+<p>On the Parquet path, a <code class="docutils literal notranslate"><span class="pre">CometS3CredentialProvider</span></code> gets one credential per bucket: Comet requests it with the path of the first file it reads from the bucket and uses it for every file there. If your policies differ by location within a bucket, for example one policy for <code class="docutils literal notranslate"><span class="pre">warehouse/sales</span></code> and another for <code class="docutils literal notrans [...]
+<div class="highlight-java notranslate"><div class="highlight"><pre><span></span><span class="kd">public</span><span class="w"> </span><span class="kd">final</span><span class="w"> </span><span class="kd">class</span> <span class="nc">MyLocationProvider</span><span class="w"> </span><span class="kd">implements</span><span class="w"> </span><span class="n">CometS3LocationScopedCredentialProvider</span><span class="w"> </span><span class="p">{</span>
+<span class="w"> </span><span class="nd">@Override</span>
+<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="nf">getPolicyLocations</span><span class="p">(</span><span class="n">String</span><span class="w"> </span><span class="n">bucket</span><span class="p">)</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">Exception</ [...]
+<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">policyService</span><span class="p">.</span><span class="na">locationsWithPolicies</span><span class="p">(</span><span class="n">bucket</span><span class="p">);</span><span class="w"> </span><span class="c1">// ["warehouse/sales", "warehouse/finance"]</span>
+<span class="w"> </span><span class="p">}</span>
+
+<span class="w"> </span><span class="nd">@Override</span>
+<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="n">CometS3Credentials</span><span class="w"> </span><span class="nf">getCredentialsForPath</span><span class="p">(</span><span class="n">CometS3CredentialContext</span><span class="w"> </span><span class="n">ctx</span><span class="p">)</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">Exception</span><span class="w"> </span><span class="p">{</span>
+<span class="w"> </span><span class="c1">// ctx.getPath() is a returned location with a leading slash, or "/" for the bucket root.</span>
+<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">sessionForLocation</span><span class="p">(</span><span class="n">ctx</span><span class="p">.</span><span class="na">getBucket</span><span class="p">(),</span><span class="w"> </span><span class="n">ctx</span><span class="p">.</span><span class="na">getPath</span><span class="p">(),</span><span class="w"> </span><span class="n">ctx</span><span class="p">.</span><span class="na">getMode</spa [...]
+<span class="w"> </span><span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
+</div>
+<p>Comet serves each request with the credential of the longest location that covers its path. A location covers a path when the path is the location itself or lies below it, compared one <code class="docutils literal notranslate"><span class="pre">/</span></code>-separated segment at a time, so <code class="docutils literal notranslate"><span class="pre">warehouse/sales</span></code> covers <code class="docutils literal notranslate"><span class="pre">warehouse/sales/part-0.parquet</span [...]
+<p>Comet requests a location’s credential by calling <code class="docutils literal notranslate"><span class="pre">getCredentialsForPath</span></code> with the location as the path, as you returned it but with a leading slash. Every request under a location shares that credential, so it must authorize every path the location is the longest match for, and your cache can key on the location. Locations apply to Comet’s native Parquet reads only; Iceberg reads call <code class="docutils liter [...]
+<p><strong>When Comet asks.</strong> Comet calls <code class="docutils literal notranslate"><span class="pre">getPolicyLocations</span></code> when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its [...]
+<p><strong>Failures.</strong> If <code class="docutils literal notranslate"><span class="pre">getPolicyLocations</span></code> throws or returns <code class="docutils literal notranslate"><span class="pre">null</span></code>, or returns a location that is <code class="docutils literal notranslate"><span class="pre">null</span></code> or invalid, the read fails. A location is invalid if, once decoded, it is not valid UTF-8 or has a segment that is empty, <code class="docutils literal notr [...]
+<p><strong>Backward compatibility.</strong> Providers that implement only <code class="docutils literal notranslate"><span class="pre">CometS3CredentialProvider</span></code> are unaffected. Comet calls nothing new on them and keeps one credential per bucket, as before.</p>
+</section>
<section id="composing-multiple-credential-backends">
<h3>Composing multiple credential backends<a class="headerlink" href="#composing-multiple-credential-backends" title="Link to this heading">#</a></h3>
<p>A single configured provider class is the dispatcher. If a vendor needs to route across several credential backends (per bucket, per path prefix, per tenant), the dispatch lives inside the vendor’s class:</p>
@@ -1100,6 +1123,7 @@ under the License.
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#caching-refresh-and-distribution">Caching, refresh, and distribution</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#returned-fields">Returned fields</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#returns-or-throws">Returns or throws</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#credentials-per-location">Credentials per location</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#composing-multiple-credential-backends">Composing multiple credential backends</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#reference-implementation-iceberg-rest-vended-credentials">Reference implementation: Iceberg REST vended credentials</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#access-mode">Access mode</a></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org