You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/25 23:10:32 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 36ab57c689d4551b060637ec311f0e8da3b96a62
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4c04c9e10e Publish built docs triggered by 36ab57c689d4551b060637ec311f0e8da3b96a62
4c04c9e10e is described below
commit 4c04c9e10ecea31f5b82fe8b83fd02549ff411fb
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Sep 25 23:10:26 2026 +0000
Publish built docs triggered by 36ab57c689d4551b060637ec311f0e8da3b96a62
---
_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg | 2 +-
_sources/contributor-guide/s3-credential-provider-design.md.txt | 6 +++---
_sources/user-guide/latest/s3-credential-providers.md.txt | 2 +-
contributor-guide/s3-credential-provider-design.html | 6 +++---
searchindex.js | 2 +-
user-guide/latest/s3-credential-providers.html | 2 +-
6 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
index 1fb2c82c2a..a319c8efdb 100644
--- a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
+++ b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/s3-credential-provider-design.md.txt b/_sources/contributor-guide/s3-credential-provider-design.md.txt
index 8fa449dd55..e67168055f 100644
--- a/_sources/contributor-guide/s3-credential-provider-design.md.txt
+++ b/_sources/contributor-guide/s3-credential-provider-design.md.txt
@@ -82,13 +82,13 @@ A Comet-side cache would have to either expose a tuning knob (TTL, max size, evi
- It is cached and registered like any other S3 store, one per key, so later scans on the bucket share it and the cache and registry keep one store per identity. As for any store, scans that miss the cache at the same moment each build one, and the last one cached wins.
- Each request is served by the store of the longest location covering its path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. Routing is per request, so a partition whose files span several locations reads each file with its own location's credential.
- A location's store is an `AmazonS3` whose bridge is bound to the location itself, so the vendor sees one stable path per location. The bridge is derived from the bucket's bridge, sharing its provider registration, so creating it calls no `ensureInitialized` and loads no classes on the Tokio worker that usually creates it. It is built on first use, usually inside an async read, from an `S3StoreTemplate` that resolved the region when the bucket's store was created, so building never bloc [...]
-- The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, so the store fetches the locations again and retries the request once if its path now routes elsewhere. Every attempt, successful or not, starts a new snapshot generation, so requests routed from the same generation share one attempt and a failed attempt fails them all instead of each calling the provider in turn. The retry budget is per request, with no state that outlives it.
+- The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, and so is a failure to get a location's credential, because a provider with no policy for a path throws rather than vending a credential S3 would reject. On either, the store fetches the locations again and retries the request once if its path now routes elsewhere. The bridge gives its failures a `CredentialProviderError` source, which `object_store` passes through to the read [...]
The locations come from asking for the bucket's whole list rather than which prefixes one session covers. Asking per session leaves Comet to discover the other scopes from 403s, which needs mutable per-store state and cannot route a partition that spans scopes. With the whole list up front, routing is a function of the path.
-This is consistent with [Why no Comet-side cache](#why-no-comet-side-cache): the store caches no credentials, and every request still calls `getCredentialsForPath`. What it keeps is the provider's location list and a store for each location that has been read, so it grows with the vendor's policy list, not with the paths read. The list is only fetched again after a 403, so a change that causes none is not seen until the executor builds a new store.
+This is consistent with [Why no Comet-side cache](#why-no-comet-side-cache): the store caches no credentials, and every request still calls `getCredentialsForPath`. What it keeps is the provider's location list and a store for each location that has been read, so it grows with the vendor's policy list, not with the paths read. The list is only fetched again after a 403 or a credential failure, so a change that causes neither is not seen until the executor builds a new store.
-The dispatcher returns `null` for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the 403 retry, since Comet only reads through these stores.
+The dispatcher returns `null` for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the retry, since Comet only reads through these stores.
## Path-specific behavior
diff --git a/_sources/user-guide/latest/s3-credential-providers.md.txt b/_sources/user-guide/latest/s3-credential-providers.md.txt
index e97e22c4eb..df7fe62ac7 100644
--- a/_sources/user-guide/latest/s3-credential-providers.md.txt
+++ b/_sources/user-guide/latest/s3-credential-providers.md.txt
@@ -222,7 +222,7 @@ Comet serves each request with the credential of the longest location that cover
Comet requests a location's credential by calling `getCredentialsForPath` with the location as the path, as you returned it but with a leading slash. Every request under a location shares that credential, so it must authorize every path the location is the longest match for, and your cache can key on the location. Locations apply to Comet's native Parquet reads only; Iceberg reads call `getCredentialsForPath` as they do for any provider.
-**When Comet asks.** Comet calls `getPolicyLocations` when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its path now falls under a different location, so a location added while a job runs is picked [...]
+**When Comet asks.** Comet calls `getPolicyLocations` when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, or because `getCredentialsForPath` threw for the location Comet sent it to, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its path now fall [...]
**Failures.** If `getPolicyLocations` throws or returns `null`, or returns a location that is `null` or invalid, the read fails. A location is invalid if, once decoded, it is not valid UTF-8 or has a segment that is empty, `.`, `..`, or contains a control character, so a URI such as `s3://bucket/a` is invalid too. Comet does not fall back to a broader credential.
diff --git a/contributor-guide/s3-credential-provider-design.html b/contributor-guide/s3-credential-provider-design.html
index c7eeaccb39..7919d84732 100644
--- a/contributor-guide/s3-credential-provider-design.html
+++ b/contributor-guide/s3-credential-provider-design.html
@@ -610,11 +610,11 @@ under the License.
<li><p>It is cached and registered like any other S3 store, one per key, so later scans on the bucket share it and the cache and registry keep one store per identity. As for any store, scans that miss the cache at the same moment each build one, and the last one cached wins.</p></li>
<li><p>Each request is served by the store of the longest location covering its path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. Routing is per request, so a partition whose files span several locations reads each file with its own location’s credential.</p></li>
<li><p>A location’s store is an <code class="docutils literal notranslate"><span class="pre">AmazonS3</span></code> whose bridge is bound to the location itself, so the vendor sees one stable path per location. The bridge is derived from the bucket’s bridge, sharing its provider registration, so creating it calls no <code class="docutils literal notranslate"><span class="pre">ensureInitialized</span></code> and loads no classes on the Tokio worker that usually creates it. It is built on [...]
-<li><p>The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, so the store fetches the locations again and retries the request once if its path now routes elsewhere. Every attempt, successful or not, starts a new snapshot generation, so requests routed from the same generation share one attempt and a failed attempt fails them all instead of each calling the provider in turn. The retry budget is per request, with no state that outliv [...]
+<li><p>The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, and so is a failure to get a location’s credential, because a provider with no policy for a path throws rather than vending a credential S3 would reject. On either, the store fetches the locations again and retries the request once if its path now routes elsewhere. The bridge gives its failures a <code class="docutils literal notranslate"><span class="pre">CredentialProvi [...]
</ul>
<p>The locations come from asking for the bucket’s whole list rather than which prefixes one session covers. Asking per session leaves Comet to discover the other scopes from 403s, which needs mutable per-store state and cannot route a partition that spans scopes. With the whole list up front, routing is a function of the path.</p>
-<p>This is consistent with <a class="reference internal" href="#why-no-comet-side-cache">Why no Comet-side cache</a>: the store caches no credentials, and every request still calls <code class="docutils literal notranslate"><span class="pre">getCredentialsForPath</span></code>. What it keeps is the provider’s location list and a store for each location that has been read, so it grows with the vendor’s policy list, not with the paths read. The list is only fetched again after a 403, so a [...]
-<p>The dispatcher returns <code class="docutils literal notranslate"><span class="pre">null</span></code> for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the 403 retry, since Comet only reads through these stores.</p>
+<p>This is consistent with <a class="reference internal" href="#why-no-comet-side-cache">Why no Comet-side cache</a>: the store caches no credentials, and every request still calls <code class="docutils literal notranslate"><span class="pre">getCredentialsForPath</span></code>. What it keeps is the provider’s location list and a store for each location that has been read, so it grows with the vendor’s policy list, not with the paths read. The list is only fetched again after a 403 or a c [...]
+<p>The dispatcher returns <code class="docutils literal notranslate"><span class="pre">null</span></code> for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the retry, since Comet only reads through these stores.</p>
</section>
<section id="path-specific-behavior">
<h2>Path-specific behavior<a class="headerlink" href="#path-specific-behavior" title="Link to this heading">#</a></h2>
diff --git a/searchindex.js b/searchindex.js
index 7075329cdc..396ec6b30e 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [74, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [74, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/s3-credential-providers.html b/user-guide/latest/s3-credential-providers.html
index b580f1f2aa..71bf77f1fd 100644
--- a/user-guide/latest/s3-credential-providers.html
+++ b/user-guide/latest/s3-credential-providers.html
@@ -963,7 +963,7 @@ under the License.
</div>
<p>Comet serves each request with the credential of the longest location that covers its path. A location covers a path when the path is the location itself or lies below it, compared one <code class="docutils literal notranslate"><span class="pre">/</span></code>-separated segment at a time, so <code class="docutils literal notranslate"><span class="pre">warehouse/sales</span></code> covers <code class="docutils literal notranslate"><span class="pre">warehouse/sales/part-0.parquet</span [...]
<p>Comet requests a location’s credential by calling <code class="docutils literal notranslate"><span class="pre">getCredentialsForPath</span></code> with the location as the path, as you returned it but with a leading slash. Every request under a location shares that credential, so it must authorize every path the location is the longest match for, and your cache can key on the location. Locations apply to Comet’s native Parquet reads only; Iceberg reads call <code class="docutils liter [...]
-<p><strong>When Comet asks.</strong> Comet calls <code class="docutils literal notranslate"><span class="pre">getPolicyLocations</span></code> when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its [...]
+<p><strong>When Comet asks.</strong> Comet calls <code class="docutils literal notranslate"><span class="pre">getPolicyLocations</span></code> when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, or because <code class="docutils literal notranslate"><span class="pre">getCredentialsForPath</span></code> [...]
<p><strong>Failures.</strong> If <code class="docutils literal notranslate"><span class="pre">getPolicyLocations</span></code> throws or returns <code class="docutils literal notranslate"><span class="pre">null</span></code>, or returns a location that is <code class="docutils literal notranslate"><span class="pre">null</span></code> or invalid, the read fails. A location is invalid if, once decoded, it is not valid UTF-8 or has a segment that is empty, <code class="docutils literal notr [...]
<p><strong>Backward compatibility.</strong> Providers that implement only <code class="docutils literal notranslate"><span class="pre">CometS3CredentialProvider</span></code> are unaffected. Comet calls nothing new on them and keeps one credential per bucket, as before.</p>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org