You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/12/04 19:13:37 UTC
(datafusion) branch asf-site updated: Publish built docs triggered by d150af7b2b7aeadff47d74f8e3e44a2bbab80bd7
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 3e67a5e4ab Publish built docs triggered by d150af7b2b7aeadff47d74f8e3e44a2bbab80bd7
3e67a5e4ab is described below
commit 3e67a5e4ab8f9e1d8f5ba0e631309cc16d4ff56d
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Dec 4 19:13:33 2025 +0000
Publish built docs triggered by d150af7b2b7aeadff47d74f8e3e44a2bbab80bd7
---
_sources/library-user-guide/upgrading.md.txt | 17 +++++++++++++++++
library-user-guide/upgrading.html | 14 ++++++++++++++
searchindex.js | 2 +-
3 files changed, 32 insertions(+), 1 deletion(-)
diff --git a/_sources/library-user-guide/upgrading.md.txt b/_sources/library-user-guide/upgrading.md.txt
index 4f2b7345a0..00e55ac9e7 100644
--- a/_sources/library-user-guide/upgrading.md.txt
+++ b/_sources/library-user-guide/upgrading.md.txt
@@ -40,6 +40,23 @@ directly on the `Field`. For example:
+ let field = Arc::clone(df_schema.field("my_column"));
```
+### ListingTableProvider now caches `LIST` commands
+
+In prior versions, `ListingTableProvider` would issue `LIST` commands to
+the underlying object store each time it needed to list files for a query.
+To improve performance, `ListingTableProvider` now caches the results of
+`LIST` commands for the lifetime of the `ListingTableProvider` instance.
+
+Note that by default the cache has no expiration time, so if files are added or removed
+from the underlying object store, the `ListingTableProvider` will not see
+those changes until the `ListingTableProvider` instance is dropped and recreated.
+
+You will be able to configure the maximum cache size and cache expiration time via a configuration option:
+
+See <https://github.com/apache/datafusion/issues/19056> for more details.
+
+Note that the internal API has changed to use a trait `ListFilesCache` instead of a type alias.
+
### Removal of `pyarrow` feature
The `pyarrow` feature flag has been removed. This feature has been migrated to
diff --git a/library-user-guide/upgrading.html b/library-user-guide/upgrading.html
index 7f3682050d..e52e7cfcf3 100644
--- a/library-user-guide/upgrading.html
+++ b/library-user-guide/upgrading.html
@@ -420,6 +420,19 @@ directly on the <code class="docutils literal notranslate"><span class="pre">Fie
</pre></div>
</div>
</section>
+<section id="listingtableprovider-now-caches-list-commands">
+<h3>ListingTableProvider now caches <code class="docutils literal notranslate"><span class="pre">LIST</span></code> commands<a class="headerlink" href="#listingtableprovider-now-caches-list-commands" title="Link to this heading">#</a></h3>
+<p>In prior versions, <code class="docutils literal notranslate"><span class="pre">ListingTableProvider</span></code> would issue <code class="docutils literal notranslate"><span class="pre">LIST</span></code> commands to
+the underlying object store each time it needed to list files for a query.
+To improve performance, <code class="docutils literal notranslate"><span class="pre">ListingTableProvider</span></code> now caches the results of
+<code class="docutils literal notranslate"><span class="pre">LIST</span></code> commands for the lifetime of the <code class="docutils literal notranslate"><span class="pre">ListingTableProvider</span></code> instance.</p>
+<p>Note that by default the cache has no expiration time, so if files are added or removed
+from the underlying object store, the <code class="docutils literal notranslate"><span class="pre">ListingTableProvider</span></code> will not see
+those changes until the <code class="docutils literal notranslate"><span class="pre">ListingTableProvider</span></code> instance is dropped and recreated.</p>
+<p>You will be able to configure the maximum cache size and cache expiration time via a configuration option:</p>
+<p>See <a class="github reference external" href="https://github.com/apache/datafusion/issues/19056">apache/datafusion#19056</a> for more details.</p>
+<p>Note that the internal API has changed to use a trait <code class="docutils literal notranslate"><span class="pre">ListFilesCache</span></code> instead of a type alias.</p>
+</section>
<section id="removal-of-pyarrow-feature">
<h3>Removal of <code class="docutils literal notranslate"><span class="pre">pyarrow</span></code> feature<a class="headerlink" href="#removal-of-pyarrow-feature" title="Link to this heading">#</a></h3>
<p>The <code class="docutils literal notranslate"><span class="pre">pyarrow</span></code> feature flag has been removed. This feature has been migrated to
@@ -2009,6 +2022,7 @@ take care of constructing the <code class="docutils literal notranslate"><span c
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion-52-0-0">DataFusion <code class="docutils literal notranslate"><span class="pre">52.0.0</span></code></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#changes-to-dfschema-api">Changes to DFSchema API</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#listingtableprovider-now-caches-list-commands">ListingTableProvider now caches <code class="docutils literal notranslate"><span class="pre">LIST</span></code> commands</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#removal-of-pyarrow-feature">Removal of <code class="docutils literal notranslate"><span class="pre">pyarrow</span></code> feature</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#adaptive-filter-representation-in-parquet-filter-pushdown">Adaptive filter representation in Parquet filter pushdown</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#statistics-handling-moved-from-filesource-to-filescanconfig">Statistics handling moved from <code class="docutils literal notranslate"><span class="pre">FileSource</span></code> to <code class="docutils literal notranslate"><span class="pre">FileScanConfig</span></code></a></li>
diff --git a/searchindex.js b/searchindex.js
index 6c70e673e1..b8ae5750b8 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"!=":[[60,"op-neq"]],"!~":[[60,"op-re-not-match"]],"!~*":[[60,"op-re-not-match-i"]],"!~~":[[60,"id19"]],"!~~*":[[60,"id20"]],"#":[[60,"op-bit-xor"]],"%":[[60,"op-modulo"]],"&":[[60,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[13,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[60,"op-multiply"]],"+":[[60,"op-plus"]],"-":[[60,"op-minus"]],"/":[[60,"op-divide"]],"<":[[60,"op-lt"]],"< [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"!=":[[60,"op-neq"]],"!~":[[60,"op-re-not-match"]],"!~*":[[60,"op-re-not-match-i"]],"!~~":[[60,"id19"]],"!~~*":[[60,"id20"]],"#":[[60,"op-bit-xor"]],"%":[[60,"op-modulo"]],"&":[[60,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[13,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[60,"op-multiply"]],"+":[[60,"op-plus"]],"-":[[60,"op-minus"]],"/":[[60,"op-divide"]],"<":[[60,"op-lt"]],"< [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org