You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/06/21 11:41:43 UTC
(datafusion) branch asf-site updated: Publish built docs triggered by 7cecc1827936428b2f620d1d9b27cdd26f020605
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 61afa41453 Publish built docs triggered by 7cecc1827936428b2f620d1d9b27cdd26f020605
61afa41453 is described below
commit 61afa41453108bee2a1f8a3f1a06e9deb1b452ef
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sat Jun 21 11:41:38 2025 +0000
Publish built docs triggered by 7cecc1827936428b2f620d1d9b27cdd26f020605
---
_sources/library-user-guide/upgrading.md.txt | 29 +++++++++++++++++++++++
library-user-guide/upgrading.html | 35 ++++++++++++++++++++++++++++
searchindex.js | 2 +-
3 files changed, 65 insertions(+), 1 deletion(-)
diff --git a/_sources/library-user-guide/upgrading.md.txt b/_sources/library-user-guide/upgrading.md.txt
index 613c2be43d..182e728c5b 100644
--- a/_sources/library-user-guide/upgrading.md.txt
+++ b/_sources/library-user-guide/upgrading.md.txt
@@ -59,6 +59,35 @@ See [#16317] for details.
[#16317]: https://github.com/apache/datafusion/pull/16317
+### New `datafusion.execution.spill_compression` configuration option
+
+DataFusion 49.0.0 adds support for compressing spill files when data is written to disk during spilling query execution. A new configuration option `datafusion.execution.spill_compression` controls the compression codec used.
+
+**Configuration:**
+
+- **Key**: `datafusion.execution.spill_compression`
+- **Default**: `uncompressed`
+- **Valid values**: `uncompressed`, `lz4_frame`, `zstd`
+
+**Usage:**
+
+```rust
+use datafusion::prelude::*;
+use datafusion_common::config::SpillCompression;
+
+let config = SessionConfig::default()
+ .with_spill_compression(SpillCompression::Zstd);
+let ctx = SessionContext::new_with_config(config);
+```
+
+Or via SQL:
+
+```sql
+SET datafusion.execution.spill_compression = 'zstd';
+```
+
+For more details about this configuration option, including performance trade-offs between different compression codecs, see the [Configuration Settings](../user-guide/configs.md) documentation.
+
## DataFusion `48.0.0`
### `Expr::Literal` has optional metadata
diff --git a/library-user-guide/upgrading.html b/library-user-guide/upgrading.html
index baa1e14504..100e3017b5 100644
--- a/library-user-guide/upgrading.html
+++ b/library-user-guide/upgrading.html
@@ -584,6 +584,17 @@
</code>
</a>
</li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#new-datafusion-execution-spill-compression-configuration-option">
+ New
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ datafusion.execution.spill_compression
+ </span>
+ </code>
+ configuration option
+ </a>
+ </li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -994,6 +1005,30 @@ is more efficient.</p>
</div>
<p>See <a class="reference external" href="https://github.com/apache/datafusion/pull/16317">#16317</a> for details.</p>
</section>
+<section id="new-datafusion-execution-spill-compression-configuration-option">
+<h3>New <code class="docutils literal notranslate"><span class="pre">datafusion.execution.spill_compression</span></code> configuration option<a class="headerlink" href="#new-datafusion-execution-spill-compression-configuration-option" title="Link to this heading">¶</a></h3>
+<p>DataFusion 49.0.0 adds support for compressing spill files when data is written to disk during spilling query execution. A new configuration option <code class="docutils literal notranslate"><span class="pre">datafusion.execution.spill_compression</span></code> controls the compression codec used.</p>
+<p><strong>Configuration:</strong></p>
+<ul class="simple">
+<li><p><strong>Key</strong>: <code class="docutils literal notranslate"><span class="pre">datafusion.execution.spill_compression</span></code></p></li>
+<li><p><strong>Default</strong>: <code class="docutils literal notranslate"><span class="pre">uncompressed</span></code></p></li>
+<li><p><strong>Valid values</strong>: <code class="docutils literal notranslate"><span class="pre">uncompressed</span></code>, <code class="docutils literal notranslate"><span class="pre">lz4_frame</span></code>, <code class="docutils literal notranslate"><span class="pre">zstd</span></code></p></li>
+</ul>
+<p><strong>Usage:</strong></p>
+<div class="highlight-rust notranslate"><div class="highlight"><pre><span></span><span class="k">use</span><span class="w"> </span><span class="n">datafusion</span><span class="p">::</span><span class="n">prelude</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
+<span class="k">use</span><span class="w"> </span><span class="n">datafusion_common</span><span class="p">::</span><span class="n">config</span><span class="p">::</span><span class="n">SpillCompression</span><span class="p">;</span>
+
+<span class="kd">let</span><span class="w"> </span><span class="n">config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SessionConfig</span><span class="p">::</span><span class="n">default</span><span class="p">()</span>
+<span class="w"> </span><span class="p">.</span><span class="n">with_spill_compression</span><span class="p">(</span><span class="n">SpillCompression</span><span class="p">::</span><span class="n">Zstd</span><span class="p">);</span>
+<span class="kd">let</span><span class="w"> </span><span class="n">ctx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SessionContext</span><span class="p">::</span><span class="n">new_with_config</span><span class="p">(</span><span class="n">config</span><span class="p">);</span>
+</pre></div>
+</div>
+<p>Or via SQL:</p>
+<div class="highlight-sql notranslate"><div class="highlight"><pre><span></span><span class="k">SET</span><span class="w"> </span><span class="n">datafusion</span><span class="p">.</span><span class="n">execution</span><span class="p">.</span><span class="n">spill_compression</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'zstd'</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>For more details about this configuration option, including performance trade-offs between different compression codecs, see the <a class="reference internal" href="../user-guide/configs.html"><span class="std std-doc">Configuration Settings</span></a> documentation.</p>
+</section>
</section>
<section id="datafusion-48-0-0">
<h2>DataFusion <code class="docutils literal notranslate"><span class="pre">48.0.0</span></code><a class="headerlink" href="#datafusion-48-0-0" title="Link to this heading">¶</a></h2>
diff --git a/searchindex.js b/searchindex.js
index 9d0564efad..14f339befc 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"!=":[[57,"op-neq"]],"!~":[[57,"op-re-not-match"]],"!~*":[[57,"op-re-not-match-i"]],"!~~":[[57,"id19"]],"!~~*":[[57,"id20"]],"#":[[57,"op-bit-xor"]],"%":[[57,"op-modulo"]],"&":[[57,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[12,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[57,"op-multiply"]],"+":[[57,"op-plus"]],"-":[[57,"op-minus"]],"/":[[57,"op-divide"]],"<":[[57,"op-lt"]],"< [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"!=":[[57,"op-neq"]],"!~":[[57,"op-re-not-match"]],"!~*":[[57,"op-re-not-match-i"]],"!~~":[[57,"id19"]],"!~~*":[[57,"id20"]],"#":[[57,"op-bit-xor"]],"%":[[57,"op-modulo"]],"&":[[57,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[12,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[57,"op-multiply"]],"+":[[57,"op-plus"]],"-":[[57,"op-minus"]],"/":[[57,"op-divide"]],"<":[[57,"op-lt"]],"< [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org