You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/03/20 22:06:35 UTC
(datafusion-site) branch asf-staging updated: Commit build products
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new 8fdbb27 Commit build products
8fdbb27 is described below
commit 8fdbb27881dacf01bf4a568b11b6d148832fba4e
Author: Build Pelican (action) <pr...@infra.apache.org>
AuthorDate: Fri Mar 20 22:06:31 2026 +0000
Commit build products
---
blog/2019/02/04/datafusion-donation/index.html | 8 +-
blog/2021/04/12/ballista-donation/index.html | 8 +-
blog/2021/08/18/ballista-0.5.0/index.html | 8 +-
blog/2021/08/18/datafusion-5.0.0/index.html | 8 +-
.../11/19/2021-11-8-datafusion-6.0.0.md/index.html | 8 +-
blog/2022/02/28/datafusion-7.0.0/index.html | 10 +-
blog/2022/03/21/datafusion-contrib/index.html | 8 +-
blog/2022/05/16/datafusion-8.0.0/index.html | 8 +-
blog/2022/10/25/datafusion-13.0.0/index.html | 8 +-
blog/2022/10/28/ballista-0.9.0/index.html | 8 +-
blog/2023/01/19/datafusion-16.0.0/index.html | 10 +-
blog/2023/06/24/datafusion-25.0.0/index.html | 8 +-
.../2023/08/05/datafusion_fast_grouping/index.html | 8 +-
blog/2024/01/19/datafusion-34.0.0/index.html | 10 +-
blog/2024/03/06/comet-donation/index.html | 8 +-
blog/2024/05/07/datafusion-tlp/index.html | 8 +-
blog/2024/07/20/datafusion-comet-0.1.0/index.html | 8 +-
blog/2024/07/24/datafusion-40.0.0/index.html | 8 +-
.../2024/08/20/python-datafusion-40.0.0/index.html | 10 +-
blog/2024/08/28/datafusion-comet-0.2.0/index.html | 8 +-
.../index.html | 8 +-
.../index.html | 12 +-
blog/2024/09/27/datafusion-comet-0.3.0/index.html | 8 +-
.../index.html | 8 +-
.../datafusion-python-udf-comparisons/index.html | 16 +-
blog/2024/11/20/datafusion-comet-0.4.0/index.html | 8 +-
.../2024/12/14/datafusion-python-43.1.0/index.html | 12 +-
blog/2025/01/17/datafusion-comet-0.5.0/index.html | 8 +-
.../02/02/datafusion-ballista-43.0.0/index.html | 8 +-
blog/2025/02/17/datafusion-comet-0.6.0/index.html | 8 +-
blog/2025/02/20/datafusion-45.0.0/index.html | 8 +-
blog/2025/03/11/ordering-analysis/index.html | 8 +-
blog/2025/03/20/datafusion-comet-0.7.0/index.html | 8 +-
blog/2025/03/20/parquet-pruning/index.html | 8 +-
blog/2025/03/21/parquet-pushdown/index.html | 8 +-
blog/2025/03/24/datafusion-46.0.0/index.html | 8 +-
.../2025/03/30/datafusion-python-46.0.0/index.html | 10 +-
blog/2025/04/10/fastest-tpch-generator/index.html | 8 +-
.../04/19/user-defined-window-functions/index.html | 8 +-
blog/2025/05/06/datafusion-comet-0.8.0/index.html | 8 +-
.../optimizing-sql-dataframes-part-one/index.html | 8 +-
.../optimizing-sql-dataframes-part-two/index.html | 8 +-
blog/2025/06/30/cancellation/index.html | 8 +-
blog/2025/07/01/datafusion-comet-0.9.0/index.html | 8 +-
blog/2025/07/11/datafusion-47.0.0/index.html | 8 +-
.../07/14/user-defined-parquet-indexes/index.html | 8 +-
blog/2025/07/16/datafusion-48.0.0/index.html | 8 +-
blog/2025/07/28/datafusion-49.0.0/index.html | 8 +-
.../2025/08/15/external-parquet-indexes/index.html | 8 +-
blog/2025/09/10/dynamic-filters/index.html | 8 +-
blog/2025/09/16/datafusion-comet-0.10.0/index.html | 8 +-
.../09/21/custom-types-using-metadata/index.html | 8 +-
blog/2025/09/29/datafusion-50.0.0/index.html | 8 +-
blog/2025/10/21/datafusion-comet-0.11.0/index.html | 8 +-
blog/2025/11/25/datafusion-51.0.0/index.html | 8 +-
blog/2025/12/04/datafusion-comet-0.12.0/index.html | 8 +-
.../15/avoid-consecutive-repartitions/index.html | 8 +-
.../01/{08 => 12}/datafusion-52.0.0/index.html | 10 +-
blog/2026/01/12/extending-sql/index.html | 8 +-
blog/2026/01/30/datafusion-comet-0.13.0/index.html | 228 +++
blog/2026/02/02/datafusion_case/index.html | 503 +++++
blog/2026/03/20/limit-pruning/index.html | 413 +++++
blog/2026/03/20/writing-table-providers/index.html | 743 ++++++++
blog/about.html | 96 +-
blog/author/pepijn-van-eeckhoudt.html | 50 +
blog/author/pmc.html | 36 +-
.../{pepijn-van-eeckhoudt.html => tim-saucer.html} | 36 +-
.../{pepijn-van-eeckhoudt.html => xudong.html} | 14 +-
blog/category/blog.html | 178 +-
blog/css/bootstrap.min.css | 6 +-
blog/css/dark-mode.css | 100 +
blog/feed.xml | 132 +-
blog/feeds/all-en.atom.xml | 1923 +++++++++++++++++---
blog/feeds/blog.atom.xml | 1923 +++++++++++++++++---
blog/feeds/pepijn-van-eeckhoudt.atom.xml | 397 +++-
blog/feeds/pepijn-van-eeckhoudt.rss.xml | 42 +-
blog/feeds/pmc.atom.xml | 124 +-
blog/feeds/pmc.rss.xml | 26 +-
blog/feeds/tim-saucer.atom.xml | 597 ++++++
blog/feeds/tim-saucer.rss.xml | 24 +
blog/feeds/timsaucer.atom.xml | 16 +-
blog/feeds/xiangpeng-hao-andrew-lamb.atom.xml | 4 +-
blog/feeds/xudong.atom.xml | 313 ++++
...pepijn-van-eeckhoudt.rss.xml => xudong.rss.xml} | 6 +-
blog/images/case/evaluate_selection.svg | 4 +
blog/images/case/merge.svg | 4 +
blog/images/case/merge_n.svg | 4 +
blog/images/case/merging.svg | 4 +
blog/images/case/no_projection.svg | 4 +
blog/images/case/original_loop.svg | 4 +
blog/images/case/projection.svg | 4 +
blog/images/case/results.png | Bin 0 -> 111593 bytes
blog/images/limit-pruning/before-after.svg | 74 +
.../limit-pruning/fully-matched-detection.svg | 54 +
.../limit-pruning/limit-rewrite-algorithm.svg | 71 +
blog/images/limit-pruning/pruning-phases.svg | 77 +
blog/images/limit-pruning/pruning-pipeline.svg | 52 +
blog/images/limit-pruning/row-group-states.svg | 75 +
blog/images/limit-pruning/wasted-io.svg | 101 +
blog/index.html | 224 ++-
blog/js/bootstrap.bundle.min.js | 8 +-
blog/js/bootstrap.bundle.min.js.map | 2 +-
blog/js/dark-mode.js | 46 +
103 files changed, 8546 insertions(+), 696 deletions(-)
diff --git a/blog/2019/02/04/datafusion-donation/index.html b/blog/2019/02/04/datafusion-donation/index.html
index ec84ab2..92825ba 100644
--- a/blog/2019/02/04/datafusion-donation/index.html
+++ b/blog/2019/02/04/datafusion-donation/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2021/04/12/ballista-donation/index.html b/blog/2021/04/12/ballista-donation/index.html
index f833fbe..ea1e055 100644
--- a/blog/2021/04/12/ballista-donation/index.html
+++ b/blog/2021/04/12/ballista-donation/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2021/08/18/ballista-0.5.0/index.html b/blog/2021/08/18/ballista-0.5.0/index.html
index 211bb76..5b118b0 100644
--- a/blog/2021/08/18/ballista-0.5.0/index.html
+++ b/blog/2021/08/18/ballista-0.5.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2021/08/18/datafusion-5.0.0/index.html b/blog/2021/08/18/datafusion-5.0.0/index.html
index 77a5cfc..10cce8b 100644
--- a/blog/2021/08/18/datafusion-5.0.0/index.html
+++ b/blog/2021/08/18/datafusion-5.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html b/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html
index 77ea2fe..c8ba04b 100644
--- a/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html
+++ b/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2022/02/28/datafusion-7.0.0/index.html b/blog/2022/02/28/datafusion-7.0.0/index.html
index 0e86152..278bcb7 100644
--- a/blog/2022/02/28/datafusion-7.0.0/index.html
+++ b/blog/2022/02/28/datafusion-7.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
@@ -125,7 +131,7 @@ git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc
<li>Switch from <code>std::sync::Mutex</code> to <code>parking_lot::Mutex</code> <a href="https://github.com/apache/arrow-datafusion/pull/1720">#1720</a></li>
<li>New Features</li>
<li>Support for memory tracking and spilling to disk<ul>
-<li>MemoryMananger and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
+<li>MemoryManager and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
<li>Out of core sort <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
<li>New metrics</li>
<li><code>Gauge</code> and <code>CurrentMemoryUsage</code> <a href="https://github.com/apache/arrow-datafusion/pull/1682">#1682</a></li>
diff --git a/blog/2022/03/21/datafusion-contrib/index.html b/blog/2022/03/21/datafusion-contrib/index.html
index 1bc8e00..407b018 100644
--- a/blog/2022/03/21/datafusion-contrib/index.html
+++ b/blog/2022/03/21/datafusion-contrib/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2022/05/16/datafusion-8.0.0/index.html b/blog/2022/05/16/datafusion-8.0.0/index.html
index eff4015..0d47fcf 100644
--- a/blog/2022/05/16/datafusion-8.0.0/index.html
+++ b/blog/2022/05/16/datafusion-8.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2022/10/25/datafusion-13.0.0/index.html b/blog/2022/10/25/datafusion-13.0.0/index.html
index 6deba34..22c72b1 100644
--- a/blog/2022/10/25/datafusion-13.0.0/index.html
+++ b/blog/2022/10/25/datafusion-13.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2022/10/28/ballista-0.9.0/index.html b/blog/2022/10/28/ballista-0.9.0/index.html
index ced73b8..bdd9489 100644
--- a/blog/2022/10/28/ballista-0.9.0/index.html
+++ b/blog/2022/10/28/ballista-0.9.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2023/01/19/datafusion-16.0.0/index.html b/blog/2023/01/19/datafusion-16.0.0/index.html
index ca6c896..11e2f81 100644
--- a/blog/2023/01/19/datafusion-16.0.0/index.html
+++ b/blog/2023/01/19/datafusion-16.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
@@ -192,7 +198,7 @@ required synchronous access to all relevant catalog information.</p>
<li>Automatic coercions ast between Date and Timestamp <a href="https://github.com/apache/arrow-datafusion/issues/4726">#4726</a></li>
<li>Support type coercion for timestamp and utf8 <a href="https://github.com/apache/arrow-datafusion/issues/4312">#4312</a></li>
<li>Full support for time32 and time64 literal values (<code>ScalarValue</code>) <a href="https://github.com/apache/arrow-datafusion/issues/4156">#4156</a></li>
-<li>New functions, incuding <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
+<li>New functions, including <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
<li>Compressed CSV/JSON support <a href="https://github.com/apache/arrow-datafusion/issues/3642">#3642</a></li>
</ul>
<p>The community has also invested in new <a href="https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/tests/sqllogictests/README.md">sqllogic based</a> tests to keep improving DataFusion's quality with less effort.</p>
diff --git a/blog/2023/06/24/datafusion-25.0.0/index.html b/blog/2023/06/24/datafusion-25.0.0/index.html
index 4381b26..00a8759 100644
--- a/blog/2023/06/24/datafusion-25.0.0/index.html
+++ b/blog/2023/06/24/datafusion-25.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2023/08/05/datafusion_fast_grouping/index.html b/blog/2023/08/05/datafusion_fast_grouping/index.html
index bda185e..64c2565 100644
--- a/blog/2023/08/05/datafusion_fast_grouping/index.html
+++ b/blog/2023/08/05/datafusion_fast_grouping/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/01/19/datafusion-34.0.0/index.html b/blog/2024/01/19/datafusion-34.0.0/index.html
index 7f8ca1c..1da4e58 100644
--- a/blog/2024/01/19/datafusion-34.0.0/index.html
+++ b/blog/2024/01/19/datafusion-34.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
@@ -256,7 +262,7 @@ LIMIT 3;
3 rows in set. Query took 0.053 seconds.
</code></pre>
<h3 id="growth-of-datafusion">Growth of DataFusion 📈<a class="headerlink" href="#growth-of-datafusion" title="Permanent link">¶</a></h3>
-<p>DataFusion has been appearing more publically in the wild. For example
+<p>DataFusion has been appearing more publicly in the wild. For example
* New projects built using DataFusion such as <a href="https://lancedb.com/">lancedb</a>, <a href="https://glaredb.com/">GlareDB</a>, <a href="https://www.arroyo.dev/">Arroyo</a>, and <a href="https://github.com/cmu-db/optd">optd</a>.
* Public talks such as <a href="https://www.youtube.com/watch?v=AJU9rdRNk9I">Apache Arrow Datafusion: Vectorized
Execution Framework For Maximum Performance</a> in <a href="https://www.bagevent.com/event/8432178">CommunityOverCode Asia 2023</a>
diff --git a/blog/2024/03/06/comet-donation/index.html b/blog/2024/03/06/comet-donation/index.html
index f2daf6e..9fed027 100644
--- a/blog/2024/03/06/comet-donation/index.html
+++ b/blog/2024/03/06/comet-donation/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/05/07/datafusion-tlp/index.html b/blog/2024/05/07/datafusion-tlp/index.html
index b84c2aa..708b9bd 100644
--- a/blog/2024/05/07/datafusion-tlp/index.html
+++ b/blog/2024/05/07/datafusion-tlp/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/07/20/datafusion-comet-0.1.0/index.html b/blog/2024/07/20/datafusion-comet-0.1.0/index.html
index c1fce1b..9157495 100644
--- a/blog/2024/07/20/datafusion-comet-0.1.0/index.html
+++ b/blog/2024/07/20/datafusion-comet-0.1.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/07/24/datafusion-40.0.0/index.html b/blog/2024/07/24/datafusion-40.0.0/index.html
index 0ecb192..9e85394 100644
--- a/blog/2024/07/24/datafusion-40.0.0/index.html
+++ b/blog/2024/07/24/datafusion-40.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/08/20/python-datafusion-40.0.0/index.html b/blog/2024/08/20/python-datafusion-40.0.0/index.html
index 8bc65cc..f0da687 100644
--- a/blog/2024/08/20/python-datafusion-40.0.0/index.html
+++ b/blog/2024/08/20/python-datafusion-40.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
@@ -105,7 +111,7 @@ to their Rust counterparts.</li>
<p>The most significant difference is that we have added wrapper functions and classes for most of the
user facing interface. These wrappers, written in Python, contain both documentation and type
annotations.</p>
-<p>This documenation is now available on the <a href="https://datafusion.apache.org/python/autoapi/datafusion/index.html">DataFusion in Python API</a> website. There you can browse
+<p>This documentation is now available on the <a href="https://datafusion.apache.org/python/autoapi/datafusion/index.html">DataFusion in Python API</a> website. There you can browse
the available functions and classes to see the breadth of available functionality.</p>
<p>Modern IDEs use language servers such as
<a href="https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance">Pylance</a> or
diff --git a/blog/2024/08/28/datafusion-comet-0.2.0/index.html b/blog/2024/08/28/datafusion-comet-0.2.0/index.html
index 91be95a..79fdfef 100644
--- a/blog/2024/08/28/datafusion-comet-0.2.0/index.html
+++ b/blog/2024/08/28/datafusion-comet-0.2.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/09/13/string-view-german-style-strings-part-1/index.html b/blog/2024/09/13/string-view-german-style-strings-part-1/index.html
index cb15171..e12854a 100644
--- a/blog/2024/09/13/string-view-german-style-strings-part-1/index.html
+++ b/blog/2024/09/13/string-view-german-style-strings-part-1/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/09/13/string-view-german-style-strings-part-2/index.html b/blog/2024/09/13/string-view-german-style-strings-part-2/index.html
index eea23f7..8e844df 100644
--- a/blog/2024/09/13/string-view-german-style-strings-part-2/index.html
+++ b/blog/2024/09/13/string-view-german-style-strings-part-2/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
@@ -107,8 +113,8 @@ Figure 1 illustrates the difference between the output of both string representa
<h1 id="when-to-gc">When to GC?<a class="headerlink" href="#when-to-gc" title="Permanent link">¶</a></h1>
<p>Zero-copy <code>take/filter</code> is great for generating large arrays quickly, but it is suboptimal for highly selective filters, where most of the strings are filtered out. When the cardinality drops, StringViewArray buffers become sparse—only a small subset of the bytes in the buffer’s memory are referred to by any <code>view</code>. This leads to excessive memory usage, especially in a <a href="https://github.com/apache/datafusion/issues/11628">filter-then-coalesce scenario</a>. [...]
<p>To release unused memory, we implemented a <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html#method.gc">garbage collection (GC)</a> routine to consolidate the data into a new buffer to release the old sparse buffer(s). As the GC operation copies strings, similarly to StringArray, we must be careful about when to call it. If we call GC too early, we cause unnecessary copying, losing much of the benefit of StringViewArray. If we call GC too late, we hold [...]
-<p><code>arrow-rs</code> implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/coalesce_batches/struct.CoalesceBatchesExec.html"><code>CoalseceBatchesExec</code></a> operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, which aligns perfectly with the scenario of G [...]
-We, therefore,<a href="https://github.com/apache/datafusion/pull/11587"> implemented the GC procedure</a> inside <code>CoalseceBatchesExec</code>[^5] with a heuristic that estimates when the buffers are too sparse.</p>
+<p><code>arrow-rs</code> implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/coalesce_batches/struct.CoalesceBatchesExec.html"><code>CoalesceBatchesExec</code></a> operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, which aligns perfectly with the scenario of G [...]
+We, therefore,<a href="https://github.com/apache/datafusion/pull/11587"> implemented the GC procedure</a> inside <code>CoalesceBatchesExec</code>[^5] with a heuristic that estimates when the buffers are too sparse.</p>
<h2 id="the-art-of-function-inlining-not-too-much-not-too-little">The art of function inlining: not too much, not too little<a class="headerlink" href="#the-art-of-function-inlining-not-too-much-not-too-little" title="Permanent link">¶</a></h2>
<p>Like string inlining, <em>function</em> inlining is the process of embedding a short function into the caller to avoid the overhead of function calls (caller/callee save).
Usually, the Rust compiler does a good job of deciding when to inline. However, it is possible to override its default using the <a href="https://doc.rust-lang.org/reference/attributes/codegen.html#the-inline-attribute"><code>#[inline(always)]</code> directive</a>.
diff --git a/blog/2024/09/27/datafusion-comet-0.3.0/index.html b/blog/2024/09/27/datafusion-comet-0.3.0/index.html
index bac52fc..5f4be49 100644
--- a/blog/2024/09/27/datafusion-comet-0.3.0/index.html
+++ b/blog/2024/09/27/datafusion-comet-0.3.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html b/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html
index 7a72182..31bb498 100644
--- a/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html
+++ b/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/11/19/datafusion-python-udf-comparisons/index.html b/blog/2024/11/19/datafusion-python-udf-comparisons/index.html
index 8b0330e..ee61d3a 100644
--- a/blog/2024/11/19/datafusion-python-udf-comparisons/index.html
+++ b/blog/2024/11/19/datafusion-python-udf-comparisons/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
@@ -149,7 +155,7 @@ than a join can be significantly faster. This is worth profiling for your specif
<p>I have a DataFrame with many values that I want to aggregate. I have already analyzed it and
determined there is a noise level below which I do not want to include in my analysis. I want to
compute a sum of only values that are above my noise threshold.</p>
-<p>This can be done fairly easy without leaning on a User Defined Aggegate Function (UDAF). You can
+<p>This can be done fairly easy without leaning on a User Defined Aggregate Function (UDAF). You can
simply filter the DataFrame and then aggregate using the built-in <code>sum</code> function. Here, we
demonstrate doing this as a UDF primarily as an example of how to write UDAFs. We will use the
PyArrow compute approach.</p>
@@ -310,7 +316,7 @@ Python, is to primarily demonstrate how to make the Python to Rust with Python w
transition. In the second implementation you can see how we can iterate through all of the arrays
ourselves.</p>
<p>In this first example, we are hard coding the values of interest, but in the following section
-we demonstrate passing these in during initalization.</p>
+we demonstrate passing these in during initialization.</p>
<pre><code class="language-rust">#[pyfunction]
pub fn tuple_filter_fn(
py: Python<'_>,
@@ -533,13 +539,13 @@ how much they have ordered total. We want to ignore small orders, which we defin
import pyarrow as pa
import pyarrow.compute as pc
-IGNORE_THESHOLD = 5000.0
+IGNORE_THRESHOLD = 5000.0
class AboveThresholdAccum(Accumulator):
def __init__(self) -> None:
self._sum = 0.0
def update(self, values: pa.Array) -> None:
- over_threshold = pc.greater(values, pa.scalar(IGNORE_THESHOLD))
+ over_threshold = pc.greater(values, pa.scalar(IGNORE_THRESHOLD))
sum_above = pc.sum(values.filter(over_threshold)).as_py()
if sum_above is None:
sum_above = 0.0
diff --git a/blog/2024/11/20/datafusion-comet-0.4.0/index.html b/blog/2024/11/20/datafusion-comet-0.4.0/index.html
index 0e35c4d..6a76617 100644
--- a/blog/2024/11/20/datafusion-comet-0.4.0/index.html
+++ b/blog/2024/11/20/datafusion-comet-0.4.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2024/12/14/datafusion-python-43.1.0/index.html b/blog/2024/12/14/datafusion-python-43.1.0/index.html
index 4ce52c0..7e39e5f 100644
--- a/blog/2024/12/14/datafusion-python-43.1.0/index.html
+++ b/blog/2024/12/14/datafusion-python-43.1.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
@@ -95,7 +101,7 @@ consistent method for exposing these data structures across libraries.</p>
<p>In <a href="https://github.com/apache/datafusion-python/pull/825">PR #825</a>, we introduced support for both importing and exporting Arrow data in
<code>datafusion-python</code>. With this improvement, you can now use a single function call to import
a table from <strong>any</strong> Python library that implements the <a href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html">Arrow PyCapsule Interface</a>.
-Many popular libaries, such as <a href="https://pandas.pydata.org/">Pandas</a> and <a href="https://pola.rs/">Polars</a>
+Many popular libraries, such as <a href="https://pandas.pydata.org/">Pandas</a> and <a href="https://pola.rs/">Polars</a>
already support these interfaces.</p>
<p>Suppose you have a Pandas and Polars DataFrames named <code>df_pandas</code> or <code>df_polars</code>, respectively:</p>
<pre><code class="language-python">ctx = SessionContext()
@@ -155,7 +161,7 @@ of the blog post describing how these enhancements can lead to 20-200% performan
gains in some tests.</p>
<p>During our testing we identified some cases where we needed to adjust workflows to
account for the fact that StringView is now the default type for string based operations.
-First, when performing manipulations on string objects there is a perfomance loss when
+First, when performing manipulations on string objects there is a performance loss when
needing to cast from string to string view or vice versa. To reap the best performance,
ideally all of your string type data will use StringView. For most users this should be
transparent. However if you specify a schema for reading or creating data, then you
diff --git a/blog/2025/01/17/datafusion-comet-0.5.0/index.html b/blog/2025/01/17/datafusion-comet-0.5.0/index.html
index 7efd3fe..bbb8a45 100644
--- a/blog/2025/01/17/datafusion-comet-0.5.0/index.html
+++ b/blog/2025/01/17/datafusion-comet-0.5.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/02/02/datafusion-ballista-43.0.0/index.html b/blog/2025/02/02/datafusion-ballista-43.0.0/index.html
index edacf22..598fa0a 100644
--- a/blog/2025/02/02/datafusion-ballista-43.0.0/index.html
+++ b/blog/2025/02/02/datafusion-ballista-43.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/02/17/datafusion-comet-0.6.0/index.html b/blog/2025/02/17/datafusion-comet-0.6.0/index.html
index dd05ae4..61ea0cc 100644
--- a/blog/2025/02/17/datafusion-comet-0.6.0/index.html
+++ b/blog/2025/02/17/datafusion-comet-0.6.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/02/20/datafusion-45.0.0/index.html b/blog/2025/02/20/datafusion-45.0.0/index.html
index bb25782..1a1b3af 100644
--- a/blog/2025/02/20/datafusion-45.0.0/index.html
+++ b/blog/2025/02/20/datafusion-45.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/03/11/ordering-analysis/index.html b/blog/2025/03/11/ordering-analysis/index.html
index 228f13b..bbc09a0 100644
--- a/blog/2025/03/11/ordering-analysis/index.html
+++ b/blog/2025/03/11/ordering-analysis/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/03/20/datafusion-comet-0.7.0/index.html b/blog/2025/03/20/datafusion-comet-0.7.0/index.html
index 03fe624..3d89a28 100644
--- a/blog/2025/03/20/datafusion-comet-0.7.0/index.html
+++ b/blog/2025/03/20/datafusion-comet-0.7.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/03/20/parquet-pruning/index.html b/blog/2025/03/20/parquet-pruning/index.html
index 9ec0197..b1f1e5e 100644
--- a/blog/2025/03/20/parquet-pruning/index.html
+++ b/blog/2025/03/20/parquet-pruning/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/03/21/parquet-pushdown/index.html b/blog/2025/03/21/parquet-pushdown/index.html
index feab768..d06845f 100644
--- a/blog/2025/03/21/parquet-pushdown/index.html
+++ b/blog/2025/03/21/parquet-pushdown/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/03/24/datafusion-46.0.0/index.html b/blog/2025/03/24/datafusion-46.0.0/index.html
index 0d4a300..5258ad0 100644
--- a/blog/2025/03/24/datafusion-46.0.0/index.html
+++ b/blog/2025/03/24/datafusion-46.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/03/30/datafusion-python-46.0.0/index.html b/blog/2025/03/30/datafusion-python-46.0.0/index.html
index e369b04..e16f2cb 100644
--- a/blog/2025/03/30/datafusion-python-46.0.0/index.html
+++ b/blog/2025/03/30/datafusion-python-46.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
@@ -117,7 +123,7 @@ to register the view and then use it in another place:</p>
<pre><code class="language-python">ctx.register_view("view1", df1)
</code></pre>
<p>And then in another portion of your code which has access to the same session context
-you can retrive the DataFrame with:</p>
+you can retrieve the DataFrame with:</p>
<pre><code>df2 = ctx.table("view1")
</code></pre>
<h2 id="asynchronous-iteration-of-record-batches">Asynchronous Iteration of Record Batches<a class="headerlink" href="#asynchronous-iteration-of-record-batches" title="Permanent link">¶</a></h2>
diff --git a/blog/2025/04/10/fastest-tpch-generator/index.html b/blog/2025/04/10/fastest-tpch-generator/index.html
index e16a9c3..59c1944 100644
--- a/blog/2025/04/10/fastest-tpch-generator/index.html
+++ b/blog/2025/04/10/fastest-tpch-generator/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/04/19/user-defined-window-functions/index.html b/blog/2025/04/19/user-defined-window-functions/index.html
index b7db6f0..cab7d12 100644
--- a/blog/2025/04/19/user-defined-window-functions/index.html
+++ b/blog/2025/04/19/user-defined-window-functions/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/05/06/datafusion-comet-0.8.0/index.html b/blog/2025/05/06/datafusion-comet-0.8.0/index.html
index a4a041f..93f4cd5 100644
--- a/blog/2025/05/06/datafusion-comet-0.8.0/index.html
+++ b/blog/2025/05/06/datafusion-comet-0.8.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/06/15/optimizing-sql-dataframes-part-one/index.html b/blog/2025/06/15/optimizing-sql-dataframes-part-one/index.html
index 31b832f..745ccae 100644
--- a/blog/2025/06/15/optimizing-sql-dataframes-part-one/index.html
+++ b/blog/2025/06/15/optimizing-sql-dataframes-part-one/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/06/15/optimizing-sql-dataframes-part-two/index.html b/blog/2025/06/15/optimizing-sql-dataframes-part-two/index.html
index fa8e013..7a2cff4 100644
--- a/blog/2025/06/15/optimizing-sql-dataframes-part-two/index.html
+++ b/blog/2025/06/15/optimizing-sql-dataframes-part-two/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/06/30/cancellation/index.html b/blog/2025/06/30/cancellation/index.html
index ce1cf00..78faf74 100644
--- a/blog/2025/06/30/cancellation/index.html
+++ b/blog/2025/06/30/cancellation/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/07/01/datafusion-comet-0.9.0/index.html b/blog/2025/07/01/datafusion-comet-0.9.0/index.html
index e2eadc2..c2cf0da 100644
--- a/blog/2025/07/01/datafusion-comet-0.9.0/index.html
+++ b/blog/2025/07/01/datafusion-comet-0.9.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/07/11/datafusion-47.0.0/index.html b/blog/2025/07/11/datafusion-47.0.0/index.html
index f8b3779..c8d6cc1 100644
--- a/blog/2025/07/11/datafusion-47.0.0/index.html
+++ b/blog/2025/07/11/datafusion-47.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/07/14/user-defined-parquet-indexes/index.html b/blog/2025/07/14/user-defined-parquet-indexes/index.html
index 1fa7c44..ebbe48c 100644
--- a/blog/2025/07/14/user-defined-parquet-indexes/index.html
+++ b/blog/2025/07/14/user-defined-parquet-indexes/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/07/16/datafusion-48.0.0/index.html b/blog/2025/07/16/datafusion-48.0.0/index.html
index 9016444..e0d9d86 100644
--- a/blog/2025/07/16/datafusion-48.0.0/index.html
+++ b/blog/2025/07/16/datafusion-48.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/07/28/datafusion-49.0.0/index.html b/blog/2025/07/28/datafusion-49.0.0/index.html
index b3b8222..b57deb9 100644
--- a/blog/2025/07/28/datafusion-49.0.0/index.html
+++ b/blog/2025/07/28/datafusion-49.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/08/15/external-parquet-indexes/index.html b/blog/2025/08/15/external-parquet-indexes/index.html
index 80fe7b9..6dfd16c 100644
--- a/blog/2025/08/15/external-parquet-indexes/index.html
+++ b/blog/2025/08/15/external-parquet-indexes/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/09/10/dynamic-filters/index.html b/blog/2025/09/10/dynamic-filters/index.html
index 8c9a491..82306f8 100644
--- a/blog/2025/09/10/dynamic-filters/index.html
+++ b/blog/2025/09/10/dynamic-filters/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/09/16/datafusion-comet-0.10.0/index.html b/blog/2025/09/16/datafusion-comet-0.10.0/index.html
index 3226fc5..f4e0f4a 100644
--- a/blog/2025/09/16/datafusion-comet-0.10.0/index.html
+++ b/blog/2025/09/16/datafusion-comet-0.10.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/09/21/custom-types-using-metadata/index.html b/blog/2025/09/21/custom-types-using-metadata/index.html
index a258628..0c8fdd4 100644
--- a/blog/2025/09/21/custom-types-using-metadata/index.html
+++ b/blog/2025/09/21/custom-types-using-metadata/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/09/29/datafusion-50.0.0/index.html b/blog/2025/09/29/datafusion-50.0.0/index.html
index 02e0931..d663199 100644
--- a/blog/2025/09/29/datafusion-50.0.0/index.html
+++ b/blog/2025/09/29/datafusion-50.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/10/21/datafusion-comet-0.11.0/index.html b/blog/2025/10/21/datafusion-comet-0.11.0/index.html
index 07e2e49..b6d48fc 100644
--- a/blog/2025/10/21/datafusion-comet-0.11.0/index.html
+++ b/blog/2025/10/21/datafusion-comet-0.11.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/11/25/datafusion-51.0.0/index.html b/blog/2025/11/25/datafusion-51.0.0/index.html
index 46a987b..6cfb389 100644
--- a/blog/2025/11/25/datafusion-51.0.0/index.html
+++ b/blog/2025/11/25/datafusion-51.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/12/04/datafusion-comet-0.12.0/index.html b/blog/2025/12/04/datafusion-comet-0.12.0/index.html
index 109cc0d..332bdf0 100644
--- a/blog/2025/12/04/datafusion-comet-0.12.0/index.html
+++ b/blog/2025/12/04/datafusion-comet-0.12.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2025/12/15/avoid-consecutive-repartitions/index.html b/blog/2025/12/15/avoid-consecutive-repartitions/index.html
index 2d673a9..fdba062 100644
--- a/blog/2025/12/15/avoid-consecutive-repartitions/index.html
+++ b/blog/2025/12/15/avoid-consecutive-repartitions/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2026/01/08/datafusion-52.0.0/index.html b/blog/2026/01/12/datafusion-52.0.0/index.html
similarity index 97%
rename from blog/2026/01/08/datafusion-52.0.0/index.html
rename to blog/2026/01/12/datafusion-52.0.0/index.html
index c299530..c83d796 100644
--- a/blog/2026/01/08/datafusion-52.0.0/index.html
+++ b/blog/2026/01/12/datafusion-52.0.0/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
@@ -42,7 +48,7 @@
<h1>
Apache DataFusion 52.0.0 Released
</h1>
- <p>Posted on: Thu 08 January 2026 by pmc</p>
+ <p>Posted on: Mon 12 January 2026 by pmc</p>
<aside class="toc-container d-md-none mb-2">
<div class="toc"><span class="toctitle">Contents</span><ul>
diff --git a/blog/2026/01/12/extending-sql/index.html b/blog/2026/01/12/extending-sql/index.html
index 01007ff..d8a68e3 100644
--- a/blog/2026/01/12/extending-sql/index.html
+++ b/blog/2026/01/12/extending-sql/index.html
@@ -10,6 +10,8 @@
<link href="/blog/css/headerlink.css" rel="stylesheet">
<link href="/blog/highlight/default.min.css" rel="stylesheet">
<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
<script src="/blog/highlight/highlight.js"></script>
<script>hljs.highlightAll();</script> </head>
<body class="d-flex flex-column h-100">
@@ -17,7 +19,7 @@
<!-- nav bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
<div class="container-fluid">
- <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
@@ -31,6 +33,10 @@
<a class="nav-link" href="/blog/feed.xml">RSS</a>
</li>
</ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
</div>
</div>
</nav>
diff --git a/blog/2026/01/30/datafusion-comet-0.13.0/index.html b/blog/2026/01/30/datafusion-comet-0.13.0/index.html
new file mode 100644
index 0000000..ffc0400
--- /dev/null
+++ b/blog/2026/01/30/datafusion-comet-0.13.0/index.html
@@ -0,0 +1,228 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion Comet 0.13.0 Release - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
+ </div>
+ </div>
+</nav>
+<!-- article contents -->
+<div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>
+ Apache DataFusion Comet 0.13.0 Release
+ </h1>
+ <p>Posted on: Fri 30 January 2026 by pmc</p>
+
+ <aside class="toc-container d-md-none mb-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#key-features">Key Features</a><ul>
+<li><a href="#native-parquet-write-support-experimental">Native Parquet Write Support (Experimental)</a></li>
+<li><a href="#native-iceberg-improvements">Native Iceberg Improvements</a></li>
+<li><a href="#native-csv-reading-experimental">Native CSV Reading (Experimental)</a></li>
+<li><a href="#new-expressions">New Expressions</a></li>
+<li><a href="#ansi-mode-support">ANSI Mode Support</a></li>
+<li><a href="#native-shuffle-improvements">Native Shuffle Improvements</a></li>
+</ul>
+</li>
+<li><a href="#performance-improvements">Performance Improvements</a></li>
+<li><a href="#deprecations">Deprecations</a></li>
+<li><a href="#compatibility">Compatibility</a></li>
+</ul>
+</div>
+ </aside>
+
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development work and is the result of merging 169 PRs from 15
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.13.0.md">change log</a> for more information.</p>
+<h2 id="key-features">Key Features<a class="headerlink" href="#key-features" title="Permanent link">¶</a></h2>
+<h3 id="native-parquet-write-support-experimental">Native Parquet Write Support (Experimental)<a class="headerlink" href="#native-parquet-write-support-experimental" title="Permanent link">¶</a></h3>
+<p>This release introduces experimental native Parquet write capabilities, allowing Comet to intercept and execute Parquet write operations natively through DataFusion. Key capabilities include:</p>
+<ul>
+<li>File commit protocol support for reliable writes</li>
+<li>Remote HDFS writing via OpenDAL integration</li>
+<li>Complex type support (arrays, maps, structs)</li>
+<li>Proper handling of object store settings</li>
+</ul>
+<p>To enable native Parquet writes, set:</p>
+<pre><code>spark.comet.allowIncompatibleOp.DataWritingCommandExec=true
+spark.comet.parquet.write.enabled=true
+</code></pre>
+<p><strong>Note</strong>: This feature is highly experimental and should not be used in production environments. It is currently categorized as a testing feature and is disabled by default.</p>
+<h3 id="native-iceberg-improvements">Native Iceberg Improvements<a class="headerlink" href="#native-iceberg-improvements" title="Permanent link">¶</a></h3>
+<p>Comet's fully-native Iceberg integration received significant enhancements in this release:</p>
+<p><strong>REST Catalog Support</strong>: Native Iceberg scans now support REST catalogs, enabling integration with catalog services like Apache Polaris and Tabular. Configure with:</p>
+<pre><code class="language-shell">--conf spark.sql.catalog.rest_cat=org.apache.iceberg.spark.SparkCatalog
+--conf spark.sql.catalog.rest_cat.catalog-impl=org.apache.iceberg.rest.RESTCatalog
+--conf spark.sql.catalog.rest_cat.uri=http://localhost:8181
+--conf spark.comet.scan.icebergNative.enabled=true
+</code></pre>
+<p><strong>Session Token Authentication</strong>: Added support for session tokens in native Iceberg scans for secure S3 access.</p>
+<p><strong>Performance Optimizations</strong>:</p>
+<ul>
+<li>Deduplicated serialized metadata reducing memory overhead</li>
+<li>Switched from JSON to protobuf for partition value serialization</li>
+<li>Removed IcebergFileStream in favor of iceberg-rust's built-in parallelization</li>
+<li>Reduced metadata serialization points</li>
+<li>Added SchemaAdapter caching</li>
+</ul>
+<p>To enable fully-native Iceberg scanning:</p>
+<pre><code>spark.comet.scan.icebergNative.enabled=true
+</code></pre>
+<p>The native reader supports Iceberg table spec v1 and v2, all primitive and complex types, schema evolution, time travel, positional and equality deletes, filter pushdown, and various storage backends (local, HDFS, S3).</p>
+<h3 id="native-csv-reading-experimental">Native CSV Reading (Experimental)<a class="headerlink" href="#native-csv-reading-experimental" title="Permanent link">¶</a></h3>
+<p>Experimental support for native CSV file reading has been added, expanding Comet's file format capabilities beyond Parquet.</p>
+<h3 id="new-expressions">New Expressions<a class="headerlink" href="#new-expressions" title="Permanent link">¶</a></h3>
+<p>The release adds support for numerous expressions:</p>
+<ul>
+<li>Array functions: <code>explode</code>, <code>explode_outer</code>, <code>size</code></li>
+<li>Date/time functions: <code>unix_date</code>, <code>date_format</code>, <code>datediff</code>, <code>last_day</code>, <code>unix_timestamp</code></li>
+<li>String functions: <code>left</code></li>
+<li>JSON functions: <code>from_json</code> (partial support)</li>
+</ul>
+<h3 id="ansi-mode-support">ANSI Mode Support<a class="headerlink" href="#ansi-mode-support" title="Permanent link">¶</a></h3>
+<p>Sum and average aggregate expressions now support ANSI mode for both integer and decimal inputs, enabling overflow checking in strict SQL mode.</p>
+<h3 id="native-shuffle-improvements">Native Shuffle Improvements<a class="headerlink" href="#native-shuffle-improvements" title="Permanent link">¶</a></h3>
+<ul>
+<li>Round-robin partitioning is now supported in native shuffle</li>
+<li>Spill metrics are now reported correctly</li>
+<li>Configurable shuffle writer buffer size via <code>spark.comet.shuffle.write.bufferSize</code></li>
+</ul>
+<h2 id="performance-improvements">Performance Improvements<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>This release includes extensive performance optimizations:</p>
+<ul>
+<li><strong>String to integer casting</strong>: Significant speedups through optimized parsing</li>
+<li><strong>String functions</strong>: Optimized <code>lpad</code>/<code>rpad</code> to remove unnecessary memory allocations</li>
+<li><strong>Date operations</strong>: Improved <code>normalize_nan</code> and date truncate performance</li>
+<li><strong>Query planning</strong>: Cached query plans to avoid per-partition serialization overhead</li>
+<li><strong>Memory efficiency</strong>: Reduced GC pressure in protobuf serialization</li>
+<li><strong>Hash operations</strong>: Optimized complex-type hash implementations including murmur3 support for nested types</li>
+<li><strong>Runtime efficiency</strong>: Eliminated busy-polling of Tokio stream for plans without CometScan</li>
+<li><strong>Metrics overhead</strong>: Reduced timer and syscall overhead in native shuffle writer</li>
+</ul>
+<h2 id="deprecations">Deprecations<a class="headerlink" href="#deprecations" title="Permanent link">¶</a></h2>
+<p>The <code>native_comet</code> scan mode is now deprecated in favor of <code>native_iceberg_compat</code> and will be removed in a future release. The <code>auto</code> scan mode no longer falls back to <code>native_comet</code>.</p>
+<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
+<p>This release upgrades to DataFusion 51, Arrow 57, and the latest iceberg-rust. The minimum supported Rust version is now 1.88.</p>
+<p>Supported platforms include Spark 3.4.3, 3.5.4-3.5.7, and Spark 4.0.x with various JDK and Scala combinations.</p>
+<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p>
+
+<!--
+ Comments Section
+ Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div id="comments">
+ <hr>
+ <h3>Comments</h3>
+
+ <!-- Local loader script -->
+ <script src="/content/js/giscus-consent.js" defer></script>
+
+ <!-- Consent UI -->
+ <div id="giscus-consent">
+ <p>
+ We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
+ To respect your privacy, Giscus and comments will load only if you click "Show Comments"
+ </p>
+
+ <div class="consent-actions">
+ <button id="giscus-load" type="button">Show Comments</button>
+ <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+ </div>
+
+ <noscript>JavaScript is required to load comments from Giscus.</noscript>
+ </div>
+
+ <!-- Container where Giscus will render -->
+ <div id="comment-thread"></div>
+</div> </div>
+ <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#key-features">Key Features</a><ul>
+<li><a href="#native-parquet-write-support-experimental">Native Parquet Write Support (Experimental)</a></li>
+<li><a href="#native-iceberg-improvements">Native Iceberg Improvements</a></li>
+<li><a href="#native-csv-reading-experimental">Native CSV Reading (Experimental)</a></li>
+<li><a href="#new-expressions">New Expressions</a></li>
+<li><a href="#ansi-mode-support">ANSI Mode Support</a></li>
+<li><a href="#native-shuffle-improvements">Native Shuffle Improvements</a></li>
+</ul>
+</li>
+<li><a href="#performance-improvements">Performance Improvements</a></li>
+<li><a href="#deprecations">Deprecations</a></li>
+<li><a href="#compatibility">Compatibility</a></li>
+</ul>
+</div>
+ </aside>
+ </div>
+ </div>
+</div>
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/2026/02/02/datafusion_case/index.html b/blog/2026/02/02/datafusion_case/index.html
new file mode 100644
index 0000000..1a4e951
--- /dev/null
+++ b/blog/2026/02/02/datafusion_case/index.html
@@ -0,0 +1,503 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Optimizing SQL CASE Expression Evaluation - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
+ </div>
+ </div>
+</nav>
+<!-- article contents -->
+<div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>
+ Optimizing SQL CASE Expression Evaluation
+ </h1>
+ <p>Posted on: Mon 02 February 2026 by Pepijn Van Eeckhoudt</p>
+
+ <aside class="toc-container d-md-none mb-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#background-case-expression-evaluation">Background: CASE Expression Evaluation</a></li>
+<li><a href="#case-evaluation-in-datafusion-5000">CASE Evaluation in DataFusion 50.0.0</a><ul>
+<li><a href="#opportunity-1-early-exit">Opportunity 1: Early Exit</a></li>
+<li><a href="#opportunity-2-optimize-repeated-filtering-scattering-and-merging">Opportunity 2: Optimize Repeated Filtering, Scattering, and Merging</a></li>
+<li><a href="#opportunity-3-filter-only-necessary-columns">Opportunity 3: Filter only Necessary Columns</a></li>
+</ul>
+</li>
+<li><a href="#performance-optimizations">Performance Optimizations</a><ul>
+<li><a href="#optimization-1-short-circuit-early-exit">Optimization 1: Short-Circuit Early Exit</a></li>
+<li><a href="#optimization-2-optimized-result-merging">Optimization 2: Optimized Result Merging</a></li>
+<li><a href="#optimization-3-column-projection">Optimization 3: Column Projection</a></li>
+<li><a href="#optimization-4-eliminating-scatter-in-two-branch-case">Optimization 4: Eliminating Scatter in Two-Branch Case</a></li>
+<li><a href="#optimization-5-table-lookup-of-constants">Optimization 5: Table Lookup of Constants</a></li>
+</ul>
+</li>
+<li><a href="#results">Results</a></li>
+<li><a href="#summary">Summary</a></li>
+<li><a href="#about-datafusion">About DataFusion</a></li>
+<li><a href="#how-to-get-involved">How to Get Involved</a></li>
+</ul>
+</div>
+ </aside>
+
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a set of expressions is evaluated for each row based on arbitrary boolean expressions.
+Its deceptively simple syntax hides significant implementation complexity.
+Over the past few <a href="https://datafusion.apache.org/">Apache DataFusion</a> releases, a series of improvements to <code>CASE</code> expression evaluator have been merged that reduce both CPU time and memory allocations.
+This post provides an overview of the original implementation, its performance bottlenecks, and the steps taken to address them.</p>
+<h2 id="background-case-expression-evaluation">Background: CASE Expression Evaluation<a class="headerlink" href="#background-case-expression-evaluation" title="Permanent link">¶</a></h2>
+<p>SQL supports two forms of CASE expressions:</p>
+<ol>
+<li><strong>Simple</strong>: <code>CASE expr WHEN value1 THEN result1 WHEN value2 THEN result2 ... END</code></li>
+<li><strong>Searched</strong>: <code>CASE WHEN condition1 THEN result1 WHEN condition2 THEN result2 ... END</code></li>
+</ol>
+<p>The simple form evaluates an expression once for each input row and then tests that value against the expressions (typically constants) in each <code>WHEN</code> clause using equality comparisons.</p>
+<p>Here's an example of the simple form:</p>
+<pre><code class="language-sql">CASE status
+ WHEN 'pending' THEN 1
+ WHEN 'active' THEN 2
+ WHEN 'complete' THEN 3
+ ELSE 0
+END
+</code></pre>
+<p>In this <code>CASE</code> expression, <code>status</code> is evaluated once per row, and then its value is tested for equality with the values <code>'pending'</code>, <code>'active'</code>, and <code>'complete'</code> in that order.
+The <code>CASE</code> expression evaluates to the value of the <code>THEN</code> expression corresponding to the first matching <code>WHEN</code> expression.</p>
+<p>The searched <code>CASE</code> form is a more flexible variant.
+It evaluates completely independent boolean expressions for each branch.
+This allows you to test different columns with different operators per branch as shown in the following example:</p>
+<pre><code class="language-sql">CASE
+ WHEN age > 65 THEN 'senior'
+ WHEN childCount != 0 THEN 'parent'
+ WHEN age < 21 THEN 'minor'
+ ELSE 'adult'
+END
+</code></pre>
+<p>In both forms, branches are evaluated sequentially with short-circuit semantics: for each row, once a <code>WHEN</code> condition matches, the corresponding <code>THEN</code> expression is evaluated.
+Any further branches are not evaluated for that row.
+This lazy evaluation model is critical for correctness.
+It lets you safely write <code>CASE</code> expressions like <code>CASE WHEN d != 0 THEN n / d ELSE NULL END</code> that are guaranteed to not trigger divide-by-zero errors.</p>
+<p>Besides <code>CASE</code>, there are a few <a href="https://datafusion.apache.org/user-guide/sql/scalar_functions.html#conditional-functions">conditional scalar functions</a> that provide similar, more restricted capabilities.
+These include <code>COALESCE</code>, <code>IFNULL</code>, and <code>NVL2</code>.
+You can consider each of these functions as the equivalent of a macro for <code>CASE</code>.
+For example, <code>COALESCE(expr1, expr2, expr3)</code> expands to:</p>
+<pre><code class="language-sql">CASE
+ WHEN expr1 IS NOT NULL THEN expr1
+ WHEN expr2 IS NOT NULL THEN expr2
+ ELSE expr3
+END
+</code></pre>
+<p>Since <a href="https://datafusion.apache.org/">Apache DataFusion</a> rewrites these conditional functions to their equivalent <code>CASE</code> expression, any optimizations related to <code>CASE</code> described in this post also apply to conditional function evaluation.</p>
+<h2 id="case-evaluation-in-datafusion-5000"><code>CASE</code> Evaluation in DataFusion 50.0.0<a class="headerlink" href="#case-evaluation-in-datafusion-5000" title="Permanent link">¶</a></h2>
+<p>For the remainder of this post, we'll be looking at 'searched CASE' evaluation.
+'Simple CASE' uses a distinct, but very similar implementation.
+The same set of improvements has been applied to both.</p>
+<p>The baseline implementation in DataFusion 50.0.0 evaluated <code>CASE</code> using a common, straightforward approach:</p>
+<ol>
+<li>Start with an output array <code>out</code> with the same length as the input batch, filled with nulls. Additionally, create a bit vector <code>remainder</code> with the same length and each value set to <code>true</code>.</li>
+<li>For each <code>WHEN</code>/<code>THEN</code> branch:<ul>
+<li>Evaluate the <code>WHEN</code> condition for the remaining unmatched rows using <a href="https://docs.rs/datafusion/latest/datafusion/physical_expr/trait.PhysicalExpr.html#method.evaluate_selection"><code>PhysicalExpr::evaluate_selection</code></a>, passing in the input batch and the <code>remainder</code> mask.</li>
+<li>If any rows matched, evaluate the <code>THEN</code> expression for those rows using <code>PhysicalExpr::evaluate_selection</code>.</li>
+<li>Merge the results into the <code>out</code> array using the <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/zip/fn.zip.html"><code>zip</code></a> kernel.</li>
+<li>Update the <code>remainder</code> mask to exclude the matched rows.</li>
+</ul>
+</li>
+<li>If there's an <code>ELSE</code> clause, evaluate it for any remaining unmatched rows and merge using <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/zip/fn.zip.html"><code>zip</code></a>.</li>
+</ol>
+<p>Here's a simplified version of the Rust code for the original loop:</p>
+<pre><code class="language-rust">let mut out = new_null_array(&return_type, batch.num_rows());
+let mut remainder = BooleanArray::from(vec![true; batch.num_rows()]);
+
+for (when_expr, then_expr) in &self.when_then_expr {
+ // Determine for which remaining rows the WHEN condition matches
+ let when = when_expr.evaluate_selection(batch, &remainder)?
+ .into_array(batch.num_rows())?;
+ // Ensure any `NULL` values are treated as false
+ let when_and_rem = and(&when, &remainder)?;
+
+ if when_and_rem.true_count() == 0 {
+ continue;
+ }
+
+ // Evaluate the THEN expression for matching rows
+ let then = then_expr.evaluate_selection(batch, &when_and_rem)?;
+ // Merge results into output array
+ out = zip(&when_and_rem, &then_value, &out)?;
+ // Update remainder mask to exclude matched rows
+ remainder = and_not(&remainder, &when_and_rem)?;
+}
+</code></pre>
+<p>Let's examine one iteration of this loop for the following <code>CASE</code> expression:</p>
+<pre><code class="language-sql">CASE
+ WHEN col = 'b' THEN 100
+ ELSE 200
+END
+</code></pre>
+<p>Schematically, it will look as follows:</p>
+<figure>
+<img alt="Schematic representation of data flow in the original CASE implementation" class="img-responsive" src="/blog/images/case/original_loop.svg" width="100%"/>
+<figcaption>One iteration of the `CASE` evaluation loop</figcaption>
+</figure>
+<p>This implementation works perfectly fine, but there's significant room for optimization, mostly related to the usage of <code>evaluate_selection</code>.
+To understand why, we need to dig a little deeper into the implementation of that function.
+Here's a simplified version of it that captures the relevant parts:</p>
+<pre><code class="language-rust">pub trait PhysicalExpr {
+ fn evaluate_selection(
+ &self,
+ batch: &RecordBatch,
+ selection: &BooleanArray,
+ ) -> Result<ColumnarValue> {
+ // Reduce record batch to only include rows that match selection
+ let filtered_batch = filter_record_batch(batch, selection)?;
+ // Perform regular evaluation on filtered batch
+ let filtered_result = self.evaluate(&filtered_batch)?;
+ // Expand result array to match original batch length
+ scatter(selection, filtered_result)
+ }
+}
+</code></pre>
+<p>Going back to the same example as before, the data flow in <code>evaluate_selection</code> looks like this:</p>
+<figure>
+<img alt="Schematic representation of `evaluate_selection` evaluation" class="img-responsive" src="/blog/images/case/evaluate_selection.svg" width="100%"/>
+<figcaption>evaluate_selection data flow</figcaption>
+</figure>
+<p>The <code>evaluate_selection</code> method first filters the input batch to only include rows that match the <code>selection</code> mask.
+It then calls the regular <code>evaluate</code> method using the filtered batch as input.
+Finally, to return a result array with the same number of rows as <code>batch</code>, the <code>scatter</code> function is called.
+This function produces a new array padded with <code>null</code> values for any rows that didn't match the <code>selection</code> mask.</p>
+<p>So how can we improve the performance of the simple evaluation strategy and use of <code>evaluate_selection</code>?</p>
+<h3 id="opportunity-1-early-exit">Opportunity 1: Early Exit<a class="headerlink" href="#opportunity-1-early-exit" title="Permanent link">¶</a></h3>
+<p>The <code>CASE</code> evaluation loop always iterates through all branches, even when every row has already been matched.
+In queries where early branches match all rows, this results in unnecessary work being done for the remaining branches.</p>
+<h3 id="opportunity-2-optimize-repeated-filtering-scattering-and-merging">Opportunity 2: Optimize Repeated Filtering, Scattering, and Merging<a class="headerlink" href="#opportunity-2-optimize-repeated-filtering-scattering-and-merging" title="Permanent link">¶</a></h3>
+<p>Each iteration performs a number of operations that are very well-optimized, but still take up a significant amount of CPU time:</p>
+<ul>
+<li><strong>Filtering</strong>: <code>PhysicalExpr::evaluate_selection</code> filters the entire <code>RecordBatch</code> for each branch. For the <code>WHEN</code> expression, this is done even if the selection mask was entirely empty.</li>
+<li><strong>Scattering</strong>: <code>PhysicalExpr::evaluate_selection</code> scatters the filtered result back to the original <code>RecordBatch</code> length.</li>
+<li><strong>Merging</strong>: The <code>zip</code> kernel is called once per branch to merge partial results into the output array</li>
+</ul>
+<p>Each of these operations needs to allocate memory for new arrays and shuffle quite a bit of data around.</p>
+<h3 id="opportunity-3-filter-only-necessary-columns">Opportunity 3: Filter only Necessary Columns<a class="headerlink" href="#opportunity-3-filter-only-necessary-columns" title="Permanent link">¶</a></h3>
+<p>The <code>PhysicalExpr::evaluate_selection</code> method filters the entire record batch, including columns that the current branch's <code>WHEN</code> and <code>THEN</code> expressions don't reference.
+For wide tables (many columns) with narrow expressions (few column references), this is wasteful.</p>
+<p>Suppose you have a table with 26 columns named <code>a</code> through <code>z</code>, and the following simple <code>CASE</code> expression:</p>
+<pre><code class="language-sql">CASE
+ WHEN a > 1000 THEN 'large'
+ WHEN a >= 0 THEN 'positive'
+ ELSE 'negative'
+END
+</code></pre>
+<p>The implementation would filter all 26 columns even though only a single column is needed for the entire <code>CASE</code> expression evaluation.
+Again this involves a non-negligible amount of allocation and data copying.</p>
+<h2 id="performance-optimizations">Performance Optimizations<a class="headerlink" href="#performance-optimizations" title="Permanent link">¶</a></h2>
+<h3 id="optimization-1-short-circuit-early-exit">Optimization 1: Short-Circuit Early Exit<a class="headerlink" href="#optimization-1-short-circuit-early-exit" title="Permanent link">¶</a></h3>
+<p>The first optimization is straightforward.
+As soon as we detect that all rows of the batch have been matched, we break out of the evaluation loop:</p>
+<pre><code class="language-rust">let mut remainder_count = batch.num_rows();
+
+for (when_expr, then_expr) in &self.when_then_expr {
+ if remainder_count == 0 {
+ break; // All rows matched, exit early
+ }
+
+ // ... evaluate branch ...
+
+ let when_match_count = when_value.true_count();
+ remainder_count -= when_match_count;
+}
+</code></pre>
+<p>Additionally, we avoid evaluating the <code>ELSE</code> clause when no rows remain:</p>
+<pre><code class="language-rust">if let Some(else_expr) = &self.else_expr {
+ remainder = or(&base_nulls, &remainder)?;
+ if remainder.true_count() > 0 {
+ // ... evaluate else ...
+ }
+}
+</code></pre>
+<p>For queries where early branches match all rows, this eliminates unnecessary branch evaluations and <code>ELSE</code> clause processing.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/17898">PR #17898</a></p>
+<h3 id="optimization-2-optimized-result-merging">Optimization 2: Optimized Result Merging<a class="headerlink" href="#optimization-2-optimized-result-merging" title="Permanent link">¶</a></h3>
+<p>The second optimization fundamentally restructures how the results of each loop iteration will be merged.
+The diagram below illustrates the optimized data flow when evaluating the <code>CASE WHEN col = 'b' THEN 100 ELSE 200 END</code> from before:</p>
+<figure>
+<img alt="Schematic representation of optimized evaluation loop" class="img-responsive" src="/blog/images/case/merging.svg" width="100%"/>
+<figcaption>optimized evaluation loop</figcaption>
+</figure>
+<p>In the reworked implementation, the <code>evaluate_selection</code> function is no longer used.
+The key insight is that we can defer all merging until the end of the evaluation loop by tracking result provenance.
+This was implemented with the following changes:</p>
+<ol>
+<li>Augment the input batch with a column containing row indices.</li>
+<li>Reduce the augmented batch after each loop iteration to only contain the remaining rows.</li>
+<li>Use the row index column to track which partial result array contains the value for each row.</li>
+<li>Perform a single merge operation at the end instead of a <code>zip</code> operation after each loop iteration.</li>
+</ol>
+<p>These changes make it unnecessary to <code>scatter</code> and <code>zip</code> results in each loop iteration.
+Instead, when all rows have been matched, we then merge the partial results using <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge_n.html"><code>arrow_select::merge::merge_n</code></a>.</p>
+<p>The diagram below illustrates how <code>merge_n</code> works for an example where three <code>WHEN/THEN</code> branches produced results.
+The first branch produced the result <code>A</code> for row 2, the second produced <code>B</code> for row 1, and the third produced <code>C</code> and <code>D</code> for rows 4 and 5.</p>
+<figure>
+<img alt="Schematic illustration of the merge_n algorithm" class="img-responsive" src="/blog/images/case/merge_n.svg" width="100%"/>
+<figcaption>merge_n example</figcaption>
+</figure>
+<p>The <code>merge_n</code> algorithm scans through the indices array.
+For each non-empty cell, it takes one value from the corresponding values array.
+In the example above, we first encounter <code>1</code>.
+This takes the first element from the values array with index <code>1</code>, resulting in <code>B</code>.
+The next cell contains <code>0</code> which takes <code>A</code>, from the first array.
+Finally, we encounter <code>2</code> twice.
+This takes the first and second element from the last values array respectively.</p>
+<p>This algorithm was initially implemented in DataFusion for the <code>CASE</code> implementation, but in the meantime has been generalized and moved into the <code>arrow-rs</code> crate as <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge_n.html"><code>arrow_select::merge::merge_n</code></a>.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18152">PR #18152</a></p>
+<h3 id="optimization-3-column-projection">Optimization 3: Column Projection<a class="headerlink" href="#optimization-3-column-projection" title="Permanent link">¶</a></h3>
+<p>The third optimization addresses the "filtering unused columns" overhead through projection.</p>
+<p>Look at the following query example where the <code>mailing_address</code> table has the columns <code>name</code>, <code>surname</code>, <code>street</code>, <code>number</code>, <code>city</code>, <code>state</code>, <code>country</code>:</p>
+<pre><code class="language-sql">SELECT *, CASE WHEN country = 'USA' THEN state ELSE country END AS region
+FROM mailing_address
+</code></pre>
+<p>You can see that the <code>CASE</code> expression only references the columns <code>country</code> and <code>state</code>, but because all columns are being queried, projection pushdown cannot reduce the number of columns being fed in to the projection operator.</p>
+<figure>
+<img alt="Schematic illustration of CASE evaluation without projection" class="img-responsive" src="/blog/images/case/no_projection.svg" width="100%"/>
+<figcaption>CASE evaluation without projection</figcaption>
+</figure>
+<p>During <code>CASE</code> evaluation, the batch must be filtered using the <code>WHEN</code> expression to evaluate the <code>THEN</code> expression values.
+As the diagram above shows, this filtering creates a reduced copy of all columns.</p>
+<p>This unnecessary copying can be avoided by first narrowing the batch to only include the columns that are actually needed.</p>
+<figure>
+<img alt="Schematic illustration of CASE evaluation with projection" class="img-responsive" src="/blog/images/case/projection.svg" width="100%"/>
+<figcaption>CASE evaluation with projection</figcaption>
+</figure>
+<p>At first glance, this might not seem beneficial, since we're introducing an additional processing step.
+Luckily projection of a record batch only requires a shallow copy of the record batch.
+The column arrays themselves are not copied, and the only work that is actually done is incrementing the reference counts of the columns.</p>
+<p><strong>Impact</strong>: For wide tables with narrow CASE expressions, this dramatically reduces filtering overhead by removing the copying of unused columns.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18329">PR #18329</a></p>
+<h3 id="optimization-4-eliminating-scatter-in-two-branch-case">Optimization 4: Eliminating Scatter in Two-Branch Case<a class="headerlink" href="#optimization-4-eliminating-scatter-in-two-branch-case" title="Permanent link">¶</a></h3>
+<p>Some of the earlier examples in this post use expressions of the form <code>CASE WHEN condition THEN expr1 ELSE expr2 END</code> to explain how the general evaluation loop works.
+For this kind of two-branch <code>CASE</code> expression, <a href="https://datafusion.apache.org/">Apache DataFusion</a> has a more optimized implementation that unrolls the loop.
+This specialized <code>ExpressionOrExpression</code> fast path still used <code>evaluate_selection()</code> for both branches which uses <code>scatter</code> and <code>zip</code> to combine the results incurring the same performance overhead as the general implementation.</p>
+<p>The revised implementation eliminates the use of <code>evaluate_selection</code> as follows:</p>
+<pre><code class="language-rust">// Compute the `WHEN` condition for the entire batch
+let when_filter = create_filter(&when_value);
+
+// Compute a compact array of `THEN` values for the matching rows
+let then_batch = filter_record_batch(batch, &when_filter)?;
+let then_value = then_expr.evaluate(&then_batch)?;
+
+// Compute a compact array of `ELSE` values for the non-matching rows
+let else_filter = create_filter(&not(&when_value)?);
+let else_batch = filter_record_batch(batch, &else_filter)?;
+let else_value = else_expr.evaluate(&else_batch)?;
+</code></pre>
+<p>This produces two compact arrays, one for the THEN values and one for the ELSE values, which are then merged with the <code>merge</code> function.
+In contrast to <code>zip</code>, <code>merge</code> does not require both of its value inputs to have the same length.
+Instead it requires that the sum of the length of the value inputs matches the length of the mask array.</p>
+<figure>
+<img alt="Schematic illustration of the merge algorithm" class="img-responsive" src="/blog/images/case/merge.svg" width="100%"/>
+<figcaption>merge example</figcaption>
+</figure>
+<p>This eliminates unnecessary <code>scatter</code> operations and memory allocations for one of the most common <code>CASE</code> expression patterns.</p>
+<p>Just like <code>merge_n</code>, this operation has been moved into <code>arrow-rs</code> as <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge.html"><code>arrow_select::merge::merge</code></a>.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18444">PR #18444</a></p>
+<h3 id="optimization-5-table-lookup-of-constants">Optimization 5: Table Lookup of Constants<a class="headerlink" href="#optimization-5-table-lookup-of-constants" title="Permanent link">¶</a></h3>
+<p>Up until now, we've discussed the implementations for generic <code>CASE</code> expressions that use non-constant expressions for both <code>WHEN</code> and <code>THEN</code>.
+Another common use of <code>CASE</code> is to perform a mapping from one set of constants to another.
+For instance, you can expand numeric constants to human-readable strings using the following <code>CASE</code> example.</p>
+<pre><code class="language-sql">CASE status
+ WHEN 0 THEN 'idle'
+ WHEN 1 THEN 'running'
+ WHEN 2 THEN 'paused'
+ WHEN 3 THEN 'stopped'
+ ELSE 'unknown'
+END
+</code></pre>
+<p>A final <code>CASE</code> optimization recognizes this pattern and compiles the <code>CASE</code> expression into a hash table.
+Rather than evaluating the <code>WHEN</code> and <code>THEN</code> expressions, the input expression is evaluated once, and the result array is computed using a vectorized hash table lookup.
+This approach avoids the need to filter the input batch and combine partial results entirely.
+The result array is computed in a single pass over the input values, and the computation time does not grow significantly with the number of <code>WHEN</code> branches in the <code>CASE</code> expression.</p>
+<p>This optimization was implemented by Raz Luvaton (<a href="https://github.com/rluvaton"><code>@rluvaton</code></a>) in <a href="https://github.com/apache/datafusion/pull/18183">PR #18183</a></p>
+<h2 id="results">Results<a class="headerlink" href="#results" title="Permanent link">¶</a></h2>
+<p>The degree to which the performance optimizations described in this post will benefit your queries is highly dependent on both your data and your queries.
+To give some idea of the impact, we ran the following query on the TPC_H <code>orders</code> table with a scale factor of 100:</p>
+<pre><code class="language-sql">SELECT
+ *,
+ case o_orderstatus
+ when 'O' then 'ordered'
+ when 'F' then 'filled'
+ when 'P' then 'pending'
+ else 'other'
+ end
+from orders
+</code></pre>
+<p>This query was first run with DataFusion 50.0.0 to get a baseline measurement.
+The same query was then run with each optimization applied in turn.
+The recorded times are presented as the blue series in the chart below.
+The green series shows the time measurement for the <code>SELECT * FROM orders</code> to give an idea of the cost the addition of a <code>CASE</code> expression in a query incurs.
+All measurements were made with a target partition count of <code>1</code>.</p>
+<figure>
+<img alt="Performance measurements chart" class="img-responsive" src="/blog/images/case/results.png" width="100%"/>
+<figcaption>Performance measurements</figcaption>
+</figure>
+<p>What you can see in the chart is that the effect of the various optimizations compounds up to the <code>project</code> measurement.
+Up to that point these results are applicable to any <code>CASE</code> expression.
+The final improvement in the <code>hash</code> measurement is only applicable to simple <code>CASE</code> expressions with constant <code>WHEN</code> and <code>THEN</code> expressions.</p>
+<p>The cumulative effect of these optimizations is a 63-71% reduction in CPU time spent evaluating <code>CASE</code> expressions compared to the baseline.</p>
+<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h2>
+<p>Through a number of targeted optimizations, we've transformed <code>CASE</code> expression evaluation from a simple, but unoptimized implementation into a highly optimized one.
+The optimizations described in this post compound: a <code>CASE</code> expression on a wide table with multiple branches and early matches benefits from all four optimizations simultaneously.
+The result is significantly reduced CPU time and memory allocation in SQL constructs that are essential for ETL-like queries.</p>
+<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries,
+and machine learning and streaming applications.
+While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion’s primary design goal</a> is to accelerate the creation of other data-centric systems, it provides a reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe library</a>, <a href="https://datafusion.apache.org/python/">Python library</a>, and <a href="https://datafusion.apache.org/user-guide/cli/">command-line SQL tool</a>.</p>
+<p>DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone.
+Without DataFusion, highly performant vectorized query engines would remain the domain of a few large companies and world-class research institutions.
+With DataFusion, we can all build on top of a shared foundation and focus on what makes our projects unique.</p>
+<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
+<p>DataFusion is not a project built or driven by a single person, company, or foundation.
+Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code.
+A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p>
+
+<!--
+ Comments Section
+ Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div id="comments">
+ <hr>
+ <h3>Comments</h3>
+
+ <!-- Local loader script -->
+ <script src="/content/js/giscus-consent.js" defer></script>
+
+ <!-- Consent UI -->
+ <div id="giscus-consent">
+ <p>
+ We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
+ To respect your privacy, Giscus and comments will load only if you click "Show Comments"
+ </p>
+
+ <div class="consent-actions">
+ <button id="giscus-load" type="button">Show Comments</button>
+ <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+ </div>
+
+ <noscript>JavaScript is required to load comments from Giscus.</noscript>
+ </div>
+
+ <!-- Container where Giscus will render -->
+ <div id="comment-thread"></div>
+</div> </div>
+ <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#background-case-expression-evaluation">Background: CASE Expression Evaluation</a></li>
+<li><a href="#case-evaluation-in-datafusion-5000">CASE Evaluation in DataFusion 50.0.0</a><ul>
+<li><a href="#opportunity-1-early-exit">Opportunity 1: Early Exit</a></li>
+<li><a href="#opportunity-2-optimize-repeated-filtering-scattering-and-merging">Opportunity 2: Optimize Repeated Filtering, Scattering, and Merging</a></li>
+<li><a href="#opportunity-3-filter-only-necessary-columns">Opportunity 3: Filter only Necessary Columns</a></li>
+</ul>
+</li>
+<li><a href="#performance-optimizations">Performance Optimizations</a><ul>
+<li><a href="#optimization-1-short-circuit-early-exit">Optimization 1: Short-Circuit Early Exit</a></li>
+<li><a href="#optimization-2-optimized-result-merging">Optimization 2: Optimized Result Merging</a></li>
+<li><a href="#optimization-3-column-projection">Optimization 3: Column Projection</a></li>
+<li><a href="#optimization-4-eliminating-scatter-in-two-branch-case">Optimization 4: Eliminating Scatter in Two-Branch Case</a></li>
+<li><a href="#optimization-5-table-lookup-of-constants">Optimization 5: Table Lookup of Constants</a></li>
+</ul>
+</li>
+<li><a href="#results">Results</a></li>
+<li><a href="#summary">Summary</a></li>
+<li><a href="#about-datafusion">About DataFusion</a></li>
+<li><a href="#how-to-get-involved">How to Get Involved</a></li>
+</ul>
+</div>
+ </aside>
+ </div>
+ </div>
+</div>
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/2026/03/20/limit-pruning/index.html b/blog/2026/03/20/limit-pruning/index.html
new file mode 100644
index 0000000..c17b76c
--- /dev/null
+++ b/blog/2026/03/20/limit-pruning/index.html
@@ -0,0 +1,413 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
+ </div>
+ </div>
+</nav>
+<!-- article contents -->
+<div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>
+ Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack
+ </h1>
+ <p>Posted on: Fri 20 March 2026 by xudong</p>
+
+ <aside class="toc-container d-md-none mb-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#datafusions-pruning-pipeline">DataFusion's Pruning Pipeline</a><ul>
+<li><a href="#phase-1-high-level-discovery">Phase 1: High-Level Discovery</a></li>
+<li><a href="#phase-2-row-group-statistics">Phase 2: Row Group Statistics</a></li>
+<li><a href="#phase-3-granular-pruning">Phase 3: Granular Pruning</a></li>
+</ul>
+</li>
+<li><a href="#the-problem-limit-was-ignored">The Problem: LIMIT Was Ignored</a></li>
+<li><a href="#the-solution-limit-aware-pruning">The Solution: Limit-Aware Pruning</a></li>
+<li><a href="#mechanism-detecting-fully-matched-row-groups">Mechanism: Detecting Fully Matched Row Groups</a></li>
+<li><a href="#mechanism-rewriting-the-access-plan">Mechanism: Rewriting the Access Plan</a></li>
+<li><a href="#case-study-alpine-wildlife-query">Case Study: Alpine Wildlife Query</a></li>
+<li><a href="#observing-limit-pruning-via-metrics">Observing Limit Pruning via Metrics</a></li>
+<li><a href="#future-directions">Future Directions</a></li>
+<li><a href="#summary">Summary</a></li>
+<li><a href="#about-datafusion">About DataFusion</a></li>
+<li><a href="#how-to-get-involved">How to Get Involved</a></li>
+</ul>
+</div>
+ </aside>
+
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make. This sounds obvious, but making it happen in practice requires careful engineering at every layer of the query engine. <a href="https://datafusion.apache.org/">Apache DataFusion</a> achieves this through a multi-layer <strong>pruning pipeline</strong> — a series of stages that progressively narrow down the data before decoding a single row.</p>
+<p>In this post, we describe a new optimization called <strong>limit pruning</strong> that makes this pipeline aware of SQL <code>LIMIT</code> clauses. By identifying row groups where <em>every</em> row is guaranteed to match the predicate, DataFusion can satisfy a <code>LIMIT</code> query without ever touching partially matching row groups — eliminating wasted I/O entirely.</p>
+<p>For example, given a query like:</p>
+<pre><code class="language-sql">SELECT * FROM tracking_data
+WHERE species LIKE 'Alpine%' AND s >= 50
+LIMIT 3
+</code></pre>
+<p>If the pruning pipeline already knows that certain row groups fully satisfy the <code>WHERE</code> clause, those groups alone may contain enough rows to fill the <code>LIMIT</code> — making it unnecessary to scan anything else.</p>
+<p>This work was inspired by the "Pruning for LIMIT Queries" section of Snowflake's paper <a href="https://arxiv.org/pdf/2504.11540"><em>Pruning in Snowflake: Working Smarter, Not Harder</em></a>.</p>
+<h2 id="datafusions-pruning-pipeline">DataFusion's Pruning Pipeline<a class="headerlink" href="#datafusions-pruning-pipeline" title="Permanent link">¶</a></h2>
+<p>Before diving into limit pruning, let's understand the full pruning pipeline. DataFusion scans Parquet data through a series of increasingly fine-grained filters, each one eliminating data so the next stage processes less:</p>
+<figure>
+<img alt="Three phases of DataFusion's pruning pipeline" src="/blog/images/limit-pruning/pruning-phases.svg" width="80%"/>
+<figcaption>Figure 1: The three phases of DataFusion's pruning pipeline — from directories down to individual rows.</figcaption>
+</figure>
+<h3 id="phase-1-high-level-discovery">Phase 1: High-Level Discovery<a class="headerlink" href="#phase-1-high-level-discovery" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong>Partition Pruning</strong>: The <a href="https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html">ListingTable</a> component evaluates filters that depend only on partition columns — things like <code>year</code>, <code>month</code>, or <code>region</code> encoded in directory paths (e.g., <code>s3://data/year=2024/month=01/</code>). Irrelevant directories are eliminated before we even open a file.</li>
+<li><strong>File Stats Pruning</strong>: The <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.FilePruner.html">FilePruner</a> checks file-level min/max and null-count statistics. If these statistics prove that a file cannot satisfy the predicate, we drop it entirely — no need to read row group metadata.</li>
+</ul>
+<h3 id="phase-2-row-group-statistics">Phase 2: Row Group Statistics<a class="headerlink" href="#phase-2-row-group-statistics" title="Permanent link">¶</a></h3>
+<p>For each surviving file, DataFusion reads row group metadata and potentially <a href="https://parquet.apache.org/docs/file-format/bloomfilter/">bloom filters</a> and classifies each row group into one of three states (the example data shown in the figures below, such as "Snow Vole" and "Alpine Ibex", is adapted from the <a href="https://arxiv.org/pdf/2504.11540">Snowflake pruning paper</a>):</p>
+<figure>
+<img alt="Row group classification: not matching, partially matching, fully matching" src="/blog/images/limit-pruning/row-group-states.svg" width="80%"/>
+<figcaption>Figure 2: Row groups are classified into three states based on their statistics.</figcaption>
+</figure>
+<ul>
+<li><strong>Not Matching (Skipped)</strong>: Statistics prove no rows can match. The row group is ignored completely.</li>
+<li><strong>Partially Matching</strong>: Statistics cannot rule out matching rows, but also cannot guarantee them. These groups might be scanned and verified row by row later.</li>
+<li><strong>Fully Matching</strong>: Statistics prove that <em>every single row</em> in the group satisfies the predicate. This state is key to making limit pruning possible.</li>
+</ul>
+<h3 id="phase-3-granular-pruning">Phase 3: Granular Pruning<a class="headerlink" href="#phase-3-granular-pruning" title="Permanent link">¶</a></h3>
+<p>The final phase goes even deeper:</p>
+<ul>
+<li><strong><a href="https://parquet.apache.org/docs/file-format/pageindex/">Page Index Pruning</a></strong>: Parquet pages have their own min/max statistics. DataFusion uses these to skip individual data pages within a surviving row group.</li>
+<li><strong><a href="https://arrow.apache.org/blog/2025/12/11/parquet-late-materialization-deep-dive/">Late Materialization</a> (Row Filtering)</strong>: Instead of decoding all columns at once, DataFusion decodes the cheapest, most selective columns first. It filters rows using those columns, then only decodes the remaining columns for surviving rows.</li>
+</ul>
+<h2 id="the-problem-limit-was-ignored">The Problem: LIMIT Was Ignored<a class="headerlink" href="#the-problem-limit-was-ignored" title="Permanent link">¶</a></h2>
+<p>Before limit pruning, all of these stages worked well — but the pruning pipeline had <strong>no awareness of <code>LIMIT</code></strong>. Consider a query like:</p>
+<pre><code class="language-sql">SELECT * FROM tracking_data
+WHERE species LIKE 'Alpine%' AND s >= 50
+LIMIT 3
+</code></pre>
+<p>Even when fully matched row groups alone contain enough rows to satisfy the <code>LIMIT</code>, DataFusion would still decode partially matching groups and filter out rows that did not match, wasting resources decoding rows just to immediately discard them.</p>
+<figure>
+<img alt="Traditional pruning decodes partially matching groups with no LIMIT awareness" src="/blog/images/limit-pruning/wasted-io.svg" width="80%"/>
+<figcaption>Figure 3: Without limit awareness, partially matching groups are scanned and filtered even when fully matched groups already have enough rows. The left section shows 5 fully matched rows (enough to satisfy <code>LIMIT 5</code>), while the right section with the dashed red border represents a partially matching group that is still decoded — wasting CPU and I/O on rows that may not match at all.</figcaption>
+</figure>
+<p>If five fully matched rows in a fully matched group already satisfy <code>LIMIT 5</code>, why bother decoding groups where we're not even sure any rows qualify?</p>
+<h2 id="the-solution-limit-aware-pruning">The Solution: Limit-Aware Pruning<a class="headerlink" href="#the-solution-limit-aware-pruning" title="Permanent link">¶</a></h2>
+<p>The solution adds a new step in the pruning pipeline — right after row group pruning and before page index pruning:</p>
+<figure>
+<img alt="Pruning pipeline with limit pruning highlighted" src="/blog/images/limit-pruning/pruning-pipeline.svg" width="80%"/>
+<figcaption>Figure 4: Limit pruning is inserted between row group and page index pruning.</figcaption>
+</figure>
+<p>The idea is simple: <strong>if fully matched row groups already contain enough rows to satisfy the <code>LIMIT</code>, rewrite the access plan to scan only those groups and skip everything else.</strong></p>
+<p>This optimization is applied only when the query is a pure limit query with no <code>ORDER BY</code>, because reordering which groups we scan could change the output ordering of the results. In the implementation, this check is expressed as:</p>
+<pre><code class="language-rust">// Prune by limit if limit is set and order is not sensitive
+if let (Some(limit), false) = (limit, preserve_order) {
+ row_groups.prune_by_limit(limit, rg_metadata, &file_metrics);
+}
+</code></pre>
+<h2 id="mechanism-detecting-fully-matched-row-groups">Mechanism: Detecting Fully Matched Row Groups<a class="headerlink" href="#mechanism-detecting-fully-matched-row-groups" title="Permanent link">¶</a></h2>
+<p>The core insight is <strong>predicate negation</strong>. To determine if every row in a row group satisfies the predicate, we:</p>
+<ol>
+<li>Negate the original predicate</li>
+<li>Simplify the negated expression</li>
+<li>Evaluate the negation against the row group's statistics</li>
+<li>If the negation is <em>pruned</em> (proven impossible), then the original predicate holds for every row</li>
+</ol>
+<p>Since DataFusion already had expression simplification (step 2) and statistics-based pruning (step 3), implementing this was relatively straightforward — the key addition was composing these existing capabilities with predicate negation.</p>
+<figure>
+<img alt="Fully matched detection via predicate negation" src="/blog/images/limit-pruning/fully-matched-detection.svg" width="80%"/>
+<figcaption>Figure 5: If the negated predicate is impossible according to row group stats, all rows must match the original predicate.</figcaption>
+</figure>
+<p>In DataFusion's codebase, this logic lives in <code>identify_fully_matched_row_groups</code> (<a href="https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/row_group_filter.rs">row_group_filter.rs</a>):</p>
+<pre><code class="language-rust">fn identify_fully_matched_row_groups(
+ &mut self,
+ candidate_row_group_indices: &[usize],
+ arrow_schema: &Schema,
+ parquet_schema: &SchemaDescriptor,
+ groups: &[RowGroupMetaData],
+ predicate: &PruningPredicate,
+ metrics: &ParquetFileMetrics,
+) {
+ // Create the inverted predicate: NOT(original)
+ let inverted_expr = Arc::new(NotExpr::new(
+ Arc::clone(predicate.orig_expr()),
+ ));
+
+ // Simplify: e.g., NOT(c1 = 0) → c1 != 0
+ let simplifier = PhysicalExprSimplifier::new(arrow_schema);
+ let Ok(inverted_expr) = simplifier.simplify(inverted_expr) else {
+ return;
+ };
+
+ let Ok(inverted_predicate) = PruningPredicate::try_new(
+ inverted_expr,
+ Arc::clone(predicate.schema()),
+ ) else {
+ return;
+ };
+
+ // Evaluate inverted predicate against row group stats
+ let Ok(inverted_values) =
+ inverted_predicate.prune(&inverted_pruning_stats)
+ else {
+ return;
+ };
+
+ for (i, &original_idx) in
+ candidate_row_group_indices.iter().enumerate()
+ {
+ // If negation is pruned (false), all rows match original
+ if !inverted_values[i] {
+ self.is_fully_matched[original_idx] = true;
+ }
+ }
+}
+</code></pre>
+<h2 id="mechanism-rewriting-the-access-plan">Mechanism: Rewriting the Access Plan<a class="headerlink" href="#mechanism-rewriting-the-access-plan" title="Permanent link">¶</a></h2>
+<p>Once we know which row groups are fully matched, the limit pruning algorithm is straightforward:</p>
+<figure>
+<img alt="Limit pruning access plan rewrite algorithm" src="/blog/images/limit-pruning/limit-rewrite-algorithm.svg" width="80%"/>
+<figcaption>Figure 6: The algorithm iterates fully matched groups, accumulating row counts until the limit is satisfied.</figcaption>
+</figure>
+<p>The implementation in <code>prune_by_limit</code> (<a href="https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/row_group_filter.rs">row_group_filter.rs</a>):</p>
+<pre><code class="language-rust">pub fn prune_by_limit(
+ &mut self,
+ limit: usize,
+ rg_metadata: &[RowGroupMetaData],
+ metrics: &ParquetFileMetrics,
+) {
+ let mut fully_matched_indexes: Vec<usize> = Vec::new();
+ let mut fully_matched_rows: usize = 0;
+
+ for &idx in self.access_plan.row_group_indexes().iter() {
+ if self.is_fully_matched[idx] {
+ fully_matched_indexes.push(idx);
+ fully_matched_rows += rg_metadata[idx].num_rows() as usize;
+ if fully_matched_rows >= limit {
+ break;
+ }
+ }
+ }
+
+ // Rewrite the plan if we have enough rows
+ if fully_matched_rows >= limit {
+ let mut new_plan = ParquetAccessPlan::new_none(rg_metadata.len());
+ for &idx in &fully_matched_indexes {
+ new_plan.scan(idx);
+ }
+ self.access_plan = new_plan;
+ }
+}
+</code></pre>
+<p>Key properties of this algorithm:</p>
+<ul>
+<li>It preserves the original row group ordering</li>
+<li>If fully matched groups don't have enough rows, the plan is unchanged — no harm done</li>
+<li>The cost is minimal: a single pass over the row group list</li>
+</ul>
+<h2 id="case-study-alpine-wildlife-query">Case Study: Alpine Wildlife Query<a class="headerlink" href="#case-study-alpine-wildlife-query" title="Permanent link">¶</a></h2>
+<p>Let's walk through a concrete example adapted from the <a href="https://arxiv.org/pdf/2504.11540">Snowflake pruning paper</a>. Given a wildlife tracking dataset with four row groups:</p>
+<pre><code class="language-sql">SELECT * FROM tracking_data
+WHERE species LIKE 'Alpine%' AND s >= 50
+LIMIT 3
+</code></pre>
+<table class="table">
+<thead>
+<tr>
+<th>Row Group</th>
+<th>Species Range</th>
+<th>S Range</th>
+<th>State</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>RG1</td>
+<td>Snow Vole, Brown Bear, Gray Wolf</td>
+<td>7–133</td>
+<td><strong>Not Matching</strong> (no 'Alpine%')</td>
+</tr>
+<tr>
+<td>RG2</td>
+<td>Lynx, Red Fox, Alpine Bat</td>
+<td>6–71</td>
+<td><strong>Partially Matching</strong></td>
+</tr>
+<tr>
+<td>RG3</td>
+<td>Alpine Ibex, Alpine Goat, Alpine Sheep</td>
+<td>76–101</td>
+<td><strong>Fully Matching</strong></td>
+</tr>
+<tr>
+<td>RG4</td>
+<td>Mixed species</td>
+<td>Mixed</td>
+<td><strong>Partially Matching</strong></td>
+</tr>
+</tbody>
+</table>
+<figure>
+<img alt="Before and after limit pruning comparison" src="/blog/images/limit-pruning/before-after.svg" width="80%"/>
+<figcaption>Figure 7: Before limit pruning, RG2 is scanned for zero hits. After limit pruning, only RG3 is scanned.</figcaption>
+</figure>
+<p><strong>Before limit pruning</strong>: DataFusion scans RG2 (0 hits — wasted I/O), then RG3 (3 hits, early return). RG2 was decoded entirely for nothing.</p>
+<p><strong>With limit pruning</strong>: The system detects that RG3 has 3 fully matched rows, which satisfies <code>LIMIT 3</code>. It rewrites the access plan to scan only RG3, skipping RG2 and RG4 entirely. One row group scanned. Zero waste.</p>
+<h2 id="observing-limit-pruning-via-metrics">Observing Limit Pruning via Metrics<a class="headerlink" href="#observing-limit-pruning-via-metrics" title="Permanent link">¶</a></h2>
+<p>DataFusion exposes limit pruning activity through query metrics. When running a query with <code>EXPLAIN ANALYZE</code>, you will see entries like:</p>
+<pre><code>row_groups_pruned_statistics=4 total → 3 matched -> 1 fully matched
+limit_pruned_row_groups=3 total → 1 matched
+</code></pre>
+<p>This tells us:
+- 4 row groups were evaluated, 3 survived statistics pruning, 1 was identified as fully matching
+- Of the 3 row groups that entered limit pruning, only 1 survived — 2 were pruned by the limit optimization</p>
+<h2 id="future-directions">Future Directions<a class="headerlink" href="#future-directions" title="Permanent link">¶</a></h2>
+<p>There are two natural extensions of this work:</p>
+<p><strong><a href="https://github.com/apache/datafusion/issues/19193">Page-Level Limit Pruning</a></strong>: Today, "fully matched" detection operates at the row group level. If we extend this to use page index statistics, we could stop decoding pages <em>within</em> a row group once the limit is met. This would pay dividends for wide row groups where only a few pages hold matching data.</p>
+<p><strong><a href="https://github.com/apache/datafusion/issues/19028">Row Filter Hints</a></strong>: Even when a row group is fully matched, the current row filter still evaluates predicates row by row. If we pass the fully matched groups info into the row filter builder, we can skip predicate evaluation entirely for guaranteed groups — saving CPU cycles on predicate evaluation.</p>
+<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h2>
+<p>DataFusion's pruning pipeline trims redundant I/O from the partition level all the way down to individual rows. Limit pruning adds a new step that creates an early exit when fully matched row groups already satisfy the <code>LIMIT</code>. The result is fewer row groups scanned, less data decoded, and faster queries.</p>
+<p>The key insights are:
+1. <strong>Predicate negation</strong> can identify row groups where <em>all</em> rows match — not just "some might match"
+2. <strong>Row count accumulation</strong> across fully matched groups enables early termination</p>
+<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries, and machine learning and streaming applications.</p>
+<p>DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone.</p>
+<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
+<p>If you are interested in contributing, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a href="https://datafu [...]
+
+<!--
+ Comments Section
+ Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div id="comments">
+ <hr>
+ <h3>Comments</h3>
+
+ <!-- Local loader script -->
+ <script src="/content/js/giscus-consent.js" defer></script>
+
+ <!-- Consent UI -->
+ <div id="giscus-consent">
+ <p>
+ We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
+ To respect your privacy, Giscus and comments will load only if you click "Show Comments"
+ </p>
+
+ <div class="consent-actions">
+ <button id="giscus-load" type="button">Show Comments</button>
+ <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+ </div>
+
+ <noscript>JavaScript is required to load comments from Giscus.</noscript>
+ </div>
+
+ <!-- Container where Giscus will render -->
+ <div id="comment-thread"></div>
+</div> </div>
+ <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#datafusions-pruning-pipeline">DataFusion's Pruning Pipeline</a><ul>
+<li><a href="#phase-1-high-level-discovery">Phase 1: High-Level Discovery</a></li>
+<li><a href="#phase-2-row-group-statistics">Phase 2: Row Group Statistics</a></li>
+<li><a href="#phase-3-granular-pruning">Phase 3: Granular Pruning</a></li>
+</ul>
+</li>
+<li><a href="#the-problem-limit-was-ignored">The Problem: LIMIT Was Ignored</a></li>
+<li><a href="#the-solution-limit-aware-pruning">The Solution: Limit-Aware Pruning</a></li>
+<li><a href="#mechanism-detecting-fully-matched-row-groups">Mechanism: Detecting Fully Matched Row Groups</a></li>
+<li><a href="#mechanism-rewriting-the-access-plan">Mechanism: Rewriting the Access Plan</a></li>
+<li><a href="#case-study-alpine-wildlife-query">Case Study: Alpine Wildlife Query</a></li>
+<li><a href="#observing-limit-pruning-via-metrics">Observing Limit Pruning via Metrics</a></li>
+<li><a href="#future-directions">Future Directions</a></li>
+<li><a href="#summary">Summary</a></li>
+<li><a href="#about-datafusion">About DataFusion</a></li>
+<li><a href="#how-to-get-involved">How to Get Involved</a></li>
+</ul>
+</div>
+ </aside>
+ </div>
+ </div>
+</div>
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/2026/03/20/writing-table-providers/index.html b/blog/2026/03/20/writing-table-providers/index.html
new file mode 100644
index 0000000..0f5312f
--- /dev/null
+++ b/blog/2026/03/20/writing-table-providers/index.html
@@ -0,0 +1,743 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>"Writing Custom Table Providers in Apache DataFusion" - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
+ </div>
+ </div>
+</nav>
+<!-- article contents -->
+<div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>
+ "Writing Custom Table Providers in Apache DataFusion"
+ </h1>
+ <p>Posted on: Fri 20 March 2026 by Tim Saucer</p>
+
+ <aside class="toc-container d-md-none mb-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#the-three-layers">The Three Layers</a></li>
+<li><a href="#layer-1-tableprovider">Layer 1: TableProvider</a><ul>
+<li><a href="#keep-scan-lightweight">Keep scan() Lightweight</a></li>
+<li><a href="#existing-implementations-to-learn-from">Existing Implementations to Learn From</a></li>
+</ul>
+</li>
+<li><a href="#layer-2-executionplan">Layer 2: ExecutionPlan</a><ul>
+<li><a href="#partitioning-strategies">Partitioning Strategies</a></li>
+<li><a href="#keep-execute-lightweight-too">Keep execute() Lightweight Too</a></li>
+<li><a href="#existing-implementations-to-learn-from_1">Existing Implementations to Learn From</a></li>
+</ul>
+</li>
+<li><a href="#layer-3-sendablerecordbatchstream">Layer 3: SendableRecordBatchStream</a><ul>
+<li><a href="#using-recordbatchstreamadapter">Using RecordBatchStreamAdapter</a></li>
+<li><a href="#cpu-intensive-work-use-a-separate-thread-pool">CPU-Intensive Work: Use a Separate Thread Pool</a></li>
+</ul>
+</li>
+<li><a href="#where-should-the-work-happen">Where Should the Work Happen?</a><ul>
+<li><a href="#why-this-matters">Why This Matters</a></li>
+</ul>
+</li>
+<li><a href="#filter-pushdown-doing-less-work">Filter Pushdown: Doing Less Work</a><ul>
+<li><a href="#how-filter-pushdown-works">How Filter Pushdown Works</a></li>
+<li><a href="#why-filter-pushdown-matters">Why Filter Pushdown Matters</a></li>
+<li><a href="#using-explain-to-debug-your-table-provider">Using EXPLAIN to Debug Your Table Provider</a></li>
+</ul>
+</li>
+<li><a href="#putting-it-all-together">Putting It All Together</a></li>
+<li><a href="#choosing-the-right-starting-point">Choosing the Right Starting Point</a></li>
+<li><a href="#further-reading">Further Reading</a></li>
+</ul>
+</div>
+ </aside>
+
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you need to
+understand and explains where your work should actually happen.</p>
+<h2 id="the-three-layers">The Three Layers<a class="headerlink" href="#the-three-layers" title="Permanent link">¶</a></h2>
+<hr/>
+<p>When DataFusion executes a query against a table, three abstractions collaborate
+to produce results:</p>
+<ol>
+<li><strong>[<code>TableProvider</code>]</strong> -- Describes the table (schema, capabilities) and
+ produces an execution plan when queried.</li>
+<li><strong>[<code>ExecutionPlan</code>]</strong> -- Describes <em>how</em> to compute the result: partitioning,
+ ordering, and child plan relationships.</li>
+<li><strong>[<code>SendableRecordBatchStream</code>]</strong> -- The async stream that <em>actually does the
+ work</em>, yielding <code>RecordBatch</code>es one at a time.</li>
+</ol>
+<p>Think of these as a funnel: <code>TableProvider::scan()</code> is called once during
+planning to create an <code>ExecutionPlan</code>, then <code>ExecutionPlan::execute()</code> is called
+once per partition to create a stream, and those streams are where rows are
+actually produced during execution.</p>
+<h2 id="layer-1-tableprovider">Layer 1: TableProvider<a class="headerlink" href="#layer-1-tableprovider" title="Permanent link">¶</a></h2>
+<hr/>
+<p>A [<code>TableProvider</code>] represents a queryable data source. For a minimal read-only
+table, you need four methods:</p>
+<pre><code class="language-rust">impl TableProvider for MyTable {
+ fn as_any(&self) -> &dyn Any { self }
+
+ fn schema(&self) -> SchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ filters: &[Expr],
+ limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ // Build and return an ExecutionPlan -- keep this lightweight!
+ Ok(Arc::new(MyExecPlan::new(
+ Arc::clone(&self.schema),
+ projection,
+ limit,
+ )))
+ }
+}
+</code></pre>
+<p>The <code>scan</code> method is the heart of <code>TableProvider</code>. It receives three pushdown
+hints from the optimizer, each reducing the amount of data your source needs
+to produce:</p>
+<ul>
+<li><strong><code>projection</code></strong> -- Which columns are needed. This reduces the <strong>width</strong> of
+ the output. If your source supports it, read only these columns rather than
+ the full schema.</li>
+<li><strong><code>filters</code></strong> -- Predicates the engine would like you to apply during the
+ scan. This reduces the <strong>number of rows</strong> by skipping data that does not
+ match. Implement <code>supports_filters_pushdown</code> to advertise which filters you
+ can handle.</li>
+<li><strong><code>limit</code></strong> -- A row count cap. This also reduces the <strong>number of rows</strong> --
+ if you can stop reading early once you have produced enough rows, this avoids
+ unnecessary work.</li>
+</ul>
+<h3 id="keep-scan-lightweight">Keep <code>scan()</code> Lightweight<a class="headerlink" href="#keep-scan-lightweight" title="Permanent link">¶</a></h3>
+<p>This is a critical point: <strong><code>scan()</code> runs during planning, not execution.</strong> It
+should return quickly. Best practices are to avoid performing I/O, network
+calls, or heavy computation here. The <code>scan</code> method's job is to <em>describe</em> how
+the data will be produced, not to produce it. All the real work belongs in the
+stream (Layer 3).</p>
+<p>A common pitfall is to fetch data or open connections in <code>scan()</code>. This blocks
+the planning thread and can cause timeouts or deadlocks, especially if the query
+involves multiple tables or subqueries that all need to be planned before
+execution begins.</p>
+<h3 id="existing-implementations-to-learn-from">Existing Implementations to Learn From<a class="headerlink" href="#existing-implementations-to-learn-from" title="Permanent link">¶</a></h3>
+<p>DataFusion ships several <code>TableProvider</code> implementations that are excellent
+references:</p>
+<ul>
+<li><strong>[<code>MemTable</code>]</strong> -- Holds data in memory as <code>Vec<RecordBatch></code>. The simplest
+ possible provider; great for tests and small datasets.</li>
+<li><strong>[<code>StreamTable</code>]</strong> -- Wraps a user-provided stream factory. Useful when your
+ data arrives as a continuous stream (e.g., from Kafka or a socket).</li>
+<li><strong>[<code>SortedTableProvider</code>]</strong> -- Wraps another <code>TableProvider</code> and advertises a
+ known sort order, enabling the optimizer to skip redundant sorts.</li>
+</ul>
+<h2 id="layer-2-executionplan">Layer 2: ExecutionPlan<a class="headerlink" href="#layer-2-executionplan" title="Permanent link">¶</a></h2>
+<hr/>
+<p>An [<code>ExecutionPlan</code>] is a node in the physical query plan tree. Your table
+provider's <code>scan()</code> method returns one. The required methods are:</p>
+<pre><code class="language-rust">impl ExecutionPlan for MyExecPlan {
+ fn name(&self) -> &str { "MyExecPlan" }
+
+ fn as_any(&self) -> &dyn Any { self }
+
+ fn properties(&self) -> &PlanProperties {
+ &self.properties
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
+ vec![] // Leaf node -- no children
+ }
+
+ fn with_new_children(
+ self: Arc<Self>,
+ children: Vec<Arc<dyn ExecutionPlan>>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ assert!(children.is_empty());
+ Ok(self)
+ }
+
+ fn execute(
+ &self,
+ partition: usize,
+ context: Arc<TaskContext>,
+ ) -> Result<SendableRecordBatchStream> {
+ // This is where you build and return your stream
+ // ...
+ }
+}
+</code></pre>
+<p>The key properties to set correctly in [<code>PlanProperties</code>] are <strong>output
+partitioning</strong> and <strong>output ordering</strong>.</p>
+<p><strong>Output partitioning</strong> tells the engine how many partitions your data has,
+which determines parallelism. If your source naturally partitions data (e.g.,
+by file or by shard), expose that here.</p>
+<p><strong>Output ordering</strong> declares whether your data is naturally sorted. This
+enables the optimizer to avoid inserting a <code>SortExec</code> when a query requires
+ordered data. Getting this right can be a significant performance win.</p>
+<h3 id="partitioning-strategies">Partitioning Strategies<a class="headerlink" href="#partitioning-strategies" title="Permanent link">¶</a></h3>
+<p>Since <code>execute()</code> is called once per partition, partitioning directly controls
+the parallelism of your table scan. Each partition runs on its own task, so
+more partitions means more concurrent work -- up to the number of available
+cores.</p>
+<p>Consider how your data source naturally divides its data:</p>
+<ul>
+<li><strong>By file or object:</strong> If you are reading from S3, each file can be a
+ partition. DataFusion will read them in parallel.</li>
+<li><strong>By shard or region:</strong> If your source is a sharded database, each shard
+ maps naturally to a partition.</li>
+<li><strong>By key range:</strong> If your data is keyed (e.g., by timestamp or customer ID),
+ you can split it into ranges.</li>
+</ul>
+<p>Getting partitioning right matters because it affects everything downstream in
+the plan. When DataFusion needs to perform an aggregation or join, it
+repartitions data by hashing the relevant columns. If your source already
+produces data partitioned by the join or group-by key, DataFusion can skip the
+repartition step entirely -- avoiding a potentially expensive shuffle.</p>
+<p>For example, if you are building a table provider for a system that stores
+data partitioned by <code>customer_id</code>, and a common query groups by <code>customer_id</code>:</p>
+<pre><code class="language-sql">SELECT customer_id, SUM(amount)
+FROM my_table
+GROUP BY customer_id;
+</code></pre>
+<p>If you declare your output partitioning as <code>Hash([customer_id], N)</code>, the
+optimizer recognizes that the data is already distributed correctly for the
+aggregation and eliminates the <code>RepartitionExec</code> that would otherwise appear
+in the plan. You can verify this with <code>EXPLAIN</code> (more on this below).</p>
+<p>Conversely, if you report <code>UnknownPartitioning</code>, DataFusion must assume the
+worst case and will always insert repartitioning operators as needed.</p>
+<h3 id="keep-execute-lightweight-too">Keep <code>execute()</code> Lightweight Too<a class="headerlink" href="#keep-execute-lightweight-too" title="Permanent link">¶</a></h3>
+<p>Like <code>scan()</code>, the <code>execute()</code> method should construct and return a stream
+without doing heavy work. The actual data production happens when the stream
+is polled. Do not block on async operations here -- build the stream and let
+the runtime drive it.</p>
+<h3 id="existing-implementations-to-learn-from_1">Existing Implementations to Learn From<a class="headerlink" href="#existing-implementations-to-learn-from_1" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong>[<code>StreamingTableExec</code>]</strong> -- Executes a streaming table scan. It takes a
+ stream factory (a closure that produces streams) and handles partitioning.
+ Good reference for wrapping external streams.</li>
+<li><strong>[<code>DataSourceExec</code>]</strong> -- The execution plan behind DataFusion's built-in file
+ scanning (Parquet, CSV, JSON). It demonstrates sophisticated partitioning,
+ filter pushdown, and projection pushdown.</li>
+</ul>
+<h2 id="layer-3-sendablerecordbatchstream">Layer 3: SendableRecordBatchStream<a class="headerlink" href="#layer-3-sendablerecordbatchstream" title="Permanent link">¶</a></h2>
+<hr/>
+<p>[<code>SendableRecordBatchStream</code>] is where the real work happens. It is defined as:</p>
+<pre><code class="language-rust">type SendableRecordBatchStream =
+ Pin<Box<dyn RecordBatchStream<Item = Result<RecordBatch>> + Send>>;
+</code></pre>
+<p>This is an async stream of <code>RecordBatch</code>es that can be sent across threads. When
+the DataFusion runtime polls this stream, your code runs: reading files, calling
+APIs, transforming data, etc.</p>
+<h3 id="using-recordbatchstreamadapter">Using RecordBatchStreamAdapter<a class="headerlink" href="#using-recordbatchstreamadapter" title="Permanent link">¶</a></h3>
+<p>The easiest way to create a <code>SendableRecordBatchStream</code> is with
+[<code>RecordBatchStreamAdapter</code>]. It bridges any <code>futures::Stream<Item =
+Result<RecordBatch>></code> into the <code>SendableRecordBatchStream</code> type:</p>
+<pre><code class="language-rust">use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+
+fn execute(
+ &self,
+ partition: usize,
+ context: Arc<TaskContext>,
+) -> Result<SendableRecordBatchStream> {
+ let schema = self.schema();
+ let config = self.config.clone();
+
+ let stream = futures::stream::once(async move {
+ // ALL the heavy work happens here, inside the stream:
+ // - Open connections
+ // - Read data from external sources
+ // - Transform and batch the results
+ let batches = fetch_data_from_source(&config).await?;
+ Ok(batches)
+ })
+ .flat_map(|result| match result {
+ Ok(batch) => futures::stream::iter(vec![Ok(batch)]),
+ Err(e) => futures::stream::iter(vec![Err(e)]),
+ });
+
+ Ok(Box::pin(RecordBatchStreamAdapter::new(schema, stream)))
+}
+</code></pre>
+<h3 id="cpu-intensive-work-use-a-separate-thread-pool">CPU-Intensive Work: Use a Separate Thread Pool<a class="headerlink" href="#cpu-intensive-work-use-a-separate-thread-pool" title="Permanent link">¶</a></h3>
+<p>If your stream performs CPU-intensive work (parsing, decompression, complex
+transformations), avoid blocking the tokio runtime. Instead, offload to a
+dedicated thread pool and send results back through a channel:</p>
+<pre><code class="language-rust">fn execute(
+ &self,
+ partition: usize,
+ context: Arc<TaskContext>,
+) -> Result<SendableRecordBatchStream> {
+ let schema = self.schema();
+ let config = self.config.clone();
+
+ let (tx, rx) = tokio::sync::mpsc::channel(2);
+
+ // Spawn CPU-heavy work on a blocking thread pool
+ tokio::task::spawn_blocking(move || {
+ let batches = generate_data(&config);
+ for batch in batches {
+ if tx.blocking_send(Ok(batch)).is_err() {
+ break; // Receiver dropped, query was cancelled
+ }
+ }
+ });
+
+ let stream = tokio_stream::wrappers::ReceiverStream::new(rx);
+ Ok(Box::pin(RecordBatchStreamAdapter::new(schema, stream)))
+}
+</code></pre>
+<p>This pattern keeps the async runtime responsive while your data generation
+runs on its own threads.</p>
+<h2 id="where-should-the-work-happen">Where Should the Work Happen?<a class="headerlink" href="#where-should-the-work-happen" title="Permanent link">¶</a></h2>
+<hr/>
+<p>This table summarizes what belongs at each layer:</p>
+<table class="table">
+<thead>
+<tr>
+<th>Layer</th>
+<th>Runs During</th>
+<th>Should Do</th>
+<th>Should NOT Do</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>TableProvider::scan()</code></td>
+<td>Planning</td>
+<td>Build an <code>ExecutionPlan</code> with metadata</td>
+<td>I/O, network calls, heavy computation</td>
+</tr>
+<tr>
+<td><code>ExecutionPlan::execute()</code></td>
+<td>Execution (once per partition)</td>
+<td>Construct a stream, set up channels</td>
+<td>Block on async work, read data</td>
+</tr>
+<tr>
+<td><code>RecordBatchStream</code> (polling)</td>
+<td>Execution</td>
+<td>All I/O, computation, data production</td>
+<td>--</td>
+</tr>
+</tbody>
+</table>
+<p>The guiding principle: <strong>push work as late as possible.</strong> Planning should be
+fast so the optimizer can do its job. Execution setup should be fast so all
+partitions can start promptly. The stream is where you spend time producing
+data.</p>
+<h3 id="why-this-matters">Why This Matters<a class="headerlink" href="#why-this-matters" title="Permanent link">¶</a></h3>
+<p>When <code>scan()</code> does heavy work, several problems arise:</p>
+<ol>
+<li><strong>Planning becomes slow.</strong> If a query touches 10 tables and each <code>scan()</code>
+ takes 500ms, planning alone takes 5 seconds before any data flows.</li>
+<li><strong>The optimizer cannot help.</strong> The optimizer runs between planning and
+ execution. If you have already fetched data during planning, optimizations
+ like predicate pushdown or partition pruning cannot reduce the work.</li>
+<li><strong>Resource management breaks down.</strong> DataFusion manages concurrency and
+ memory during execution. Work done during planning bypasses these controls.</li>
+</ol>
+<h2 id="filter-pushdown-doing-less-work">Filter Pushdown: Doing Less Work<a class="headerlink" href="#filter-pushdown-doing-less-work" title="Permanent link">¶</a></h2>
+<hr/>
+<p>One of the most impactful optimizations you can add to a custom table provider
+is <strong>filter pushdown</strong> -- letting the source skip data that the query does not
+need, rather than reading everything and filtering it afterward.</p>
+<h3 id="how-filter-pushdown-works">How Filter Pushdown Works<a class="headerlink" href="#how-filter-pushdown-works" title="Permanent link">¶</a></h3>
+<p>When DataFusion plans a query with a <code>WHERE</code> clause, it passes the filter
+predicates to your <code>scan()</code> method as the <code>filters</code> parameter. By default,
+DataFusion assumes your provider cannot handle any filters and inserts a
+<code>FilterExec</code> node above your scan to apply them. But if your source <em>can</em>
+evaluate some predicates during scanning -- for example, by skipping files,
+partitions, or row groups that cannot match -- you can eliminate a huge amount
+of unnecessary I/O.</p>
+<p>To opt in, implement <code>supports_filters_pushdown</code>:</p>
+<pre><code class="language-rust">fn supports_filters_pushdown(
+ &self,
+ filters: &[&Expr],
+) -> Result<Vec<TableProviderFilterPushDown>> {
+ Ok(filters.iter().map(|f| {
+ match f {
+ // We can fully evaluate equality filters on
+ // the partition column at the source
+ Expr::BinaryExpr(BinaryExpr {
+ left, op: Operator::Eq, right
+ }) if is_partition_column(left) || is_partition_column(right) => {
+ TableProviderFilterPushDown::Exact
+ }
+ // All other filters: let DataFusion handle them
+ _ => TableProviderFilterPushDown::Unsupported,
+ }
+ }).collect())
+}
+</code></pre>
+<p>The three possible responses for each filter are:</p>
+<ul>
+<li><strong><code>Exact</code></strong> -- Your source guarantees that no output rows will have a false
+ value for this predicate. Because the filter is fully evaluated at the source,
+ DataFusion will <strong>not</strong> add a <code>FilterExec</code> for it.</li>
+<li><strong><code>Inexact</code></strong> -- Your source has the ability to reduce the data produced, but
+ the output may still include rows that do not satisfy the predicate. For
+ example, you might skip entire files based on metadata statistics but not
+ filter individual rows within a file. DataFusion will still add a <code>FilterExec</code>
+ above your scan to remove any remaining rows that slipped through.</li>
+<li><strong><code>Unsupported</code></strong> -- Your source ignores this filter entirely. DataFusion
+ handles it.</li>
+</ul>
+<h3 id="why-filter-pushdown-matters">Why Filter Pushdown Matters<a class="headerlink" href="#why-filter-pushdown-matters" title="Permanent link">¶</a></h3>
+<p>Consider a table with 1 billion rows partitioned by <code>region</code>, and a query:</p>
+<pre><code class="language-sql">SELECT * FROM events WHERE region = 'us-east-1' AND event_type = 'click';
+</code></pre>
+<p><strong>Without filter pushdown:</strong> Your table provider reads all 1 billion rows
+across all regions. DataFusion then applies both filters, discarding the vast
+majority of the data.</p>
+<p><strong>With filter pushdown on <code>region</code>:</strong> Your <code>scan()</code> method sees the
+<code>region = 'us-east-1'</code> filter and constructs an execution plan that only reads
+the <code>us-east-1</code> partition. If that partition holds 100 million rows, you have
+just eliminated 90% of the I/O. DataFusion still applies the <code>event_type</code>
+filter via <code>FilterExec</code> if you reported it as <code>Unsupported</code>.</p>
+<h3 id="using-explain-to-debug-your-table-provider">Using EXPLAIN to Debug Your Table Provider<a class="headerlink" href="#using-explain-to-debug-your-table-provider" title="Permanent link">¶</a></h3>
+<p>The <code>EXPLAIN</code> statement is your best tool for understanding what DataFusion is
+actually doing with your table provider. It shows the physical plan that
+DataFusion will execute, including any operators it inserted:</p>
+<pre><code class="language-sql">EXPLAIN SELECT * FROM events WHERE region = 'us-east-1' AND event_type = 'click';
+</code></pre>
+<p>If you are using DataFrames, call <code>.explain(false, false)</code> for the logical plan
+or <code>.explain(false, true)</code> for the physical plan. You can also print the plans
+in verbose mode with <code>.explain(true, true)</code>.</p>
+<p><strong>Before filter pushdown</strong>, the plan might look like:</p>
+<pre><code class="language-text">FilterExec: region@0 = us-east-1 AND event_type@1 = click
+ MyExecPlan: partitions=50
+</code></pre>
+<p>Here DataFusion is reading all 50 partitions and filtering everything
+afterward. The <code>FilterExec</code> above your scan is doing all the predicate work.</p>
+<p><strong>After implementing pushdown for <code>region</code></strong> (reported as <code>Exact</code>):</p>
+<pre><code class="language-text">FilterExec: event_type@1 = click
+ MyExecPlan: partitions=5, filter=[region = us-east-1]
+</code></pre>
+<p>Now your exec reads only the 5 partitions for <code>us-east-1</code>, and the remaining
+<code>FilterExec</code> only handles the <code>event_type</code> predicate. The <code>region</code> filter has
+been fully absorbed by your scan.</p>
+<p><strong>After implementing pushdown for both filters</strong> (both <code>Exact</code>):</p>
+<pre><code class="language-text">MyExecPlan: partitions=5, filter=[region = us-east-1 AND event_type = click]
+</code></pre>
+<p>No <code>FilterExec</code> at all -- your source handles everything.</p>
+<p>Similarly, <code>EXPLAIN</code> will reveal whether DataFusion is inserting unnecessary
+<code>SortExec</code> or <code>RepartitionExec</code> nodes that you could eliminate by declaring
+better output properties. Whenever your queries seem slower than expected,
+<code>EXPLAIN</code> is the first place to look.</p>
+<h2 id="putting-it-all-together">Putting It All Together<a class="headerlink" href="#putting-it-all-together" title="Permanent link">¶</a></h2>
+<hr/>
+<p>Here is a minimal but complete example of a custom table provider that generates
+data lazily during streaming:</p>
+<pre><code class="language-rust">use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Int64Array, StringArray};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use arrow::record_batch::RecordBatch;
+use datafusion::catalog::TableProvider;
+use datafusion::common::Result;
+use datafusion::datasource::TableType;
+use datafusion::execution::context::SessionState;
+use datafusion::execution::SendableRecordBatchStream;
+use datafusion::logical_expr::Expr;
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{
+ ExecutionPlan, Partitioning, PlanProperties,
+};
+use futures::stream;
+
+/// A table provider that generates sequential numbers on demand.
+struct CountingTable {
+ schema: SchemaRef,
+ num_partitions: usize,
+ rows_per_partition: usize,
+}
+
+impl CountingTable {
+ fn new(num_partitions: usize, rows_per_partition: usize) -> Self {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("partition", DataType::Int64, false),
+ Field::new("value", DataType::Int64, false),
+ ]));
+ Self { schema, num_partitions, rows_per_partition }
+ }
+}
+
+#[async_trait::async_trait]
+impl TableProvider for CountingTable {
+ fn as_any(&self) -> &dyn Any { self }
+ fn schema(&self) -> SchemaRef { Arc::clone(&self.schema) }
+ fn table_type(&self) -> TableType { TableType::Base }
+
+ async fn scan(
+ &self,
+ _state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ // Light work only: build the plan with metadata
+ Ok(Arc::new(CountingExec {
+ schema: Arc::clone(&self.schema),
+ num_partitions: self.num_partitions,
+ rows_per_partition: limit
+ .unwrap_or(self.rows_per_partition)
+ .min(self.rows_per_partition),
+ properties: PlanProperties::new(
+ EquivalenceProperties::new(Arc::clone(&self.schema)),
+ Partitioning::UnknownPartitioning(self.num_partitions),
+ EmissionType::Incremental,
+ Boundedness::Bounded,
+ ),
+ }))
+ }
+}
+
+struct CountingExec {
+ schema: SchemaRef,
+ num_partitions: usize,
+ rows_per_partition: usize,
+ properties: PlanProperties,
+}
+
+impl ExecutionPlan for CountingExec {
+ fn name(&self) -> &str { "CountingExec" }
+ fn as_any(&self) -> &dyn Any { self }
+ fn properties(&self) -> &PlanProperties { &self.properties }
+ fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> { vec![] }
+
+ fn with_new_children(
+ self: Arc<Self>,
+ _children: Vec<Arc<dyn ExecutionPlan>>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(self)
+ }
+
+ fn execute(
+ &self,
+ partition: usize,
+ _context: Arc<TaskContext>,
+ ) -> Result<SendableRecordBatchStream> {
+ let schema = Arc::clone(&self.schema);
+ let rows = self.rows_per_partition;
+
+ // The heavy work (data generation) happens inside the stream,
+ // not here in execute().
+ let batch_stream = stream::once(async move {
+ let partitions = Int64Array::from(
+ vec![partition as i64; rows],
+ );
+ let values = Int64Array::from(
+ (0..rows as i64).collect::<Vec<_>>(),
+ );
+ let batch = RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![Arc::new(partitions), Arc::new(values)],
+ )?;
+ Ok(batch)
+ });
+
+ Ok(Box::pin(RecordBatchStreamAdapter::new(
+ Arc::clone(&self.schema),
+ batch_stream,
+ )))
+ }
+}
+</code></pre>
+<h2 id="choosing-the-right-starting-point">Choosing the Right Starting Point<a class="headerlink" href="#choosing-the-right-starting-point" title="Permanent link">¶</a></h2>
+<hr/>
+<p>Not every custom data source requires implementing all three layers from
+scratch. DataFusion provides building blocks that let you plug in at whatever
+level makes sense:</p>
+<table class="table">
+<thead>
+<tr>
+<th>If your data is...</th>
+<th>Start with</th>
+<th>You implement</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Already in <code>RecordBatch</code>es in memory</td>
+<td>[<code>MemTable</code>]</td>
+<td>Nothing -- just construct it</td>
+</tr>
+<tr>
+<td>An async stream of batches</td>
+<td>[<code>StreamTable</code>]</td>
+<td>A stream factory</td>
+</tr>
+<tr>
+<td>A table with known sort order</td>
+<td>[<code>SortedTableProvider</code>] wrapping another provider</td>
+<td>The inner provider</td>
+</tr>
+<tr>
+<td>A custom source needing full control</td>
+<td><code>TableProvider</code> + <code>ExecutionPlan</code> + stream</td>
+<td>All three layers</td>
+</tr>
+</tbody>
+</table>
+<p>For most integrations, [<code>StreamTable</code>] combined with
+[<code>RecordBatchStreamAdapter</code>] provides a good balance of simplicity and
+flexibility. You provide a closure that returns a stream, and DataFusion handles
+the rest.</p>
+<h2 id="further-reading">Further Reading<a class="headerlink" href="#further-reading" title="Permanent link">¶</a></h2>
+<hr/>
+<ul>
+<li>[TableProvider API docs][<code>TableProvider</code>]</li>
+<li>[ExecutionPlan API docs][<code>ExecutionPlan</code>]</li>
+<li>[SendableRecordBatchStream API docs][<code>SendableRecordBatchStream</code>]</li>
+<li><a href="https://github.com/apache/datafusion/issues/16821">GitHub issue discussing table provider examples</a></li>
+<li><a href="https://github.com/apache/datafusion/tree/main/datafusion-examples/examples">DataFusion examples directory</a> --
+ contains working examples including custom table providers</li>
+</ul>
+<hr/>
+<p><em>Note: Portions of this blog post were written with the assistance of an AI agent.</em></p>
+
+<!--
+ Comments Section
+ Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div id="comments">
+ <hr>
+ <h3>Comments</h3>
+
+ <!-- Local loader script -->
+ <script src="/content/js/giscus-consent.js" defer></script>
+
+ <!-- Consent UI -->
+ <div id="giscus-consent">
+ <p>
+ We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
+ To respect your privacy, Giscus and comments will load only if you click "Show Comments"
+ </p>
+
+ <div class="consent-actions">
+ <button id="giscus-load" type="button">Show Comments</button>
+ <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+ </div>
+
+ <noscript>JavaScript is required to load comments from Giscus.</noscript>
+ </div>
+
+ <!-- Container where Giscus will render -->
+ <div id="comment-thread"></div>
+</div> </div>
+ <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#the-three-layers">The Three Layers</a></li>
+<li><a href="#layer-1-tableprovider">Layer 1: TableProvider</a><ul>
+<li><a href="#keep-scan-lightweight">Keep scan() Lightweight</a></li>
+<li><a href="#existing-implementations-to-learn-from">Existing Implementations to Learn From</a></li>
+</ul>
+</li>
+<li><a href="#layer-2-executionplan">Layer 2: ExecutionPlan</a><ul>
+<li><a href="#partitioning-strategies">Partitioning Strategies</a></li>
+<li><a href="#keep-execute-lightweight-too">Keep execute() Lightweight Too</a></li>
+<li><a href="#existing-implementations-to-learn-from_1">Existing Implementations to Learn From</a></li>
+</ul>
+</li>
+<li><a href="#layer-3-sendablerecordbatchstream">Layer 3: SendableRecordBatchStream</a><ul>
+<li><a href="#using-recordbatchstreamadapter">Using RecordBatchStreamAdapter</a></li>
+<li><a href="#cpu-intensive-work-use-a-separate-thread-pool">CPU-Intensive Work: Use a Separate Thread Pool</a></li>
+</ul>
+</li>
+<li><a href="#where-should-the-work-happen">Where Should the Work Happen?</a><ul>
+<li><a href="#why-this-matters">Why This Matters</a></li>
+</ul>
+</li>
+<li><a href="#filter-pushdown-doing-less-work">Filter Pushdown: Doing Less Work</a><ul>
+<li><a href="#how-filter-pushdown-works">How Filter Pushdown Works</a></li>
+<li><a href="#why-filter-pushdown-matters">Why Filter Pushdown Matters</a></li>
+<li><a href="#using-explain-to-debug-your-table-provider">Using EXPLAIN to Debug Your Table Provider</a></li>
+</ul>
+</li>
+<li><a href="#putting-it-all-together">Putting It All Together</a></li>
+<li><a href="#choosing-the-right-starting-point">Choosing the Right Starting Point</a></li>
+<li><a href="#further-reading">Further Reading</a></li>
+</ul>
+</div>
+ </aside>
+ </div>
+ </div>
+</div>
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/about.html b/blog/about.html
index 5d9b82e..91122d6 100644
--- a/blog/about.html
+++ b/blog/about.html
@@ -1,34 +1,66 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
- <title>Apache DataFusion Blog - About</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
- <link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>About - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
-
-</head>
-
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li class="active"><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
- <h1>About</h1>
-
-
- <p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is a very fast, extensible query engine for building high-quality data-centric systems in Rust, using the Apache Arrow in-memory format.</p>
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
+ </div>
+ </div>
+</nav>
+ <div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>About</h1>
+ <p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is a very fast, extensible query engine for building high-quality data-centric systems in Rust, using the Apache Arrow in-memory format.</p>
<p>To update this site, please propose a change to the <a href="https://github.com/apache/datafusion-site">source code</a>.</p>
-
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
-</html>
\ No newline at end of file
+ </div>
+ </div>
+ </div>
+ </div>
+
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/author/pepijn-van-eeckhoudt.html b/blog/author/pepijn-van-eeckhoudt.html
index a85b6e3..25789ed 100644
--- a/blog/author/pepijn-van-eeckhoudt.html
+++ b/blog/author/pepijn-van-eeckhoudt.html
@@ -20,6 +20,56 @@
<h2>Articles by Pepijn Van Eeckhoudt</h2>
<ol id="post-list">
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="bookmark" title="Permalink to Optimizing SQL CASE Expression Evaluation">Optimizing SQL CASE Expression Evaluation</a></h2> </header>
+ <footer class="post-info">
+ <time class="published" datetime="2026-02-02T00:00:00+00:00"> Mon 02 February 2026 </time>
+ <address class="vcard author">By
+ <a class="url fn" href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a …</p> </div><!-- /.entry-content -->
+ </article></li>
<li><article class="hentry">
<header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="bookmark" title="Permalink to Using Rust async for Query Execution and Cancelling Long-Running Queries">Using Rust async for Query Execution and Cancelling Long-Running Queries</a></h2> </header>
<footer class="post-info">
diff --git a/blog/author/pmc.html b/blog/author/pmc.html
index 412bda6..eab42dc 100644
--- a/blog/author/pmc.html
+++ b/blog/author/pmc.html
@@ -21,9 +21,41 @@
<ol id="post-list">
<li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/08/datafusion-52.0.0" rel="bookmark" title="Permalink to Apache DataFusion 52.0.0 Released">Apache DataFusion 52.0.0 Released</a></h2> </header>
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.13.0 Release">Apache DataFusion Comet 0.13.0 Release</a></h2> </header>
<footer class="post-info">
- <time class="published" datetime="2026-01-08T00:00:00+00:00"> Thu 08 January 2026 </time>
+ <time class="published" datetime="2026-01-30T00:00:00+00:00"> Fri 30 January 2026 </time>
+ <address class="vcard author">By
+ <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development …</p> </div><!-- /.entry-content -->
+ </article></li>
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="bookmark" title="Permalink to Apache DataFusion 52.0.0 Released">Apache DataFusion 52.0.0 Released</a></h2> </header>
+ <footer class="post-info">
+ <time class="published" datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
<address class="vcard author">By
<a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
</address>
diff --git a/blog/author/pepijn-van-eeckhoudt.html b/blog/author/tim-saucer.html
similarity index 69%
copy from blog/author/pepijn-van-eeckhoudt.html
copy to blog/author/tim-saucer.html
index a85b6e3..c9faea3 100644
--- a/blog/author/pepijn-van-eeckhoudt.html
+++ b/blog/author/tim-saucer.html
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<html lang="en">
<head>
- <title>Apache DataFusion Blog - Articles by Pepijn Van Eeckhoudt</title>
+ <title>Apache DataFusion Blog - Articles by Tim Saucer</title>
<meta charset="utf-8" />
<meta name="generator" content="Pelican" />
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
@@ -17,15 +17,15 @@
<li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
</ul></nav><!-- /#menu -->
<section id="content">
-<h2>Articles by Pepijn Van Eeckhoudt</h2>
+<h2>Articles by Tim Saucer</h2>
<ol id="post-list">
<li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="bookmark" title="Permalink to Using Rust async for Query Execution and Cancelling Long-Running Queries">Using Rust async for Query Execution and Cancelling Long-Running Queries</a></h2> </header>
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/20/writing-table-providers" rel="bookmark" title="Permalink to "Writing Custom Table Providers in Apache DataFusion"">"Writing Custom Table Providers in Apache DataFusion"</a></h2> </header>
<footer class="post-info">
- <time class="published" datetime="2025-06-30T00:00:00+00:00"> Mon 30 June 2025 </time>
+ <time class="published" datetime="2026-03-20T00:00:00+00:00"> Fri 20 March 2026 </time>
<address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
+ <a class="url fn" href="https://datafusion.apache.org/blog/author/tim-saucer.html">Tim Saucer</a>
</address>
</footer><!-- /.post-info -->
<div class="entry-content"> <!--
@@ -47,28 +47,10 @@ limitations under the License.
{% endcomment %}
-->
-<style>
-figure {
- margin: 20px 0;
-}
-
-figure img {
- display: block;
- max-width: 80%;
- margin: auto;
-}
-
-figcaption {
- font-style: italic;
- color: #555;
- font-size: 0.9em;
- max-width: 80%;
- margin: auto;
- text-align: center;
-}
-</style>
-<p>Have you ever tried to cancel a query that just wouldn't stop?
-In this post, we'll review how Rust's <a href="https://doc.rust-lang.org/book/ch17-00-async-await.html"><code>async</code> programming model</a> works, how …</p> </div><!-- /.entry-content -->
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you …</p> </div><!-- /.entry-content -->
</article></li>
</ol><!-- /#posts-list -->
</section><!-- /#content -->
diff --git a/blog/author/pepijn-van-eeckhoudt.html b/blog/author/xudong.html
similarity index 76%
copy from blog/author/pepijn-van-eeckhoudt.html
copy to blog/author/xudong.html
index a85b6e3..b64321c 100644
--- a/blog/author/pepijn-van-eeckhoudt.html
+++ b/blog/author/xudong.html
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<html lang="en">
<head>
- <title>Apache DataFusion Blog - Articles by Pepijn Van Eeckhoudt</title>
+ <title>Apache DataFusion Blog - Articles by xudong</title>
<meta charset="utf-8" />
<meta name="generator" content="Pelican" />
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
@@ -17,15 +17,15 @@
<li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
</ul></nav><!-- /#menu -->
<section id="content">
-<h2>Articles by Pepijn Van Eeckhoudt</h2>
+<h2>Articles by xudong</h2>
<ol id="post-list">
<li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="bookmark" title="Permalink to Using Rust async for Query Execution and Cancelling Long-Running Queries">Using Rust async for Query Execution and Cancelling Long-Running Queries</a></h2> </header>
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="bookmark" title="Permalink to Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack">Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</a></h2> </header>
<footer class="post-info">
- <time class="published" datetime="2025-06-30T00:00:00+00:00"> Mon 30 June 2025 </time>
+ <time class="published" datetime="2026-03-20T00:00:00+00:00"> Fri 20 March 2026 </time>
<address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
+ <a class="url fn" href="https://datafusion.apache.org/blog/author/xudong.html">xudong</a>
</address>
</footer><!-- /.post-info -->
<div class="entry-content"> <!--
@@ -67,8 +67,8 @@ figcaption {
text-align: center;
}
</style>
-<p>Have you ever tried to cancel a query that just wouldn't stop?
-In this post, we'll review how Rust's <a href="https://doc.rust-lang.org/book/ch17-00-async-await.html"><code>async</code> programming model</a> works, how …</p> </div><!-- /.entry-content -->
+<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p> </div><!-- /.entry-content -->
</article></li>
</ol><!-- /#posts-list -->
</section><!-- /#content -->
diff --git a/blog/category/blog.html b/blog/category/blog.html
index 969d39d..9c32175 100644
--- a/blog/category/blog.html
+++ b/blog/category/blog.html
@@ -22,11 +22,11 @@
<ol id="post-list">
<li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="bookmark" title="Permalink to Extending SQL in DataFusion: from ->> to TABLESAMPLE">Extending SQL in DataFusion: from ->> to TABLESAMPLE</a></h2> </header>
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="bookmark" title="Permalink to Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack">Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</a></h2> </header>
<footer class="post-info">
- <time class="published" datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
+ <time class="published" datetime="2026-03-20T00:00:00+00:00"> Fri 20 March 2026 </time>
<address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/geoffrey-claude-datadog.html">Geoffrey Claude (Datadog)</a>
+ <a class="url fn" href="https://datafusion.apache.org/blog/author/xudong.html">xudong</a>
</address>
</footer><!-- /.post-info -->
<div class="entry-content"> <!--
@@ -48,13 +48,147 @@ limitations under the License.
{% endcomment %}
-->
-<p>If you embed <a href="https://datafusion.apache.org/">DataFusion</a> in your product, your users will eventually run SQL that DataFusion does not recognize. Not because the query is unreasonable, but because SQL in practice includes many dialects and system-specific statements.</p>
-<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p> </div><!-- /.entry-content -->
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p> </div><!-- /.entry-content -->
+ </article></li>
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/20/writing-table-providers" rel="bookmark" title="Permalink to "Writing Custom Table Providers in Apache DataFusion"">"Writing Custom Table Providers in Apache DataFusion"</a></h2> </header>
+ <footer class="post-info">
+ <time class="published" datetime="2026-03-20T00:00:00+00:00"> Fri 20 March 2026 </time>
+ <address class="vcard author">By
+ <a class="url fn" href="https://datafusion.apache.org/blog/author/tim-saucer.html">Tim Saucer</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you …</p> </div><!-- /.entry-content -->
</article></li>
<li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/08/datafusion-52.0.0" rel="bookmark" title="Permalink to Apache DataFusion 52.0.0 Released">Apache DataFusion 52.0.0 Released</a></h2> </header>
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="bookmark" title="Permalink to Optimizing SQL CASE Expression Evaluation">Optimizing SQL CASE Expression Evaluation</a></h2> </header>
<footer class="post-info">
- <time class="published" datetime="2026-01-08T00:00:00+00:00"> Thu 08 January 2026 </time>
+ <time class="published" datetime="2026-02-02T00:00:00+00:00"> Mon 02 February 2026 </time>
+ <address class="vcard author">By
+ <a class="url fn" href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a …</p> </div><!-- /.entry-content -->
+ </article></li>
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.13.0 Release">Apache DataFusion Comet 0.13.0 Release</a></h2> </header>
+ <footer class="post-info">
+ <time class="published" datetime="2026-01-30T00:00:00+00:00"> Fri 30 January 2026 </time>
+ <address class="vcard author">By
+ <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development …</p> </div><!-- /.entry-content -->
+ </article></li>
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="bookmark" title="Permalink to Apache DataFusion 52.0.0 Released">Apache DataFusion 52.0.0 Released</a></h2> </header>
+ <footer class="post-info">
+ <time class="published" datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
<address class="vcard author">By
<a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
</address>
@@ -84,6 +218,36 @@ changes is available in the <a href="https://github.com/apache/datafusion/blob/b
making this release possible.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
<p>We continue to …</p> </div><!-- /.entry-content -->
+ </article></li>
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="bookmark" title="Permalink to Extending SQL in DataFusion: from ->> to TABLESAMPLE">Extending SQL in DataFusion: from ->> to TABLESAMPLE</a></h2> </header>
+ <footer class="post-info">
+ <time class="published" datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
+ <address class="vcard author">By
+ <a class="url fn" href="https://datafusion.apache.org/blog/author/geoffrey-claude-datadog.html">Geoffrey Claude (Datadog)</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a href="https://datafusion.apache.org/">DataFusion</a> in your product, your users will eventually run SQL that DataFusion does not recognize. Not because the query is unreasonable, but because SQL in practice includes many dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p> </div><!-- /.entry-content -->
</article></li>
<li><article class="hentry">
<header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="bookmark" title="Permalink to Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution">Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</a></h2> </header>
diff --git a/blog/css/bootstrap.min.css b/blog/css/bootstrap.min.css
index 3993414..1d8bac4 100644
--- a/blog/css/bootstrap.min.css
+++ b/blog/css/bootstrap.min.css
@@ -1,6 +1,6 @@
@charset "UTF-8";/*!
- * Bootstrap v5.3.3 (https://getbootstrap.com/)
- * Copyright 2011-2024 The Bootstrap Authors
+ * Bootstrap v5.3.8 (https://getbootstrap.com/)
+ * Copyright 2011-2025 The Bootstrap Authors
* Licensed under MIT (https://github.com/twbs/bootstrap/blob/main/LICENSE)
- */:root,[data-bs-theme=light]{--bs-blue:#0d6efd;--bs-indigo:#6610f2;--bs-purple:#6f42c1;--bs-pink:#d63384;--bs-red:#dc3545;--bs-orange:#fd7e14;--bs-yellow:#ffc107;--bs-green:#198754;--bs-teal:#20c997;--bs-cyan:#0dcaf0;--bs-black:#000;--bs-white:#fff;--bs-gray:#6c757d;--bs-gray-dark:#343a40;--bs-gray-100:#f8f9fa;--bs-gray-200:#e9ecef;--bs-gray-300:#dee2e6;--bs-gray-400:#ced4da;--bs-gray-500:#adb5bd;--bs-gray-600:#6c757d;--bs-gray-700:#495057;--bs-gray-800:#343a40;--bs-gray-900:#212529;-- [...]
+ */:root,[data-bs-theme=light]{--bs-blue:#0d6efd;--bs-indigo:#6610f2;--bs-purple:#6f42c1;--bs-pink:#d63384;--bs-red:#dc3545;--bs-orange:#fd7e14;--bs-yellow:#ffc107;--bs-green:#198754;--bs-teal:#20c997;--bs-cyan:#0dcaf0;--bs-black:#000;--bs-white:#fff;--bs-gray:#6c757d;--bs-gray-dark:#343a40;--bs-gray-100:#f8f9fa;--bs-gray-200:#e9ecef;--bs-gray-300:#dee2e6;--bs-gray-400:#ced4da;--bs-gray-500:#adb5bd;--bs-gray-600:#6c757d;--bs-gray-700:#495057;--bs-gray-800:#343a40;--bs-gray-900:#212529;-- [...]
/*# sourceMappingURL=bootstrap.min.css.map */
\ No newline at end of file
diff --git a/blog/css/dark-mode.css b/blog/css/dark-mode.css
new file mode 100644
index 0000000..95b2156
--- /dev/null
+++ b/blog/css/dark-mode.css
@@ -0,0 +1,100 @@
+/* Dark Mode Styles */
+:root {
+ --bg-primary: #ffffff;
+ --text-primary: #212529;
+ --text-secondary: #6c757d;
+ --navbar-bg: #343a40;
+ --code-bg: #f8f9fa;
+}
+
+[data-theme="dark"] {
+ --bg-primary: #0d1117;
+ --text-primary: #e6edf3;
+ --text-secondary: #8b949e;
+ --navbar-bg: #0d1117;
+ --code-bg: #161b22;
+}
+
+body {
+ background-color: var(--bg-primary) !important;
+ color: var(--text-primary) !important;
+}
+
+[data-theme="dark"] .bg-white {
+ background-color: var(--bg-primary) !important;
+ color: var(--text-primary) !important;
+}
+
+[data-theme="dark"] a {
+ color: #58a6ff !important;
+}
+
+[data-theme="dark"] a:hover {
+ color: #79c0ff !important;
+}
+
+[data-theme="dark"] .navbar-dark {
+ background-color: var(--navbar-bg) !important;
+}
+
+[data-theme="dark"] pre {
+ background-color: var(--code-bg) !important;
+ border-radius: 6px;
+}
+
+.dark-mode-toggle {
+ background: none;
+ border: none;
+ color: rgba(255, 255, 255, 0.8);
+ font-size: 1.2rem;
+ cursor: pointer;
+ padding: 0;
+ margin-left: 1rem;
+ line-height: 1;
+ display: inline-flex;
+ align-items: center;
+ transition: color 0.15s ease;
+}
+
+.dark-mode-toggle:hover {
+ color: rgba(255, 255, 255, 1);
+}
+
+.dark-mode-toggle:focus {
+ outline: 2px solid #79c0ff;
+ outline-offset: 2px;
+}
+
+.dark-mode-toggle:focus:not(:focus-visible) {
+ outline: none;
+}
+
+.sun-icon {
+ display: none;
+}
+
+.moon-icon {
+ display: inline;
+}
+
+.sun-icon,
+.moon-icon {
+ line-height: 1;
+}
+
+[data-theme="dark"] .sun-icon {
+ display: inline;
+}
+
+[data-theme="dark"] .moon-icon {
+ display: none;
+}
+
+[data-theme="dark"] .hljs {
+ background: var(--code-bg) !important;
+ color: var(--text-primary) !important;
+}
+
+[data-theme="dark"] .hljs-comment {
+ color: var(--text-secondary) !important;
+}
diff --git a/blog/feed.xml b/blog/feed.xml
index 95c2b03..dafc16b 100644
--- a/blog/feed.xml
+++ b/blog/feed.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon, 12 Jan 2026 00:00:00 +0000</lastBuildDate><item><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link>https://datafusion.apache.org/blog/2026/01/12/extending-sql</link><description><!--
+<rss version="2.0"><channel><title>Apache DataFusion Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Fri, 20 Mar 2026 00:00:00 +0000</lastBuildDate><item><title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</title><link>https://datafusion.apache.org/blog/2026/03/20/limit-pruning</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -18,8 +18,112 @@ limitations under the License.
{% endcomment %}
-->
-<p>If you embed <a href="https://datafusion.apache.org/">DataFusion</a> in your product, your users will eventually run SQL that DataFusion does not recognize. Not because the query is unreasonable, but because SQL in practice includes many dialects and system-specific statements.</p>
-<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Geoffrey Claude (Datadog)</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/extending-sql</guid><category>blog</category></item><item><title>Apache DataFusion 52.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/01/08/da [...]
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">xudong</dc:creator><pubDate>Fri, 20 Mar 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-03-20:/blog/2026/03/20/limit-pruning</guid><category>blog</category></item><item><title>"Writing Custom Table Providers in Apache DataFusion"</title><link>https://dat [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Tim Saucer</dc:creator><pubDate>Fri, 20 Mar 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-03-20:/blog/2026/03/20/writing-table-providers</guid><category>blog</category></item><item><title>Optimizing SQL CASE Expression Evaluation</title><link>https://datafusion.apache.org/b [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Pepijn Van Eeckhoudt</dc:creator><pubDate>Mon, 02 Feb 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-02-02:/blog/2026/02/02/datafusion_case</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.13.0 Release</title><link>https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0</link><des [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 30 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-30:/blog/2026/01/30/datafusion-comet-0.13.0</guid><category>blog</category></item><item><title>Apache DataFusion 52.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0</link><de [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -43,7 +147,27 @@ some of the major improvements since <a href="https://datafusion.apache.org/b
changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
making this release possible.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 08 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-08:/blog/2026/01/08/datafusion-52.0.0</guid><category>blog</category></item><item><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link>https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repar [...]
+<p>We continue to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/datafusion-52.0.0</guid><category>blog</category></item><item><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link>https://datafusion.apache.org/blog/2026/01/12/extending-sql</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a href="https://datafusion.apache.org/">DataFusion</a> in your product, your users will eventually run SQL that DataFusion does not recognize. Not because the query is unreasonable, but because SQL in practice includes many dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Geoffrey Claude (Datadog)</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/extending-sql</guid><category>blog</category></item><item><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</titl [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/all-en.atom.xml b/blog/feeds/all-en.atom.xml
index 8dc477b..00a7bfd 100644
--- a/blog/feeds/all-en.atom.xml
+++ b/blog/feeds/all-en.atom.xml
@@ -1,5 +1,1660 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-01-12T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https://datafusion.apache.org/blog/2026/01/12/extend [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-03-20T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</title><link href="https://datafusion.apache [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make. This sounds obvious, but making it happen in practice requires careful engineering at every layer of the query engine. <a href="https://datafusion.apache.org/">Apache DataFusion</a> achieves this through a multi-layer <strong>pruning pipeline</strong> — a series of stages that progressively narrow down the data before decoding a single row.</p>
+<p>In this post, we describe a new optimization called <strong>limit pruning</strong> that makes this pipeline aware of SQL <code>LIMIT</code> clauses. By identifying row groups where <em>every</em> row is guaranteed to match the predicate, DataFusion can satisfy a <code>LIMIT</code> query without ever touching partially matching row groups — eliminating wasted I/O entirely.</p>
+<p>For example, given a query like:</p>
+<pre><code class="language-sql">SELECT * FROM tracking_data
+WHERE species LIKE 'Alpine%' AND s &gt;= 50
+LIMIT 3
+</code></pre>
+<p>If the pruning pipeline already knows that certain row groups fully satisfy the <code>WHERE</code> clause, those groups alone may contain enough rows to fill the <code>LIMIT</code> — making it unnecessary to scan anything else.</p>
+<p>This work was inspired by the "Pruning for LIMIT Queries" section of Snowflake's paper <a href="https://arxiv.org/pdf/2504.11540"><em>Pruning in Snowflake: Working Smarter, Not Harder</em></a>.</p>
+<h2 id="datafusions-pruning-pipeline">DataFusion's Pruning Pipeline<a class="headerlink" href="#datafusions-pruning-pipeline" title="Permanent link">¶</a></h2>
+<p>Before diving into limit pruning, let's understand the full pruning pipeline. DataFusion scans Parquet data through a series of increasingly fine-grained filters, each one eliminating data so the next stage processes less:</p>
+<figure>
+<img alt="Three phases of DataFusion's pruning pipeline" src="/blog/images/limit-pruning/pruning-phases.svg" width="80%"/>
+<figcaption>Figure 1: The three phases of DataFusion's pruning pipeline — from directories down to individual rows.</figcaption>
+</figure>
+<h3 id="phase-1-high-level-discovery">Phase 1: High-Level Discovery<a class="headerlink" href="#phase-1-high-level-discovery" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong>Partition Pruning</strong>: The <a href="https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html">ListingTable</a> component evaluates filters that depend only on partition columns — things like <code>year</code>, <code>month</code>, or <code>region</code> encoded in directory paths (e.g., <code>s3://data/year=2024/month=01/</code>). Irrelevant directories are elimin [...]
+<li><strong>File Stats Pruning</strong>: The <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.FilePruner.html">FilePruner</a> checks file-level min/max and null-count statistics. If these statistics prove that a file cannot satisfy the predicate, we drop it entirely — no need to read row group metadata.</li>
+</ul>
+<h3 id="phase-2-row-group-statistics">Phase 2: Row Group Statistics<a class="headerlink" href="#phase-2-row-group-statistics" title="Permanent link">¶</a></h3>
+<p>For each surviving file, DataFusion reads row group metadata and potentially <a href="https://parquet.apache.org/docs/file-format/bloomfilter/">bloom filters</a> and classifies each row group into one of three states (the example data shown in the figures below, such as "Snow Vole" and "Alpine Ibex", is adapted from the <a href="https://arxiv.org/pdf/2504.11540">Snowflake pruning paper</a>):</p>
+<figure>
+<img alt="Row group classification: not matching, partially matching, fully matching" src="/blog/images/limit-pruning/row-group-states.svg" width="80%"/>
+<figcaption>Figure 2: Row groups are classified into three states based on their statistics.</figcaption>
+</figure>
+<ul>
+<li><strong>Not Matching (Skipped)</strong>: Statistics prove no rows can match. The row group is ignored completely.</li>
+<li><strong>Partially Matching</strong>: Statistics cannot rule out matching rows, but also cannot guarantee them. These groups might be scanned and verified row by row later.</li>
+<li><strong>Fully Matching</strong>: Statistics prove that <em>every single row</em> in the group satisfies the predicate. This state is key to making limit pruning possible.</li>
+</ul>
+<h3 id="phase-3-granular-pruning">Phase 3: Granular Pruning<a class="headerlink" href="#phase-3-granular-pruning" title="Permanent link">¶</a></h3>
+<p>The final phase goes even deeper:</p>
+<ul>
+<li><strong><a href="https://parquet.apache.org/docs/file-format/pageindex/">Page Index Pruning</a></strong>: Parquet pages have their own min/max statistics. DataFusion uses these to skip individual data pages within a surviving row group.</li>
+<li><strong><a href="https://arrow.apache.org/blog/2025/12/11/parquet-late-materialization-deep-dive/">Late Materialization</a> (Row Filtering)</strong>: Instead of decoding all columns at once, DataFusion decodes the cheapest, most selective columns first. It filters rows using those columns, then only decodes the remaining columns for surviving rows.</li>
+</ul>
+<h2 id="the-problem-limit-was-ignored">The Problem: LIMIT Was Ignored<a class="headerlink" href="#the-problem-limit-was-ignored" title="Permanent link">¶</a></h2>
+<p>Before limit pruning, all of these stages worked well — but the pruning pipeline had <strong>no awareness of <code>LIMIT</code></strong>. Consider a query like:</p>
+<pre><code class="language-sql">SELECT * FROM tracking_data
+WHERE species LIKE 'Alpine%' AND s &gt;= 50
+LIMIT 3
+</code></pre>
+<p>Even when fully matched row groups alone contain enough rows to satisfy the <code>LIMIT</code>, DataFusion would still decode partially matching groups and filter out rows that did not match, wasting resources decoding rows just to immediately discard them.</p>
+<figure>
+<img alt="Traditional pruning decodes partially matching groups with no LIMIT awareness" src="/blog/images/limit-pruning/wasted-io.svg" width="80%"/>
+<figcaption>Figure 3: Without limit awareness, partially matching groups are scanned and filtered even when fully matched groups already have enough rows. The left section shows 5 fully matched rows (enough to satisfy <code>LIMIT 5</code>), while the right section with the dashed red border represents a partially matching group that is still decoded — wasting CPU and I/O on rows that may not match at all.</figcaption>
+</figure>
+<p>If five fully matched rows in a fully matched group already satisfy <code>LIMIT 5</code>, why bother decoding groups where we're not even sure any rows qualify?</p>
+<h2 id="the-solution-limit-aware-pruning">The Solution: Limit-Aware Pruning<a class="headerlink" href="#the-solution-limit-aware-pruning" title="Permanent link">¶</a></h2>
+<p>The solution adds a new step in the pruning pipeline — right after row group pruning and before page index pruning:</p>
+<figure>
+<img alt="Pruning pipeline with limit pruning highlighted" src="/blog/images/limit-pruning/pruning-pipeline.svg" width="80%"/>
+<figcaption>Figure 4: Limit pruning is inserted between row group and page index pruning.</figcaption>
+</figure>
+<p>The idea is simple: <strong>if fully matched row groups already contain enough rows to satisfy the <code>LIMIT</code>, rewrite the access plan to scan only those groups and skip everything else.</strong></p>
+<p>This optimization is applied only when the query is a pure limit query with no <code>ORDER BY</code>, because reordering which groups we scan could change the output ordering of the results. In the implementation, this check is expressed as:</p>
+<pre><code class="language-rust">// Prune by limit if limit is set and order is not sensitive
+if let (Some(limit), false) = (limit, preserve_order) {
+ row_groups.prune_by_limit(limit, rg_metadata, &amp;file_metrics);
+}
+</code></pre>
+<h2 id="mechanism-detecting-fully-matched-row-groups">Mechanism: Detecting Fully Matched Row Groups<a class="headerlink" href="#mechanism-detecting-fully-matched-row-groups" title="Permanent link">¶</a></h2>
+<p>The core insight is <strong>predicate negation</strong>. To determine if every row in a row group satisfies the predicate, we:</p>
+<ol>
+<li>Negate the original predicate</li>
+<li>Simplify the negated expression</li>
+<li>Evaluate the negation against the row group's statistics</li>
+<li>If the negation is <em>pruned</em> (proven impossible), then the original predicate holds for every row</li>
+</ol>
+<p>Since DataFusion already had expression simplification (step 2) and statistics-based pruning (step 3), implementing this was relatively straightforward — the key addition was composing these existing capabilities with predicate negation.</p>
+<figure>
+<img alt="Fully matched detection via predicate negation" src="/blog/images/limit-pruning/fully-matched-detection.svg" width="80%"/>
+<figcaption>Figure 5: If the negated predicate is impossible according to row group stats, all rows must match the original predicate.</figcaption>
+</figure>
+<p>In DataFusion's codebase, this logic lives in <code>identify_fully_matched_row_groups</code> (<a href="https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/row_group_filter.rs">row_group_filter.rs</a>):</p>
+<pre><code class="language-rust">fn identify_fully_matched_row_groups(
+ &amp;mut self,
+ candidate_row_group_indices: &amp;[usize],
+ arrow_schema: &amp;Schema,
+ parquet_schema: &amp;SchemaDescriptor,
+ groups: &amp;[RowGroupMetaData],
+ predicate: &amp;PruningPredicate,
+ metrics: &amp;ParquetFileMetrics,
+) {
+ // Create the inverted predicate: NOT(original)
+ let inverted_expr = Arc::new(NotExpr::new(
+ Arc::clone(predicate.orig_expr()),
+ ));
+
+ // Simplify: e.g., NOT(c1 = 0) → c1 != 0
+ let simplifier = PhysicalExprSimplifier::new(arrow_schema);
+ let Ok(inverted_expr) = simplifier.simplify(inverted_expr) else {
+ return;
+ };
+
+ let Ok(inverted_predicate) = PruningPredicate::try_new(
+ inverted_expr,
+ Arc::clone(predicate.schema()),
+ ) else {
+ return;
+ };
+
+ // Evaluate inverted predicate against row group stats
+ let Ok(inverted_values) =
+ inverted_predicate.prune(&amp;inverted_pruning_stats)
+ else {
+ return;
+ };
+
+ for (i, &amp;original_idx) in
+ candidate_row_group_indices.iter().enumerate()
+ {
+ // If negation is pruned (false), all rows match original
+ if !inverted_values[i] {
+ self.is_fully_matched[original_idx] = true;
+ }
+ }
+}
+</code></pre>
+<h2 id="mechanism-rewriting-the-access-plan">Mechanism: Rewriting the Access Plan<a class="headerlink" href="#mechanism-rewriting-the-access-plan" title="Permanent link">¶</a></h2>
+<p>Once we know which row groups are fully matched, the limit pruning algorithm is straightforward:</p>
+<figure>
+<img alt="Limit pruning access plan rewrite algorithm" src="/blog/images/limit-pruning/limit-rewrite-algorithm.svg" width="80%"/>
+<figcaption>Figure 6: The algorithm iterates fully matched groups, accumulating row counts until the limit is satisfied.</figcaption>
+</figure>
+<p>The implementation in <code>prune_by_limit</code> (<a href="https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/row_group_filter.rs">row_group_filter.rs</a>):</p>
+<pre><code class="language-rust">pub fn prune_by_limit(
+ &amp;mut self,
+ limit: usize,
+ rg_metadata: &amp;[RowGroupMetaData],
+ metrics: &amp;ParquetFileMetrics,
+) {
+ let mut fully_matched_indexes: Vec&lt;usize&gt; = Vec::new();
+ let mut fully_matched_rows: usize = 0;
+
+ for &amp;idx in self.access_plan.row_group_indexes().iter() {
+ if self.is_fully_matched[idx] {
+ fully_matched_indexes.push(idx);
+ fully_matched_rows += rg_metadata[idx].num_rows() as usize;
+ if fully_matched_rows &gt;= limit {
+ break;
+ }
+ }
+ }
+
+ // Rewrite the plan if we have enough rows
+ if fully_matched_rows &gt;= limit {
+ let mut new_plan = ParquetAccessPlan::new_none(rg_metadata.len());
+ for &amp;idx in &amp;fully_matched_indexes {
+ new_plan.scan(idx);
+ }
+ self.access_plan = new_plan;
+ }
+}
+</code></pre>
+<p>Key properties of this algorithm:</p>
+<ul>
+<li>It preserves the original row group ordering</li>
+<li>If fully matched groups don't have enough rows, the plan is unchanged — no harm done</li>
+<li>The cost is minimal: a single pass over the row group list</li>
+</ul>
+<h2 id="case-study-alpine-wildlife-query">Case Study: Alpine Wildlife Query<a class="headerlink" href="#case-study-alpine-wildlife-query" title="Permanent link">¶</a></h2>
+<p>Let's walk through a concrete example adapted from the <a href="https://arxiv.org/pdf/2504.11540">Snowflake pruning paper</a>. Given a wildlife tracking dataset with four row groups:</p>
+<pre><code class="language-sql">SELECT * FROM tracking_data
+WHERE species LIKE 'Alpine%' AND s &gt;= 50
+LIMIT 3
+</code></pre>
+<table class="table">
+<thead>
+<tr>
+<th>Row Group</th>
+<th>Species Range</th>
+<th>S Range</th>
+<th>State</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>RG1</td>
+<td>Snow Vole, Brown Bear, Gray Wolf</td>
+<td>7–133</td>
+<td><strong>Not Matching</strong> (no 'Alpine%')</td>
+</tr>
+<tr>
+<td>RG2</td>
+<td>Lynx, Red Fox, Alpine Bat</td>
+<td>6–71</td>
+<td><strong>Partially Matching</strong></td>
+</tr>
+<tr>
+<td>RG3</td>
+<td>Alpine Ibex, Alpine Goat, Alpine Sheep</td>
+<td>76–101</td>
+<td><strong>Fully Matching</strong></td>
+</tr>
+<tr>
+<td>RG4</td>
+<td>Mixed species</td>
+<td>Mixed</td>
+<td><strong>Partially Matching</strong></td>
+</tr>
+</tbody>
+</table>
+<figure>
+<img alt="Before and after limit pruning comparison" src="/blog/images/limit-pruning/before-after.svg" width="80%"/>
+<figcaption>Figure 7: Before limit pruning, RG2 is scanned for zero hits. After limit pruning, only RG3 is scanned.</figcaption>
+</figure>
+<p><strong>Before limit pruning</strong>: DataFusion scans RG2 (0 hits — wasted I/O), then RG3 (3 hits, early return). RG2 was decoded entirely for nothing.</p>
+<p><strong>With limit pruning</strong>: The system detects that RG3 has 3 fully matched rows, which satisfies <code>LIMIT 3</code>. It rewrites the access plan to scan only RG3, skipping RG2 and RG4 entirely. One row group scanned. Zero waste.</p>
+<h2 id="observing-limit-pruning-via-metrics">Observing Limit Pruning via Metrics<a class="headerlink" href="#observing-limit-pruning-via-metrics" title="Permanent link">¶</a></h2>
+<p>DataFusion exposes limit pruning activity through query metrics. When running a query with <code>EXPLAIN ANALYZE</code>, you will see entries like:</p>
+<pre><code>row_groups_pruned_statistics=4 total → 3 matched -&gt; 1 fully matched
+limit_pruned_row_groups=3 total → 1 matched
+</code></pre>
+<p>This tells us:
+- 4 row groups were evaluated, 3 survived statistics pruning, 1 was identified as fully matching
+- Of the 3 row groups that entered limit pruning, only 1 survived — 2 were pruned by the limit optimization</p>
+<h2 id="future-directions">Future Directions<a class="headerlink" href="#future-directions" title="Permanent link">¶</a></h2>
+<p>There are two natural extensions of this work:</p>
+<p><strong><a href="https://github.com/apache/datafusion/issues/19193">Page-Level Limit Pruning</a></strong>: Today, "fully matched" detection operates at the row group level. If we extend this to use page index statistics, we could stop decoding pages <em>within</em> a row group once the limit is met. This would pay dividends for wide row groups where only a few pages hold matching data.</p>
+<p><strong><a href="https://github.com/apache/datafusion/issues/19028">Row Filter Hints</a></strong>: Even when a row group is fully matched, the current row filter still evaluates predicates row by row. If we pass the fully matched groups info into the row filter builder, we can skip predicate evaluation entirely for guaranteed groups — saving CPU cycles on predicate evaluation.</p>
+<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h2>
+<p>DataFusion's pruning pipeline trims redundant I/O from the partition level all the way down to individual rows. Limit pruning adds a new step that creates an early exit when fully matched row groups already satisfy the <code>LIMIT</code>. The result is fewer row groups scanned, less data decoded, and faster queries.</p>
+<p>The key insights are:
+1. <strong>Predicate negation</strong> can identify row groups where <em>all</em> rows match — not just "some might match"
+2. <strong>Row count accumulation</strong> across fully matched groups enables early termination</p>
+<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries, and machine learning and streaming applications.</p>
+<p>DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone.</p>
+<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
+<p>If you are interested in contributing, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you need to
+understand and explains where your work should actually happen.</p>
+<h2 id="the-three-layers">The Three Layers<a class="headerlink" href="#the-three-layers" title="Permanent link">¶</a></h2>
+<hr/>
+<p>When DataFusion executes a query against a table, three abstractions collaborate
+to produce results:</p>
+<ol>
+<li><strong>[<code>TableProvider</code>]</strong> -- Describes the table (schema, capabilities) and
+ produces an execution plan when queried.</li>
+<li><strong>[<code>ExecutionPlan</code>]</strong> -- Describes <em>how</em> to compute the result: partitioning,
+ ordering, and child plan relationships.</li>
+<li><strong>[<code>SendableRecordBatchStream</code>]</strong> -- The async stream that <em>actually does the
+ work</em>, yielding <code>RecordBatch</code>es one at a time.</li>
+</ol>
+<p>Think of these as a funnel: <code>TableProvider::scan()</code> is called once during
+planning to create an <code>ExecutionPlan</code>, then <code>ExecutionPlan::execute()</code> is called
+once per partition to create a stream, and those streams are where rows are
+actually produced during execution.</p>
+<h2 id="layer-1-tableprovider">Layer 1: TableProvider<a class="headerlink" href="#layer-1-tableprovider" title="Permanent link">¶</a></h2>
+<hr/>
+<p>A [<code>TableProvider</code>] represents a queryable data source. For a minimal read-only
+table, you need four methods:</p>
+<pre><code class="language-rust">impl TableProvider for MyTable {
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+
+ fn schema(&amp;self) -&gt; SchemaRef {
+ Arc::clone(&amp;self.schema)
+ }
+
+ fn table_type(&amp;self) -&gt; TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &amp;self,
+ state: &amp;dyn Session,
+ projection: Option&lt;&amp;Vec&lt;usize&gt;&gt;,
+ filters: &amp;[Expr],
+ limit: Option&lt;usize&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ // Build and return an ExecutionPlan -- keep this lightweight!
+ Ok(Arc::new(MyExecPlan::new(
+ Arc::clone(&amp;self.schema),
+ projection,
+ limit,
+ )))
+ }
+}
+</code></pre>
+<p>The <code>scan</code> method is the heart of <code>TableProvider</code>. It receives three pushdown
+hints from the optimizer, each reducing the amount of data your source needs
+to produce:</p>
+<ul>
+<li><strong><code>projection</code></strong> -- Which columns are needed. This reduces the <strong>width</strong> of
+ the output. If your source supports it, read only these columns rather than
+ the full schema.</li>
+<li><strong><code>filters</code></strong> -- Predicates the engine would like you to apply during the
+ scan. This reduces the <strong>number of rows</strong> by skipping data that does not
+ match. Implement <code>supports_filters_pushdown</code> to advertise which filters you
+ can handle.</li>
+<li><strong><code>limit</code></strong> -- A row count cap. This also reduces the <strong>number of rows</strong> --
+ if you can stop reading early once you have produced enough rows, this avoids
+ unnecessary work.</li>
+</ul>
+<h3 id="keep-scan-lightweight">Keep <code>scan()</code> Lightweight<a class="headerlink" href="#keep-scan-lightweight" title="Permanent link">¶</a></h3>
+<p>This is a critical point: <strong><code>scan()</code> runs during planning, not execution.</strong> It
+should return quickly. Best practices are to avoid performing I/O, network
+calls, or heavy computation here. The <code>scan</code> method's job is to <em>describe</em> how
+the data will be produced, not to produce it. All the real work belongs in the
+stream (Layer 3).</p>
+<p>A common pitfall is to fetch data or open connections in <code>scan()</code>. This blocks
+the planning thread and can cause timeouts or deadlocks, especially if the query
+involves multiple tables or subqueries that all need to be planned before
+execution begins.</p>
+<h3 id="existing-implementations-to-learn-from">Existing Implementations to Learn From<a class="headerlink" href="#existing-implementations-to-learn-from" title="Permanent link">¶</a></h3>
+<p>DataFusion ships several <code>TableProvider</code> implementations that are excellent
+references:</p>
+<ul>
+<li><strong>[<code>MemTable</code>]</strong> -- Holds data in memory as <code>Vec&lt;RecordBatch&gt;</code>. The simplest
+ possible provider; great for tests and small datasets.</li>
+<li><strong>[<code>StreamTable</code>]</strong> -- Wraps a user-provided stream factory. Useful when your
+ data arrives as a continuous stream (e.g., from Kafka or a socket).</li>
+<li><strong>[<code>SortedTableProvider</code>]</strong> -- Wraps another <code>TableProvider</code> and advertises a
+ known sort order, enabling the optimizer to skip redundant sorts.</li>
+</ul>
+<h2 id="layer-2-executionplan">Layer 2: ExecutionPlan<a class="headerlink" href="#layer-2-executionplan" title="Permanent link">¶</a></h2>
+<hr/>
+<p>An [<code>ExecutionPlan</code>] is a node in the physical query plan tree. Your table
+provider's <code>scan()</code> method returns one. The required methods are:</p>
+<pre><code class="language-rust">impl ExecutionPlan for MyExecPlan {
+ fn name(&amp;self) -&gt; &amp;str { "MyExecPlan" }
+
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+
+ fn properties(&amp;self) -&gt; &amp;PlanProperties {
+ &amp;self.properties
+ }
+
+ fn children(&amp;self) -&gt; Vec&lt;&amp;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ vec![] // Leaf node -- no children
+ }
+
+ fn with_new_children(
+ self: Arc&lt;Self&gt;,
+ children: Vec&lt;Arc&lt;dyn ExecutionPlan&gt;&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ assert!(children.is_empty());
+ Ok(self)
+ }
+
+ fn execute(
+ &amp;self,
+ partition: usize,
+ context: Arc&lt;TaskContext&gt;,
+ ) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ // This is where you build and return your stream
+ // ...
+ }
+}
+</code></pre>
+<p>The key properties to set correctly in [<code>PlanProperties</code>] are <strong>output
+partitioning</strong> and <strong>output ordering</strong>.</p>
+<p><strong>Output partitioning</strong> tells the engine how many partitions your data has,
+which determines parallelism. If your source naturally partitions data (e.g.,
+by file or by shard), expose that here.</p>
+<p><strong>Output ordering</strong> declares whether your data is naturally sorted. This
+enables the optimizer to avoid inserting a <code>SortExec</code> when a query requires
+ordered data. Getting this right can be a significant performance win.</p>
+<h3 id="partitioning-strategies">Partitioning Strategies<a class="headerlink" href="#partitioning-strategies" title="Permanent link">¶</a></h3>
+<p>Since <code>execute()</code> is called once per partition, partitioning directly controls
+the parallelism of your table scan. Each partition runs on its own task, so
+more partitions means more concurrent work -- up to the number of available
+cores.</p>
+<p>Consider how your data source naturally divides its data:</p>
+<ul>
+<li><strong>By file or object:</strong> If you are reading from S3, each file can be a
+ partition. DataFusion will read them in parallel.</li>
+<li><strong>By shard or region:</strong> If your source is a sharded database, each shard
+ maps naturally to a partition.</li>
+<li><strong>By key range:</strong> If your data is keyed (e.g., by timestamp or customer ID),
+ you can split it into ranges.</li>
+</ul>
+<p>Getting partitioning right matters because it affects everything downstream in
+the plan. When DataFusion needs to perform an aggregation or join, it
+repartitions data by hashing the relevant columns. If your source already
+produces data partitioned by the join or group-by key, DataFusion can skip the
+repartition step entirely -- avoiding a potentially expensive shuffle.</p>
+<p>For example, if you are building a table provider for a system that stores
+data partitioned by <code>customer_id</code>, and a common query groups by <code>customer_id</code>:</p>
+<pre><code class="language-sql">SELECT customer_id, SUM(amount)
+FROM my_table
+GROUP BY customer_id;
+</code></pre>
+<p>If you declare your output partitioning as <code>Hash([customer_id], N)</code>, the
+optimizer recognizes that the data is already distributed correctly for the
+aggregation and eliminates the <code>RepartitionExec</code> that would otherwise appear
+in the plan. You can verify this with <code>EXPLAIN</code> (more on this below).</p>
+<p>Conversely, if you report <code>UnknownPartitioning</code>, DataFusion must assume the
+worst case and will always insert repartitioning operators as needed.</p>
+<h3 id="keep-execute-lightweight-too">Keep <code>execute()</code> Lightweight Too<a class="headerlink" href="#keep-execute-lightweight-too" title="Permanent link">¶</a></h3>
+<p>Like <code>scan()</code>, the <code>execute()</code> method should construct and return a stream
+without doing heavy work. The actual data production happens when the stream
+is polled. Do not block on async operations here -- build the stream and let
+the runtime drive it.</p>
+<h3 id="existing-implementations-to-learn-from_1">Existing Implementations to Learn From<a class="headerlink" href="#existing-implementations-to-learn-from_1" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong>[<code>StreamingTableExec</code>]</strong> -- Executes a streaming table scan. It takes a
+ stream factory (a closure that produces streams) and handles partitioning.
+ Good reference for wrapping external streams.</li>
+<li><strong>[<code>DataSourceExec</code>]</strong> -- The execution plan behind DataFusion's built-in file
+ scanning (Parquet, CSV, JSON). It demonstrates sophisticated partitioning,
+ filter pushdown, and projection pushdown.</li>
+</ul>
+<h2 id="layer-3-sendablerecordbatchstream">Layer 3: SendableRecordBatchStream<a class="headerlink" href="#layer-3-sendablerecordbatchstream" title="Permanent link">¶</a></h2>
+<hr/>
+<p>[<code>SendableRecordBatchStream</code>] is where the real work happens. It is defined as:</p>
+<pre><code class="language-rust">type SendableRecordBatchStream =
+ Pin&lt;Box&lt;dyn RecordBatchStream&lt;Item = Result&lt;RecordBatch&gt;&gt; + Send&gt;&gt;;
+</code></pre>
+<p>This is an async stream of <code>RecordBatch</code>es that can be sent across threads. When
+the DataFusion runtime polls this stream, your code runs: reading files, calling
+APIs, transforming data, etc.</p>
+<h3 id="using-recordbatchstreamadapter">Using RecordBatchStreamAdapter<a class="headerlink" href="#using-recordbatchstreamadapter" title="Permanent link">¶</a></h3>
+<p>The easiest way to create a <code>SendableRecordBatchStream</code> is with
+[<code>RecordBatchStreamAdapter</code>]. It bridges any <code>futures::Stream&lt;Item =
+Result&lt;RecordBatch&gt;&gt;</code> into the <code>SendableRecordBatchStream</code> type:</p>
+<pre><code class="language-rust">use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+
+fn execute(
+ &amp;self,
+ partition: usize,
+ context: Arc&lt;TaskContext&gt;,
+) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ let schema = self.schema();
+ let config = self.config.clone();
+
+ let stream = futures::stream::once(async move {
+ // ALL the heavy work happens here, inside the stream:
+ // - Open connections
+ // - Read data from external sources
+ // - Transform and batch the results
+ let batches = fetch_data_from_source(&amp;config).await?;
+ Ok(batches)
+ })
+ .flat_map(|result| match result {
+ Ok(batch) =&gt; futures::stream::iter(vec![Ok(batch)]),
+ Err(e) =&gt; futures::stream::iter(vec![Err(e)]),
+ });
+
+ Ok(Box::pin(RecordBatchStreamAdapter::new(schema, stream)))
+}
+</code></pre>
+<h3 id="cpu-intensive-work-use-a-separate-thread-pool">CPU-Intensive Work: Use a Separate Thread Pool<a class="headerlink" href="#cpu-intensive-work-use-a-separate-thread-pool" title="Permanent link">¶</a></h3>
+<p>If your stream performs CPU-intensive work (parsing, decompression, complex
+transformations), avoid blocking the tokio runtime. Instead, offload to a
+dedicated thread pool and send results back through a channel:</p>
+<pre><code class="language-rust">fn execute(
+ &amp;self,
+ partition: usize,
+ context: Arc&lt;TaskContext&gt;,
+) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ let schema = self.schema();
+ let config = self.config.clone();
+
+ let (tx, rx) = tokio::sync::mpsc::channel(2);
+
+ // Spawn CPU-heavy work on a blocking thread pool
+ tokio::task::spawn_blocking(move || {
+ let batches = generate_data(&amp;config);
+ for batch in batches {
+ if tx.blocking_send(Ok(batch)).is_err() {
+ break; // Receiver dropped, query was cancelled
+ }
+ }
+ });
+
+ let stream = tokio_stream::wrappers::ReceiverStream::new(rx);
+ Ok(Box::pin(RecordBatchStreamAdapter::new(schema, stream)))
+}
+</code></pre>
+<p>This pattern keeps the async runtime responsive while your data generation
+runs on its own threads.</p>
+<h2 id="where-should-the-work-happen">Where Should the Work Happen?<a class="headerlink" href="#where-should-the-work-happen" title="Permanent link">¶</a></h2>
+<hr/>
+<p>This table summarizes what belongs at each layer:</p>
+<table class="table">
+<thead>
+<tr>
+<th>Layer</th>
+<th>Runs During</th>
+<th>Should Do</th>
+<th>Should NOT Do</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>TableProvider::scan()</code></td>
+<td>Planning</td>
+<td>Build an <code>ExecutionPlan</code> with metadata</td>
+<td>I/O, network calls, heavy computation</td>
+</tr>
+<tr>
+<td><code>ExecutionPlan::execute()</code></td>
+<td>Execution (once per partition)</td>
+<td>Construct a stream, set up channels</td>
+<td>Block on async work, read data</td>
+</tr>
+<tr>
+<td><code>RecordBatchStream</code> (polling)</td>
+<td>Execution</td>
+<td>All I/O, computation, data production</td>
+<td>--</td>
+</tr>
+</tbody>
+</table>
+<p>The guiding principle: <strong>push work as late as possible.</strong> Planning should be
+fast so the optimizer can do its job. Execution setup should be fast so all
+partitions can start promptly. The stream is where you spend time producing
+data.</p>
+<h3 id="why-this-matters">Why This Matters<a class="headerlink" href="#why-this-matters" title="Permanent link">¶</a></h3>
+<p>When <code>scan()</code> does heavy work, several problems arise:</p>
+<ol>
+<li><strong>Planning becomes slow.</strong> If a query touches 10 tables and each <code>scan()</code>
+ takes 500ms, planning alone takes 5 seconds before any data flows.</li>
+<li><strong>The optimizer cannot help.</strong> The optimizer runs between planning and
+ execution. If you have already fetched data during planning, optimizations
+ like predicate pushdown or partition pruning cannot reduce the work.</li>
+<li><strong>Resource management breaks down.</strong> DataFusion manages concurrency and
+ memory during execution. Work done during planning bypasses these controls.</li>
+</ol>
+<h2 id="filter-pushdown-doing-less-work">Filter Pushdown: Doing Less Work<a class="headerlink" href="#filter-pushdown-doing-less-work" title="Permanent link">¶</a></h2>
+<hr/>
+<p>One of the most impactful optimizations you can add to a custom table provider
+is <strong>filter pushdown</strong> -- letting the source skip data that the query does not
+need, rather than reading everything and filtering it afterward.</p>
+<h3 id="how-filter-pushdown-works">How Filter Pushdown Works<a class="headerlink" href="#how-filter-pushdown-works" title="Permanent link">¶</a></h3>
+<p>When DataFusion plans a query with a <code>WHERE</code> clause, it passes the filter
+predicates to your <code>scan()</code> method as the <code>filters</code> parameter. By default,
+DataFusion assumes your provider cannot handle any filters and inserts a
+<code>FilterExec</code> node above your scan to apply them. But if your source <em>can</em>
+evaluate some predicates during scanning -- for example, by skipping files,
+partitions, or row groups that cannot match -- you can eliminate a huge amount
+of unnecessary I/O.</p>
+<p>To opt in, implement <code>supports_filters_pushdown</code>:</p>
+<pre><code class="language-rust">fn supports_filters_pushdown(
+ &amp;self,
+ filters: &amp;[&amp;Expr],
+) -&gt; Result&lt;Vec&lt;TableProviderFilterPushDown&gt;&gt; {
+ Ok(filters.iter().map(|f| {
+ match f {
+ // We can fully evaluate equality filters on
+ // the partition column at the source
+ Expr::BinaryExpr(BinaryExpr {
+ left, op: Operator::Eq, right
+ }) if is_partition_column(left) || is_partition_column(right) =&gt; {
+ TableProviderFilterPushDown::Exact
+ }
+ // All other filters: let DataFusion handle them
+ _ =&gt; TableProviderFilterPushDown::Unsupported,
+ }
+ }).collect())
+}
+</code></pre>
+<p>The three possible responses for each filter are:</p>
+<ul>
+<li><strong><code>Exact</code></strong> -- Your source guarantees that no output rows will have a false
+ value for this predicate. Because the filter is fully evaluated at the source,
+ DataFusion will <strong>not</strong> add a <code>FilterExec</code> for it.</li>
+<li><strong><code>Inexact</code></strong> -- Your source has the ability to reduce the data produced, but
+ the output may still include rows that do not satisfy the predicate. For
+ example, you might skip entire files based on metadata statistics but not
+ filter individual rows within a file. DataFusion will still add a <code>FilterExec</code>
+ above your scan to remove any remaining rows that slipped through.</li>
+<li><strong><code>Unsupported</code></strong> -- Your source ignores this filter entirely. DataFusion
+ handles it.</li>
+</ul>
+<h3 id="why-filter-pushdown-matters">Why Filter Pushdown Matters<a class="headerlink" href="#why-filter-pushdown-matters" title="Permanent link">¶</a></h3>
+<p>Consider a table with 1 billion rows partitioned by <code>region</code>, and a query:</p>
+<pre><code class="language-sql">SELECT * FROM events WHERE region = 'us-east-1' AND event_type = 'click';
+</code></pre>
+<p><strong>Without filter pushdown:</strong> Your table provider reads all 1 billion rows
+across all regions. DataFusion then applies both filters, discarding the vast
+majority of the data.</p>
+<p><strong>With filter pushdown on <code>region</code>:</strong> Your <code>scan()</code> method sees the
+<code>region = 'us-east-1'</code> filter and constructs an execution plan that only reads
+the <code>us-east-1</code> partition. If that partition holds 100 million rows, you have
+just eliminated 90% of the I/O. DataFusion still applies the <code>event_type</code>
+filter via <code>FilterExec</code> if you reported it as <code>Unsupported</code>.</p>
+<h3 id="using-explain-to-debug-your-table-provider">Using EXPLAIN to Debug Your Table Provider<a class="headerlink" href="#using-explain-to-debug-your-table-provider" title="Permanent link">¶</a></h3>
+<p>The <code>EXPLAIN</code> statement is your best tool for understanding what DataFusion is
+actually doing with your table provider. It shows the physical plan that
+DataFusion will execute, including any operators it inserted:</p>
+<pre><code class="language-sql">EXPLAIN SELECT * FROM events WHERE region = 'us-east-1' AND event_type = 'click';
+</code></pre>
+<p>If you are using DataFrames, call <code>.explain(false, false)</code> for the logical plan
+or <code>.explain(false, true)</code> for the physical plan. You can also print the plans
+in verbose mode with <code>.explain(true, true)</code>.</p>
+<p><strong>Before filter pushdown</strong>, the plan might look like:</p>
+<pre><code class="language-text">FilterExec: region@0 = us-east-1 AND event_type@1 = click
+ MyExecPlan: partitions=50
+</code></pre>
+<p>Here DataFusion is reading all 50 partitions and filtering everything
+afterward. The <code>FilterExec</code> above your scan is doing all the predicate work.</p>
+<p><strong>After implementing pushdown for <code>region</code></strong> (reported as <code>Exact</code>):</p>
+<pre><code class="language-text">FilterExec: event_type@1 = click
+ MyExecPlan: partitions=5, filter=[region = us-east-1]
+</code></pre>
+<p>Now your exec reads only the 5 partitions for <code>us-east-1</code>, and the remaining
+<code>FilterExec</code> only handles the <code>event_type</code> predicate. The <code>region</code> filter has
+been fully absorbed by your scan.</p>
+<p><strong>After implementing pushdown for both filters</strong> (both <code>Exact</code>):</p>
+<pre><code class="language-text">MyExecPlan: partitions=5, filter=[region = us-east-1 AND event_type = click]
+</code></pre>
+<p>No <code>FilterExec</code> at all -- your source handles everything.</p>
+<p>Similarly, <code>EXPLAIN</code> will reveal whether DataFusion is inserting unnecessary
+<code>SortExec</code> or <code>RepartitionExec</code> nodes that you could eliminate by declaring
+better output properties. Whenever your queries seem slower than expected,
+<code>EXPLAIN</code> is the first place to look.</p>
+<h2 id="putting-it-all-together">Putting It All Together<a class="headerlink" href="#putting-it-all-together" title="Permanent link">¶</a></h2>
+<hr/>
+<p>Here is a minimal but complete example of a custom table provider that generates
+data lazily during streaming:</p>
+<pre><code class="language-rust">use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Int64Array, StringArray};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use arrow::record_batch::RecordBatch;
+use datafusion::catalog::TableProvider;
+use datafusion::common::Result;
+use datafusion::datasource::TableType;
+use datafusion::execution::context::SessionState;
+use datafusion::execution::SendableRecordBatchStream;
+use datafusion::logical_expr::Expr;
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{
+ ExecutionPlan, Partitioning, PlanProperties,
+};
+use futures::stream;
+
+/// A table provider that generates sequential numbers on demand.
+struct CountingTable {
+ schema: SchemaRef,
+ num_partitions: usize,
+ rows_per_partition: usize,
+}
+
+impl CountingTable {
+ fn new(num_partitions: usize, rows_per_partition: usize) -&gt; Self {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("partition", DataType::Int64, false),
+ Field::new("value", DataType::Int64, false),
+ ]));
+ Self { schema, num_partitions, rows_per_partition }
+ }
+}
+
+#[async_trait::async_trait]
+impl TableProvider for CountingTable {
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+ fn schema(&amp;self) -&gt; SchemaRef { Arc::clone(&amp;self.schema) }
+ fn table_type(&amp;self) -&gt; TableType { TableType::Base }
+
+ async fn scan(
+ &amp;self,
+ _state: &amp;dyn Session,
+ projection: Option&lt;&amp;Vec&lt;usize&gt;&gt;,
+ _filters: &amp;[Expr],
+ limit: Option&lt;usize&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ // Light work only: build the plan with metadata
+ Ok(Arc::new(CountingExec {
+ schema: Arc::clone(&amp;self.schema),
+ num_partitions: self.num_partitions,
+ rows_per_partition: limit
+ .unwrap_or(self.rows_per_partition)
+ .min(self.rows_per_partition),
+ properties: PlanProperties::new(
+ EquivalenceProperties::new(Arc::clone(&amp;self.schema)),
+ Partitioning::UnknownPartitioning(self.num_partitions),
+ EmissionType::Incremental,
+ Boundedness::Bounded,
+ ),
+ }))
+ }
+}
+
+struct CountingExec {
+ schema: SchemaRef,
+ num_partitions: usize,
+ rows_per_partition: usize,
+ properties: PlanProperties,
+}
+
+impl ExecutionPlan for CountingExec {
+ fn name(&amp;self) -&gt; &amp;str { "CountingExec" }
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+ fn properties(&amp;self) -&gt; &amp;PlanProperties { &amp;self.properties }
+ fn children(&amp;self) -&gt; Vec&lt;&amp;Arc&lt;dyn ExecutionPlan&gt;&gt; { vec![] }
+
+ fn with_new_children(
+ self: Arc&lt;Self&gt;,
+ _children: Vec&lt;Arc&lt;dyn ExecutionPlan&gt;&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ Ok(self)
+ }
+
+ fn execute(
+ &amp;self,
+ partition: usize,
+ _context: Arc&lt;TaskContext&gt;,
+ ) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ let schema = Arc::clone(&amp;self.schema);
+ let rows = self.rows_per_partition;
+
+ // The heavy work (data generation) happens inside the stream,
+ // not here in execute().
+ let batch_stream = stream::once(async move {
+ let partitions = Int64Array::from(
+ vec![partition as i64; rows],
+ );
+ let values = Int64Array::from(
+ (0..rows as i64).collect::&lt;Vec&lt;_&gt;&gt;(),
+ );
+ let batch = RecordBatch::try_new(
+ Arc::clone(&amp;schema),
+ vec![Arc::new(partitions), Arc::new(values)],
+ )?;
+ Ok(batch)
+ });
+
+ Ok(Box::pin(RecordBatchStreamAdapter::new(
+ Arc::clone(&amp;self.schema),
+ batch_stream,
+ )))
+ }
+}
+</code></pre>
+<h2 id="choosing-the-right-starting-point">Choosing the Right Starting Point<a class="headerlink" href="#choosing-the-right-starting-point" title="Permanent link">¶</a></h2>
+<hr/>
+<p>Not every custom data source requires implementing all three layers from
+scratch. DataFusion provides building blocks that let you plug in at whatever
+level makes sense:</p>
+<table class="table">
+<thead>
+<tr>
+<th>If your data is...</th>
+<th>Start with</th>
+<th>You implement</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Already in <code>RecordBatch</code>es in memory</td>
+<td>[<code>MemTable</code>]</td>
+<td>Nothing -- just construct it</td>
+</tr>
+<tr>
+<td>An async stream of batches</td>
+<td>[<code>StreamTable</code>]</td>
+<td>A stream factory</td>
+</tr>
+<tr>
+<td>A table with known sort order</td>
+<td>[<code>SortedTableProvider</code>] wrapping another provider</td>
+<td>The inner provider</td>
+</tr>
+<tr>
+<td>A custom source needing full control</td>
+<td><code>TableProvider</code> + <code>ExecutionPlan</code> + stream</td>
+<td>All three layers</td>
+</tr>
+</tbody>
+</table>
+<p>For most integrations, [<code>StreamTable</code>] combined with
+[<code>RecordBatchStreamAdapter</code>] provides a good balance of simplicity and
+flexibility. You provide a closure that returns a stream, and DataFusion handles
+the rest.</p>
+<h2 id="further-reading">Further Reading<a class="headerlink" href="#further-reading" title="Permanent link">¶</a></h2>
+<hr/>
+<ul>
+<li>[TableProvider API docs][<code>TableProvider</code>]</li>
+<li>[ExecutionPlan API docs][<code>ExecutionPlan</code>]</li>
+<li>[SendableRecordBatchStream API docs][<code>SendableRecordBatchStream</code>]</li>
+<li><a href="https://github.com/apache/datafusion/issues/16821">GitHub issue discussing table provider examples</a></li>
+<li><a href="https://github.com/apache/datafusion/tree/main/datafusion-examples/examples">DataFusion examples directory</a> --
+ contains working examples including custom table providers</li>
+</ul>
+<hr/>
+<p><em>Note: Portions of this blog post were written with the assistance of an AI agent.</em></p></content><category term="blog"></category></entry><entry><title>Optimizing SQL CASE Expression Evaluation</title><link href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="alternate"></link><published>2026-02-02T00:00:00+00:00</published><updated>2026-02-02T00:00:00+00:00</updated><author><name>Pepijn Van Eeckhoudt</name></author><id>tag:datafusio [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a set of expressions is evaluated for each row based on arbitrary boolean expressions.
+Its deceptively simple syntax hides significant implementation complexity.
+Over the past few <a href="https://datafusion.apache.org/">Apache DataFusion</a> releases, a series of improvements to <code>CASE</code> expression evaluator have been merged that reduce both CPU time and memory allocations.
+This post provides an overview of the original implementation, its performance bottlenecks, and the steps taken to address them.</p>
+<h2 id="background-case-expression-evaluation">Background: CASE Expression Evaluation<a class="headerlink" href="#background-case-expression-evaluation" title="Permanent link">¶</a></h2>
+<p>SQL supports two forms of CASE expressions:</p>
+<ol>
+<li><strong>Simple</strong>: <code>CASE expr WHEN value1 THEN result1 WHEN value2 THEN result2 ... END</code></li>
+<li><strong>Searched</strong>: <code>CASE WHEN condition1 THEN result1 WHEN condition2 THEN result2 ... END</code></li>
+</ol>
+<p>The simple form evaluates an expression once for each input row and then tests that value against the expressions (typically constants) in each <code>WHEN</code> clause using equality comparisons.</p>
+<p>Here's an example of the simple form:</p>
+<pre><code class="language-sql">CASE status
+ WHEN 'pending' THEN 1
+ WHEN 'active' THEN 2
+ WHEN 'complete' THEN 3
+ ELSE 0
+END
+</code></pre>
+<p>In this <code>CASE</code> expression, <code>status</code> is evaluated once per row, and then its value is tested for equality with the values <code>'pending'</code>, <code>'active'</code>, and <code>'complete'</code> in that order.
+The <code>CASE</code> expression evaluates to the value of the <code>THEN</code> expression corresponding to the first matching <code>WHEN</code> expression.</p>
+<p>The searched <code>CASE</code> form is a more flexible variant.
+It evaluates completely independent boolean expressions for each branch.
+This allows you to test different columns with different operators per branch as shown in the following example:</p>
+<pre><code class="language-sql">CASE
+ WHEN age &gt; 65 THEN 'senior'
+ WHEN childCount != 0 THEN 'parent'
+ WHEN age &lt; 21 THEN 'minor'
+ ELSE 'adult'
+END
+</code></pre>
+<p>In both forms, branches are evaluated sequentially with short-circuit semantics: for each row, once a <code>WHEN</code> condition matches, the corresponding <code>THEN</code> expression is evaluated.
+Any further branches are not evaluated for that row.
+This lazy evaluation model is critical for correctness.
+It lets you safely write <code>CASE</code> expressions like <code>CASE WHEN d != 0 THEN n / d ELSE NULL END</code> that are guaranteed to not trigger divide-by-zero errors.</p>
+<p>Besides <code>CASE</code>, there are a few <a href="https://datafusion.apache.org/user-guide/sql/scalar_functions.html#conditional-functions">conditional scalar functions</a> that provide similar, more restricted capabilities.
+These include <code>COALESCE</code>, <code>IFNULL</code>, and <code>NVL2</code>.
+You can consider each of these functions as the equivalent of a macro for <code>CASE</code>.
+For example, <code>COALESCE(expr1, expr2, expr3)</code> expands to:</p>
+<pre><code class="language-sql">CASE
+ WHEN expr1 IS NOT NULL THEN expr1
+ WHEN expr2 IS NOT NULL THEN expr2
+ ELSE expr3
+END
+</code></pre>
+<p>Since <a href="https://datafusion.apache.org/">Apache DataFusion</a> rewrites these conditional functions to their equivalent <code>CASE</code> expression, any optimizations related to <code>CASE</code> described in this post also apply to conditional function evaluation.</p>
+<h2 id="case-evaluation-in-datafusion-5000"><code>CASE</code> Evaluation in DataFusion 50.0.0<a class="headerlink" href="#case-evaluation-in-datafusion-5000" title="Permanent link">¶</a></h2>
+<p>For the remainder of this post, we'll be looking at 'searched CASE' evaluation.
+'Simple CASE' uses a distinct, but very similar implementation.
+The same set of improvements has been applied to both.</p>
+<p>The baseline implementation in DataFusion 50.0.0 evaluated <code>CASE</code> using a common, straightforward approach:</p>
+<ol>
+<li>Start with an output array <code>out</code> with the same length as the input batch, filled with nulls. Additionally, create a bit vector <code>remainder</code> with the same length and each value set to <code>true</code>.</li>
+<li>For each <code>WHEN</code>/<code>THEN</code> branch:<ul>
+<li>Evaluate the <code>WHEN</code> condition for the remaining unmatched rows using <a href="https://docs.rs/datafusion/latest/datafusion/physical_expr/trait.PhysicalExpr.html#method.evaluate_selection"><code>PhysicalExpr::evaluate_selection</code></a>, passing in the input batch and the <code>remainder</code> mask.</li>
+<li>If any rows matched, evaluate the <code>THEN</code> expression for those rows using <code>PhysicalExpr::evaluate_selection</code>.</li>
+<li>Merge the results into the <code>out</code> array using the <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/zip/fn.zip.html"><code>zip</code></a> kernel.</li>
+<li>Update the <code>remainder</code> mask to exclude the matched rows.</li>
+</ul>
+</li>
+<li>If there's an <code>ELSE</code> clause, evaluate it for any remaining unmatched rows and merge using <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/zip/fn.zip.html"><code>zip</code></a>.</li>
+</ol>
+<p>Here's a simplified version of the Rust code for the original loop:</p>
+<pre><code class="language-rust">let mut out = new_null_array(&amp;return_type, batch.num_rows());
+let mut remainder = BooleanArray::from(vec![true; batch.num_rows()]);
+
+for (when_expr, then_expr) in &amp;self.when_then_expr {
+ // Determine for which remaining rows the WHEN condition matches
+ let when = when_expr.evaluate_selection(batch, &amp;remainder)?
+ .into_array(batch.num_rows())?;
+ // Ensure any `NULL` values are treated as false
+ let when_and_rem = and(&amp;when, &amp;remainder)?;
+
+ if when_and_rem.true_count() == 0 {
+ continue;
+ }
+
+ // Evaluate the THEN expression for matching rows
+ let then = then_expr.evaluate_selection(batch, &amp;when_and_rem)?;
+ // Merge results into output array
+ out = zip(&amp;when_and_rem, &amp;then_value, &amp;out)?;
+ // Update remainder mask to exclude matched rows
+ remainder = and_not(&amp;remainder, &amp;when_and_rem)?;
+}
+</code></pre>
+<p>Let's examine one iteration of this loop for the following <code>CASE</code> expression:</p>
+<pre><code class="language-sql">CASE
+ WHEN col = 'b' THEN 100
+ ELSE 200
+END
+</code></pre>
+<p>Schematically, it will look as follows:</p>
+<figure>
+<img alt="Schematic representation of data flow in the original CASE implementation" class="img-responsive" src="/blog/images/case/original_loop.svg" width="100%"/>
+<figcaption>One iteration of the `CASE` evaluation loop</figcaption>
+</figure>
+<p>This implementation works perfectly fine, but there's significant room for optimization, mostly related to the usage of <code>evaluate_selection</code>.
+To understand why, we need to dig a little deeper into the implementation of that function.
+Here's a simplified version of it that captures the relevant parts:</p>
+<pre><code class="language-rust">pub trait PhysicalExpr {
+ fn evaluate_selection(
+ &amp;self,
+ batch: &amp;RecordBatch,
+ selection: &amp;BooleanArray,
+ ) -&gt; Result&lt;ColumnarValue&gt; {
+ // Reduce record batch to only include rows that match selection
+ let filtered_batch = filter_record_batch(batch, selection)?;
+ // Perform regular evaluation on filtered batch
+ let filtered_result = self.evaluate(&amp;filtered_batch)?;
+ // Expand result array to match original batch length
+ scatter(selection, filtered_result)
+ }
+}
+</code></pre>
+<p>Going back to the same example as before, the data flow in <code>evaluate_selection</code> looks like this:</p>
+<figure>
+<img alt="Schematic representation of `evaluate_selection` evaluation" class="img-responsive" src="/blog/images/case/evaluate_selection.svg" width="100%"/>
+<figcaption>evaluate_selection data flow</figcaption>
+</figure>
+<p>The <code>evaluate_selection</code> method first filters the input batch to only include rows that match the <code>selection</code> mask.
+It then calls the regular <code>evaluate</code> method using the filtered batch as input.
+Finally, to return a result array with the same number of rows as <code>batch</code>, the <code>scatter</code> function is called.
+This function produces a new array padded with <code>null</code> values for any rows that didn't match the <code>selection</code> mask.</p>
+<p>So how can we improve the performance of the simple evaluation strategy and use of <code>evaluate_selection</code>?</p>
+<h3 id="opportunity-1-early-exit">Opportunity 1: Early Exit<a class="headerlink" href="#opportunity-1-early-exit" title="Permanent link">¶</a></h3>
+<p>The <code>CASE</code> evaluation loop always iterates through all branches, even when every row has already been matched.
+In queries where early branches match all rows, this results in unnecessary work being done for the remaining branches.</p>
+<h3 id="opportunity-2-optimize-repeated-filtering-scattering-and-merging">Opportunity 2: Optimize Repeated Filtering, Scattering, and Merging<a class="headerlink" href="#opportunity-2-optimize-repeated-filtering-scattering-and-merging" title="Permanent link">¶</a></h3>
+<p>Each iteration performs a number of operations that are very well-optimized, but still take up a significant amount of CPU time:</p>
+<ul>
+<li><strong>Filtering</strong>: <code>PhysicalExpr::evaluate_selection</code> filters the entire <code>RecordBatch</code> for each branch. For the <code>WHEN</code> expression, this is done even if the selection mask was entirely empty.</li>
+<li><strong>Scattering</strong>: <code>PhysicalExpr::evaluate_selection</code> scatters the filtered result back to the original <code>RecordBatch</code> length.</li>
+<li><strong>Merging</strong>: The <code>zip</code> kernel is called once per branch to merge partial results into the output array</li>
+</ul>
+<p>Each of these operations needs to allocate memory for new arrays and shuffle quite a bit of data around.</p>
+<h3 id="opportunity-3-filter-only-necessary-columns">Opportunity 3: Filter only Necessary Columns<a class="headerlink" href="#opportunity-3-filter-only-necessary-columns" title="Permanent link">¶</a></h3>
+<p>The <code>PhysicalExpr::evaluate_selection</code> method filters the entire record batch, including columns that the current branch's <code>WHEN</code> and <code>THEN</code> expressions don't reference.
+For wide tables (many columns) with narrow expressions (few column references), this is wasteful.</p>
+<p>Suppose you have a table with 26 columns named <code>a</code> through <code>z</code>, and the following simple <code>CASE</code> expression:</p>
+<pre><code class="language-sql">CASE
+ WHEN a &gt; 1000 THEN 'large'
+ WHEN a &gt;= 0 THEN 'positive'
+ ELSE 'negative'
+END
+</code></pre>
+<p>The implementation would filter all 26 columns even though only a single column is needed for the entire <code>CASE</code> expression evaluation.
+Again this involves a non-negligible amount of allocation and data copying.</p>
+<h2 id="performance-optimizations">Performance Optimizations<a class="headerlink" href="#performance-optimizations" title="Permanent link">¶</a></h2>
+<h3 id="optimization-1-short-circuit-early-exit">Optimization 1: Short-Circuit Early Exit<a class="headerlink" href="#optimization-1-short-circuit-early-exit" title="Permanent link">¶</a></h3>
+<p>The first optimization is straightforward.
+As soon as we detect that all rows of the batch have been matched, we break out of the evaluation loop:</p>
+<pre><code class="language-rust">let mut remainder_count = batch.num_rows();
+
+for (when_expr, then_expr) in &amp;self.when_then_expr {
+ if remainder_count == 0 {
+ break; // All rows matched, exit early
+ }
+
+ // ... evaluate branch ...
+
+ let when_match_count = when_value.true_count();
+ remainder_count -= when_match_count;
+}
+</code></pre>
+<p>Additionally, we avoid evaluating the <code>ELSE</code> clause when no rows remain:</p>
+<pre><code class="language-rust">if let Some(else_expr) = &amp;self.else_expr {
+ remainder = or(&amp;base_nulls, &amp;remainder)?;
+ if remainder.true_count() &gt; 0 {
+ // ... evaluate else ...
+ }
+}
+</code></pre>
+<p>For queries where early branches match all rows, this eliminates unnecessary branch evaluations and <code>ELSE</code> clause processing.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/17898">PR #17898</a></p>
+<h3 id="optimization-2-optimized-result-merging">Optimization 2: Optimized Result Merging<a class="headerlink" href="#optimization-2-optimized-result-merging" title="Permanent link">¶</a></h3>
+<p>The second optimization fundamentally restructures how the results of each loop iteration will be merged.
+The diagram below illustrates the optimized data flow when evaluating the <code>CASE WHEN col = 'b' THEN 100 ELSE 200 END</code> from before:</p>
+<figure>
+<img alt="Schematic representation of optimized evaluation loop" class="img-responsive" src="/blog/images/case/merging.svg" width="100%"/>
+<figcaption>optimized evaluation loop</figcaption>
+</figure>
+<p>In the reworked implementation, the <code>evaluate_selection</code> function is no longer used.
+The key insight is that we can defer all merging until the end of the evaluation loop by tracking result provenance.
+This was implemented with the following changes:</p>
+<ol>
+<li>Augment the input batch with a column containing row indices.</li>
+<li>Reduce the augmented batch after each loop iteration to only contain the remaining rows.</li>
+<li>Use the row index column to track which partial result array contains the value for each row.</li>
+<li>Perform a single merge operation at the end instead of a <code>zip</code> operation after each loop iteration.</li>
+</ol>
+<p>These changes make it unnecessary to <code>scatter</code> and <code>zip</code> results in each loop iteration.
+Instead, when all rows have been matched, we then merge the partial results using <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge_n.html"><code>arrow_select::merge::merge_n</code></a>.</p>
+<p>The diagram below illustrates how <code>merge_n</code> works for an example where three <code>WHEN/THEN</code> branches produced results.
+The first branch produced the result <code>A</code> for row 2, the second produced <code>B</code> for row 1, and the third produced <code>C</code> and <code>D</code> for rows 4 and 5.</p>
+<figure>
+<img alt="Schematic illustration of the merge_n algorithm" class="img-responsive" src="/blog/images/case/merge_n.svg" width="100%"/>
+<figcaption>merge_n example</figcaption>
+</figure>
+<p>The <code>merge_n</code> algorithm scans through the indices array.
+For each non-empty cell, it takes one value from the corresponding values array.
+In the example above, we first encounter <code>1</code>.
+This takes the first element from the values array with index <code>1</code>, resulting in <code>B</code>.
+The next cell contains <code>0</code> which takes <code>A</code>, from the first array.
+Finally, we encounter <code>2</code> twice.
+This takes the first and second element from the last values array respectively.</p>
+<p>This algorithm was initially implemented in DataFusion for the <code>CASE</code> implementation, but in the meantime has been generalized and moved into the <code>arrow-rs</code> crate as <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge_n.html"><code>arrow_select::merge::merge_n</code></a>.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18152">PR #18152</a></p>
+<h3 id="optimization-3-column-projection">Optimization 3: Column Projection<a class="headerlink" href="#optimization-3-column-projection" title="Permanent link">¶</a></h3>
+<p>The third optimization addresses the "filtering unused columns" overhead through projection.</p>
+<p>Look at the following query example where the <code>mailing_address</code> table has the columns <code>name</code>, <code>surname</code>, <code>street</code>, <code>number</code>, <code>city</code>, <code>state</code>, <code>country</code>:</p>
+<pre><code class="language-sql">SELECT *, CASE WHEN country = 'USA' THEN state ELSE country END AS region
+FROM mailing_address
+</code></pre>
+<p>You can see that the <code>CASE</code> expression only references the columns <code>country</code> and <code>state</code>, but because all columns are being queried, projection pushdown cannot reduce the number of columns being fed in to the projection operator.</p>
+<figure>
+<img alt="Schematic illustration of CASE evaluation without projection" class="img-responsive" src="/blog/images/case/no_projection.svg" width="100%"/>
+<figcaption>CASE evaluation without projection</figcaption>
+</figure>
+<p>During <code>CASE</code> evaluation, the batch must be filtered using the <code>WHEN</code> expression to evaluate the <code>THEN</code> expression values.
+As the diagram above shows, this filtering creates a reduced copy of all columns.</p>
+<p>This unnecessary copying can be avoided by first narrowing the batch to only include the columns that are actually needed.</p>
+<figure>
+<img alt="Schematic illustration of CASE evaluation with projection" class="img-responsive" src="/blog/images/case/projection.svg" width="100%"/>
+<figcaption>CASE evaluation with projection</figcaption>
+</figure>
+<p>At first glance, this might not seem beneficial, since we're introducing an additional processing step.
+Luckily projection of a record batch only requires a shallow copy of the record batch.
+The column arrays themselves are not copied, and the only work that is actually done is incrementing the reference counts of the columns.</p>
+<p><strong>Impact</strong>: For wide tables with narrow CASE expressions, this dramatically reduces filtering overhead by removing the copying of unused columns.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18329">PR #18329</a></p>
+<h3 id="optimization-4-eliminating-scatter-in-two-branch-case">Optimization 4: Eliminating Scatter in Two-Branch Case<a class="headerlink" href="#optimization-4-eliminating-scatter-in-two-branch-case" title="Permanent link">¶</a></h3>
+<p>Some of the earlier examples in this post use expressions of the form <code>CASE WHEN condition THEN expr1 ELSE expr2 END</code> to explain how the general evaluation loop works.
+For this kind of two-branch <code>CASE</code> expression, <a href="https://datafusion.apache.org/">Apache DataFusion</a> has a more optimized implementation that unrolls the loop.
+This specialized <code>ExpressionOrExpression</code> fast path still used <code>evaluate_selection()</code> for both branches which uses <code>scatter</code> and <code>zip</code> to combine the results incurring the same performance overhead as the general implementation.</p>
+<p>The revised implementation eliminates the use of <code>evaluate_selection</code> as follows:</p>
+<pre><code class="language-rust">// Compute the `WHEN` condition for the entire batch
+let when_filter = create_filter(&amp;when_value);
+
+// Compute a compact array of `THEN` values for the matching rows
+let then_batch = filter_record_batch(batch, &amp;when_filter)?;
+let then_value = then_expr.evaluate(&amp;then_batch)?;
+
+// Compute a compact array of `ELSE` values for the non-matching rows
+let else_filter = create_filter(&amp;not(&amp;when_value)?);
+let else_batch = filter_record_batch(batch, &amp;else_filter)?;
+let else_value = else_expr.evaluate(&amp;else_batch)?;
+</code></pre>
+<p>This produces two compact arrays, one for the THEN values and one for the ELSE values, which are then merged with the <code>merge</code> function.
+In contrast to <code>zip</code>, <code>merge</code> does not require both of its value inputs to have the same length.
+Instead it requires that the sum of the length of the value inputs matches the length of the mask array.</p>
+<figure>
+<img alt="Schematic illustration of the merge algorithm" class="img-responsive" src="/blog/images/case/merge.svg" width="100%"/>
+<figcaption>merge example</figcaption>
+</figure>
+<p>This eliminates unnecessary <code>scatter</code> operations and memory allocations for one of the most common <code>CASE</code> expression patterns.</p>
+<p>Just like <code>merge_n</code>, this operation has been moved into <code>arrow-rs</code> as <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge.html"><code>arrow_select::merge::merge</code></a>.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18444">PR #18444</a></p>
+<h3 id="optimization-5-table-lookup-of-constants">Optimization 5: Table Lookup of Constants<a class="headerlink" href="#optimization-5-table-lookup-of-constants" title="Permanent link">¶</a></h3>
+<p>Up until now, we've discussed the implementations for generic <code>CASE</code> expressions that use non-constant expressions for both <code>WHEN</code> and <code>THEN</code>.
+Another common use of <code>CASE</code> is to perform a mapping from one set of constants to another.
+For instance, you can expand numeric constants to human-readable strings using the following <code>CASE</code> example.</p>
+<pre><code class="language-sql">CASE status
+ WHEN 0 THEN 'idle'
+ WHEN 1 THEN 'running'
+ WHEN 2 THEN 'paused'
+ WHEN 3 THEN 'stopped'
+ ELSE 'unknown'
+END
+</code></pre>
+<p>A final <code>CASE</code> optimization recognizes this pattern and compiles the <code>CASE</code> expression into a hash table.
+Rather than evaluating the <code>WHEN</code> and <code>THEN</code> expressions, the input expression is evaluated once, and the result array is computed using a vectorized hash table lookup.
+This approach avoids the need to filter the input batch and combine partial results entirely.
+The result array is computed in a single pass over the input values, and the computation time does not grow significantly with the number of <code>WHEN</code> branches in the <code>CASE</code> expression.</p>
+<p>This optimization was implemented by Raz Luvaton (<a href="https://github.com/rluvaton"><code>@rluvaton</code></a>) in <a href="https://github.com/apache/datafusion/pull/18183">PR #18183</a></p>
+<h2 id="results">Results<a class="headerlink" href="#results" title="Permanent link">¶</a></h2>
+<p>The degree to which the performance optimizations described in this post will benefit your queries is highly dependent on both your data and your queries.
+To give some idea of the impact, we ran the following query on the TPC_H <code>orders</code> table with a scale factor of 100:</p>
+<pre><code class="language-sql">SELECT
+ *,
+ case o_orderstatus
+ when 'O' then 'ordered'
+ when 'F' then 'filled'
+ when 'P' then 'pending'
+ else 'other'
+ end
+from orders
+</code></pre>
+<p>This query was first run with DataFusion 50.0.0 to get a baseline measurement.
+The same query was then run with each optimization applied in turn.
+The recorded times are presented as the blue series in the chart below.
+The green series shows the time measurement for the <code>SELECT * FROM orders</code> to give an idea of the cost the addition of a <code>CASE</code> expression in a query incurs.
+All measurements were made with a target partition count of <code>1</code>.</p>
+<figure>
+<img alt="Performance measurements chart" class="img-responsive" src="/blog/images/case/results.png" width="100%"/>
+<figcaption>Performance measurements</figcaption>
+</figure>
+<p>What you can see in the chart is that the effect of the various optimizations compounds up to the <code>project</code> measurement.
+Up to that point these results are applicable to any <code>CASE</code> expression.
+The final improvement in the <code>hash</code> measurement is only applicable to simple <code>CASE</code> expressions with constant <code>WHEN</code> and <code>THEN</code> expressions.</p>
+<p>The cumulative effect of these optimizations is a 63-71% reduction in CPU time spent evaluating <code>CASE</code> expressions compared to the baseline.</p>
+<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h2>
+<p>Through a number of targeted optimizations, we've transformed <code>CASE</code> expression evaluation from a simple, but unoptimized implementation into a highly optimized one.
+The optimizations described in this post compound: a <code>CASE</code> expression on a wide table with multiple branches and early matches benefits from all four optimizations simultaneously.
+The result is significantly reduced CPU time and memory allocation in SQL constructs that are essential for ETL-like queries.</p>
+<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries,
+and machine learning and streaming applications.
+While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion’s primary design goal</a> is to accelerate the creation of other data-centric systems, it provides a reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe library</a>, <a href="https://datafusion.apache.org/python/">Python library</a>, and <a href="https://datafusion.apache.org/u [...]
+<p>DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone.
+Without DataFusion, highly performant vectorized query engines would remain the domain of a few large companies and world-class research institutions.
+With DataFusion, we can all build on top of a shared foundation and focus on what makes our projects unique.</p>
+<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
+<p>DataFusion is not a project built or driven by a single person, company, or foundation.
+Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code.
+A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.13.0 Release</title><link href="https://datafusion.a [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development work and is the result of merging 169 PRs from 15
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.13.0.md">change log</a> for more information.</p>
+<h2 id="key-features">Key Features<a class="headerlink" href="#key-features" title="Permanent link">¶</a></h2>
+<h3 id="native-parquet-write-support-experimental">Native Parquet Write Support (Experimental)<a class="headerlink" href="#native-parquet-write-support-experimental" title="Permanent link">¶</a></h3>
+<p>This release introduces experimental native Parquet write capabilities, allowing Comet to intercept and execute Parquet write operations natively through DataFusion. Key capabilities include:</p>
+<ul>
+<li>File commit protocol support for reliable writes</li>
+<li>Remote HDFS writing via OpenDAL integration</li>
+<li>Complex type support (arrays, maps, structs)</li>
+<li>Proper handling of object store settings</li>
+</ul>
+<p>To enable native Parquet writes, set:</p>
+<pre><code>spark.comet.allowIncompatibleOp.DataWritingCommandExec=true
+spark.comet.parquet.write.enabled=true
+</code></pre>
+<p><strong>Note</strong>: This feature is highly experimental and should not be used in production environments. It is currently categorized as a testing feature and is disabled by default.</p>
+<h3 id="native-iceberg-improvements">Native Iceberg Improvements<a class="headerlink" href="#native-iceberg-improvements" title="Permanent link">¶</a></h3>
+<p>Comet's fully-native Iceberg integration received significant enhancements in this release:</p>
+<p><strong>REST Catalog Support</strong>: Native Iceberg scans now support REST catalogs, enabling integration with catalog services like Apache Polaris and Tabular. Configure with:</p>
+<pre><code class="language-shell">--conf spark.sql.catalog.rest_cat=org.apache.iceberg.spark.SparkCatalog
+--conf spark.sql.catalog.rest_cat.catalog-impl=org.apache.iceberg.rest.RESTCatalog
+--conf spark.sql.catalog.rest_cat.uri=http://localhost:8181
+--conf spark.comet.scan.icebergNative.enabled=true
+</code></pre>
+<p><strong>Session Token Authentication</strong>: Added support for session tokens in native Iceberg scans for secure S3 access.</p>
+<p><strong>Performance Optimizations</strong>:</p>
+<ul>
+<li>Deduplicated serialized metadata reducing memory overhead</li>
+<li>Switched from JSON to protobuf for partition value serialization</li>
+<li>Removed IcebergFileStream in favor of iceberg-rust's built-in parallelization</li>
+<li>Reduced metadata serialization points</li>
+<li>Added SchemaAdapter caching</li>
+</ul>
+<p>To enable fully-native Iceberg scanning:</p>
+<pre><code>spark.comet.scan.icebergNative.enabled=true
+</code></pre>
+<p>The native reader supports Iceberg table spec v1 and v2, all primitive and complex types, schema evolution, time travel, positional and equality deletes, filter pushdown, and various storage backends (local, HDFS, S3).</p>
+<h3 id="native-csv-reading-experimental">Native CSV Reading (Experimental)<a class="headerlink" href="#native-csv-reading-experimental" title="Permanent link">¶</a></h3>
+<p>Experimental support for native CSV file reading has been added, expanding Comet's file format capabilities beyond Parquet.</p>
+<h3 id="new-expressions">New Expressions<a class="headerlink" href="#new-expressions" title="Permanent link">¶</a></h3>
+<p>The release adds support for numerous expressions:</p>
+<ul>
+<li>Array functions: <code>explode</code>, <code>explode_outer</code>, <code>size</code></li>
+<li>Date/time functions: <code>unix_date</code>, <code>date_format</code>, <code>datediff</code>, <code>last_day</code>, <code>unix_timestamp</code></li>
+<li>String functions: <code>left</code></li>
+<li>JSON functions: <code>from_json</code> (partial support)</li>
+</ul>
+<h3 id="ansi-mode-support">ANSI Mode Support<a class="headerlink" href="#ansi-mode-support" title="Permanent link">¶</a></h3>
+<p>Sum and average aggregate expressions now support ANSI mode for both integer and decimal inputs, enabling overflow checking in strict SQL mode.</p>
+<h3 id="native-shuffle-improvements">Native Shuffle Improvements<a class="headerlink" href="#native-shuffle-improvements" title="Permanent link">¶</a></h3>
+<ul>
+<li>Round-robin partitioning is now supported in native shuffle</li>
+<li>Spill metrics are now reported correctly</li>
+<li>Configurable shuffle writer buffer size via <code>spark.comet.shuffle.write.bufferSize</code></li>
+</ul>
+<h2 id="performance-improvements">Performance Improvements<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>This release includes extensive performance optimizations:</p>
+<ul>
+<li><strong>String to integer casting</strong>: Significant speedups through optimized parsing</li>
+<li><strong>String functions</strong>: Optimized <code>lpad</code>/<code>rpad</code> to remove unnecessary memory allocations</li>
+<li><strong>Date operations</strong>: Improved <code>normalize_nan</code> and date truncate performance</li>
+<li><strong>Query planning</strong>: Cached query plans to avoid per-partition serialization overhead</li>
+<li><strong>Memory efficiency</strong>: Reduced GC pressure in protobuf serialization</li>
+<li><strong>Hash operations</strong>: Optimized complex-type hash implementations including murmur3 support for nested types</li>
+<li><strong>Runtime efficiency</strong>: Eliminated busy-polling of Tokio stream for plans without CometScan</li>
+<li><strong>Metrics overhead</strong>: Reduced timer and syscall overhead in native shuffle writer</li>
+</ul>
+<h2 id="deprecations">Deprecations<a class="headerlink" href="#deprecations" title="Permanent link">¶</a></h2>
+<p>The <code>native_comet</code> scan mode is now deprecated in favor of <code>native_iceberg_compat</code> and will be removed in a future release. The <code>auto</code> scan mode no longer falls back to <code>native_comet</code>.</p>
+<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
+<p>This release upgrades to DataFusion 51, Arrow 57, and the latest iceberg-rust. The minimum supported Rust version is now 1.88.</p>
+<p>Supported platforms include Spark 3.4.3, 3.5.4-3.5.7, and Spark 4.0.x with various JDK and Scala combinations.</p>
+<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="alternate"></link><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusio [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/52.0.0">DataFusion 52.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/">DataFusion 51.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>We continue to …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/52.0.0">DataFusion 52.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/">DataFusion 51.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>We continue to make significant performance improvements in DataFusion as explained below.</p>
+<h3 id="faster-case-expressions">Faster <code>CASE</code> Expressions<a class="headerlink" href="#faster-case-expressions" title="Permanent link">¶</a></h3>
+<p>DataFusion 52 has lookup-table-based evaluation for certain <code>CASE</code> expressions
+to avoid repeated evaluation for accelerating common ETL patterns such as</p>
+<pre><code class="language-sql">CASE company
+ WHEN 1 THEN 'Apple'
+ WHEN 5 THEN 'Samsung'
+ WHEN 2 THEN 'Motorola'
+ WHEN 3 THEN 'LG'
+ ELSE 'Other'
+END
+</code></pre>
+<p>This is the final work in our <code>CASE</code> performance epic (<a href="https://github.com/apache/datafusion/issues/18075">#18075</a>), which has
+improved <code>CASE</code> evaluation significantly. Related PRs <a href="https://github.com/apache/datafusion/pull/18183">#18183</a>. Thanks to
+<a href="https://github.com/rluvaton">rluvaton</a> and <a href="https://github.com/pepijnve">pepijnve</a> for the implementation.</p>
+<h3 id="minmax-aggregate-dynamic-filters"><code>MIN</code>/<code>MAX</code> Aggregate Dynamic Filters<a class="headerlink" href="#minmax-aggregate-dynamic-filters" title="Permanent link">¶</a></h3>
+<p>DataFusion now creates dynamic filters for queries with <code>MIN</code>/<code>MAX</code> aggregates
+that have filters, but no <code>GROUP BY</code>. These dynamic filters are used during scan
+to prune files and rows as tighter bounds are discovered during execution, as
+explained in the <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/#hash-join-dynamic-filters">Dynamic Filtering Blog</a>. For example, the following query:</p>
+<pre><code class="language-sql">SELECT min(l_shipdate)
+FROM lineitem
+WHERE l_returnflag = 'R';
+</code></pre>
+<p>Is now executed like this </p>
+<pre><code class="language-sql">SELECT min(l_shipdate)
+FROM lineitem
+-- '__current_min' is updated dynamically during execution
+WHERE l_returnflag = 'R' AND l_shipdate &lt; __current_min;
+</code></pre>
+<p>Thanks to <a href="https://github.com/2010YOUY01">2010YOUY01</a> for implementing this feature, with reviews from
+<a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/adriangb">adriangb</a>, and <a href="https://github.com/LiaCastaneda">LiaCastaneda</a>. Related PRs: <a href="https://github.com/apache/datafusion/pull/18644">#18644</a></p>
+<h3 id="new-merge-join">New Merge Join<a class="headerlink" href="#new-merge-join" title="Permanent link">¶</a></h3>
+<p>DataFusion 52 includes a rewrite of the sort-merge join (SMJ) operator, with
+speedups of three orders of magnitude in some pathological cases such as the
+case in <a href="https://github.com/apache/datafusion/issues/18487">#18487</a>, which also affected <a href="https://datafusion.apache.org/comet/">Apache Comet</a> workloads. Benchmarks in
+<a href="https://github.com/apache/datafusion/pull/18875">#18875</a> show dramatic gains for TPC-H Q21 (minutes to milliseconds) while
+leaving other queries unchanged or modestly faster. Thanks to <a href="https://github.com/mbutrovich">mbutrovich</a> for
+the implementation and reviews from <a href="https://github.com/Dandandan">Dandandan</a>.</p>
+<h3 id="caching-improvements">Caching Improvements<a class="headerlink" href="#caching-improvements" title="Permanent link">¶</a></h3>
+<p>This release also includes several additional caching improvements.</p>
+<p>A new statistics cache for File Metadata avoids repeatedly (re)calculating
+statistics for files. This significantly improves planning time
+for certain queries. You can see the contents of the new cache using the
+<a href="https://datafusion.apache.org/user-guide/cli/functions.html#statistics-cache">statistics_cache</a> function in the CLI:</p>
+<pre><code class="language-sql">select * from statistics_cache();
++------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
+| path | file_modified | file_size_bytes | e_tag | version | num_rows | num_columns | table_size_bytes | statistics_size_bytes |
++------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
+| .../hits.parquet | 2022-06-25T22:22:22 | 14779976446 | 0-5e24d1ee16380-370f48 | NULL | Exact(99997497) | 105 | Exact(36445943240) | 0 |
++------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
+</code></pre>
+<p>Thanks to <a href="https://github.com/bharath-techie">bharath-techie</a> and <a href="https://github.com/nuno-faria">nuno-faria</a> for implementing the statistics cache,
+with reviews from <a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/alamb">alamb</a>, and <a href="https://github.com/alchemist51">alchemist51</a>.
+Related PRs: <a href="https://github.com/apache/datafusion/pull/18971">#18971</a>, <a href="https://github.com/apache/datafusion/pull/19054">#19054</a></p>
+<p>A prefix-aware list-files cache accelerates evaluating partition predicates for
+Hive partitioned tables.</p>
+<pre><code class="language-sql">-- Read the hive partitioned dataset from Overture Maps (100s of Parquet files)
+CREATE EXTERNAL TABLE overturemaps
+STORED AS PARQUET LOCATION 's3://overturemaps-us-west-2/release/2025-12-17.0/';
+-- Find all files where the path contains `theme=base without requiring another LIST call
+select count(*) from overturemaps where theme='base';
+</code></pre>
+<p>You can see the
+contents of the new cache using the <a href="https://datafusion.apache.org/user-guide/cli/functions.html#list-files-cache">list_files_cache</a> function in the CLI:</p>
+<pre><code class="language-sql">create external table overturemaps
+stored as parquet
+location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure';
+0 row(s) fetched.
+&gt; select table, path, metadata_size_bytes, expires_in, unnest(metadata_list)['file_size_bytes'] as file_size_bytes, unnest(metadata_list)['e_tag'] as e_tag from list_files_cache() limit 10;
++--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
+| table | path | metadata_size_bytes | expires_in | file_size_bytes | e_tag |
++--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 999055952 | "35fc8fbe8400960b54c66fbb408c48e8-60" |
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 975592768 | "8a16e10b722681cdc00242564b502965-59" |
+...
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1016732378 | "6d70857a0473ed9ed3fc6e149814168b-61" |
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 991363784 | "c9cafb42fcbb413f851691c895dd7c2b-60" |
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1032469715 | "7540252d0d67158297a67038a3365e0f-62" |
++--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
+</code></pre>
+<p>Thanks to <a href="https://github.com/BlakeOrth">BlakeOrth</a> and <a href="https://github.com/Yuvraj-cyborg">Yuvraj-cyborg</a> for implementing the list-files cache work,
+with reviews from <a href="https://github.com/gabotechs">gabotechs</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/alchemist51">alchemist51</a>, <a href="https://github.com/martin-g">martin-g</a>, and <a href="https://github.com/BlakeOrth">BlakeOrth</a>.
+Related PRs: <a href="https://github.com/apache/datafusion/pull/18146">#18146</a>, <a href="https://github.com/apache/datafusion/pull/18855">#18855</a>, <a href="https://github.com/apache/datafusion/pull/19366">#19366</a>, <a href="https://github.com/apache/datafusion/pull/19298">#19298</a>, </p>
+<h3 id="improved-hash-join-filter-pushdown">Improved Hash Join Filter Pushdown<a class="headerlink" href="#improved-hash-join-filter-pushdown" title="Permanent link">¶</a></h3>
+<p>Starting in DataFusion 51, filtering information from <code>HashJoinExec</code> is passed
+dynamically to scans, as explained in the <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/#hash-join-dynamic-filters">Dynamic Filtering Blog</a> using a
+technique referred to as <a href="https://dl.acm.org/doi/10.1109/ICDE.2008.4497486">Sideways Information Passing</a> in Database research
+literature. The initial implementation passed min/max values for the join keys.
+DataFusion 52 extends the optimization (<a href="https://github.com/apache/datafusion/issues/17171">#17171</a> / <a href="https://github.com/apache/datafusion/pull/18393">#18393</a>) to pass the
+contents of the build side hash map. These filters are evaluated on the probe
+side scan to prune files, row groups, and individual rows. When the build side
+contains <code>20</code> or fewer rows (configurable) the contents of the hash map are
+transformed to an <code>IN</code> expression and used for <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.html">statistics-based pruning</a> which
+can avoid reading entire files or row groups that contain no matching join keys.
+Thanks to <a href="https://github.com/adriangb">adriangb</a> for implementing this feature, with reviews from
+<a href="https://github.com/LiaCastaneda">LiaCastaneda</a>, <a href="https://github.com/asolimando">asolimando</a>, <a href="https://github.com/comphead">comphead</a>, and <a href="https://github.com/mbutrovich">mbutrovich</a>.</p>
+<h2 id="major-features">Major Features ✨<a class="headerlink" href="#major-features" title="Permanent link">¶</a></h2>
+<h3 id="arrow-ipc-stream-file-support">Arrow IPC Stream file support<a class="headerlink" href="#arrow-ipc-stream-file-support" title="Permanent link">¶</a></h3>
+<p>DataFusion can now read Arrow IPC stream files (<a href="https://github.com/apache/datafusion/pull/18457">#18457</a>). This expands
+interoperability with systems that emit Arrow streams directly, making it
+simpler to ingest Arrow-native data without conversion. Thanks to <a href="https://github.com/corasaurus-hex">corasaurus-hex</a>
+for implementing this feature, with reviews from <a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/Jefffrey">Jefffrey</a>,
+<a href="https://github.com/jdcasale">jdcasale</a>, <a href="https://github.com/2010YOUY01">2010YOUY01</a>, and <a href="https://github.com/timsaucer">timsaucer</a>.</p>
+<pre><code class="language-sql">CREATE EXTERNAL TABLE ipc_events
+STORED AS ARROW
+LOCATION 's3://bucket/events.arrow';
+</code></pre>
+<p>Related PRs: <a href="https://github.com/apache/datafusion/pull/18457">#18457</a></p>
+<h3 id="more-extensible-sql-planning-with-relationplanner">More Extensible SQL Planning with <code>RelationPlanner</code><a class="headerlink" href="#more-extensible-sql-planning-with-relationplanner" title="Permanent link">¶</a></h3>
+<p>DataFusion now has an API for extending the SQL planner for relations, as
+explained in the <a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql/">Extending SQL in DataFusion Blog</a>. In addition to the existing
+expression and types extension points, this new API now allows extending <code>FROM</code>
+clauses. Using these APIs it is straightforward to provide SQL support for
+almost any dialect, including vendor-specific syntax. Example use cases include:</p>
+<pre><code class="language-sql">-- Postgres-style JSON operators
+SELECT payload-&gt;'user'-&gt;&gt;'id' FROM logs;
+-- MySQL-specific types
+SELECT DATETIME '2001-01-01 18:00:00';
+-- Statistical sampling
+SELECT * FROM sensor_data TABLESAMPLE BERNOULLI(10 PERCENT);
+</code></pre>
+<p>Thanks to <a href="https://github.com/geoffreyclaude">geoffreyclaude</a> for implementing relation planner extensions, and to
+<a href="https://github.com/theirix">theirix</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/NGA-TRAN">NGA-TRAN</a>, and <a href="https://github.com/gabotechs">gabotechs</a> for reviews and feedback on the
+design. Related PRs: <a href="https://github.com/apache/datafusion/pull/17843">#17843</a></p>
+<h3 id="expression-evaluation-pushdown-to-scans">Expression Evaluation Pushdown to Scans<a class="headerlink" href="#expression-evaluation-pushdown-to-scans" title="Permanent link">¶</a></h3>
+<p>DataFusion now pushes down expression evaluation into TableProviders using
+<a href="https://docs.rs/datafusion/52.0.0/datafusion/physical_expr_adapter/trait.PhysicalExprAdapter.html">PhysicalExprAdapter</a>, replacing the older SchemaAdapter approach (<a href="https://github.com/apache/datafusion/issues/14993">#14993</a>,
+<a href="https://github.com/apache/datafusion/issues/16800">#16800</a>). Predicates and expressions can now be customized for each
+individual file schema, opening additional optimization such as support for
+<a href="https://github.com/apache/datafusion/issues/16116">Variant shredding</a>. Thanks to <a href="https://github.com/adriangb">adriangb</a> for implementing PhysicalExprAdapter
+and reworking pushdown to use it. Related PRs: <a href="https://github.com/apache/datafusion/pull/18998">#18998</a>, <a href="https://github.com/apache/datafusion/pull/19345">#19345</a></p>
+<h3 id="sort-pushdown-to-scans">Sort Pushdown to Scans<a class="headerlink" href="#sort-pushdown-to-scans" title="Permanent link">¶</a></h3>
+<p>DataFusion can now push sorts into data sources (<a href="https://github.com/apache/datafusion/issues/10433">#10433</a>, <a href="https://github.com/apache/datafusion/pull/19064">#19064</a>).
+This allows table provider implementations to optimize based on
+sort knowledge for certain query patterns. For example, the provided Parquet
+data source now reverses the scan order of row groups and files when queried
+for the opposite of the file's natural sort (e.g. <code>DESC</code> when the files are sorted <code>ASC</code>).
+This reversal, combined with dynamic filtering, allows top-K queries with <code>LIMIT</code>
+on pre-sorted data to find the requested rows very quickly, pruning more files and row groups
+without even scanning them. We have seen a ~30x performance improvement on
+benchmark queries with pre-sorted data.
+Thanks to <a href="https://github.com/zhuqi-lucas">zhuqi-lucas</a> and <a href="https://github.com/xudong963">xudong963</a> for this feature, with reviews from
+<a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/adriangb">adriangb</a>, and <a href="https://github.com/alamb">alamb</a>.</p>
+<h3 id="tableprovider-supports-delete-and-update-statements"><code>TableProvider</code> supports <code>DELETE</code> and <code>UPDATE</code> statements<a class="headerlink" href="#tableprovider-supports-delete-and-update-statements" title="Permanent link">¶</a></h3>
+<p>The <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html">TableProvider</a> trait now includes hooks for <code>DELETE</code> and <code>UPDATE</code>
+statements and the basic MemTable implements them (<a href="https://github.com/apache/datafusion/pull/19142">#19142</a>). This lets
+downstream implementations and storage engines plug in their own mutation logic.
+See <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html#method.delete_from">TableProvider::delete_from</a> and <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html#method.update">TableProvider::update</a> for more details.</p>
+<p>Example:</p>
+<pre><code class="language-sql">DELETE FROM mem_table WHERE status = 'obsolete';
+</code></pre>
+<p>Thanks to <a href="https://github.com/ethan-tyler">ethan-tyler</a> for the implementation and <a href="https://github.com/alamb">alamb</a> and <a href="https://github.com/adriangb">adriangb</a> for
+reviews.</p>
+<h3 id="coalescebatchesexec-removed"><code>CoalesceBatchesExec</code> Removed<a class="headerlink" href="#coalescebatchesexec-removed" title="Permanent link">¶</a></h3>
+<p>The standalone <code>CoalesceBatchesExec</code> operator existed to ensure batches were
+large enough for subsequent vectorized execution, and was inserted after
+filter-like operators such as <code>FilterExec</code>, <code>HashJoinExec</code>, and
+<code>RepartitionExec</code>. However, using a separate operator also blocks other
+optimizations such as pushing <code>LIMIT</code> through joins and made optimizer rules
+more complex. In this release, we integrated the coalescing into the operators
+themselves (<a href="https://github.com/apache/datafusion/issues/18779">#18779</a>) using Arrow's <a href="https://docs.rs/arrow/57.2.0/arrow/compute/kernels/coalesce/">coalesce kernel</a>. This reduces plan
+complexity while keeping batch sizes efficient, and allows additional focused
+optimization work in the Arrow kernel, such as <a href="https://github.com/Dandandan">Dandandan</a>'s recent work with
+filtering in <a href="https://github.com/apache/arrow-rs/pull/8951">arrow-rs/#8951</a>.</p>
+<p>Related PRs: <a href="https://github.com/apache/datafusion/pull/18540">#18540</a>, <a href="https://github.com/apache/datafusion/pull/18604">#18604</a>, <a href="https://github.com/apache/datafusion/pull/18630">#18630</a>, <a href="https://github.com/apache/datafusion/pull/18972">#18972</a>, <a href="https://github.com/apache/datafusion/pull/19002">#19002</a>, <a href="https://github.com/apache/datafusion/pull/19342" [...]
+Thanks to <a href="https://github.com/Tim-53">Tim-53</a>, <a href="https://github.com/Dandandan">Dandandan</a>, <a href="https://github.com/jizezhang">jizezhang</a>, and <a href="https://github.com/feniljain">feniljain</a> for implementing
+this feature, with reviews from <a href="https://github.com/Jefffrey">Jefffrey</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/martin-g">martin-g</a>,
+<a href="https://github.com/geoffreyclaude">geoffreyclaude</a>, <a href="https://github.com/milenkovicm">milenkovicm</a>, and <a href="https://github.com/jizezhang">jizezhang</a>.</p>
+<h2 id="upgrade-guide-and-changelog">Upgrade Guide and Changelog<a class="headerlink" href="#upgrade-guide-and-changelog" title="Permanent link">¶</a></h2>
+<p>As always, upgrading to 52.0.0 should be straightforward for most users. Please review the
+<a href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade Guide</a>
+for details on breaking changes and code snippets to help with the transition.
+For a comprehensive list of all changes, please refer to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>.</p>
+<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses
+<a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
+create new, fast, data-centric systems such as databases, dataframe libraries,
+and machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion's primary
+design goal</a> is to accelerate the creation of other data-centric systems, it
+provides a reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe
+library</a>, <a href="https://datafusion.apache.org/python/">Python library</a>, and <a href="https://datafusion.apache.org/user-guide/cli/">command-line SQL tool</a>.</p>
+<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
+<p>DataFusion is not a project built or driven by a single person, company, or
+foundation. Rather, our community of users and contributors works together to
+build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us, we would love to have you. You can try out
+DataFusion on some of your own data and projects and let us know how it goes,
+contribute suggestions, documentation, bug reports, or a PR with documentation,
+tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="alternate"></link><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>Ge [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -283,245 +1938,7 @@ println!("{}", df.logical_plan().display_indent());
<li><strong>Try it out</strong>: Implement one of the extension points and share your experience</li>
<li><strong>File issues or join the conversation</strong>: <a href="https://github.com/apache/datafusion/">GitHub</a> for bugs and feature requests, <a href="https://datafusion.apache.org/contributor-guide/communication.html">Slack or Discord</a> for discussion</li>
</ul>
-<!-- Reference links --></content><category term="blog"></category></entry><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/08/datafusion-52.0.0" rel="alternate"></link><published>2026-01-08T00:00:00+00:00</published><updated>2026-01-08T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2026-01-08:/blog/2026/01/08/datafusion-52.0.0</id><summary type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/52.0.0">DataFusion 52.0.0</a>. This post highlights
-some of the major improvements since <a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/">DataFusion 51.0.0</a>. The complete list of
-changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
-making this release possible.</p>
-<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue to …</p></summary><content type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/52.0.0">DataFusion 52.0.0</a>. This post highlights
-some of the major improvements since <a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/">DataFusion 51.0.0</a>. The complete list of
-changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
-making this release possible.</p>
-<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue to make significant performance improvements in DataFusion as explained below.</p>
-<h3 id="faster-case-expressions">Faster <code>CASE</code> Expressions<a class="headerlink" href="#faster-case-expressions" title="Permanent link">¶</a></h3>
-<p>DataFusion 52 has lookup-table-based evaluation for certain <code>CASE</code> expressions
-to avoid repeated evaluation for accelerating common ETL patterns such as</p>
-<pre><code class="language-sql">CASE company
- WHEN 1 THEN 'Apple'
- WHEN 5 THEN 'Samsung'
- WHEN 2 THEN 'Motorola'
- WHEN 3 THEN 'LG'
- ELSE 'Other'
-END
-</code></pre>
-<p>This is the final work in our <code>CASE</code> performance epic (<a href="https://github.com/apache/datafusion/issues/18075">#18075</a>), which has
-improved <code>CASE</code> evaluation significantly. Related PRs <a href="https://github.com/apache/datafusion/pull/18183">#18183</a>. Thanks to
-<a href="https://github.com/rluvaton">rluvaton</a> and <a href="https://github.com/pepijnve">pepijnve</a> for the implementation.</p>
-<h3 id="minmax-aggregate-dynamic-filters"><code>MIN</code>/<code>MAX</code> Aggregate Dynamic Filters<a class="headerlink" href="#minmax-aggregate-dynamic-filters" title="Permanent link">¶</a></h3>
-<p>DataFusion now creates dynamic filters for queries with <code>MIN</code>/<code>MAX</code> aggregates
-that have filters, but no <code>GROUP BY</code>. These dynamic filters are used during scan
-to prune files and rows as tighter bounds are discovered during execution, as
-explained in the <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/#hash-join-dynamic-filters">Dynamic Filtering Blog</a>. For example, the following query:</p>
-<pre><code class="language-sql">SELECT min(l_shipdate)
-FROM lineitem
-WHERE l_returnflag = 'R';
-</code></pre>
-<p>Is now executed like this </p>
-<pre><code class="language-sql">SELECT min(l_shipdate)
-FROM lineitem
--- '__current_min' is updated dynamically during execution
-WHERE l_returnflag = 'R' AND l_shipdate &lt; __current_min;
-</code></pre>
-<p>Thanks to <a href="https://github.com/2010YOUY01">2010YOUY01</a> for implementing this feature, with reviews from
-<a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/adriangb">adriangb</a>, and <a href="https://github.com/LiaCastaneda">LiaCastaneda</a>. Related PRs: <a href="https://github.com/apache/datafusion/pull/18644">#18644</a></p>
-<h3 id="new-merge-join">New Merge Join<a class="headerlink" href="#new-merge-join" title="Permanent link">¶</a></h3>
-<p>DataFusion 52 includes a rewrite of the sort-merge join (SMJ) operator, with
-speedups of three orders of magnitude in some pathological cases such as the
-case in <a href="https://github.com/apache/datafusion/issues/18487">#18487</a>, which also affected <a href="https://datafusion.apache.org/comet/">Apache Comet</a> workloads. Benchmarks in
-<a href="https://github.com/apache/datafusion/pull/18875">#18875</a> show dramatic gains for TPC-H Q21 (minutes to milliseconds) while
-leaving other queries unchanged or modestly faster. Thanks to <a href="https://github.com/mbutrovich">mbutrovich</a> for
-the implementation and reviews from <a href="https://github.com/Dandandan">Dandandan</a>.</p>
-<h3 id="caching-improvements">Caching Improvements<a class="headerlink" href="#caching-improvements" title="Permanent link">¶</a></h3>
-<p>This release also includes several additional caching improvements.</p>
-<p>A new statistics cache for File Metadata avoids repeatedly (re)calculating
-statistics for files. This significantly improves planning time
-for certain queries. You can see the contents of the new cache using the
-<a href="https://datafusion.apache.org/user-guide/cli/functions.html#statistics-cache">statistics_cache</a> function in the CLI:</p>
-<pre><code class="language-sql">select * from statistics_cache();
-+------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
-| path | file_modified | file_size_bytes | e_tag | version | num_rows | num_columns | table_size_bytes | statistics_size_bytes |
-+------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
-| .../hits.parquet | 2022-06-25T22:22:22 | 14779976446 | 0-5e24d1ee16380-370f48 | NULL | Exact(99997497) | 105 | Exact(36445943240) | 0 |
-+------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
-</code></pre>
-<p>Thanks to <a href="https://github.com/bharath-techie">bharath-techie</a> and <a href="https://github.com/nuno-faria">nuno-faria</a> for implementing the statistics cache,
-with reviews from <a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/alamb">alamb</a>, and <a href="https://github.com/alchemist51">alchemist51</a>.
-Related PRs: <a href="https://github.com/apache/datafusion/pull/18971">#18971</a>, <a href="https://github.com/apache/datafusion/pull/19054">#19054</a></p>
-<p>A prefix-aware list-files cache accelerates evaluating partition predicates for
-Hive partitioned tables.</p>
-<pre><code class="language-sql">-- Read the hive partitioned dataset from Overture Maps (100s of Parquet files)
-CREATE EXTERNAL TABLE overturemaps
-STORED AS PARQUET LOCATION 's3://overturemaps-us-west-2/release/2025-12-17.0/';
--- Find all files where the path contains `theme=base without requiring another LIST call
-select count(*) from overturemaps where theme='base';
-</code></pre>
-<p>You can see the
-contents of the new cache using the <a href="https://datafusion.apache.org/user-guide/cli/functions.html#list-files-cache">list_files_cache</a> function in the CLI:</p>
-<pre><code class="language-sql">create external table overturemaps
-stored as parquet
-location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure';
-0 row(s) fetched.
-&gt; select table, path, metadata_size_bytes, expires_in, unnest(metadata_list)['file_size_bytes'] as file_size_bytes, unnest(metadata_list)['e_tag'] as e_tag from list_files_cache() limit 10;
-+--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
-| table | path | metadata_size_bytes | expires_in | file_size_bytes | e_tag |
-+--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 999055952 | "35fc8fbe8400960b54c66fbb408c48e8-60" |
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 975592768 | "8a16e10b722681cdc00242564b502965-59" |
-...
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1016732378 | "6d70857a0473ed9ed3fc6e149814168b-61" |
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 991363784 | "c9cafb42fcbb413f851691c895dd7c2b-60" |
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1032469715 | "7540252d0d67158297a67038a3365e0f-62" |
-+--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
-</code></pre>
-<p>Thanks to <a href="https://github.com/BlakeOrth">BlakeOrth</a> and <a href="https://github.com/Yuvraj-cyborg">Yuvraj-cyborg</a> for implementing the list-files cache work,
-with reviews from <a href="https://github.com/gabotechs">gabotechs</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/alchemist51">alchemist51</a>, <a href="https://github.com/martin-g">martin-g</a>, and <a href="https://github.com/BlakeOrth">BlakeOrth</a>.
-Related PRs: <a href="https://github.com/apache/datafusion/pull/18146">#18146</a>, <a href="https://github.com/apache/datafusion/pull/18855">#18855</a>, <a href="https://github.com/apache/datafusion/pull/19366">#19366</a>, <a href="https://github.com/apache/datafusion/pull/19298">#19298</a>, </p>
-<h3 id="improved-hash-join-filter-pushdown">Improved Hash Join Filter Pushdown<a class="headerlink" href="#improved-hash-join-filter-pushdown" title="Permanent link">¶</a></h3>
-<p>Starting in DataFusion 51, filtering information from <code>HashJoinExec</code> is passed
-dynamically to scans, as explained in the <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/#hash-join-dynamic-filters">Dynamic Filtering Blog</a> using a
-technique referred to as <a href="https://dl.acm.org/doi/10.1109/ICDE.2008.4497486">Sideways Information Passing</a> in Database research
-literature. The initial implementation passed min/max values for the join keys.
-DataFusion 52 extends the optimization (<a href="https://github.com/apache/datafusion/issues/17171">#17171</a> / <a href="https://github.com/apache/datafusion/pull/18393">#18393</a>) to pass the
-contents of the build side hash map. These filters are evaluated on the probe
-side scan to prune files, row groups, and individual rows. When the build side
-contains <code>20</code> or fewer rows (configurable) the contents of the hash map are
-transformed to an <code>IN</code> expression and used for <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.html">statistics-based pruning</a> which
-can avoid reading entire files or row groups that contain no matching join keys.
-Thanks to <a href="https://github.com/adriangb">adriangb</a> for implementing this feature, with reviews from
-<a href="https://github.com/LiaCastaneda">LiaCastaneda</a>, <a href="https://github.com/asolimando">asolimando</a>, <a href="https://github.com/comphead">comphead</a>, and <a href="https://github.com/mbutrovich">mbutrovich</a>.</p>
-<h2 id="major-features">Major Features ✨<a class="headerlink" href="#major-features" title="Permanent link">¶</a></h2>
-<h3 id="arrow-ipc-stream-file-support">Arrow IPC Stream file support<a class="headerlink" href="#arrow-ipc-stream-file-support" title="Permanent link">¶</a></h3>
-<p>DataFusion can now read Arrow IPC stream files (<a href="https://github.com/apache/datafusion/pull/18457">#18457</a>). This expands
-interoperability with systems that emit Arrow streams directly, making it
-simpler to ingest Arrow-native data without conversion. Thanks to <a href="https://github.com/corasaurus-hex">corasaurus-hex</a>
-for implementing this feature, with reviews from <a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/Jefffrey">Jefffrey</a>,
-<a href="https://github.com/jdcasale">jdcasale</a>, <a href="https://github.com/2010YOUY01">2010YOUY01</a>, and <a href="https://github.com/timsaucer">timsaucer</a>.</p>
-<pre><code class="language-sql">CREATE EXTERNAL TABLE ipc_events
-STORED AS ARROW
-LOCATION 's3://bucket/events.arrow';
-</code></pre>
-<p>Related PRs: <a href="https://github.com/apache/datafusion/pull/18457">#18457</a></p>
-<h3 id="more-extensible-sql-planning-with-relationplanner">More Extensible SQL Planning with <code>RelationPlanner</code><a class="headerlink" href="#more-extensible-sql-planning-with-relationplanner" title="Permanent link">¶</a></h3>
-<p>DataFusion now has an API for extending the SQL planner for relations, as
-explained in the <a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql/">Extending SQL in DataFusion Blog</a>. In addition to the existing
-expression and types extension points, this new API now allows extending <code>FROM</code>
-clauses. Using these APIs it is straightforward to provide SQL support for
-almost any dialect, including vendor-specific syntax. Example use cases include:</p>
-<pre><code class="language-sql">-- Postgres-style JSON operators
-SELECT payload-&gt;'user'-&gt;&gt;'id' FROM logs;
--- MySQL-specific types
-SELECT DATETIME '2001-01-01 18:00:00';
--- Statistical sampling
-SELECT * FROM sensor_data TABLESAMPLE BERNOULLI(10 PERCENT);
-</code></pre>
-<p>Thanks to <a href="https://github.com/geoffreyclaude">geoffreyclaude</a> for implementing relation planner extensions, and to
-<a href="https://github.com/theirix">theirix</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/NGA-TRAN">NGA-TRAN</a>, and <a href="https://github.com/gabotechs">gabotechs</a> for reviews and feedback on the
-design. Related PRs: <a href="https://github.com/apache/datafusion/pull/17843">#17843</a></p>
-<h3 id="expression-evaluation-pushdown-to-scans">Expression Evaluation Pushdown to Scans<a class="headerlink" href="#expression-evaluation-pushdown-to-scans" title="Permanent link">¶</a></h3>
-<p>DataFusion now pushes down expression evaluation into TableProviders using
-<a href="https://docs.rs/datafusion/52.0.0/datafusion/physical_expr_adapter/trait.PhysicalExprAdapter.html">PhysicalExprAdapter</a>, replacing the older SchemaAdapter approach (<a href="https://github.com/apache/datafusion/issues/14993">#14993</a>,
-<a href="https://github.com/apache/datafusion/issues/16800">#16800</a>). Predicates and expressions can now be customized for each
-individual file schema, opening additional optimization such as support for
-<a href="https://github.com/apache/datafusion/issues/16116">Variant shredding</a>. Thanks to <a href="https://github.com/adriangb">adriangb</a> for implementing PhysicalExprAdapter
-and reworking pushdown to use it. Related PRs: <a href="https://github.com/apache/datafusion/pull/18998">#18998</a>, <a href="https://github.com/apache/datafusion/pull/19345">#19345</a></p>
-<h3 id="sort-pushdown-to-scans">Sort Pushdown to Scans<a class="headerlink" href="#sort-pushdown-to-scans" title="Permanent link">¶</a></h3>
-<p>DataFusion can now push sorts into data sources (<a href="https://github.com/apache/datafusion/issues/10433">#10433</a>, <a href="https://github.com/apache/datafusion/pull/19064">#19064</a>).
-This allows table provider implementations to optimize based on
-sort knowledge for certain query patterns. For example, the provided Parquet
-data source now reverses the scan order of row groups and files when queried
-for the opposite of the file's natural sort (e.g. <code>DESC</code> when the files are sorted <code>ASC</code>).
-This reversal, combined with dynamic filtering, allows top-K queries with <code>LIMIT</code>
-on pre-sorted data to find the requested rows very quickly, pruning more files and row groups
-without even scanning them. We have seen a ~30x performance improvement on
-benchmark queries with pre-sorted data.
-Thanks to <a href="https://github.com/zhuqi-lucas">zhuqi-lucas</a> and <a href="https://github.com/xudong963">xudong963</a> for this feature, with reviews from
-<a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/adriangb">adriangb</a>, and <a href="https://github.com/alamb">alamb</a>.</p>
-<h3 id="tableprovider-supports-delete-and-update-statements"><code>TableProvider</code> supports <code>DELETE</code> and <code>UPDATE</code> statements<a class="headerlink" href="#tableprovider-supports-delete-and-update-statements" title="Permanent link">¶</a></h3>
-<p>The <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html">TableProvider</a> trait now includes hooks for <code>DELETE</code> and <code>UPDATE</code>
-statements and the basic MemTable implements them (<a href="https://github.com/apache/datafusion/pull/19142">#19142</a>). This lets
-downstream implementations and storage engines plug in their own mutation logic.
-See <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html#method.delete_from">TableProvider::delete_from</a> and <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html#method.update">TableProvider::update</a> for more details.</p>
-<p>Example:</p>
-<pre><code class="language-sql">DELETE FROM mem_table WHERE status = 'obsolete';
-</code></pre>
-<p>Thanks to <a href="https://github.com/ethan-tyler">ethan-tyler</a> for the implementation and <a href="https://github.com/alamb">alamb</a> and <a href="https://github.com/adriangb">adriangb</a> for
-reviews.</p>
-<h3 id="coalescebatchesexec-removed"><code>CoalesceBatchesExec</code> Removed<a class="headerlink" href="#coalescebatchesexec-removed" title="Permanent link">¶</a></h3>
-<p>The standalone <code>CoalesceBatchesExec</code> operator existed to ensure batches were
-large enough for subsequent vectorized execution, and was inserted after
-filter-like operators such as <code>FilterExec</code>, <code>HashJoinExec</code>, and
-<code>RepartitionExec</code>. However, using a separate operator also blocks other
-optimizations such as pushing <code>LIMIT</code> through joins and made optimizer rules
-more complex. In this release, we integrated the coalescing into the operators
-themselves (<a href="https://github.com/apache/datafusion/issues/18779">#18779</a>) using Arrow's <a href="https://docs.rs/arrow/57.2.0/arrow/compute/kernels/coalesce/">coalesce kernel</a>. This reduces plan
-complexity while keeping batch sizes efficient, and allows additional focused
-optimization work in the Arrow kernel, such as <a href="https://github.com/Dandandan">Dandandan</a>'s recent work with
-filtering in <a href="https://github.com/apache/arrow-rs/pull/8951">arrow-rs/#8951</a>.</p>
-<p>Related PRs: <a href="https://github.com/apache/datafusion/pull/18540">#18540</a>, <a href="https://github.com/apache/datafusion/pull/18604">#18604</a>, <a href="https://github.com/apache/datafusion/pull/18630">#18630</a>, <a href="https://github.com/apache/datafusion/pull/18972">#18972</a>, <a href="https://github.com/apache/datafusion/pull/19002">#19002</a>, <a href="https://github.com/apache/datafusion/pull/19342" [...]
-Thanks to <a href="https://github.com/Tim-53">Tim-53</a>, <a href="https://github.com/Dandandan">Dandandan</a>, <a href="https://github.com/jizezhang">jizezhang</a>, and <a href="https://github.com/feniljain">feniljain</a> for implementing
-this feature, with reviews from <a href="https://github.com/Jefffrey">Jefffrey</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/martin-g">martin-g</a>,
-<a href="https://github.com/geoffreyclaude">geoffreyclaude</a>, <a href="https://github.com/milenkovicm">milenkovicm</a>, and <a href="https://github.com/jizezhang">jizezhang</a>.</p>
-<h2 id="upgrade-guide-and-changelog">Upgrade Guide and Changelog<a class="headerlink" href="#upgrade-guide-and-changelog" title="Permanent link">¶</a></h2>
-<p>As always, upgrading to 52.0.0 should be straightforward for most users. Please review the
-<a href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade Guide</a>
-for details on breaking changes and code snippets to help with the transition.
-For a comprehensive list of all changes, please refer to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>.</p>
-<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
-<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses
-<a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
-create new, fast, data-centric systems such as databases, dataframe libraries,
-and machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion's primary
-design goal</a> is to accelerate the creation of other data-centric systems, it
-provides a reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe
-library</a>, <a href="https://datafusion.apache.org/python/">Python library</a>, and <a href="https://datafusion.apache.org/user-guide/cli/">command-line SQL tool</a>.</p>
-<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
-<p>DataFusion is not a project built or driven by a single person, company, or
-foundation. Rather, our community of users and contributors works together to
-build a shared technology that none of us could have built alone.</p>
-<p>If you are interested in joining us, we would love to have you. You can try out
-DataFusion on some of your own data and projects and let us know how it goes,
-contribute suggestions, documentation, bug reports, or a PR with documentation,
-tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="alternate"></link><published>2025-12-15T00:00:00+00:00</published><updated>202 [...]
+<!-- Reference links --></content><category term="blog"></category></entry><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="alternate"></link><published>2025-12-15T00:00:00+00:00</published><updated>2025-12-15T00:00:00+00:00</updated><author><name>Gene Bordegaray</name></author><id>tag:datafusion.apache.org,2025-12-15:/blog/202 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6545,7 +7962,7 @@ to register the view and then use it in another place:</p>
<pre><code class="language-python">ctx.register_view("view1", df1)
</code></pre>
<p>And then in another portion of your code which has access to the same session context
-you can retrive the DataFrame with:</p>
+you can retrieve the DataFrame with:</p>
<pre><code>df2 = ctx.table("view1")
</code></pre>
<h2 id="asynchronous-iteration-of-record-batches">Asynchronous Iteration of Record Batches<a class="headerlink" href="#asynchronous-iteration-of-record-batches" title="Permanent link">¶</a></h2>
@@ -8179,7 +9596,7 @@ consistent method for exposing these data structures across libraries.</p>
<p>In <a href="https://github.com/apache/datafusion-python/pull/825">PR #825</a>, we introduced support for both importing and exporting Arrow data in
<code>datafusion-python</code>. With this improvement, you can now use a single function call to import
a table from <strong>any</strong> Python library that implements the <a href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html">Arrow PyCapsule Interface</a>.
-Many popular libaries, such as <a href="https://pandas.pydata.org/">Pandas</a> and <a href="https://pola.rs/">Polars</a>
+Many popular libraries, such as <a href="https://pandas.pydata.org/">Pandas</a> and <a href="https://pola.rs/">Polars</a>
already support these interfaces.</p>
<p>Suppose you have a Pandas and Polars DataFrames named <code>df_pandas</code> or <code>df_polars</code>, respectively:</p>
<pre><code class="language-python">ctx = SessionContext()
@@ -8239,7 +9656,7 @@ of the blog post describing how these enhancements can lead to 20-200% performan
gains in some tests.</p>
<p>During our testing we identified some cases where we needed to adjust workflows to
account for the fact that StringView is now the default type for string based operations.
-First, when performing manipulations on string objects there is a perfomance loss when
+First, when performing manipulations on string objects there is a performance loss when
needing to cast from string to string view or vice versa. To reap the best performance,
ideally all of your string type data will use StringView. For most users this should be
transparent. However if you specify a schema for reading or creating data, then you
@@ -8476,7 +9893,7 @@ than a join can be significantly faster. This is worth profiling for your specif
<p>I have a DataFrame with many values that I want to aggregate. I have already analyzed it and
determined there is a noise level below which I do not want to include in my analysis. I want to
compute a sum of only values that are above my noise threshold.</p>
-<p>This can be done fairly easy without leaning on a User Defined Aggegate Function (UDAF). You can
+<p>This can be done fairly easy without leaning on a User Defined Aggregate Function (UDAF). You can
simply filter the DataFrame and then aggregate using the built-in <code>sum</code> function. Here, we
demonstrate doing this as a UDF primarily as an example of how to write UDAFs. We will use the
PyArrow compute approach.</p>
@@ -8637,7 +10054,7 @@ Python, is to primarily demonstrate how to make the Python to Rust with Python w
transition. In the second implementation you can see how we can iterate through all of the arrays
ourselves.</p>
<p>In this first example, we are hard coding the values of interest, but in the following section
-we demonstrate passing these in during initalization.</p>
+we demonstrate passing these in during initialization.</p>
<pre><code class="language-rust">#[pyfunction]
pub fn tuple_filter_fn(
py: Python&lt;'_&gt;,
@@ -8860,13 +10277,13 @@ how much they have ordered total. We want to ignore small orders, which we defin
import pyarrow as pa
import pyarrow.compute as pc
-IGNORE_THESHOLD = 5000.0
+IGNORE_THRESHOLD = 5000.0
class AboveThresholdAccum(Accumulator):
def __init__(self) -&gt; None:
self._sum = 0.0
def update(self, values: pa.Array) -&gt; None:
- over_threshold = pc.greater(values, pa.scalar(IGNORE_THESHOLD))
+ over_threshold = pc.greater(values, pa.scalar(IGNORE_THRESHOLD))
sum_above = pc.sum(values.filter(over_threshold)).as_py()
if sum_above is None:
sum_above = 0.0
@@ -9441,8 +10858,8 @@ Figure 1 illustrates the difference between the output of both string representa
<h1 id="when-to-gc">When to GC?<a class="headerlink" href="#when-to-gc" title="Permanent link">¶</a></h1>
<p>Zero-copy <code>take/filter</code> is great for generating large arrays quickly, but it is suboptimal for highly selective filters, where most of the strings are filtered out. When the cardinality drops, StringViewArray buffers become sparse—only a small subset of the bytes in the buffer’s memory are referred to by any <code>view</code>. This leads to excessive memory usage, especially in a <a href="https://github.com/apache/datafusion/issues/11628"> [...]
<p>To release unused memory, we implemented a <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html#method.gc">garbage collection (GC)</a> routine to consolidate the data into a new buffer to release the old sparse buffer(s). As the GC operation copies strings, similarly to StringArray, we must be careful about when to call it. If we call GC too early, we cause unnecessary copying, losing much of the benefit of StringViewArray. If we call GC [...]
-<p><code>arrow-rs</code> implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/coalesce_batches/struct.CoalesceBatchesExec.html"><code>CoalseceBatchesExec</code></a> operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, whi [...]
-We, therefore,<a href="https://github.com/apache/datafusion/pull/11587"> implemented the GC procedure</a> inside <code>CoalseceBatchesExec</code>[^5] with a heuristic that estimates when the buffers are too sparse.</p>
+<p><code>arrow-rs</code> implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/coalesce_batches/struct.CoalesceBatchesExec.html"><code>CoalesceBatchesExec</code></a> operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, whi [...]
+We, therefore,<a href="https://github.com/apache/datafusion/pull/11587"> implemented the GC procedure</a> inside <code>CoalesceBatchesExec</code>[^5] with a heuristic that estimates when the buffers are too sparse.</p>
<h2 id="the-art-of-function-inlining-not-too-much-not-too-little">The art of function inlining: not too much, not too little<a class="headerlink" href="#the-art-of-function-inlining-not-too-much-not-too-little" title="Permanent link">¶</a></h2>
<p>Like string inlining, <em>function</em> inlining is the process of embedding a short function into the caller to avoid the overhead of function calls (caller/callee save).
Usually, the Rust compiler does a good job of deciding when to inline. However, it is possible to override its default using the <a href="https://doc.rust-lang.org/reference/attributes/codegen.html#the-inline-attribute"><code>#[inline(always)]</code> directive</a>.
@@ -9641,7 +11058,7 @@ to their Rust counterparts.</li>
<p>The most significant difference is that we have added wrapper functions and classes for most of the
user facing interface. These wrappers, written in Python, contain both documentation and type
annotations.</p>
-<p>This documenation is now available on the <a href="https://datafusion.apache.org/python/autoapi/datafusion/index.html">DataFusion in Python API</a> website. There you can browse
+<p>This documentation is now available on the <a href="https://datafusion.apache.org/python/autoapi/datafusion/index.html">DataFusion in Python API</a> website. There you can browse
the available functions and classes to see the breadth of available functionality.</p>
<p>Modern IDEs use language servers such as
<a href="https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance">Pylance</a> or
@@ -10565,7 +11982,7 @@ LIMIT 3;
3 rows in set. Query took 0.053 seconds.
</code></pre>
<h3 id="growth-of-datafusion">Growth of DataFusion 📈<a class="headerlink" href="#growth-of-datafusion" title="Permanent link">¶</a></h3>
-<p>DataFusion has been appearing more publically in the wild. For example
+<p>DataFusion has been appearing more publicly in the wild. For example
* New projects built using DataFusion such as <a href="https://lancedb.com/">lancedb</a>, <a href="https://glaredb.com/">GlareDB</a>, <a href="https://www.arroyo.dev/">Arroyo</a>, and <a href="https://github.com/cmu-db/optd">optd</a>.
* Public talks such as <a href="https://www.youtube.com/watch?v=AJU9rdRNk9I">Apache Arrow Datafusion: Vectorized
Execution Framework For Maximum Performance</a> in <a href="https://www.bagevent.com/event/8432178">CommunityOverCode Asia 2023</a>
@@ -11317,7 +12734,7 @@ required synchronous access to all relevant catalog information.</p>
<li>Automatic coercions ast between Date and Timestamp <a href="https://github.com/apache/arrow-datafusion/issues/4726">#4726</a></li>
<li>Support type coercion for timestamp and utf8 <a href="https://github.com/apache/arrow-datafusion/issues/4312">#4312</a></li>
<li>Full support for time32 and time64 literal values (<code>ScalarValue</code>) <a href="https://github.com/apache/arrow-datafusion/issues/4156">#4156</a></li>
-<li>New functions, incuding <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
+<li>New functions, including <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
<li>Compressed CSV/JSON support <a href="https://github.com/apache/arrow-datafusion/issues/3642">#3642</a></li>
</ul>
<p>The community has also invested in new <a href="https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/tests/sqllogictests/README.md">sqllogic based</a> tests to keep improving DataFusion's quality with less effort.</p>
@@ -12169,7 +13586,7 @@ git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc
<li>Switch from <code>std::sync::Mutex</code> to <code>parking_lot::Mutex</code> <a href="https://github.com/apache/arrow-datafusion/pull/1720">#1720</a></li>
<li>New Features</li>
<li>Support for memory tracking and spilling to disk<ul>
-<li>MemoryMananger and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
+<li>MemoryManager and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
<li>Out of core sort <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
<li>New metrics</li>
<li><code>Gauge</code> and <code>CurrentMemoryUsage</code> <a href="https://github.com/apache/arrow-datafusion/pull/1682">#1682</a></li>
diff --git a/blog/feeds/blog.atom.xml b/blog/feeds/blog.atom.xml
index 9ca668b..7f8f4a9 100644
--- a/blog/feeds/blog.atom.xml
+++ b/blog/feeds/blog.atom.xml
@@ -1,5 +1,1660 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/blog.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-01-12T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https://datafusion.apache.org/blog/2026/01/12/e [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/blog.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-03-20T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</title><link href="https://datafusion.a [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make. This sounds obvious, but making it happen in practice requires careful engineering at every layer of the query engine. <a href="https://datafusion.apache.org/">Apache DataFusion</a> achieves this through a multi-layer <strong>pruning pipeline</strong> — a series of stages that progressively narrow down the data before decoding a single row.</p>
+<p>In this post, we describe a new optimization called <strong>limit pruning</strong> that makes this pipeline aware of SQL <code>LIMIT</code> clauses. By identifying row groups where <em>every</em> row is guaranteed to match the predicate, DataFusion can satisfy a <code>LIMIT</code> query without ever touching partially matching row groups — eliminating wasted I/O entirely.</p>
+<p>For example, given a query like:</p>
+<pre><code class="language-sql">SELECT * FROM tracking_data
+WHERE species LIKE 'Alpine%' AND s &gt;= 50
+LIMIT 3
+</code></pre>
+<p>If the pruning pipeline already knows that certain row groups fully satisfy the <code>WHERE</code> clause, those groups alone may contain enough rows to fill the <code>LIMIT</code> — making it unnecessary to scan anything else.</p>
+<p>This work was inspired by the "Pruning for LIMIT Queries" section of Snowflake's paper <a href="https://arxiv.org/pdf/2504.11540"><em>Pruning in Snowflake: Working Smarter, Not Harder</em></a>.</p>
+<h2 id="datafusions-pruning-pipeline">DataFusion's Pruning Pipeline<a class="headerlink" href="#datafusions-pruning-pipeline" title="Permanent link">¶</a></h2>
+<p>Before diving into limit pruning, let's understand the full pruning pipeline. DataFusion scans Parquet data through a series of increasingly fine-grained filters, each one eliminating data so the next stage processes less:</p>
+<figure>
+<img alt="Three phases of DataFusion's pruning pipeline" src="/blog/images/limit-pruning/pruning-phases.svg" width="80%"/>
+<figcaption>Figure 1: The three phases of DataFusion's pruning pipeline — from directories down to individual rows.</figcaption>
+</figure>
+<h3 id="phase-1-high-level-discovery">Phase 1: High-Level Discovery<a class="headerlink" href="#phase-1-high-level-discovery" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong>Partition Pruning</strong>: The <a href="https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html">ListingTable</a> component evaluates filters that depend only on partition columns — things like <code>year</code>, <code>month</code>, or <code>region</code> encoded in directory paths (e.g., <code>s3://data/year=2024/month=01/</code>). Irrelevant directories are elimin [...]
+<li><strong>File Stats Pruning</strong>: The <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.FilePruner.html">FilePruner</a> checks file-level min/max and null-count statistics. If these statistics prove that a file cannot satisfy the predicate, we drop it entirely — no need to read row group metadata.</li>
+</ul>
+<h3 id="phase-2-row-group-statistics">Phase 2: Row Group Statistics<a class="headerlink" href="#phase-2-row-group-statistics" title="Permanent link">¶</a></h3>
+<p>For each surviving file, DataFusion reads row group metadata and potentially <a href="https://parquet.apache.org/docs/file-format/bloomfilter/">bloom filters</a> and classifies each row group into one of three states (the example data shown in the figures below, such as "Snow Vole" and "Alpine Ibex", is adapted from the <a href="https://arxiv.org/pdf/2504.11540">Snowflake pruning paper</a>):</p>
+<figure>
+<img alt="Row group classification: not matching, partially matching, fully matching" src="/blog/images/limit-pruning/row-group-states.svg" width="80%"/>
+<figcaption>Figure 2: Row groups are classified into three states based on their statistics.</figcaption>
+</figure>
+<ul>
+<li><strong>Not Matching (Skipped)</strong>: Statistics prove no rows can match. The row group is ignored completely.</li>
+<li><strong>Partially Matching</strong>: Statistics cannot rule out matching rows, but also cannot guarantee them. These groups might be scanned and verified row by row later.</li>
+<li><strong>Fully Matching</strong>: Statistics prove that <em>every single row</em> in the group satisfies the predicate. This state is key to making limit pruning possible.</li>
+</ul>
+<h3 id="phase-3-granular-pruning">Phase 3: Granular Pruning<a class="headerlink" href="#phase-3-granular-pruning" title="Permanent link">¶</a></h3>
+<p>The final phase goes even deeper:</p>
+<ul>
+<li><strong><a href="https://parquet.apache.org/docs/file-format/pageindex/">Page Index Pruning</a></strong>: Parquet pages have their own min/max statistics. DataFusion uses these to skip individual data pages within a surviving row group.</li>
+<li><strong><a href="https://arrow.apache.org/blog/2025/12/11/parquet-late-materialization-deep-dive/">Late Materialization</a> (Row Filtering)</strong>: Instead of decoding all columns at once, DataFusion decodes the cheapest, most selective columns first. It filters rows using those columns, then only decodes the remaining columns for surviving rows.</li>
+</ul>
+<h2 id="the-problem-limit-was-ignored">The Problem: LIMIT Was Ignored<a class="headerlink" href="#the-problem-limit-was-ignored" title="Permanent link">¶</a></h2>
+<p>Before limit pruning, all of these stages worked well — but the pruning pipeline had <strong>no awareness of <code>LIMIT</code></strong>. Consider a query like:</p>
+<pre><code class="language-sql">SELECT * FROM tracking_data
+WHERE species LIKE 'Alpine%' AND s &gt;= 50
+LIMIT 3
+</code></pre>
+<p>Even when fully matched row groups alone contain enough rows to satisfy the <code>LIMIT</code>, DataFusion would still decode partially matching groups and filter out rows that did not match, wasting resources decoding rows just to immediately discard them.</p>
+<figure>
+<img alt="Traditional pruning decodes partially matching groups with no LIMIT awareness" src="/blog/images/limit-pruning/wasted-io.svg" width="80%"/>
+<figcaption>Figure 3: Without limit awareness, partially matching groups are scanned and filtered even when fully matched groups already have enough rows. The left section shows 5 fully matched rows (enough to satisfy <code>LIMIT 5</code>), while the right section with the dashed red border represents a partially matching group that is still decoded — wasting CPU and I/O on rows that may not match at all.</figcaption>
+</figure>
+<p>If five fully matched rows in a fully matched group already satisfy <code>LIMIT 5</code>, why bother decoding groups where we're not even sure any rows qualify?</p>
+<h2 id="the-solution-limit-aware-pruning">The Solution: Limit-Aware Pruning<a class="headerlink" href="#the-solution-limit-aware-pruning" title="Permanent link">¶</a></h2>
+<p>The solution adds a new step in the pruning pipeline — right after row group pruning and before page index pruning:</p>
+<figure>
+<img alt="Pruning pipeline with limit pruning highlighted" src="/blog/images/limit-pruning/pruning-pipeline.svg" width="80%"/>
+<figcaption>Figure 4: Limit pruning is inserted between row group and page index pruning.</figcaption>
+</figure>
+<p>The idea is simple: <strong>if fully matched row groups already contain enough rows to satisfy the <code>LIMIT</code>, rewrite the access plan to scan only those groups and skip everything else.</strong></p>
+<p>This optimization is applied only when the query is a pure limit query with no <code>ORDER BY</code>, because reordering which groups we scan could change the output ordering of the results. In the implementation, this check is expressed as:</p>
+<pre><code class="language-rust">// Prune by limit if limit is set and order is not sensitive
+if let (Some(limit), false) = (limit, preserve_order) {
+ row_groups.prune_by_limit(limit, rg_metadata, &amp;file_metrics);
+}
+</code></pre>
+<h2 id="mechanism-detecting-fully-matched-row-groups">Mechanism: Detecting Fully Matched Row Groups<a class="headerlink" href="#mechanism-detecting-fully-matched-row-groups" title="Permanent link">¶</a></h2>
+<p>The core insight is <strong>predicate negation</strong>. To determine if every row in a row group satisfies the predicate, we:</p>
+<ol>
+<li>Negate the original predicate</li>
+<li>Simplify the negated expression</li>
+<li>Evaluate the negation against the row group's statistics</li>
+<li>If the negation is <em>pruned</em> (proven impossible), then the original predicate holds for every row</li>
+</ol>
+<p>Since DataFusion already had expression simplification (step 2) and statistics-based pruning (step 3), implementing this was relatively straightforward — the key addition was composing these existing capabilities with predicate negation.</p>
+<figure>
+<img alt="Fully matched detection via predicate negation" src="/blog/images/limit-pruning/fully-matched-detection.svg" width="80%"/>
+<figcaption>Figure 5: If the negated predicate is impossible according to row group stats, all rows must match the original predicate.</figcaption>
+</figure>
+<p>In DataFusion's codebase, this logic lives in <code>identify_fully_matched_row_groups</code> (<a href="https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/row_group_filter.rs">row_group_filter.rs</a>):</p>
+<pre><code class="language-rust">fn identify_fully_matched_row_groups(
+ &amp;mut self,
+ candidate_row_group_indices: &amp;[usize],
+ arrow_schema: &amp;Schema,
+ parquet_schema: &amp;SchemaDescriptor,
+ groups: &amp;[RowGroupMetaData],
+ predicate: &amp;PruningPredicate,
+ metrics: &amp;ParquetFileMetrics,
+) {
+ // Create the inverted predicate: NOT(original)
+ let inverted_expr = Arc::new(NotExpr::new(
+ Arc::clone(predicate.orig_expr()),
+ ));
+
+ // Simplify: e.g., NOT(c1 = 0) → c1 != 0
+ let simplifier = PhysicalExprSimplifier::new(arrow_schema);
+ let Ok(inverted_expr) = simplifier.simplify(inverted_expr) else {
+ return;
+ };
+
+ let Ok(inverted_predicate) = PruningPredicate::try_new(
+ inverted_expr,
+ Arc::clone(predicate.schema()),
+ ) else {
+ return;
+ };
+
+ // Evaluate inverted predicate against row group stats
+ let Ok(inverted_values) =
+ inverted_predicate.prune(&amp;inverted_pruning_stats)
+ else {
+ return;
+ };
+
+ for (i, &amp;original_idx) in
+ candidate_row_group_indices.iter().enumerate()
+ {
+ // If negation is pruned (false), all rows match original
+ if !inverted_values[i] {
+ self.is_fully_matched[original_idx] = true;
+ }
+ }
+}
+</code></pre>
+<h2 id="mechanism-rewriting-the-access-plan">Mechanism: Rewriting the Access Plan<a class="headerlink" href="#mechanism-rewriting-the-access-plan" title="Permanent link">¶</a></h2>
+<p>Once we know which row groups are fully matched, the limit pruning algorithm is straightforward:</p>
+<figure>
+<img alt="Limit pruning access plan rewrite algorithm" src="/blog/images/limit-pruning/limit-rewrite-algorithm.svg" width="80%"/>
+<figcaption>Figure 6: The algorithm iterates fully matched groups, accumulating row counts until the limit is satisfied.</figcaption>
+</figure>
+<p>The implementation in <code>prune_by_limit</code> (<a href="https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/row_group_filter.rs">row_group_filter.rs</a>):</p>
+<pre><code class="language-rust">pub fn prune_by_limit(
+ &amp;mut self,
+ limit: usize,
+ rg_metadata: &amp;[RowGroupMetaData],
+ metrics: &amp;ParquetFileMetrics,
+) {
+ let mut fully_matched_indexes: Vec&lt;usize&gt; = Vec::new();
+ let mut fully_matched_rows: usize = 0;
+
+ for &amp;idx in self.access_plan.row_group_indexes().iter() {
+ if self.is_fully_matched[idx] {
+ fully_matched_indexes.push(idx);
+ fully_matched_rows += rg_metadata[idx].num_rows() as usize;
+ if fully_matched_rows &gt;= limit {
+ break;
+ }
+ }
+ }
+
+ // Rewrite the plan if we have enough rows
+ if fully_matched_rows &gt;= limit {
+ let mut new_plan = ParquetAccessPlan::new_none(rg_metadata.len());
+ for &amp;idx in &amp;fully_matched_indexes {
+ new_plan.scan(idx);
+ }
+ self.access_plan = new_plan;
+ }
+}
+</code></pre>
+<p>Key properties of this algorithm:</p>
+<ul>
+<li>It preserves the original row group ordering</li>
+<li>If fully matched groups don't have enough rows, the plan is unchanged — no harm done</li>
+<li>The cost is minimal: a single pass over the row group list</li>
+</ul>
+<h2 id="case-study-alpine-wildlife-query">Case Study: Alpine Wildlife Query<a class="headerlink" href="#case-study-alpine-wildlife-query" title="Permanent link">¶</a></h2>
+<p>Let's walk through a concrete example adapted from the <a href="https://arxiv.org/pdf/2504.11540">Snowflake pruning paper</a>. Given a wildlife tracking dataset with four row groups:</p>
+<pre><code class="language-sql">SELECT * FROM tracking_data
+WHERE species LIKE 'Alpine%' AND s &gt;= 50
+LIMIT 3
+</code></pre>
+<table class="table">
+<thead>
+<tr>
+<th>Row Group</th>
+<th>Species Range</th>
+<th>S Range</th>
+<th>State</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>RG1</td>
+<td>Snow Vole, Brown Bear, Gray Wolf</td>
+<td>7–133</td>
+<td><strong>Not Matching</strong> (no 'Alpine%')</td>
+</tr>
+<tr>
+<td>RG2</td>
+<td>Lynx, Red Fox, Alpine Bat</td>
+<td>6–71</td>
+<td><strong>Partially Matching</strong></td>
+</tr>
+<tr>
+<td>RG3</td>
+<td>Alpine Ibex, Alpine Goat, Alpine Sheep</td>
+<td>76–101</td>
+<td><strong>Fully Matching</strong></td>
+</tr>
+<tr>
+<td>RG4</td>
+<td>Mixed species</td>
+<td>Mixed</td>
+<td><strong>Partially Matching</strong></td>
+</tr>
+</tbody>
+</table>
+<figure>
+<img alt="Before and after limit pruning comparison" src="/blog/images/limit-pruning/before-after.svg" width="80%"/>
+<figcaption>Figure 7: Before limit pruning, RG2 is scanned for zero hits. After limit pruning, only RG3 is scanned.</figcaption>
+</figure>
+<p><strong>Before limit pruning</strong>: DataFusion scans RG2 (0 hits — wasted I/O), then RG3 (3 hits, early return). RG2 was decoded entirely for nothing.</p>
+<p><strong>With limit pruning</strong>: The system detects that RG3 has 3 fully matched rows, which satisfies <code>LIMIT 3</code>. It rewrites the access plan to scan only RG3, skipping RG2 and RG4 entirely. One row group scanned. Zero waste.</p>
+<h2 id="observing-limit-pruning-via-metrics">Observing Limit Pruning via Metrics<a class="headerlink" href="#observing-limit-pruning-via-metrics" title="Permanent link">¶</a></h2>
+<p>DataFusion exposes limit pruning activity through query metrics. When running a query with <code>EXPLAIN ANALYZE</code>, you will see entries like:</p>
+<pre><code>row_groups_pruned_statistics=4 total → 3 matched -&gt; 1 fully matched
+limit_pruned_row_groups=3 total → 1 matched
+</code></pre>
+<p>This tells us:
+- 4 row groups were evaluated, 3 survived statistics pruning, 1 was identified as fully matching
+- Of the 3 row groups that entered limit pruning, only 1 survived — 2 were pruned by the limit optimization</p>
+<h2 id="future-directions">Future Directions<a class="headerlink" href="#future-directions" title="Permanent link">¶</a></h2>
+<p>There are two natural extensions of this work:</p>
+<p><strong><a href="https://github.com/apache/datafusion/issues/19193">Page-Level Limit Pruning</a></strong>: Today, "fully matched" detection operates at the row group level. If we extend this to use page index statistics, we could stop decoding pages <em>within</em> a row group once the limit is met. This would pay dividends for wide row groups where only a few pages hold matching data.</p>
+<p><strong><a href="https://github.com/apache/datafusion/issues/19028">Row Filter Hints</a></strong>: Even when a row group is fully matched, the current row filter still evaluates predicates row by row. If we pass the fully matched groups info into the row filter builder, we can skip predicate evaluation entirely for guaranteed groups — saving CPU cycles on predicate evaluation.</p>
+<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h2>
+<p>DataFusion's pruning pipeline trims redundant I/O from the partition level all the way down to individual rows. Limit pruning adds a new step that creates an early exit when fully matched row groups already satisfy the <code>LIMIT</code>. The result is fewer row groups scanned, less data decoded, and faster queries.</p>
+<p>The key insights are:
+1. <strong>Predicate negation</strong> can identify row groups where <em>all</em> rows match — not just "some might match"
+2. <strong>Row count accumulation</strong> across fully matched groups enables early termination</p>
+<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries, and machine learning and streaming applications.</p>
+<p>DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone.</p>
+<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
+<p>If you are interested in contributing, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you need to
+understand and explains where your work should actually happen.</p>
+<h2 id="the-three-layers">The Three Layers<a class="headerlink" href="#the-three-layers" title="Permanent link">¶</a></h2>
+<hr/>
+<p>When DataFusion executes a query against a table, three abstractions collaborate
+to produce results:</p>
+<ol>
+<li><strong>[<code>TableProvider</code>]</strong> -- Describes the table (schema, capabilities) and
+ produces an execution plan when queried.</li>
+<li><strong>[<code>ExecutionPlan</code>]</strong> -- Describes <em>how</em> to compute the result: partitioning,
+ ordering, and child plan relationships.</li>
+<li><strong>[<code>SendableRecordBatchStream</code>]</strong> -- The async stream that <em>actually does the
+ work</em>, yielding <code>RecordBatch</code>es one at a time.</li>
+</ol>
+<p>Think of these as a funnel: <code>TableProvider::scan()</code> is called once during
+planning to create an <code>ExecutionPlan</code>, then <code>ExecutionPlan::execute()</code> is called
+once per partition to create a stream, and those streams are where rows are
+actually produced during execution.</p>
+<h2 id="layer-1-tableprovider">Layer 1: TableProvider<a class="headerlink" href="#layer-1-tableprovider" title="Permanent link">¶</a></h2>
+<hr/>
+<p>A [<code>TableProvider</code>] represents a queryable data source. For a minimal read-only
+table, you need four methods:</p>
+<pre><code class="language-rust">impl TableProvider for MyTable {
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+
+ fn schema(&amp;self) -&gt; SchemaRef {
+ Arc::clone(&amp;self.schema)
+ }
+
+ fn table_type(&amp;self) -&gt; TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &amp;self,
+ state: &amp;dyn Session,
+ projection: Option&lt;&amp;Vec&lt;usize&gt;&gt;,
+ filters: &amp;[Expr],
+ limit: Option&lt;usize&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ // Build and return an ExecutionPlan -- keep this lightweight!
+ Ok(Arc::new(MyExecPlan::new(
+ Arc::clone(&amp;self.schema),
+ projection,
+ limit,
+ )))
+ }
+}
+</code></pre>
+<p>The <code>scan</code> method is the heart of <code>TableProvider</code>. It receives three pushdown
+hints from the optimizer, each reducing the amount of data your source needs
+to produce:</p>
+<ul>
+<li><strong><code>projection</code></strong> -- Which columns are needed. This reduces the <strong>width</strong> of
+ the output. If your source supports it, read only these columns rather than
+ the full schema.</li>
+<li><strong><code>filters</code></strong> -- Predicates the engine would like you to apply during the
+ scan. This reduces the <strong>number of rows</strong> by skipping data that does not
+ match. Implement <code>supports_filters_pushdown</code> to advertise which filters you
+ can handle.</li>
+<li><strong><code>limit</code></strong> -- A row count cap. This also reduces the <strong>number of rows</strong> --
+ if you can stop reading early once you have produced enough rows, this avoids
+ unnecessary work.</li>
+</ul>
+<h3 id="keep-scan-lightweight">Keep <code>scan()</code> Lightweight<a class="headerlink" href="#keep-scan-lightweight" title="Permanent link">¶</a></h3>
+<p>This is a critical point: <strong><code>scan()</code> runs during planning, not execution.</strong> It
+should return quickly. Best practices are to avoid performing I/O, network
+calls, or heavy computation here. The <code>scan</code> method's job is to <em>describe</em> how
+the data will be produced, not to produce it. All the real work belongs in the
+stream (Layer 3).</p>
+<p>A common pitfall is to fetch data or open connections in <code>scan()</code>. This blocks
+the planning thread and can cause timeouts or deadlocks, especially if the query
+involves multiple tables or subqueries that all need to be planned before
+execution begins.</p>
+<h3 id="existing-implementations-to-learn-from">Existing Implementations to Learn From<a class="headerlink" href="#existing-implementations-to-learn-from" title="Permanent link">¶</a></h3>
+<p>DataFusion ships several <code>TableProvider</code> implementations that are excellent
+references:</p>
+<ul>
+<li><strong>[<code>MemTable</code>]</strong> -- Holds data in memory as <code>Vec&lt;RecordBatch&gt;</code>. The simplest
+ possible provider; great for tests and small datasets.</li>
+<li><strong>[<code>StreamTable</code>]</strong> -- Wraps a user-provided stream factory. Useful when your
+ data arrives as a continuous stream (e.g., from Kafka or a socket).</li>
+<li><strong>[<code>SortedTableProvider</code>]</strong> -- Wraps another <code>TableProvider</code> and advertises a
+ known sort order, enabling the optimizer to skip redundant sorts.</li>
+</ul>
+<h2 id="layer-2-executionplan">Layer 2: ExecutionPlan<a class="headerlink" href="#layer-2-executionplan" title="Permanent link">¶</a></h2>
+<hr/>
+<p>An [<code>ExecutionPlan</code>] is a node in the physical query plan tree. Your table
+provider's <code>scan()</code> method returns one. The required methods are:</p>
+<pre><code class="language-rust">impl ExecutionPlan for MyExecPlan {
+ fn name(&amp;self) -&gt; &amp;str { "MyExecPlan" }
+
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+
+ fn properties(&amp;self) -&gt; &amp;PlanProperties {
+ &amp;self.properties
+ }
+
+ fn children(&amp;self) -&gt; Vec&lt;&amp;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ vec![] // Leaf node -- no children
+ }
+
+ fn with_new_children(
+ self: Arc&lt;Self&gt;,
+ children: Vec&lt;Arc&lt;dyn ExecutionPlan&gt;&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ assert!(children.is_empty());
+ Ok(self)
+ }
+
+ fn execute(
+ &amp;self,
+ partition: usize,
+ context: Arc&lt;TaskContext&gt;,
+ ) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ // This is where you build and return your stream
+ // ...
+ }
+}
+</code></pre>
+<p>The key properties to set correctly in [<code>PlanProperties</code>] are <strong>output
+partitioning</strong> and <strong>output ordering</strong>.</p>
+<p><strong>Output partitioning</strong> tells the engine how many partitions your data has,
+which determines parallelism. If your source naturally partitions data (e.g.,
+by file or by shard), expose that here.</p>
+<p><strong>Output ordering</strong> declares whether your data is naturally sorted. This
+enables the optimizer to avoid inserting a <code>SortExec</code> when a query requires
+ordered data. Getting this right can be a significant performance win.</p>
+<h3 id="partitioning-strategies">Partitioning Strategies<a class="headerlink" href="#partitioning-strategies" title="Permanent link">¶</a></h3>
+<p>Since <code>execute()</code> is called once per partition, partitioning directly controls
+the parallelism of your table scan. Each partition runs on its own task, so
+more partitions means more concurrent work -- up to the number of available
+cores.</p>
+<p>Consider how your data source naturally divides its data:</p>
+<ul>
+<li><strong>By file or object:</strong> If you are reading from S3, each file can be a
+ partition. DataFusion will read them in parallel.</li>
+<li><strong>By shard or region:</strong> If your source is a sharded database, each shard
+ maps naturally to a partition.</li>
+<li><strong>By key range:</strong> If your data is keyed (e.g., by timestamp or customer ID),
+ you can split it into ranges.</li>
+</ul>
+<p>Getting partitioning right matters because it affects everything downstream in
+the plan. When DataFusion needs to perform an aggregation or join, it
+repartitions data by hashing the relevant columns. If your source already
+produces data partitioned by the join or group-by key, DataFusion can skip the
+repartition step entirely -- avoiding a potentially expensive shuffle.</p>
+<p>For example, if you are building a table provider for a system that stores
+data partitioned by <code>customer_id</code>, and a common query groups by <code>customer_id</code>:</p>
+<pre><code class="language-sql">SELECT customer_id, SUM(amount)
+FROM my_table
+GROUP BY customer_id;
+</code></pre>
+<p>If you declare your output partitioning as <code>Hash([customer_id], N)</code>, the
+optimizer recognizes that the data is already distributed correctly for the
+aggregation and eliminates the <code>RepartitionExec</code> that would otherwise appear
+in the plan. You can verify this with <code>EXPLAIN</code> (more on this below).</p>
+<p>Conversely, if you report <code>UnknownPartitioning</code>, DataFusion must assume the
+worst case and will always insert repartitioning operators as needed.</p>
+<h3 id="keep-execute-lightweight-too">Keep <code>execute()</code> Lightweight Too<a class="headerlink" href="#keep-execute-lightweight-too" title="Permanent link">¶</a></h3>
+<p>Like <code>scan()</code>, the <code>execute()</code> method should construct and return a stream
+without doing heavy work. The actual data production happens when the stream
+is polled. Do not block on async operations here -- build the stream and let
+the runtime drive it.</p>
+<h3 id="existing-implementations-to-learn-from_1">Existing Implementations to Learn From<a class="headerlink" href="#existing-implementations-to-learn-from_1" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong>[<code>StreamingTableExec</code>]</strong> -- Executes a streaming table scan. It takes a
+ stream factory (a closure that produces streams) and handles partitioning.
+ Good reference for wrapping external streams.</li>
+<li><strong>[<code>DataSourceExec</code>]</strong> -- The execution plan behind DataFusion's built-in file
+ scanning (Parquet, CSV, JSON). It demonstrates sophisticated partitioning,
+ filter pushdown, and projection pushdown.</li>
+</ul>
+<h2 id="layer-3-sendablerecordbatchstream">Layer 3: SendableRecordBatchStream<a class="headerlink" href="#layer-3-sendablerecordbatchstream" title="Permanent link">¶</a></h2>
+<hr/>
+<p>[<code>SendableRecordBatchStream</code>] is where the real work happens. It is defined as:</p>
+<pre><code class="language-rust">type SendableRecordBatchStream =
+ Pin&lt;Box&lt;dyn RecordBatchStream&lt;Item = Result&lt;RecordBatch&gt;&gt; + Send&gt;&gt;;
+</code></pre>
+<p>This is an async stream of <code>RecordBatch</code>es that can be sent across threads. When
+the DataFusion runtime polls this stream, your code runs: reading files, calling
+APIs, transforming data, etc.</p>
+<h3 id="using-recordbatchstreamadapter">Using RecordBatchStreamAdapter<a class="headerlink" href="#using-recordbatchstreamadapter" title="Permanent link">¶</a></h3>
+<p>The easiest way to create a <code>SendableRecordBatchStream</code> is with
+[<code>RecordBatchStreamAdapter</code>]. It bridges any <code>futures::Stream&lt;Item =
+Result&lt;RecordBatch&gt;&gt;</code> into the <code>SendableRecordBatchStream</code> type:</p>
+<pre><code class="language-rust">use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+
+fn execute(
+ &amp;self,
+ partition: usize,
+ context: Arc&lt;TaskContext&gt;,
+) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ let schema = self.schema();
+ let config = self.config.clone();
+
+ let stream = futures::stream::once(async move {
+ // ALL the heavy work happens here, inside the stream:
+ // - Open connections
+ // - Read data from external sources
+ // - Transform and batch the results
+ let batches = fetch_data_from_source(&amp;config).await?;
+ Ok(batches)
+ })
+ .flat_map(|result| match result {
+ Ok(batch) =&gt; futures::stream::iter(vec![Ok(batch)]),
+ Err(e) =&gt; futures::stream::iter(vec![Err(e)]),
+ });
+
+ Ok(Box::pin(RecordBatchStreamAdapter::new(schema, stream)))
+}
+</code></pre>
+<h3 id="cpu-intensive-work-use-a-separate-thread-pool">CPU-Intensive Work: Use a Separate Thread Pool<a class="headerlink" href="#cpu-intensive-work-use-a-separate-thread-pool" title="Permanent link">¶</a></h3>
+<p>If your stream performs CPU-intensive work (parsing, decompression, complex
+transformations), avoid blocking the tokio runtime. Instead, offload to a
+dedicated thread pool and send results back through a channel:</p>
+<pre><code class="language-rust">fn execute(
+ &amp;self,
+ partition: usize,
+ context: Arc&lt;TaskContext&gt;,
+) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ let schema = self.schema();
+ let config = self.config.clone();
+
+ let (tx, rx) = tokio::sync::mpsc::channel(2);
+
+ // Spawn CPU-heavy work on a blocking thread pool
+ tokio::task::spawn_blocking(move || {
+ let batches = generate_data(&amp;config);
+ for batch in batches {
+ if tx.blocking_send(Ok(batch)).is_err() {
+ break; // Receiver dropped, query was cancelled
+ }
+ }
+ });
+
+ let stream = tokio_stream::wrappers::ReceiverStream::new(rx);
+ Ok(Box::pin(RecordBatchStreamAdapter::new(schema, stream)))
+}
+</code></pre>
+<p>This pattern keeps the async runtime responsive while your data generation
+runs on its own threads.</p>
+<h2 id="where-should-the-work-happen">Where Should the Work Happen?<a class="headerlink" href="#where-should-the-work-happen" title="Permanent link">¶</a></h2>
+<hr/>
+<p>This table summarizes what belongs at each layer:</p>
+<table class="table">
+<thead>
+<tr>
+<th>Layer</th>
+<th>Runs During</th>
+<th>Should Do</th>
+<th>Should NOT Do</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>TableProvider::scan()</code></td>
+<td>Planning</td>
+<td>Build an <code>ExecutionPlan</code> with metadata</td>
+<td>I/O, network calls, heavy computation</td>
+</tr>
+<tr>
+<td><code>ExecutionPlan::execute()</code></td>
+<td>Execution (once per partition)</td>
+<td>Construct a stream, set up channels</td>
+<td>Block on async work, read data</td>
+</tr>
+<tr>
+<td><code>RecordBatchStream</code> (polling)</td>
+<td>Execution</td>
+<td>All I/O, computation, data production</td>
+<td>--</td>
+</tr>
+</tbody>
+</table>
+<p>The guiding principle: <strong>push work as late as possible.</strong> Planning should be
+fast so the optimizer can do its job. Execution setup should be fast so all
+partitions can start promptly. The stream is where you spend time producing
+data.</p>
+<h3 id="why-this-matters">Why This Matters<a class="headerlink" href="#why-this-matters" title="Permanent link">¶</a></h3>
+<p>When <code>scan()</code> does heavy work, several problems arise:</p>
+<ol>
+<li><strong>Planning becomes slow.</strong> If a query touches 10 tables and each <code>scan()</code>
+ takes 500ms, planning alone takes 5 seconds before any data flows.</li>
+<li><strong>The optimizer cannot help.</strong> The optimizer runs between planning and
+ execution. If you have already fetched data during planning, optimizations
+ like predicate pushdown or partition pruning cannot reduce the work.</li>
+<li><strong>Resource management breaks down.</strong> DataFusion manages concurrency and
+ memory during execution. Work done during planning bypasses these controls.</li>
+</ol>
+<h2 id="filter-pushdown-doing-less-work">Filter Pushdown: Doing Less Work<a class="headerlink" href="#filter-pushdown-doing-less-work" title="Permanent link">¶</a></h2>
+<hr/>
+<p>One of the most impactful optimizations you can add to a custom table provider
+is <strong>filter pushdown</strong> -- letting the source skip data that the query does not
+need, rather than reading everything and filtering it afterward.</p>
+<h3 id="how-filter-pushdown-works">How Filter Pushdown Works<a class="headerlink" href="#how-filter-pushdown-works" title="Permanent link">¶</a></h3>
+<p>When DataFusion plans a query with a <code>WHERE</code> clause, it passes the filter
+predicates to your <code>scan()</code> method as the <code>filters</code> parameter. By default,
+DataFusion assumes your provider cannot handle any filters and inserts a
+<code>FilterExec</code> node above your scan to apply them. But if your source <em>can</em>
+evaluate some predicates during scanning -- for example, by skipping files,
+partitions, or row groups that cannot match -- you can eliminate a huge amount
+of unnecessary I/O.</p>
+<p>To opt in, implement <code>supports_filters_pushdown</code>:</p>
+<pre><code class="language-rust">fn supports_filters_pushdown(
+ &amp;self,
+ filters: &amp;[&amp;Expr],
+) -&gt; Result&lt;Vec&lt;TableProviderFilterPushDown&gt;&gt; {
+ Ok(filters.iter().map(|f| {
+ match f {
+ // We can fully evaluate equality filters on
+ // the partition column at the source
+ Expr::BinaryExpr(BinaryExpr {
+ left, op: Operator::Eq, right
+ }) if is_partition_column(left) || is_partition_column(right) =&gt; {
+ TableProviderFilterPushDown::Exact
+ }
+ // All other filters: let DataFusion handle them
+ _ =&gt; TableProviderFilterPushDown::Unsupported,
+ }
+ }).collect())
+}
+</code></pre>
+<p>The three possible responses for each filter are:</p>
+<ul>
+<li><strong><code>Exact</code></strong> -- Your source guarantees that no output rows will have a false
+ value for this predicate. Because the filter is fully evaluated at the source,
+ DataFusion will <strong>not</strong> add a <code>FilterExec</code> for it.</li>
+<li><strong><code>Inexact</code></strong> -- Your source has the ability to reduce the data produced, but
+ the output may still include rows that do not satisfy the predicate. For
+ example, you might skip entire files based on metadata statistics but not
+ filter individual rows within a file. DataFusion will still add a <code>FilterExec</code>
+ above your scan to remove any remaining rows that slipped through.</li>
+<li><strong><code>Unsupported</code></strong> -- Your source ignores this filter entirely. DataFusion
+ handles it.</li>
+</ul>
+<h3 id="why-filter-pushdown-matters">Why Filter Pushdown Matters<a class="headerlink" href="#why-filter-pushdown-matters" title="Permanent link">¶</a></h3>
+<p>Consider a table with 1 billion rows partitioned by <code>region</code>, and a query:</p>
+<pre><code class="language-sql">SELECT * FROM events WHERE region = 'us-east-1' AND event_type = 'click';
+</code></pre>
+<p><strong>Without filter pushdown:</strong> Your table provider reads all 1 billion rows
+across all regions. DataFusion then applies both filters, discarding the vast
+majority of the data.</p>
+<p><strong>With filter pushdown on <code>region</code>:</strong> Your <code>scan()</code> method sees the
+<code>region = 'us-east-1'</code> filter and constructs an execution plan that only reads
+the <code>us-east-1</code> partition. If that partition holds 100 million rows, you have
+just eliminated 90% of the I/O. DataFusion still applies the <code>event_type</code>
+filter via <code>FilterExec</code> if you reported it as <code>Unsupported</code>.</p>
+<h3 id="using-explain-to-debug-your-table-provider">Using EXPLAIN to Debug Your Table Provider<a class="headerlink" href="#using-explain-to-debug-your-table-provider" title="Permanent link">¶</a></h3>
+<p>The <code>EXPLAIN</code> statement is your best tool for understanding what DataFusion is
+actually doing with your table provider. It shows the physical plan that
+DataFusion will execute, including any operators it inserted:</p>
+<pre><code class="language-sql">EXPLAIN SELECT * FROM events WHERE region = 'us-east-1' AND event_type = 'click';
+</code></pre>
+<p>If you are using DataFrames, call <code>.explain(false, false)</code> for the logical plan
+or <code>.explain(false, true)</code> for the physical plan. You can also print the plans
+in verbose mode with <code>.explain(true, true)</code>.</p>
+<p><strong>Before filter pushdown</strong>, the plan might look like:</p>
+<pre><code class="language-text">FilterExec: region@0 = us-east-1 AND event_type@1 = click
+ MyExecPlan: partitions=50
+</code></pre>
+<p>Here DataFusion is reading all 50 partitions and filtering everything
+afterward. The <code>FilterExec</code> above your scan is doing all the predicate work.</p>
+<p><strong>After implementing pushdown for <code>region</code></strong> (reported as <code>Exact</code>):</p>
+<pre><code class="language-text">FilterExec: event_type@1 = click
+ MyExecPlan: partitions=5, filter=[region = us-east-1]
+</code></pre>
+<p>Now your exec reads only the 5 partitions for <code>us-east-1</code>, and the remaining
+<code>FilterExec</code> only handles the <code>event_type</code> predicate. The <code>region</code> filter has
+been fully absorbed by your scan.</p>
+<p><strong>After implementing pushdown for both filters</strong> (both <code>Exact</code>):</p>
+<pre><code class="language-text">MyExecPlan: partitions=5, filter=[region = us-east-1 AND event_type = click]
+</code></pre>
+<p>No <code>FilterExec</code> at all -- your source handles everything.</p>
+<p>Similarly, <code>EXPLAIN</code> will reveal whether DataFusion is inserting unnecessary
+<code>SortExec</code> or <code>RepartitionExec</code> nodes that you could eliminate by declaring
+better output properties. Whenever your queries seem slower than expected,
+<code>EXPLAIN</code> is the first place to look.</p>
+<h2 id="putting-it-all-together">Putting It All Together<a class="headerlink" href="#putting-it-all-together" title="Permanent link">¶</a></h2>
+<hr/>
+<p>Here is a minimal but complete example of a custom table provider that generates
+data lazily during streaming:</p>
+<pre><code class="language-rust">use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Int64Array, StringArray};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use arrow::record_batch::RecordBatch;
+use datafusion::catalog::TableProvider;
+use datafusion::common::Result;
+use datafusion::datasource::TableType;
+use datafusion::execution::context::SessionState;
+use datafusion::execution::SendableRecordBatchStream;
+use datafusion::logical_expr::Expr;
+use datafusion::physical_expr::EquivalenceProperties;
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+use datafusion::physical_plan::{
+ ExecutionPlan, Partitioning, PlanProperties,
+};
+use futures::stream;
+
+/// A table provider that generates sequential numbers on demand.
+struct CountingTable {
+ schema: SchemaRef,
+ num_partitions: usize,
+ rows_per_partition: usize,
+}
+
+impl CountingTable {
+ fn new(num_partitions: usize, rows_per_partition: usize) -&gt; Self {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("partition", DataType::Int64, false),
+ Field::new("value", DataType::Int64, false),
+ ]));
+ Self { schema, num_partitions, rows_per_partition }
+ }
+}
+
+#[async_trait::async_trait]
+impl TableProvider for CountingTable {
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+ fn schema(&amp;self) -&gt; SchemaRef { Arc::clone(&amp;self.schema) }
+ fn table_type(&amp;self) -&gt; TableType { TableType::Base }
+
+ async fn scan(
+ &amp;self,
+ _state: &amp;dyn Session,
+ projection: Option&lt;&amp;Vec&lt;usize&gt;&gt;,
+ _filters: &amp;[Expr],
+ limit: Option&lt;usize&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ // Light work only: build the plan with metadata
+ Ok(Arc::new(CountingExec {
+ schema: Arc::clone(&amp;self.schema),
+ num_partitions: self.num_partitions,
+ rows_per_partition: limit
+ .unwrap_or(self.rows_per_partition)
+ .min(self.rows_per_partition),
+ properties: PlanProperties::new(
+ EquivalenceProperties::new(Arc::clone(&amp;self.schema)),
+ Partitioning::UnknownPartitioning(self.num_partitions),
+ EmissionType::Incremental,
+ Boundedness::Bounded,
+ ),
+ }))
+ }
+}
+
+struct CountingExec {
+ schema: SchemaRef,
+ num_partitions: usize,
+ rows_per_partition: usize,
+ properties: PlanProperties,
+}
+
+impl ExecutionPlan for CountingExec {
+ fn name(&amp;self) -&gt; &amp;str { "CountingExec" }
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+ fn properties(&amp;self) -&gt; &amp;PlanProperties { &amp;self.properties }
+ fn children(&amp;self) -&gt; Vec&lt;&amp;Arc&lt;dyn ExecutionPlan&gt;&gt; { vec![] }
+
+ fn with_new_children(
+ self: Arc&lt;Self&gt;,
+ _children: Vec&lt;Arc&lt;dyn ExecutionPlan&gt;&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ Ok(self)
+ }
+
+ fn execute(
+ &amp;self,
+ partition: usize,
+ _context: Arc&lt;TaskContext&gt;,
+ ) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ let schema = Arc::clone(&amp;self.schema);
+ let rows = self.rows_per_partition;
+
+ // The heavy work (data generation) happens inside the stream,
+ // not here in execute().
+ let batch_stream = stream::once(async move {
+ let partitions = Int64Array::from(
+ vec![partition as i64; rows],
+ );
+ let values = Int64Array::from(
+ (0..rows as i64).collect::&lt;Vec&lt;_&gt;&gt;(),
+ );
+ let batch = RecordBatch::try_new(
+ Arc::clone(&amp;schema),
+ vec![Arc::new(partitions), Arc::new(values)],
+ )?;
+ Ok(batch)
+ });
+
+ Ok(Box::pin(RecordBatchStreamAdapter::new(
+ Arc::clone(&amp;self.schema),
+ batch_stream,
+ )))
+ }
+}
+</code></pre>
+<h2 id="choosing-the-right-starting-point">Choosing the Right Starting Point<a class="headerlink" href="#choosing-the-right-starting-point" title="Permanent link">¶</a></h2>
+<hr/>
+<p>Not every custom data source requires implementing all three layers from
+scratch. DataFusion provides building blocks that let you plug in at whatever
+level makes sense:</p>
+<table class="table">
+<thead>
+<tr>
+<th>If your data is...</th>
+<th>Start with</th>
+<th>You implement</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Already in <code>RecordBatch</code>es in memory</td>
+<td>[<code>MemTable</code>]</td>
+<td>Nothing -- just construct it</td>
+</tr>
+<tr>
+<td>An async stream of batches</td>
+<td>[<code>StreamTable</code>]</td>
+<td>A stream factory</td>
+</tr>
+<tr>
+<td>A table with known sort order</td>
+<td>[<code>SortedTableProvider</code>] wrapping another provider</td>
+<td>The inner provider</td>
+</tr>
+<tr>
+<td>A custom source needing full control</td>
+<td><code>TableProvider</code> + <code>ExecutionPlan</code> + stream</td>
+<td>All three layers</td>
+</tr>
+</tbody>
+</table>
+<p>For most integrations, [<code>StreamTable</code>] combined with
+[<code>RecordBatchStreamAdapter</code>] provides a good balance of simplicity and
+flexibility. You provide a closure that returns a stream, and DataFusion handles
+the rest.</p>
+<h2 id="further-reading">Further Reading<a class="headerlink" href="#further-reading" title="Permanent link">¶</a></h2>
+<hr/>
+<ul>
+<li>[TableProvider API docs][<code>TableProvider</code>]</li>
+<li>[ExecutionPlan API docs][<code>ExecutionPlan</code>]</li>
+<li>[SendableRecordBatchStream API docs][<code>SendableRecordBatchStream</code>]</li>
+<li><a href="https://github.com/apache/datafusion/issues/16821">GitHub issue discussing table provider examples</a></li>
+<li><a href="https://github.com/apache/datafusion/tree/main/datafusion-examples/examples">DataFusion examples directory</a> --
+ contains working examples including custom table providers</li>
+</ul>
+<hr/>
+<p><em>Note: Portions of this blog post were written with the assistance of an AI agent.</em></p></content><category term="blog"></category></entry><entry><title>Optimizing SQL CASE Expression Evaluation</title><link href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="alternate"></link><published>2026-02-02T00:00:00+00:00</published><updated>2026-02-02T00:00:00+00:00</updated><author><name>Pepijn Van Eeckhoudt</name></author><id>tag:datafusio [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a set of expressions is evaluated for each row based on arbitrary boolean expressions.
+Its deceptively simple syntax hides significant implementation complexity.
+Over the past few <a href="https://datafusion.apache.org/">Apache DataFusion</a> releases, a series of improvements to <code>CASE</code> expression evaluator have been merged that reduce both CPU time and memory allocations.
+This post provides an overview of the original implementation, its performance bottlenecks, and the steps taken to address them.</p>
+<h2 id="background-case-expression-evaluation">Background: CASE Expression Evaluation<a class="headerlink" href="#background-case-expression-evaluation" title="Permanent link">¶</a></h2>
+<p>SQL supports two forms of CASE expressions:</p>
+<ol>
+<li><strong>Simple</strong>: <code>CASE expr WHEN value1 THEN result1 WHEN value2 THEN result2 ... END</code></li>
+<li><strong>Searched</strong>: <code>CASE WHEN condition1 THEN result1 WHEN condition2 THEN result2 ... END</code></li>
+</ol>
+<p>The simple form evaluates an expression once for each input row and then tests that value against the expressions (typically constants) in each <code>WHEN</code> clause using equality comparisons.</p>
+<p>Here's an example of the simple form:</p>
+<pre><code class="language-sql">CASE status
+ WHEN 'pending' THEN 1
+ WHEN 'active' THEN 2
+ WHEN 'complete' THEN 3
+ ELSE 0
+END
+</code></pre>
+<p>In this <code>CASE</code> expression, <code>status</code> is evaluated once per row, and then its value is tested for equality with the values <code>'pending'</code>, <code>'active'</code>, and <code>'complete'</code> in that order.
+The <code>CASE</code> expression evaluates to the value of the <code>THEN</code> expression corresponding to the first matching <code>WHEN</code> expression.</p>
+<p>The searched <code>CASE</code> form is a more flexible variant.
+It evaluates completely independent boolean expressions for each branch.
+This allows you to test different columns with different operators per branch as shown in the following example:</p>
+<pre><code class="language-sql">CASE
+ WHEN age &gt; 65 THEN 'senior'
+ WHEN childCount != 0 THEN 'parent'
+ WHEN age &lt; 21 THEN 'minor'
+ ELSE 'adult'
+END
+</code></pre>
+<p>In both forms, branches are evaluated sequentially with short-circuit semantics: for each row, once a <code>WHEN</code> condition matches, the corresponding <code>THEN</code> expression is evaluated.
+Any further branches are not evaluated for that row.
+This lazy evaluation model is critical for correctness.
+It lets you safely write <code>CASE</code> expressions like <code>CASE WHEN d != 0 THEN n / d ELSE NULL END</code> that are guaranteed to not trigger divide-by-zero errors.</p>
+<p>Besides <code>CASE</code>, there are a few <a href="https://datafusion.apache.org/user-guide/sql/scalar_functions.html#conditional-functions">conditional scalar functions</a> that provide similar, more restricted capabilities.
+These include <code>COALESCE</code>, <code>IFNULL</code>, and <code>NVL2</code>.
+You can consider each of these functions as the equivalent of a macro for <code>CASE</code>.
+For example, <code>COALESCE(expr1, expr2, expr3)</code> expands to:</p>
+<pre><code class="language-sql">CASE
+ WHEN expr1 IS NOT NULL THEN expr1
+ WHEN expr2 IS NOT NULL THEN expr2
+ ELSE expr3
+END
+</code></pre>
+<p>Since <a href="https://datafusion.apache.org/">Apache DataFusion</a> rewrites these conditional functions to their equivalent <code>CASE</code> expression, any optimizations related to <code>CASE</code> described in this post also apply to conditional function evaluation.</p>
+<h2 id="case-evaluation-in-datafusion-5000"><code>CASE</code> Evaluation in DataFusion 50.0.0<a class="headerlink" href="#case-evaluation-in-datafusion-5000" title="Permanent link">¶</a></h2>
+<p>For the remainder of this post, we'll be looking at 'searched CASE' evaluation.
+'Simple CASE' uses a distinct, but very similar implementation.
+The same set of improvements has been applied to both.</p>
+<p>The baseline implementation in DataFusion 50.0.0 evaluated <code>CASE</code> using a common, straightforward approach:</p>
+<ol>
+<li>Start with an output array <code>out</code> with the same length as the input batch, filled with nulls. Additionally, create a bit vector <code>remainder</code> with the same length and each value set to <code>true</code>.</li>
+<li>For each <code>WHEN</code>/<code>THEN</code> branch:<ul>
+<li>Evaluate the <code>WHEN</code> condition for the remaining unmatched rows using <a href="https://docs.rs/datafusion/latest/datafusion/physical_expr/trait.PhysicalExpr.html#method.evaluate_selection"><code>PhysicalExpr::evaluate_selection</code></a>, passing in the input batch and the <code>remainder</code> mask.</li>
+<li>If any rows matched, evaluate the <code>THEN</code> expression for those rows using <code>PhysicalExpr::evaluate_selection</code>.</li>
+<li>Merge the results into the <code>out</code> array using the <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/zip/fn.zip.html"><code>zip</code></a> kernel.</li>
+<li>Update the <code>remainder</code> mask to exclude the matched rows.</li>
+</ul>
+</li>
+<li>If there's an <code>ELSE</code> clause, evaluate it for any remaining unmatched rows and merge using <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/zip/fn.zip.html"><code>zip</code></a>.</li>
+</ol>
+<p>Here's a simplified version of the Rust code for the original loop:</p>
+<pre><code class="language-rust">let mut out = new_null_array(&amp;return_type, batch.num_rows());
+let mut remainder = BooleanArray::from(vec![true; batch.num_rows()]);
+
+for (when_expr, then_expr) in &amp;self.when_then_expr {
+ // Determine for which remaining rows the WHEN condition matches
+ let when = when_expr.evaluate_selection(batch, &amp;remainder)?
+ .into_array(batch.num_rows())?;
+ // Ensure any `NULL` values are treated as false
+ let when_and_rem = and(&amp;when, &amp;remainder)?;
+
+ if when_and_rem.true_count() == 0 {
+ continue;
+ }
+
+ // Evaluate the THEN expression for matching rows
+ let then = then_expr.evaluate_selection(batch, &amp;when_and_rem)?;
+ // Merge results into output array
+ out = zip(&amp;when_and_rem, &amp;then_value, &amp;out)?;
+ // Update remainder mask to exclude matched rows
+ remainder = and_not(&amp;remainder, &amp;when_and_rem)?;
+}
+</code></pre>
+<p>Let's examine one iteration of this loop for the following <code>CASE</code> expression:</p>
+<pre><code class="language-sql">CASE
+ WHEN col = 'b' THEN 100
+ ELSE 200
+END
+</code></pre>
+<p>Schematically, it will look as follows:</p>
+<figure>
+<img alt="Schematic representation of data flow in the original CASE implementation" class="img-responsive" src="/blog/images/case/original_loop.svg" width="100%"/>
+<figcaption>One iteration of the `CASE` evaluation loop</figcaption>
+</figure>
+<p>This implementation works perfectly fine, but there's significant room for optimization, mostly related to the usage of <code>evaluate_selection</code>.
+To understand why, we need to dig a little deeper into the implementation of that function.
+Here's a simplified version of it that captures the relevant parts:</p>
+<pre><code class="language-rust">pub trait PhysicalExpr {
+ fn evaluate_selection(
+ &amp;self,
+ batch: &amp;RecordBatch,
+ selection: &amp;BooleanArray,
+ ) -&gt; Result&lt;ColumnarValue&gt; {
+ // Reduce record batch to only include rows that match selection
+ let filtered_batch = filter_record_batch(batch, selection)?;
+ // Perform regular evaluation on filtered batch
+ let filtered_result = self.evaluate(&amp;filtered_batch)?;
+ // Expand result array to match original batch length
+ scatter(selection, filtered_result)
+ }
+}
+</code></pre>
+<p>Going back to the same example as before, the data flow in <code>evaluate_selection</code> looks like this:</p>
+<figure>
+<img alt="Schematic representation of `evaluate_selection` evaluation" class="img-responsive" src="/blog/images/case/evaluate_selection.svg" width="100%"/>
+<figcaption>evaluate_selection data flow</figcaption>
+</figure>
+<p>The <code>evaluate_selection</code> method first filters the input batch to only include rows that match the <code>selection</code> mask.
+It then calls the regular <code>evaluate</code> method using the filtered batch as input.
+Finally, to return a result array with the same number of rows as <code>batch</code>, the <code>scatter</code> function is called.
+This function produces a new array padded with <code>null</code> values for any rows that didn't match the <code>selection</code> mask.</p>
+<p>So how can we improve the performance of the simple evaluation strategy and use of <code>evaluate_selection</code>?</p>
+<h3 id="opportunity-1-early-exit">Opportunity 1: Early Exit<a class="headerlink" href="#opportunity-1-early-exit" title="Permanent link">¶</a></h3>
+<p>The <code>CASE</code> evaluation loop always iterates through all branches, even when every row has already been matched.
+In queries where early branches match all rows, this results in unnecessary work being done for the remaining branches.</p>
+<h3 id="opportunity-2-optimize-repeated-filtering-scattering-and-merging">Opportunity 2: Optimize Repeated Filtering, Scattering, and Merging<a class="headerlink" href="#opportunity-2-optimize-repeated-filtering-scattering-and-merging" title="Permanent link">¶</a></h3>
+<p>Each iteration performs a number of operations that are very well-optimized, but still take up a significant amount of CPU time:</p>
+<ul>
+<li><strong>Filtering</strong>: <code>PhysicalExpr::evaluate_selection</code> filters the entire <code>RecordBatch</code> for each branch. For the <code>WHEN</code> expression, this is done even if the selection mask was entirely empty.</li>
+<li><strong>Scattering</strong>: <code>PhysicalExpr::evaluate_selection</code> scatters the filtered result back to the original <code>RecordBatch</code> length.</li>
+<li><strong>Merging</strong>: The <code>zip</code> kernel is called once per branch to merge partial results into the output array</li>
+</ul>
+<p>Each of these operations needs to allocate memory for new arrays and shuffle quite a bit of data around.</p>
+<h3 id="opportunity-3-filter-only-necessary-columns">Opportunity 3: Filter only Necessary Columns<a class="headerlink" href="#opportunity-3-filter-only-necessary-columns" title="Permanent link">¶</a></h3>
+<p>The <code>PhysicalExpr::evaluate_selection</code> method filters the entire record batch, including columns that the current branch's <code>WHEN</code> and <code>THEN</code> expressions don't reference.
+For wide tables (many columns) with narrow expressions (few column references), this is wasteful.</p>
+<p>Suppose you have a table with 26 columns named <code>a</code> through <code>z</code>, and the following simple <code>CASE</code> expression:</p>
+<pre><code class="language-sql">CASE
+ WHEN a &gt; 1000 THEN 'large'
+ WHEN a &gt;= 0 THEN 'positive'
+ ELSE 'negative'
+END
+</code></pre>
+<p>The implementation would filter all 26 columns even though only a single column is needed for the entire <code>CASE</code> expression evaluation.
+Again this involves a non-negligible amount of allocation and data copying.</p>
+<h2 id="performance-optimizations">Performance Optimizations<a class="headerlink" href="#performance-optimizations" title="Permanent link">¶</a></h2>
+<h3 id="optimization-1-short-circuit-early-exit">Optimization 1: Short-Circuit Early Exit<a class="headerlink" href="#optimization-1-short-circuit-early-exit" title="Permanent link">¶</a></h3>
+<p>The first optimization is straightforward.
+As soon as we detect that all rows of the batch have been matched, we break out of the evaluation loop:</p>
+<pre><code class="language-rust">let mut remainder_count = batch.num_rows();
+
+for (when_expr, then_expr) in &amp;self.when_then_expr {
+ if remainder_count == 0 {
+ break; // All rows matched, exit early
+ }
+
+ // ... evaluate branch ...
+
+ let when_match_count = when_value.true_count();
+ remainder_count -= when_match_count;
+}
+</code></pre>
+<p>Additionally, we avoid evaluating the <code>ELSE</code> clause when no rows remain:</p>
+<pre><code class="language-rust">if let Some(else_expr) = &amp;self.else_expr {
+ remainder = or(&amp;base_nulls, &amp;remainder)?;
+ if remainder.true_count() &gt; 0 {
+ // ... evaluate else ...
+ }
+}
+</code></pre>
+<p>For queries where early branches match all rows, this eliminates unnecessary branch evaluations and <code>ELSE</code> clause processing.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/17898">PR #17898</a></p>
+<h3 id="optimization-2-optimized-result-merging">Optimization 2: Optimized Result Merging<a class="headerlink" href="#optimization-2-optimized-result-merging" title="Permanent link">¶</a></h3>
+<p>The second optimization fundamentally restructures how the results of each loop iteration will be merged.
+The diagram below illustrates the optimized data flow when evaluating the <code>CASE WHEN col = 'b' THEN 100 ELSE 200 END</code> from before:</p>
+<figure>
+<img alt="Schematic representation of optimized evaluation loop" class="img-responsive" src="/blog/images/case/merging.svg" width="100%"/>
+<figcaption>optimized evaluation loop</figcaption>
+</figure>
+<p>In the reworked implementation, the <code>evaluate_selection</code> function is no longer used.
+The key insight is that we can defer all merging until the end of the evaluation loop by tracking result provenance.
+This was implemented with the following changes:</p>
+<ol>
+<li>Augment the input batch with a column containing row indices.</li>
+<li>Reduce the augmented batch after each loop iteration to only contain the remaining rows.</li>
+<li>Use the row index column to track which partial result array contains the value for each row.</li>
+<li>Perform a single merge operation at the end instead of a <code>zip</code> operation after each loop iteration.</li>
+</ol>
+<p>These changes make it unnecessary to <code>scatter</code> and <code>zip</code> results in each loop iteration.
+Instead, when all rows have been matched, we then merge the partial results using <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge_n.html"><code>arrow_select::merge::merge_n</code></a>.</p>
+<p>The diagram below illustrates how <code>merge_n</code> works for an example where three <code>WHEN/THEN</code> branches produced results.
+The first branch produced the result <code>A</code> for row 2, the second produced <code>B</code> for row 1, and the third produced <code>C</code> and <code>D</code> for rows 4 and 5.</p>
+<figure>
+<img alt="Schematic illustration of the merge_n algorithm" class="img-responsive" src="/blog/images/case/merge_n.svg" width="100%"/>
+<figcaption>merge_n example</figcaption>
+</figure>
+<p>The <code>merge_n</code> algorithm scans through the indices array.
+For each non-empty cell, it takes one value from the corresponding values array.
+In the example above, we first encounter <code>1</code>.
+This takes the first element from the values array with index <code>1</code>, resulting in <code>B</code>.
+The next cell contains <code>0</code> which takes <code>A</code>, from the first array.
+Finally, we encounter <code>2</code> twice.
+This takes the first and second element from the last values array respectively.</p>
+<p>This algorithm was initially implemented in DataFusion for the <code>CASE</code> implementation, but in the meantime has been generalized and moved into the <code>arrow-rs</code> crate as <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge_n.html"><code>arrow_select::merge::merge_n</code></a>.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18152">PR #18152</a></p>
+<h3 id="optimization-3-column-projection">Optimization 3: Column Projection<a class="headerlink" href="#optimization-3-column-projection" title="Permanent link">¶</a></h3>
+<p>The third optimization addresses the "filtering unused columns" overhead through projection.</p>
+<p>Look at the following query example where the <code>mailing_address</code> table has the columns <code>name</code>, <code>surname</code>, <code>street</code>, <code>number</code>, <code>city</code>, <code>state</code>, <code>country</code>:</p>
+<pre><code class="language-sql">SELECT *, CASE WHEN country = 'USA' THEN state ELSE country END AS region
+FROM mailing_address
+</code></pre>
+<p>You can see that the <code>CASE</code> expression only references the columns <code>country</code> and <code>state</code>, but because all columns are being queried, projection pushdown cannot reduce the number of columns being fed in to the projection operator.</p>
+<figure>
+<img alt="Schematic illustration of CASE evaluation without projection" class="img-responsive" src="/blog/images/case/no_projection.svg" width="100%"/>
+<figcaption>CASE evaluation without projection</figcaption>
+</figure>
+<p>During <code>CASE</code> evaluation, the batch must be filtered using the <code>WHEN</code> expression to evaluate the <code>THEN</code> expression values.
+As the diagram above shows, this filtering creates a reduced copy of all columns.</p>
+<p>This unnecessary copying can be avoided by first narrowing the batch to only include the columns that are actually needed.</p>
+<figure>
+<img alt="Schematic illustration of CASE evaluation with projection" class="img-responsive" src="/blog/images/case/projection.svg" width="100%"/>
+<figcaption>CASE evaluation with projection</figcaption>
+</figure>
+<p>At first glance, this might not seem beneficial, since we're introducing an additional processing step.
+Luckily projection of a record batch only requires a shallow copy of the record batch.
+The column arrays themselves are not copied, and the only work that is actually done is incrementing the reference counts of the columns.</p>
+<p><strong>Impact</strong>: For wide tables with narrow CASE expressions, this dramatically reduces filtering overhead by removing the copying of unused columns.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18329">PR #18329</a></p>
+<h3 id="optimization-4-eliminating-scatter-in-two-branch-case">Optimization 4: Eliminating Scatter in Two-Branch Case<a class="headerlink" href="#optimization-4-eliminating-scatter-in-two-branch-case" title="Permanent link">¶</a></h3>
+<p>Some of the earlier examples in this post use expressions of the form <code>CASE WHEN condition THEN expr1 ELSE expr2 END</code> to explain how the general evaluation loop works.
+For this kind of two-branch <code>CASE</code> expression, <a href="https://datafusion.apache.org/">Apache DataFusion</a> has a more optimized implementation that unrolls the loop.
+This specialized <code>ExpressionOrExpression</code> fast path still used <code>evaluate_selection()</code> for both branches which uses <code>scatter</code> and <code>zip</code> to combine the results incurring the same performance overhead as the general implementation.</p>
+<p>The revised implementation eliminates the use of <code>evaluate_selection</code> as follows:</p>
+<pre><code class="language-rust">// Compute the `WHEN` condition for the entire batch
+let when_filter = create_filter(&amp;when_value);
+
+// Compute a compact array of `THEN` values for the matching rows
+let then_batch = filter_record_batch(batch, &amp;when_filter)?;
+let then_value = then_expr.evaluate(&amp;then_batch)?;
+
+// Compute a compact array of `ELSE` values for the non-matching rows
+let else_filter = create_filter(&amp;not(&amp;when_value)?);
+let else_batch = filter_record_batch(batch, &amp;else_filter)?;
+let else_value = else_expr.evaluate(&amp;else_batch)?;
+</code></pre>
+<p>This produces two compact arrays, one for the THEN values and one for the ELSE values, which are then merged with the <code>merge</code> function.
+In contrast to <code>zip</code>, <code>merge</code> does not require both of its value inputs to have the same length.
+Instead it requires that the sum of the length of the value inputs matches the length of the mask array.</p>
+<figure>
+<img alt="Schematic illustration of the merge algorithm" class="img-responsive" src="/blog/images/case/merge.svg" width="100%"/>
+<figcaption>merge example</figcaption>
+</figure>
+<p>This eliminates unnecessary <code>scatter</code> operations and memory allocations for one of the most common <code>CASE</code> expression patterns.</p>
+<p>Just like <code>merge_n</code>, this operation has been moved into <code>arrow-rs</code> as <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge.html"><code>arrow_select::merge::merge</code></a>.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18444">PR #18444</a></p>
+<h3 id="optimization-5-table-lookup-of-constants">Optimization 5: Table Lookup of Constants<a class="headerlink" href="#optimization-5-table-lookup-of-constants" title="Permanent link">¶</a></h3>
+<p>Up until now, we've discussed the implementations for generic <code>CASE</code> expressions that use non-constant expressions for both <code>WHEN</code> and <code>THEN</code>.
+Another common use of <code>CASE</code> is to perform a mapping from one set of constants to another.
+For instance, you can expand numeric constants to human-readable strings using the following <code>CASE</code> example.</p>
+<pre><code class="language-sql">CASE status
+ WHEN 0 THEN 'idle'
+ WHEN 1 THEN 'running'
+ WHEN 2 THEN 'paused'
+ WHEN 3 THEN 'stopped'
+ ELSE 'unknown'
+END
+</code></pre>
+<p>A final <code>CASE</code> optimization recognizes this pattern and compiles the <code>CASE</code> expression into a hash table.
+Rather than evaluating the <code>WHEN</code> and <code>THEN</code> expressions, the input expression is evaluated once, and the result array is computed using a vectorized hash table lookup.
+This approach avoids the need to filter the input batch and combine partial results entirely.
+The result array is computed in a single pass over the input values, and the computation time does not grow significantly with the number of <code>WHEN</code> branches in the <code>CASE</code> expression.</p>
+<p>This optimization was implemented by Raz Luvaton (<a href="https://github.com/rluvaton"><code>@rluvaton</code></a>) in <a href="https://github.com/apache/datafusion/pull/18183">PR #18183</a></p>
+<h2 id="results">Results<a class="headerlink" href="#results" title="Permanent link">¶</a></h2>
+<p>The degree to which the performance optimizations described in this post will benefit your queries is highly dependent on both your data and your queries.
+To give some idea of the impact, we ran the following query on the TPC_H <code>orders</code> table with a scale factor of 100:</p>
+<pre><code class="language-sql">SELECT
+ *,
+ case o_orderstatus
+ when 'O' then 'ordered'
+ when 'F' then 'filled'
+ when 'P' then 'pending'
+ else 'other'
+ end
+from orders
+</code></pre>
+<p>This query was first run with DataFusion 50.0.0 to get a baseline measurement.
+The same query was then run with each optimization applied in turn.
+The recorded times are presented as the blue series in the chart below.
+The green series shows the time measurement for the <code>SELECT * FROM orders</code> to give an idea of the cost the addition of a <code>CASE</code> expression in a query incurs.
+All measurements were made with a target partition count of <code>1</code>.</p>
+<figure>
+<img alt="Performance measurements chart" class="img-responsive" src="/blog/images/case/results.png" width="100%"/>
+<figcaption>Performance measurements</figcaption>
+</figure>
+<p>What you can see in the chart is that the effect of the various optimizations compounds up to the <code>project</code> measurement.
+Up to that point these results are applicable to any <code>CASE</code> expression.
+The final improvement in the <code>hash</code> measurement is only applicable to simple <code>CASE</code> expressions with constant <code>WHEN</code> and <code>THEN</code> expressions.</p>
+<p>The cumulative effect of these optimizations is a 63-71% reduction in CPU time spent evaluating <code>CASE</code> expressions compared to the baseline.</p>
+<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h2>
+<p>Through a number of targeted optimizations, we've transformed <code>CASE</code> expression evaluation from a simple, but unoptimized implementation into a highly optimized one.
+The optimizations described in this post compound: a <code>CASE</code> expression on a wide table with multiple branches and early matches benefits from all four optimizations simultaneously.
+The result is significantly reduced CPU time and memory allocation in SQL constructs that are essential for ETL-like queries.</p>
+<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries,
+and machine learning and streaming applications.
+While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion’s primary design goal</a> is to accelerate the creation of other data-centric systems, it provides a reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe library</a>, <a href="https://datafusion.apache.org/python/">Python library</a>, and <a href="https://datafusion.apache.org/u [...]
+<p>DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone.
+Without DataFusion, highly performant vectorized query engines would remain the domain of a few large companies and world-class research institutions.
+With DataFusion, we can all build on top of a shared foundation and focus on what makes our projects unique.</p>
+<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
+<p>DataFusion is not a project built or driven by a single person, company, or foundation.
+Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code.
+A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.13.0 Release</title><link href="https://datafusion.a [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development work and is the result of merging 169 PRs from 15
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.13.0.md">change log</a> for more information.</p>
+<h2 id="key-features">Key Features<a class="headerlink" href="#key-features" title="Permanent link">¶</a></h2>
+<h3 id="native-parquet-write-support-experimental">Native Parquet Write Support (Experimental)<a class="headerlink" href="#native-parquet-write-support-experimental" title="Permanent link">¶</a></h3>
+<p>This release introduces experimental native Parquet write capabilities, allowing Comet to intercept and execute Parquet write operations natively through DataFusion. Key capabilities include:</p>
+<ul>
+<li>File commit protocol support for reliable writes</li>
+<li>Remote HDFS writing via OpenDAL integration</li>
+<li>Complex type support (arrays, maps, structs)</li>
+<li>Proper handling of object store settings</li>
+</ul>
+<p>To enable native Parquet writes, set:</p>
+<pre><code>spark.comet.allowIncompatibleOp.DataWritingCommandExec=true
+spark.comet.parquet.write.enabled=true
+</code></pre>
+<p><strong>Note</strong>: This feature is highly experimental and should not be used in production environments. It is currently categorized as a testing feature and is disabled by default.</p>
+<h3 id="native-iceberg-improvements">Native Iceberg Improvements<a class="headerlink" href="#native-iceberg-improvements" title="Permanent link">¶</a></h3>
+<p>Comet's fully-native Iceberg integration received significant enhancements in this release:</p>
+<p><strong>REST Catalog Support</strong>: Native Iceberg scans now support REST catalogs, enabling integration with catalog services like Apache Polaris and Tabular. Configure with:</p>
+<pre><code class="language-shell">--conf spark.sql.catalog.rest_cat=org.apache.iceberg.spark.SparkCatalog
+--conf spark.sql.catalog.rest_cat.catalog-impl=org.apache.iceberg.rest.RESTCatalog
+--conf spark.sql.catalog.rest_cat.uri=http://localhost:8181
+--conf spark.comet.scan.icebergNative.enabled=true
+</code></pre>
+<p><strong>Session Token Authentication</strong>: Added support for session tokens in native Iceberg scans for secure S3 access.</p>
+<p><strong>Performance Optimizations</strong>:</p>
+<ul>
+<li>Deduplicated serialized metadata reducing memory overhead</li>
+<li>Switched from JSON to protobuf for partition value serialization</li>
+<li>Removed IcebergFileStream in favor of iceberg-rust's built-in parallelization</li>
+<li>Reduced metadata serialization points</li>
+<li>Added SchemaAdapter caching</li>
+</ul>
+<p>To enable fully-native Iceberg scanning:</p>
+<pre><code>spark.comet.scan.icebergNative.enabled=true
+</code></pre>
+<p>The native reader supports Iceberg table spec v1 and v2, all primitive and complex types, schema evolution, time travel, positional and equality deletes, filter pushdown, and various storage backends (local, HDFS, S3).</p>
+<h3 id="native-csv-reading-experimental">Native CSV Reading (Experimental)<a class="headerlink" href="#native-csv-reading-experimental" title="Permanent link">¶</a></h3>
+<p>Experimental support for native CSV file reading has been added, expanding Comet's file format capabilities beyond Parquet.</p>
+<h3 id="new-expressions">New Expressions<a class="headerlink" href="#new-expressions" title="Permanent link">¶</a></h3>
+<p>The release adds support for numerous expressions:</p>
+<ul>
+<li>Array functions: <code>explode</code>, <code>explode_outer</code>, <code>size</code></li>
+<li>Date/time functions: <code>unix_date</code>, <code>date_format</code>, <code>datediff</code>, <code>last_day</code>, <code>unix_timestamp</code></li>
+<li>String functions: <code>left</code></li>
+<li>JSON functions: <code>from_json</code> (partial support)</li>
+</ul>
+<h3 id="ansi-mode-support">ANSI Mode Support<a class="headerlink" href="#ansi-mode-support" title="Permanent link">¶</a></h3>
+<p>Sum and average aggregate expressions now support ANSI mode for both integer and decimal inputs, enabling overflow checking in strict SQL mode.</p>
+<h3 id="native-shuffle-improvements">Native Shuffle Improvements<a class="headerlink" href="#native-shuffle-improvements" title="Permanent link">¶</a></h3>
+<ul>
+<li>Round-robin partitioning is now supported in native shuffle</li>
+<li>Spill metrics are now reported correctly</li>
+<li>Configurable shuffle writer buffer size via <code>spark.comet.shuffle.write.bufferSize</code></li>
+</ul>
+<h2 id="performance-improvements">Performance Improvements<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>This release includes extensive performance optimizations:</p>
+<ul>
+<li><strong>String to integer casting</strong>: Significant speedups through optimized parsing</li>
+<li><strong>String functions</strong>: Optimized <code>lpad</code>/<code>rpad</code> to remove unnecessary memory allocations</li>
+<li><strong>Date operations</strong>: Improved <code>normalize_nan</code> and date truncate performance</li>
+<li><strong>Query planning</strong>: Cached query plans to avoid per-partition serialization overhead</li>
+<li><strong>Memory efficiency</strong>: Reduced GC pressure in protobuf serialization</li>
+<li><strong>Hash operations</strong>: Optimized complex-type hash implementations including murmur3 support for nested types</li>
+<li><strong>Runtime efficiency</strong>: Eliminated busy-polling of Tokio stream for plans without CometScan</li>
+<li><strong>Metrics overhead</strong>: Reduced timer and syscall overhead in native shuffle writer</li>
+</ul>
+<h2 id="deprecations">Deprecations<a class="headerlink" href="#deprecations" title="Permanent link">¶</a></h2>
+<p>The <code>native_comet</code> scan mode is now deprecated in favor of <code>native_iceberg_compat</code> and will be removed in a future release. The <code>auto</code> scan mode no longer falls back to <code>native_comet</code>.</p>
+<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
+<p>This release upgrades to DataFusion 51, Arrow 57, and the latest iceberg-rust. The minimum supported Rust version is now 1.88.</p>
+<p>Supported platforms include Spark 3.4.3, 3.5.4-3.5.7, and Spark 4.0.x with various JDK and Scala combinations.</p>
+<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="alternate"></link><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusio [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/52.0.0">DataFusion 52.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/">DataFusion 51.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>We continue to …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/52.0.0">DataFusion 52.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/">DataFusion 51.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>We continue to make significant performance improvements in DataFusion as explained below.</p>
+<h3 id="faster-case-expressions">Faster <code>CASE</code> Expressions<a class="headerlink" href="#faster-case-expressions" title="Permanent link">¶</a></h3>
+<p>DataFusion 52 has lookup-table-based evaluation for certain <code>CASE</code> expressions
+to avoid repeated evaluation for accelerating common ETL patterns such as</p>
+<pre><code class="language-sql">CASE company
+ WHEN 1 THEN 'Apple'
+ WHEN 5 THEN 'Samsung'
+ WHEN 2 THEN 'Motorola'
+ WHEN 3 THEN 'LG'
+ ELSE 'Other'
+END
+</code></pre>
+<p>This is the final work in our <code>CASE</code> performance epic (<a href="https://github.com/apache/datafusion/issues/18075">#18075</a>), which has
+improved <code>CASE</code> evaluation significantly. Related PRs <a href="https://github.com/apache/datafusion/pull/18183">#18183</a>. Thanks to
+<a href="https://github.com/rluvaton">rluvaton</a> and <a href="https://github.com/pepijnve">pepijnve</a> for the implementation.</p>
+<h3 id="minmax-aggregate-dynamic-filters"><code>MIN</code>/<code>MAX</code> Aggregate Dynamic Filters<a class="headerlink" href="#minmax-aggregate-dynamic-filters" title="Permanent link">¶</a></h3>
+<p>DataFusion now creates dynamic filters for queries with <code>MIN</code>/<code>MAX</code> aggregates
+that have filters, but no <code>GROUP BY</code>. These dynamic filters are used during scan
+to prune files and rows as tighter bounds are discovered during execution, as
+explained in the <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/#hash-join-dynamic-filters">Dynamic Filtering Blog</a>. For example, the following query:</p>
+<pre><code class="language-sql">SELECT min(l_shipdate)
+FROM lineitem
+WHERE l_returnflag = 'R';
+</code></pre>
+<p>Is now executed like this </p>
+<pre><code class="language-sql">SELECT min(l_shipdate)
+FROM lineitem
+-- '__current_min' is updated dynamically during execution
+WHERE l_returnflag = 'R' AND l_shipdate &lt; __current_min;
+</code></pre>
+<p>Thanks to <a href="https://github.com/2010YOUY01">2010YOUY01</a> for implementing this feature, with reviews from
+<a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/adriangb">adriangb</a>, and <a href="https://github.com/LiaCastaneda">LiaCastaneda</a>. Related PRs: <a href="https://github.com/apache/datafusion/pull/18644">#18644</a></p>
+<h3 id="new-merge-join">New Merge Join<a class="headerlink" href="#new-merge-join" title="Permanent link">¶</a></h3>
+<p>DataFusion 52 includes a rewrite of the sort-merge join (SMJ) operator, with
+speedups of three orders of magnitude in some pathological cases such as the
+case in <a href="https://github.com/apache/datafusion/issues/18487">#18487</a>, which also affected <a href="https://datafusion.apache.org/comet/">Apache Comet</a> workloads. Benchmarks in
+<a href="https://github.com/apache/datafusion/pull/18875">#18875</a> show dramatic gains for TPC-H Q21 (minutes to milliseconds) while
+leaving other queries unchanged or modestly faster. Thanks to <a href="https://github.com/mbutrovich">mbutrovich</a> for
+the implementation and reviews from <a href="https://github.com/Dandandan">Dandandan</a>.</p>
+<h3 id="caching-improvements">Caching Improvements<a class="headerlink" href="#caching-improvements" title="Permanent link">¶</a></h3>
+<p>This release also includes several additional caching improvements.</p>
+<p>A new statistics cache for File Metadata avoids repeatedly (re)calculating
+statistics for files. This significantly improves planning time
+for certain queries. You can see the contents of the new cache using the
+<a href="https://datafusion.apache.org/user-guide/cli/functions.html#statistics-cache">statistics_cache</a> function in the CLI:</p>
+<pre><code class="language-sql">select * from statistics_cache();
++------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
+| path | file_modified | file_size_bytes | e_tag | version | num_rows | num_columns | table_size_bytes | statistics_size_bytes |
++------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
+| .../hits.parquet | 2022-06-25T22:22:22 | 14779976446 | 0-5e24d1ee16380-370f48 | NULL | Exact(99997497) | 105 | Exact(36445943240) | 0 |
++------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
+</code></pre>
+<p>Thanks to <a href="https://github.com/bharath-techie">bharath-techie</a> and <a href="https://github.com/nuno-faria">nuno-faria</a> for implementing the statistics cache,
+with reviews from <a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/alamb">alamb</a>, and <a href="https://github.com/alchemist51">alchemist51</a>.
+Related PRs: <a href="https://github.com/apache/datafusion/pull/18971">#18971</a>, <a href="https://github.com/apache/datafusion/pull/19054">#19054</a></p>
+<p>A prefix-aware list-files cache accelerates evaluating partition predicates for
+Hive partitioned tables.</p>
+<pre><code class="language-sql">-- Read the hive partitioned dataset from Overture Maps (100s of Parquet files)
+CREATE EXTERNAL TABLE overturemaps
+STORED AS PARQUET LOCATION 's3://overturemaps-us-west-2/release/2025-12-17.0/';
+-- Find all files where the path contains `theme=base without requiring another LIST call
+select count(*) from overturemaps where theme='base';
+</code></pre>
+<p>You can see the
+contents of the new cache using the <a href="https://datafusion.apache.org/user-guide/cli/functions.html#list-files-cache">list_files_cache</a> function in the CLI:</p>
+<pre><code class="language-sql">create external table overturemaps
+stored as parquet
+location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure';
+0 row(s) fetched.
+&gt; select table, path, metadata_size_bytes, expires_in, unnest(metadata_list)['file_size_bytes'] as file_size_bytes, unnest(metadata_list)['e_tag'] as e_tag from list_files_cache() limit 10;
++--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
+| table | path | metadata_size_bytes | expires_in | file_size_bytes | e_tag |
++--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 999055952 | "35fc8fbe8400960b54c66fbb408c48e8-60" |
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 975592768 | "8a16e10b722681cdc00242564b502965-59" |
+...
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1016732378 | "6d70857a0473ed9ed3fc6e149814168b-61" |
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 991363784 | "c9cafb42fcbb413f851691c895dd7c2b-60" |
+| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1032469715 | "7540252d0d67158297a67038a3365e0f-62" |
++--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
+</code></pre>
+<p>Thanks to <a href="https://github.com/BlakeOrth">BlakeOrth</a> and <a href="https://github.com/Yuvraj-cyborg">Yuvraj-cyborg</a> for implementing the list-files cache work,
+with reviews from <a href="https://github.com/gabotechs">gabotechs</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/alchemist51">alchemist51</a>, <a href="https://github.com/martin-g">martin-g</a>, and <a href="https://github.com/BlakeOrth">BlakeOrth</a>.
+Related PRs: <a href="https://github.com/apache/datafusion/pull/18146">#18146</a>, <a href="https://github.com/apache/datafusion/pull/18855">#18855</a>, <a href="https://github.com/apache/datafusion/pull/19366">#19366</a>, <a href="https://github.com/apache/datafusion/pull/19298">#19298</a>, </p>
+<h3 id="improved-hash-join-filter-pushdown">Improved Hash Join Filter Pushdown<a class="headerlink" href="#improved-hash-join-filter-pushdown" title="Permanent link">¶</a></h3>
+<p>Starting in DataFusion 51, filtering information from <code>HashJoinExec</code> is passed
+dynamically to scans, as explained in the <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/#hash-join-dynamic-filters">Dynamic Filtering Blog</a> using a
+technique referred to as <a href="https://dl.acm.org/doi/10.1109/ICDE.2008.4497486">Sideways Information Passing</a> in Database research
+literature. The initial implementation passed min/max values for the join keys.
+DataFusion 52 extends the optimization (<a href="https://github.com/apache/datafusion/issues/17171">#17171</a> / <a href="https://github.com/apache/datafusion/pull/18393">#18393</a>) to pass the
+contents of the build side hash map. These filters are evaluated on the probe
+side scan to prune files, row groups, and individual rows. When the build side
+contains <code>20</code> or fewer rows (configurable) the contents of the hash map are
+transformed to an <code>IN</code> expression and used for <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.html">statistics-based pruning</a> which
+can avoid reading entire files or row groups that contain no matching join keys.
+Thanks to <a href="https://github.com/adriangb">adriangb</a> for implementing this feature, with reviews from
+<a href="https://github.com/LiaCastaneda">LiaCastaneda</a>, <a href="https://github.com/asolimando">asolimando</a>, <a href="https://github.com/comphead">comphead</a>, and <a href="https://github.com/mbutrovich">mbutrovich</a>.</p>
+<h2 id="major-features">Major Features ✨<a class="headerlink" href="#major-features" title="Permanent link">¶</a></h2>
+<h3 id="arrow-ipc-stream-file-support">Arrow IPC Stream file support<a class="headerlink" href="#arrow-ipc-stream-file-support" title="Permanent link">¶</a></h3>
+<p>DataFusion can now read Arrow IPC stream files (<a href="https://github.com/apache/datafusion/pull/18457">#18457</a>). This expands
+interoperability with systems that emit Arrow streams directly, making it
+simpler to ingest Arrow-native data without conversion. Thanks to <a href="https://github.com/corasaurus-hex">corasaurus-hex</a>
+for implementing this feature, with reviews from <a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/Jefffrey">Jefffrey</a>,
+<a href="https://github.com/jdcasale">jdcasale</a>, <a href="https://github.com/2010YOUY01">2010YOUY01</a>, and <a href="https://github.com/timsaucer">timsaucer</a>.</p>
+<pre><code class="language-sql">CREATE EXTERNAL TABLE ipc_events
+STORED AS ARROW
+LOCATION 's3://bucket/events.arrow';
+</code></pre>
+<p>Related PRs: <a href="https://github.com/apache/datafusion/pull/18457">#18457</a></p>
+<h3 id="more-extensible-sql-planning-with-relationplanner">More Extensible SQL Planning with <code>RelationPlanner</code><a class="headerlink" href="#more-extensible-sql-planning-with-relationplanner" title="Permanent link">¶</a></h3>
+<p>DataFusion now has an API for extending the SQL planner for relations, as
+explained in the <a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql/">Extending SQL in DataFusion Blog</a>. In addition to the existing
+expression and types extension points, this new API now allows extending <code>FROM</code>
+clauses. Using these APIs it is straightforward to provide SQL support for
+almost any dialect, including vendor-specific syntax. Example use cases include:</p>
+<pre><code class="language-sql">-- Postgres-style JSON operators
+SELECT payload-&gt;'user'-&gt;&gt;'id' FROM logs;
+-- MySQL-specific types
+SELECT DATETIME '2001-01-01 18:00:00';
+-- Statistical sampling
+SELECT * FROM sensor_data TABLESAMPLE BERNOULLI(10 PERCENT);
+</code></pre>
+<p>Thanks to <a href="https://github.com/geoffreyclaude">geoffreyclaude</a> for implementing relation planner extensions, and to
+<a href="https://github.com/theirix">theirix</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/NGA-TRAN">NGA-TRAN</a>, and <a href="https://github.com/gabotechs">gabotechs</a> for reviews and feedback on the
+design. Related PRs: <a href="https://github.com/apache/datafusion/pull/17843">#17843</a></p>
+<h3 id="expression-evaluation-pushdown-to-scans">Expression Evaluation Pushdown to Scans<a class="headerlink" href="#expression-evaluation-pushdown-to-scans" title="Permanent link">¶</a></h3>
+<p>DataFusion now pushes down expression evaluation into TableProviders using
+<a href="https://docs.rs/datafusion/52.0.0/datafusion/physical_expr_adapter/trait.PhysicalExprAdapter.html">PhysicalExprAdapter</a>, replacing the older SchemaAdapter approach (<a href="https://github.com/apache/datafusion/issues/14993">#14993</a>,
+<a href="https://github.com/apache/datafusion/issues/16800">#16800</a>). Predicates and expressions can now be customized for each
+individual file schema, opening additional optimization such as support for
+<a href="https://github.com/apache/datafusion/issues/16116">Variant shredding</a>. Thanks to <a href="https://github.com/adriangb">adriangb</a> for implementing PhysicalExprAdapter
+and reworking pushdown to use it. Related PRs: <a href="https://github.com/apache/datafusion/pull/18998">#18998</a>, <a href="https://github.com/apache/datafusion/pull/19345">#19345</a></p>
+<h3 id="sort-pushdown-to-scans">Sort Pushdown to Scans<a class="headerlink" href="#sort-pushdown-to-scans" title="Permanent link">¶</a></h3>
+<p>DataFusion can now push sorts into data sources (<a href="https://github.com/apache/datafusion/issues/10433">#10433</a>, <a href="https://github.com/apache/datafusion/pull/19064">#19064</a>).
+This allows table provider implementations to optimize based on
+sort knowledge for certain query patterns. For example, the provided Parquet
+data source now reverses the scan order of row groups and files when queried
+for the opposite of the file's natural sort (e.g. <code>DESC</code> when the files are sorted <code>ASC</code>).
+This reversal, combined with dynamic filtering, allows top-K queries with <code>LIMIT</code>
+on pre-sorted data to find the requested rows very quickly, pruning more files and row groups
+without even scanning them. We have seen a ~30x performance improvement on
+benchmark queries with pre-sorted data.
+Thanks to <a href="https://github.com/zhuqi-lucas">zhuqi-lucas</a> and <a href="https://github.com/xudong963">xudong963</a> for this feature, with reviews from
+<a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/adriangb">adriangb</a>, and <a href="https://github.com/alamb">alamb</a>.</p>
+<h3 id="tableprovider-supports-delete-and-update-statements"><code>TableProvider</code> supports <code>DELETE</code> and <code>UPDATE</code> statements<a class="headerlink" href="#tableprovider-supports-delete-and-update-statements" title="Permanent link">¶</a></h3>
+<p>The <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html">TableProvider</a> trait now includes hooks for <code>DELETE</code> and <code>UPDATE</code>
+statements and the basic MemTable implements them (<a href="https://github.com/apache/datafusion/pull/19142">#19142</a>). This lets
+downstream implementations and storage engines plug in their own mutation logic.
+See <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html#method.delete_from">TableProvider::delete_from</a> and <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html#method.update">TableProvider::update</a> for more details.</p>
+<p>Example:</p>
+<pre><code class="language-sql">DELETE FROM mem_table WHERE status = 'obsolete';
+</code></pre>
+<p>Thanks to <a href="https://github.com/ethan-tyler">ethan-tyler</a> for the implementation and <a href="https://github.com/alamb">alamb</a> and <a href="https://github.com/adriangb">adriangb</a> for
+reviews.</p>
+<h3 id="coalescebatchesexec-removed"><code>CoalesceBatchesExec</code> Removed<a class="headerlink" href="#coalescebatchesexec-removed" title="Permanent link">¶</a></h3>
+<p>The standalone <code>CoalesceBatchesExec</code> operator existed to ensure batches were
+large enough for subsequent vectorized execution, and was inserted after
+filter-like operators such as <code>FilterExec</code>, <code>HashJoinExec</code>, and
+<code>RepartitionExec</code>. However, using a separate operator also blocks other
+optimizations such as pushing <code>LIMIT</code> through joins and made optimizer rules
+more complex. In this release, we integrated the coalescing into the operators
+themselves (<a href="https://github.com/apache/datafusion/issues/18779">#18779</a>) using Arrow's <a href="https://docs.rs/arrow/57.2.0/arrow/compute/kernels/coalesce/">coalesce kernel</a>. This reduces plan
+complexity while keeping batch sizes efficient, and allows additional focused
+optimization work in the Arrow kernel, such as <a href="https://github.com/Dandandan">Dandandan</a>'s recent work with
+filtering in <a href="https://github.com/apache/arrow-rs/pull/8951">arrow-rs/#8951</a>.</p>
+<p>Related PRs: <a href="https://github.com/apache/datafusion/pull/18540">#18540</a>, <a href="https://github.com/apache/datafusion/pull/18604">#18604</a>, <a href="https://github.com/apache/datafusion/pull/18630">#18630</a>, <a href="https://github.com/apache/datafusion/pull/18972">#18972</a>, <a href="https://github.com/apache/datafusion/pull/19002">#19002</a>, <a href="https://github.com/apache/datafusion/pull/19342" [...]
+Thanks to <a href="https://github.com/Tim-53">Tim-53</a>, <a href="https://github.com/Dandandan">Dandandan</a>, <a href="https://github.com/jizezhang">jizezhang</a>, and <a href="https://github.com/feniljain">feniljain</a> for implementing
+this feature, with reviews from <a href="https://github.com/Jefffrey">Jefffrey</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/martin-g">martin-g</a>,
+<a href="https://github.com/geoffreyclaude">geoffreyclaude</a>, <a href="https://github.com/milenkovicm">milenkovicm</a>, and <a href="https://github.com/jizezhang">jizezhang</a>.</p>
+<h2 id="upgrade-guide-and-changelog">Upgrade Guide and Changelog<a class="headerlink" href="#upgrade-guide-and-changelog" title="Permanent link">¶</a></h2>
+<p>As always, upgrading to 52.0.0 should be straightforward for most users. Please review the
+<a href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade Guide</a>
+for details on breaking changes and code snippets to help with the transition.
+For a comprehensive list of all changes, please refer to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>.</p>
+<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses
+<a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
+create new, fast, data-centric systems such as databases, dataframe libraries,
+and machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion's primary
+design goal</a> is to accelerate the creation of other data-centric systems, it
+provides a reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe
+library</a>, <a href="https://datafusion.apache.org/python/">Python library</a>, and <a href="https://datafusion.apache.org/user-guide/cli/">command-line SQL tool</a>.</p>
+<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
+<p>DataFusion is not a project built or driven by a single person, company, or
+foundation. Rather, our community of users and contributors works together to
+build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us, we would love to have you. You can try out
+DataFusion on some of your own data and projects and let us know how it goes,
+contribute suggestions, documentation, bug reports, or a PR with documentation,
+tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="alternate"></link><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>Ge [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -283,245 +1938,7 @@ println!("{}", df.logical_plan().display_indent());
<li><strong>Try it out</strong>: Implement one of the extension points and share your experience</li>
<li><strong>File issues or join the conversation</strong>: <a href="https://github.com/apache/datafusion/">GitHub</a> for bugs and feature requests, <a href="https://datafusion.apache.org/contributor-guide/communication.html">Slack or Discord</a> for discussion</li>
</ul>
-<!-- Reference links --></content><category term="blog"></category></entry><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/08/datafusion-52.0.0" rel="alternate"></link><published>2026-01-08T00:00:00+00:00</published><updated>2026-01-08T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2026-01-08:/blog/2026/01/08/datafusion-52.0.0</id><summary type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/52.0.0">DataFusion 52.0.0</a>. This post highlights
-some of the major improvements since <a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/">DataFusion 51.0.0</a>. The complete list of
-changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
-making this release possible.</p>
-<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue to …</p></summary><content type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/52.0.0">DataFusion 52.0.0</a>. This post highlights
-some of the major improvements since <a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/">DataFusion 51.0.0</a>. The complete list of
-changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
-making this release possible.</p>
-<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue to make significant performance improvements in DataFusion as explained below.</p>
-<h3 id="faster-case-expressions">Faster <code>CASE</code> Expressions<a class="headerlink" href="#faster-case-expressions" title="Permanent link">¶</a></h3>
-<p>DataFusion 52 has lookup-table-based evaluation for certain <code>CASE</code> expressions
-to avoid repeated evaluation for accelerating common ETL patterns such as</p>
-<pre><code class="language-sql">CASE company
- WHEN 1 THEN 'Apple'
- WHEN 5 THEN 'Samsung'
- WHEN 2 THEN 'Motorola'
- WHEN 3 THEN 'LG'
- ELSE 'Other'
-END
-</code></pre>
-<p>This is the final work in our <code>CASE</code> performance epic (<a href="https://github.com/apache/datafusion/issues/18075">#18075</a>), which has
-improved <code>CASE</code> evaluation significantly. Related PRs <a href="https://github.com/apache/datafusion/pull/18183">#18183</a>. Thanks to
-<a href="https://github.com/rluvaton">rluvaton</a> and <a href="https://github.com/pepijnve">pepijnve</a> for the implementation.</p>
-<h3 id="minmax-aggregate-dynamic-filters"><code>MIN</code>/<code>MAX</code> Aggregate Dynamic Filters<a class="headerlink" href="#minmax-aggregate-dynamic-filters" title="Permanent link">¶</a></h3>
-<p>DataFusion now creates dynamic filters for queries with <code>MIN</code>/<code>MAX</code> aggregates
-that have filters, but no <code>GROUP BY</code>. These dynamic filters are used during scan
-to prune files and rows as tighter bounds are discovered during execution, as
-explained in the <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/#hash-join-dynamic-filters">Dynamic Filtering Blog</a>. For example, the following query:</p>
-<pre><code class="language-sql">SELECT min(l_shipdate)
-FROM lineitem
-WHERE l_returnflag = 'R';
-</code></pre>
-<p>Is now executed like this </p>
-<pre><code class="language-sql">SELECT min(l_shipdate)
-FROM lineitem
--- '__current_min' is updated dynamically during execution
-WHERE l_returnflag = 'R' AND l_shipdate &lt; __current_min;
-</code></pre>
-<p>Thanks to <a href="https://github.com/2010YOUY01">2010YOUY01</a> for implementing this feature, with reviews from
-<a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/adriangb">adriangb</a>, and <a href="https://github.com/LiaCastaneda">LiaCastaneda</a>. Related PRs: <a href="https://github.com/apache/datafusion/pull/18644">#18644</a></p>
-<h3 id="new-merge-join">New Merge Join<a class="headerlink" href="#new-merge-join" title="Permanent link">¶</a></h3>
-<p>DataFusion 52 includes a rewrite of the sort-merge join (SMJ) operator, with
-speedups of three orders of magnitude in some pathological cases such as the
-case in <a href="https://github.com/apache/datafusion/issues/18487">#18487</a>, which also affected <a href="https://datafusion.apache.org/comet/">Apache Comet</a> workloads. Benchmarks in
-<a href="https://github.com/apache/datafusion/pull/18875">#18875</a> show dramatic gains for TPC-H Q21 (minutes to milliseconds) while
-leaving other queries unchanged or modestly faster. Thanks to <a href="https://github.com/mbutrovich">mbutrovich</a> for
-the implementation and reviews from <a href="https://github.com/Dandandan">Dandandan</a>.</p>
-<h3 id="caching-improvements">Caching Improvements<a class="headerlink" href="#caching-improvements" title="Permanent link">¶</a></h3>
-<p>This release also includes several additional caching improvements.</p>
-<p>A new statistics cache for File Metadata avoids repeatedly (re)calculating
-statistics for files. This significantly improves planning time
-for certain queries. You can see the contents of the new cache using the
-<a href="https://datafusion.apache.org/user-guide/cli/functions.html#statistics-cache">statistics_cache</a> function in the CLI:</p>
-<pre><code class="language-sql">select * from statistics_cache();
-+------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
-| path | file_modified | file_size_bytes | e_tag | version | num_rows | num_columns | table_size_bytes | statistics_size_bytes |
-+------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
-| .../hits.parquet | 2022-06-25T22:22:22 | 14779976446 | 0-5e24d1ee16380-370f48 | NULL | Exact(99997497) | 105 | Exact(36445943240) | 0 |
-+------------------+---------------------+-----------------+------------------------+---------+-----------------+-------------+--------------------+-----------------------+
-</code></pre>
-<p>Thanks to <a href="https://github.com/bharath-techie">bharath-techie</a> and <a href="https://github.com/nuno-faria">nuno-faria</a> for implementing the statistics cache,
-with reviews from <a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/alamb">alamb</a>, and <a href="https://github.com/alchemist51">alchemist51</a>.
-Related PRs: <a href="https://github.com/apache/datafusion/pull/18971">#18971</a>, <a href="https://github.com/apache/datafusion/pull/19054">#19054</a></p>
-<p>A prefix-aware list-files cache accelerates evaluating partition predicates for
-Hive partitioned tables.</p>
-<pre><code class="language-sql">-- Read the hive partitioned dataset from Overture Maps (100s of Parquet files)
-CREATE EXTERNAL TABLE overturemaps
-STORED AS PARQUET LOCATION 's3://overturemaps-us-west-2/release/2025-12-17.0/';
--- Find all files where the path contains `theme=base without requiring another LIST call
-select count(*) from overturemaps where theme='base';
-</code></pre>
-<p>You can see the
-contents of the new cache using the <a href="https://datafusion.apache.org/user-guide/cli/functions.html#list-files-cache">list_files_cache</a> function in the CLI:</p>
-<pre><code class="language-sql">create external table overturemaps
-stored as parquet
-location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure';
-0 row(s) fetched.
-&gt; select table, path, metadata_size_bytes, expires_in, unnest(metadata_list)['file_size_bytes'] as file_size_bytes, unnest(metadata_list)['e_tag'] as e_tag from list_files_cache() limit 10;
-+--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
-| table | path | metadata_size_bytes | expires_in | file_size_bytes | e_tag |
-+--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 999055952 | "35fc8fbe8400960b54c66fbb408c48e8-60" |
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 975592768 | "8a16e10b722681cdc00242564b502965-59" |
-...
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1016732378 | "6d70857a0473ed9ed3fc6e149814168b-61" |
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 991363784 | "c9cafb42fcbb413f851691c895dd7c2b-60" |
-| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1032469715 | "7540252d0d67158297a67038a3365e0f-62" |
-+--------------+-----------------------------------------------------+---------------------+-----------------------------------+-----------------+---------------------------------------+
-</code></pre>
-<p>Thanks to <a href="https://github.com/BlakeOrth">BlakeOrth</a> and <a href="https://github.com/Yuvraj-cyborg">Yuvraj-cyborg</a> for implementing the list-files cache work,
-with reviews from <a href="https://github.com/gabotechs">gabotechs</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/alchemist51">alchemist51</a>, <a href="https://github.com/martin-g">martin-g</a>, and <a href="https://github.com/BlakeOrth">BlakeOrth</a>.
-Related PRs: <a href="https://github.com/apache/datafusion/pull/18146">#18146</a>, <a href="https://github.com/apache/datafusion/pull/18855">#18855</a>, <a href="https://github.com/apache/datafusion/pull/19366">#19366</a>, <a href="https://github.com/apache/datafusion/pull/19298">#19298</a>, </p>
-<h3 id="improved-hash-join-filter-pushdown">Improved Hash Join Filter Pushdown<a class="headerlink" href="#improved-hash-join-filter-pushdown" title="Permanent link">¶</a></h3>
-<p>Starting in DataFusion 51, filtering information from <code>HashJoinExec</code> is passed
-dynamically to scans, as explained in the <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/#hash-join-dynamic-filters">Dynamic Filtering Blog</a> using a
-technique referred to as <a href="https://dl.acm.org/doi/10.1109/ICDE.2008.4497486">Sideways Information Passing</a> in Database research
-literature. The initial implementation passed min/max values for the join keys.
-DataFusion 52 extends the optimization (<a href="https://github.com/apache/datafusion/issues/17171">#17171</a> / <a href="https://github.com/apache/datafusion/pull/18393">#18393</a>) to pass the
-contents of the build side hash map. These filters are evaluated on the probe
-side scan to prune files, row groups, and individual rows. When the build side
-contains <code>20</code> or fewer rows (configurable) the contents of the hash map are
-transformed to an <code>IN</code> expression and used for <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.html">statistics-based pruning</a> which
-can avoid reading entire files or row groups that contain no matching join keys.
-Thanks to <a href="https://github.com/adriangb">adriangb</a> for implementing this feature, with reviews from
-<a href="https://github.com/LiaCastaneda">LiaCastaneda</a>, <a href="https://github.com/asolimando">asolimando</a>, <a href="https://github.com/comphead">comphead</a>, and <a href="https://github.com/mbutrovich">mbutrovich</a>.</p>
-<h2 id="major-features">Major Features ✨<a class="headerlink" href="#major-features" title="Permanent link">¶</a></h2>
-<h3 id="arrow-ipc-stream-file-support">Arrow IPC Stream file support<a class="headerlink" href="#arrow-ipc-stream-file-support" title="Permanent link">¶</a></h3>
-<p>DataFusion can now read Arrow IPC stream files (<a href="https://github.com/apache/datafusion/pull/18457">#18457</a>). This expands
-interoperability with systems that emit Arrow streams directly, making it
-simpler to ingest Arrow-native data without conversion. Thanks to <a href="https://github.com/corasaurus-hex">corasaurus-hex</a>
-for implementing this feature, with reviews from <a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/Jefffrey">Jefffrey</a>,
-<a href="https://github.com/jdcasale">jdcasale</a>, <a href="https://github.com/2010YOUY01">2010YOUY01</a>, and <a href="https://github.com/timsaucer">timsaucer</a>.</p>
-<pre><code class="language-sql">CREATE EXTERNAL TABLE ipc_events
-STORED AS ARROW
-LOCATION 's3://bucket/events.arrow';
-</code></pre>
-<p>Related PRs: <a href="https://github.com/apache/datafusion/pull/18457">#18457</a></p>
-<h3 id="more-extensible-sql-planning-with-relationplanner">More Extensible SQL Planning with <code>RelationPlanner</code><a class="headerlink" href="#more-extensible-sql-planning-with-relationplanner" title="Permanent link">¶</a></h3>
-<p>DataFusion now has an API for extending the SQL planner for relations, as
-explained in the <a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql/">Extending SQL in DataFusion Blog</a>. In addition to the existing
-expression and types extension points, this new API now allows extending <code>FROM</code>
-clauses. Using these APIs it is straightforward to provide SQL support for
-almost any dialect, including vendor-specific syntax. Example use cases include:</p>
-<pre><code class="language-sql">-- Postgres-style JSON operators
-SELECT payload-&gt;'user'-&gt;&gt;'id' FROM logs;
--- MySQL-specific types
-SELECT DATETIME '2001-01-01 18:00:00';
--- Statistical sampling
-SELECT * FROM sensor_data TABLESAMPLE BERNOULLI(10 PERCENT);
-</code></pre>
-<p>Thanks to <a href="https://github.com/geoffreyclaude">geoffreyclaude</a> for implementing relation planner extensions, and to
-<a href="https://github.com/theirix">theirix</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/NGA-TRAN">NGA-TRAN</a>, and <a href="https://github.com/gabotechs">gabotechs</a> for reviews and feedback on the
-design. Related PRs: <a href="https://github.com/apache/datafusion/pull/17843">#17843</a></p>
-<h3 id="expression-evaluation-pushdown-to-scans">Expression Evaluation Pushdown to Scans<a class="headerlink" href="#expression-evaluation-pushdown-to-scans" title="Permanent link">¶</a></h3>
-<p>DataFusion now pushes down expression evaluation into TableProviders using
-<a href="https://docs.rs/datafusion/52.0.0/datafusion/physical_expr_adapter/trait.PhysicalExprAdapter.html">PhysicalExprAdapter</a>, replacing the older SchemaAdapter approach (<a href="https://github.com/apache/datafusion/issues/14993">#14993</a>,
-<a href="https://github.com/apache/datafusion/issues/16800">#16800</a>). Predicates and expressions can now be customized for each
-individual file schema, opening additional optimization such as support for
-<a href="https://github.com/apache/datafusion/issues/16116">Variant shredding</a>. Thanks to <a href="https://github.com/adriangb">adriangb</a> for implementing PhysicalExprAdapter
-and reworking pushdown to use it. Related PRs: <a href="https://github.com/apache/datafusion/pull/18998">#18998</a>, <a href="https://github.com/apache/datafusion/pull/19345">#19345</a></p>
-<h3 id="sort-pushdown-to-scans">Sort Pushdown to Scans<a class="headerlink" href="#sort-pushdown-to-scans" title="Permanent link">¶</a></h3>
-<p>DataFusion can now push sorts into data sources (<a href="https://github.com/apache/datafusion/issues/10433">#10433</a>, <a href="https://github.com/apache/datafusion/pull/19064">#19064</a>).
-This allows table provider implementations to optimize based on
-sort knowledge for certain query patterns. For example, the provided Parquet
-data source now reverses the scan order of row groups and files when queried
-for the opposite of the file's natural sort (e.g. <code>DESC</code> when the files are sorted <code>ASC</code>).
-This reversal, combined with dynamic filtering, allows top-K queries with <code>LIMIT</code>
-on pre-sorted data to find the requested rows very quickly, pruning more files and row groups
-without even scanning them. We have seen a ~30x performance improvement on
-benchmark queries with pre-sorted data.
-Thanks to <a href="https://github.com/zhuqi-lucas">zhuqi-lucas</a> and <a href="https://github.com/xudong963">xudong963</a> for this feature, with reviews from
-<a href="https://github.com/martin-g">martin-g</a>, <a href="https://github.com/adriangb">adriangb</a>, and <a href="https://github.com/alamb">alamb</a>.</p>
-<h3 id="tableprovider-supports-delete-and-update-statements"><code>TableProvider</code> supports <code>DELETE</code> and <code>UPDATE</code> statements<a class="headerlink" href="#tableprovider-supports-delete-and-update-statements" title="Permanent link">¶</a></h3>
-<p>The <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html">TableProvider</a> trait now includes hooks for <code>DELETE</code> and <code>UPDATE</code>
-statements and the basic MemTable implements them (<a href="https://github.com/apache/datafusion/pull/19142">#19142</a>). This lets
-downstream implementations and storage engines plug in their own mutation logic.
-See <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html#method.delete_from">TableProvider::delete_from</a> and <a href="https://docs.rs/datafusion/52.0.0/datafusion/datasource/trait.TableProvider.html#method.update">TableProvider::update</a> for more details.</p>
-<p>Example:</p>
-<pre><code class="language-sql">DELETE FROM mem_table WHERE status = 'obsolete';
-</code></pre>
-<p>Thanks to <a href="https://github.com/ethan-tyler">ethan-tyler</a> for the implementation and <a href="https://github.com/alamb">alamb</a> and <a href="https://github.com/adriangb">adriangb</a> for
-reviews.</p>
-<h3 id="coalescebatchesexec-removed"><code>CoalesceBatchesExec</code> Removed<a class="headerlink" href="#coalescebatchesexec-removed" title="Permanent link">¶</a></h3>
-<p>The standalone <code>CoalesceBatchesExec</code> operator existed to ensure batches were
-large enough for subsequent vectorized execution, and was inserted after
-filter-like operators such as <code>FilterExec</code>, <code>HashJoinExec</code>, and
-<code>RepartitionExec</code>. However, using a separate operator also blocks other
-optimizations such as pushing <code>LIMIT</code> through joins and made optimizer rules
-more complex. In this release, we integrated the coalescing into the operators
-themselves (<a href="https://github.com/apache/datafusion/issues/18779">#18779</a>) using Arrow's <a href="https://docs.rs/arrow/57.2.0/arrow/compute/kernels/coalesce/">coalesce kernel</a>. This reduces plan
-complexity while keeping batch sizes efficient, and allows additional focused
-optimization work in the Arrow kernel, such as <a href="https://github.com/Dandandan">Dandandan</a>'s recent work with
-filtering in <a href="https://github.com/apache/arrow-rs/pull/8951">arrow-rs/#8951</a>.</p>
-<p>Related PRs: <a href="https://github.com/apache/datafusion/pull/18540">#18540</a>, <a href="https://github.com/apache/datafusion/pull/18604">#18604</a>, <a href="https://github.com/apache/datafusion/pull/18630">#18630</a>, <a href="https://github.com/apache/datafusion/pull/18972">#18972</a>, <a href="https://github.com/apache/datafusion/pull/19002">#19002</a>, <a href="https://github.com/apache/datafusion/pull/19342" [...]
-Thanks to <a href="https://github.com/Tim-53">Tim-53</a>, <a href="https://github.com/Dandandan">Dandandan</a>, <a href="https://github.com/jizezhang">jizezhang</a>, and <a href="https://github.com/feniljain">feniljain</a> for implementing
-this feature, with reviews from <a href="https://github.com/Jefffrey">Jefffrey</a>, <a href="https://github.com/alamb">alamb</a>, <a href="https://github.com/martin-g">martin-g</a>,
-<a href="https://github.com/geoffreyclaude">geoffreyclaude</a>, <a href="https://github.com/milenkovicm">milenkovicm</a>, and <a href="https://github.com/jizezhang">jizezhang</a>.</p>
-<h2 id="upgrade-guide-and-changelog">Upgrade Guide and Changelog<a class="headerlink" href="#upgrade-guide-and-changelog" title="Permanent link">¶</a></h2>
-<p>As always, upgrading to 52.0.0 should be straightforward for most users. Please review the
-<a href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade Guide</a>
-for details on breaking changes and code snippets to help with the transition.
-For a comprehensive list of all changes, please refer to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>.</p>
-<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
-<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses
-<a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
-create new, fast, data-centric systems such as databases, dataframe libraries,
-and machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion's primary
-design goal</a> is to accelerate the creation of other data-centric systems, it
-provides a reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe
-library</a>, <a href="https://datafusion.apache.org/python/">Python library</a>, and <a href="https://datafusion.apache.org/user-guide/cli/">command-line SQL tool</a>.</p>
-<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
-<p>DataFusion is not a project built or driven by a single person, company, or
-foundation. Rather, our community of users and contributors works together to
-build a shared technology that none of us could have built alone.</p>
-<p>If you are interested in joining us, we would love to have you. You can try out
-DataFusion on some of your own data and projects and let us know how it goes,
-contribute suggestions, documentation, bug reports, or a PR with documentation,
-tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="alternate"></link><published>2025-12-15T00:00:00+00:00</published><updated>202 [...]
+<!-- Reference links --></content><category term="blog"></category></entry><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="alternate"></link><published>2025-12-15T00:00:00+00:00</published><updated>2025-12-15T00:00:00+00:00</updated><author><name>Gene Bordegaray</name></author><id>tag:datafusion.apache.org,2025-12-15:/blog/202 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6545,7 +7962,7 @@ to register the view and then use it in another place:</p>
<pre><code class="language-python">ctx.register_view("view1", df1)
</code></pre>
<p>And then in another portion of your code which has access to the same session context
-you can retrive the DataFrame with:</p>
+you can retrieve the DataFrame with:</p>
<pre><code>df2 = ctx.table("view1")
</code></pre>
<h2 id="asynchronous-iteration-of-record-batches">Asynchronous Iteration of Record Batches<a class="headerlink" href="#asynchronous-iteration-of-record-batches" title="Permanent link">¶</a></h2>
@@ -8179,7 +9596,7 @@ consistent method for exposing these data structures across libraries.</p>
<p>In <a href="https://github.com/apache/datafusion-python/pull/825">PR #825</a>, we introduced support for both importing and exporting Arrow data in
<code>datafusion-python</code>. With this improvement, you can now use a single function call to import
a table from <strong>any</strong> Python library that implements the <a href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html">Arrow PyCapsule Interface</a>.
-Many popular libaries, such as <a href="https://pandas.pydata.org/">Pandas</a> and <a href="https://pola.rs/">Polars</a>
+Many popular libraries, such as <a href="https://pandas.pydata.org/">Pandas</a> and <a href="https://pola.rs/">Polars</a>
already support these interfaces.</p>
<p>Suppose you have a Pandas and Polars DataFrames named <code>df_pandas</code> or <code>df_polars</code>, respectively:</p>
<pre><code class="language-python">ctx = SessionContext()
@@ -8239,7 +9656,7 @@ of the blog post describing how these enhancements can lead to 20-200% performan
gains in some tests.</p>
<p>During our testing we identified some cases where we needed to adjust workflows to
account for the fact that StringView is now the default type for string based operations.
-First, when performing manipulations on string objects there is a perfomance loss when
+First, when performing manipulations on string objects there is a performance loss when
needing to cast from string to string view or vice versa. To reap the best performance,
ideally all of your string type data will use StringView. For most users this should be
transparent. However if you specify a schema for reading or creating data, then you
@@ -8476,7 +9893,7 @@ than a join can be significantly faster. This is worth profiling for your specif
<p>I have a DataFrame with many values that I want to aggregate. I have already analyzed it and
determined there is a noise level below which I do not want to include in my analysis. I want to
compute a sum of only values that are above my noise threshold.</p>
-<p>This can be done fairly easy without leaning on a User Defined Aggegate Function (UDAF). You can
+<p>This can be done fairly easy without leaning on a User Defined Aggregate Function (UDAF). You can
simply filter the DataFrame and then aggregate using the built-in <code>sum</code> function. Here, we
demonstrate doing this as a UDF primarily as an example of how to write UDAFs. We will use the
PyArrow compute approach.</p>
@@ -8637,7 +10054,7 @@ Python, is to primarily demonstrate how to make the Python to Rust with Python w
transition. In the second implementation you can see how we can iterate through all of the arrays
ourselves.</p>
<p>In this first example, we are hard coding the values of interest, but in the following section
-we demonstrate passing these in during initalization.</p>
+we demonstrate passing these in during initialization.</p>
<pre><code class="language-rust">#[pyfunction]
pub fn tuple_filter_fn(
py: Python&lt;'_&gt;,
@@ -8860,13 +10277,13 @@ how much they have ordered total. We want to ignore small orders, which we defin
import pyarrow as pa
import pyarrow.compute as pc
-IGNORE_THESHOLD = 5000.0
+IGNORE_THRESHOLD = 5000.0
class AboveThresholdAccum(Accumulator):
def __init__(self) -&gt; None:
self._sum = 0.0
def update(self, values: pa.Array) -&gt; None:
- over_threshold = pc.greater(values, pa.scalar(IGNORE_THESHOLD))
+ over_threshold = pc.greater(values, pa.scalar(IGNORE_THRESHOLD))
sum_above = pc.sum(values.filter(over_threshold)).as_py()
if sum_above is None:
sum_above = 0.0
@@ -9441,8 +10858,8 @@ Figure 1 illustrates the difference between the output of both string representa
<h1 id="when-to-gc">When to GC?<a class="headerlink" href="#when-to-gc" title="Permanent link">¶</a></h1>
<p>Zero-copy <code>take/filter</code> is great for generating large arrays quickly, but it is suboptimal for highly selective filters, where most of the strings are filtered out. When the cardinality drops, StringViewArray buffers become sparse—only a small subset of the bytes in the buffer’s memory are referred to by any <code>view</code>. This leads to excessive memory usage, especially in a <a href="https://github.com/apache/datafusion/issues/11628"> [...]
<p>To release unused memory, we implemented a <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html#method.gc">garbage collection (GC)</a> routine to consolidate the data into a new buffer to release the old sparse buffer(s). As the GC operation copies strings, similarly to StringArray, we must be careful about when to call it. If we call GC too early, we cause unnecessary copying, losing much of the benefit of StringViewArray. If we call GC [...]
-<p><code>arrow-rs</code> implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/coalesce_batches/struct.CoalesceBatchesExec.html"><code>CoalseceBatchesExec</code></a> operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, whi [...]
-We, therefore,<a href="https://github.com/apache/datafusion/pull/11587"> implemented the GC procedure</a> inside <code>CoalseceBatchesExec</code>[^5] with a heuristic that estimates when the buffers are too sparse.</p>
+<p><code>arrow-rs</code> implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/coalesce_batches/struct.CoalesceBatchesExec.html"><code>CoalesceBatchesExec</code></a> operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, whi [...]
+We, therefore,<a href="https://github.com/apache/datafusion/pull/11587"> implemented the GC procedure</a> inside <code>CoalesceBatchesExec</code>[^5] with a heuristic that estimates when the buffers are too sparse.</p>
<h2 id="the-art-of-function-inlining-not-too-much-not-too-little">The art of function inlining: not too much, not too little<a class="headerlink" href="#the-art-of-function-inlining-not-too-much-not-too-little" title="Permanent link">¶</a></h2>
<p>Like string inlining, <em>function</em> inlining is the process of embedding a short function into the caller to avoid the overhead of function calls (caller/callee save).
Usually, the Rust compiler does a good job of deciding when to inline. However, it is possible to override its default using the <a href="https://doc.rust-lang.org/reference/attributes/codegen.html#the-inline-attribute"><code>#[inline(always)]</code> directive</a>.
@@ -9641,7 +11058,7 @@ to their Rust counterparts.</li>
<p>The most significant difference is that we have added wrapper functions and classes for most of the
user facing interface. These wrappers, written in Python, contain both documentation and type
annotations.</p>
-<p>This documenation is now available on the <a href="https://datafusion.apache.org/python/autoapi/datafusion/index.html">DataFusion in Python API</a> website. There you can browse
+<p>This documentation is now available on the <a href="https://datafusion.apache.org/python/autoapi/datafusion/index.html">DataFusion in Python API</a> website. There you can browse
the available functions and classes to see the breadth of available functionality.</p>
<p>Modern IDEs use language servers such as
<a href="https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance">Pylance</a> or
@@ -10565,7 +11982,7 @@ LIMIT 3;
3 rows in set. Query took 0.053 seconds.
</code></pre>
<h3 id="growth-of-datafusion">Growth of DataFusion 📈<a class="headerlink" href="#growth-of-datafusion" title="Permanent link">¶</a></h3>
-<p>DataFusion has been appearing more publically in the wild. For example
+<p>DataFusion has been appearing more publicly in the wild. For example
* New projects built using DataFusion such as <a href="https://lancedb.com/">lancedb</a>, <a href="https://glaredb.com/">GlareDB</a>, <a href="https://www.arroyo.dev/">Arroyo</a>, and <a href="https://github.com/cmu-db/optd">optd</a>.
* Public talks such as <a href="https://www.youtube.com/watch?v=AJU9rdRNk9I">Apache Arrow Datafusion: Vectorized
Execution Framework For Maximum Performance</a> in <a href="https://www.bagevent.com/event/8432178">CommunityOverCode Asia 2023</a>
@@ -11317,7 +12734,7 @@ required synchronous access to all relevant catalog information.</p>
<li>Automatic coercions ast between Date and Timestamp <a href="https://github.com/apache/arrow-datafusion/issues/4726">#4726</a></li>
<li>Support type coercion for timestamp and utf8 <a href="https://github.com/apache/arrow-datafusion/issues/4312">#4312</a></li>
<li>Full support for time32 and time64 literal values (<code>ScalarValue</code>) <a href="https://github.com/apache/arrow-datafusion/issues/4156">#4156</a></li>
-<li>New functions, incuding <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
+<li>New functions, including <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
<li>Compressed CSV/JSON support <a href="https://github.com/apache/arrow-datafusion/issues/3642">#3642</a></li>
</ul>
<p>The community has also invested in new <a href="https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/tests/sqllogictests/README.md">sqllogic based</a> tests to keep improving DataFusion's quality with less effort.</p>
@@ -12169,7 +13586,7 @@ git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc
<li>Switch from <code>std::sync::Mutex</code> to <code>parking_lot::Mutex</code> <a href="https://github.com/apache/arrow-datafusion/pull/1720">#1720</a></li>
<li>New Features</li>
<li>Support for memory tracking and spilling to disk<ul>
-<li>MemoryMananger and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
+<li>MemoryManager and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
<li>Out of core sort <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
<li>New metrics</li>
<li><code>Gauge</code> and <code>CurrentMemoryUsage</code> <a href="https://github.com/apache/arrow-datafusion/pull/1682">#1682</a></li>
diff --git a/blog/feeds/pepijn-van-eeckhoudt.atom.xml b/blog/feeds/pepijn-van-eeckhoudt.atom.xml
index fc58e1a..ad0b541 100644
--- a/blog/feeds/pepijn-van-eeckhoudt.atom.xml
+++ b/blog/feeds/pepijn-van-eeckhoudt.atom.xml
@@ -1,5 +1,400 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Pepijn Van Eeckhoudt</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/pepijn-van-eeckhoudt.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-06-30T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</title><link href="h [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Pepijn Van Eeckhoudt</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/pepijn-van-eeckhoudt.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-02-02T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Optimizing SQL CASE Expression Evaluation</title><link href="https://datafusion.apache.org/bl [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a set of expressions is evaluated for each row based on arbitrary boolean expressions.
+Its deceptively simple syntax hides significant implementation complexity.
+Over the past few <a href="https://datafusion.apache.org/">Apache DataFusion</a> releases, a series of improvements to <code>CASE</code> expression evaluator have been merged that reduce both CPU time and memory allocations.
+This post provides an overview of the original implementation, its performance bottlenecks, and the steps taken to address them.</p>
+<h2 id="background-case-expression-evaluation">Background: CASE Expression Evaluation<a class="headerlink" href="#background-case-expression-evaluation" title="Permanent link">¶</a></h2>
+<p>SQL supports two forms of CASE expressions:</p>
+<ol>
+<li><strong>Simple</strong>: <code>CASE expr WHEN value1 THEN result1 WHEN value2 THEN result2 ... END</code></li>
+<li><strong>Searched</strong>: <code>CASE WHEN condition1 THEN result1 WHEN condition2 THEN result2 ... END</code></li>
+</ol>
+<p>The simple form evaluates an expression once for each input row and then tests that value against the expressions (typically constants) in each <code>WHEN</code> clause using equality comparisons.</p>
+<p>Here's an example of the simple form:</p>
+<pre><code class="language-sql">CASE status
+ WHEN 'pending' THEN 1
+ WHEN 'active' THEN 2
+ WHEN 'complete' THEN 3
+ ELSE 0
+END
+</code></pre>
+<p>In this <code>CASE</code> expression, <code>status</code> is evaluated once per row, and then its value is tested for equality with the values <code>'pending'</code>, <code>'active'</code>, and <code>'complete'</code> in that order.
+The <code>CASE</code> expression evaluates to the value of the <code>THEN</code> expression corresponding to the first matching <code>WHEN</code> expression.</p>
+<p>The searched <code>CASE</code> form is a more flexible variant.
+It evaluates completely independent boolean expressions for each branch.
+This allows you to test different columns with different operators per branch as shown in the following example:</p>
+<pre><code class="language-sql">CASE
+ WHEN age &gt; 65 THEN 'senior'
+ WHEN childCount != 0 THEN 'parent'
+ WHEN age &lt; 21 THEN 'minor'
+ ELSE 'adult'
+END
+</code></pre>
+<p>In both forms, branches are evaluated sequentially with short-circuit semantics: for each row, once a <code>WHEN</code> condition matches, the corresponding <code>THEN</code> expression is evaluated.
+Any further branches are not evaluated for that row.
+This lazy evaluation model is critical for correctness.
+It lets you safely write <code>CASE</code> expressions like <code>CASE WHEN d != 0 THEN n / d ELSE NULL END</code> that are guaranteed to not trigger divide-by-zero errors.</p>
+<p>Besides <code>CASE</code>, there are a few <a href="https://datafusion.apache.org/user-guide/sql/scalar_functions.html#conditional-functions">conditional scalar functions</a> that provide similar, more restricted capabilities.
+These include <code>COALESCE</code>, <code>IFNULL</code>, and <code>NVL2</code>.
+You can consider each of these functions as the equivalent of a macro for <code>CASE</code>.
+For example, <code>COALESCE(expr1, expr2, expr3)</code> expands to:</p>
+<pre><code class="language-sql">CASE
+ WHEN expr1 IS NOT NULL THEN expr1
+ WHEN expr2 IS NOT NULL THEN expr2
+ ELSE expr3
+END
+</code></pre>
+<p>Since <a href="https://datafusion.apache.org/">Apache DataFusion</a> rewrites these conditional functions to their equivalent <code>CASE</code> expression, any optimizations related to <code>CASE</code> described in this post also apply to conditional function evaluation.</p>
+<h2 id="case-evaluation-in-datafusion-5000"><code>CASE</code> Evaluation in DataFusion 50.0.0<a class="headerlink" href="#case-evaluation-in-datafusion-5000" title="Permanent link">¶</a></h2>
+<p>For the remainder of this post, we'll be looking at 'searched CASE' evaluation.
+'Simple CASE' uses a distinct, but very similar implementation.
+The same set of improvements has been applied to both.</p>
+<p>The baseline implementation in DataFusion 50.0.0 evaluated <code>CASE</code> using a common, straightforward approach:</p>
+<ol>
+<li>Start with an output array <code>out</code> with the same length as the input batch, filled with nulls. Additionally, create a bit vector <code>remainder</code> with the same length and each value set to <code>true</code>.</li>
+<li>For each <code>WHEN</code>/<code>THEN</code> branch:<ul>
+<li>Evaluate the <code>WHEN</code> condition for the remaining unmatched rows using <a href="https://docs.rs/datafusion/latest/datafusion/physical_expr/trait.PhysicalExpr.html#method.evaluate_selection"><code>PhysicalExpr::evaluate_selection</code></a>, passing in the input batch and the <code>remainder</code> mask.</li>
+<li>If any rows matched, evaluate the <code>THEN</code> expression for those rows using <code>PhysicalExpr::evaluate_selection</code>.</li>
+<li>Merge the results into the <code>out</code> array using the <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/zip/fn.zip.html"><code>zip</code></a> kernel.</li>
+<li>Update the <code>remainder</code> mask to exclude the matched rows.</li>
+</ul>
+</li>
+<li>If there's an <code>ELSE</code> clause, evaluate it for any remaining unmatched rows and merge using <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/zip/fn.zip.html"><code>zip</code></a>.</li>
+</ol>
+<p>Here's a simplified version of the Rust code for the original loop:</p>
+<pre><code class="language-rust">let mut out = new_null_array(&amp;return_type, batch.num_rows());
+let mut remainder = BooleanArray::from(vec![true; batch.num_rows()]);
+
+for (when_expr, then_expr) in &amp;self.when_then_expr {
+ // Determine for which remaining rows the WHEN condition matches
+ let when = when_expr.evaluate_selection(batch, &amp;remainder)?
+ .into_array(batch.num_rows())?;
+ // Ensure any `NULL` values are treated as false
+ let when_and_rem = and(&amp;when, &amp;remainder)?;
+
+ if when_and_rem.true_count() == 0 {
+ continue;
+ }
+
+ // Evaluate the THEN expression for matching rows
+ let then = then_expr.evaluate_selection(batch, &amp;when_and_rem)?;
+ // Merge results into output array
+ out = zip(&amp;when_and_rem, &amp;then_value, &amp;out)?;
+ // Update remainder mask to exclude matched rows
+ remainder = and_not(&amp;remainder, &amp;when_and_rem)?;
+}
+</code></pre>
+<p>Let's examine one iteration of this loop for the following <code>CASE</code> expression:</p>
+<pre><code class="language-sql">CASE
+ WHEN col = 'b' THEN 100
+ ELSE 200
+END
+</code></pre>
+<p>Schematically, it will look as follows:</p>
+<figure>
+<img alt="Schematic representation of data flow in the original CASE implementation" class="img-responsive" src="/blog/images/case/original_loop.svg" width="100%"/>
+<figcaption>One iteration of the `CASE` evaluation loop</figcaption>
+</figure>
+<p>This implementation works perfectly fine, but there's significant room for optimization, mostly related to the usage of <code>evaluate_selection</code>.
+To understand why, we need to dig a little deeper into the implementation of that function.
+Here's a simplified version of it that captures the relevant parts:</p>
+<pre><code class="language-rust">pub trait PhysicalExpr {
+ fn evaluate_selection(
+ &amp;self,
+ batch: &amp;RecordBatch,
+ selection: &amp;BooleanArray,
+ ) -&gt; Result&lt;ColumnarValue&gt; {
+ // Reduce record batch to only include rows that match selection
+ let filtered_batch = filter_record_batch(batch, selection)?;
+ // Perform regular evaluation on filtered batch
+ let filtered_result = self.evaluate(&amp;filtered_batch)?;
+ // Expand result array to match original batch length
+ scatter(selection, filtered_result)
+ }
+}
+</code></pre>
+<p>Going back to the same example as before, the data flow in <code>evaluate_selection</code> looks like this:</p>
+<figure>
+<img alt="Schematic representation of `evaluate_selection` evaluation" class="img-responsive" src="/blog/images/case/evaluate_selection.svg" width="100%"/>
+<figcaption>evaluate_selection data flow</figcaption>
+</figure>
+<p>The <code>evaluate_selection</code> method first filters the input batch to only include rows that match the <code>selection</code> mask.
+It then calls the regular <code>evaluate</code> method using the filtered batch as input.
+Finally, to return a result array with the same number of rows as <code>batch</code>, the <code>scatter</code> function is called.
+This function produces a new array padded with <code>null</code> values for any rows that didn't match the <code>selection</code> mask.</p>
+<p>So how can we improve the performance of the simple evaluation strategy and use of <code>evaluate_selection</code>?</p>
+<h3 id="opportunity-1-early-exit">Opportunity 1: Early Exit<a class="headerlink" href="#opportunity-1-early-exit" title="Permanent link">¶</a></h3>
+<p>The <code>CASE</code> evaluation loop always iterates through all branches, even when every row has already been matched.
+In queries where early branches match all rows, this results in unnecessary work being done for the remaining branches.</p>
+<h3 id="opportunity-2-optimize-repeated-filtering-scattering-and-merging">Opportunity 2: Optimize Repeated Filtering, Scattering, and Merging<a class="headerlink" href="#opportunity-2-optimize-repeated-filtering-scattering-and-merging" title="Permanent link">¶</a></h3>
+<p>Each iteration performs a number of operations that are very well-optimized, but still take up a significant amount of CPU time:</p>
+<ul>
+<li><strong>Filtering</strong>: <code>PhysicalExpr::evaluate_selection</code> filters the entire <code>RecordBatch</code> for each branch. For the <code>WHEN</code> expression, this is done even if the selection mask was entirely empty.</li>
+<li><strong>Scattering</strong>: <code>PhysicalExpr::evaluate_selection</code> scatters the filtered result back to the original <code>RecordBatch</code> length.</li>
+<li><strong>Merging</strong>: The <code>zip</code> kernel is called once per branch to merge partial results into the output array</li>
+</ul>
+<p>Each of these operations needs to allocate memory for new arrays and shuffle quite a bit of data around.</p>
+<h3 id="opportunity-3-filter-only-necessary-columns">Opportunity 3: Filter only Necessary Columns<a class="headerlink" href="#opportunity-3-filter-only-necessary-columns" title="Permanent link">¶</a></h3>
+<p>The <code>PhysicalExpr::evaluate_selection</code> method filters the entire record batch, including columns that the current branch's <code>WHEN</code> and <code>THEN</code> expressions don't reference.
+For wide tables (many columns) with narrow expressions (few column references), this is wasteful.</p>
+<p>Suppose you have a table with 26 columns named <code>a</code> through <code>z</code>, and the following simple <code>CASE</code> expression:</p>
+<pre><code class="language-sql">CASE
+ WHEN a &gt; 1000 THEN 'large'
+ WHEN a &gt;= 0 THEN 'positive'
+ ELSE 'negative'
+END
+</code></pre>
+<p>The implementation would filter all 26 columns even though only a single column is needed for the entire <code>CASE</code> expression evaluation.
+Again this involves a non-negligible amount of allocation and data copying.</p>
+<h2 id="performance-optimizations">Performance Optimizations<a class="headerlink" href="#performance-optimizations" title="Permanent link">¶</a></h2>
+<h3 id="optimization-1-short-circuit-early-exit">Optimization 1: Short-Circuit Early Exit<a class="headerlink" href="#optimization-1-short-circuit-early-exit" title="Permanent link">¶</a></h3>
+<p>The first optimization is straightforward.
+As soon as we detect that all rows of the batch have been matched, we break out of the evaluation loop:</p>
+<pre><code class="language-rust">let mut remainder_count = batch.num_rows();
+
+for (when_expr, then_expr) in &amp;self.when_then_expr {
+ if remainder_count == 0 {
+ break; // All rows matched, exit early
+ }
+
+ // ... evaluate branch ...
+
+ let when_match_count = when_value.true_count();
+ remainder_count -= when_match_count;
+}
+</code></pre>
+<p>Additionally, we avoid evaluating the <code>ELSE</code> clause when no rows remain:</p>
+<pre><code class="language-rust">if let Some(else_expr) = &amp;self.else_expr {
+ remainder = or(&amp;base_nulls, &amp;remainder)?;
+ if remainder.true_count() &gt; 0 {
+ // ... evaluate else ...
+ }
+}
+</code></pre>
+<p>For queries where early branches match all rows, this eliminates unnecessary branch evaluations and <code>ELSE</code> clause processing.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/17898">PR #17898</a></p>
+<h3 id="optimization-2-optimized-result-merging">Optimization 2: Optimized Result Merging<a class="headerlink" href="#optimization-2-optimized-result-merging" title="Permanent link">¶</a></h3>
+<p>The second optimization fundamentally restructures how the results of each loop iteration will be merged.
+The diagram below illustrates the optimized data flow when evaluating the <code>CASE WHEN col = 'b' THEN 100 ELSE 200 END</code> from before:</p>
+<figure>
+<img alt="Schematic representation of optimized evaluation loop" class="img-responsive" src="/blog/images/case/merging.svg" width="100%"/>
+<figcaption>optimized evaluation loop</figcaption>
+</figure>
+<p>In the reworked implementation, the <code>evaluate_selection</code> function is no longer used.
+The key insight is that we can defer all merging until the end of the evaluation loop by tracking result provenance.
+This was implemented with the following changes:</p>
+<ol>
+<li>Augment the input batch with a column containing row indices.</li>
+<li>Reduce the augmented batch after each loop iteration to only contain the remaining rows.</li>
+<li>Use the row index column to track which partial result array contains the value for each row.</li>
+<li>Perform a single merge operation at the end instead of a <code>zip</code> operation after each loop iteration.</li>
+</ol>
+<p>These changes make it unnecessary to <code>scatter</code> and <code>zip</code> results in each loop iteration.
+Instead, when all rows have been matched, we then merge the partial results using <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge_n.html"><code>arrow_select::merge::merge_n</code></a>.</p>
+<p>The diagram below illustrates how <code>merge_n</code> works for an example where three <code>WHEN/THEN</code> branches produced results.
+The first branch produced the result <code>A</code> for row 2, the second produced <code>B</code> for row 1, and the third produced <code>C</code> and <code>D</code> for rows 4 and 5.</p>
+<figure>
+<img alt="Schematic illustration of the merge_n algorithm" class="img-responsive" src="/blog/images/case/merge_n.svg" width="100%"/>
+<figcaption>merge_n example</figcaption>
+</figure>
+<p>The <code>merge_n</code> algorithm scans through the indices array.
+For each non-empty cell, it takes one value from the corresponding values array.
+In the example above, we first encounter <code>1</code>.
+This takes the first element from the values array with index <code>1</code>, resulting in <code>B</code>.
+The next cell contains <code>0</code> which takes <code>A</code>, from the first array.
+Finally, we encounter <code>2</code> twice.
+This takes the first and second element from the last values array respectively.</p>
+<p>This algorithm was initially implemented in DataFusion for the <code>CASE</code> implementation, but in the meantime has been generalized and moved into the <code>arrow-rs</code> crate as <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge_n.html"><code>arrow_select::merge::merge_n</code></a>.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18152">PR #18152</a></p>
+<h3 id="optimization-3-column-projection">Optimization 3: Column Projection<a class="headerlink" href="#optimization-3-column-projection" title="Permanent link">¶</a></h3>
+<p>The third optimization addresses the "filtering unused columns" overhead through projection.</p>
+<p>Look at the following query example where the <code>mailing_address</code> table has the columns <code>name</code>, <code>surname</code>, <code>street</code>, <code>number</code>, <code>city</code>, <code>state</code>, <code>country</code>:</p>
+<pre><code class="language-sql">SELECT *, CASE WHEN country = 'USA' THEN state ELSE country END AS region
+FROM mailing_address
+</code></pre>
+<p>You can see that the <code>CASE</code> expression only references the columns <code>country</code> and <code>state</code>, but because all columns are being queried, projection pushdown cannot reduce the number of columns being fed in to the projection operator.</p>
+<figure>
+<img alt="Schematic illustration of CASE evaluation without projection" class="img-responsive" src="/blog/images/case/no_projection.svg" width="100%"/>
+<figcaption>CASE evaluation without projection</figcaption>
+</figure>
+<p>During <code>CASE</code> evaluation, the batch must be filtered using the <code>WHEN</code> expression to evaluate the <code>THEN</code> expression values.
+As the diagram above shows, this filtering creates a reduced copy of all columns.</p>
+<p>This unnecessary copying can be avoided by first narrowing the batch to only include the columns that are actually needed.</p>
+<figure>
+<img alt="Schematic illustration of CASE evaluation with projection" class="img-responsive" src="/blog/images/case/projection.svg" width="100%"/>
+<figcaption>CASE evaluation with projection</figcaption>
+</figure>
+<p>At first glance, this might not seem beneficial, since we're introducing an additional processing step.
+Luckily projection of a record batch only requires a shallow copy of the record batch.
+The column arrays themselves are not copied, and the only work that is actually done is incrementing the reference counts of the columns.</p>
+<p><strong>Impact</strong>: For wide tables with narrow CASE expressions, this dramatically reduces filtering overhead by removing the copying of unused columns.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18329">PR #18329</a></p>
+<h3 id="optimization-4-eliminating-scatter-in-two-branch-case">Optimization 4: Eliminating Scatter in Two-Branch Case<a class="headerlink" href="#optimization-4-eliminating-scatter-in-two-branch-case" title="Permanent link">¶</a></h3>
+<p>Some of the earlier examples in this post use expressions of the form <code>CASE WHEN condition THEN expr1 ELSE expr2 END</code> to explain how the general evaluation loop works.
+For this kind of two-branch <code>CASE</code> expression, <a href="https://datafusion.apache.org/">Apache DataFusion</a> has a more optimized implementation that unrolls the loop.
+This specialized <code>ExpressionOrExpression</code> fast path still used <code>evaluate_selection()</code> for both branches which uses <code>scatter</code> and <code>zip</code> to combine the results incurring the same performance overhead as the general implementation.</p>
+<p>The revised implementation eliminates the use of <code>evaluate_selection</code> as follows:</p>
+<pre><code class="language-rust">// Compute the `WHEN` condition for the entire batch
+let when_filter = create_filter(&amp;when_value);
+
+// Compute a compact array of `THEN` values for the matching rows
+let then_batch = filter_record_batch(batch, &amp;when_filter)?;
+let then_value = then_expr.evaluate(&amp;then_batch)?;
+
+// Compute a compact array of `ELSE` values for the non-matching rows
+let else_filter = create_filter(&amp;not(&amp;when_value)?);
+let else_batch = filter_record_batch(batch, &amp;else_filter)?;
+let else_value = else_expr.evaluate(&amp;else_batch)?;
+</code></pre>
+<p>This produces two compact arrays, one for the THEN values and one for the ELSE values, which are then merged with the <code>merge</code> function.
+In contrast to <code>zip</code>, <code>merge</code> does not require both of its value inputs to have the same length.
+Instead it requires that the sum of the length of the value inputs matches the length of the mask array.</p>
+<figure>
+<img alt="Schematic illustration of the merge algorithm" class="img-responsive" src="/blog/images/case/merge.svg" width="100%"/>
+<figcaption>merge example</figcaption>
+</figure>
+<p>This eliminates unnecessary <code>scatter</code> operations and memory allocations for one of the most common <code>CASE</code> expression patterns.</p>
+<p>Just like <code>merge_n</code>, this operation has been moved into <code>arrow-rs</code> as <a href="https://docs.rs/arrow-select/57.1.0/arrow_select/merge/fn.merge.html"><code>arrow_select::merge::merge</code></a>.</p>
+<p>This optimization was implemented by Pepijn Van Eeckhoudt (<a href="https://github.com/pepijnve"><code>@pepijnve</code></a>) in <a href="https://github.com/apache/datafusion/pull/18444">PR #18444</a></p>
+<h3 id="optimization-5-table-lookup-of-constants">Optimization 5: Table Lookup of Constants<a class="headerlink" href="#optimization-5-table-lookup-of-constants" title="Permanent link">¶</a></h3>
+<p>Up until now, we've discussed the implementations for generic <code>CASE</code> expressions that use non-constant expressions for both <code>WHEN</code> and <code>THEN</code>.
+Another common use of <code>CASE</code> is to perform a mapping from one set of constants to another.
+For instance, you can expand numeric constants to human-readable strings using the following <code>CASE</code> example.</p>
+<pre><code class="language-sql">CASE status
+ WHEN 0 THEN 'idle'
+ WHEN 1 THEN 'running'
+ WHEN 2 THEN 'paused'
+ WHEN 3 THEN 'stopped'
+ ELSE 'unknown'
+END
+</code></pre>
+<p>A final <code>CASE</code> optimization recognizes this pattern and compiles the <code>CASE</code> expression into a hash table.
+Rather than evaluating the <code>WHEN</code> and <code>THEN</code> expressions, the input expression is evaluated once, and the result array is computed using a vectorized hash table lookup.
+This approach avoids the need to filter the input batch and combine partial results entirely.
+The result array is computed in a single pass over the input values, and the computation time does not grow significantly with the number of <code>WHEN</code> branches in the <code>CASE</code> expression.</p>
+<p>This optimization was implemented by Raz Luvaton (<a href="https://github.com/rluvaton"><code>@rluvaton</code></a>) in <a href="https://github.com/apache/datafusion/pull/18183">PR #18183</a></p>
+<h2 id="results">Results<a class="headerlink" href="#results" title="Permanent link">¶</a></h2>
+<p>The degree to which the performance optimizations described in this post will benefit your queries is highly dependent on both your data and your queries.
+To give some idea of the impact, we ran the following query on the TPC_H <code>orders</code> table with a scale factor of 100:</p>
+<pre><code class="language-sql">SELECT
+ *,
+ case o_orderstatus
+ when 'O' then 'ordered'
+ when 'F' then 'filled'
+ when 'P' then 'pending'
+ else 'other'
+ end
+from orders
+</code></pre>
+<p>This query was first run with DataFusion 50.0.0 to get a baseline measurement.
+The same query was then run with each optimization applied in turn.
+The recorded times are presented as the blue series in the chart below.
+The green series shows the time measurement for the <code>SELECT * FROM orders</code> to give an idea of the cost the addition of a <code>CASE</code> expression in a query incurs.
+All measurements were made with a target partition count of <code>1</code>.</p>
+<figure>
+<img alt="Performance measurements chart" class="img-responsive" src="/blog/images/case/results.png" width="100%"/>
+<figcaption>Performance measurements</figcaption>
+</figure>
+<p>What you can see in the chart is that the effect of the various optimizations compounds up to the <code>project</code> measurement.
+Up to that point these results are applicable to any <code>CASE</code> expression.
+The final improvement in the <code>hash</code> measurement is only applicable to simple <code>CASE</code> expressions with constant <code>WHEN</code> and <code>THEN</code> expressions.</p>
+<p>The cumulative effect of these optimizations is a 63-71% reduction in CPU time spent evaluating <code>CASE</code> expressions compared to the baseline.</p>
+<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h2>
+<p>Through a number of targeted optimizations, we've transformed <code>CASE</code> expression evaluation from a simple, but unoptimized implementation into a highly optimized one.
+The optimizations described in this post compound: a <code>CASE</code> expression on a wide table with multiple branches and early matches benefits from all four optimizations simultaneously.
+The result is significantly reduced CPU time and memory allocation in SQL constructs that are essential for ETL-like queries.</p>
+<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries,
+and machine learning and streaming applications.
+While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion’s primary design goal</a> is to accelerate the creation of other data-centric systems, it provides a reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe library</a>, <a href="https://datafusion.apache.org/python/">Python library</a>, and <a href="https://datafusion.apache.org/u [...]
+<p>DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone.
+Without DataFusion, highly performant vectorized query engines would remain the domain of a few large companies and world-class research institutions.
+With DataFusion, we can all build on top of a shared foundation and focus on what makes our projects unique.</p>
+<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
+<p>DataFusion is not a project built or driven by a single person, company, or foundation.
+Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code.
+A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</titl [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/pepijn-van-eeckhoudt.rss.xml b/blog/feeds/pepijn-van-eeckhoudt.rss.xml
index 91f38b2..75a3953 100644
--- a/blog/feeds/pepijn-van-eeckhoudt.rss.xml
+++ b/blog/feeds/pepijn-van-eeckhoudt.rss.xml
@@ -1,5 +1,45 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - Pepijn Van Eeckhoudt</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon, 30 Jun 2025 00:00:00 +0000</lastBuildDate><item><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</title><link>https://datafusion.apache.org/blog/2025/06/30/cancellation</link><description><!--
+<rss version="2.0"><channel><title>Apache DataFusion Blog - Pepijn Van Eeckhoudt</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon, 02 Feb 2026 00:00:00 +0000</lastBuildDate><item><title>Optimizing SQL CASE Expression Evaluation</title><link>https://datafusion.apache.org/blog/2026/02/02/datafusion_case</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style>
+figure {
+ margin: 20px 0;
+}
+
+figure img {
+ display: block;
+ max-width: 80%;
+ margin: auto;
+}
+
+figcaption {
+ font-style: italic;
+ color: #555;
+ font-size: 0.9em;
+ max-width: 80%;
+ margin: auto;
+ text-align: center;
+}
+</style>
+<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
+It allows you to control which expression from a …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Pepijn Van Eeckhoudt</dc:creator><pubDate>Mon, 02 Feb 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-02-02:/blog/2026/02/02/datafusion_case</guid><category>blog</category></item><item><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</title><link>https://datafusion.apache.org/blog/2025/06/30/ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/pmc.atom.xml b/blog/feeds/pmc.atom.xml
index 3f50ce1..470aa0e 100644
--- a/blog/feeds/pmc.atom.xml
+++ b/blog/feeds/pmc.atom.xml
@@ -1,5 +1,121 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - pmc</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-01-08T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/08/datafusion-52.0.0" rel="alte [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - pmc</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-01-30T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion Comet 0.13.0 Release</title><link href="https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0 [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development work and is the result of merging 169 PRs from 15
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.13.0.md">change log</a> for more information.</p>
+<h2 id="key-features">Key Features<a class="headerlink" href="#key-features" title="Permanent link">¶</a></h2>
+<h3 id="native-parquet-write-support-experimental">Native Parquet Write Support (Experimental)<a class="headerlink" href="#native-parquet-write-support-experimental" title="Permanent link">¶</a></h3>
+<p>This release introduces experimental native Parquet write capabilities, allowing Comet to intercept and execute Parquet write operations natively through DataFusion. Key capabilities include:</p>
+<ul>
+<li>File commit protocol support for reliable writes</li>
+<li>Remote HDFS writing via OpenDAL integration</li>
+<li>Complex type support (arrays, maps, structs)</li>
+<li>Proper handling of object store settings</li>
+</ul>
+<p>To enable native Parquet writes, set:</p>
+<pre><code>spark.comet.allowIncompatibleOp.DataWritingCommandExec=true
+spark.comet.parquet.write.enabled=true
+</code></pre>
+<p><strong>Note</strong>: This feature is highly experimental and should not be used in production environments. It is currently categorized as a testing feature and is disabled by default.</p>
+<h3 id="native-iceberg-improvements">Native Iceberg Improvements<a class="headerlink" href="#native-iceberg-improvements" title="Permanent link">¶</a></h3>
+<p>Comet's fully-native Iceberg integration received significant enhancements in this release:</p>
+<p><strong>REST Catalog Support</strong>: Native Iceberg scans now support REST catalogs, enabling integration with catalog services like Apache Polaris and Tabular. Configure with:</p>
+<pre><code class="language-shell">--conf spark.sql.catalog.rest_cat=org.apache.iceberg.spark.SparkCatalog
+--conf spark.sql.catalog.rest_cat.catalog-impl=org.apache.iceberg.rest.RESTCatalog
+--conf spark.sql.catalog.rest_cat.uri=http://localhost:8181
+--conf spark.comet.scan.icebergNative.enabled=true
+</code></pre>
+<p><strong>Session Token Authentication</strong>: Added support for session tokens in native Iceberg scans for secure S3 access.</p>
+<p><strong>Performance Optimizations</strong>:</p>
+<ul>
+<li>Deduplicated serialized metadata reducing memory overhead</li>
+<li>Switched from JSON to protobuf for partition value serialization</li>
+<li>Removed IcebergFileStream in favor of iceberg-rust's built-in parallelization</li>
+<li>Reduced metadata serialization points</li>
+<li>Added SchemaAdapter caching</li>
+</ul>
+<p>To enable fully-native Iceberg scanning:</p>
+<pre><code>spark.comet.scan.icebergNative.enabled=true
+</code></pre>
+<p>The native reader supports Iceberg table spec v1 and v2, all primitive and complex types, schema evolution, time travel, positional and equality deletes, filter pushdown, and various storage backends (local, HDFS, S3).</p>
+<h3 id="native-csv-reading-experimental">Native CSV Reading (Experimental)<a class="headerlink" href="#native-csv-reading-experimental" title="Permanent link">¶</a></h3>
+<p>Experimental support for native CSV file reading has been added, expanding Comet's file format capabilities beyond Parquet.</p>
+<h3 id="new-expressions">New Expressions<a class="headerlink" href="#new-expressions" title="Permanent link">¶</a></h3>
+<p>The release adds support for numerous expressions:</p>
+<ul>
+<li>Array functions: <code>explode</code>, <code>explode_outer</code>, <code>size</code></li>
+<li>Date/time functions: <code>unix_date</code>, <code>date_format</code>, <code>datediff</code>, <code>last_day</code>, <code>unix_timestamp</code></li>
+<li>String functions: <code>left</code></li>
+<li>JSON functions: <code>from_json</code> (partial support)</li>
+</ul>
+<h3 id="ansi-mode-support">ANSI Mode Support<a class="headerlink" href="#ansi-mode-support" title="Permanent link">¶</a></h3>
+<p>Sum and average aggregate expressions now support ANSI mode for both integer and decimal inputs, enabling overflow checking in strict SQL mode.</p>
+<h3 id="native-shuffle-improvements">Native Shuffle Improvements<a class="headerlink" href="#native-shuffle-improvements" title="Permanent link">¶</a></h3>
+<ul>
+<li>Round-robin partitioning is now supported in native shuffle</li>
+<li>Spill metrics are now reported correctly</li>
+<li>Configurable shuffle writer buffer size via <code>spark.comet.shuffle.write.bufferSize</code></li>
+</ul>
+<h2 id="performance-improvements">Performance Improvements<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>This release includes extensive performance optimizations:</p>
+<ul>
+<li><strong>String to integer casting</strong>: Significant speedups through optimized parsing</li>
+<li><strong>String functions</strong>: Optimized <code>lpad</code>/<code>rpad</code> to remove unnecessary memory allocations</li>
+<li><strong>Date operations</strong>: Improved <code>normalize_nan</code> and date truncate performance</li>
+<li><strong>Query planning</strong>: Cached query plans to avoid per-partition serialization overhead</li>
+<li><strong>Memory efficiency</strong>: Reduced GC pressure in protobuf serialization</li>
+<li><strong>Hash operations</strong>: Optimized complex-type hash implementations including murmur3 support for nested types</li>
+<li><strong>Runtime efficiency</strong>: Eliminated busy-polling of Tokio stream for plans without CometScan</li>
+<li><strong>Metrics overhead</strong>: Reduced timer and syscall overhead in native shuffle writer</li>
+</ul>
+<h2 id="deprecations">Deprecations<a class="headerlink" href="#deprecations" title="Permanent link">¶</a></h2>
+<p>The <code>native_comet</code> scan mode is now deprecated in favor of <code>native_iceberg_compat</code> and will be removed in a future release. The <code>auto</code> scan mode no longer falls back to <code>native_comet</code>.</p>
+<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
+<p>This release upgrades to DataFusion 51, Arrow 57, and the latest iceberg-rust. The minimum supported Rust version is now 1.88.</p>
+<p>Supported platforms include Spark 3.4.3, 3.5.4-3.5.7, and Spark 4.0.x with various JDK and Scala combinations.</p>
+<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="alternate"></link><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusio [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -3724,7 +3840,7 @@ LIMIT 3;
3 rows in set. Query took 0.053 seconds.
</code></pre>
<h3 id="growth-of-datafusion">Growth of DataFusion 📈<a class="headerlink" href="#growth-of-datafusion" title="Permanent link">¶</a></h3>
-<p>DataFusion has been appearing more publically in the wild. For example
+<p>DataFusion has been appearing more publicly in the wild. For example
* New projects built using DataFusion such as <a href="https://lancedb.com/">lancedb</a>, <a href="https://glaredb.com/">GlareDB</a>, <a href="https://www.arroyo.dev/">Arroyo</a>, and <a href="https://github.com/cmu-db/optd">optd</a>.
* Public talks such as <a href="https://www.youtube.com/watch?v=AJU9rdRNk9I">Apache Arrow Datafusion: Vectorized
Execution Framework For Maximum Performance</a> in <a href="https://www.bagevent.com/event/8432178">CommunityOverCode Asia 2023</a>
@@ -4153,7 +4269,7 @@ required synchronous access to all relevant catalog information.</p>
<li>Automatic coercions ast between Date and Timestamp <a href="https://github.com/apache/arrow-datafusion/issues/4726">#4726</a></li>
<li>Support type coercion for timestamp and utf8 <a href="https://github.com/apache/arrow-datafusion/issues/4312">#4312</a></li>
<li>Full support for time32 and time64 literal values (<code>ScalarValue</code>) <a href="https://github.com/apache/arrow-datafusion/issues/4156">#4156</a></li>
-<li>New functions, incuding <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
+<li>New functions, including <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
<li>Compressed CSV/JSON support <a href="https://github.com/apache/arrow-datafusion/issues/3642">#3642</a></li>
</ul>
<p>The community has also invested in new <a href="https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/tests/sqllogictests/README.md">sqllogic based</a> tests to keep improving DataFusion's quality with less effort.</p>
@@ -5005,7 +5121,7 @@ git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc
<li>Switch from <code>std::sync::Mutex</code> to <code>parking_lot::Mutex</code> <a href="https://github.com/apache/arrow-datafusion/pull/1720">#1720</a></li>
<li>New Features</li>
<li>Support for memory tracking and spilling to disk<ul>
-<li>MemoryMananger and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
+<li>MemoryManager and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
<li>Out of core sort <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
<li>New metrics</li>
<li><code>Gauge</code> and <code>CurrentMemoryUsage</code> <a href="https://github.com/apache/arrow-datafusion/pull/1682">#1682</a></li>
diff --git a/blog/feeds/pmc.rss.xml b/blog/feeds/pmc.rss.xml
index 4b9925d..20bbffa 100644
--- a/blog/feeds/pmc.rss.xml
+++ b/blog/feeds/pmc.rss.xml
@@ -1,5 +1,27 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - pmc</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Thu, 08 Jan 2026 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion 52.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/01/08/datafusion-52.0.0</link><description><!--
+<rss version="2.0"><channel><title>Apache DataFusion Blog - pmc</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Fri, 30 Jan 2026 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet 0.13.0 Release</title><link>https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately eight weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 30 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-30:/blog/2026/01/30/datafusion-comet-0.13.0</guid><category>blog</category></item><item><title>Apache DataFusion 52.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0</link><de [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -23,7 +45,7 @@ some of the major improvements since <a href="https://datafusion.apache.org/b
changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
making this release possible.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 08 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-08:/blog/2026/01/08/datafusion-52.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.12.0 Release</title><link>https://datafusion.apache.org/blog/2025/12/04/datafusion-comet-0.12.0</link><description><!--
+<p>We continue to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/datafusion-52.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.12.0 Release</title><link>https://datafusion.apache.org/blog/2025/12/04/datafusion-comet-0.12.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/tim-saucer.atom.xml b/blog/feeds/tim-saucer.atom.xml
new file mode 100644
index 0000000..81bd8f6
--- /dev/null
+++ b/blog/feeds/tim-saucer.atom.xml
@@ -0,0 +1,597 @@
+<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Tim Saucer</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/tim-saucer.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-03-20T00:00:00+00:00</updated><subtitle></subtitle><entry><title>"Writing Custom Table Providers in Apache DataFusion"</title><link href="https://datafusion.apache.org/blog/2026/ [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
+in a custom format, behind an API, or in a system that DataFusion does not
+natively support, you can teach DataFusion to read it by implementing a
+<strong>custom table provider</strong>. This post walks through the three layers you need to
+understand and explains where your work should actually happen.</p>
+<h2 id="the-three-layers">The Three Layers<a class="headerlink" href="#the-three-layers" title="Permanent link">¶</a></h2>
+<hr/>
+<p>When DataFusion executes a query against a table, three abstractions collaborate
+to produce results:</p>
+<ol>
+<li><strong>[<code>TableProvider</code>]</strong> -- Describes the table (schema, capabilities) and
+ produces an execution plan when queried.</li>
+<li><strong>[<code>ExecutionPlan</code>]</strong> -- Describes <em>how</em> to compute the result: partitioning,
+ ordering, and child plan relationships.</li>
+<li><strong>[<code>SendableRecordBatchStream</code>]</strong> -- The async stream that <em>actually does the
+ work</em>, yielding <code>RecordBatch</code>es one at a time.</li>
+</ol>
+<p>Think of these as a funnel: <code>TableProvider::scan()</code> is called once during
+planning to create an <code>ExecutionPlan</code>, then <code>ExecutionPlan::execute()</code> is called
+once per partition to create a stream, and those streams are where rows are
+actually produced during execution.</p>
+<h2 id="layer-1-tableprovider">Layer 1: TableProvider<a class="headerlink" href="#layer-1-tableprovider" title="Permanent link">¶</a></h2>
+<hr/>
+<p>A [<code>TableProvider</code>] represents a queryable data source. For a minimal read-only
+table, you need four methods:</p>
+<pre><code class="language-rust">impl TableProvider for MyTable {
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+
+ fn schema(&amp;self) -&gt; SchemaRef {
+ Arc::clone(&amp;self.schema)
+ }
+
+ fn table_type(&amp;self) -&gt; TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &amp;self,
+ state: &amp;dyn Session,
+ projection: Option&lt;&amp;Vec&lt;usize&gt;&gt;,
+ filters: &amp;[Expr],
+ limit: Option&lt;usize&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ // Build and return an ExecutionPlan -- keep this lightweight!
+ Ok(Arc::new(MyExecPlan::new(
+ Arc::clone(&amp;self.schema),
+ projection,
+ limit,
+ )))
+ }
+}
+</code></pre>
+<p>The <code>scan</code> method is the heart of <code>TableProvider</code>. It receives three pushdown
+hints from the optimizer, each reducing the amount of data your source needs
+to produce:</p>
+<ul>
+<li><strong><code>projection</code></strong> -- Which columns are needed. This reduces the <strong>width</strong> of
+ the output. If your source supports it, read only these columns rather than
+ the full schema.</li>
+<li><strong><code>filters</code></strong> -- Predicates the engine would like you to apply during the
+ scan. This reduces the <strong>number of rows</strong> by skipping data that does not
+ match. Implement <code>supports_filters_pushdown</code> to advertise which filters you
+ can handle.</li>
+<li><strong><code>limit</code></strong> -- A row count cap. This also reduces the <strong>number of rows</strong> --
+ if you can stop reading early once you have produced enough rows, this avoids
+ unnecessary work.</li>
+</ul>
+<h3 id="keep-scan-lightweight">Keep <code>scan()</code> Lightweight<a class="headerlink" href="#keep-scan-lightweight" title="Permanent link">¶</a></h3>
+<p>This is a critical point: <strong><code>scan()</code> runs during planning, not execution.</strong> It
+should return quickly. Best practices are to avoid performing I/O, network
+calls, or heavy computation here. The <code>scan</code> method's job is to <em>describe</em> how
+the data will be produced, not to produce it. All the real work belongs in the
+stream (Layer 3).</p>
+<p>A common pitfall is to fetch data or open connections in <code>scan()</code>. This blocks
+the planning thread and can cause timeouts or deadlocks, especially if the query
+involves multiple tables or subqueries that all need to be planned before
+execution begins.</p>
+<h3 id="existing-implementations-to-learn-from">Existing Implementations to Learn From<a class="headerlink" href="#existing-implementations-to-learn-from" title="Permanent link">¶</a></h3>
+<p>DataFusion ships several <code>TableProvider</code> implementations that are excellent
+references:</p>
+<ul>
+<li><strong>[<code>MemTable</code>]</strong> -- Holds data in memory as <code>Vec&lt;RecordBatch&gt;</code>. The simplest
+ possible provider; great for tests and small datasets.</li>
+<li><strong>[<code>StreamTable</code>]</strong> -- Wraps a user-provided stream factory. Useful when your
+ data arrives as a continuous stream (e.g., from Kafka or a socket).</li>
+<li><strong>[<code>SortedTableProvider</code>]</strong> -- Wraps another <code>TableProvider</code> and advertises a
+ known sort order, enabling the optimizer to skip redundant sorts.</li>
+</ul>
+<h2 id="layer-2-executionplan">Layer 2: ExecutionPlan<a class="headerlink" href="#layer-2-executionplan" title="Permanent link">¶</a></h2>
+<hr/>
+<p>An [<code>ExecutionPlan</code>] is a node in the physical query plan tree. Your table
+provider's <code>scan()</code> method returns one. The required methods are:</p>
+<pre><code class="language-rust">impl ExecutionPlan for MyExecPlan {
+ fn name(&amp;self) -&gt; &amp;str { "MyExecPlan" }
+
+ fn as_any(&amp;self) -&gt; &amp;dyn Any { self }
+
+ fn properties(&amp;self) -&gt; &amp;PlanProperties {
+ &amp;self.properties
+ }
+
+ fn children(&amp;self) -&gt; Vec&lt;&amp;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ vec![] // Leaf node -- no children
+ }
+
+ fn with_new_children(
+ self: Arc&lt;Self&gt;,
+ children: Vec&lt;Arc&lt;dyn ExecutionPlan&gt;&gt;,
+ ) -&gt; Result&lt;Arc&lt;dyn ExecutionPlan&gt;&gt; {
+ assert!(children.is_empty());
+ Ok(self)
+ }
+
+ fn execute(
+ &amp;self,
+ partition: usize,
+ context: Arc&lt;TaskContext&gt;,
+ ) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ // This is where you build and return your stream
+ // ...
+ }
+}
+</code></pre>
+<p>The key properties to set correctly in [<code>PlanProperties</code>] are <strong>output
+partitioning</strong> and <strong>output ordering</strong>.</p>
+<p><strong>Output partitioning</strong> tells the engine how many partitions your data has,
+which determines parallelism. If your source naturally partitions data (e.g.,
+by file or by shard), expose that here.</p>
+<p><strong>Output ordering</strong> declares whether your data is naturally sorted. This
+enables the optimizer to avoid inserting a <code>SortExec</code> when a query requires
+ordered data. Getting this right can be a significant performance win.</p>
+<h3 id="partitioning-strategies">Partitioning Strategies<a class="headerlink" href="#partitioning-strategies" title="Permanent link">¶</a></h3>
+<p>Since <code>execute()</code> is called once per partition, partitioning directly controls
+the parallelism of your table scan. Each partition runs on its own task, so
+more partitions means more concurrent work -- up to the number of available
+cores.</p>
+<p>Consider how your data source naturally divides its data:</p>
+<ul>
+<li><strong>By file or object:</strong> If you are reading from S3, each file can be a
+ partition. DataFusion will read them in parallel.</li>
+<li><strong>By shard or region:</strong> If your source is a sharded database, each shard
+ maps naturally to a partition.</li>
+<li><strong>By key range:</strong> If your data is keyed (e.g., by timestamp or customer ID),
+ you can split it into ranges.</li>
+</ul>
+<p>Getting partitioning right matters because it affects everything downstream in
+the plan. When DataFusion needs to perform an aggregation or join, it
+repartitions data by hashing the relevant columns. If your source already
+produces data partitioned by the join or group-by key, DataFusion can skip the
+repartition step entirely -- avoiding a potentially expensive shuffle.</p>
+<p>For example, if you are building a table provider for a system that stores
+data partitioned by <code>customer_id</code>, and a common query groups by <code>customer_id</code>:</p>
+<pre><code class="language-sql">SELECT customer_id, SUM(amount)
+FROM my_table
+GROUP BY customer_id;
+</code></pre>
+<p>If you declare your output partitioning as <code>Hash([customer_id], N)</code>, the
+optimizer recognizes that the data is already distributed correctly for the
+aggregation and eliminates the <code>RepartitionExec</code> that would otherwise appear
+in the plan. You can verify this with <code>EXPLAIN</code> (more on this below).</p>
+<p>Conversely, if you report <code>UnknownPartitioning</code>, DataFusion must assume the
+worst case and will always insert repartitioning operators as needed.</p>
+<h3 id="keep-execute-lightweight-too">Keep <code>execute()</code> Lightweight Too<a class="headerlink" href="#keep-execute-lightweight-too" title="Permanent link">¶</a></h3>
+<p>Like <code>scan()</code>, the <code>execute()</code> method should construct and return a stream
+without doing heavy work. The actual data production happens when the stream
+is polled. Do not block on async operations here -- build the stream and let
+the runtime drive it.</p>
+<h3 id="existing-implementations-to-learn-from_1">Existing Implementations to Learn From<a class="headerlink" href="#existing-implementations-to-learn-from_1" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong>[<code>StreamingTableExec</code>]</strong> -- Executes a streaming table scan. It takes a
+ stream factory (a closure that produces streams) and handles partitioning.
+ Good reference for wrapping external streams.</li>
+<li><strong>[<code>DataSourceExec</code>]</strong> -- The execution plan behind DataFusion's built-in file
+ scanning (Parquet, CSV, JSON). It demonstrates sophisticated partitioning,
+ filter pushdown, and projection pushdown.</li>
+</ul>
+<h2 id="layer-3-sendablerecordbatchstream">Layer 3: SendableRecordBatchStream<a class="headerlink" href="#layer-3-sendablerecordbatchstream" title="Permanent link">¶</a></h2>
+<hr/>
+<p>[<code>SendableRecordBatchStream</code>] is where the real work happens. It is defined as:</p>
+<pre><code class="language-rust">type SendableRecordBatchStream =
+ Pin&lt;Box&lt;dyn RecordBatchStream&lt;Item = Result&lt;RecordBatch&gt;&gt; + Send&gt;&gt;;
+</code></pre>
+<p>This is an async stream of <code>RecordBatch</code>es that can be sent across threads. When
+the DataFusion runtime polls this stream, your code runs: reading files, calling
+APIs, transforming data, etc.</p>
+<h3 id="using-recordbatchstreamadapter">Using RecordBatchStreamAdapter<a class="headerlink" href="#using-recordbatchstreamadapter" title="Permanent link">¶</a></h3>
+<p>The easiest way to create a <code>SendableRecordBatchStream</code> is with
+[<code>RecordBatchStreamAdapter</code>]. It bridges any <code>futures::Stream&lt;Item =
+Result&lt;RecordBatch&gt;&gt;</code> into the <code>SendableRecordBatchStream</code> type:</p>
+<pre><code class="language-rust">use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
+
+fn execute(
+ &amp;self,
+ partition: usize,
+ context: Arc&lt;TaskContext&gt;,
+) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ let schema = self.schema();
+ let config = self.config.clone();
+
+ let stream = futures::stream::once(async move {
+ // ALL the heavy work happens here, inside the stream:
+ // - Open connections
+ // - Read data from external sources
+ // - Transform and batch the results
+ let batches = fetch_data_from_source(&amp;config).await?;
+ Ok(batches)
+ })
+ .flat_map(|result| match result {
+ Ok(batch) =&gt; futures::stream::iter(vec![Ok(batch)]),
+ Err(e) =&gt; futures::stream::iter(vec![Err(e)]),
+ });
+
+ Ok(Box::pin(RecordBatchStreamAdapter::new(schema, stream)))
+}
+</code></pre>
+<h3 id="cpu-intensive-work-use-a-separate-thread-pool">CPU-Intensive Work: Use a Separate Thread Pool<a class="headerlink" href="#cpu-intensive-work-use-a-separate-thread-pool" title="Permanent link">¶</a></h3>
+<p>If your stream performs CPU-intensive work (parsing, decompression, complex
+transformations), avoid blocking the tokio runtime. Instead, offload to a
+dedicated thread pool and send results back through a channel:</p>
+<pre><code class="language-rust">fn execute(
+ &amp;self,
+ partition: usize,
+ context: Arc&lt;TaskContext&gt;,
+) -&gt; Result&lt;SendableRecordBatchStream&gt; {
+ let schema = self.schema();
+ let config = self.config.clone();
+
+ let (tx, rx) = tokio::sync::mpsc::channel(2);
+
+ // Spawn CPU-heavy work on a blocking thread pool
+ tokio::task::spawn_blocking(move || {
+ let batches = generate_data(&amp;config);
+ for batch in batches {
+ if tx.blocking_send(Ok(batch)).is_err() {
+ break; // Receiver dropped, query was cancelled
+ }
+ }
+ });
+
+ let stream = tokio_stream::wrappers::ReceiverStream::new(rx);
+ Ok(Box::pin(RecordBatchStreamAdapter::new(schema, stream)))
+}
+</code></pre>
+<p>This pattern keeps the async runtime responsive while your data generation
+runs on its own threads.</p>
+<h2 id="where-should-the-work-happen">Where Should the Work Happen?<a class="headerlink" href="#where-should-the-work-happen" title="Permanent link">¶</a></h2>
+<hr/>
+<p>This table summarizes what belongs at each layer:</p>
+<table class="table">
+<thead>
+<tr>
+<th>Layer</th>
+<th>Runs During</th>
+<th>Should Do</th>
+<th>Should NOT Do</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>TableProvider::scan()</code></td>
+<td>Planning</td>
+<td>Build an <code>ExecutionPlan</code> with metadata</td>
+<td>I/O, network calls, heavy computation</td>
+</tr>
+<tr>
+<td><code>ExecutionPlan::execute()</code></td>
+<td>Execution (once per partition)</td>
+<td>Construct a stream, set up channels</td>
+<td>Block on async work, read data</td>
+</tr>
+<tr>
+<td><code>RecordBatchStream</code> (polling)</td>
+<td>Execution</td>
+<td>All I/O, computation, data production</td>
+<td>--</td>
+</tr>
+</tbody>
+</table>
+<p>The guiding principle: <strong>push work as late as possible.</strong> Planning should be
+fast so the optimizer can do its job. Execution setup should be fast so all
+partitions can start promptly. The stream is where you spend time producing
+data.</p>
+<h3 id="why-this-matters">Why This Matters<a class="headerlink" href="#why-this-matters" title="Permanent link">¶</a></h3>
+<p>When <code>scan()</code> does heavy work, several problems arise:</p>
+<ol>
+<li><strong>Planning becomes slow.</strong> If a query touches 10 tables and each <code>scan()</code>
+ takes 500ms, planning alone takes 5 seconds before any data flows.</li>
+<li><strong>The optimizer cannot help.</strong> The optimizer runs between planning and
+ execution. If you have already fetched data during planning, optimizations
+ like predicate pushdown or partition pruning cannot reduce the work.</li>
+<li><strong>Resource management breaks down.</strong> DataFusion manages concurrency and
+ memory during execution. Work done during planning bypasses these controls.</li>
+</ol>
+<h2 id="filter-pushdown-doing-less-work">Filter Pushdown: Doing Less Work<a class="headerlink" href="#filter-pushdown-doing-less-work" title="Permanent link">¶</a></h2>
+<hr/>
+<p>One of the most impactful optimizations you can add to a custom table provider
+is <strong>filter pushdown</strong> -- letting the source skip data that the query does not
+need, rather than reading everything and filtering it afterward.</p>
+<h3 id="how-filter-pushdown-works">How Filter Pushdown Works<a class="headerlink" href="#how-filter-pushdown-works" title="Permanent link">¶</a></h3>
+<p>When DataFusion plans a query with a <code>WHERE</code> clause, it passes the filter
+predicates to your <code>scan()</code> method as the <code>filters</code> parameter. By default,
+DataFusion assumes your provider cannot handle any filters and inserts a
+<code>FilterExec</code> node above your scan to apply them. But if your source <em>can</em>
+evaluate some predicates during scanning -- for example, by skipping files,
+partitions, or row groups that cannot match -- you can eliminate a huge amount
+of unnecessary I/O.</p>
+<p>To opt in, implement <code>supports_filters_pushdown</code>:</p>
+<pre><code class="language-rust">fn supports_filters_pushdown(
+ &amp;self,
+ filters: &amp;[&amp;Expr],
+) -&gt; Result&lt;Vec&lt;TableProviderFilterPushDown&gt;&gt; {
+ Ok(filters.iter().map(|f| {
+ match f {
+ // We can fully evaluate equality filters on
+ // the partition column at the source
+ Expr::BinaryExpr(BinaryExpr {
+ left, op: Operator::Eq, right
+ }) if is_partition_column(left) || is_partition_column(right) =&gt; {
+ TableProviderFilterPushDown::Exact
+ }
+ // All other filters: let DataFusion handle them
+ _ =&gt; TableProviderFilterPushDown::Unsupported,
+ }
+ }).collect())
+}
+</code></pre>
+<p>The three possible responses for each filter are:</p>
+<ul>
+<li><strong><code>Exact</code></strong> -- Your source guarantees that no output rows will have a false
+ value for this predicate. Because the filter is fully evaluated at the source,
+ DataFusion will <strong>not</strong> add a <code>FilterExec</code> for it.</li>
+<li><strong><code>Inexact</code></strong> -- Your source has the ability to reduce the data produced, but
+ the output may still include rows that do not satisfy the predicate. For
+ example, you might skip entire files based on metadata statistics but not
+ filter individual rows within a file. DataFusion will still add a <code>FilterExec</code>
+ above your scan to remove any remaining rows that slipped through.</li>
+<li><strong><code>Unsupported</code></strong> -- Your source ignores this filter entirely. DataFusion
+ handles it.</li>
+</ul>
+<h3 id="why-filter-pushdown-matters">Why Filter Pushdown Matters<a class="headerlink" href="#why-filter-pushdown-matters" title="Permanent link">¶</a></h3>
+<p>Consider a table with 1 billion rows partitioned by <code>region</code>, and a query:</p>
+<pre><code class="language-sql">SELECT * FROM events WHERE region = 'us-east-1' AND event_type = 'click';
+</code></pre>
+<p><strong>Without filter pushdown:</strong> Your table provider reads all 1 billion rows
+across all regions. DataFusion then applies both filters, discarding the vast
+majority of the data.</p>
+<p><strong>With filter pushdown on <code>region</code>:</strong> Your <code>scan()</code> method sees the
+<code>region = 'us-east-1'</code> filter and constructs an execution plan that only reads
+the <code>us-east-1</code> partition. If that partition holds 100 million rows, you have
+just eliminated 90% of the I/O. DataFusion still applies the <code>event_type</code>
+filter via <code>FilterExec</code> if you reported it as <code>Unsupported</code>.</p>
... 1653 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org