You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/18 16:54:33 UTC
(datafusion-site) branch asf-staging updated: Commit build products
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new c50d0f4 Commit build products
c50d0f4 is described below
commit c50d0f4f76f0c6da255e7d109f7ab6c07478326d
Author: Build Pelican (action) <pr...@infra.apache.org>
AuthorDate: Sat Apr 18 16:54:28 2026 +0000
Commit build products
---
blog/2024/01/19/datafusion-34.0.0/index.html | 8 +-
blog/2024/03/06/comet-donation/index.html | 4 +-
.../2024/08/20/python-datafusion-40.0.0/index.html | 8 +-
blog/2025/03/11/ordering-analysis/index.html | 12 +-
.../2025/03/30/datafusion-python-46.0.0/index.html | 4 +-
blog/2025/04/10/fastest-tpch-generator/index.html | 10 -
blog/2025/06/30/cancellation/index.html | 2 +-
blog/2025/07/11/datafusion-47.0.0/index.html | 2 +-
.../07/14/user-defined-parquet-indexes/index.html | 14 +-
.../15/avoid-consecutive-repartitions/index.html | 24 +-
blog/2026/04/02/datafusion-53.0.0/index.html | 289 ++++
blog/2026/04/18/datafusion-comet-0.15.0/index.html | 248 ++++
blog/author/aditya-singh-rathore-andrew-lamb.html | 80 +-
...-badaracco-pydantic-andrew-lamb-influxdata.html | 76 +-
blog/author/agrove.html | 98 +-
blog/author/alamb-akurmustafa.html | 98 +-
blog/author/alamb-dandandan-tustvold.html | 76 +-
.../andrew-lamb-achraf-b-and-sean-smith.html | 88 +-
blog/author/andrew-lamb-influxdata.html | 76 +-
.../andrew-lamb-staff-engineer-at-influxdata.html | 76 +-
blog/author/gene-bordegaray.html | 86 +-
blog/author/geoffrey-claude-datadog.html | 80 +-
blog/author/milenkovicm.html | 76 +-
blog/author/mustafa-akur-andrew-lamb.html | 78 +-
...anci-and-berkay-sahin-on-behalf-of-the-pmc.html | 76 +-
blog/author/pepijn-van-eeckhoudt.html | 98 +-
blog/author/pmc.html | 895 +++++------
...at-tu-darmstadt-and-andrew-lamb-influxdata.html | 80 +-
blog/author/tim-saucer-rerunio.html | 76 +-
...-dunningtonwherobots-andrew-lambinfluxdata.html | 76 +-
blog/author/timsaucer.html | 146 +-
blog/author/xiangpeng-hao-andrew-lamb.html | 98 +-
blog/author/xiangpeng-hao.html | 98 +-
blog/author/xudong.html | 76 +-
blog/category/blog.html | 1547 ++++++++++----------
blog/feed.xml | 195 ++-
.../aditya-singh-rathore-andrew-lamb.atom.xml | 4 +-
.../feeds/aditya-singh-rathore-andrew-lamb.rss.xml | 4 +-
...aracco-pydantic-andrew-lamb-influxdata.atom.xml | 4 +-
...daracco-pydantic-andrew-lamb-influxdata.rss.xml | 4 +-
blog/feeds/agrove.atom.xml | 6 +-
blog/feeds/agrove.rss.xml | 6 +-
blog/feeds/alamb-akurmustafa.atom.xml | 6 +-
blog/feeds/alamb-akurmustafa.rss.xml | 6 +-
blog/feeds/alamb-dandandan-tustvold.atom.xml | 4 +-
blog/feeds/alamb-dandandan-tustvold.rss.xml | 4 +-
blog/feeds/all-en.atom.xml | 552 +++++--
.../andrew-lamb-achraf-b-and-sean-smith.atom.xml | 28 +-
.../andrew-lamb-achraf-b-and-sean-smith.rss.xml | 16 +-
blog/feeds/andrew-lamb-influxdata.atom.xml | 4 +-
blog/feeds/andrew-lamb-influxdata.rss.xml | 4 +-
...drew-lamb-staff-engineer-at-influxdata.atom.xml | 4 +-
...ndrew-lamb-staff-engineer-at-influxdata.rss.xml | 4 +-
blog/feeds/blog.atom.xml | 552 +++++--
blog/feeds/gene-bordegaray.atom.xml | 32 +-
blog/feeds/gene-bordegaray.rss.xml | 8 +-
blog/feeds/geoffrey-claude-datadog.atom.xml | 4 +-
blog/feeds/geoffrey-claude-datadog.rss.xml | 4 +-
blog/feeds/milenkovicm.atom.xml | 4 +-
blog/feeds/milenkovicm.rss.xml | 4 +-
blog/feeds/mustafa-akur-andrew-lamb.atom.xml | 18 +-
blog/feeds/mustafa-akur-andrew-lamb.rss.xml | 6 +-
...-and-berkay-sahin-on-behalf-of-the-pmc.atom.xml | 4 +-
...i-and-berkay-sahin-on-behalf-of-the-pmc.rss.xml | 4 +-
blog/feeds/pepijn-van-eeckhoudt.atom.xml | 8 +-
blog/feeds/pepijn-van-eeckhoudt.rss.xml | 6 +-
blog/feeds/pmc.atom.xml | 400 ++++-
blog/feeds/pmc.rss.xml | 119 +-
...u-darmstadt-and-andrew-lamb-influxdata.atom.xml | 18 +-
...tu-darmstadt-and-andrew-lamb-influxdata.rss.xml | 4 +-
blog/feeds/tim-saucer-rerunio.atom.xml | 4 +-
blog/feeds/tim-saucer-rerunio.rss.xml | 4 +-
...ningtonwherobots-andrew-lambinfluxdata.atom.xml | 4 +-
...nningtonwherobots-andrew-lambinfluxdata.rss.xml | 4 +-
blog/feeds/timsaucer.atom.xml | 22 +-
blog/feeds/timsaucer.rss.xml | 10 +-
blog/feeds/xiangpeng-hao-andrew-lamb.atom.xml | 6 +-
blog/feeds/xiangpeng-hao-andrew-lamb.rss.xml | 6 +-
blog/feeds/xiangpeng-hao.atom.xml | 6 +-
blog/feeds/xiangpeng-hao.rss.xml | 6 +-
blog/feeds/xudong.atom.xml | 4 +-
blog/feeds/xudong.rss.xml | 4 +-
.../datafusion-53.0.0/field-access-pushdown.svg | 58 +
.../datafusion-53.0.0/join-filter-pushdown.svg | 99 ++
.../performance_over_time_clickbench.png | Bin 0 -> 57883 bytes
blog/index.html | 105 +-
86 files changed, 4581 insertions(+), 2690 deletions(-)
diff --git a/blog/2024/01/19/datafusion-34.0.0/index.html b/blog/2024/01/19/datafusion-34.0.0/index.html
index e4cc5fb..7253aeb 100644
--- a/blog/2024/01/19/datafusion-34.0.0/index.html
+++ b/blog/2024/01/19/datafusion-34.0.0/index.html
@@ -147,16 +147,16 @@ more than 2x faster on <a href="https://benchmark.clickhouse.com/">ClickBench</a
Average runtime on 25.0.0: 7.2s (for the queries that actually ran)
Average runtime on 34.0.0: 3.6s (for the same queries that ran in 25.0.0)
-->
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare-new.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare-new.png"/>
<figcaption>
<b>Figure 1</b>: Performance improvement between <code>25.0.0</code> and <code>34.0.0</code> on ClickBench.
Note that DataFusion <code>25.0.0</code>, could not run several queries due to
unsupported SQL (Q9, Q11, Q12, Q14) or memory requirements (Q33).
</figcaption>
</figure>
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare.png"/>
<figcaption>
<b>Figure 2</b>: Total query runtime for DataFusion <code>34.0.0</code> and DataFusion <code>25.0.0</code>.
</figcaption>
diff --git a/blog/2024/03/06/comet-donation/index.html b/blog/2024/03/06/comet-donation/index.html
index e2b7af5..1e728d5 100644
--- a/blog/2024/03/06/comet-donation/index.html
+++ b/blog/2024/03/06/comet-donation/index.html
@@ -84,8 +84,8 @@ a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/datafu
accelerate Spark workloads. It is designed as a drop-in
replacement for Spark's JVM based SQL execution engine and offers significant
performance improvements for some workloads as shown below.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-comet/comet-architecture.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-comet/comet-architecture.png"/>
<figcaption>
<b>Figure 1</b>: With Comet, users interact with the same Spark ecosystem, tools
and APIs such as Spark SQL. Queries still run through Spark's query optimizer and planner.
diff --git a/blog/2024/08/20/python-datafusion-40.0.0/index.html b/blog/2024/08/20/python-datafusion-40.0.0/index.html
index a8e0ab9..31522e0 100644
--- a/blog/2024/08/20/python-datafusion-40.0.0/index.html
+++ b/blog/2024/08/20/python-datafusion-40.0.0/index.html
@@ -118,8 +118,8 @@ the available functions and classes to see the breadth of available functionalit
<a href="https://jedi.readthedocs.io/en/latest/">Jedi</a> to perform analysis of python code, provide useful
hints, and identify usage errors. These are major tools in the python user community. With this
release, users can fully use these tools in their workflow.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Enhanced tooltips in an IDE." class="img-fluid" src="/blog/images/python-datafusion-40.0.0/vscode_hover_tooltip.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Enhanced tooltips in an IDE." class="img-fluid" src="/blog/images/python-datafusion-40.0.0/vscode_hover_tooltip.png"/>
<figcaption>
<b>Figure 1</b>: With the enhanced python wrappers, users can see helpful tool tips with
type annotations directly in modern IDEs.
@@ -127,8 +127,8 @@ release, users can fully use these tools in their workflow.</p>
</figure>
<p>By having the type annotations, these IDEs can also identify quickly when a user has incorrectly
used a function's arguments as shown in Figure 2.</p>
-<figure style="text-align: center;">
-<img alt="Fig 2: Error checking in static analysis" class="img-fluid" src="/blog/images/python-datafusion-40.0.0/pylance_error_checking.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 2: Error checking in static analysis" class="img-fluid" src="/blog/images/python-datafusion-40.0.0/pylance_error_checking.png"/>
<figcaption>
<b>Figure 2</b>: Modern Python language servers can perform static analysis and quickly find
errors in the arguments to functions.
diff --git a/blog/2025/03/11/ordering-analysis/index.html b/blog/2025/03/11/ordering-analysis/index.html
index f894d45..588fff5 100644
--- a/blog/2025/03/11/ordering-analysis/index.html
+++ b/blog/2025/03/11/ordering-analysis/index.html
@@ -100,7 +100,7 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
</blockquote>
<p>There are various use cases where this type of analysis can be useful such as the following examples.</p>
@@ -181,7 +181,7 @@ Being logically streamable does not guarantee that a query will execute in a str
</tr>
</table>
<p><br/></p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>How can a table have multiple orderings?</strong> At first glance it may seem counterintuitive for a table to have more than one valid ordering. However, during query execution such scenarios can arise.
For example consider the following query:
@@ -242,7 +242,7 @@ DataFusion leverages these functional dependencies to infer new orderings as dat
<li>Equivalent Expression Groups (will be explained shortly)</li>
<li>Succinct Valid Orderings (will be explained shortly)</li>
</ul>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Note:</strong> These properties are implemented in the <code>EquivalenceProperties</code> structure in <code>DataFusion</code>, please see the <a href="https://github.com/apache/datafusion/blob/f47ea73b87eec4af044f9b9923baf042682615b2/datafusion/physical-expr/src/equivalence/properties/mod.rs#L134" target="_blank">source</a> for more details<br/>
</blockquote>
<p>These properties allow us to analyze whether the ordering requirement is satisfied by the data already.</p>
@@ -252,7 +252,7 @@ DataFusion leverages these functional dependencies to infer new orderings as dat
<ul>
<li>Columns <code>hostname</code> and <code>currency</code> are constant because every row in the table has the same value (<code>'app.example.com'</code> for <code>hostname</code>, and <code>'USD'</code> for <code>currency</code>) for these columns.</li>
</ul>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Note:</strong> Constant expressions can arise during query execution. For example, in following query:<br/>
<code>SELECT hostname FROM logs</code><br/><code>WHERE hostname='app.example.com'</code> <br/>
after filtering is done, for subsequent operators the <code>hostname</code> column will be constant.
@@ -260,7 +260,7 @@ DataFusion leverages these functional dependencies to infer new orderings as dat
<h3 id="2-equivalent-expression-groups">2. Equivalent Expression Groups<a class="headerlink" href="#2-equivalent-expression-groups" title="Permanent link">¶</a></h3>
<p>Equivalent expression groups are expressions that always hold the same value across rows. These expressions can be thought of as clones of each other and may arise from operations like <code>Filter</code>, <code>Join</code>, or <code>Projection</code>.</p>
<p>In the example table, the expressions <code>price</code> and <code>price_cloned</code> form one equivalence group, and <code>time</code> and <code>time_cloned</code> form another equivalence group.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Note:</strong> Equivalent expression groups can arise during the query execution. For example, in the following query:<br/>
<code>SELECT time, time as time_cloned FROM logs</code> <br/>
after the projection is done, for subsequent operators <code>time</code> and <code>time_cloned</code> will form an equivalence group. As another example, in the following query:<br/>
@@ -328,7 +328,7 @@ Following third and fourth constraints for the simplified table, the succinct va
<code>[amount ASC, price ASC]</code>, <br/>
<code>[time_bin ASC]</code>,<br/>
<code>[time ASC]</code> </p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<p><strong>How can DataFusion find orderings?</strong></p>
DataFusion's <code>CREATE EXTERNAL TABLE</code> has a <code>WITH ORDER</code> clause (see <a href="https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table">docs</a>) to specify the known orderings of the table during table creation. For example the following query:<br/>
<pre><code>
diff --git a/blog/2025/03/30/datafusion-python-46.0.0/index.html b/blog/2025/03/30/datafusion-python-46.0.0/index.html
index 70206cd..655d006 100644
--- a/blog/2025/03/30/datafusion-python-46.0.0/index.html
+++ b/blog/2025/03/30/datafusion-python-46.0.0/index.html
@@ -182,8 +182,8 @@ for the user to click on it to expand the data.</li>
</ul>
<p>In the below view you can see an example of some of these features such as the
expandable text and scroll bars.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Example html rendering in a jupyter notebook." class="img-fluid" src="/blog/images/python-datafusion-46.0.0/html_rendering.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Example html rendering in a jupyter notebook." class="img-fluid" src="/blog/images/python-datafusion-46.0.0/html_rendering.png"/>
<figcaption>
<b>Figure 1</b>: With the html rendering enhancements, tables are more easily
viewable in jupyter notebooks.
diff --git a/blog/2025/04/10/fastest-tpch-generator/index.html b/blog/2025/04/10/fastest-tpch-generator/index.html
index cee56da..b756b9c 100644
--- a/blog/2025/04/10/fastest-tpch-generator/index.html
+++ b/blog/2025/04/10/fastest-tpch-generator/index.html
@@ -88,16 +88,6 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
diff --git a/blog/2025/06/30/cancellation/index.html b/blog/2025/06/30/cancellation/index.html
index 3fe09a8..cfa4ca0 100644
--- a/blog/2025/06/30/cancellation/index.html
+++ b/blog/2025/06/30/cancellation/index.html
@@ -355,7 +355,7 @@ The task will then yield, after which the Tokio runtime resets the counter.</p>
</figure>
<p>If we assume a task budget of 1 unit, each time Tokio schedules the task would result in the following sequence of function calls.</p>
<figure>
-<img alt="Sequence diagram showing how the tokio task budget is used and reset." class="img-fluid" src="/blog/images/task-cancellation/tokio_budget.png" style="width: 100%; max-width: 100%"/>
+<img alt="Sequence diagram showing how the tokio task budget is used and reset." class="img-fluid" src="/blog/images/task-cancellation/tokio_budget.png"/>
<figcaption>Tokio task budget system, assuming the task budget is set to 1, for the plan above.</figcaption>
</figure>
<p>The aggregation stream would try to poll the merge stream in a loop.
diff --git a/blog/2025/07/11/datafusion-47.0.0/index.html b/blog/2025/07/11/datafusion-47.0.0/index.html
index 6b5316b..f8bc9ed 100644
--- a/blog/2025/07/11/datafusion-47.0.0/index.html
+++ b/blog/2025/07/11/datafusion-47.0.0/index.html
@@ -229,7 +229,7 @@ the SQL <code>varchar</code> columns will be mapped to the <a href="https://docs
logging, or metrics) across thread boundaries without depending on any specific instrumentation library.
You can use the <a href="https://docs.rs/datafusion/latest/datafusion/common/runtime/trait.JoinSetTracer.html">JoinSetTracer</a> API to instrument DataFusion plans with your own tracing or logging libraries, or
use pre-integrated community crates such as the <a href="https://github.com/datafusion-contrib/datafusion-tracing">datafusion-tracing</a> crate.</p>
-<div style="text-align: center;">
+<div class="text-center">
<a href="https://github.com/datafusion-contrib/datafusion-tracing">
<img alt="DataFusion telemetry project logo" class="img-fluid" src="/blog/images/datafusion-47.0.0/datafusion-telemetry.png" width="50%"/>
</a>
diff --git a/blog/2025/07/14/user-defined-parquet-indexes/index.html b/blog/2025/07/14/user-defined-parquet-indexes/index.html
index 52d37ee..e1368e1 100644
--- a/blog/2025/07/14/user-defined-parquet-indexes/index.html
+++ b/blog/2025/07/14/user-defined-parquet-indexes/index.html
@@ -189,24 +189,24 @@ limitations under the License.
<hr/>
<p>A <strong>distinct value index</strong> stores the unique values of a specific column. This type of index is effective for columns with a small number of distinct values and can be used to quickly skip files that do not match the query. These indexes are popular in several engines, such as the <a href="https://clickhouse.com/docs/optimize/skipping-indexes#set">"set" Skip Index in ClickHouse</a> and the <a href="https://docs.influxdata.com/influxdb3/enterprise/admin/distinct-value-cach [...]
<p>For example, if the files contain a column named <code>Category</code> like this:</p>
-<table class="table" style="border-collapse:collapse;">
+<table class="table">
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><b><code>Category</code></b></td>
+<td><b><code>Category</code></b></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>foo</code></td>
+<td><code>foo</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>bar</code></td>
+<td><code>bar</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>...</code></td>
+<td><code>...</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>baz</code></td>
+<td><code>baz</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>foo</code></td>
+<td><code>foo</code></td>
</tr>
</table>
<p>The distinct value index will contain the values <code>foo</code>, <code>bar</code>, and <code>baz</code>. In contrast, traditional min/max statistics would store only the minimum (<code>bar</code>) and maximum (<code>foo</code>) values, so a query like</p>
diff --git a/blog/2025/12/15/avoid-consecutive-repartitions/index.html b/blog/2025/12/15/avoid-consecutive-repartitions/index.html
index c405879..1849c80 100644
--- a/blog/2025/12/15/avoid-consecutive-repartitions/index.html
+++ b/blog/2025/12/15/avoid-consecutive-repartitions/index.html
@@ -69,16 +69,16 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is exposed by a user-friendly interface and query languages (typically a dialect of SQL).
<br/><br/>
Starting a journey learning about database internals can be daunting. With so many topics that are whole PhD degrees themselves, finding a place to start is difficult. In this blog post, I will share my early journey in the database world and a quick lesson on one of the first topics I dove into. If you are new to the space, this post will help you get your first foot into the database world, and if you are already a veteran, you may still learn something new.
</div>
-<div style="flex: 0 0 40%; text-align: center;">
-<img alt="Database System Components" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/database_system_diagram.png" width="100%"/>
+<div class="col-md-5 text-center">
+<img alt="Database System Components" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/database_system_diagram.png"/>
</div>
</div>
<hr/>
@@ -119,29 +119,29 @@ DataFusion achieves multi-core parallelism through the use of "exchange operator
<h3 id="what-is-repartitioning">What is Repartitioning?<a class="headerlink" href="#what-is-repartitioning" title="Permanent link">¶</a></h3>
<p>Partitioning is a "divide-and-conquer" approach to executing a query. Each partition is a subset of the data that is being processed on a single core. Repartitioning is an operation that redistributes data across different partitions to balance workloads, reduce data skew, and increase parallelism. Two repartitioning methods are used in DataFusion: round-robin and hash.</p>
<h4 id="round-robin-repartitioning"><strong>Round-Robin Repartitioning</strong><a class="headerlink" href="#round-robin-repartitioning" title="Permanent link">¶</a></h4>
-<div style="display: flex; align-items: top; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-start mb-3">
+<div class="col-md-9">
Round-robin repartitioning is the simplest partitioning strategy. Incoming data is processed in batches (chunks of rows), and these batches are distributed across partitions cyclically or sequentially, with each new batch assigned to the next available partition.
<br/><br/>
Round-robin repartitioning is useful when the data grouping isn't known or when aiming for an even distribution across partitions. Because it simply assigns batches in order without inspecting their contents, it is a low-overhead way to increase parallelism for downstream operations.
</div>
-<div style="flex: 0 0 25%; text-align: center;">
-<img alt="Round-Robin Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/round_robin_repartitioning.png" width="100%"/>
+<div class="col-md-3 text-center">
+<img alt="Round-Robin Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/round_robin_repartitioning.png"/>
</div>
</div>
<h4 id="hash-repartitioning"><strong>Hash Repartitioning</strong><a class="headerlink" href="#hash-repartitioning" title="Permanent link">¶</a></h4>
-<div style="display: flex; align-items: top; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-start mb-3">
+<div class="col-md-9">
Hash repartitioning distributes data based on a hash function applied to one or more columns, called the partitioning key. Rows with the same hash value are placed in the same partition.
<br/><br/>
Hash repartitioning is useful when working with grouped data. Imagine you have a database containing information on company sales, and you are looking to find the total revenue each store produced. Hash repartitioning would make this query much more efficient. Rather than iterating over the data on a single thread and keeping a running sum for each store, it would be better to hash repartition on the store column and have multiple threads calculate individual store sales.
</div>
-<div style="flex: 0 0 25%; text-align: center;">
-<img alt="Hash Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/hash_repartitioning.png" width="100%"/>
+<div class="col-md-3 text-center">
+<img alt="Hash Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/hash_repartitioning.png"/>
</div>
</div>
<p>Note, the benefit of hash opposed to round-robin partitioning in this scenario. Hash repartitioning consolidates all rows with the same store value in distinct partitions. Because of this property we can compute the complete results for each store in parallel and merge them to get the final outcome. This parallel processing wouldn’t be possible with only round-robin partitioning as the same store value may be spread across multiple partitions, making the aggregation results part [...]
diff --git a/blog/2026/04/02/datafusion-53.0.0/index.html b/blog/2026/04/02/datafusion-53.0.0/index.html
new file mode 100644
index 0000000..51c6d49
--- /dev/null
+++ b/blog/2026/04/02/datafusion-53.0.0/index.html
@@ -0,0 +1,289 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion 53.0.0 Released - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
+ </div>
+ </div>
+</nav>
+<!-- article contents -->
+<div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>
+ Apache DataFusion 53.0.0 Released
+ </h1>
+ <p>Posted on: Thu 02 April 2026 by pmc</p>
+
+ <aside class="toc-container d-md-none mb-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#performance-improvements">Performance Improvements 🚀</a><ul>
+<li><a href="#limit-aware-parquet-row-group-pruning">LIMIT-Aware Parquet Row Group Pruning</a></li>
+<li><a href="#improved-filter-pushdown">Improved Filter Pushdown</a></li>
+<li><a href="#faster-query-planning">Faster Query Planning</a></li>
+<li><a href="#faster-functions">Faster Functions</a></li>
+<li><a href="#nested-field-pushdown">Nested Field Pushdown</a></li>
+</ul>
+</li>
+<li><a href="#new-features">New Features ✨</a></li>
+<li><a href="#stability-and-release-engineering">Stability and Release Engineering 🦺</a></li>
+<li><a href="#upgrade-notes">Upgrade Notes</a></li>
+<li><a href="#known-issues">Known Issues</a></li>
+<li><a href="#thank-you">Thank You</a></li>
+</ul>
+</div>
+ </aside>
+
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/53.0.0">DataFusion 53.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0/">DataFusion 52.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md#credits">114 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p><img alt="Performance over time" class="img-fluid" src="/blog/images/datafusion-53.0.0/performance_over_time_clickbench.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: Average and median normalized execution times for DataFusion 53.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+<a href="https://alamb.github.io/datafusion-benchmarking/">DataFusion Benchmarking Page</a>
+for more details.</p>
+<p>DataFusion 53 continues the project-wide focus on performance. This release
+reduces planning overhead, skips more unnecessary I/O, and pushes more work
+into earlier and cheaper stages of execution.</p>
+<h3 id="limit-aware-parquet-row-group-pruning"><code>LIMIT</code>-Aware Parquet Row Group Pruning<a class="headerlink" href="#limit-aware-parquet-row-group-pruning" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 includes a new optimization that makes Parquet pruning aware of
+<code>LIMIT</code>. This optimization is described in full in <a href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning/">limit pruning blog post</a>. If
+DataFusion can prove that an entire row group matches the predicate, and those
+fully matching row groups contain enough rows to satisfy the <code>LIMIT</code>, partially
+matching row groups are skipped entirely.</p>
+<figure>
+<img alt="Pruning pipeline with limit pruning highlighted" class="img-fluid" src="/blog/images/limit-pruning/pruning-pipeline.svg" width="80%"/>
+<figcaption><b>Figure 2</b>: Limit pruning is inserted between row group and page index pruning.</figcaption>
+</figure>
+<p>Thanks to <a href="https://github.com/xudong963">@xudong963</a> for implementing this feature. Related PRs: <a href="https://github.com/apache/datafusion/pull/18868">#18868</a></p>
+<h3 id="improved-filter-pushdown">Improved Filter Pushdown<a class="headerlink" href="#improved-filter-pushdown" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 pushes filters down through more join types and through <code>UnionExec</code>,
+and expands support for pushing down <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters">dynamic filters</a>. More
+pushdown means fewer rows flow into joins, repartitions, and later operators,
+which reduces CPU, memory, and I/O.</p>
+<p>For example:</p>
+<pre><code class="language-sql">SELECT *
+FROM (
+ SELECT *
+ FROM t1
+ LEFT ANTI JOIN t2 ON t1.k = t2.k
+) a
+JOIN t1 b ON a.k = b.k
+WHERE b.v = 1;
+</code></pre>
+<p>Now DataFusion can often transform the physical plan so filters and
+<a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters">dynamic filters</a> are pushed deeper into the plan, even through subqueries and
+nested joins. In this example, the filter on <code>b.v</code> helps produce dynamic filters
+that can be pushed into both sides of the nested anti join.</p>
+<figure>
+<img alt="Before and after diagram of dynamic filter pushdown through a subquery with nested joins" class="img-fluid" src="/blog/images/datafusion-53.0.0/join-filter-pushdown.svg" width="100%"/>
+<figcaption><b>Figure 3</b>: DataFusion 53 pushes dynamic filters through subqueries and into both sides of nested joins.</figcaption>
+</figure>
+<p>Thanks to <a href="https://github.com/nuno-faria">@nuno-faria</a>, <a href="https://github.com/haohuaijin">@haohuaijin</a>, and <a href="https://github.com/jackkleeman">@jackkleeman</a> for
+driving this work. Related PRs: <a href="https://github.com/apache/datafusion/pull/19918">#19918</a>, <a href="https://github.com/apache/datafusion/pull/20145">#20145</a>, <a href="https://github.com/apache/datafusion/pull/20192">#20192</a></p>
+<h3 id="faster-query-planning">Faster Query Planning<a class="headerlink" href="#faster-query-planning" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 improves query planning performance by making immutable pieces of
+execution plans cheaper to clone. This helps applications that need extremely
+low latency, plan many or complex queries, or use prepared statements or
+parameterized queries. In some benchmarks, overall execution time drops from
+roughly 4-5 ms to about 100 us.</p>
+<p>Thanks to <a href="https://github.com/askalt">@askalt</a> for leading this work. Related PRs: <a href="https://github.com/apache/datafusion/pull/19792">#19792</a>, <a href="https://github.com/apache/datafusion/pull/19893">#19893</a></p>
+<h3 id="faster-functions">Faster Functions<a class="headerlink" href="#faster-functions" title="Permanent link">¶</a></h3>
+<p>DataFusion includes <a href="https://datafusion.apache.org/user-guide/sql/scalar_functions.html">235 built-in functions</a>. Improving the performance of these
+functions benefits a wide range of workloads. This release improves the performance of 42 of those
+functions, such as <a href="https://github.com/apache/datafusion/pull/20295">strpos</a>, <a href="https://github.com/apache/datafusion/pull/20344">replace</a>, <a href="https://github.com/apache/datafusion/pull/20317">concat</a>, <a href="https://github.com/apache/datafusion/pull/20305">translate</a>, <a href="https://github.com/apache/datafusion/pull/20374">array_has</a>,
+<a href="https://github.com/apache/datafusion/pull/20504">array_agg</a>, <a href="https://github.com/apache/datafusion/pull/19980">left</a>, <a href="https://github.com/apache/datafusion/pull/20069">right</a>, and <a href="https://github.com/apache/datafusion/pull/20097">case_when</a>.</p>
+<p>Thanks to the contributors who drove this work, especially <a href="https://github.com/neilconway">@neilconway</a>,
+<a href="https://github.com/theirix">@theirix</a>, <a href="https://github.com/lyne7-sc">@lyne7-sc</a>, <a href="https://github.com/kumarUjjawal">@kumarUjjawal</a>, <a href="https://github.com/pepijnve">@pepijnve</a>, <a href="https://github.com/zhangxffff">@zhangxffff</a>, and
+<a href="https://github.com/UBarney">@UBarney</a>.</p>
+<h3 id="nested-field-pushdown">Nested Field Pushdown<a class="headerlink" href="#nested-field-pushdown" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 pushes expressions such as <code>get_field</code> down the plan and into data
+sources. This is especially important for nested data such as structs in
+Parquet files. Instead of reading an entire struct column and then extracting
+the field of interest, DataFusion 53 pushes the field extraction into the scan.</p>
+<p>For example, the following query reads a struct column <code>s</code> and extracts the
+<code>label</code> field for rows where the <code>value</code> field is greater than 150:</p>
+<pre><code class="language-sql">SELECT id, s['label']
+FROM t
+WHERE s['value'] > 150;
+</code></pre>
+<figure>
+<img alt="Before and after diagram of field access pushdown into a data source" class="img-fluid" src="/blog/images/datafusion-53.0.0/field-access-pushdown.svg" width="80%"/>
+<figcaption><b>Figure 4</b>: DataFusion 53 pushes field-access expressions closer to the scan.</figcaption>
+</figure>
+<p>Special thanks to <a href="https://github.com/adriangb">@adriangb</a> for designing and implementing this optimizer
+work. Related PRs: <a href="https://github.com/apache/datafusion/pull/20065">#20065</a>, <a href="https://github.com/apache/datafusion/pull/20117">#20117</a>, <a href="https://github.com/apache/datafusion/pull/20239">#20239</a></p>
+<h2 id="new-features">New Features ✨<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h2>
+<ul>
+<li>
+<p><strong>JSON Array File Support</strong>: DataFusion 53 can now read JSON arrays such as
+ <code>[{...}, {...}]</code> directly as multiple rows, including streaming inputs from
+ object stores. Thanks to <a href="https://github.com/zhuqi-lucas">@zhuqi-lucas</a> for implementing this feature.
+ Related PRs: <a href="https://github.com/apache/datafusion/pull/19924">#19924</a></p>
+</li>
+<li>
+<p><strong>Support for <code>:</code> operator</strong>: DataFusion can plan queries such as
+ <code>SELECT payload:'user_id' FROM events;</code>, enabling better <a href="https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/">Parquet Variant</a>
+ support via <a href="https://github.com/datafusion-contrib/datafusion-variant">datafusion-variant</a>. Thanks to <a href="https://github.com/Samyak2">@Samyak2</a>. Related PRs:
+ <a href="https://github.com/apache/datafusion/pull/20717">#20717</a></p>
+</li>
+<li>
+<p><strong>New SQL</strong>: DataFusion supports additional set-comparison subqueries, null-aware
+ anti join, and deletion predicates. Thanks to <a href="https://github.com/waynexia">@waynexia</a>, <a href="https://github.com/viirya">@viirya</a>, and
+ <a href="https://github.com/askalt">@askalt</a> for key contributions in this area. Related PRs: <a href="https://github.com/apache/datafusion/pull/19109">#19109</a>,
+ <a href="https://github.com/apache/datafusion/pull/19635">#19635</a>, <a href="https://github.com/apache/datafusion/pull/20137">#20137</a></p>
+</li>
+<li>
+<p><strong>Spark-Compatible Functions</strong>: This release includes almost 20 new or improved
+ Spark-compatible functions and behaviors in the <a href="https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html">datafusion-spark crate</a>.
+ It includes functions such as <a href="https://github.com/apache/datafusion/pull/19699">collect_list</a>, <a href="https://github.com/apache/datafusion/pull/19845">date_diff</a>,
+ <a href="https://github.com/apache/datafusion/pull/19880">from_utc_timestamp</a>, <a href="https://github.com/apache/datafusion/pull/20412">json_tuple</a>, <a href="https://github.com/apache/datafusion/pull/20440">arrays_zip</a>, <a href="https://github.com/apache/datafusion/pull/20479">bin</a>, and <a href="https://github.com/apache/datafusion/pull/20685">array_contains</a>.
+ Thanks to the contributors who drove this work, especially <a href="https://github.com/cht42">@cht42</a>,
+ <a href="https://github.com/CuteChuanChuan">@CuteChuanChuan</a>, <a href="https://github.com/SubhamSinghal">@SubhamSinghal</a>, <a href="https://github.com/kazantsev-maksim">@kazantsev-maksim</a>, <a href="https://github.com/unknowntpo">@unknowntpo</a>,
+ <a href="https://github.com/aryan-212">@aryan-212</a>, <a href="https://github.com/hsiang-c">@hsiang-c</a>, and <a href="https://github.com/davidlghellin">@davidlghellin</a>.</p>
+</li>
+</ul>
+<h2 id="stability-and-release-engineering">Stability and Release Engineering 🦺<a class="headerlink" href="#stability-and-release-engineering" title="Permanent link">¶</a></h2>
+<p>The community spent significant time this release cycle stabilizing the release
+branch and improving the release process. While such improvements are not as
+headline-friendly as new features, they are highly important for real
+deployments. We are discussing ways to improve the process on <a href="https://github.com/apache/datafusion/issues/21034">#21034</a> and would
+welcome suggestions and contributions to help with release engineering work in
+the future.</p>
+<p>Thanks to <a href="https://github.com/comphead">@comphead</a> for running this release, and to <a href="https://github.com/jonathanc-n">@jonathanc-n</a>, <a href="https://github.com/alamb">@alamb</a>,
+<a href="https://github.com/xanderbailey">@xanderbailey</a>, <a href="https://github.com/haohuaijin">@haohuaijin</a>, <a href="https://github.com/friendlymatthew">@friendlymatthew</a>, <a href="https://github.com/fwojciec">@fwojciec</a>,
+<a href="https://github.com/Kontinuation">@Kontinuation</a>, <a href="https://github.com/nathanb9">@nathanb9</a>, and many others who helped stabilize the release
+branch.</p>
+<h2 id="upgrade-notes">Upgrade Notes<a class="headerlink" href="#upgrade-notes" title="Permanent link">¶</a></h2>
+<p>DataFusion 53 includes some breaking changes, including updates to the SQL
+parser, optimizer behavior, and some physical-plan APIs. Please see the <a href="https://datafusion.apache.org/library-user-guide/upgrading/index.html">upgrade
+guide</a> and <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a> for the full details before upgrading.</p>
+<h2 id="known-issues">Known Issues<a class="headerlink" href="#known-issues" title="Permanent link">¶</a></h2>
+<p>A small number of issues were discovered after the 53.0.0 release,
+and we expect to publish DataFusion 53.1.0 soon. See the <a href="https://github.com/apache/datafusion/issues/21079">53.1.0 release tracking
+issue</a> for the latest status.</p>
+<h2 id="thank-you">Thank You<a class="headerlink" href="#thank-you" title="Permanent link">¶</a></h2>
+<p>Thank you to everyone in the DataFusion community who contributed code, reviews,
+testing, bug reports, documentation, and release engineering work for 53.0.0.
+This release contains direct contributions from 114 different people, and we are
+grateful for the time and effort that everyone put in to make it happen.</p>
+
+<!--
+ Comments Section
+ Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div>
+ <hr>
+ <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
+
+ <!-- Local loader script -->
+ <script src="/blog/js/giscus-consent.js" defer></script>
+
+ <!-- Consent UI -->
+ <div id="giscus-consent">
+ <p>
+ We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
+ To respect your privacy, Giscus and comments will load only if you click "Show Comments"
+ </p>
+
+ <div class="consent-actions">
+ <button id="giscus-load" type="button">Show Comments</button>
+ <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+ </div>
+
+ <noscript>JavaScript is required to load comments from Giscus.</noscript>
+ </div>
+
+ <!-- Container where Giscus will render -->
+ <div id="comment-thread"></div>
+</div> </div>
+ <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#performance-improvements">Performance Improvements 🚀</a><ul>
+<li><a href="#limit-aware-parquet-row-group-pruning">LIMIT-Aware Parquet Row Group Pruning</a></li>
+<li><a href="#improved-filter-pushdown">Improved Filter Pushdown</a></li>
+<li><a href="#faster-query-planning">Faster Query Planning</a></li>
+<li><a href="#faster-functions">Faster Functions</a></li>
+<li><a href="#nested-field-pushdown">Nested Field Pushdown</a></li>
+</ul>
+</li>
+<li><a href="#new-features">New Features ✨</a></li>
+<li><a href="#stability-and-release-engineering">Stability and Release Engineering 🦺</a></li>
+<li><a href="#upgrade-notes">Upgrade Notes</a></li>
+<li><a href="#known-issues">Known Issues</a></li>
+<li><a href="#thank-you">Thank You</a></li>
+</ul>
+</div>
+ </aside>
+ </div>
+ </div>
+</div>
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/2026/04/18/datafusion-comet-0.15.0/index.html b/blog/2026/04/18/datafusion-comet-0.15.0/index.html
new file mode 100644
index 0000000..bcf9b78
--- /dev/null
+++ b/blog/2026/04/18/datafusion-comet-0.15.0/index.html
@@ -0,0 +1,248 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion Comet 0.15.0 Release - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
+ </div>
+ </div>
+</nav>
+<!-- article contents -->
+<div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>
+ Apache DataFusion Comet 0.15.0 Release
+ </h1>
+ <p>Posted on: Sat 18 April 2026 by pmc</p>
+
+ <aside class="toc-container d-md-none mb-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#key-features">Key Features</a><ul>
+<li><a href="#native-iceberg-reader-enabled-by-default">Native Iceberg Reader Enabled by Default</a></li>
+<li><a href="#auto-mode-enables-native_datafusion-scan">Auto Mode Enables native_datafusion Scan</a></li>
+<li><a href="#new-expressions-and-function-support">New Expressions and Function Support</a></li>
+<li><a href="#expanded-metrics-and-observability">Expanded Metrics and Observability</a></li>
+</ul>
+</li>
+<li><a href="#performance-improvements">Performance Improvements</a></li>
+<li><a href="#stability-and-correctness">Stability and Correctness</a></li>
+<li><a href="#dependency-upgrades">Dependency Upgrades</a></li>
+<li><a href="#deprecations-and-removals">Deprecations and Removals</a></li>
+<li><a href="#compatibility">Compatibility</a></li>
+</ul>
+</div>
+ </aside>
+
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.15.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately four weeks of development work and is the result of merging 142 PRs from 19
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.15.0.md">change log</a> for more information.</p>
+<h2 id="key-features">Key Features<a class="headerlink" href="#key-features" title="Permanent link">¶</a></h2>
+<h3 id="native-iceberg-reader-enabled-by-default">Native Iceberg Reader Enabled by Default<a class="headerlink" href="#native-iceberg-reader-enabled-by-default" title="Permanent link">¶</a></h3>
+<p>This release marks a major milestone for Iceberg users: <strong>Comet's fully-native Iceberg reader is now enabled by
+default</strong>. Workloads that read Iceberg tables will automatically benefit from native Rust-based scans built on
+iceberg-rust, with no additional configuration required.</p>
+<p>To support this change, the release bundles a broad set of Iceberg-focused improvements:</p>
+<ul>
+<li><strong>Correct classloader handling</strong>: Iceberg classes are now loaded via the thread context classloader, resolving
+ class-loading issues in environments where the executor classloader differs from the application classloader.</li>
+<li><strong>Continuous Iceberg CI</strong>: Iceberg Spark integration tests now run on every PR and push to <code>main</code>, providing
+ continuous validation of the native Iceberg code path. Test diffs for Spark 3.4 were updated to keep the matrix
+ green across supported Spark versions.</li>
+<li><strong>iceberg-rust upgrade</strong>: Comet picks up the latest iceberg-rust, pulling in fixes for Parquet reader edge cases
+ discovered in earlier testing.</li>
+<li><strong>Refreshed documentation</strong>: The Iceberg user guide has been rewritten to reflect current capabilities, and the
+ contributor guide now documents how to run the Iceberg Spark test suites locally.</li>
+</ul>
+<p>Users who need to fall back to the previous behavior can still opt out, but we encourage the community to exercise
+the native reader and report any issues.</p>
+<h3 id="auto-mode-enables-native_datafusion-scan">Auto Mode Enables <code>native_datafusion</code> Scan<a class="headerlink" href="#auto-mode-enables-native_datafusion-scan" title="Permanent link">¶</a></h3>
+<p>The <code>auto</code> scan mode now enables the <code>native_datafusion</code> scan where supported, giving users the benefits of the
+native Parquet reader without having to explicitly opt in. This is part of the ongoing effort to make
+<code>native_datafusion</code> the default Parquet path once the deprecation of <code>native_iceberg_compat</code> completes.</p>
+<h3 id="new-expressions-and-function-support">New Expressions and Function Support<a class="headerlink" href="#new-expressions-and-function-support" title="Permanent link">¶</a></h3>
+<p>This release adds support for the following:</p>
+<ul>
+<li><strong>Date/time functions</strong>: <code>days</code>, <code>hours</code>, <code>date_from_unix_date</code></li>
+<li><strong>String/JSON functions</strong>: native <code>get_json_object</code> with improved performance over the fallback path</li>
+<li><strong>Hash/math functions</strong>: <code>crc32c</code>, <code>bin</code></li>
+<li><strong>Array functions</strong>: <code>sort_array</code></li>
+<li><strong>Window functions</strong>: <code>LEAD</code> and <code>LAG</code> with <code>IGNORE NULLS</code></li>
+<li><strong>Aggregates</strong>: SQL <code>FILTER (WHERE ...)</code> clauses now execute natively; <code>Corr</code> aggregate enabled</li>
+</ul>
+<h3 id="expanded-metrics-and-observability">Expanded Metrics and Observability<a class="headerlink" href="#expanded-metrics-and-observability" title="Permanent link">¶</a></h3>
+<p>Comet metrics can now be exposed through Spark's external monitoring system, making it easier to integrate Comet
+execution statistics with existing observability dashboards. Native DataFusion scans also now report accurate
+<code>filesScanned</code> and <code>bytesScanned</code> input metrics, matching Spark's native Parquet scan reporting.</p>
+<h2 id="performance-improvements">Performance Improvements<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>Performance was a major theme of this release, with a series of targeted optimizations across the shuffle, scan,
+and execution layers:</p>
+<ul>
+<li><strong>Shuffle read path</strong>: The native shuffle reader no longer uses FFI on the read side, removing a per-batch cost
+ that was particularly visible in shuffle-heavy queries.</li>
+<li><strong>Broadcast exchanges</strong>: Batches are now coalesced before broadcasting, reducing the number of small batches
+ crossing the JVM/native boundary.</li>
+<li><strong>Columnar-to-row (C2R)</strong>: Native C2R conversion is now exercised for a broader set of query shapes.</li>
+<li><strong>FFI-safe operators</strong>: More operators are marked as FFI-safe, avoiding unnecessary deep copies when crossing
+ the JVM/native boundary.</li>
+<li><strong>Shared memory pools</strong>: Unified memory pools are now shared across native execution contexts within a Spark
+ task, improving memory accounting and reducing fragmentation.</li>
+<li><strong>Object store caching</strong>: Object stores and bucket region lookups are cached, dramatically reducing DNS query
+ volume on workloads that open many files.</li>
+<li><strong><code>get_ranges</code> performance</strong>: Picked up an upstream <code>opendal</code> fix that restores fast range reads from object
+ storage.</li>
+<li><strong>Spill I/O</strong>: Removed a redundant <code>BufReader</code> wrapper when copying spill files to shuffle output.</li>
+</ul>
+<p>Together, these changes reduce CPU and memory overhead for shuffle-heavy, broadcast-heavy, and
+object-storage-bound workloads.</p>
+<h2 id="stability-and-correctness">Stability and Correctness<a class="headerlink" href="#stability-and-correctness" title="Permanent link">¶</a></h2>
+<p>A significant portion of this release is dedicated to stability and Spark compatibility. Highlights include:</p>
+<ul>
+<li><strong>Cast string to timestamp</strong>: Multiple fixes for UTC timestamps, timezone handling, special formats
+ (<code>epoch</code>, <code>now</code>, etc.), and compatibility with Spark's semantics.</li>
+<li><strong>Cast decimal to string</strong>: Added legacy mode handling to match Spark's output formatting.</li>
+<li><strong>String to decimal</strong>: Support for full-width characters, null characters, and negative scale.</li>
+<li><strong>Decimal arithmetic</strong>: Fixes for decimal division and additional test coverage for ANSI overflow handling,
+ including scalar decimal overflow.</li>
+<li><strong>Array expressions</strong>: Corrected <code>GetArrayItem</code> null handling for dynamic indices; <code>array_append</code> return type
+ fixed and marked <code>Compatible</code>; audited <code>array_insert</code> for correctness; <code>array_compact</code> marked <code>Compatible</code>;
+ array-to-array cast enabled.</li>
+<li><strong>DateTrunc/TimestampTrunc</strong>: Fixed native crashes when the input is a literal.</li>
+<li><strong>Ambiguous local times</strong>: Correct handling of ambiguous and non-existent local times across DST transitions.</li>
+<li><strong>Case-insensitive Parquet fields</strong>: <code>native_datafusion</code> now correctly detects duplicate/ambiguous fields in
+ case-insensitive mode and falls back where appropriate.</li>
+<li><strong>Shuffle planning</strong>: Shuffle fallback decisions are now "sticky" across planning passes, and Comet columnar
+ shuffle is skipped for stages containing DPP scans to avoid mismatched partitioning.</li>
+<li><strong>Error propagation</strong>: Native error messages are now propagated through <code>SparkException</code> even when the
+ <code>errorClass</code> is empty, and file-not-found errors flow through the standard Spark error JSON path.</li>
+<li><strong>Trigonometric compatibility</strong>: <code>tan</code> and <code>atan2</code> are now Spark-compatible.</li>
+</ul>
+<h2 id="dependency-upgrades">Dependency Upgrades<a class="headerlink" href="#dependency-upgrades" title="Permanent link">¶</a></h2>
+<p>This release upgrades to <strong>DataFusion 53.1</strong> and <strong>Arrow 58.1</strong>, and picks up the latest <code>iceberg-rust</code> release
+with additional reader fixes. The <code>jni</code> crate was upgraded to 0.22.4.</p>
+<h2 id="deprecations-and-removals">Deprecations and Removals<a class="headerlink" href="#deprecations-and-removals" title="Permanent link">¶</a></h2>
+<p>The <code>SupportsComet</code> interface has been removed, along with the Java-based Iceberg integration path (which is
+fully superseded by the native Iceberg reader). The <code>native_iceberg_compat</code> scan remains deprecated and is
+expected to be removed in a future release in favor of <code>native_datafusion</code>.</p>
+<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
+<p>Supported platforms include Spark 3.4.3, 3.5.4–3.5.8, and Spark 4.0.x with various JDK and Scala combinations.</p>
+<p>The community encourages users to test Comet with existing Spark and Iceberg workloads and welcomes contributions
+to ongoing development.</p>
+
+<!--
+ Comments Section
+ Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div>
+ <hr>
+ <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
+
+ <!-- Local loader script -->
+ <script src="/blog/js/giscus-consent.js" defer></script>
+
+ <!-- Consent UI -->
+ <div id="giscus-consent">
+ <p>
+ We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
+ To respect your privacy, Giscus and comments will load only if you click "Show Comments"
+ </p>
+
+ <div class="consent-actions">
+ <button id="giscus-load" type="button">Show Comments</button>
+ <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+ </div>
+
+ <noscript>JavaScript is required to load comments from Giscus.</noscript>
+ </div>
+
+ <!-- Container where Giscus will render -->
+ <div id="comment-thread"></div>
+</div> </div>
+ <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#key-features">Key Features</a><ul>
+<li><a href="#native-iceberg-reader-enabled-by-default">Native Iceberg Reader Enabled by Default</a></li>
+<li><a href="#auto-mode-enables-native_datafusion-scan">Auto Mode Enables native_datafusion Scan</a></li>
+<li><a href="#new-expressions-and-function-support">New Expressions and Function Support</a></li>
+<li><a href="#expanded-metrics-and-observability">Expanded Metrics and Observability</a></li>
+</ul>
+</li>
+<li><a href="#performance-improvements">Performance Improvements</a></li>
+<li><a href="#stability-and-correctness">Stability and Correctness</a></li>
+<li><a href="#dependency-upgrades">Dependency Upgrades</a></li>
+<li><a href="#deprecations-and-removals">Deprecations and Removals</a></li>
+<li><a href="#compatibility">Compatibility</a></li>
+</ul>
+</div>
+ </aside>
+ </div>
+ </div>
+</div>
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/author/aditya-singh-rathore-andrew-lamb.html b/blog/author/aditya-singh-rathore-andrew-lamb.html
index 7bda344..6a5a64b 100644
--- a/blog/author/aditya-singh-rathore-andrew-lamb.html
+++ b/blog/author/aditya-singh-rathore-andrew-lamb.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Aditya Singh Rathore, Andrew Lamb</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Aditya Singh Rathore, Andrew Lamb</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
-
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Aditya Singh Rathore, Andrew Lamb</h2>
-
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions" rel="bookmark" title="Permalink to User defined Window Functions in DataFusion">User defined Window Functions in DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-04-19T00:00:00+00:00"> Sat 19 April 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/aditya-singh-rathore-andrew-lamb.html">Aditya Singh Rathore, Andrew Lamb</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+ </head>
+
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Aditya Singh Rathore, Andrew Lamb</h2>
+
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions" rel="bookmark" title="Permalink to User defined Window Functions in DataFusion">User defined Window Functions in DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -48,15 +43,22 @@ limitations under the License.
-->
<p>Window functions are a powerful feature in SQL, allowing for complex analytical computations over a subset of data. However, efficiently implementing them, especially sliding windows, can be quite challenging. With <a href="https://datafusion.apache.org/">Apache DataFusion</a>'s user-defined window functions, developers can easily take advantage of all the effort put into DataFusion's implementation.</p>
-<p>In …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>In …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-04-19T00:00:00+00:00"> Sat 19 April 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/aditya-singh-rathore-andrew-lamb.html">Aditya Singh Rathore, Andrew Lamb</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.html b/blog/author/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.html
index 5861a86..3f050fb 100644
--- a/blog/author/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.html
+++ b/blog/author/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters" rel="bookmark" title="Permalink to Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries">Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-09-10T00:00:00+00:00"> Wed 10 September 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.html">Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters" rel="bookmark" title="Permalink to Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries">Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -58,15 +53,22 @@ optimizations work, and how we implemented them with the <a href="https://datafu
community to improve performance by an order of magnitude for some query
patterns.</p>
<h2 id="motivation-and-results">Motivation and Results<a class="headerlink" href="#motivation-and-results" title="Permanent link">¶</a></h2>
-<p>The …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>The …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-09-10T00:00:00+00:00"> Wed 10 September 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.html">Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/agrove.html b/blog/author/agrove.html
index ea4495c..638e6a7 100644
--- a/blog/author/agrove.html
+++ b/blog/author/agrove.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by agrove</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by agrove</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by agrove</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by agrove</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2021/04/12/ballista-donation" rel="bookmark" title="Permalink to Ballista: A Distributed Scheduler for Apache Arrow">Ballista: A Distributed Scheduler for Apache Arrow</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2021-04-12T00:00:00+00:00"> Mon 12 April 2021 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/agrove.html">agrove</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2021/04/12/ballista-donation" rel="bookmark" title="Permalink to Ballista: A Distributed Scheduler for Apache Arrow">Ballista: A Distributed Scheduler for Apache Arrow</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -50,17 +45,17 @@ limitations under the License.
<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
to the Apache Arrow project. </p>
<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="bookmark" title="Permalink to DataFusion: A Rust-native Query Engine for Apache Arrow">DataFusion: A Rust-native Query Engine for Apache Arrow</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2019-02-04T00:00:00+00:00"> Mon 04 February 2019 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/agrove.html">agrove</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></section>
+ <footer>
+ <p>Published: <time datetime="2021-04-12T00:00:00+00:00"> Mon 12 April 2021 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/agrove.html">agrove</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="bookmark" title="Permalink to DataFusion: A Rust-native Query Engine for Apache Arrow">DataFusion: A Rust-native Query Engine for Apache Arrow</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -80,15 +75,22 @@ limitations under the License.
-->
<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></section>
+ <footer>
+ <p>Published: <time datetime="2019-02-04T00:00:00+00:00"> Mon 04 February 2019 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/agrove.html">agrove</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/alamb-akurmustafa.html b/blog/author/alamb-akurmustafa.html
index 89575c9..4d0b69c 100644
--- a/blog/author/alamb-akurmustafa.html
+++ b/blog/author/alamb-akurmustafa.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by alamb, akurmustafa</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by alamb, akurmustafa</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by alamb, akurmustafa</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by alamb, akurmustafa</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one" rel="bookmark" title="Permalink to Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview">Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-06-15T00:00:00+00:00"> Sun 15 June 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/alamb-akurmustafa.html">alamb, akurmustafa</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one" rel="bookmark" title="Permalink to Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview">Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -55,17 +50,17 @@ science,”</a> according to Father
Pavlo, or some behind-the-scenes player. We believe this perception is because:</p>
<ol>
<li>
-<p>One must implement the rest of a …</p></li></ol> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="bookmark" title="Permalink to Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion">Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-06-15T00:00:00+00:00"> Sun 15 June 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/alamb-akurmustafa.html">alamb, akurmustafa</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content">
+<p>One must implement the rest of a …</p></li></ol></section>
+ <footer>
+ <p>Published: <time datetime="2025-06-15T00:00:00+00:00"> Sun 15 June 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/alamb-akurmustafa.html">alamb, akurmustafa</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="bookmark" title="Permalink to Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion">Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</a></h2> </header>
+ <section>
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -88,15 +83,22 @@ limitations under the License.
<p>In the <a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one">first part of this post</a>, we discussed what a Query Optimizer is, what
role it plays, and described how industrial optimizers are organized. In this
second post, we describe various optimizations that are found in <a href="https://datafusion.apache.org/">Apache
-DataFusion</a> and …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+DataFusion</a> and …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-06-15T00:00:00+00:00"> Sun 15 June 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/alamb-akurmustafa.html">alamb, akurmustafa</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/alamb-dandandan-tustvold.html b/blog/author/alamb-dandandan-tustvold.html
index 2b5e9f7..99785d5 100644
--- a/blog/author/alamb-dandandan-tustvold.html
+++ b/blog/author/alamb-dandandan-tustvold.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by alamb, Dandandan, tustvold</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by alamb, Dandandan, tustvold</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by alamb, Dandandan, tustvold</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by alamb, Dandandan, tustvold</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping" rel="bookmark" title="Permalink to Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0">Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2023-08-05T00:00:00+00:00"> Sat 05 August 2023 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/alamb-dandandan-tustvold.html">alamb, Dandandan, tustvold</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping" rel="bookmark" title="Permalink to Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0">Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -52,15 +47,22 @@ limitations under the License.
<p>Andrew Lamb, Daniël Heres, Raphael Taylor-Davies,</p>
<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
<h2 id="tldr">TLDR<a class="headerlink" href="#tldr" title="Permanent link">¶</a></h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p></section>
+ <footer>
+ <p>Published: <time datetime="2023-08-05T00:00:00+00:00"> Sat 05 August 2023 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/alamb-dandandan-tustvold.html">alamb, Dandandan, tustvold</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/andrew-lamb-achraf-b-and-sean-smith.html b/blog/author/andrew-lamb-achraf-b-and-sean-smith.html
index 01e538e..c2e703d 100644
--- a/blog/author/andrew-lamb-achraf-b-and-sean-smith.html
+++ b/blog/author/andrew-lamb-achraf-b-and-sean-smith.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Andrew Lamb, Achraf B, and Sean Smith</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Andrew Lamb, Achraf B, and Sean Smith</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Andrew Lamb, Achraf B, and Sean Smith</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Andrew Lamb, Achraf B, and Sean Smith</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/04/10/fastest-tpch-generator" rel="bookmark" title="Permalink to tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust">tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-04-10T00:00:00+00:00"> Thu 10 April 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/andrew-lamb-achraf-b-and-sean-smith.html">Andrew Lamb, Achraf B, and Sean Smith</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/04/10/fastest-tpch-generator" rel="bookmark" title="Permalink to tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust">tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -45,27 +40,26 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
-development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
+faster than any other implementation we know of.</p>
+<p>It is now possible to create …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-04-10T00:00:00+00:00"> Thu 10 April 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/andrew-lamb-achraf-b-and-sean-smith.html">Andrew Lamb, Achraf B, and Sean Smith</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/andrew-lamb-influxdata.html b/blog/author/andrew-lamb-influxdata.html
index 7302b32..58c4ab6 100644
--- a/blog/author/andrew-lamb-influxdata.html
+++ b/blog/author/andrew-lamb-influxdata.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Andrew Lamb (InfluxData)</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Andrew Lamb (InfluxData)</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Andrew Lamb (InfluxData)</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Andrew Lamb (InfluxData)</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes" rel="bookmark" title="Permalink to Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet">Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-08-15T00:00:00+00:00"> Fri 15 August 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/andrew-lamb-influxdata.html">Andrew Lamb (InfluxData)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes" rel="bookmark" title="Permalink to Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet">Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -53,15 +48,22 @@ metadata and is limited to indexing structures provided by the format. In fact,
caching parsed metadata and using custom external indexes along with
Parquet's hierarchical data organization can significantly speed up query
processing.</p>
-<p>In this blog, I describe …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>In this blog, I describe …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-08-15T00:00:00+00:00"> Fri 15 August 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/andrew-lamb-influxdata.html">Andrew Lamb (InfluxData)</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/andrew-lamb-staff-engineer-at-influxdata.html b/blog/author/andrew-lamb-staff-engineer-at-influxdata.html
index 3688e41..28a8dc8 100644
--- a/blog/author/andrew-lamb-staff-engineer-at-influxdata.html
+++ b/blog/author/andrew-lamb-staff-engineer-at-influxdata.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Andrew Lamb, Staff Engineer at InfluxData</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Andrew Lamb, Staff Engineer at InfluxData</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Andrew Lamb, Staff Engineer at InfluxData</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Andrew Lamb, Staff Engineer at InfluxData</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" rel="bookmark" title="Permalink to Apache DataFusion is now the fastest single node engine for querying Apache Parquet files">Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-11-18T00:00:00+00:00"> Mon 18 November 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/andrew-lamb-staff-engineer-at-influxdata.html">Andrew Lamb, Staff Engineer at InfluxData</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" rel="bookmark" title="Permalink to Apache DataFusion is now the fastest single node engine for querying Apache Parquet files">Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -51,15 +46,22 @@ limitations under the License.
fastest engine for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">ClickBench</a>. It is faster
than <a href="https://duckdb.org/">DuckDB</a>, <a href="https://clickhouse.com/chdb">chDB</a> and <a href="https://clickhouse.com/">Clickhouse</a> using the same hardware. It also marks
the first time a <a href="https://www.rust-lang.org/">Rust</a>-based engine holds the top spot, which has previously
-been …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+been …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-11-18T00:00:00+00:00"> Mon 18 November 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/andrew-lamb-staff-engineer-at-influxdata.html">Andrew Lamb, Staff Engineer at InfluxData</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/gene-bordegaray.html b/blog/author/gene-bordegaray.html
index b1ac65e..08011bb 100644
--- a/blog/author/gene-bordegaray.html
+++ b/blog/author/gene-bordegaray.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Gene Bordegaray</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Gene Bordegaray</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
-
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Gene Bordegaray</h2>
-
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="bookmark" title="Permalink to Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution">Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-12-15T00:00:00+00:00"> Mon 15 December 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/gene-bordegaray.html">Gene Bordegaray</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+ </head>
+
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Gene Bordegaray</h2>
+
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="bookmark" title="Permalink to Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution">Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -46,18 +41,25 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
-
-Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
+
+Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div></section>
+ <footer>
+ <p>Published: <time datetime="2025-12-15T00:00:00+00:00"> Mon 15 December 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/gene-bordegaray.html">Gene Bordegaray</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/geoffrey-claude-datadog.html b/blog/author/geoffrey-claude-datadog.html
index f6df445..c62b73a 100644
--- a/blog/author/geoffrey-claude-datadog.html
+++ b/blog/author/geoffrey-claude-datadog.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Geoffrey Claude (Datadog)</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Geoffrey Claude (Datadog)</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
-
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Geoffrey Claude (Datadog)</h2>
-
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="bookmark" title="Permalink to Extending SQL in DataFusion: from ->> to TABLESAMPLE">Extending SQL in DataFusion: from ->> to TABLESAMPLE</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/geoffrey-claude-datadog.html">Geoffrey Claude (Datadog)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+ </head>
+
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Geoffrey Claude (Datadog)</h2>
+
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="bookmark" title="Permalink to Extending SQL in DataFusion: from ->> to TABLESAMPLE">Extending SQL in DataFusion: from ->> to TABLESAMPLE</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -48,15 +43,22 @@ limitations under the License.
-->
<p>If you embed <a href="https://datafusion.apache.org/">DataFusion</a> in your product, your users will eventually run SQL that DataFusion does not recognize. Not because the query is unreasonable, but because SQL in practice includes many dialects and system-specific statements.</p>
-<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/geoffrey-claude-datadog.html">Geoffrey Claude (Datadog)</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/milenkovicm.html b/blog/author/milenkovicm.html
index 968154a..72ae6bb 100644
--- a/blog/author/milenkovicm.html
+++ b/blog/author/milenkovicm.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by milenkovicm</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by milenkovicm</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by milenkovicm</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by milenkovicm</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0" rel="bookmark" title="Permalink to Apache DataFusion Ballista 43.0.0 Released">Apache DataFusion Ballista 43.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-02-02T00:00:00+00:00"> Sun 02 February 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/milenkovicm.html">milenkovicm</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0" rel="bookmark" title="Permalink to Apache DataFusion Ballista 43.0.0 Released">Apache DataFusion Ballista 43.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -50,15 +45,22 @@ limitations under the License.
<p>We are pleased to announce version <a href="https://github.com/apache/datafusion-ballista/blob/main/CHANGELOG.md#4300-2025-01-07">43.0.0</a> of the <a href="https://datafusion.apache.org/ballista/">DataFusion Ballista</a>. Ballista allows existing <a href="https://datafusion.apache.org">DataFusion</a> applications to be scaled out on a cluster for use cases that are not practical to run on a single node.</p>
<h2 id="highlights-of-this-release">Highlights of this release<a class="headerlink" href="#highlights-of-this-release" title="Permanent link">¶</a></h2>
<h3 id="seamless-integration-with-datafusion">Seamless Integration with DataFusion<a class="headerlink" href="#seamless-integration-with-datafusion" title="Permanent link">¶</a></h3>
-<p>The primary objective of …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>The primary objective of …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-02-02T00:00:00+00:00"> Sun 02 February 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/milenkovicm.html">milenkovicm</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/mustafa-akur-andrew-lamb.html b/blog/author/mustafa-akur-andrew-lamb.html
index 8f7abce..5302e0c 100644
--- a/blog/author/mustafa-akur-andrew-lamb.html
+++ b/blog/author/mustafa-akur-andrew-lamb.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Mustafa Akur, Andrew Lamb</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Mustafa Akur, Andrew Lamb</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Mustafa Akur, Andrew Lamb</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Mustafa Akur, Andrew Lamb</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="bookmark" title="Permalink to Using Ordering for Better Plans in Apache DataFusion">Using Ordering for Better Plans in Apache DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-11T00:00:00+00:00"> Tue 11 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/mustafa-akur-andrew-lamb.html">Mustafa Akur, Andrew Lamb</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="bookmark" title="Permalink to Using Ordering for Better Plans in Apache DataFusion">Using Ordering for Better Plans in Apache DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -50,16 +45,23 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
-<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-11T00:00:00+00:00"> Tue 11 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/mustafa-akur-andrew-lamb.html">Mustafa Akur, Andrew Lamb</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html b/blog/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html
index 87bc9be..5bb10e3 100644
--- a/blog/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html
+++ b/blog/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Oznur Hanci and Berkay Sahin on behalf of the PMC</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Oznur Hanci and Berkay Sahin on behalf of the PMC</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Oznur Hanci and Berkay Sahin on behalf of the PMC</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Oznur Hanci and Berkay Sahin on behalf of the PMC</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0" rel="bookmark" title="Permalink to Apache DataFusion 46.0.0 Released">Apache DataFusion 46.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-24T00:00:00+00:00"> Mon 24 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html">Oznur Hanci and Berkay Sahin on behalf of the PMC</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0" rel="bookmark" title="Permalink to Apache DataFusion 46.0.0 Released">Apache DataFusion 46.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -45,15 +40,22 @@ limitations under the License.
{% endcomment %}
-->
-<p>We’re excited to announce the release of <strong>Apache DataFusion 46.0.0</strong>! This new version represents a significant milestone for the project, packing in a wide range of improvements and fixes. You can find the complete details in the full <a href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>. We’ll highlight the most important changes below …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>We’re excited to announce the release of <strong>Apache DataFusion 46.0.0</strong>! This new version represents a significant milestone for the project, packing in a wide range of improvements and fixes. You can find the complete details in the full <a href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>. We’ll highlight the most important changes below …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-24T00:00:00+00:00"> Mon 24 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html">Oznur Hanci and Berkay Sahin on behalf of the PMC</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/pepijn-van-eeckhoudt.html b/blog/author/pepijn-van-eeckhoudt.html
index 25789ed..030036b 100644
--- a/blog/author/pepijn-van-eeckhoudt.html
+++ b/blog/author/pepijn-van-eeckhoudt.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Pepijn Van Eeckhoudt</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Pepijn Van Eeckhoudt</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Pepijn Van Eeckhoudt</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Pepijn Van Eeckhoudt</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="bookmark" title="Permalink to Optimizing SQL CASE Expression Evaluation">Optimizing SQL CASE Expression Evaluation</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-02-02T00:00:00+00:00"> Mon 02 February 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="bookmark" title="Permalink to Optimizing SQL CASE Expression Evaluation">Optimizing SQL CASE Expression Evaluation</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -68,17 +63,17 @@ figcaption {
}
</style>
<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
-It allows you to control which expression from a …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="bookmark" title="Permalink to Using Rust async for Query Execution and Cancelling Long-Running Queries">Using Rust async for Query Execution and Cancelling Long-Running Queries</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-06-30T00:00:00+00:00"> Mon 30 June 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+It allows you to control which expression from a …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-02-02T00:00:00+00:00"> Mon 02 February 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="bookmark" title="Permalink to Using Rust async for Query Execution and Cancelling Long-Running Queries">Using Rust async for Query Execution and Cancelling Long-Running Queries</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -118,15 +113,22 @@ figcaption {
}
</style>
<p>Have you ever tried to cancel a query that just wouldn't stop?
-In this post, we'll review how Rust's <a href="https://doc.rust-lang.org/book/ch17-00-async-await.html"><code>async</code> programming model</a> works, how …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+In this post, we'll review how Rust's <a href="https://doc.rust-lang.org/book/ch17-00-async-await.html"><code>async</code> programming model</a> works, how …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-06-30T00:00:00+00:00"> Mon 30 June 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/pmc.html b/blog/author/pmc.html
index cb10f9b..41c6e6f 100644
--- a/blog/author/pmc.html
+++ b/blog/author/pmc.html
@@ -1,34 +1,96 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by pmc</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by pmc</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
-
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by pmc</h2>
-
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/18/datafusion-comet-0.14.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.14.0 Release">Apache DataFusion Comet 0.14.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-03-18T00:00:00+00:00"> Wed 18 March 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+ </head>
+
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by pmc</h2>
+
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/04/18/datafusion-comet-0.15.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.15.0 Release">Apache DataFusion Comet 0.15.0 Release</a></h2> </header>
+ <section><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.15.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately four weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-04-18T00:00:00+00:00"> Sat 18 April 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/04/02/datafusion-53.0.0" rel="bookmark" title="Permalink to Apache DataFusion 53.0.0 Released">Apache DataFusion 53.0.0 Released</a></h2> </header>
+ <section><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/53.0.0">DataFusion 53.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0/">DataFusion 52.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md#credits">114 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p><img alt="Performance over time" class="img-fluid" src="/blog/images/datafusion-53.0.0/performance_over_time_clickbench.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: Average …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-04-02T00:00:00+00:00"> Thu 02 April 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/03/18/datafusion-comet-0.14.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.14.0 Release">Apache DataFusion Comet 0.14.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -50,17 +112,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.14.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately eight weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.13.0 Release">Apache DataFusion Comet 0.13.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-01-30T00:00:00+00:00"> Fri 30 January 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately eight weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-03-18T00:00:00+00:00"> Wed 18 March 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.13.0 Release">Apache DataFusion Comet 0.13.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -82,17 +144,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately eight weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="bookmark" title="Permalink to Apache DataFusion 52.0.0 Released">Apache DataFusion 52.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately eight weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-01-30T00:00:00+00:00"> Fri 30 January 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="bookmark" title="Permalink to Apache DataFusion 52.0.0 Released">Apache DataFusion 52.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -116,17 +178,17 @@ some of the major improvements since <a href="https://datafusion.apache.org/blog
changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
making this release possible.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/12/04/datafusion-comet-0.12.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.12.0 Release">Apache DataFusion Comet 0.12.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-12-04T00:00:00+00:00"> Thu 04 December 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>We continue to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/12/04/datafusion-comet-0.12.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.12.0 Release">Apache DataFusion Comet 0.12.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -148,17 +210,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.12.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately four weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0" rel="bookmark" title="Permalink to Apache DataFusion 51.0.0 Released">Apache DataFusion 51.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-11-25T00:00:00+00:00"> Tue 25 November 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately four weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-12-04T00:00:00+00:00"> Thu 04 December 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0" rel="bookmark" title="Permalink to Apache DataFusion 51.0.0 Released">Apache DataFusion 51.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -183,17 +245,17 @@ some of the major improvements since <a href="https://datafusion.apache.org/blog
changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md#credits">128 contributors</a> for
making this release possible.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.11.0 Release">Apache DataFusion Comet 0.11.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-10-21T00:00:00+00:00"> Tue 21 October 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>We continue …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-11-25T00:00:00+00:00"> Tue 25 November 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.11.0 Release">Apache DataFusion Comet 0.11.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -215,17 +277,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately five weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0" rel="bookmark" title="Permalink to Apache DataFusion 50.0.0 Released">Apache DataFusion 50.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-09-29T00:00:00+00:00"> Mon 29 September 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately five weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-10-21T00:00:00+00:00"> Tue 21 October 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0" rel="bookmark" title="Permalink to Apache DataFusion 50.0.0 Released">Apache DataFusion 50.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -250,17 +312,17 @@ limitations under the License.
highlights some of the major improvements since the release of <a href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0/">DataFusion
49.0.0</a>. The complete list of changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-50/dev/changelog/50.0.0.md">changelog</a>.
Thanks to <a href="https://github.com/apache/datafusion/blob/branch-50/dev/changelog/50.0.0.md#credits">numerous contributors</a> for making this release possible!</p>
-<h2 id="performance-improvements">Performance …</h2> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/09/16/datafusion-comet-0.10.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.10.0 Release">Apache DataFusion Comet 0.10.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-09-16T00:00:00+00:00"> Tue 16 September 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<h2 id="performance-improvements">Performance …</h2></section>
+ <footer>
+ <p>Published: <time datetime="2025-09-29T00:00:00+00:00"> Mon 29 September 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/09/16/datafusion-comet-0.10.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.10.0 Release">Apache DataFusion Comet 0.10.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -282,17 +344,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.10.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately ten weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0" rel="bookmark" title="Permalink to Apache DataFusion 49.0.0 Released">Apache DataFusion 49.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-28T00:00:00+00:00"> Mon 28 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately ten weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-09-16T00:00:00+00:00"> Tue 16 September 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0" rel="bookmark" title="Permalink to Apache DataFusion 49.0.0 Released">Apache DataFusion 49.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -316,17 +378,17 @@ limitations under the License.
<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/49.0.0">DataFusion 49.0.0</a>. This blog post highlights some of
the major improvements since the release of <a href="https://datafusion.apache.org/blog/2025/07/18/datafusion-48.0.0/">DataFusion 48.0.0</a>. The complete list of changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-49/dev/changelog/49.0.0.md">changelog</a>.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>DataFusion continues to focus on enhancing performance, as …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0" rel="bookmark" title="Permalink to Apache DataFusion 48.0.0 Released">Apache DataFusion 48.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-16T00:00:00+00:00"> Wed 16 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">PMC</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>DataFusion continues to focus on enhancing performance, as …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-28T00:00:00+00:00"> Mon 28 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0" rel="bookmark" title="Permalink to Apache DataFusion 48.0.0 Released">Apache DataFusion 48.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -348,17 +410,17 @@ limitations under the License.
improvements and fixes. You can find the complete details in the full
<a href="https://github.com/apache/datafusion/blob/branch-48/dev/changelog/48.0.0.md">changelog</a>. We’ll highlight the most
important changes below and guide you through upgrading.</p>
-<h2 id="breaking-changes">Breaking …</h2> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/11/datafusion-47.0.0" rel="bookmark" title="Permalink to Apache DataFusion 47.0.0 Released">Apache DataFusion 47.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-11T00:00:00+00:00"> Fri 11 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">PMC</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<h2 id="breaking-changes">Breaking …</h2></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-16T00:00:00+00:00"> Wed 16 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">PMC</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/11/datafusion-47.0.0" rel="bookmark" title="Permalink to Apache DataFusion 47.0.0 Released">Apache DataFusion 47.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -379,17 +441,17 @@ limitations under the License.
<p>We’re excited to announce the release of <strong>Apache DataFusion 47.0.0</strong>! This new version represents a significant
milestone for the project, packing in a wide range of improvements and fixes. You can find the complete details in the
full <a href="https://github.com/apache/datafusion/blob/branch-47/dev/changelog/47.0.0.md">changelog</a>. We’ll highlight the most
-important changes below …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.9.0 Release">Apache DataFusion Comet 0.9.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-01T00:00:00+00:00"> Tue 01 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+important changes below …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-11T00:00:00+00:00"> Fri 11 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">PMC</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.9.0 Release">Apache DataFusion Comet 0.9.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -411,17 +473,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.9.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately ten weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.8.0 Release">Apache DataFusion Comet 0.8.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-05-06T00:00:00+00:00"> Tue 06 May 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately ten weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-01T00:00:00+00:00"> Tue 01 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.8.0 Release">Apache DataFusion Comet 0.8.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -443,17 +505,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.8.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately six weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.7.0 Release">Apache DataFusion Comet 0.7.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-20T00:00:00+00:00"> Thu 20 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately six weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-05-06T00:00:00+00:00"> Tue 06 May 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.7.0 Release">Apache DataFusion Comet 0.7.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -475,17 +537,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/02/20/datafusion-45.0.0" rel="bookmark" title="Permalink to Apache DataFusion 45.0.0 Released">Apache DataFusion 45.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-02-20T00:00:00+00:00"> Thu 20 February 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-20T00:00:00+00:00"> Thu 20 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/02/20/datafusion-45.0.0" rel="bookmark" title="Permalink to Apache DataFusion 45.0.0 Released">Apache DataFusion 45.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -509,17 +571,17 @@ limitations under the License.
<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
what the community is thinking about in the next 6 months. It has been an exciting
-period of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.6.0 Release">Apache DataFusion Comet 0.6.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-02-17T00:00:00+00:00"> Mon 17 February 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+period of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-02-20T00:00:00+00:00"> Thu 20 February 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.6.0 Release">Apache DataFusion Comet 0.6.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -541,17 +603,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.6.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/01/17/datafusion-comet-0.5.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.5.0 Release">Apache DataFusion Comet 0.5.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-01-17T00:00:00+00:00"> Fri 17 January 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-02-17T00:00:00+00:00"> Mon 17 February 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/01/17/datafusion-comet-0.5.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.5.0 Release">Apache DataFusion Comet 0.5.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -573,17 +635,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.5.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.4.0 Release">Apache DataFusion Comet 0.4.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-11-20T00:00:00+00:00"> Wed 20 November 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-01-17T00:00:00+00:00"> Fri 17 January 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.4.0 Release">Apache DataFusion Comet 0.4.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -605,17 +667,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.3.0 Release">Apache DataFusion Comet 0.3.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-09-27T00:00:00+00:00"> Fri 27 September 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-11-20T00:00:00+00:00"> Wed 20 November 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.3.0 Release">Apache DataFusion Comet 0.3.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -637,17 +699,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.3.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.2.0 Release">Apache DataFusion Comet 0.2.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-08-28T00:00:00+00:00"> Wed 28 August 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-09-27T00:00:00+00:00"> Fri 27 September 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.2.0 Release">Apache DataFusion Comet 0.2.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -669,17 +731,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.2.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0" rel="bookmark" title="Permalink to Apache DataFusion 40.0.0 Released">Apache DataFusion 40.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-07-24T00:00:00+00:00"> Wed 24 July 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-08-28T00:00:00+00:00"> Wed 28 August 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0" rel="bookmark" title="Permalink to Apache DataFusion 40.0.0 Released">Apache DataFusion 40.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -703,17 +765,17 @@ limitations under the License.
<p>We are proud to announce <a href="https://crates.io/crates/datafusion/40.0.0">DataFusion 40.0.0</a>. This blog highlights some of the
many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0/">DataFusion 34.0.0</a> and a preview of
what the community is thinking about in the next 6 months. We are hoping to make
-more regular blog posts …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.1.0 Release">Apache DataFusion Comet 0.1.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-07-20T00:00:00+00:00"> Sat 20 July 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+more regular blog posts …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-07-24T00:00:00+00:00"> Wed 24 July 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.1.0 Release">Apache DataFusion Comet 0.1.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -735,17 +797,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce the first official source release of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp" rel="bookmark" title="Permalink to Announcing Apache Arrow DataFusion is now Apache DataFusion">Announcing Apache Arrow DataFusion is now Apache DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-05-07T00:00:00+00:00"> Tue 07 May 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-07-20T00:00:00+00:00"> Sat 20 July 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp" rel="bookmark" title="Permalink to Announcing Apache Arrow DataFusion is now Apache DataFusion">Announcing Apache Arrow DataFusion is now Apache DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -770,17 +832,17 @@ limitations under the License.
April 16, 2024 the Apache Arrow DataFusion subproject is now a top level
<a href="https://www.apache.org/">Apache Software Foundation</a> project.</p>
<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h2>
-<p>Apache DataFusion is a fast, extensible query engine for building …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/03/06/comet-donation" rel="bookmark" title="Permalink to Announcing Apache Arrow DataFusion Comet">Announcing Apache Arrow DataFusion Comet</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-03-06T00:00:00+00:00"> Wed 06 March 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Apache DataFusion is a fast, extensible query engine for building …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-05-07T00:00:00+00:00"> Tue 07 May 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/03/06/comet-donation" rel="bookmark" title="Permalink to Announcing Apache Arrow DataFusion Comet">Announcing Apache Arrow DataFusion Comet</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -804,17 +866,17 @@ limitations under the License.
a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/datafusion">Apache Arrow DataFusion</a>.</p>
<p>Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to
accelerate Spark workloads. It is designed as a drop-in
-replacement for Spark's JVM …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024">Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-01-19T00:00:00+00:00"> Fri 19 January 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+replacement for Spark's JVM …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-03-06T00:00:00+00:00"> Wed 06 March 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024">Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -837,17 +899,17 @@ limitations under the License.
<p>We recently <a href="https://crates.io/crates/datafusion/34.0.0">released DataFusion 34.0.0</a>. This blog highlights some of the major
improvements since we <a href="https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/.">released DataFusion 26.0.0</a> (spoiler alert there are many)
and a preview of where the community plans to focus in the next 6 months.</p>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2023/06/24/datafusion-25.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 26.0.0">Apache Arrow DataFusion 26.0.0</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2023-06-24T00:00:00+00:00"> Sat 24 June 2023 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-01-19T00:00:00+00:00"> Fri 19 January 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2023/06/24/datafusion-25.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 26.0.0">Apache Arrow DataFusion 26.0.0</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -872,17 +934,17 @@ performance improved and we are <a href="https://github.com/apache/arrow-datafus
top level Apache Project.</p>
<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h2>
<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database
-toolkit …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 16.0.0 Project Update">Apache Arrow DataFusion 16.0.0 Project Update</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2023-01-19T00:00:00+00:00"> Thu 19 January 2023 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+toolkit …</p></section>
+ <footer>
+ <p>Published: <time datetime="2023-06-24T00:00:00+00:00"> Sat 24 June 2023 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 16.0.0 Project Update">Apache Arrow DataFusion 16.0.0 Project Update</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -909,17 +971,17 @@ in-memory format. It is targeted primarily at developers creating data
intensive analytics, and offers mature
<a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>,
a DataFrame API, and many extension points.</p>
-<p>Systems based on DataFusion perform very well in benchmarks …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0" rel="bookmark" title="Permalink to Apache Arrow Ballista 0.9.0 Release">Apache Arrow Ballista 0.9.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-10-28T00:00:00+00:00"> Fri 28 October 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Systems based on DataFusion perform very well in benchmarks …</p></section>
+ <footer>
+ <p>Published: <time datetime="2023-01-19T00:00:00+00:00"> Thu 19 January 2023 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0" rel="bookmark" title="Permalink to Apache Arrow Ballista 0.9.0 Release">Apache Arrow Ballista 0.9.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -943,17 +1005,17 @@ limitations under the License.
<p>Ballista 0.9.0 is now available and is the most significant release since the project was <a href="http://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to Apache
Arrow in 2021.</p>
<p>This release represents 4 weeks of work, with 66 commits from 14 contributors:</p>
-<pre><code> 22 Andy …</code></pre> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 13.0.0 Project Update">Apache Arrow DataFusion 13.0.0 Project Update</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-10-25T00:00:00+00:00"> Tue 25 October 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<pre><code> 22 Andy …</code></pre></section>
+ <footer>
+ <p>Published: <time datetime="2022-10-28T00:00:00+00:00"> Fri 28 October 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 13.0.0 Project Update">Apache Arrow DataFusion 13.0.0 Project Update</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -974,17 +1036,17 @@ limitations under the License.
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> <a href="https://crates.io/crates/datafusion"><code>13.0.0</code></a> is released, and this blog contains an update on the project for the 5 months since our <a href="https://arrow.apache.org/blog/2022/05/16/datafusion-8.0.0/">last update in May 2022</a>.</p>
-<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/05/16/datafusion-8.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 8.0.0 Release">Apache Arrow DataFusion 8.0.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-05-16T00:00:00+00:00"> Mon 16 May 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL …</p></section>
+ <footer>
+ <p>Published: <time datetime="2022-10-25T00:00:00+00:00"> Tue 25 October 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/05/16/datafusion-8.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 8.0.0 Release">Apache Arrow DataFusion 8.0.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1007,17 +1069,17 @@ limitations under the License.
<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that
uses Apache Arrow as its in-memory format.</p>
<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>,
-a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/03/21/datafusion-contrib" rel="bookmark" title="Permalink to Introducing Apache Arrow DataFusion Contrib">Introducing Apache Arrow DataFusion Contrib</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-03-21T00:00:00+00:00"> Mon 21 March 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></section>
+ <footer>
+ <p>Published: <time datetime="2022-05-16T00:00:00+00:00"> Mon 16 May 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/03/21/datafusion-contrib" rel="bookmark" title="Permalink to Introducing Apache Arrow DataFusion Contrib">Introducing Apache Arrow DataFusion Contrib</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1037,17 +1099,17 @@ limitations under the License.
-->
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
<p>Apache Arrow <a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/02/28/datafusion-7.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 7.0.0 Release">Apache Arrow DataFusion 7.0.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-02-28T00:00:00+00:00"> Mon 28 February 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is …</p></section>
+ <footer>
+ <p>Published: <time datetime="2022-03-21T00:00:00+00:00"> Mon 21 March 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/02/28/datafusion-7.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 7.0.0 Release">Apache Arrow DataFusion 7.0.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1067,17 +1129,17 @@ limitations under the License.
-->
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" rel="bookmark" title="Permalink to Apache Arrow DataFusion 6.0.0 Release">Apache Arrow DataFusion 6.0.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2021-11-19T00:00:00+00:00"> Fri 19 November 2021 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></section>
+ <footer>
+ <p>Published: <time datetime="2022-02-28T00:00:00+00:00"> Mon 28 February 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" rel="bookmark" title="Permalink to Apache Arrow DataFusion 6.0.0 Release">Apache Arrow DataFusion 6.0.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1101,17 +1163,17 @@ query engine which leverages the unique features of
<a href="https://www.rust-lang.org/">Rust</a> and <a href="https://arrow.apache.org/">Apache
Arrow</a> to provide a system that is high
performance, easy to connect, easy to embed, and high quality.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2021/08/18/ballista-0.5.0" rel="bookmark" title="Permalink to Apache Arrow Ballista 0.5.0 Release">Apache Arrow Ballista 0.5.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2021-08-18T00:00:00+00:00"> Wed 18 August 2021 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers …</p></section>
+ <footer>
+ <p>Published: <time datetime="2021-11-19T00:00:00+00:00"> Fri 19 November 2021 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2021/08/18/ballista-0.5.0" rel="bookmark" title="Permalink to Apache Arrow Ballista 0.5.0 Release">Apache Arrow Ballista 0.5.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1133,17 +1195,17 @@ limitations under the License.
<p>Ballista extends DataFusion to provide support for distributed queries. This is the first release of Ballista since
the project was <a href="https://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to the Apache Arrow project
and includes 80 commits from 11 contributors.</p>
-<pre><code>git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust …</code></pre> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2021/08/18/datafusion-5.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 5.0.0 Release">Apache Arrow DataFusion 5.0.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2021-08-18T00:00:00+00:00"> Wed 18 August 2021 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<pre><code>git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust …</code></pre></section>
+ <footer>
+ <p>Published: <time datetime="2021-08-18T00:00:00+00:00"> Wed 18 August 2021 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2021/08/18/datafusion-5.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 5.0.0 Release">Apache Arrow DataFusion 5.0.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1166,15 +1228,22 @@ and includes 211 commits from the following 31 distinct contributors.</p>
<pre><code>$ git shortlog -sn 4.0.0..5.0.0 datafusion datafusion-cli datafusion-examples
61 Jiayu Liu
47 Andrew Lamb
- 27 …</code></pre> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+ 27 …</code></pre></section>
+ <footer>
+ <p>Published: <time datetime="2021-08-18T00:00:00+00:00"> Wed 18 August 2021 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/qi-zhu-cloudera-jigao-luo-systems-group-at-tu-darmstadt-and-andrew-lamb-influxdata.html b/blog/author/qi-zhu-cloudera-jigao-luo-systems-group-at-tu-darmstadt-and-andrew-lamb-influxdata.html
index 032e9f4..9865b16 100644
--- a/blog/author/qi-zhu-cloudera-jigao-luo-systems-group-at-tu-darmstadt-and-andrew-lamb-influxdata.html
+++ b/blog/author/qi-zhu-cloudera-jigao-luo-systems-group-at-tu-darmstadt-and-andrew-lamb-influxdata.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
-
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</h2>
-
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes" rel="bookmark" title="Permalink to Embedding User-Defined Indexes in Apache Parquet Files">Embedding User-Defined Indexes in Apache Parquet Files</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-14T00:00:00+00:00"> Mon 14 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/qi-zhu-cloudera-jigao-luo-systems-group-at-tu-darmstadt-and-andrew-lamb-influxdata.html">Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+ </head>
+
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</h2>
+
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes" rel="bookmark" title="Permalink to Embedding User-Defined Indexes in Apache Parquet Files">Embedding User-Defined Indexes in Apache Parquet Files</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -47,15 +42,22 @@ limitations under the License.
{% endcomment %}
-->
-<p>It’s a common misconception that <a href="https://parquet.apache.org/">Apache Parquet</a> files are limited to basic Min/Max/Null Count statistics and Bloom filters, and that adding more advanced indexes requires changing the specification or creating a new file format. In fact, footer metadata and offset-based addressing already provide everything needed to embed …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>It’s a common misconception that <a href="https://parquet.apache.org/">Apache Parquet</a> files are limited to basic Min/Max/Null Count statistics and Bloom filters, and that adding more advanced indexes requires changing the specification or creating a new file format. In fact, footer metadata and offset-based addressing already provide everything needed to embed …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-14T00:00:00+00:00"> Mon 14 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/qi-zhu-cloudera-jigao-luo-systems-group-at-tu-darmstadt-and-andrew-lamb-influxdata.html">Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/tim-saucer-rerunio.html b/blog/author/tim-saucer-rerunio.html
index ddfd065..bd51d23 100644
--- a/blog/author/tim-saucer-rerunio.html
+++ b/blog/author/tim-saucer-rerunio.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Tim Saucer (rerun.io)</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Tim Saucer (rerun.io)</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Tim Saucer (rerun.io)</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Tim Saucer (rerun.io)</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/31/writing-table-providers" rel="bookmark" title="Permalink to Writing Custom Table Providers in Apache DataFusion">Writing Custom Table Providers in Apache DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-03-31T00:00:00+00:00"> Tue 31 March 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/tim-saucer-rerunio.html">Tim Saucer (rerun.io)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/03/31/writing-table-providers" rel="bookmark" title="Permalink to Writing Custom Table Providers in Apache DataFusion">Writing Custom Table Providers in Apache DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -50,15 +45,22 @@ limitations under the License.
<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
in a custom format, behind an API, or in a system that DataFusion does not
natively support, you can teach DataFusion to read it by implementing a
-<strong>custom table provider</strong>. This post walks through the three layers you …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<strong>custom table provider</strong>. This post walks through the three layers you …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-03-31T00:00:00+00:00"> Tue 31 March 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/tim-saucer-rerunio.html">Tim Saucer (rerun.io)</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/tim-saucerrerunio-dewey-dunningtonwherobots-andrew-lambinfluxdata.html b/blog/author/tim-saucerrerunio-dewey-dunningtonwherobots-andrew-lambinfluxdata.html
index e03bb2b..c785c49 100644
--- a/blog/author/tim-saucerrerunio-dewey-dunningtonwherobots-andrew-lambinfluxdata.html
+++ b/blog/author/tim-saucerrerunio-dewey-dunningtonwherobots-andrew-lambinfluxdata.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata" rel="bookmark" title="Permalink to Implementing User Defined Types and Custom Metadata in DataFusion">Implementing User Defined Types and Custom Metadata in DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-09-21T00:00:00+00:00"> Sun 21 September 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/tim-saucerrerunio-dewey-dunningtonwherobots-andrew-lambinfluxdata.html">Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata" rel="bookmark" title="Permalink to Implementing User Defined Types and Custom Metadata in DataFusion">Implementing User Defined Types and Custom Metadata in DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -50,15 +45,22 @@ defined types and metadata. The user defined function APIs let users access
metadata on the input columns to functions and produce metadata in the output.</p>
<h2 id="user-defined-types-extension-types">User defined types == extension types<a class="headerlink" href="#user-defined-types-extension-types" title="Permanent link">¶</a></h2>
<p>DataFusion directly uses <a href="https://arrow.apache.org">Apache Arrow</a>'s <a href="https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html">DataTypes</a> as its type system. This
-has …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+has …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-09-21T00:00:00+00:00"> Sun 21 September 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/tim-saucerrerunio-dewey-dunningtonwherobots-andrew-lambinfluxdata.html">Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/timsaucer.html b/blog/author/timsaucer.html
index 2711c4a..ebb0622 100644
--- a/blog/author/timsaucer.html
+++ b/blog/author/timsaucer.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by timsaucer</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by timsaucer</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
-
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by timsaucer</h2>
-
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/30/datafusion-python-46.0.0" rel="bookmark" title="Permalink to Apache DataFusion Python 46.0.0 Released">Apache DataFusion Python 46.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-30T00:00:00+00:00"> Sun 30 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+ </head>
+
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by timsaucer</h2>
+
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/30/datafusion-python-46.0.0" rel="bookmark" title="Permalink to Apache DataFusion Python 46.0.0 Released">Apache DataFusion Python 46.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -50,17 +45,17 @@ limitations under the License.
<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/46.0.0/">datafusion-python 46.0.0</a> has been released. This release
brings in all of the new features of the core <a href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0">DataFusion 46.0.0</a> library. Since the last
blog post for <a href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0/">datafusion-python 43.1.0</a>, a large number of improvements have been made
-that can …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0" rel="bookmark" title="Permalink to Apache DataFusion Python 43.1.0 Released">Apache DataFusion Python 43.1.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-12-14T00:00:00+00:00"> Sat 14 December 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+that can …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-30T00:00:00+00:00"> Sun 30 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0" rel="bookmark" title="Permalink to Apache DataFusion Python 43.1.0 Released">Apache DataFusion Python 43.1.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -82,17 +77,17 @@ limitations under the License.
<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/43.1.0/">datafusion-python 43.1.0</a> has been released. This release
brings in all of the new features of the core <a href="https://github.com/apache/datafusion/blob/main/dev/changelog/43.0.0.md">DataFusion 43.0.0</a> library. Since the last
blog post for <a href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0/">datafusion-python 40.1.0</a>, a large number of improvements have been made
-that can …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons" rel="bookmark" title="Permalink to Comparing approaches to User Defined Functions in Apache DataFusion using Python">Comparing approaches to User Defined Functions in Apache DataFusion using Python</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-11-19T00:00:00+00:00"> Tue 19 November 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+that can …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-12-14T00:00:00+00:00"> Sat 14 December 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons" rel="bookmark" title="Permalink to Comparing approaches to User Defined Functions in Apache DataFusion using Python">Comparing approaches to User Defined Functions in Apache DataFusion using Python</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -115,17 +110,17 @@ limitations under the License.
<p>For a few months now I’ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
fast query engine written in Rust. From my experience the language that nearly all data scientists
are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0" rel="bookmark" title="Permalink to Apache DataFusion Python 40.1.0 Released, Significant usability updates">Apache DataFusion Python 40.1.0 Released, Significant usability updates</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-08-20T00:00:00+00:00"> Tue 20 August 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-11-19T00:00:00+00:00"> Tue 19 November 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0" rel="bookmark" title="Permalink to Apache DataFusion Python 40.1.0 Released, Significant usability updates">Apache DataFusion Python 40.1.0 Released, Significant usability updates</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -147,15 +142,22 @@ limitations under the License.
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/40.1.0/">DataFusion in Python 40.1.0</a> has been released. In addition to
bringing in all of the new features of the core <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> package, this release
-contains <em>significant</em> updates to the user interface and documentation. We listened to the python …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+contains <em>significant</em> updates to the user interface and documentation. We listened to the python …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-08-20T00:00:00+00:00"> Tue 20 August 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/xiangpeng-hao-andrew-lamb.html b/blog/author/xiangpeng-hao-andrew-lamb.html
index e45bc05..d4a4740 100644
--- a/blog/author/xiangpeng-hao-andrew-lamb.html
+++ b/blog/author/xiangpeng-hao-andrew-lamb.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Xiangpeng Hao, Andrew Lamb</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Xiangpeng Hao, Andrew Lamb</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Xiangpeng Hao, Andrew Lamb</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Xiangpeng Hao, Andrew Lamb</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1" rel="bookmark" title="Permalink to Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet">Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-09-13T00:00:00+00:00"> Fri 13 September 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xiangpeng-hao-andrew-lamb.html">Xiangpeng Hao, Andrew Lamb</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1" rel="bookmark" title="Permalink to Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet">Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -48,17 +43,17 @@ limitations under the License.
-->
<p><em>Editor's Note: This is the first of a <a href="../string-view-german-style-strings-part-2/">two part</a> blog series that was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2" rel="bookmark" title="Permalink to Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations">Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-09-13T00:00:00+00:00"> Fri 13 September 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xiangpeng-hao-andrew-lamb.html">Xiangpeng Hao, Andrew Lamb</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-09-13T00:00:00+00:00"> Fri 13 September 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xiangpeng-hao-andrew-lamb.html">Xiangpeng Hao, Andrew Lamb</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2" rel="bookmark" title="Permalink to Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations">Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -79,15 +74,22 @@ limitations under the License.
<p><em>Editor's Note: This blog series was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
<p>In the <a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1">first post</a>, we discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies.
-In this second …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+In this second …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-09-13T00:00:00+00:00"> Fri 13 September 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xiangpeng-hao-andrew-lamb.html">Xiangpeng Hao, Andrew Lamb</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/xiangpeng-hao.html b/blog/author/xiangpeng-hao.html
index 74122f0..85e79f4 100644
--- a/blog/author/xiangpeng-hao.html
+++ b/blog/author/xiangpeng-hao.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by Xiangpeng Hao</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by Xiangpeng Hao</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by Xiangpeng Hao</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by Xiangpeng Hao</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown" rel="bookmark" title="Permalink to Efficient Filter Pushdown in Parquet">Efficient Filter Pushdown in Parquet</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-21T00:00:00+00:00"> Fri 21 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xiangpeng-hao.html">Xiangpeng Hao</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <style>
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown" rel="bookmark" title="Permalink to Efficient Filter Pushdown in Parquet">Efficient Filter Pushdown in Parquet</a></h2> </header>
+ <section><style>
figure {
margin: 20px 0;
}
@@ -67,17 +62,17 @@ limitations under the License.
<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-pushdown/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
<hr/>
-<p>In the <a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning">previous post …</a></p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="bookmark" title="Permalink to Parquet Pruning in DataFusion: Read Only What Matters">Parquet Pruning in DataFusion: Read Only What Matters</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-20T00:00:00+00:00"> Thu 20 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xiangpeng-hao.html">Xiangpeng Hao</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>In the <a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning">previous post …</a></p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-21T00:00:00+00:00"> Fri 21 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xiangpeng-hao.html">Xiangpeng Hao</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="bookmark" title="Permalink to Parquet Pruning in DataFusion: Read Only What Matters">Parquet Pruning in DataFusion: Read Only What Matters</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -99,15 +94,22 @@ limitations under the License.
<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
<hr/>
<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
-<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-20T00:00:00+00:00"> Thu 20 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xiangpeng-hao.html">Xiangpeng Hao</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/author/xudong.html b/blog/author/xudong.html
index b64321c..25a85ad 100644
--- a/blog/author/xudong.html
+++ b/blog/author/xudong.html
@@ -1,34 +1,29 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - Articles by xudong</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - Articles by xudong</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles by xudong</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles by xudong</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="bookmark" title="Permalink to Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack">Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-03-20T00:00:00+00:00"> Fri 20 March 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xudong.html">xudong</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="bookmark" title="Permalink to Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack">Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -68,15 +63,22 @@ figcaption {
}
</style>
<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
-<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-03-20T00:00:00+00:00"> Fri 20 March 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xudong.html">xudong</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/category/blog.html b/blog/category/blog.html
index 11fff8d..22c7aaa 100644
--- a/blog/category/blog.html
+++ b/blog/category/blog.html
@@ -1,35 +1,97 @@
<!DOCTYPE html>
<html lang="en">
-<head>
- <title>Apache DataFusion Blog - blog category</title>
- <meta charset="utf-8" />
- <meta name="generator" content="Pelican" />
+ <head>
+ <title>Apache DataFusion Blog - blog category</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://datafusion.apache.org/blog/feed.xml" type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS Feed" />
<link href="https://datafusion.apache.org/blog/feeds/blog.atom.xml" type="application/atom+xml" rel="alternate" title="Apache DataFusion Blog Categories Atom Feed" />
-</head>
+ </head>
-<body id="index" class="home">
- <header id="banner" class="body">
- <h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1>
- </header><!-- /#banner -->
- <nav id="menu"><ul>
- <li><a href="https://datafusion.apache.org/blog/pages/about.html">About</a></li>
- <li><a href="https://datafusion.apache.org/blog/pages/index.html">index</a></li>
- <li class="active"><a href="https://datafusion.apache.org/blog/category/blog.html">blog</a></li>
- </ul></nav><!-- /#menu -->
-<section id="content">
-<h2>Articles in the blog category</h2>
+ <body>
+ <header>
+ <hgroup><h1><a href="https://datafusion.apache.org/blog/">Apache DataFusion Blog</a></h1></hgroup>
+ <nav><ul>
+ <li><a href="https://datafusion.apache.org/blog/pages/about.html" >About</a></li>
+ <li><a href="https://datafusion.apache.org/blog/pages/index.html" >index</a></li>
+ <li><a href="https://datafusion.apache.org/blog/category/blog.html" aria-current="page" >blog</a></li>
+ </ul></nav>
+ </header>
+ <main>
+ <h2>Articles in the blog category</h2>
-<ol id="post-list">
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/31/writing-table-providers" rel="bookmark" title="Permalink to Writing Custom Table Providers in Apache DataFusion">Writing Custom Table Providers in Apache DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-03-31T00:00:00+00:00"> Tue 31 March 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/tim-saucer-rerunio.html">Tim Saucer (rerun.io)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/04/18/datafusion-comet-0.15.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.15.0 Release">Apache DataFusion Comet 0.15.0 Release</a></h2> </header>
+ <section><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.15.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately four weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-04-18T00:00:00+00:00"> Sat 18 April 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/04/02/datafusion-53.0.0" rel="bookmark" title="Permalink to Apache DataFusion 53.0.0 Released">Apache DataFusion 53.0.0 Released</a></h2> </header>
+ <section><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/53.0.0">DataFusion 53.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0/">DataFusion 52.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md#credits">114 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p><img alt="Performance over time" class="img-fluid" src="/blog/images/datafusion-53.0.0/performance_over_time_clickbench.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: Average …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-04-02T00:00:00+00:00"> Thu 02 April 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/03/31/writing-table-providers" rel="bookmark" title="Permalink to Writing Custom Table Providers in Apache DataFusion">Writing Custom Table Providers in Apache DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -51,17 +113,17 @@ limitations under the License.
<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
in a custom format, behind an API, or in a system that DataFusion does not
natively support, you can teach DataFusion to read it by implementing a
-<strong>custom table provider</strong>. This post walks through the three layers you …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="bookmark" title="Permalink to Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack">Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-03-20T00:00:00+00:00"> Fri 20 March 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xudong.html">xudong</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<strong>custom table provider</strong>. This post walks through the three layers you …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-03-31T00:00:00+00:00"> Tue 31 March 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/tim-saucer-rerunio.html">Tim Saucer (rerun.io)</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="bookmark" title="Permalink to Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack">Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -101,17 +163,17 @@ figcaption {
}
</style>
<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
-<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/03/18/datafusion-comet-0.14.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.14.0 Release">Apache DataFusion Comet 0.14.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-03-18T00:00:00+00:00"> Wed 18 March 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-03-20T00:00:00+00:00"> Fri 20 March 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xudong.html">xudong</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/03/18/datafusion-comet-0.14.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.14.0 Release">Apache DataFusion Comet 0.14.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -133,17 +195,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.14.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately eight weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="bookmark" title="Permalink to Optimizing SQL CASE Expression Evaluation">Optimizing SQL CASE Expression Evaluation</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-02-02T00:00:00+00:00"> Mon 02 February 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately eight weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-03-18T00:00:00+00:00"> Wed 18 March 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="bookmark" title="Permalink to Optimizing SQL CASE Expression Evaluation">Optimizing SQL CASE Expression Evaluation</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -183,17 +245,17 @@ figcaption {
}
</style>
<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
-It allows you to control which expression from a …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.13.0 Release">Apache DataFusion Comet 0.13.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-01-30T00:00:00+00:00"> Fri 30 January 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+It allows you to control which expression from a …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-02-02T00:00:00+00:00"> Mon 02 February 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.13.0 Release">Apache DataFusion Comet 0.13.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -215,17 +277,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately eight weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="bookmark" title="Permalink to Apache DataFusion 52.0.0 Released">Apache DataFusion 52.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately eight weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-01-30T00:00:00+00:00"> Fri 30 January 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="bookmark" title="Permalink to Apache DataFusion 52.0.0 Released">Apache DataFusion 52.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -249,17 +311,17 @@ some of the major improvements since <a href="https://datafusion.apache.org/blog
changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
making this release possible.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="bookmark" title="Permalink to Extending SQL in DataFusion: from ->> to TABLESAMPLE">Extending SQL in DataFusion: from ->> to TABLESAMPLE</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/geoffrey-claude-datadog.html">Geoffrey Claude (Datadog)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>We continue to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="bookmark" title="Permalink to Extending SQL in DataFusion: from ->> to TABLESAMPLE">Extending SQL in DataFusion: from ->> to TABLESAMPLE</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -279,17 +341,17 @@ limitations under the License.
-->
<p>If you embed <a href="https://datafusion.apache.org/">DataFusion</a> in your product, your users will eventually run SQL that DataFusion does not recognize. Not because the query is unreasonable, but because SQL in practice includes many dialects and system-specific statements.</p>
-<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="bookmark" title="Permalink to Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution">Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-12-15T00:00:00+00:00"> Mon 15 December 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/gene-bordegaray.html">Gene Bordegaray</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p></section>
+ <footer>
+ <p>Published: <time datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/geoffrey-claude-datadog.html">Geoffrey Claude (Datadog)</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="bookmark" title="Permalink to Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution">Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -307,20 +369,20 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
-Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/12/04/datafusion-comet-0.12.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.12.0 Release">Apache DataFusion Comet 0.12.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-12-04T00:00:00+00:00"> Thu 04 December 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div></section>
+ <footer>
+ <p>Published: <time datetime="2025-12-15T00:00:00+00:00"> Mon 15 December 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/gene-bordegaray.html">Gene Bordegaray</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/12/04/datafusion-comet-0.12.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.12.0 Release">Apache DataFusion Comet 0.12.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -342,17 +404,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.12.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately four weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0" rel="bookmark" title="Permalink to Apache DataFusion 51.0.0 Released">Apache DataFusion 51.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-11-25T00:00:00+00:00"> Tue 25 November 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately four weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-12-04T00:00:00+00:00"> Thu 04 December 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0" rel="bookmark" title="Permalink to Apache DataFusion 51.0.0 Released">Apache DataFusion 51.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -377,17 +439,17 @@ some of the major improvements since <a href="https://datafusion.apache.org/blog
changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md#credits">128 contributors</a> for
making this release possible.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.11.0 Release">Apache DataFusion Comet 0.11.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-10-21T00:00:00+00:00"> Tue 21 October 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>We continue …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-11-25T00:00:00+00:00"> Tue 25 November 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.11.0 Release">Apache DataFusion Comet 0.11.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -409,17 +471,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately five weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0" rel="bookmark" title="Permalink to Apache DataFusion 50.0.0 Released">Apache DataFusion 50.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-09-29T00:00:00+00:00"> Mon 29 September 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately five weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-10-21T00:00:00+00:00"> Tue 21 October 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0" rel="bookmark" title="Permalink to Apache DataFusion 50.0.0 Released">Apache DataFusion 50.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -444,17 +506,17 @@ limitations under the License.
highlights some of the major improvements since the release of <a href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0/">DataFusion
49.0.0</a>. The complete list of changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-50/dev/changelog/50.0.0.md">changelog</a>.
Thanks to <a href="https://github.com/apache/datafusion/blob/branch-50/dev/changelog/50.0.0.md#credits">numerous contributors</a> for making this release possible!</p>
-<h2 id="performance-improvements">Performance …</h2> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata" rel="bookmark" title="Permalink to Implementing User Defined Types and Custom Metadata in DataFusion">Implementing User Defined Types and Custom Metadata in DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-09-21T00:00:00+00:00"> Sun 21 September 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/tim-saucerrerunio-dewey-dunningtonwherobots-andrew-lambinfluxdata.html">Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<h2 id="performance-improvements">Performance …</h2></section>
+ <footer>
+ <p>Published: <time datetime="2025-09-29T00:00:00+00:00"> Mon 29 September 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata" rel="bookmark" title="Permalink to Implementing User Defined Types and Custom Metadata in DataFusion">Implementing User Defined Types and Custom Metadata in DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -476,17 +538,17 @@ defined types and metadata. The user defined function APIs let users access
metadata on the input columns to functions and produce metadata in the output.</p>
<h2 id="user-defined-types-extension-types">User defined types == extension types<a class="headerlink" href="#user-defined-types-extension-types" title="Permanent link">¶</a></h2>
<p>DataFusion directly uses <a href="https://arrow.apache.org">Apache Arrow</a>'s <a href="https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html">DataTypes</a> as its type system. This
-has …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/09/16/datafusion-comet-0.10.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.10.0 Release">Apache DataFusion Comet 0.10.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-09-16T00:00:00+00:00"> Tue 16 September 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+has …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-09-21T00:00:00+00:00"> Sun 21 September 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/tim-saucerrerunio-dewey-dunningtonwherobots-andrew-lambinfluxdata.html">Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/09/16/datafusion-comet-0.10.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.10.0 Release">Apache DataFusion Comet 0.10.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -508,17 +570,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.10.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately ten weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters" rel="bookmark" title="Permalink to Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries">Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-09-10T00:00:00+00:00"> Wed 10 September 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.html">Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately ten weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-09-16T00:00:00+00:00"> Tue 16 September 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters" rel="bookmark" title="Permalink to Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries">Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -548,17 +610,17 @@ optimizations work, and how we implemented them with the <a href="https://datafu
community to improve performance by an order of magnitude for some query
patterns.</p>
<h2 id="motivation-and-results">Motivation and Results<a class="headerlink" href="#motivation-and-results" title="Permanent link">¶</a></h2>
-<p>The …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes" rel="bookmark" title="Permalink to Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet">Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-08-15T00:00:00+00:00"> Fri 15 August 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/andrew-lamb-influxdata.html">Andrew Lamb (InfluxData)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>The …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-09-10T00:00:00+00:00"> Wed 10 September 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.html">Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes" rel="bookmark" title="Permalink to Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet">Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -583,17 +645,17 @@ metadata and is limited to indexing structures provided by the format. In fact,
caching parsed metadata and using custom external indexes along with
Parquet's hierarchical data organization can significantly speed up query
processing.</p>
-<p>In this blog, I describe …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0" rel="bookmark" title="Permalink to Apache DataFusion 49.0.0 Released">Apache DataFusion 49.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-28T00:00:00+00:00"> Mon 28 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>In this blog, I describe …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-08-15T00:00:00+00:00"> Fri 15 August 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/andrew-lamb-influxdata.html">Andrew Lamb (InfluxData)</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0" rel="bookmark" title="Permalink to Apache DataFusion 49.0.0 Released">Apache DataFusion 49.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -617,17 +679,17 @@ limitations under the License.
<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/49.0.0">DataFusion 49.0.0</a>. This blog post highlights some of
the major improvements since the release of <a href="https://datafusion.apache.org/blog/2025/07/18/datafusion-48.0.0/">DataFusion 48.0.0</a>. The complete list of changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-49/dev/changelog/49.0.0.md">changelog</a>.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>DataFusion continues to focus on enhancing performance, as …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0" rel="bookmark" title="Permalink to Apache DataFusion 48.0.0 Released">Apache DataFusion 48.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-16T00:00:00+00:00"> Wed 16 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">PMC</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>DataFusion continues to focus on enhancing performance, as …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-28T00:00:00+00:00"> Mon 28 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0" rel="bookmark" title="Permalink to Apache DataFusion 48.0.0 Released">Apache DataFusion 48.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -649,17 +711,17 @@ limitations under the License.
improvements and fixes. You can find the complete details in the full
<a href="https://github.com/apache/datafusion/blob/branch-48/dev/changelog/48.0.0.md">changelog</a>. We’ll highlight the most
important changes below and guide you through upgrading.</p>
-<h2 id="breaking-changes">Breaking …</h2> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes" rel="bookmark" title="Permalink to Embedding User-Defined Indexes in Apache Parquet Files">Embedding User-Defined Indexes in Apache Parquet Files</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-14T00:00:00+00:00"> Mon 14 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/qi-zhu-cloudera-jigao-luo-systems-group-at-tu-darmstadt-and-andrew-lamb-influxdata.html">Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<h2 id="breaking-changes">Breaking …</h2></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-16T00:00:00+00:00"> Wed 16 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">PMC</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes" rel="bookmark" title="Permalink to Embedding User-Defined Indexes in Apache Parquet Files">Embedding User-Defined Indexes in Apache Parquet Files</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -678,17 +740,17 @@ limitations under the License.
{% endcomment %}
-->
-<p>It’s a common misconception that <a href="https://parquet.apache.org/">Apache Parquet</a> files are limited to basic Min/Max/Null Count statistics and Bloom filters, and that adding more advanced indexes requires changing the specification or creating a new file format. In fact, footer metadata and offset-based addressing already provide everything needed to embed …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/11/datafusion-47.0.0" rel="bookmark" title="Permalink to Apache DataFusion 47.0.0 Released">Apache DataFusion 47.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-11T00:00:00+00:00"> Fri 11 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">PMC</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>It’s a common misconception that <a href="https://parquet.apache.org/">Apache Parquet</a> files are limited to basic Min/Max/Null Count statistics and Bloom filters, and that adding more advanced indexes requires changing the specification or creating a new file format. In fact, footer metadata and offset-based addressing already provide everything needed to embed …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-14T00:00:00+00:00"> Mon 14 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/qi-zhu-cloudera-jigao-luo-systems-group-at-tu-darmstadt-and-andrew-lamb-influxdata.html">Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/11/datafusion-47.0.0" rel="bookmark" title="Permalink to Apache DataFusion 47.0.0 Released">Apache DataFusion 47.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -709,17 +771,17 @@ limitations under the License.
<p>We’re excited to announce the release of <strong>Apache DataFusion 47.0.0</strong>! This new version represents a significant
milestone for the project, packing in a wide range of improvements and fixes. You can find the complete details in the
full <a href="https://github.com/apache/datafusion/blob/branch-47/dev/changelog/47.0.0.md">changelog</a>. We’ll highlight the most
-important changes below …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.9.0 Release">Apache DataFusion Comet 0.9.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-07-01T00:00:00+00:00"> Tue 01 July 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+important changes below …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-11T00:00:00+00:00"> Fri 11 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">PMC</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.9.0 Release">Apache DataFusion Comet 0.9.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -741,17 +803,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.9.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately ten weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="bookmark" title="Permalink to Using Rust async for Query Execution and Cancelling Long-Running Queries">Using Rust async for Query Execution and Cancelling Long-Running Queries</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-06-30T00:00:00+00:00"> Mon 30 June 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately ten weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-07-01T00:00:00+00:00"> Tue 01 July 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="bookmark" title="Permalink to Using Rust async for Query Execution and Cancelling Long-Running Queries">Using Rust async for Query Execution and Cancelling Long-Running Queries</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -791,17 +853,17 @@ figcaption {
}
</style>
<p>Have you ever tried to cancel a query that just wouldn't stop?
-In this post, we'll review how Rust's <a href="https://doc.rust-lang.org/book/ch17-00-async-await.html"><code>async</code> programming model</a> works, how …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one" rel="bookmark" title="Permalink to Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview">Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-06-15T00:00:00+00:00"> Sun 15 June 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/alamb-akurmustafa.html">alamb, akurmustafa</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+In this post, we'll review how Rust's <a href="https://doc.rust-lang.org/book/ch17-00-async-await.html"><code>async</code> programming model</a> works, how …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-06-30T00:00:00+00:00"> Mon 30 June 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pepijn-van-eeckhoudt.html">Pepijn Van Eeckhoudt</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one" rel="bookmark" title="Permalink to Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview">Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -828,17 +890,17 @@ science,”</a> according to Father
Pavlo, or some behind-the-scenes player. We believe this perception is because:</p>
<ol>
<li>
-<p>One must implement the rest of a …</p></li></ol> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="bookmark" title="Permalink to Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion">Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-06-15T00:00:00+00:00"> Sun 15 June 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/alamb-akurmustafa.html">alamb, akurmustafa</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content">
+<p>One must implement the rest of a …</p></li></ol></section>
+ <footer>
+ <p>Published: <time datetime="2025-06-15T00:00:00+00:00"> Sun 15 June 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/alamb-akurmustafa.html">alamb, akurmustafa</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="bookmark" title="Permalink to Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion">Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</a></h2> </header>
+ <section>
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -861,17 +923,17 @@ limitations under the License.
<p>In the <a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one">first part of this post</a>, we discussed what a Query Optimizer is, what
role it plays, and described how industrial optimizers are organized. In this
second post, we describe various optimizations that are found in <a href="https://datafusion.apache.org/">Apache
-DataFusion</a> and …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.8.0 Release">Apache DataFusion Comet 0.8.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-05-06T00:00:00+00:00"> Tue 06 May 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+DataFusion</a> and …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-06-15T00:00:00+00:00"> Sun 15 June 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/alamb-akurmustafa.html">alamb, akurmustafa</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.8.0 Release">Apache DataFusion Comet 0.8.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -893,17 +955,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.8.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately six weeks of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions" rel="bookmark" title="Permalink to User defined Window Functions in DataFusion">User defined Window Functions in DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-04-19T00:00:00+00:00"> Sat 19 April 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/aditya-singh-rathore-andrew-lamb.html">Aditya Singh Rathore, Andrew Lamb</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This release covers approximately six weeks of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-05-06T00:00:00+00:00"> Tue 06 May 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions" rel="bookmark" title="Permalink to User defined Window Functions in DataFusion">User defined Window Functions in DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -923,17 +985,17 @@ limitations under the License.
-->
<p>Window functions are a powerful feature in SQL, allowing for complex analytical computations over a subset of data. However, efficiently implementing them, especially sliding windows, can be quite challenging. With <a href="https://datafusion.apache.org/">Apache DataFusion</a>'s user-defined window functions, developers can easily take advantage of all the effort put into DataFusion's implementation.</p>
-<p>In …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/04/10/fastest-tpch-generator" rel="bookmark" title="Permalink to tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust">tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-04-10T00:00:00+00:00"> Thu 10 April 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/andrew-lamb-achraf-b-and-sean-smith.html">Andrew Lamb, Achraf B, and Sean Smith</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>In …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-04-19T00:00:00+00:00"> Sat 19 April 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/aditya-singh-rathore-andrew-lamb.html">Aditya Singh Rathore, Andrew Lamb</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/04/10/fastest-tpch-generator" rel="bookmark" title="Permalink to tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust">tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -950,29 +1012,21 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
-development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/30/datafusion-python-46.0.0" rel="bookmark" title="Permalink to Apache DataFusion Python 46.0.0 Released">Apache DataFusion Python 46.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-30T00:00:00+00:00"> Sun 30 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
+faster than any other implementation we know of.</p>
+<p>It is now possible to create …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-04-10T00:00:00+00:00"> Thu 10 April 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/andrew-lamb-achraf-b-and-sean-smith.html">Andrew Lamb, Achraf B, and Sean Smith</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/30/datafusion-python-46.0.0" rel="bookmark" title="Permalink to Apache DataFusion Python 46.0.0 Released">Apache DataFusion Python 46.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -994,17 +1048,17 @@ limitations under the License.
<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/46.0.0/">datafusion-python 46.0.0</a> has been released. This release
brings in all of the new features of the core <a href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0">DataFusion 46.0.0</a> library. Since the last
blog post for <a href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0/">datafusion-python 43.1.0</a>, a large number of improvements have been made
-that can …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0" rel="bookmark" title="Permalink to Apache DataFusion 46.0.0 Released">Apache DataFusion 46.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-24T00:00:00+00:00"> Mon 24 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html">Oznur Hanci and Berkay Sahin on behalf of the PMC</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+that can …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-30T00:00:00+00:00"> Sun 30 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0" rel="bookmark" title="Permalink to Apache DataFusion 46.0.0 Released">Apache DataFusion 46.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1021,17 +1075,17 @@ limitations under the License.
{% endcomment %}
-->
-<p>We’re excited to announce the release of <strong>Apache DataFusion 46.0.0</strong>! This new version represents a significant milestone for the project, packing in a wide range of improvements and fixes. You can find the complete details in the full <a href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>. We’ll highlight the most important changes below …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown" rel="bookmark" title="Permalink to Efficient Filter Pushdown in Parquet">Efficient Filter Pushdown in Parquet</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-21T00:00:00+00:00"> Fri 21 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xiangpeng-hao.html">Xiangpeng Hao</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <style>
+<p>We’re excited to announce the release of <strong>Apache DataFusion 46.0.0</strong>! This new version represents a significant milestone for the project, packing in a wide range of improvements and fixes. You can find the complete details in the full <a href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>. We’ll highlight the most important changes below …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-24T00:00:00+00:00"> Mon 24 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html">Oznur Hanci and Berkay Sahin on behalf of the PMC</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown" rel="bookmark" title="Permalink to Efficient Filter Pushdown in Parquet">Efficient Filter Pushdown in Parquet</a></h2> </header>
+ <section><style>
figure {
margin: 20px 0;
}
@@ -1070,17 +1124,17 @@ limitations under the License.
<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-pushdown/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
<hr/>
-<p>In the <a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning">previous post …</a></p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.7.0 Release">Apache DataFusion Comet 0.7.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-20T00:00:00+00:00"> Thu 20 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>In the <a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning">previous post …</a></p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-21T00:00:00+00:00"> Fri 21 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xiangpeng-hao.html">Xiangpeng Hao</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.7.0 Release">Apache DataFusion Comet 0.7.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1102,17 +1156,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="bookmark" title="Permalink to Parquet Pruning in DataFusion: Read Only What Matters">Parquet Pruning in DataFusion: Read Only What Matters</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-20T00:00:00+00:00"> Thu 20 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xiangpeng-hao.html">Xiangpeng Hao</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-20T00:00:00+00:00"> Thu 20 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="bookmark" title="Permalink to Parquet Pruning in DataFusion: Read Only What Matters">Parquet Pruning in DataFusion: Read Only What Matters</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1134,17 +1188,17 @@ limitations under the License.
<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
<hr/>
<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
-<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="bookmark" title="Permalink to Using Ordering for Better Plans in Apache DataFusion">Using Ordering for Better Plans in Apache DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-03-11T00:00:00+00:00"> Tue 11 March 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/mustafa-akur-andrew-lamb.html">Mustafa Akur, Andrew Lamb</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-20T00:00:00+00:00"> Thu 20 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xiangpeng-hao.html">Xiangpeng Hao</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="bookmark" title="Permalink to Using Ordering for Better Plans in Apache DataFusion">Using Ordering for Better Plans in Apache DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1166,18 +1220,18 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
-<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/02/20/datafusion-45.0.0" rel="bookmark" title="Permalink to Apache DataFusion 45.0.0 Released">Apache DataFusion 45.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-02-20T00:00:00+00:00"> Thu 20 February 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></section>
+ <footer>
+ <p>Published: <time datetime="2025-03-11T00:00:00+00:00"> Tue 11 March 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/mustafa-akur-andrew-lamb.html">Mustafa Akur, Andrew Lamb</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/02/20/datafusion-45.0.0" rel="bookmark" title="Permalink to Apache DataFusion 45.0.0 Released">Apache DataFusion 45.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1201,17 +1255,17 @@ limitations under the License.
<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
what the community is thinking about in the next 6 months. It has been an exciting
-period of development …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.6.0 Release">Apache DataFusion Comet 0.6.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-02-17T00:00:00+00:00"> Mon 17 February 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+period of development …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-02-20T00:00:00+00:00"> Thu 20 February 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.6.0 Release">Apache DataFusion Comet 0.6.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1233,17 +1287,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.6.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0" rel="bookmark" title="Permalink to Apache DataFusion Ballista 43.0.0 Released">Apache DataFusion Ballista 43.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-02-02T00:00:00+00:00"> Sun 02 February 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/milenkovicm.html">milenkovicm</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-02-17T00:00:00+00:00"> Mon 17 February 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0" rel="bookmark" title="Permalink to Apache DataFusion Ballista 43.0.0 Released">Apache DataFusion Ballista 43.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1265,17 +1319,17 @@ limitations under the License.
<p>We are pleased to announce version <a href="https://github.com/apache/datafusion-ballista/blob/main/CHANGELOG.md#4300-2025-01-07">43.0.0</a> of the <a href="https://datafusion.apache.org/ballista/">DataFusion Ballista</a>. Ballista allows existing <a href="https://datafusion.apache.org">DataFusion</a> applications to be scaled out on a cluster for use cases that are not practical to run on a single node.</p>
<h2 id="highlights-of-this-release">Highlights of this release<a class="headerlink" href="#highlights-of-this-release" title="Permanent link">¶</a></h2>
<h3 id="seamless-integration-with-datafusion">Seamless Integration with DataFusion<a class="headerlink" href="#seamless-integration-with-datafusion" title="Permanent link">¶</a></h3>
-<p>The primary objective of …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2025/01/17/datafusion-comet-0.5.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.5.0 Release">Apache DataFusion Comet 0.5.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2025-01-17T00:00:00+00:00"> Fri 17 January 2025 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>The primary objective of …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-02-02T00:00:00+00:00"> Sun 02 February 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/milenkovicm.html">milenkovicm</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2025/01/17/datafusion-comet-0.5.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.5.0 Release">Apache DataFusion Comet 0.5.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1297,17 +1351,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.5.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0" rel="bookmark" title="Permalink to Apache DataFusion Python 43.1.0 Released">Apache DataFusion Python 43.1.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-12-14T00:00:00+00:00"> Sat 14 December 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2025-01-17T00:00:00+00:00"> Fri 17 January 2025 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0" rel="bookmark" title="Permalink to Apache DataFusion Python 43.1.0 Released">Apache DataFusion Python 43.1.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1329,17 +1383,17 @@ limitations under the License.
<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/43.1.0/">datafusion-python 43.1.0</a> has been released. This release
brings in all of the new features of the core <a href="https://github.com/apache/datafusion/blob/main/dev/changelog/43.0.0.md">DataFusion 43.0.0</a> library. Since the last
blog post for <a href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0/">datafusion-python 40.1.0</a>, a large number of improvements have been made
-that can …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.4.0 Release">Apache DataFusion Comet 0.4.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-11-20T00:00:00+00:00"> Wed 20 November 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+that can …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-12-14T00:00:00+00:00"> Sat 14 December 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.4.0 Release">Apache DataFusion Comet 0.4.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1361,17 +1415,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons" rel="bookmark" title="Permalink to Comparing approaches to User Defined Functions in Apache DataFusion using Python">Comparing approaches to User Defined Functions in Apache DataFusion using Python</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-11-19T00:00:00+00:00"> Tue 19 November 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-11-20T00:00:00+00:00"> Wed 20 November 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons" rel="bookmark" title="Permalink to Comparing approaches to User Defined Functions in Apache DataFusion using Python">Comparing approaches to User Defined Functions in Apache DataFusion using Python</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1394,17 +1448,17 @@ limitations under the License.
<p>For a few months now I’ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
fast query engine written in Rust. From my experience the language that nearly all data scientists
are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" rel="bookmark" title="Permalink to Apache DataFusion is now the fastest single node engine for querying Apache Parquet files">Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-11-18T00:00:00+00:00"> Mon 18 November 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/andrew-lamb-staff-engineer-at-influxdata.html">Andrew Lamb, Staff Engineer at InfluxData</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-11-19T00:00:00+00:00"> Tue 19 November 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" rel="bookmark" title="Permalink to Apache DataFusion is now the fastest single node engine for querying Apache Parquet files">Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1427,17 +1481,17 @@ limitations under the License.
fastest engine for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">ClickBench</a>. It is faster
than <a href="https://duckdb.org/">DuckDB</a>, <a href="https://clickhouse.com/chdb">chDB</a> and <a href="https://clickhouse.com/">Clickhouse</a> using the same hardware. It also marks
the first time a <a href="https://www.rust-lang.org/">Rust</a>-based engine holds the top spot, which has previously
-been …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.3.0 Release">Apache DataFusion Comet 0.3.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-09-27T00:00:00+00:00"> Fri 27 September 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+been …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-11-18T00:00:00+00:00"> Mon 18 November 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/andrew-lamb-staff-engineer-at-influxdata.html">Andrew Lamb, Staff Engineer at InfluxData</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.3.0 Release">Apache DataFusion Comet 0.3.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1459,17 +1513,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.3.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1" rel="bookmark" title="Permalink to Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet">Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-09-13T00:00:00+00:00"> Fri 13 September 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xiangpeng-hao-andrew-lamb.html">Xiangpeng Hao, Andrew Lamb</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-09-27T00:00:00+00:00"> Fri 27 September 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1" rel="bookmark" title="Permalink to Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet">Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1489,17 +1543,17 @@ limitations under the License.
-->
<p><em>Editor's Note: This is the first of a <a href="../string-view-german-style-strings-part-2/">two part</a> blog series that was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2" rel="bookmark" title="Permalink to Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations">Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-09-13T00:00:00+00:00"> Fri 13 September 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/xiangpeng-hao-andrew-lamb.html">Xiangpeng Hao, Andrew Lamb</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-09-13T00:00:00+00:00"> Fri 13 September 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xiangpeng-hao-andrew-lamb.html">Xiangpeng Hao, Andrew Lamb</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2" rel="bookmark" title="Permalink to Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations">Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1520,17 +1574,17 @@ limitations under the License.
<p><em>Editor's Note: This blog series was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
<p>In the <a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1">first post</a>, we discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies.
-In this second …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.2.0 Release">Apache DataFusion Comet 0.2.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-08-28T00:00:00+00:00"> Wed 28 August 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+In this second …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-09-13T00:00:00+00:00"> Fri 13 September 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/xiangpeng-hao-andrew-lamb.html">Xiangpeng Hao, Andrew Lamb</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.2.0 Release">Apache DataFusion Comet 0.2.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1552,17 +1606,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.2.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0" rel="bookmark" title="Permalink to Apache DataFusion Python 40.1.0 Released, Significant usability updates">Apache DataFusion Python 40.1.0 Released, Significant usability updates</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-08-20T00:00:00+00:00"> Tue 20 August 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims to …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-08-28T00:00:00+00:00"> Wed 28 August 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0" rel="bookmark" title="Permalink to Apache DataFusion Python 40.1.0 Released, Significant usability updates">Apache DataFusion Python 40.1.0 Released, Significant usability updates</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1584,17 +1638,17 @@ limitations under the License.
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/40.1.0/">DataFusion in Python 40.1.0</a> has been released. In addition to
bringing in all of the new features of the core <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> package, this release
-contains <em>significant</em> updates to the user interface and documentation. We listened to the python …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0" rel="bookmark" title="Permalink to Apache DataFusion 40.0.0 Released">Apache DataFusion 40.0.0 Released</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-07-24T00:00:00+00:00"> Wed 24 July 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+contains <em>significant</em> updates to the user interface and documentation. We listened to the python …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-08-20T00:00:00+00:00"> Tue 20 August 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/timsaucer.html">timsaucer</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0" rel="bookmark" title="Permalink to Apache DataFusion 40.0.0 Released">Apache DataFusion 40.0.0 Released</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1618,17 +1672,17 @@ limitations under the License.
<p>We are proud to announce <a href="https://crates.io/crates/datafusion/40.0.0">DataFusion 40.0.0</a>. This blog highlights some of the
many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0/">DataFusion 34.0.0</a> and a preview of
what the community is thinking about in the next 6 months. We are hoping to make
-more regular blog posts …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.1.0 Release">Apache DataFusion Comet 0.1.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-07-20T00:00:00+00:00"> Sat 20 July 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+more regular blog posts …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-07-24T00:00:00+00:00"> Wed 24 July 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 0.1.0 Release">Apache DataFusion Comet 0.1.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1650,17 +1704,17 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce the first official source release of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp" rel="bookmark" title="Permalink to Announcing Apache Arrow DataFusion is now Apache DataFusion">Announcing Apache Arrow DataFusion is now Apache DataFusion</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-05-07T00:00:00+00:00"> Tue 07 May 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Comet runs on commodity hardware and aims …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-07-20T00:00:00+00:00"> Sat 20 July 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp" rel="bookmark" title="Permalink to Announcing Apache Arrow DataFusion is now Apache DataFusion">Announcing Apache Arrow DataFusion is now Apache DataFusion</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1685,17 +1739,17 @@ limitations under the License.
April 16, 2024 the Apache Arrow DataFusion subproject is now a top level
<a href="https://www.apache.org/">Apache Software Foundation</a> project.</p>
<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h2>
-<p>Apache DataFusion is a fast, extensible query engine for building …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/03/06/comet-donation" rel="bookmark" title="Permalink to Announcing Apache Arrow DataFusion Comet">Announcing Apache Arrow DataFusion Comet</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-03-06T00:00:00+00:00"> Wed 06 March 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Apache DataFusion is a fast, extensible query engine for building …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-05-07T00:00:00+00:00"> Tue 07 May 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/03/06/comet-donation" rel="bookmark" title="Permalink to Announcing Apache Arrow DataFusion Comet">Announcing Apache Arrow DataFusion Comet</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1719,17 +1773,17 @@ limitations under the License.
a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/datafusion">Apache Arrow DataFusion</a>.</p>
<p>Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to
accelerate Spark workloads. It is designed as a drop-in
-replacement for Spark's JVM …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024">Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2024-01-19T00:00:00+00:00"> Fri 19 January 2024 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+replacement for Spark's JVM …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-03-06T00:00:00+00:00"> Wed 06 March 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024">Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1752,17 +1806,17 @@ limitations under the License.
<p>We recently <a href="https://crates.io/crates/datafusion/34.0.0">released DataFusion 34.0.0</a>. This blog highlights some of the major
improvements since we <a href="https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/.">released DataFusion 26.0.0</a> (spoiler alert there are many)
and a preview of where the community plans to focus in the next 6 months.</p>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping" rel="bookmark" title="Permalink to Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0">Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2023-08-05T00:00:00+00:00"> Sat 05 August 2023 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/alamb-dandandan-tustvold.html">alamb, Dandandan, tustvold</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query …</p></section>
+ <footer>
+ <p>Published: <time datetime="2024-01-19T00:00:00+00:00"> Fri 19 January 2024 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping" rel="bookmark" title="Permalink to Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0">Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1786,17 +1840,17 @@ limitations under the License.
<p>Andrew Lamb, Daniël Heres, Raphael Taylor-Davies,</p>
<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
<h2 id="tldr">TLDR<a class="headerlink" href="#tldr" title="Permanent link">¶</a></h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2023/06/24/datafusion-25.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 26.0.0">Apache Arrow DataFusion 26.0.0</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2023-06-24T00:00:00+00:00"> Sat 24 June 2023 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p></section>
+ <footer>
+ <p>Published: <time datetime="2023-08-05T00:00:00+00:00"> Sat 05 August 2023 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/alamb-dandandan-tustvold.html">alamb, Dandandan, tustvold</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2023/06/24/datafusion-25.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 26.0.0">Apache Arrow DataFusion 26.0.0</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1821,17 +1875,17 @@ performance improved and we are <a href="https://github.com/apache/arrow-datafus
top level Apache Project.</p>
<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h2>
<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database
-toolkit …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 16.0.0 Project Update">Apache Arrow DataFusion 16.0.0 Project Update</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2023-01-19T00:00:00+00:00"> Thu 19 January 2023 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+toolkit …</p></section>
+ <footer>
+ <p>Published: <time datetime="2023-06-24T00:00:00+00:00"> Sat 24 June 2023 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 16.0.0 Project Update">Apache Arrow DataFusion 16.0.0 Project Update</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1858,17 +1912,17 @@ in-memory format. It is targeted primarily at developers creating data
intensive analytics, and offers mature
<a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>,
a DataFrame API, and many extension points.</p>
-<p>Systems based on DataFusion perform very well in benchmarks …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0" rel="bookmark" title="Permalink to Apache Arrow Ballista 0.9.0 Release">Apache Arrow Ballista 0.9.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-10-28T00:00:00+00:00"> Fri 28 October 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>Systems based on DataFusion perform very well in benchmarks …</p></section>
+ <footer>
+ <p>Published: <time datetime="2023-01-19T00:00:00+00:00"> Thu 19 January 2023 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0" rel="bookmark" title="Permalink to Apache Arrow Ballista 0.9.0 Release">Apache Arrow Ballista 0.9.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1892,17 +1946,17 @@ limitations under the License.
<p>Ballista 0.9.0 is now available and is the most significant release since the project was <a href="http://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to Apache
Arrow in 2021.</p>
<p>This release represents 4 weeks of work, with 66 commits from 14 contributors:</p>
-<pre><code> 22 Andy …</code></pre> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 13.0.0 Project Update">Apache Arrow DataFusion 13.0.0 Project Update</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-10-25T00:00:00+00:00"> Tue 25 October 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<pre><code> 22 Andy …</code></pre></section>
+ <footer>
+ <p>Published: <time datetime="2022-10-28T00:00:00+00:00"> Fri 28 October 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 13.0.0 Project Update">Apache Arrow DataFusion 13.0.0 Project Update</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1923,17 +1977,17 @@ limitations under the License.
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> <a href="https://crates.io/crates/datafusion"><code>13.0.0</code></a> is released, and this blog contains an update on the project for the 5 months since our <a href="https://arrow.apache.org/blog/2022/05/16/datafusion-8.0.0/">last update in May 2022</a>.</p>
-<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/05/16/datafusion-8.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 8.0.0 Release">Apache Arrow DataFusion 8.0.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-05-16T00:00:00+00:00"> Mon 16 May 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL …</p></section>
+ <footer>
+ <p>Published: <time datetime="2022-10-25T00:00:00+00:00"> Tue 25 October 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/05/16/datafusion-8.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 8.0.0 Release">Apache Arrow DataFusion 8.0.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1956,17 +2010,17 @@ limitations under the License.
<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that
uses Apache Arrow as its in-memory format.</p>
<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>,
-a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/03/21/datafusion-contrib" rel="bookmark" title="Permalink to Introducing Apache Arrow DataFusion Contrib">Introducing Apache Arrow DataFusion Contrib</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-03-21T00:00:00+00:00"> Mon 21 March 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></section>
+ <footer>
+ <p>Published: <time datetime="2022-05-16T00:00:00+00:00"> Mon 16 May 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/03/21/datafusion-contrib" rel="bookmark" title="Permalink to Introducing Apache Arrow DataFusion Contrib">Introducing Apache Arrow DataFusion Contrib</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1986,17 +2040,17 @@ limitations under the License.
-->
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
<p>Apache Arrow <a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2022/02/28/datafusion-7.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 7.0.0 Release">Apache Arrow DataFusion 7.0.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2022-02-28T00:00:00+00:00"> Mon 28 February 2022 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is …</p></section>
+ <footer>
+ <p>Published: <time datetime="2022-03-21T00:00:00+00:00"> Mon 21 March 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2022/02/28/datafusion-7.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 7.0.0 Release">Apache Arrow DataFusion 7.0.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2016,17 +2070,17 @@ limitations under the License.
-->
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" rel="bookmark" title="Permalink to Apache Arrow DataFusion 6.0.0 Release">Apache Arrow DataFusion 6.0.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2021-11-19T00:00:00+00:00"> Fri 19 November 2021 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></section>
+ <footer>
+ <p>Published: <time datetime="2022-02-28T00:00:00+00:00"> Mon 28 February 2022 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" rel="bookmark" title="Permalink to Apache Arrow DataFusion 6.0.0 Release">Apache Arrow DataFusion 6.0.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2050,17 +2104,17 @@ query engine which leverages the unique features of
<a href="https://www.rust-lang.org/">Rust</a> and <a href="https://arrow.apache.org/">Apache
Arrow</a> to provide a system that is high
performance, easy to connect, easy to embed, and high quality.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2021/08/18/ballista-0.5.0" rel="bookmark" title="Permalink to Apache Arrow Ballista 0.5.0 Release">Apache Arrow Ballista 0.5.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2021-08-18T00:00:00+00:00"> Wed 18 August 2021 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers …</p></section>
+ <footer>
+ <p>Published: <time datetime="2021-11-19T00:00:00+00:00"> Fri 19 November 2021 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2021/08/18/ballista-0.5.0" rel="bookmark" title="Permalink to Apache Arrow Ballista 0.5.0 Release">Apache Arrow Ballista 0.5.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2082,17 +2136,17 @@ limitations under the License.
<p>Ballista extends DataFusion to provide support for distributed queries. This is the first release of Ballista since
the project was <a href="https://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to the Apache Arrow project
and includes 80 commits from 11 contributors.</p>
-<pre><code>git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust …</code></pre> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2021/08/18/datafusion-5.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 5.0.0 Release">Apache Arrow DataFusion 5.0.0 Release</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2021-08-18T00:00:00+00:00"> Wed 18 August 2021 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+<pre><code>git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust …</code></pre></section>
+ <footer>
+ <p>Published: <time datetime="2021-08-18T00:00:00+00:00"> Wed 18 August 2021 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2021/08/18/datafusion-5.0.0" rel="bookmark" title="Permalink to Apache Arrow DataFusion 5.0.0 Release">Apache Arrow DataFusion 5.0.0 Release</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2115,17 +2169,17 @@ and includes 211 commits from the following 31 distinct contributors.</p>
<pre><code>$ git shortlog -sn 4.0.0..5.0.0 datafusion datafusion-cli datafusion-examples
61 Jiayu Liu
47 Andrew Lamb
- 27 …</code></pre> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2021/04/12/ballista-donation" rel="bookmark" title="Permalink to Ballista: A Distributed Scheduler for Apache Arrow">Ballista: A Distributed Scheduler for Apache Arrow</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2021-04-12T00:00:00+00:00"> Mon 12 April 2021 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/agrove.html">agrove</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+ 27 …</code></pre></section>
+ <footer>
+ <p>Published: <time datetime="2021-08-18T00:00:00+00:00"> Wed 18 August 2021 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2021/04/12/ballista-donation" rel="bookmark" title="Permalink to Ballista: A Distributed Scheduler for Apache Arrow">Ballista: A Distributed Scheduler for Apache Arrow</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2147,17 +2201,17 @@ limitations under the License.
<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
to the Apache Arrow project. </p>
<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p> </div><!-- /.entry-content -->
- </article></li>
- <li><article class="hentry">
- <header> <h2 class="entry-title"><a href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="bookmark" title="Permalink to DataFusion: A Rust-native Query Engine for Apache Arrow">DataFusion: A Rust-native Query Engine for Apache Arrow</a></h2> </header>
- <footer class="post-info">
- <time class="published" datetime="2019-02-04T00:00:00+00:00"> Mon 04 February 2019 </time>
- <address class="vcard author">By
- <a class="url fn" href="https://datafusion.apache.org/blog/author/agrove.html">agrove</a>
- </address>
- </footer><!-- /.post-info -->
- <div class="entry-content"> <!--
+on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></section>
+ <footer>
+ <p>Published: <time datetime="2021-04-12T00:00:00+00:00"> Mon 12 April 2021 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/agrove.html">agrove</a>
+ </address>
+ </footer>
+ </article>
+ <article>
+ <header> <h2><a href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="bookmark" title="Permalink to DataFusion: A Rust-native Query Engine for Apache Arrow">DataFusion: A Rust-native Query Engine for Apache Arrow</a></h2> </header>
+ <section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2177,15 +2231,22 @@ limitations under the License.
-->
<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p> </div><!-- /.entry-content -->
- </article></li>
-</ol><!-- /#posts-list -->
-</section><!-- /#content -->
- <footer id="contentinfo" class="body">
- <address id="about" class="vcard body">
- Proudly powered by <a href="https://getpelican.com/">Pelican</a>,
- which takes great advantage of <a href="https://www.python.org/">Python</a>.
- </address><!-- /#about -->
- </footer><!-- /#contentinfo -->
-</body>
+<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></section>
+ <footer>
+ <p>Published: <time datetime="2019-02-04T00:00:00+00:00"> Mon 04 February 2019 </time></p>
+ <address>By
+ <a href="https://datafusion.apache.org/blog/author/agrove.html">agrove</a>
+ </address>
+ </footer>
+ </article>
+
+
+ </main>
+ <footer>
+ <address>
+ Proudly powered by <a rel="nofollow" href="https://getpelican.com/">Pelican</a>,
+ which takes great advantage of <a rel="nofollow" href="https://www.python.org/">Python</a>.
+ </address>
+ </footer>
+ </body>
</html>
\ No newline at end of file
diff --git a/blog/feed.xml b/blog/feed.xml
index 23be371..226022a 100644
--- a/blog/feed.xml
+++ b/blog/feed.xml
@@ -1,5 +1,52 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Tue, 31 Mar 2026 00:00:00 +0000</lastBuildDate><item><title>Writing Custom Table Providers in Apache DataFusion</title><link>https://datafusion.apache.org/blog/2026/03/31/writing-table-providers</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feed.xml" rel="self"/><lastBuildDate>Sat, 18 Apr 2026 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet 0.15.0 Release</title><link>https://datafusion.apache.org/blog/2026/04/18/datafusion-comet-0.15.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.15.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately four weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Sat, 18 Apr 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-04-18:/blog/2026/04/18/datafusion-comet-0.15.0</guid><category>blog</category></item><item><title>Apache DataFusion 53.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/04/02/datafusion-53.0.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/53.0.0">DataFusion 53.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0/">DataFusion 52.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md#credits">114 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p><img alt="Performance over time" class="img-fluid" src="/blog/images/datafusion-53.0.0/performance_over_time_clickbench.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: Average …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 02 Apr 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-04-02:/blog/2026/04/02/datafusion-53.0.0</guid><category>blog</category></item><item><title>Writing Custom Table Providers in Apache DataFusion</title><link>https://datafusion.apache.org/blog/2026/03/31/writing-table-providers</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -21,7 +68,7 @@ limitations under the License.
<p>One of DataFusion's greatest strengths is its extensibility. If your data lives
in a custom format, behind an API, or in a system that DataFusion does not
natively support, you can teach DataFusion to read it by implementing a
-<strong>custom table provider</strong>. This post walks through the three layers you …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Tim Saucer (rerun.io)</dc:creator><pubDate>Tue, 31 Mar 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-03-31:/blog/2026/03/31/writing-table-providers</guid><category>blog</category></item><item><title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pru [...]
+<strong>custom table provider</strong>. This post walks through the three layers you …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Tim Saucer (rerun.io)</dc:creator><pubDate>Tue, 31 Mar 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-03-31:/blog/2026/03/31/writing-table-providers</guid><category>blog</category></item><item><title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</title><l [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -61,7 +108,7 @@ figcaption {
}
</style>
<p><em>Xudong Wang, <a href="https://www.massive.com/">Massive</a></em></p>
-<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">xudong</dc:creator><pubDate>Fri, 20 Mar 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-03-20:/blog/2026/03/20/limit-pruning</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.14.0 Release</title><link>https://datafusion.apache. [...]
+<p>Reading data efficiently means touching as little data as possible. The fastest I/O is the I/O you never make …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">xudong</dc:creator><pubDate>Fri, 20 Mar 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-03-20:/blog/2026/03/20/limit-pruning</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.14.0 Release</title><link>https://datafusion.apache.org/blog/2026/03/18/ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -83,7 +130,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.14.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately eight weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 18 Mar 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-03-18:/blog/2026/03/18/datafusion-comet-0.14.0</guid><category>blog</category></item><item><title>Optimizing SQL CASE Expression Evaluation</title><link>https://datafusion.apache.org/blog/2026/02/02/datafusion_case</li [...]
+<p>This release covers approximately eight weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 18 Mar 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-03-18:/blog/2026/03/18/datafusion-comet-0.14.0</guid><category>blog</category></item><item><title>Optimizing SQL CASE Expression Evaluation</title><link>https://datafusion.apache.org/blog/2026/02/02/datafusion_case</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -123,7 +170,7 @@ figcaption {
}
</style>
<p>SQL's <code>CASE</code> expression is one of the few explicit conditional evaluation constructs the language provides.
-It allows you to control which expression from a …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Pepijn Van Eeckhoudt</dc:creator><pubDate>Mon, 02 Feb 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-02-02:/blog/2026/02/02/datafusion_case</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.13.0 Release</title><link>https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0</link><des [...]
+It allows you to control which expression from a …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Pepijn Van Eeckhoudt</dc:creator><pubDate>Mon, 02 Feb 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-02-02:/blog/2026/02/02/datafusion_case</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.13.0 Release</title><link>https://datafusion.apache.org/blog/2026/01/30/datafusion-comet-0.13.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -145,7 +192,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.13.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately eight weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 30 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-30:/blog/2026/01/30/datafusion-comet-0.13.0</guid><category>blog</category></item><item><title>Apache DataFusion 52.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0</link><de [...]
+<p>This release covers approximately eight weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 30 Jan 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-01-30:/blog/2026/01/30/datafusion-comet-0.13.0</guid><category>blog</category></item><item><title>Apache DataFusion 52.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -169,7 +216,7 @@ some of the major improvements since <a href="https://datafusion.apache.org/b
changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-52/dev/changelog/52.0.0.md#credits">121 contributors</a> for
making this release possible.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/datafusion-52.0.0</guid><category>blog</category></item><item><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link>https://datafusion.apache.org/blog/2026/01/12/extending-sql</link><description><!--
+<p>We continue to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/datafusion-52.0.0</guid><category>blog</category></item><item><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link>https://datafusion.apache.org/blog/2026/01/12/extending-sql</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -189,7 +236,7 @@ limitations under the License.
-->
<p>If you embed <a href="https://datafusion.apache.org/">DataFusion</a> in your product, your users will eventually run SQL that DataFusion does not recognize. Not because the query is unreasonable, but because SQL in practice includes many dialects and system-specific statements.</p>
-<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Geoffrey Claude (Datadog)</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/extending-sql</guid><category>blog</category></item><item><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</titl [...]
+<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Geoffrey Claude (Datadog)</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/extending-sql</guid><category>blog</category></item><item><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link>https://data [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -207,10 +254,10 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
-Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gene Bordegaray</dc:creator><pubDate>Mon, 15 Dec 2025 00:00 [...]
+Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gene Bordegaray</dc:creator><pubDate>Mon, 15 Dec 2025 00:00 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -232,7 +279,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.12.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately four weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 04 Dec 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-12-04:/blog/2025/12/04/datafusion-comet-0.12.0</guid><category>blog</category></item><item><title>Apache DataFusion 51.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0</link><des [...]
+<p>This release covers approximately four weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 04 Dec 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-12-04:/blog/2025/12/04/datafusion-comet-0.12.0</guid><category>blog</category></item><item><title>Apache DataFusion 51.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -257,7 +304,7 @@ some of the major improvements since <a href="https://datafusion.apache.org/b
changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md#credits">128 contributors</a> for
making this release possible.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>We continue …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 25 Nov 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-11-25:/blog/2025/11/25/datafusion-51.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.11.0 Release</title><link>https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0</link><description><!--
+<p>We continue …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 25 Nov 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-11-25:/blog/2025/11/25/datafusion-51.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.11.0 Release</title><link>https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -279,7 +326,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately five weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 21 Oct 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-10-21:/blog/2025/10/21/datafusion-comet-0.11.0</guid><category>blog</category></item><item><title>Apache DataFusion 50.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0</link><des [...]
+<p>This release covers approximately five weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 21 Oct 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-10-21:/blog/2025/10/21/datafusion-comet-0.11.0</guid><category>blog</category></item><item><title>Apache DataFusion 50.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -304,7 +351,7 @@ limitations under the License.
highlights some of the major improvements since the release of <a href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0/">DataFusion
49.0.0</a>. The complete list of changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-50/dev/changelog/50.0.0.md">changelog</a>.
Thanks to <a href="https://github.com/apache/datafusion/blob/branch-50/dev/changelog/50.0.0.md#credits">numerous contributors</a> for making this release possible!</p>
-<h2 id="performance-improvements">Performance …</h2></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 29 Sep 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-09-29:/blog/2025/09/29/datafusion-50.0.0</guid><category>blog</category></item><item><title>Implementing User Defined Types and Custom Metadata in DataFusion</title><link>https://datafusion.apache.org/blog/2025/09/21/custom-types-using [...]
+<h2 id="performance-improvements">Performance …</h2></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 29 Sep 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-09-29:/blog/2025/09/29/datafusion-50.0.0</guid><category>blog</category></item><item><title>Implementing User Defined Types and Custom Metadata in DataFusion</title><link>https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata</link><des [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -326,7 +373,7 @@ defined types and metadata. The user defined function APIs let users access
metadata on the input columns to functions and produce metadata in the output.</p>
<h2 id="user-defined-types-extension-types">User defined types == extension types<a class="headerlink" href="#user-defined-types-extension-types" title="Permanent link">¶</a></h2>
<p>DataFusion directly uses <a href="https://arrow.apache.org">Apache Arrow</a>'s <a href="https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html">DataTypes</a> as its type system. This
-has …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</dc:creator><pubDate>Sun, 21 Sep 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-09-21:/blog/2025/09/21/custom-types-using-metadata</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.10.0 Release</title><link>https://datafusion.apache.org/blog/2025/09/16/datafusion-co [...]
+has …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Tim Saucer(rerun.io), Dewey Dunnington(Wherobots), Andrew Lamb(InfluxData)</dc:creator><pubDate>Sun, 21 Sep 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-09-21:/blog/2025/09/21/custom-types-using-metadata</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.10.0 Release</title><link>https://datafusion.apache.org/blog/2025/09/16/datafusion-comet-0.10.0</link><de [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -348,7 +395,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.10.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately ten weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 16 Sep 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-09-16:/blog/2025/09/16/datafusion-comet-0.10.0</guid><category>blog</category></item><item><title>Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</title><link>https://datafu [...]
+<p>This release covers approximately ten weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 16 Sep 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-09-16:/blog/2025/09/16/datafusion-comet-0.10.0</guid><category>blog</category></item><item><title>Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</title><link>https://datafusion.apache.org/blog [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -378,7 +425,7 @@ optimizations work, and how we implemented them with the <a href="https://dat
community to improve performance by an order of magnitude for some query
patterns.</p>
<h2 id="motivation-and-results">Motivation and Results<a class="headerlink" href="#motivation-and-results" title="Permanent link">¶</a></h2>
-<p>The …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</dc:creator><pubDate>Wed, 10 Sep 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-09-10:/blog/2025/09/10/dynamic-filters</guid><category>blog</category></item><item><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</title><link>https://datafu [...]
+<p>The …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</dc:creator><pubDate>Wed, 10 Sep 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-09-10:/blog/2025/09/10/dynamic-filters</guid><category>blog</category></item><item><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</title><link>https://datafusion.apache.org/blog [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -403,7 +450,7 @@ metadata and is limited to indexing structures provided by the format. In fact,
caching parsed metadata and using custom external indexes along with
Parquet's hierarchical data organization can significantly speed up query
processing.</p>
-<p>In this blog, I describe …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb (InfluxData)</dc:creator><pubDate>Fri, 15 Aug 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-08-15:/blog/2025/08/15/external-parquet-indexes</guid><category>blog</category></item><item><title>Apache DataFusion 49.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0</link><description><!--
+<p>In this blog, I describe …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb (InfluxData)</dc:creator><pubDate>Fri, 15 Aug 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-08-15:/blog/2025/08/15/external-parquet-indexes</guid><category>blog</category></item><item><title>Apache DataFusion 49.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -427,7 +474,7 @@ limitations under the License.
<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/49.0.0">DataFusion 49.0.0</a>. This blog post highlights some of
the major improvements since the release of <a href="https://datafusion.apache.org/blog/2025/07/18/datafusion-48.0.0/">DataFusion 48.0.0</a>. The complete list of changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-49/dev/changelog/49.0.0.md">changelog</a>.</p>
<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
-<p>DataFusion continues to focus on enhancing performance, as …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 28 Jul 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-07-28:/blog/2025/07/28/datafusion-49.0.0</guid><category>blog</category></item><item><title>Apache DataFusion 48.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0</link><descriptio [...]
+<p>DataFusion continues to focus on enhancing performance, as …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 28 Jul 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-07-28:/blog/2025/07/28/datafusion-49.0.0</guid><category>blog</category></item><item><title>Apache DataFusion 48.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -449,7 +496,7 @@ limitations under the License.
improvements and fixes. You can find the complete details in the full
<a href="https://github.com/apache/datafusion/blob/branch-48/dev/changelog/48.0.0.md">changelog</a>. We’ll highlight the most
important changes below and guide you through upgrading.</p>
-<h2 id="breaking-changes">Breaking …</h2></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">PMC</dc:creator><pubDate>Wed, 16 Jul 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-07-16:/blog/2025/07/16/datafusion-48.0.0</guid><category>blog</category></item><item><title>Embedding User-Defined Indexes in Apache Parquet Files</title><link>https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes</link><desc [...]
+<h2 id="breaking-changes">Breaking …</h2></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">PMC</dc:creator><pubDate>Wed, 16 Jul 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-07-16:/blog/2025/07/16/datafusion-48.0.0</guid><category>blog</category></item><item><title>Embedding User-Defined Indexes in Apache Parquet Files</title><link>https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -468,7 +515,7 @@ limitations under the License.
{% endcomment %}
-->
-<p>It’s a common misconception that <a href="https://parquet.apache.org/">Apache Parquet</a> files are limited to basic Min/Max/Null Count statistics and Bloom filters, and that adding more advanced indexes requires changing the specification or creating a new file format. In fact, footer metadata and offset-based addressing already provide everything needed to embed …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Qi Zhu (Cloudera), Jiga [...]
+<p>It’s a common misconception that <a href="https://parquet.apache.org/">Apache Parquet</a> files are limited to basic Min/Max/Null Count statistics and Bloom filters, and that adding more advanced indexes requires changing the specification or creating a new file format. In fact, footer metadata and offset-based addressing already provide everything needed to embed …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Qi Zhu (Cloudera), Jiga [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -489,7 +536,7 @@ limitations under the License.
<p>We’re excited to announce the release of <strong>Apache DataFusion 47.0.0</strong>! This new version represents a significant
milestone for the project, packing in a wide range of improvements and fixes. You can find the complete details in the
full <a href="https://github.com/apache/datafusion/blob/branch-47/dev/changelog/47.0.0.md">changelog</a>. We’ll highlight the most
-important changes below …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">PMC</dc:creator><pubDate>Fri, 11 Jul 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-07-11:/blog/2025/07/11/datafusion-47.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.9.0 Release</title><link>https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0</link><description><!--
+important changes below …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">PMC</dc:creator><pubDate>Fri, 11 Jul 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-07-11:/blog/2025/07/11/datafusion-47.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.9.0 Release</title><link>https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -511,7 +558,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.9.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately ten weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 01 Jul 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-07-01:/blog/2025/07/01/datafusion-comet-0.9.0</guid><category>blog</category></item><item><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</title><link>https://datafusion.apache.org/blog/20 [...]
+<p>This release covers approximately ten weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 01 Jul 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-07-01:/blog/2025/07/01/datafusion-comet-0.9.0</guid><category>blog</category></item><item><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</title><link>https://datafusion.apache.org/blog/2025/06/30/cancellatio [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -551,7 +598,7 @@ figcaption {
}
</style>
<p>Have you ever tried to cancel a query that just wouldn't stop?
-In this post, we'll review how Rust's <a href="https://doc.rust-lang.org/book/ch17-00-async-await.html"><code>async</code> programming model</a> works, how …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Pepijn Van Eeckhoudt</dc:creator><pubDate>Mon, 30 Jun 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-06-30:/blog/2025/06/30/cancellation</guid><category>blog</category></item><item><title>Opti [...]
+In this post, we'll review how Rust's <a href="https://doc.rust-lang.org/book/ch17-00-async-await.html"><code>async</code> programming model</a> works, how …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Pepijn Van Eeckhoudt</dc:creator><pubDate>Mon, 30 Jun 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-06-30:/blog/2025/06/30/cancellation</guid><category>blog</category></item><item><title>Optimizing SQL (and Data [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -578,7 +625,7 @@ science,”</a> according to Father
Pavlo, or some behind-the-scenes player. We believe this perception is because:</p>
<ol>
<li>
-<p>One must implement the rest of a …</p></li></ol></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, akurmustafa</dc:creator><pubDate>Sun, 15 Jun 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-06-15:/blog/2025/06/15/optimizing-sql-dataframes-part-one</guid><category>blog</category></item><item><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link> [...]
+<p>One must implement the rest of a …</p></li></ol></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, akurmustafa</dc:creator><pubDate>Sun, 15 Jun 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-06-15:/blog/2025/06/15/optimizing-sql-dataframes-part-one</guid><category>blog</category></item><item><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link>https://datafusion.a [...]
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -601,7 +648,7 @@ limitations under the License.
<p>In the <a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one">first part of this post</a>, we discussed what a Query Optimizer is, what
role it plays, and described how industrial optimizers are organized. In this
second post, we describe various optimizations that are found in <a href="https://datafusion.apache.org/">Apache
-DataFusion</a> and …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, akurmustafa</dc:creator><pubDate>Sun, 15 Jun 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-06-15:/blog/2025/06/15/optimizing-sql-dataframes-part-two</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.8.0 Release</title><link>https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0</link><description><!--
+DataFusion</a> and …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, akurmustafa</dc:creator><pubDate>Sun, 15 Jun 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-06-15:/blog/2025/06/15/optimizing-sql-dataframes-part-two</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.8.0 Release</title><link>https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -623,7 +670,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.8.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers approximately six weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 06 May 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-05-06:/blog/2025/05/06/datafusion-comet-0.8.0</guid><category>blog</category></item><item><title>User defined Window Functions in DataFusion</title><link>https://datafusion.apache.org/blog/2025/04/19/user-defined-window- [...]
+<p>This release covers approximately six weeks of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 06 May 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-05-06:/blog/2025/05/06/datafusion-comet-0.8.0</guid><category>blog</category></item><item><title>User defined Window Functions in DataFusion</title><link>https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions</link><des [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -643,7 +690,7 @@ limitations under the License.
-->
<p>Window functions are a powerful feature in SQL, allowing for complex analytical computations over a subset of data. However, efficiently implementing them, especially sliding windows, can be quite challenging. With <a href="https://datafusion.apache.org/">Apache DataFusion</a>'s user-defined window functions, developers can easily take advantage of all the effort put into DataFusion's implementation.</p>
-<p>In …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Aditya Singh Rathore, Andrew Lamb</dc:creator><pubDate>Sat, 19 Apr 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-04-19:/blog/2025/04/19/user-defined-window-functions</guid><category>blog</category></item><item><title>tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust</title><link>https://datafusion.apache.org/blog/2025/04/10/fastes [...]
+<p>In …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Aditya Singh Rathore, Andrew Lamb</dc:creator><pubDate>Sat, 19 Apr 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-04-19:/blog/2025/04/19/user-defined-window-functions</guid><category>blog</category></item><item><title>tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust</title><link>https://datafusion.apache.org/blog/2025/04/10/fastest-tpch-generator</li [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -660,19 +707,11 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
-development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb, Achraf B, and Sean Smith</dc:creator><pubDate>Thu, 10 Apr 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-04-10:/blog/2025/04/10/fastest-tpch-generator</guid><category>blog</category></item><item><title>Apache DataFusion [...]
+development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
+faster than any other implementation we know of.</p>
+<p>It is now possible to create …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb, Achraf B, and Sean Smith</dc:creator><pubDate>Thu, 10 Apr 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-04-10:/blog/2025/04/10/fastest-tpch-generator</guid><category>blog</category></item><item><title>Apache DataFusion Python 46.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/03/30/datafusion-python-46.0.0</link><descrip [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -694,7 +733,7 @@ limitations under the License.
<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/46.0.0/">datafusion-python 46.0.0</a> has been released. This release
brings in all of the new features of the core <a href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0">DataFusion 46.0.0</a> library. Since the last
blog post for <a href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0/">datafusion-python 43.1.0</a>, a large number of improvements have been made
-that can …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Sun, 30 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-30:/blog/2025/03/30/datafusion-python-46.0.0</guid><category>blog</category></item><item><title>Apache DataFusion 46.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0</link><description><!--
+that can …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Sun, 30 Mar 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-03-30:/blog/2025/03/30/datafusion-python-46.0.0</guid><category>blog</category></item><item><title>Apache DataFusion 46.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -711,7 +750,7 @@ limitations under the License.
{% endcomment %}
-->
-<p>We’re excited to announce the release of <strong>Apache DataFusion 46.0.0</strong>! This new version represents a significant milestone for the project, packing in a wide range of improvements and fixes. You can find the complete details in the full <a href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>. We’ll highlight the most important changes below …</p></description><dc:creator xmlns:dc="http://purl. [...]
+<p>We’re excited to announce the release of <strong>Apache DataFusion 46.0.0</strong>! This new version represents a significant milestone for the project, packing in a wide range of improvements and fixes. You can find the complete details in the full <a href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>. We’ll highlight the most important changes below …</p></description><dc:creator xmlns:dc="http://purl. [...]
figure {
margin: 20px 0;
}
@@ -750,7 +789,7 @@ limitations under the License.
<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-pushdown/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
<hr/>
-<p>In the <a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning">previous post …</a></p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao</dc:creator><pubDate>Fri, 21 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-21:/blog/2025/03/21/parquet-pushdown</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.7.0 Release</title><link>https://datafusion [...]
+<p>In the <a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning">previous post …</a></p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao</dc:creator><pubDate>Fri, 21 Mar 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-03-21:/blog/2025/03/21/parquet-pushdown</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.7.0 Release</title><link>https://datafusion.apache.org/blog/202 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -772,7 +811,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 20 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-20:/blog/2025/03/20/datafusion-comet-0.7.0</guid><category>blog</category></item><item><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link>https://datafusion.apache.org/blog/2025/03/20/parquet-pruning</link><d [...]
+<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 20 Mar 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-03-20:/blog/2025/03/20/datafusion-comet-0.7.0</guid><category>blog</category></item><item><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link>https://datafusion.apache.org/blog/2025/03/20/parquet-pruning</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -794,7 +833,7 @@ limitations under the License.
<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
<hr/>
<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
-<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao</dc:creator><pubDate>Thu, 20 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-20:/blog/2025/03/20/parquet-pruning</guid><category>blog</category></item><item><title>Using Ordering for Better Plans in Apache DataFusion</title><link>https://datafusion.apache.org/blog/202 [...]
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao</dc:creator><pubDate>Thu, 20 Mar 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-03-20:/blog/2025/03/20/parquet-pruning</guid><category>blog</category></item><item><title>Using Ordering for Better Plans in Apache DataFusion</title><link>https://datafusion.apache.org/blog/2025/03/11/ordering-ana [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -816,8 +855,8 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
-<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Mustafa Akur, Andrew Lamb</dc:creator><pubDate>Tue, 11 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-11:/blog/2025/03/11/ordering-analysis</guid><category>blog</category></item><item><title>Apache DataFusion 45.0.0 Released</title><link>https: [...]
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Mustafa Akur, Andrew Lamb</dc:creator><pubDate>Tue, 11 Mar 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-03-11:/blog/2025/03/11/ordering-analysis</guid><category>blog</category></item><item><title>Apache DataFusion 45.0.0 Released</title><link>https://datafusion.apache. [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -841,7 +880,7 @@ limitations under the License.
<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
what the community is thinking about in the next 6 months. It has been an exciting
-period of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 20 Feb 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-02-20:/blog/2025/02/20/datafusion-45.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.6.0 Release</title><link>https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0</link><description><!--
+period of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 20 Feb 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-02-20:/blog/2025/02/20/datafusion-45.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.6.0 Release</title><link>https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -863,7 +902,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.6.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 17 Feb 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-02-17:/blog/2025/02/17/datafusion-comet-0.6.0</guid><category>blog</category></item><item><title>Apache DataFusion Ballista 43.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0</link><d [...]
+<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 17 Feb 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-02-17:/blog/2025/02/17/datafusion-comet-0.6.0</guid><category>blog</category></item><item><title>Apache DataFusion Ballista 43.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -885,7 +924,7 @@ limitations under the License.
<p>We are pleased to announce version <a href="https://github.com/apache/datafusion-ballista/blob/main/CHANGELOG.md#4300-2025-01-07">43.0.0</a> of the <a href="https://datafusion.apache.org/ballista/">DataFusion Ballista</a>. Ballista allows existing <a href="https://datafusion.apache.org">DataFusion</a> applications to be scaled out on a cluster for use cases that are not practical to run on a single node.</p>
<h2 id="highlights-of-this-release">Highlights of this release<a class="headerlink" href="#highlights-of-this-release" title="Permanent link">¶</a></h2>
<h3 id="seamless-integration-with-datafusion">Seamless Integration with DataFusion<a class="headerlink" href="#seamless-integration-with-datafusion" title="Permanent link">¶</a></h3>
-<p>The primary objective of …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">milenkovicm</dc:creator><pubDate>Sun, 02 Feb 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-02-02:/blog/2025/02/02/datafusion-ballista-43.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.5.0 Release</title><link>https://datafusion.apache.org/blog/2025/01/17/datafusion-comet-0.5.0</link><description><!--
+<p>The primary objective of …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">milenkovicm</dc:creator><pubDate>Sun, 02 Feb 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-02-02:/blog/2025/02/02/datafusion-ballista-43.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.5.0 Release</title><link>https://datafusion.apache.org/blog/2025/01/17/datafusion-comet-0.5.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -907,7 +946,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.5.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 17 Jan 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-01-17:/blog/2025/01/17/datafusion-comet-0.5.0</guid><category>blog</category></item><item><title>Apache DataFusion Python 43.1.0 Released</title><link>https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0</link><descr [...]
+<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 17 Jan 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-01-17:/blog/2025/01/17/datafusion-comet-0.5.0</guid><category>blog</category></item><item><title>Apache DataFusion Python 43.1.0 Released</title><link>https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -929,7 +968,7 @@ limitations under the License.
<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/43.1.0/">datafusion-python 43.1.0</a> has been released. This release
brings in all of the new features of the core <a href="https://github.com/apache/datafusion/blob/main/dev/changelog/43.0.0.md">DataFusion 43.0.0</a> library. Since the last
blog post for <a href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0/">datafusion-python 40.1.0</a>, a large number of improvements have been made
-that can …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Sat, 14 Dec 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-12-14:/blog/2024/12/14/datafusion-python-43.1.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.4.0 Release</title><link>https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0</link><description><!--
+that can …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Sat, 14 Dec 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-12-14:/blog/2024/12/14/datafusion-python-43.1.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.4.0 Release</title><link>https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -951,7 +990,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 20 Nov 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-11-20:/blog/2024/11/20/datafusion-comet-0.4.0</guid><category>blog</category></item><item><title>Comparing approaches to User Defined Functions in Apache DataFusion using Python</title><link>https://datafusion.apache.org/blog/2024/11/ [...]
+<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 20 Nov 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-11-20:/blog/2024/11/20/datafusion-comet-0.4.0</guid><category>blog</category></item><item><title>Comparing approaches to User Defined Functions in Apache DataFusion using Python</title><link>https://datafusion.apache.org/blog/2024/11/19/datafusion-python [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -974,7 +1013,7 @@ limitations under the License.
<p>For a few months now I’ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
fast query engine written in Rust. From my experience the language that nearly all data scientists
are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Tue, 19 Nov 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-11-19:/blog/2024/11/19/datafusion-python-udf-comparisons</guid><category>blog</category></item><item><title>Apache DataFusion is now the fastest single node engine for querying Apache Par [...]
+for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Tue, 19 Nov 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-11-19:/blog/2024/11/19/datafusion-python-udf-comparisons</guid><category>blog</category></item><item><title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</title><l [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -997,7 +1036,7 @@ limitations under the License.
fastest engine for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">ClickBench</a>. It is faster
than <a href="https://duckdb.org/">DuckDB</a>, <a href="https://clickhouse.com/chdb">chDB</a> and <a href="https://clickhouse.com/">Clickhouse</a> using the same hardware. It also marks
the first time a <a href="https://www.rust-lang.org/">Rust</a>-based engine holds the top spot, which has previously
-been …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb, Staff Engineer at InfluxData</dc:creator><pubDate>Mon, 18 Nov 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-11-18:/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.3.0 Release</title><link>https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0</ [...]
+been …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb, Staff Engineer at InfluxData</dc:creator><pubDate>Mon, 18 Nov 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-11-18:/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.3.0 Release</title><link>https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1019,7 +1058,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.3.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 27 Sep 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-09-27:/blog/2024/09/27/datafusion-comet-0.3.0</guid><category>blog</category></item><item><title>Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</title><link>https://datafusion.apache.org/blog/2 [...]
+<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 27 Sep 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-09-27:/blog/2024/09/27/datafusion-comet-0.3.0</guid><category>blog</category></item><item><title>Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</title><link>https://datafusion.apache.org/blog/2024/09/13/string-vie [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1039,7 +1078,7 @@ limitations under the License.
-->
<p><em>Editor's Note: This is the first of a <a href="../string-view-german-style-strings-part-2/">two part</a> blog series that was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao, Andrew Lamb</dc:creator><pubDate>Fri, 13 Sep [...]
+<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao, Andrew Lamb</dc:creator><pubDate>Fri, 13 Sep [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1060,7 +1099,7 @@ limitations under the License.
<p><em>Editor's Note: This blog series was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
<p>In the <a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1">first post</a>, we discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies.
-In this second …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao, Andrew Lamb</dc:creator><pubDate>Fri, 13 Sep 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-09-13:/blog/2024/09/13/string-view-german-style-strings-part-2</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.2.0 Release</title><link>https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0</link><descripti [...]
+In this second …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao, Andrew Lamb</dc:creator><pubDate>Fri, 13 Sep 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-09-13:/blog/2024/09/13/string-view-german-style-strings-part-2</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.2.0 Release</title><link>https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1082,7 +1121,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce version 0.2.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 28 Aug 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-08-28:/blog/2024/08/28/datafusion-comet-0.2.0</guid><category>blog</category></item><item><title>Apache DataFusion Python 40.1.0 Released, Significant usability updates</title><link>https://datafusion.apache.org/blog/2024/08/20/python [...]
+<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 28 Aug 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-08-28:/blog/2024/08/28/datafusion-comet-0.2.0</guid><category>blog</category></item><item><title>Apache DataFusion Python 40.1.0 Released, Significant usability updates</title><link>https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0</ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1104,7 +1143,7 @@ limitations under the License.
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/40.1.0/">DataFusion in Python 40.1.0</a> has been released. In addition to
bringing in all of the new features of the core <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> package, this release
-contains <em>significant</em> updates to the user interface and documentation. We listened to the python …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Tue, 20 Aug 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-08-20:/blog/2024/08/20/python-datafusion-40.0.0</guid><category>blog</category></item><item><title>Apache DataFusion 40.0.0 Released</title><link>https://datafusion. [...]
+contains <em>significant</em> updates to the user interface and documentation. We listened to the python …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Tue, 20 Aug 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-08-20:/blog/2024/08/20/python-datafusion-40.0.0</guid><category>blog</category></item><item><title>Apache DataFusion 40.0.0 Released</title><link>https://datafusion.apache.org/blog/2024 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1128,7 +1167,7 @@ limitations under the License.
<p>We are proud to announce <a href="https://crates.io/crates/datafusion/40.0.0">DataFusion 40.0.0</a>. This blog highlights some of the
many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0/">DataFusion 34.0.0</a> and a preview of
what the community is thinking about in the next 6 months. We are hoping to make
-more regular blog posts …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 24 Jul 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-07-24:/blog/2024/07/24/datafusion-40.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.1.0 Release</title><link>https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0</link><description><!--
+more regular blog posts …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 24 Jul 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-07-24:/blog/2024/07/24/datafusion-40.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.1.0 Release</title><link>https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1150,7 +1189,7 @@ limitations under the License.
<p>The Apache DataFusion PMC is pleased to announce the first official source release of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Sat, 20 Jul 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-07-20:/blog/2024/07/20/datafusion-comet-0.1.0</guid><category>blog</category></item><item><title>Announcing Apache Arrow DataFusion is now Apache DataFusion</title><link>https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp</link> [...]
+<p>Comet runs on commodity hardware and aims …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Sat, 20 Jul 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-07-20:/blog/2024/07/20/datafusion-comet-0.1.0</guid><category>blog</category></item><item><title>Announcing Apache Arrow DataFusion is now Apache DataFusion</title><link>https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1175,7 +1214,7 @@ limitations under the License.
April 16, 2024 the Apache Arrow DataFusion subproject is now a top level
<a href="https://www.apache.org/">Apache Software Foundation</a> project.</p>
<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h2>
-<p>Apache DataFusion is a fast, extensible query engine for building …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 07 May 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-05-07:/blog/2024/05/07/datafusion-tlp</guid><category>blog</category></item><item><title>Announcing Apache Arrow DataFusion Comet</title><link>https://datafusion.apache.org/blog/2024/03/06/comet-donation</link><de [...]
+<p>Apache DataFusion is a fast, extensible query engine for building …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 07 May 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-05-07:/blog/2024/05/07/datafusion-tlp</guid><category>blog</category></item><item><title>Announcing Apache Arrow DataFusion Comet</title><link>https://datafusion.apache.org/blog/2024/03/06/comet-donation</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1199,7 +1238,7 @@ limitations under the License.
a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/datafusion">Apache Arrow DataFusion</a>.</p>
<p>Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to
accelerate Spark workloads. It is designed as a drop-in
-replacement for Spark's JVM …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 06 Mar 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-03-06:/blog/2024/03/06/comet-donation</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</title><link>https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0</link><description><!--
+replacement for Spark's JVM …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 06 Mar 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-03-06:/blog/2024/03/06/comet-donation</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</title><link>https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1222,7 +1261,7 @@ limitations under the License.
<p>We recently <a href="https://crates.io/crates/datafusion/34.0.0">released DataFusion 34.0.0</a>. This blog highlights some of the major
improvements since we <a href="https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/.">released DataFusion 26.0.0</a> (spoiler alert there are many)
and a preview of where the community plans to focus in the next 6 months.</p>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 19 Jan 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-01-19:/blog/2024/01/19/datafusion-34.0.0</guid><category>blog</category></item><item><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title> [...]
+<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 19 Jan 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-01-19:/blog/2024/01/19/datafusion-34.0.0</guid><category>blog</category></item><item><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link>https://datafu [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1246,7 +1285,7 @@ limitations under the License.
<p>Andrew Lamb, Daniël Heres, Raphael Taylor-Davies,</p>
<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
<h2 id="tldr">TLDR<a class="headerlink" href="#tldr" title="Permanent link">¶</a></h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, Dandandan, tustvold</dc:creator><pubDate>Sat, 05 Aug 2023 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2023-08-05:/blog/2023/08/ [...]
+<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, Dandandan, tustvold</dc:creator><pubDate>Sat, 05 Aug 2023 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2023-08-05:/blog/2023/08/05/datafusion_fast_g [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1271,7 +1310,7 @@ performance improved and we are <a href="https://github.com/apache/arrow-data
top level Apache Project.</p>
<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h2>
<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database
-toolkit …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Sat, 24 Jun 2023 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2023-06-24:/blog/2023/06/24/datafusion-25.0.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 16.0.0 Project Update</title><link>https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0</link><description><!--
+toolkit …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Sat, 24 Jun 2023 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2023-06-24:/blog/2023/06/24/datafusion-25.0.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 16.0.0 Project Update</title><link>https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1298,7 +1337,7 @@ in-memory format. It is targeted primarily at developers creating data
intensive analytics, and offers mature
<a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>,
a DataFrame API, and many extension points.</p>
-<p>Systems based on DataFusion perform very well in benchmarks …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 19 Jan 2023 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2023-01-19:/blog/2023/01/19/datafusion-16.0.0</guid><category>blog</category></item><item><title>Apache Arrow Ballista 0.9.0 Release</title><link>https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0</link><descriptio [...]
+<p>Systems based on DataFusion perform very well in benchmarks …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 19 Jan 2023 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2023-01-19:/blog/2023/01/19/datafusion-16.0.0</guid><category>blog</category></item><item><title>Apache Arrow Ballista 0.9.0 Release</title><link>https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1322,7 +1361,7 @@ limitations under the License.
<p>Ballista 0.9.0 is now available and is the most significant release since the project was <a href="http://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to Apache
Arrow in 2021.</p>
<p>This release represents 4 weeks of work, with 66 commits from 14 contributors:</p>
-<pre><code> 22 Andy …</code></pre></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 28 Oct 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2022-10-28:/blog/2022/10/28/ballista-0.9.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 13.0.0 Project Update</title><link>https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0</link><description><!--
+<pre><code> 22 Andy …</code></pre></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 28 Oct 2022 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2022-10-28:/blog/2022/10/28/ballista-0.9.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 13.0.0 Project Update</title><link>https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1343,7 +1382,7 @@ limitations under the License.
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> <a href="https://crates.io/crates/datafusion"><code>13.0.0</code></a> is released, and this blog contains an update on the project for the 5 months since our <a href="https://arrow.apache.org/blog/2022/05/16/datafusion-8.0.0/">last update in May 2022</a>.</p>
-<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 25 Oct 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2022-10-25:/blog/2022/10/25/datafusion-13.0.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 8.0.0 Release</title><link>h [...]
+<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 25 Oct 2022 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2022-10-25:/blog/2022/10/25/datafusion-13.0.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 8.0.0 Release</title><link>https://datafusion.ap [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1366,7 +1405,7 @@ limitations under the License.
<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that
uses Apache Arrow as its in-memory format.</p>
<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>,
-a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 16 May 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2022-05-16:/blog/2022/05/16/datafusion-8.0.0</guid><category>blog</category></item><item><title>Introducing Apache Arrow DataFusion Contrib</title><link>https://datafusion.apach [...]
+a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 16 May 2022 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2022-05-16:/blog/2022/05/16/datafusion-8.0.0</guid><category>blog</category></item><item><title>Introducing Apache Arrow DataFusion Contrib</title><link>https://datafusion.apache.org/blog/2022/03/2 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1386,7 +1425,7 @@ limitations under the License.
-->
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">&para;</a></h1>
<p>Apache Arrow <a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 21 Mar 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2022-03-21:/blog/2022/03/21/data [...]
+<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 21 Mar 2022 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2022-03-21:/blog/2022/03/21/datafusion-contrib</guid [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1406,7 +1445,7 @@ limitations under the License.
-->
<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">&para;</a></h1>
<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 28 Feb 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2022-02-28:/blo [...]
+<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 28 Feb 2022 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2022-02-28:/blog/2022/02/28/datafus [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1430,7 +1469,7 @@ query engine which leverages the unique features of
<a href="https://www.rust-lang.org/">Rust</a> and <a href="https://arrow.apache.org/">Apache
Arrow</a> to provide a system that is high
performance, easy to connect, easy to embed, and high quality.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 19 Nov 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2021-11-19:/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md</guid><category>blog</category></item><item><title>Apache Arrow Ballista 0.5.0 Release</title><link>https://datafusion.apache.org/blog/2021 [...]
+<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 19 Nov 2021 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2021-11-19:/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md</guid><category>blog</category></item><item><title>Apache Arrow Ballista 0.5.0 Release</title><link>https://datafusion.apache.org/blog/2021/08/18/ballista-0.5. [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1452,7 +1491,7 @@ limitations under the License.
<p>Ballista extends DataFusion to provide support for distributed queries. This is the first release of Ballista since
the project was <a href="https://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to the Apache Arrow project
and includes 80 commits from 11 contributors.</p>
-<pre><code>git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust …</code></pre></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 18 Aug 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/ballista-0.5.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 5.0.0 Release</title><link>https://datafusion.apache. [...]
+<pre><code>git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust …</code></pre></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 18 Aug 2021 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/ballista-0.5.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 5.0.0 Release</title><link>https://datafusion.apache.org/blog/2021/08/18/ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1475,7 +1514,7 @@ and includes 211 commits from the following 31 distinct contributors.</p>
<pre><code>$ git shortlog -sn 4.0.0..5.0.0 datafusion datafusion-cli datafusion-examples
61 Jiayu Liu
47 Andrew Lamb
- 27 …</code></pre></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 18 Aug 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/datafusion-5.0.0</guid><category>blog</category></item><item><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link>https://datafusion.apache.org/blog/2021/04/12/ballista-donation</link><description><!--
+ 27 …</code></pre></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 18 Aug 2021 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/datafusion-5.0.0</guid><category>blog</category></item><item><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link>https://datafusion.apache.org/blog/2021/04/12/ballista-donation</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1497,7 +1536,7 @@ limitations under the License.
<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
to the Apache Arrow project. </p>
<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 12 Apr 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2021-04-12:/blog/2021/04/12/ballista-donation</guid><category>blog</category></item><item><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link>https://datafusio [...]
+on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 12 Apr 2021 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2021-04-12:/blog/2021/04/12/ballista-donation</guid><category>blog</category></item><item><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link>https://datafusion.apache.org/blog/20 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1517,4 +1556,4 @@ limitations under the License.
-->
<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 04 Feb 2019 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 04 Feb 2019 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/aditya-singh-rathore-andrew-lamb.atom.xml b/blog/feeds/aditya-singh-rathore-andrew-lamb.atom.xml
index 36189b0..87f071f 100644
--- a/blog/feeds/aditya-singh-rathore-andrew-lamb.atom.xml
+++ b/blog/feeds/aditya-singh-rathore-andrew-lamb.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Aditya Singh Rathore, Andrew Lamb</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/aditya-singh-rathore-andrew-lamb.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-04-19T00:00:00+00:00</updated><subtitle></subtitle><entry><title>User defined Window Functions in DataFusion</title><link href="https [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Aditya Singh Rathore, Andrew Lamb</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/aditya-singh-rathore-andrew-lamb.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2025-04-19T00:00:00+00:00</updated><entry><title>User defined Window Functions in DataFusion</title><link href="https://datafusion.apache.org/blog/202 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -352,4 +352,4 @@ giving it a try. This post was designed to make it easier for new users to work
<p>When it comes to designing UDFs, I strongly recommend reviewing the
<a href="https://datafusion.apache.org/library-user-guide/adding-udfs.html">Window functions</a> documentation.</p>
<p>A heartfelt thank you to <a href="https://github.com/alamb">@alamb</a> and <a href="https://github.com/andygrove">@andygrove</a> for their invaluable reviews and thoughtful feedback—they’ve been instrumental in shaping this post.</p>
-<p>The Apache Arrow and Apache DataFusion communities are vibrant, welcoming, and full of passionate developers building something truly powerful. If you’re excited about high-performance analytics and want to be part of an open-source journey, I highly encourage you to explore the <a href="(https://datafusion.apache.org/)">official documentation</a> and dive into one of the many <a href="https://github.com/apache/datafusion/issues">open issues</a>. There’s [...]
\ No newline at end of file
+<p>The Apache Arrow and Apache DataFusion communities are vibrant, welcoming, and full of passionate developers building something truly powerful. If you’re excited about high-performance analytics and want to be part of an open-source journey, I highly encourage you to explore the <a href="(https://datafusion.apache.org/)">official documentation</a> and dive into one of the many <a href="https://github.com/apache/datafusion/issues">open issues</a>. There’s [...]
\ No newline at end of file
diff --git a/blog/feeds/aditya-singh-rathore-andrew-lamb.rss.xml b/blog/feeds/aditya-singh-rathore-andrew-lamb.rss.xml
index ae79149..aeb0c1c 100644
--- a/blog/feeds/aditya-singh-rathore-andrew-lamb.rss.xml
+++ b/blog/feeds/aditya-singh-rathore-andrew-lamb.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - Aditya Singh Rathore, Andrew Lamb</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Sat, 19 Apr 2025 00:00:00 +0000</lastBuildDate><item><title>User defined Window Functions in DataFusion</title><link>https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - Aditya Singh Rathore, Andrew Lamb</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/aditya-singh-rathore-andrew-lamb.rss.xml" rel="self"/><lastBuildDate>Sat, 19 Apr 2025 00:00:00 +0000</lastBuildDate><item><title>User defined Window Functions in DataFusion</title><link>https://datafusion.apache.org/blog/2025/04/1 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -19,4 +19,4 @@ limitations under the License.
-->
<p>Window functions are a powerful feature in SQL, allowing for complex analytical computations over a subset of data. However, efficiently implementing them, especially sliding windows, can be quite challenging. With <a href="https://datafusion.apache.org/">Apache DataFusion</a>'s user-defined window functions, developers can easily take advantage of all the effort put into DataFusion's implementation.</p>
-<p>In …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Aditya Singh Rathore, Andrew Lamb</dc:creator><pubDate>Sat, 19 Apr 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-04-19:/blog/2025/04/19/user-defined-window-functions</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+<p>In …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Aditya Singh Rathore, Andrew Lamb</dc:creator><pubDate>Sat, 19 Apr 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-04-19:/blog/2025/04/19/user-defined-window-functions</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.atom.xml b/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.atom.xml
index 32f3135..5b26ace 100644
--- a/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.atom.xml
+++ b/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-09-10T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Dynamic Filters: P [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2025-09-10T00:00:00+00:00</updated><entry><title>Dynamic Filters: Passing Information Between Operat [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -572,4 +572,4 @@ LIMIT 10;
<td style="text-align: right;">0.602</td>
</tr>
</tbody>
-</table></content><category term="blog"></category></entry></feed>
\ No newline at end of file
+</table></content><category term="blog"/></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.rss.xml b/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.rss.xml
index 1dcad1e..7cc4660 100644
--- a/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.rss.xml
+++ b/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Wed, 10 Sep 2025 00:00:00 +0000</lastBuildDate><item><title>Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</title><link>https://datafusion.apache.org/blog/2025/09/10/dynamic-filters</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/adrian-garcia-badaracco-pydantic-andrew-lamb-influxdata.rss.xml" rel="self"/><lastBuildDate>Wed, 10 Sep 2025 00:00:00 +0000</lastBuildDate><item><title>Dynamic Filters: Passing Information Between Operat [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -29,4 +29,4 @@ optimizations work, and how we implemented them with the <a href="https://dat
community to improve performance by an order of magnitude for some query
patterns.</p>
<h2 id="motivation-and-results">Motivation and Results<a class="headerlink" href="#motivation-and-results" title="Permanent link">¶</a></h2>
-<p>The …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</dc:creator><pubDate>Wed, 10 Sep 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-09-10:/blog/2025/09/10/dynamic-filters</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+<p>The …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Adrian Garcia Badaracco (Pydantic), Andrew Lamb (InfluxData)</dc:creator><pubDate>Wed, 10 Sep 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-09-10:/blog/2025/09/10/dynamic-filters</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/agrove.atom.xml b/blog/feeds/agrove.atom.xml
index 17af026..2cdfd85 100644
--- a/blog/feeds/agrove.atom.xml
+++ b/blog/feeds/agrove.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - agrove</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/agrove.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2021-04-12T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2021/04/12/balli [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - agrove</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/agrove.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2021-04-12T00:00:00+00:00</updated><entry><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2021/04/12/ballista-donation" rel="alternate"/><p [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -71,7 +71,7 @@ datasets and file bug reports for any issues that you find. You could also check
<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20component%20%3D%20%22Rust%20-%20Ballista%22">list of issues</a> and have a go at fixing one.</p>
<p>The <a href="https://github.com/apache/arrow/blob/master/rust/README.md#arrow-rust-community">Arrow Rust Community</a>
section of the Rust README provides information on other ways to interact with the Ballista contributors and
-maintainers.</p></content><category term="blog"></category></entry><entry><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="alternate"></link><published>2019-02-04T00:00:00+00:00</published><updated>2019-02-04T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</id><summary type="html"><!--
+maintainers.</p></content><category term="blog"/></entry><entry><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="alternate"/><published>2019-02-04T00:00:00+00:00</published><updated>2019-02-04T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -190,4 +190,4 @@ while let Some(batch) = results.next().unwrap() {
<li>Creating a Docker image to make it easy to use DataFusion as a standalone query tool for interactive and batch queries</li>
</ul>
<h2 id="contributors-welcome">Contributors Welcome!<a class="headerlink" href="#contributors-welcome" title="Permanent link">¶</a></h2>
-<p>If you are excited about being able to use Rust for data science and would like to contribute to this work then there are many ways to get involved. The simplest way to get started is to try out DataFusion against your own data sources and file bug reports for any issues that you find. You could also check out the current <a href="https://cwiki.apache.org/confluence/display/ARROW/Rust+JIRA+Dashboard">list of issues</a> and have a go at fixing one. You can also join t [...]
\ No newline at end of file
+<p>If you are excited about being able to use Rust for data science and would like to contribute to this work then there are many ways to get involved. The simplest way to get started is to try out DataFusion against your own data sources and file bug reports for any issues that you find. You could also check out the current <a href="https://cwiki.apache.org/confluence/display/ARROW/Rust+JIRA+Dashboard">list of issues</a> and have a go at fixing one. You can also join t [...]
\ No newline at end of file
diff --git a/blog/feeds/agrove.rss.xml b/blog/feeds/agrove.rss.xml
index ccc1aee..0324a34 100644
--- a/blog/feeds/agrove.rss.xml
+++ b/blog/feeds/agrove.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - agrove</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon, 12 Apr 2021 00:00:00 +0000</lastBuildDate><item><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link>https://datafusion.apache.org/blog/2021/04/12/ballista-donation</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - agrove</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/agrove.rss.xml" rel="self"/><lastBuildDate>Mon, 12 Apr 2021 00:00:00 +0000</lastBuildDate><item><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link>https://datafusion.apache.org/blog/2021/04/12/ballista-donation</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -21,7 +21,7 @@ limitations under the License.
<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
to the Apache Arrow project. </p>
<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 12 Apr 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2021-04-12:/blog/2021/04/12/ballista-donation</guid><category>blog</category></item><item><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link>https://datafusio [...]
+on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 12 Apr 2021 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2021-04-12:/blog/2021/04/12/ballista-donation</guid><category>blog</category></item><item><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link>https://datafusion.apache.org/blog/20 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -41,4 +41,4 @@ limitations under the License.
-->
<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 04 Feb 2019 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 04 Feb 2019 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/alamb-akurmustafa.atom.xml b/blog/feeds/alamb-akurmustafa.atom.xml
index 88d85d0..20d3515 100644
--- a/blog/feeds/alamb-akurmustafa.atom.xml
+++ b/blog/feeds/alamb-akurmustafa.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - alamb, akurmustafa</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/alamb-akurmustafa.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-06-15T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</title><link hr [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - alamb, akurmustafa</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/alamb-akurmustafa.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2025-06-15T00:00:00+00:00</updated><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</title><link href="https://datafusion.apache.org [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -217,7 +217,7 @@ optimizations</a>.</p>
<p><a id="footnote3"></a><sup>[3]</sup> <a href="https://www.vldb.org/pvldb/vol17/p1350-justen.pdf">https://www.vldb.org/pvldb/vol17/p1350-justen.pdf</a></p>
<p><a id="footnote4"></a><sup>[4]</sup> <a href="https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/24101">https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/24101</a>, <a href="https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/22111">https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/22111</a>, <a href="https://www.dagstuhl.de/en/seminars/seminar-calendar/sem [...]
<p><a id="footnote5"></a><sup>[5]</sup> And thus in academic classes, by the time you get around to an optimizer the semester is over and everyone is ready for the semester to be done. Once industrial systems mature to the point where the optimizer is a bottleneck, the shiny new-ness of the<a href="https://en.wikipedia.org/wiki/Gartner_hype_cycle"> hype cycle</a> has worn off and it is likely in the trough of disappointment.</p>
-<p><a id="footnote6"></a><sup>[6]</sup> Often systems will classify these passes into different categories, but I am simplifying here</p></content><category term="blog"></category></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="alternate"></link><published>2025-06-15T00:00:00+00:00</publi [...]
+<p><a id="footnote6"></a><sup>[6]</sup> Often systems will classify these passes into different categories, but I am simplifying here</p></content><category term="blog"/></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="alternate"/><published>2025-06-15T00:00:00+00:00</published><updated>20 [...]
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -600,4 +600,4 @@ community</a>. We welcome first time contributors as well as long time par
to the fun of building a database together.</p>
<h2 id="notes">Notes<a class="headerlink" href="#notes" title="Permanent link">¶</a></h2>
<p><a id="footnote7"></a><sup>[7]</sup> See <a href="https://btw-2015.informatik.uni-hamburg.de/res/proceedings/Hauptband/Wiss/Neumann-Unnesting_Arbitrary_Querie.pdf">Unnesting Arbitrary Queries</a> from Neumann and Kemper for a more academic treatment.</p>
-<p><a id="footnote8"></a><sup>[8]</sup> One of my favorite terms I learned from Andy Pavlo’s CMU online lectures</p></content><category term="blog"></category></entry></feed>
\ No newline at end of file
+<p><a id="footnote8"></a><sup>[8]</sup> One of my favorite terms I learned from Andy Pavlo’s CMU online lectures</p></content><category term="blog"/></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/alamb-akurmustafa.rss.xml b/blog/feeds/alamb-akurmustafa.rss.xml
index 1d2e434..6dd26bc 100644
--- a/blog/feeds/alamb-akurmustafa.rss.xml
+++ b/blog/feeds/alamb-akurmustafa.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - alamb, akurmustafa</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Sun, 15 Jun 2025 00:00:00 +0000</lastBuildDate><item><title>Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</title><link>https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - alamb, akurmustafa</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/alamb-akurmustafa.rss.xml" rel="self"/><lastBuildDate>Sun, 15 Jun 2025 00:00:00 +0000</lastBuildDate><item><title>Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</title><link>https://datafusion.apache.org/blog/ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -26,7 +26,7 @@ science,”</a> according to Father
Pavlo, or some behind-the-scenes player. We believe this perception is because:</p>
<ol>
<li>
-<p>One must implement the rest of a …</p></li></ol></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, akurmustafa</dc:creator><pubDate>Sun, 15 Jun 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-06-15:/blog/2025/06/15/optimizing-sql-dataframes-part-one</guid><category>blog</category></item><item><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link> [...]
+<p>One must implement the rest of a …</p></li></ol></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, akurmustafa</dc:creator><pubDate>Sun, 15 Jun 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-06-15:/blog/2025/06/15/optimizing-sql-dataframes-part-one</guid><category>blog</category></item><item><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link>https://datafusion.a [...]
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -49,4 +49,4 @@ limitations under the License.
<p>In the <a href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one">first part of this post</a>, we discussed what a Query Optimizer is, what
role it plays, and described how industrial optimizers are organized. In this
second post, we describe various optimizations that are found in <a href="https://datafusion.apache.org/">Apache
-DataFusion</a> and …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, akurmustafa</dc:creator><pubDate>Sun, 15 Jun 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-06-15:/blog/2025/06/15/optimizing-sql-dataframes-part-two</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+DataFusion</a> and …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, akurmustafa</dc:creator><pubDate>Sun, 15 Jun 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-06-15:/blog/2025/06/15/optimizing-sql-dataframes-part-two</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/alamb-dandandan-tustvold.atom.xml b/blog/feeds/alamb-dandandan-tustvold.atom.xml
index c38bd99..27368ee 100644
--- a/blog/feeds/alamb-dandandan-tustvold.atom.xml
+++ b/blog/feeds/alamb-dandandan-tustvold.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - alamb, Dandandan, tustvold</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/alamb-dandandan-tustvold.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2023-08-05T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - alamb, Dandandan, tustvold</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/alamb-dandandan-tustvold.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2023-08-05T00:00:00+00:00</updated><entry><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link href="https://datafusion.apache.o [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -322,4 +322,4 @@ allocation using the arrow Row format
<p>[^3]: <code>SELECT COUNT(DISTINCT "SearchPhrase") as num_phrases FROM 'hits.parquet';</code></p>
<p>[^4]: <code>SELECT COUNT(*) FROM (SELECT DISTINCT "UserID", "SearchPhrase" FROM 'hits.parquet')</code></p>
<p>[^5]: Full script at <a href="https://github.com/alamb/datafusion-duckdb-benchmark/blob/main/hash.py">hash.py</a></p>
-<p>[^6]: <a href="https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_%7B%7D.parquet">hits_0.parquet</a>, one of the files from the partitioned ClickBench dataset, which has <code>100,000</code> rows and is 117 MB in size. The entire dataset has <code>100,000,000</code> rows in a single 14 GB Parquet file. The script did not complete on the entire dataset after 40 minutes, and used 212 GB RAM at peak.</p></content><categ [...]
\ No newline at end of file
+<p>[^6]: <a href="https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_%7B%7D.parquet">hits_0.parquet</a>, one of the files from the partitioned ClickBench dataset, which has <code>100,000</code> rows and is 117 MB in size. The entire dataset has <code>100,000,000</code> rows in a single 14 GB Parquet file. The script did not complete on the entire dataset after 40 minutes, and used 212 GB RAM at peak.</p></content><categ [...]
\ No newline at end of file
diff --git a/blog/feeds/alamb-dandandan-tustvold.rss.xml b/blog/feeds/alamb-dandandan-tustvold.rss.xml
index 5ef68a4..b7b3358 100644
--- a/blog/feeds/alamb-dandandan-tustvold.rss.xml
+++ b/blog/feeds/alamb-dandandan-tustvold.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - alamb, Dandandan, tustvold</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Sat, 05 Aug 2023 00:00:00 +0000</lastBuildDate><item><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link>https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - alamb, Dandandan, tustvold</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/alamb-dandandan-tustvold.rss.xml" rel="self"/><lastBuildDate>Sat, 05 Aug 2023 00:00:00 +0000</lastBuildDate><item><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link>https://datafusion.apache.org/blo [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -23,4 +23,4 @@ limitations under the License.
<p>Andrew Lamb, Daniël Heres, Raphael Taylor-Davies,</p>
<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
<h2 id="tldr">TLDR<a class="headerlink" href="#tldr" title="Permanent link">¶</a></h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, Dandandan, tustvold</dc:creator><pubDate>Sat, 05 Aug 2023 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2023-08-05:/blog/2023/08/ [...]
\ No newline at end of file
+<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, Dandandan, tustvold</dc:creator><pubDate>Sat, 05 Aug 2023 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2023-08-05:/blog/2023/08/05/datafusion_fast_g [...]
\ No newline at end of file
diff --git a/blog/feeds/all-en.atom.xml b/blog/feeds/all-en.atom.xml
index 3a826dd..65cae42 100644
--- a/blog/feeds/all-en.atom.xml
+++ b/blog/feeds/all-en.atom.xml
@@ -1,5 +1,319 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-03-31T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Writing Custom Table Providers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2026/03/31/writing-table [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-04-18T00:00:00+00:00</updated><entry><title>Apache DataFusion Comet 0.15.0 Release</title><link href="https://datafusion.apache.org/blog/2026/04/18/datafusion-comet-0.15.0" rel="alternate"/><published>2026-0 [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.15.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately four weeks of development …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.15.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately four weeks of development work and is the result of merging 142 PRs from 19
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.15.0.md">change log</a> for more information.</p>
+<h2 id="key-features">Key Features<a class="headerlink" href="#key-features" title="Permanent link">¶</a></h2>
+<h3 id="native-iceberg-reader-enabled-by-default">Native Iceberg Reader Enabled by Default<a class="headerlink" href="#native-iceberg-reader-enabled-by-default" title="Permanent link">¶</a></h3>
+<p>This release marks a major milestone for Iceberg users: <strong>Comet's fully-native Iceberg reader is now enabled by
+default</strong>. Workloads that read Iceberg tables will automatically benefit from native Rust-based scans built on
+iceberg-rust, with no additional configuration required.</p>
+<p>To support this change, the release bundles a broad set of Iceberg-focused improvements:</p>
+<ul>
+<li><strong>Correct classloader handling</strong>: Iceberg classes are now loaded via the thread context classloader, resolving
+ class-loading issues in environments where the executor classloader differs from the application classloader.</li>
+<li><strong>Continuous Iceberg CI</strong>: Iceberg Spark integration tests now run on every PR and push to <code>main</code>, providing
+ continuous validation of the native Iceberg code path. Test diffs for Spark 3.4 were updated to keep the matrix
+ green across supported Spark versions.</li>
+<li><strong>iceberg-rust upgrade</strong>: Comet picks up the latest iceberg-rust, pulling in fixes for Parquet reader edge cases
+ discovered in earlier testing.</li>
+<li><strong>Refreshed documentation</strong>: The Iceberg user guide has been rewritten to reflect current capabilities, and the
+ contributor guide now documents how to run the Iceberg Spark test suites locally.</li>
+</ul>
+<p>Users who need to fall back to the previous behavior can still opt out, but we encourage the community to exercise
+the native reader and report any issues.</p>
+<h3 id="auto-mode-enables-native_datafusion-scan">Auto Mode Enables <code>native_datafusion</code> Scan<a class="headerlink" href="#auto-mode-enables-native_datafusion-scan" title="Permanent link">¶</a></h3>
+<p>The <code>auto</code> scan mode now enables the <code>native_datafusion</code> scan where supported, giving users the benefits of the
+native Parquet reader without having to explicitly opt in. This is part of the ongoing effort to make
+<code>native_datafusion</code> the default Parquet path once the deprecation of <code>native_iceberg_compat</code> completes.</p>
+<h3 id="new-expressions-and-function-support">New Expressions and Function Support<a class="headerlink" href="#new-expressions-and-function-support" title="Permanent link">¶</a></h3>
+<p>This release adds support for the following:</p>
+<ul>
+<li><strong>Date/time functions</strong>: <code>days</code>, <code>hours</code>, <code>date_from_unix_date</code></li>
+<li><strong>String/JSON functions</strong>: native <code>get_json_object</code> with improved performance over the fallback path</li>
+<li><strong>Hash/math functions</strong>: <code>crc32c</code>, <code>bin</code></li>
+<li><strong>Array functions</strong>: <code>sort_array</code></li>
+<li><strong>Window functions</strong>: <code>LEAD</code> and <code>LAG</code> with <code>IGNORE NULLS</code></li>
+<li><strong>Aggregates</strong>: SQL <code>FILTER (WHERE ...)</code> clauses now execute natively; <code>Corr</code> aggregate enabled</li>
+</ul>
+<h3 id="expanded-metrics-and-observability">Expanded Metrics and Observability<a class="headerlink" href="#expanded-metrics-and-observability" title="Permanent link">¶</a></h3>
+<p>Comet metrics can now be exposed through Spark's external monitoring system, making it easier to integrate Comet
+execution statistics with existing observability dashboards. Native DataFusion scans also now report accurate
+<code>filesScanned</code> and <code>bytesScanned</code> input metrics, matching Spark's native Parquet scan reporting.</p>
+<h2 id="performance-improvements">Performance Improvements<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>Performance was a major theme of this release, with a series of targeted optimizations across the shuffle, scan,
+and execution layers:</p>
+<ul>
+<li><strong>Shuffle read path</strong>: The native shuffle reader no longer uses FFI on the read side, removing a per-batch cost
+ that was particularly visible in shuffle-heavy queries.</li>
+<li><strong>Broadcast exchanges</strong>: Batches are now coalesced before broadcasting, reducing the number of small batches
+ crossing the JVM/native boundary.</li>
+<li><strong>Columnar-to-row (C2R)</strong>: Native C2R conversion is now exercised for a broader set of query shapes.</li>
+<li><strong>FFI-safe operators</strong>: More operators are marked as FFI-safe, avoiding unnecessary deep copies when crossing
+ the JVM/native boundary.</li>
+<li><strong>Shared memory pools</strong>: Unified memory pools are now shared across native execution contexts within a Spark
+ task, improving memory accounting and reducing fragmentation.</li>
+<li><strong>Object store caching</strong>: Object stores and bucket region lookups are cached, dramatically reducing DNS query
+ volume on workloads that open many files.</li>
+<li><strong><code>get_ranges</code> performance</strong>: Picked up an upstream <code>opendal</code> fix that restores fast range reads from object
+ storage.</li>
+<li><strong>Spill I/O</strong>: Removed a redundant <code>BufReader</code> wrapper when copying spill files to shuffle output.</li>
+</ul>
+<p>Together, these changes reduce CPU and memory overhead for shuffle-heavy, broadcast-heavy, and
+object-storage-bound workloads.</p>
+<h2 id="stability-and-correctness">Stability and Correctness<a class="headerlink" href="#stability-and-correctness" title="Permanent link">¶</a></h2>
+<p>A significant portion of this release is dedicated to stability and Spark compatibility. Highlights include:</p>
+<ul>
+<li><strong>Cast string to timestamp</strong>: Multiple fixes for UTC timestamps, timezone handling, special formats
+ (<code>epoch</code>, <code>now</code>, etc.), and compatibility with Spark's semantics.</li>
+<li><strong>Cast decimal to string</strong>: Added legacy mode handling to match Spark's output formatting.</li>
+<li><strong>String to decimal</strong>: Support for full-width characters, null characters, and negative scale.</li>
+<li><strong>Decimal arithmetic</strong>: Fixes for decimal division and additional test coverage for ANSI overflow handling,
+ including scalar decimal overflow.</li>
+<li><strong>Array expressions</strong>: Corrected <code>GetArrayItem</code> null handling for dynamic indices; <code>array_append</code> return type
+ fixed and marked <code>Compatible</code>; audited <code>array_insert</code> for correctness; <code>array_compact</code> marked <code>Compatible</code>;
+ array-to-array cast enabled.</li>
+<li><strong>DateTrunc/TimestampTrunc</strong>: Fixed native crashes when the input is a literal.</li>
+<li><strong>Ambiguous local times</strong>: Correct handling of ambiguous and non-existent local times across DST transitions.</li>
+<li><strong>Case-insensitive Parquet fields</strong>: <code>native_datafusion</code> now correctly detects duplicate/ambiguous fields in
+ case-insensitive mode and falls back where appropriate.</li>
+<li><strong>Shuffle planning</strong>: Shuffle fallback decisions are now "sticky" across planning passes, and Comet columnar
+ shuffle is skipped for stages containing DPP scans to avoid mismatched partitioning.</li>
+<li><strong>Error propagation</strong>: Native error messages are now propagated through <code>SparkException</code> even when the
+ <code>errorClass</code> is empty, and file-not-found errors flow through the standard Spark error JSON path.</li>
+<li><strong>Trigonometric compatibility</strong>: <code>tan</code> and <code>atan2</code> are now Spark-compatible.</li>
+</ul>
+<h2 id="dependency-upgrades">Dependency Upgrades<a class="headerlink" href="#dependency-upgrades" title="Permanent link">¶</a></h2>
+<p>This release upgrades to <strong>DataFusion 53.1</strong> and <strong>Arrow 58.1</strong>, and picks up the latest <code>iceberg-rust</code> release
+with additional reader fixes. The <code>jni</code> crate was upgraded to 0.22.4.</p>
+<h2 id="deprecations-and-removals">Deprecations and Removals<a class="headerlink" href="#deprecations-and-removals" title="Permanent link">¶</a></h2>
+<p>The <code>SupportsComet</code> interface has been removed, along with the Java-based Iceberg integration path (which is
+fully superseded by the native Iceberg reader). The <code>native_iceberg_compat</code> scan remains deprecated and is
+expected to be removed in a future release in favor of <code>native_datafusion</code>.</p>
+<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
+<p>Supported platforms include Spark 3.4.3, 3.5.4–3.5.8, and Spark 4.0.x with various JDK and Scala combinations.</p>
+<p>The community encourages users to test Comet with existing Spark and Iceberg workloads and welcomes contributions
+to ongoing development.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 53.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/04/02/datafusion-53.0.0" rel="alternate"/><published>2026-04-02T00:00:00+00:00</published><updated>2026-04-02T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2026-04-02:/blog/2026/04/02/datafusion-53.0.0</id><summary type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/53.0.0">DataFusion 53.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0/">DataFusion 52.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md#credits">114 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p><img alt="Performance over time" class="img-fluid" src="/blog/images/datafusion-53.0.0/performance_over_time_clickbench.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: Average …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/53.0.0">DataFusion 53.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0/">DataFusion 52.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md#credits">114 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p><img alt="Performance over time" class="img-fluid" src="/blog/images/datafusion-53.0.0/performance_over_time_clickbench.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: Average and median normalized execution times for DataFusion 53.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+<a href="https://alamb.github.io/datafusion-benchmarking/">DataFusion Benchmarking Page</a>
+for more details.</p>
+<p>DataFusion 53 continues the project-wide focus on performance. This release
+reduces planning overhead, skips more unnecessary I/O, and pushes more work
+into earlier and cheaper stages of execution.</p>
+<h3 id="limit-aware-parquet-row-group-pruning"><code>LIMIT</code>-Aware Parquet Row Group Pruning<a class="headerlink" href="#limit-aware-parquet-row-group-pruning" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 includes a new optimization that makes Parquet pruning aware of
+<code>LIMIT</code>. This optimization is described in full in <a href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning/">limit pruning blog post</a>. If
+DataFusion can prove that an entire row group matches the predicate, and those
+fully matching row groups contain enough rows to satisfy the <code>LIMIT</code>, partially
+matching row groups are skipped entirely.</p>
+<figure>
+<img alt="Pruning pipeline with limit pruning highlighted" class="img-fluid" src="/blog/images/limit-pruning/pruning-pipeline.svg" width="80%"/>
+<figcaption><b>Figure 2</b>: Limit pruning is inserted between row group and page index pruning.</figcaption>
+</figure>
+<p>Thanks to <a href="https://github.com/xudong963">@xudong963</a> for implementing this feature. Related PRs: <a href="https://github.com/apache/datafusion/pull/18868">#18868</a></p>
+<h3 id="improved-filter-pushdown">Improved Filter Pushdown<a class="headerlink" href="#improved-filter-pushdown" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 pushes filters down through more join types and through <code>UnionExec</code>,
+and expands support for pushing down <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters">dynamic filters</a>. More
+pushdown means fewer rows flow into joins, repartitions, and later operators,
+which reduces CPU, memory, and I/O.</p>
+<p>For example:</p>
+<pre><code class="language-sql">SELECT *
+FROM (
+ SELECT *
+ FROM t1
+ LEFT ANTI JOIN t2 ON t1.k = t2.k
+) a
+JOIN t1 b ON a.k = b.k
+WHERE b.v = 1;
+</code></pre>
+<p>Now DataFusion can often transform the physical plan so filters and
+<a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters">dynamic filters</a> are pushed deeper into the plan, even through subqueries and
+nested joins. In this example, the filter on <code>b.v</code> helps produce dynamic filters
+that can be pushed into both sides of the nested anti join.</p>
+<figure>
+<img alt="Before and after diagram of dynamic filter pushdown through a subquery with nested joins" class="img-fluid" src="/blog/images/datafusion-53.0.0/join-filter-pushdown.svg" width="100%"/>
+<figcaption><b>Figure 3</b>: DataFusion 53 pushes dynamic filters through subqueries and into both sides of nested joins.</figcaption>
+</figure>
+<p>Thanks to <a href="https://github.com/nuno-faria">@nuno-faria</a>, <a href="https://github.com/haohuaijin">@haohuaijin</a>, and <a href="https://github.com/jackkleeman">@jackkleeman</a> for
+driving this work. Related PRs: <a href="https://github.com/apache/datafusion/pull/19918">#19918</a>, <a href="https://github.com/apache/datafusion/pull/20145">#20145</a>, <a href="https://github.com/apache/datafusion/pull/20192">#20192</a></p>
+<h3 id="faster-query-planning">Faster Query Planning<a class="headerlink" href="#faster-query-planning" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 improves query planning performance by making immutable pieces of
+execution plans cheaper to clone. This helps applications that need extremely
+low latency, plan many or complex queries, or use prepared statements or
+parameterized queries. In some benchmarks, overall execution time drops from
+roughly 4-5 ms to about 100 us.</p>
+<p>Thanks to <a href="https://github.com/askalt">@askalt</a> for leading this work. Related PRs: <a href="https://github.com/apache/datafusion/pull/19792">#19792</a>, <a href="https://github.com/apache/datafusion/pull/19893">#19893</a></p>
+<h3 id="faster-functions">Faster Functions<a class="headerlink" href="#faster-functions" title="Permanent link">¶</a></h3>
+<p>DataFusion includes <a href="https://datafusion.apache.org/user-guide/sql/scalar_functions.html">235 built-in functions</a>. Improving the performance of these
+functions benefits a wide range of workloads. This release improves the performance of 42 of those
+functions, such as <a href="https://github.com/apache/datafusion/pull/20295">strpos</a>, <a href="https://github.com/apache/datafusion/pull/20344">replace</a>, <a href="https://github.com/apache/datafusion/pull/20317">concat</a>, <a href="https://github.com/apache/datafusion/pull/20305">translate</a>, <a href="https://github.com/apache/datafusion/pull/20374">array_has</a>,
+<a href="https://github.com/apache/datafusion/pull/20504">array_agg</a>, <a href="https://github.com/apache/datafusion/pull/19980">left</a>, <a href="https://github.com/apache/datafusion/pull/20069">right</a>, and <a href="https://github.com/apache/datafusion/pull/20097">case_when</a>.</p>
+<p>Thanks to the contributors who drove this work, especially <a href="https://github.com/neilconway">@neilconway</a>,
+<a href="https://github.com/theirix">@theirix</a>, <a href="https://github.com/lyne7-sc">@lyne7-sc</a>, <a href="https://github.com/kumarUjjawal">@kumarUjjawal</a>, <a href="https://github.com/pepijnve">@pepijnve</a>, <a href="https://github.com/zhangxffff">@zhangxffff</a>, and
+<a href="https://github.com/UBarney">@UBarney</a>.</p>
+<h3 id="nested-field-pushdown">Nested Field Pushdown<a class="headerlink" href="#nested-field-pushdown" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 pushes expressions such as <code>get_field</code> down the plan and into data
+sources. This is especially important for nested data such as structs in
+Parquet files. Instead of reading an entire struct column and then extracting
+the field of interest, DataFusion 53 pushes the field extraction into the scan.</p>
+<p>For example, the following query reads a struct column <code>s</code> and extracts the
+<code>label</code> field for rows where the <code>value</code> field is greater than 150:</p>
+<pre><code class="language-sql">SELECT id, s['label']
+FROM t
+WHERE s['value'] &gt; 150;
+</code></pre>
+<figure>
+<img alt="Before and after diagram of field access pushdown into a data source" class="img-fluid" src="/blog/images/datafusion-53.0.0/field-access-pushdown.svg" width="80%"/>
+<figcaption><b>Figure 4</b>: DataFusion 53 pushes field-access expressions closer to the scan.</figcaption>
+</figure>
+<p>Special thanks to <a href="https://github.com/adriangb">@adriangb</a> for designing and implementing this optimizer
+work. Related PRs: <a href="https://github.com/apache/datafusion/pull/20065">#20065</a>, <a href="https://github.com/apache/datafusion/pull/20117">#20117</a>, <a href="https://github.com/apache/datafusion/pull/20239">#20239</a></p>
+<h2 id="new-features">New Features ✨<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h2>
+<ul>
+<li>
+<p><strong>JSON Array File Support</strong>: DataFusion 53 can now read JSON arrays such as
+ <code>[{...}, {...}]</code> directly as multiple rows, including streaming inputs from
+ object stores. Thanks to <a href="https://github.com/zhuqi-lucas">@zhuqi-lucas</a> for implementing this feature.
+ Related PRs: <a href="https://github.com/apache/datafusion/pull/19924">#19924</a></p>
+</li>
+<li>
+<p><strong>Support for <code>:</code> operator</strong>: DataFusion can plan queries such as
+ <code>SELECT payload:'user_id' FROM events;</code>, enabling better <a href="https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/">Parquet Variant</a>
+ support via <a href="https://github.com/datafusion-contrib/datafusion-variant">datafusion-variant</a>. Thanks to <a href="https://github.com/Samyak2">@Samyak2</a>. Related PRs:
+ <a href="https://github.com/apache/datafusion/pull/20717">#20717</a></p>
+</li>
+<li>
+<p><strong>New SQL</strong>: DataFusion supports additional set-comparison subqueries, null-aware
+ anti join, and deletion predicates. Thanks to <a href="https://github.com/waynexia">@waynexia</a>, <a href="https://github.com/viirya">@viirya</a>, and
+ <a href="https://github.com/askalt">@askalt</a> for key contributions in this area. Related PRs: <a href="https://github.com/apache/datafusion/pull/19109">#19109</a>,
+ <a href="https://github.com/apache/datafusion/pull/19635">#19635</a>, <a href="https://github.com/apache/datafusion/pull/20137">#20137</a></p>
+</li>
+<li>
+<p><strong>Spark-Compatible Functions</strong>: This release includes almost 20 new or improved
+ Spark-compatible functions and behaviors in the <a href="https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html">datafusion-spark crate</a>.
+ It includes functions such as <a href="https://github.com/apache/datafusion/pull/19699">collect_list</a>, <a href="https://github.com/apache/datafusion/pull/19845">date_diff</a>,
+ <a href="https://github.com/apache/datafusion/pull/19880">from_utc_timestamp</a>, <a href="https://github.com/apache/datafusion/pull/20412">json_tuple</a>, <a href="https://github.com/apache/datafusion/pull/20440">arrays_zip</a>, <a href="https://github.com/apache/datafusion/pull/20479">bin</a>, and <a href="https://github.com/apache/datafusion/pull/20685">array_contains</a>.
+ Thanks to the contributors who drove this work, especially <a href="https://github.com/cht42">@cht42</a>,
+ <a href="https://github.com/CuteChuanChuan">@CuteChuanChuan</a>, <a href="https://github.com/SubhamSinghal">@SubhamSinghal</a>, <a href="https://github.com/kazantsev-maksim">@kazantsev-maksim</a>, <a href="https://github.com/unknowntpo">@unknowntpo</a>,
+ <a href="https://github.com/aryan-212">@aryan-212</a>, <a href="https://github.com/hsiang-c">@hsiang-c</a>, and <a href="https://github.com/davidlghellin">@davidlghellin</a>.</p>
+</li>
+</ul>
+<h2 id="stability-and-release-engineering">Stability and Release Engineering 🦺<a class="headerlink" href="#stability-and-release-engineering" title="Permanent link">¶</a></h2>
+<p>The community spent significant time this release cycle stabilizing the release
+branch and improving the release process. While such improvements are not as
+headline-friendly as new features, they are highly important for real
+deployments. We are discussing ways to improve the process on <a href="https://github.com/apache/datafusion/issues/21034">#21034</a> and would
+welcome suggestions and contributions to help with release engineering work in
+the future.</p>
+<p>Thanks to <a href="https://github.com/comphead">@comphead</a> for running this release, and to <a href="https://github.com/jonathanc-n">@jonathanc-n</a>, <a href="https://github.com/alamb">@alamb</a>,
+<a href="https://github.com/xanderbailey">@xanderbailey</a>, <a href="https://github.com/haohuaijin">@haohuaijin</a>, <a href="https://github.com/friendlymatthew">@friendlymatthew</a>, <a href="https://github.com/fwojciec">@fwojciec</a>,
+<a href="https://github.com/Kontinuation">@Kontinuation</a>, <a href="https://github.com/nathanb9">@nathanb9</a>, and many others who helped stabilize the release
+branch.</p>
+<h2 id="upgrade-notes">Upgrade Notes<a class="headerlink" href="#upgrade-notes" title="Permanent link">¶</a></h2>
+<p>DataFusion 53 includes some breaking changes, including updates to the SQL
+parser, optimizer behavior, and some physical-plan APIs. Please see the <a href="https://datafusion.apache.org/library-user-guide/upgrading/index.html">upgrade
+guide</a> and <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a> for the full details before upgrading.</p>
+<h2 id="known-issues">Known Issues<a class="headerlink" href="#known-issues" title="Permanent link">¶</a></h2>
+<p>A small number of issues were discovered after the 53.0.0 release,
+and we expect to publish DataFusion 53.1.0 soon. See the <a href="https://github.com/apache/datafusion/issues/21079">53.1.0 release tracking
+issue</a> for the latest status.</p>
+<h2 id="thank-you">Thank You<a class="headerlink" href="#thank-you" title="Permanent link">¶</a></h2>
+<p>Thank you to everyone in the DataFusion community who contributed code, reviews,
+testing, bug reports, documentation, and release engineering work for 53.0.0.
+This release contains direct contributions from 114 different people, and we are
+grateful for the time and effort that everyone put in to make it happen.</p></content><category term="blog"/></entry><entry><title>Writing Custom Table Providers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2026/03/31/writing-table-providers" rel="alternate"/><published>2026-03-31T00:00:00+00:00</published><updated>2026-03-31T00:00:00+00:00</updated><author><name>Tim Saucer (rerun.io)</name></author><id>tag:datafusion.apache.org,2026-03-31:/blog/2026/0 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -826,7 +1140,7 @@ can find out how to reach us on the <a href="https://datafusion.apache.org/co
contains working examples including custom table providers</li>
</ul>
<hr/>
-<p><em>Note: Portions of this blog post were written with the assistance of an AI agent.</em></p></content><category term="blog"></category></entry><entry><title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</title><link href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="alternate"></link><published>2026-03-20T00:00:00+00:00</published><updated>2026-03-20T00:00:00+00:00</updated><author><name>xudong</nam [...]
+<p><em>Note: Portions of this blog post were written with the assistance of an AI agent.</em></p></content><category term="blog"/></entry><entry><title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</title><link href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="alternate"/><published>2026-03-20T00:00:00+00:00</published><updated>2026-03-20T00:00:00+00:00</updated><author><name>xudong</name></author><id>t [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1137,7 +1451,7 @@ limit_pruned_row_groups=3 total → 1 matched
<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries, and machine learning and streaming applications.</p>
<p>DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone.</p>
<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
-<p>If you are interested in contributing, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a [...]
+<p>If you are interested in contributing, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1238,7 +1552,7 @@ that the <code>native_iceberg_compat</code> scan is now deprecated a
<p>This release upgrades to DataFusion 52.3, Arrow 57.3, and iceberg-rust 0.9.0. Published binaries now target
x86-64-v3 and neoverse-n1 CPU architectures for improved performance on modern hardware.</p>
<p>Supported platforms include Spark 3.4.3, 3.5.4-3.5.8, and Spark 4.0.x with various JDK and Scala combinations.</p>
-<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"></category></entry><entry><title>Optimizing SQL CASE Expression Evaluation</title><link href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="alternate"></link><published>2026-02-02T00:00:00+00:00</published><updated>2026-02-02T00:00:00+00:00</updated><author><name>Pepijn Van Eeckhoudt</name></a [...]
+<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"/></entry><entry><title>Optimizing SQL CASE Expression Evaluation</title><link href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="alternate"/><published>2026-02-02T00:00:00+00:00</published><updated>2026-02-02T00:00:00+00:00</updated><author><name>Pepijn Van Eeckhoudt</name></author><id>tag:da [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1633,7 +1947,7 @@ With DataFusion, we can all build on top of a shared foundation and focus on wha
<p>DataFusion is not a project built or driven by a single person, company, or foundation.
Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.</p>
<p>If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code.
-A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.13.0 Release</title><link href="https://datafusion.a [...]
+A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.13.0 Release</title><link href="https://datafusion.apache.org/ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1749,7 +2063,7 @@ spark.comet.parquet.write.enabled=true
<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
<p>This release upgrades to DataFusion 51, Arrow 57, and the latest iceberg-rust. The minimum supported Rust version is now 1.88.</p>
<p>Supported platforms include Spark 3.4.3, 3.5.4-3.5.7, and Spark 4.0.x with various JDK and Scala combinations.</p>
-<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="alternate"></link><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusio [...]
+<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="alternate"/><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,202 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1988,7 +2302,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="alternate"></link><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>Ge [...]
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="alternate"/><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>Geoffrey Claude (D [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2272,7 +2586,7 @@ println!("{}", df.logical_plan().display_indent());
<li><strong>Try it out</strong>: Implement one of the extension points and share your experience</li>
<li><strong>File issues or join the conversation</strong>: <a href="https://github.com/apache/datafusion/">GitHub</a> for bugs and feature requests, <a href="https://datafusion.apache.org/contributor-guide/communication.html">Slack or Discord</a> for discussion</li>
</ul>
-<!-- Reference links --></content><category term="blog"></category></entry><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="alternate"></link><published>2025-12-15T00:00:00+00:00</published><updated>2025-12-15T00:00:00+00:00</updated><author><name>Gene Bordegaray</name></author><id>tag:datafusion.apache.org,2025-12-15:/blog/202 [...]
+<!-- Reference links --></content><category term="blog"/></entry><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="alternate"/><published>2025-12-15T00:00:00+00:00</published><updated>2025-12-15T00:00:00+00:00</updated><author><name>Gene Bordegaray</name></author><id>tag:datafusion.apache.org,2025-12-15:/blog/2025/12/15/avoid-co [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2290,8 +2604,8 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div></summary><content type="html"><!--
{% comment %}
@@ -2311,16 +2625,16 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is exposed by a user-friendly interface and query languages (typically a dialect of SQL).
<br/><br/>
Starting a journey learning about database internals can be daunting. With so many topics that are whole PhD degrees themselves, finding a place to start is difficult. In this blog post, I will share my early journey in the database world and a quick lesson on one of the first topics I dove into. If you are new to the space, this post will help you get your first foot into the database world, and if you are already a veteran, you may still learn something new.
</div>
-<div style="flex: 0 0 40%; text-align: center;">
-<img alt="Database System Components" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/database_system_diagram.png" width="100%"/>
+<div class="col-md-5 text-center">
+<img alt="Database System Components" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/database_system_diagram.png"/>
</div>
</div>
<hr/>
@@ -2361,29 +2675,29 @@ DataFusion achieves multi-core parallelism through the use of "exchange operator
<h3 id="what-is-repartitioning">What is Repartitioning?<a class="headerlink" href="#what-is-repartitioning" title="Permanent link">&para;</a></h3>
<p>Partitioning is a "divide-and-conquer" approach to executing a query. Each partition is a subset of the data that is being processed on a single core. Repartitioning is an operation that redistributes data across different partitions to balance workloads, reduce data skew, and increase parallelism. Two repartitioning methods are used in DataFusion: round-robin and hash.</p>
<h4 id="round-robin-repartitioning"><strong>Round-Robin Repartitioning</strong><a class="headerlink" href="#round-robin-repartitioning" title="Permanent link">&para;</a></h4>
-<div style="display: flex; align-items: top; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-start mb-3">
+<div class="col-md-9">
Round-robin repartitioning is the simplest partitioning strategy. Incoming data is processed in batches (chunks of rows), and these batches are distributed across partitions cyclically or sequentially, with each new batch assigned to the next available partition.
<br/><br/>
Round-robin repartitioning is useful when the data grouping isn't known or when aiming for an even distribution across partitions. Because it simply assigns batches in order without inspecting their contents, it is a low-overhead way to increase parallelism for downstream operations.
</div>
-<div style="flex: 0 0 25%; text-align: center;">
-<img alt="Round-Robin Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/round_robin_repartitioning.png" width="100%"/>
+<div class="col-md-3 text-center">
+<img alt="Round-Robin Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/round_robin_repartitioning.png"/>
</div>
</div>
<h4 id="hash-repartitioning"><strong>Hash Repartitioning</strong><a class="headerlink" href="#hash-repartitioning" title="Permanent link">&para;</a></h4>
-<div style="display: flex; align-items: top; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-start mb-3">
+<div class="col-md-9">
Hash repartitioning distributes data based on a hash function applied to one or more columns, called the partitioning key. Rows with the same hash value are placed in the same partition.
<br/><br/>
Hash repartitioning is useful when working with grouped data. Imagine you have a database containing information on company sales, and you are looking to find the total revenue each store produced. Hash repartitioning would make this query much more efficient. Rather than iterating over the data on a single thread and keeping a running sum for each store, it would be better to hash repartition on the store column and have multiple threads calculate individual store sales.
</div>
-<div style="flex: 0 0 25%; text-align: center;">
-<img alt="Hash Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/hash_repartitioning.png" width="100%"/>
+<div class="col-md-3 text-center">
+<img alt="Hash Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/hash_repartitioning.png"/>
</div>
</div>
<p>Note, the benefit of hash opposed to round-robin partitioning in this scenario. Hash repartitioning consolidates all rows with the same store value in distinct partitions. Because of this property we can compute the complete results for each store in parallel and merge them to get the final outcome. This parallel processing wouldn&rsquo;t be possible with only round-robin partitioning as the same store value may be spread across multiple partitions, making the aggregation re [...]
@@ -2538,7 +2852,7 @@ Hash repartitioning is useful when working with grouped data. Imagine you have a
<p>I hope you gained something from my experience and have fun learning about databases.</p>
<hr/>
<h2 id="acknowledgements"><strong>Acknowledgements</strong><a class="headerlink" href="#acknowledgements" title="Permanent link">&para;</a></h2>
-<p>Thank you to <a href="https://github.com/NGA-TRAN">Nga Tran</a> for continuous mentorship and guidance, the DataFusion community, specifically <a href="https://github.com/alamb">Andrew Lamb</a>, for lending me support throughout my work, and Datadog for providing the opportunity to work on such interesting systems.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.12.0 Release</title><link href="https://da [...]
+<p>Thank you to <a href="https://github.com/NGA-TRAN">Nga Tran</a> for continuous mentorship and guidance, the DataFusion community, specifically <a href="https://github.com/alamb">Andrew Lamb</a>, for lending me support throughout my work, and Datadog for providing the opportunity to work on such interesting systems.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.12.0 Release</title><link href="https://datafusion.a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2660,7 +2974,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 51.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0" rel="alternate"></link><published>2025-11-25T00:00:00+00:00</published><updated>2025-11-25T00:00:00+00:00</updated><author><name>pmc</name></autho [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 51.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0" rel="alternate"/><published>2025-11-25T00:00:00+00:00</published><updated>2025-11-25T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafu [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2886,7 +3200,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.11.0 Release</title><link href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0" rel="alternate"></link><published>2025-10-21T00:00:00+00:00</published><updated>2025-10-21T00:00:00+00:00</updated><author><name>pmc</name></ [...]
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.11.0 Release</title><link href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0" rel="alternate"/><published>2025-10-21T00:00:00+00:00</published><updated>2025-10-21T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:d [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -3033,7 +3347,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 50.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0" rel="alternate"></link><published>2025-09-29T00:00:00+00:00</published><updated>2025-09-29T00:00:00+00:00</updated><author><name>pmc</name></autho [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 50.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0" rel="alternate"/><published>2025-09-29T00:00:00+00:00</published><updated>2025-09-29T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafu [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -3373,7 +3687,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Implementing User Defined Types and Custom Metadata in DataFusion</title><link href="https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata" rel="alternate"></link><published>2025-09-21T00:00:00+00:00</published><updated>2025-09-21T00:00:00+00:00</upd [...]
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Implementing User Defined Types and Custom Metadata in DataFusion</title><link href="https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata" rel="alternate"/><published>2025-09-21T00:00:00+00:00</published><updated>2025-09-21T00:00:00+00:00</updated><author><na [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -3643,7 +3957,7 @@ us and help the project.</p>
<li>Try out the project and provide feedback, file issues, and contribute code.</li>
<li>Work on a <a href="https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">good first issue</a>.</li>
<li>Reach out to us via the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.10.0 Release</title><link href="https://datafusion.apache.org/blog/2025/09/16/datafusion-comet-0.10.0" rel="alternate"></link><published>2025-09-16T00:00:00+00:00</published><updated>2025-09-16T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-09-16:/blog/2025/09/16/datafusion-comet-0.10.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.10.0 Release</title><link href="https://datafusion.apache.org/blog/2025/09/16/datafusion-comet-0.10.0" rel="alternate"/><published>2025-09-16T00:00:00+00:00</published><updated>2025-09-16T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-09-16:/blog/2025/09/16/datafusion-comet-0.10.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -3756,7 +4070,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</title><link href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters" rel="alternate"></link><published>2025-09-10T00:00:00+00:00</published><updated>2025-09 [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</title><link href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters" rel="alternate"/><published>2025-09-10T00:00:00+00:00</published><updated>2025-09-10T00:00:00+00: [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -4329,7 +4643,7 @@ LIMIT 10;
<td style="text-align: right;">0.602</td>
</tr>
</tbody>
-</table></content><category term="blog"></category></entry><entry><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</title><link href="https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes" rel="alternate"></link><published>2025-08-15T00:00:00+00:00</published><updated>2025-08-15T00:00:00+00:00</updated><author><name>Andrew Lamb (InfluxData)</name></author><id>tag:datafusion.apache.org,2025-08-15:/blog/2025/ [...]
+</table></content><category term="blog"/></entry><entry><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</title><link href="https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes" rel="alternate"/><published>2025-08-15T00:00:00+00:00</published><updated>2025-08-15T00:00:00+00:00</updated><author><name>Andrew Lamb (InfluxData)</name></author><id>tag:datafusion.apache.org,2025-08-15:/blog/2025/08/15/external-p [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -4919,7 +5233,7 @@ such as point lookups, filters of different selectivity, and aggregations.</p
<p><a id="footnote7"></a><code>7</code>: For example, <a href="https://github.com/zhuqi-lucas">Qi Zhu</a> was able to speed up reads by over 2x
simply by rewriting the Parquet files with Offset Indexes and no compression (see <a href="https://github.com/apache/datafusion/issues/16149#issuecomment-2918761743">issue #16149 comment</a> for details).
There is likely significant additional performance available by using Bloom Filters and resorting the data
-to be clustered in a more optimal way for the queries.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 49.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0" rel="alternate"></link><published>2025-07-28T00:00:00+00:00</published><updated>2025-07-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-07-28:/blog/2025/07/28/datafusion-49.0.0</id><summary type="ht [...]
+to be clustered in a more optimal way for the queries.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 49.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0" rel="alternate"/><published>2025-07-28T00:00:00+00:00</published><updated>2025-07-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-07-28:/blog/2025/07/28/datafusion-49.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -5227,7 +5541,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 48.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0" rel="alternate"></link><published>2025-07-16T00:00:00+00:00</published><updated>2025-07-16T00:00:00+00:00</updated><author><name>PMC</name></author><id> [...]
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 48.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0" rel="alternate"/><published>2025-07-16T00:00:00+00:00</published><updated>2025-07-16T00:00:00+00:00</updated><author><name>PMC</name></author><id>tag:datafusion.a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -5423,7 +5737,7 @@ community discussions – whether you have questions, want to share how you’re
contribute, we’d love to hear from you. A list of open issues suitable for beginners
is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p>
-<p>Happy querying!</p></content><category term="blog"></category></entry><entry><title>Embedding User-Defined Indexes in Apache Parquet Files</title><link href="https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes" rel="alternate"></link><published>2025-07-14T00:00:00+00:00</published><updated>2025-07-14T00:00:00+00:00</updated><author><name>Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</name></author><id>tag: [...]
+<p>Happy querying!</p></content><category term="blog"/></entry><entry><title>Embedding User-Defined Indexes in Apache Parquet Files</title><link href="https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes" rel="alternate"/><published>2025-07-14T00:00:00+00:00</published><updated>2025-07-14T00:00:00+00:00</updated><author><name>Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</name></author><id>tag:datafusion.apach [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -5556,24 +5870,24 @@ limitations under the License.
<hr/>
<p>A <strong>distinct value index</strong> stores the unique values of a specific column. This type of index is effective for columns with a small number of distinct values and can be used to quickly skip files that do not match the query. These indexes are popular in several engines, such as the <a href="https://clickhouse.com/docs/optimize/skipping-indexes#set">"set" Skip Index in ClickHouse</a> and the <a href="https://docs.influxdata.com/influxdb3/ent [...]
<p>For example, if the files contain a column named <code>Category</code> like this:</p>
-<table class="table" style="border-collapse:collapse;">
+<table class="table">
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><b><code>Category</code></b></td>
+<td><b><code>Category</code></b></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>foo</code></td>
+<td><code>foo</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>bar</code></td>
+<td><code>bar</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>...</code></td>
+<td><code>...</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>baz</code></td>
+<td><code>baz</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>foo</code></td>
+<td><code>foo</code></td>
</tr>
</table>
<p>The distinct value index will contain the values <code>foo</code>, <code>bar</code>, and <code>baz</code>. In contrast, traditional min/max statistics would store only the minimum (<code>bar</code>) and maximum (<code>foo</code>) values, so a query like</p>
@@ -5913,7 +6227,7 @@ it out, we would love for you to join us.</p>
<p><a id="footnote3"></a><code>3</code>: <a href="https://dl.gi.de/items/2a8571f8-0ef2-481c-8ee9-05f82ee258c8">Seamless Integration of Parquet Files into Data Processing. / Rey, Alice; Freitag, Michael; Neumann, Thomas. / BTW 2023</a></p>
<p><a id="footnote4"></a><code>4</code>: For more information about external indexes, see <a href="https://www.youtube.com/watch?v=74YsJT1-Rdk">this talk</a> and the <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/parquet_index.rs">parquet_index.rs</a> and <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_parquet_index.rs">advanced_parquet_index.rs& [...]
<p><a id="footnote5"></a><code>5</code>: For information about rewriting files to optimize for specific queries, such as resorting, repartitioning, and tuning data page and row group sizes, see <a href="https://github.com/XiangpengHao/liquid-cache/issues/227">XiangpengHao/liquid‑cache#227</a> and the conversation between <a href="https://github.com/JigaoLuo">JigaoLuo</a> and <a href="https://github.com/XiangpengHao">XiangpengHao [...]
-<p><a id="footnote6"></a><code>6</code>: An index can also be stored inline in the key-value metadata. This approach is simple to implement and ensures the index is available once the footer is read, without additional I/O. However, it requires the index to be serialized as a UTF-8 string, which may be less efficient and increases the size of the footer metadata, impacting all Parquet readers, even those that ignore the index.</p></content><category te [...]
+<p><a id="footnote6"></a><code>6</code>: An index can also be stored inline in the key-value metadata. This approach is simple to implement and ensures the index is available once the footer is read, without additional I/O. However, it requires the index to be serialized as a UTF-8 string, which may be less efficient and increases the size of the footer metadata, impacting all Parquet readers, even those that ignore the index.</p></content><category te [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6097,7 +6411,7 @@ the SQL <code>varchar</code> columns will be mapped to the <a hre
logging, or metrics) across thread boundaries without depending on any specific instrumentation library.
You can use the <a href="https://docs.rs/datafusion/latest/datafusion/common/runtime/trait.JoinSetTracer.html">JoinSetTracer</a> API to instrument DataFusion plans with your own tracing or logging libraries, or
use pre-integrated community crates such as the <a href="https://github.com/datafusion-contrib/datafusion-tracing">datafusion-tracing</a> crate.</p>
-<div style="text-align: center;">
+<div class="text-center">
<a href="https://github.com/datafusion-contrib/datafusion-tracing">
<img alt="DataFusion telemetry project logo" class="img-fluid" src="/blog/images/datafusion-47.0.0/datafusion-telemetry.png" width="50%"/>
</a>
@@ -6155,7 +6469,7 @@ community discussions – whether you have questions, want to share how you’re
contribute, we’d love to hear from you. A list of open issues suitable for beginners
is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p>
-<p>Happy querying!</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.9.0 Release</title><link href="https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0" rel="alternate"></link><published>2025-07-01T00:00:00+00:00</published><updated>2025-07-01T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-07-01:/blog/2025/07/01/datafusion-comet-0.9.0</id><summary type="html"><!--
+<p>Happy querying!</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.9.0 Release</title><link href="https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0" rel="alternate"/><published>2025-07-01T00:00:00+00:00</published><updated>2025-07-01T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-07-01:/blog/2025/07/01/datafusion-comet-0.9.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6302,7 +6616,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</title><link href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="alternate"></link><published>2025-06-30T00:00:00+00:00</published><updated>2025-06-30T00:00:00+00:00</updat [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</title><link href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="alternate"/><published>2025-06-30T00:00:00+00:00</published><updated>2025-06-30T00:00:00+00:00</updated><author><name [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6621,7 +6935,7 @@ The task will then yield, after which the Tokio runtime resets the counter.</
</figure>
<p>If we assume a task budget of 1 unit, each time Tokio schedules the task would result in the following sequence of function calls.</p>
<figure>
-<img alt="Sequence diagram showing how the tokio task budget is used and reset." class="img-fluid" src="/blog/images/task-cancellation/tokio_budget.png" style="width: 100%; max-width: 100%"/>
+<img alt="Sequence diagram showing how the tokio task budget is used and reset." class="img-fluid" src="/blog/images/task-cancellation/tokio_budget.png"/>
<figcaption>Tokio task budget system, assuming the task budget is set to 1, for the plan above.</figcaption>
</figure>
<p>The aggregation stream would try to poll the merge stream in a loop.
@@ -6709,7 +7023,7 @@ components, rather than as a single tightly integrated system.</p>
<p>The <a href="https://datafusion.apache.org/contributor-guide/communication.html">DataFusion community</a> is always looking for new contributors to help
improve the project. If you are interested in learning more about how query
execution works, help document or improve the DataFusion codebase, or just try
-it out, we would love for you to join us.</p></content><category term="blog"></category></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one" rel="alternate"></link><published>2025-06-15T00:00:00+00:00</published><updated>2025-06-15T00:00:00+00:00</updated><author><name>alamb, akurmustafa</name></author><id>tag:datafusion.apache.org [...]
+it out, we would love for you to join us.</p></content><category term="blog"/></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one" rel="alternate"/><published>2025-06-15T00:00:00+00:00</published><updated>2025-06-15T00:00:00+00:00</updated><author><name>alamb, akurmustafa</name></author><id>tag:datafusion.apache.org,2025-06-15:/blo [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6927,7 +7241,7 @@ optimizations</a>.</p>
<p><a id="footnote3"></a><sup>[3]</sup> <a href="https://www.vldb.org/pvldb/vol17/p1350-justen.pdf">https://www.vldb.org/pvldb/vol17/p1350-justen.pdf</a></p>
<p><a id="footnote4"></a><sup>[4]</sup> <a href="https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/24101">https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/24101</a>, <a href="https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/22111">https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/22111</a>, <a href="https://www.dagstuhl.de/en/seminars/seminar-calendar/sem [...]
<p><a id="footnote5"></a><sup>[5]</sup> And thus in academic classes, by the time you get around to an optimizer the semester is over and everyone is ready for the semester to be done. Once industrial systems mature to the point where the optimizer is a bottleneck, the shiny new-ness of the<a href="https://en.wikipedia.org/wiki/Gartner_hype_cycle"> hype cycle</a> has worn off and it is likely in the trough of disappointment.</p>
-<p><a id="footnote6"></a><sup>[6]</sup> Often systems will classify these passes into different categories, but I am simplifying here</p></content><category term="blog"></category></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="alternate"></link><published>2025-06-15T00:00:00+00:00</publi [...]
+<p><a id="footnote6"></a><sup>[6]</sup> Often systems will classify these passes into different categories, but I am simplifying here</p></content><category term="blog"/></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="alternate"/><published>2025-06-15T00:00:00+00:00</published><updated>20 [...]
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -7310,7 +7624,7 @@ community</a>. We welcome first time contributors as well as long time par
to the fun of building a database together.</p>
<h2 id="notes">Notes<a class="headerlink" href="#notes" title="Permanent link">¶</a></h2>
<p><a id="footnote7"></a><sup>[7]</sup> See <a href="https://btw-2015.informatik.uni-hamburg.de/res/proceedings/Hauptband/Wiss/Neumann-Unnesting_Arbitrary_Querie.pdf">Unnesting Arbitrary Queries</a> from Neumann and Kemper for a more academic treatment.</p>
-<p><a id="footnote8"></a><sup>[8]</sup> One of my favorite terms I learned from Andy Pavlo’s CMU online lectures</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.8.0 Release</title><link href="https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0" rel="alternate"></link><published>2025-05-06T00:00:00+00:00</published><updated>2025-05-06T00:00:00+00:00</updated><author><name>pmc</name></author> [...]
+<p><a id="footnote8"></a><sup>[8]</sup> One of my favorite terms I learned from Andy Pavlo’s CMU online lectures</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.8.0 Release</title><link href="https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0" rel="alternate"/><published>2025-05-06T00:00:00+00:00</published><updated>2025-05-06T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusi [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -7397,7 +7711,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>User defined Window Functions in DataFusion</title><link href="https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions" rel="alternate"></link><published>2025-04-19T00:00:00+00:00</published><updated>2025-04-19T00:00:00+00:00</updated><author>< [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>User defined Window Functions in DataFusion</title><link href="https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions" rel="alternate"/><published>2025-04-19T00:00:00+00:00</published><updated>2025-04-19T00:00:00+00:00</updated><author><name>Aditya Sing [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -7750,7 +8064,7 @@ giving it a try. This post was designed to make it easier for new users to work
<p>When it comes to designing UDFs, I strongly recommend reviewing the
<a href="https://datafusion.apache.org/library-user-guide/adding-udfs.html">Window functions</a> documentation.</p>
<p>A heartfelt thank you to <a href="https://github.com/alamb">@alamb</a> and <a href="https://github.com/andygrove">@andygrove</a> for their invaluable reviews and thoughtful feedback—they’ve been instrumental in shaping this post.</p>
-<p>The Apache Arrow and Apache DataFusion communities are vibrant, welcoming, and full of passionate developers building something truly powerful. If you’re excited about high-performance analytics and want to be part of an open-source journey, I highly encourage you to explore the <a href="(https://datafusion.apache.org/)">official documentation</a> and dive into one of the many <a href="https://github.com/apache/datafusion/issues">open issues</a>. There’s [...]
+<p>The Apache Arrow and Apache DataFusion communities are vibrant, welcoming, and full of passionate developers building something truly powerful. If you’re excited about high-performance analytics and want to be part of an open-source journey, I highly encourage you to explore the <a href="(https://datafusion.apache.org/)">official documentation</a> and dive into one of the many <a href="https://github.com/apache/datafusion/issues">open issues</a>. There’s [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -7767,19 +8081,11 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
-development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over …</p></summary><content type="html"><!--
+development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
+faster than any other implementation we know of.</p>
+<p>It is now possible to create …</p></summary><content type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -7796,16 +8102,6 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
@@ -8220,7 +8516,7 @@ DataFusion (see <a href="https://github.com/apache/datafusion/issues/14608"&g
<p><em>2</em>: It is possible to embed the dbgen code, which appears to be the approach taken by DuckDB. This approach was tried in GlareDB (<a href="https://github.com/GlareDB/glaredb/pull/3313">GlareDB/glaredb#3313</a>), but ultimately shelved given the amount of effort needed to adapt and isolate the dbgen code.</p>
<p><em>3</em>: It is pretty amazing to imagine the machine required to generate SF300 that had 1.8TB (!!) of RAM</p>
<p><em>4</em>: We tried to <a href="https://github.com/clflushopt/tpchgen-rs/pull/34">use Rayon (see discussion here)</a>, but could not easily keep memory bounded.</p>
-<p><em>5</em>: <code>tpchgen-cli</code> memory usage is a function of the number of threads: each thread needs some buffer space</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Python 46.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/03/30/datafusion-python-46.0.0" rel="alternate"></link><published>2025-03-30T00:00:00+00:00</published><updated>2025-03-30T00:00:00+00:00</updated><author><na [...]
+<p><em>5</em>: <code>tpchgen-cli</code> memory usage is a function of the number of threads: each thread needs some buffer space</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Python 46.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/03/30/datafusion-python-46.0.0" rel="alternate"/><published>2025-03-30T00:00:00+00:00</published><updated>2025-03-30T00:00:00+00:00</updated><author><name>timsaucer</na [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -8355,8 +8651,8 @@ for the user to click on it to expand the data.</li>
</ul>
<p>In the below view you can see an example of some of these features such as the
expandable text and scroll bars.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Example html rendering in a jupyter notebook." class="img-fluid" src="/blog/images/python-datafusion-46.0.0/html_rendering.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Example html rendering in a jupyter notebook." class="img-fluid" src="/blog/images/python-datafusion-46.0.0/html_rendering.png"/>
<figcaption>
<b>Figure 1</b>: With the html rendering enhancements, tables are more easily
viewable in jupyter notebooks.
@@ -8435,7 +8731,7 @@ to have you join us and help the project.</p>
<li>
<p>Join us on <a href="https://s.apache.org/slack-invite">ASF Slack</a> or the <a href="https://discord.gg/Qw5gKqHxUM">Arrow Rust Discord Server</a>.</p>
</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache DataFusion 46.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0" rel="alternate"></link><published>2025-03-24T00:00:00+00:00</published><updated>2025-03-24T00:00:00+00:00</updated><author><name>Oznur Hanci and Berkay Sahin on behalf of the PMC</name></author><id>tag:datafusion.apache.org,2025-03-24:/blog/2025/03/24/datafusion-46.0.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache DataFusion 46.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0" rel="alternate"/><published>2025-03-24T00:00:00+00:00</published><updated>2025-03-24T00:00:00+00:00</updated><author><name>Oznur Hanci and Berkay Sahin on behalf of the PMC</name></author><id>tag:datafusion.apache.org,2025-03-24:/blog/2025/03/24/datafusion-46.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -8539,7 +8835,7 @@ SELECT col2, col1 FROM t2;
<p>Upgrading to 46.0.0 should be straightforward for most users, but do review the <a href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade Guide for DataFusion 46.0.0</a> for detailed steps and code changes. The upgrade guide covers the breaking changes mentioned (like replacing old exec nodes with <code>DataSourceExec</code>, updating UDF invocation to <code>invoke_with_args</code>, etc.) and provides code snippets to h [...]
<h2 id="get-involved">Get Involved<a class="headerlink" href="#get-involved" title="Permanent link">¶</a></h2>
<p>Apache DataFusion is an open-source project, and we welcome involvement from anyone interested. Now is a great time to take 46.0.0 for a spin: try it out on your workloads, and let us know if you encounter any issues or have suggestions. You can report bugs or request features on our GitHub issue tracker, or better yet, submit a pull request. Join our community discussions – whether you have questions, want to share how you’re using DataFusion, or are looking to contribute, we’d [...]
-<p>Happy querying!</p></content><category term="blog"></category></entry><entry><title>Efficient Filter Pushdown in Parquet</title><link href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown" rel="alternate"></link><published>2025-03-21T00:00:00+00:00</published><updated>2025-03-21T00:00:00+00:00</updated><author><name>Xiangpeng Hao</name></author><id>tag:datafusion.apache.org,2025-03-21:/blog/2025/03/21/parquet-pushdown</id><summary type="html"><style>
+<p>Happy querying!</p></content><category term="blog"/></entry><entry><title>Efficient Filter Pushdown in Parquet</title><link href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown" rel="alternate"/><published>2025-03-21T00:00:00+00:00</published><updated>2025-03-21T00:00:00+00:00</updated><author><name>Xiangpeng Hao</name></author><id>tag:datafusion.apache.org,2025-03-21:/blog/2025/03/21/parquet-pushdown</id><summary type="html"><style>
figure {
margin: 20px 0;
}
@@ -8829,7 +9125,7 @@ and queries that get slower are likely due to noise.</p>
<p>Despite being simple in theory, filter pushdown in Parquet is non-trivial to implement.
It requires understanding both the Parquet format and reader implementation details.
The challenge lies in efficiently navigating through the dynamics of decoding, filter evaluation, and memory management.</p>
-<p>If you are interested in this level of optimization and want to help test, document and implement this type of optimization, come find us in the <a href="https://datafusion.apache.org/contributor-guide/communication.html">DataFusion Community</a>. We would love to have you. </p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.7.0 Release</title><link href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0 [...]
+<p>If you are interested in this level of optimization and want to help test, document and implement this type of optimization, come find us in the <a href="https://datafusion.apache.org/contributor-guide/communication.html">DataFusion Community</a>. We would love to have you. </p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.7.0 Release</title><link href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0" rel= [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -8931,7 +9227,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="alternate"></link><published>2025-03-20T00:00:00+00:00</published><updated>2025-03-20T00:00:00+00:00</updated><author><name [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="alternate"/><published>2025-03-20T00:00:00+00:00</published><updated>2025-03-20T00:00:00+00:00</updated><author><name>Xiangpeng Hao</ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9019,7 +9315,7 @@ The last step is to <a href="https://github.com/apache/datafusion/blob/31701b
<p>We are working to remove the <a href="https://github.com/apache/arrow-rs/issues/5523#issuecomment-2429470872">remaining performance issues</a> and enable it by default, which we will discuss in the next blog post.</p>
<h3 id="conclusion">Conclusion<a class="headerlink" href="#conclusion" title="Permanent link">¶</a></h3>
<p>DataFusion employs a multi-step approach to Parquet pruning, from column projection to row group stats, page stats, and potentially row-level filtering.
-Each step may reduce the amount of data to be read and processed, significantly improving query performance.</p></content><category term="blog"></category></entry><entry><title>Using Ordering for Better Plans in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="alternate"></link><published>2025-03-11T00:00:00+00:00</published><updated>2025-03-11T00:00:00+00:00</updated><author><name>Mustafa Akur, Andrew Lamb</name></author><i [...]
+Each step may reduce the amount of data to be read and processed, significantly improving query performance.</p></content><category term="blog"/></entry><entry><title>Using Ordering for Better Plans in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="alternate"/><published>2025-03-11T00:00:00+00:00</published><updated>2025-03-11T00:00:00+00:00</updated><author><name>Mustafa Akur, Andrew Lamb</name></author><id>tag:datafusion [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9041,7 +9337,7 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></summary><content type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -9064,7 +9360,7 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
</blockquote>
<p>There are various use cases where this type of analysis can be useful such as the following examples.</p>
@@ -9145,7 +9441,7 @@ Being logically streamable does not guarantee that a query will execute in a str
</tr>
</table>
<p><br/></p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>How can a table have multiple orderings?</strong> At first glance it may seem counterintuitive for a table to have more than one valid ordering. However, during query execution such scenarios can arise.
For example consider the following query:
@@ -9206,7 +9502,7 @@ DataFusion leverages these functional dependencies to infer new orderings as dat
<li>Equivalent Expression Groups (will be explained shortly)</li>
<li>Succinct Valid Orderings (will be explained shortly)</li>
</ul>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Note:</strong> These properties are implemented in the <code>EquivalenceProperties</code> structure in <code>DataFusion</code>, please see the <a href="https://github.com/apache/datafusion/blob/f47ea73b87eec4af044f9b9923baf042682615b2/datafusion/physical-expr/src/equivalence/properties/mod.rs#L134" target="_blank">source</a> for more details<br/>
</blockquote>
<p>These properties allow us to analyze whether the ordering requirement is satisfied by the data already.</p>
@@ -9216,7 +9512,7 @@ DataFusion leverages these functional dependencies to infer new orderings as dat
<ul>
<li>Columns <code>hostname</code> and <code>currency</code> are constant because every row in the table has the same value (<code>'app.example.com'</code> for <code>hostname</code>, and <code>'USD'</code> for <code>currency</code>) for these columns.</li>
</ul>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Note:</strong> Constant expressions can arise during query execution. For example, in following query:<br/>
<code>SELECT hostname FROM logs</code><br/><code>WHERE hostname='app.example.com'</code> <br/>
after filtering is done, for subsequent operators the <code>hostname</code> column will be constant.
@@ -9224,7 +9520,7 @@ DataFusion leverages these functional dependencies to infer new orderings as dat
<h3 id="2-equivalent-expression-groups">2. Equivalent Expression Groups<a class="headerlink" href="#2-equivalent-expression-groups" title="Permanent link">¶</a></h3>
<p>Equivalent expression groups are expressions that always hold the same value across rows. These expressions can be thought of as clones of each other and may arise from operations like <code>Filter</code>, <code>Join</code>, or <code>Projection</code>.</p>
<p>In the example table, the expressions <code>price</code> and <code>price_cloned</code> form one equivalence group, and <code>time</code> and <code>time_cloned</code> form another equivalence group.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Note:</strong> Equivalent expression groups can arise during the query execution. For example, in the following query:<br/>
<code>SELECT time, time as time_cloned FROM logs</code> <br/>
after the projection is done, for subsequent operators <code>time</code> and <code>time_cloned</code> will form an equivalence group. As another example, in the following query:<br/>
@@ -9292,7 +9588,7 @@ Following third and fourth constraints for the simplified table, the succinct va
<code>[amount ASC, price ASC]</code>, <br/>
<code>[time_bin ASC]</code>,<br/>
<code>[time ASC]</code> </p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<p><strong>How can DataFusion find orderings?</strong></p>
DataFusion's <code>CREATE EXTERNAL TABLE</code> has a <code>WITH ORDER</code> clause (see <a href="https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table">docs</a>) to specify the known orderings of the table during table creation. For example the following query:<br/>
<pre><code>
@@ -9412,7 +9708,7 @@ We go back to step 4 until the ordering requirement list is exhausted or its len
<p id="footnote4"><sup>[4]</sup>This means that, if ordering <code>[amount ASC, price ASC]</code> is a valid ordering for the table. We shouldn't enlist <code>[amount ASC]</code> as valid ordering. Validity of it can be deduced from the ordering: <code>[amount ASC, price ASC]</code>
<p id="footnote5"><sup>[5]</sup>Leading ordering requirement is the first ordering requirement in the list of lexicographic ordering requirement expression. As an example for the requirement: <code>[amount ASC, time_bin ASC, prices ASC, time ASC]</code>, leading ordering requirement is: <code>amount ASC</code>.</p>
<p id="footnote6"><sup>[6]</sup>Leading valid orderings are the first ordering for each valid ordering list in the table. As an example, for the valid orderings: <code>[amount ASC, prices ASC], [time_bin ASC], [time ASC]</code>, leading valid orderings will be: <code>amount ASC, time_bin ASC, time ASC</code>. </p>
-<p id="optimal"><sup>*</sup>Best depends on the use case, <code>DataFusion</code> has many various flags to communicate what user thinks the best plan is (e.g. streamable, fastest, lowest memory, etc.). See <a href="https://datafusion.apache.org/user-guide/configs.html" target="_blank">configurations</a> for detail.</p></p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 45.0.0 Released</title><link h [...]
+<p id="optimal"><sup>*</sup>Best depends on the use case, <code>DataFusion</code> has many various flags to communicate what user thinks the best plan is (e.g. streamable, fastest, lowest memory, etc.). See <a href="https://datafusion.apache.org/user-guide/configs.html" target="_blank">configurations</a> for detail.</p></p></content><category term="blog"/></entry><entry><title>Apache DataFusion 45.0.0 Released</title><link href="https [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9602,7 +9898,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
-can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="alternate"></link><published>2025-02-17T00:00:00+00:00</published><updated>2025-02-17T00:00:00+00:00</updated><author><name>pmc</name></author [...]
+can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="alternate"/><published>2025-02-17T00:00:00+00:00</published><updated>2025-02-17T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafus [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9682,7 +9978,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Ballista 43.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0" rel="alternate"></link><published>2025-02-02T00:00:00+00:00</published><updated>2025-02-02T00:00:00+00:00</updated><author><name [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Ballista 43.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0" rel="alternate"/><published>2025-02-02T00:00:00+00:00</published><updated>2025-02-02T00:00:00+00:00</updated><author><name>milenkovicm</na [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9774,7 +10070,7 @@ async fn main() -&gt; datafusion::error::Result&lt;()&gt; {
<h2 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">¶</a></h2>
<p>Moving forward, Ballista will adopt the same release cadence as DataFusion, providing synchronized updates across the ecosystem.
Currently, there is no established long-term roadmap for Ballista. A plan will be formulated in the coming months based on community feedback and the availability of additional maintainers.</p>
-<p>In the short term, development efforts will concentrate on closing the feature gap between DataFusion and Ballista. Key priorities include implementing support for <code>INSERT INTO</code>, enabling table <code>URL</code> functionality, and achieving deeper integration with the Python ecosystem.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.5.0 Release</title><link href="https://datafusion.apache.org/b [...]
+<p>In the short term, development efforts will concentrate on closing the feature gap between DataFusion and Ballista. Key priorities include implementing support for <code>INSERT INTO</code>, enabling table <code>URL</code> functionality, and achieving deeper integration with the Python ecosystem.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.5.0 Release</title><link href="https://datafusion.apache.org/blog/2025/0 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9869,7 +10165,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Python 43.1.0 Released</title><link href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0" rel="alternate"></link><published>2024-12-14T00:00:00+00:00</published><updated>2024-12-14T00:00:00+00:00</updated><author><name>tim [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Python 43.1.0 Released</title><link href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0" rel="alternate"/><published>2024-12-14T00:00:00+00:00</published><updated>2024-12-14T00:00:00+00:00</updated><author><name>timsaucer</name></a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10022,7 +10318,7 @@ page.</p>
<li>
<p>Try out the project and provide feedback, file issues, and contribute code.</p>
</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.4.0 Release</title><link href="https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0" rel="alternate"></link><published>2024-11-20T00:00:00+00:00</published><updated>2024-11-20T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-11-20:/blog/2024/11/20/datafusion-comet-0.4.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.4.0 Release</title><link href="https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0" rel="alternate"/><published>2024-11-20T00:00:00+00:00</published><updated>2024-11-20T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-11-20:/blog/2024/11/20/datafusion-comet-0.4.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10122,7 +10418,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Comparing approaches to User Defined Functions in Apache DataFusion using Python</title><link href="https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons" rel="alternate"></link><published>2024-11-19T00:00:00+00:00</published><updated>202 [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Comparing approaches to User Defined Functions in Apache DataFusion using Python</title><link href="https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons" rel="alternate"/><published>2024-11-19T00:00:00+00:00</published><updated>2024-11-19T00:00:00 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10686,7 +10982,7 @@ for their helpful reviews and feedback.</p>
<p>Lastly, the Apache Arrow and DataFusion community is an active group of very helpful people working
to make a great tool. If you want to get involved, please take a look at the
<a href="https://datafusion.apache.org/python/">online documentation</a> and jump in to help with one of the
-<a href="https://github.com/apache/datafusion-python/issues">open issues</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</title><link href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" rel="alternate"></link><published>2024-11-18T00:00:00+00:00</published><updated>2024-11-18T00:00:00+00:00</updated><author [...]
+<a href="https://github.com/apache/datafusion-python/issues">open issues</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</title><link href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" rel="alternate"/><published>2024-11-18T00:00:00+00:00</published><updated>2024-11-18T00:00:00+00:00</updated><author><name>Andrew La [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10912,7 +11208,7 @@ online</a>.</p>
<p>[^2]: Want to try your hand at a custom format for ClickBench fame / glory?: <a href="https://github.com/apache/datafusion/issues/13448">Make DataFusion the fastest engine in ClickBench with custom file format</a></p>
<p>[^3]: We have contributors from North America, South American, Europe, Asia, Africa and Australia</p>
<p>[^4]: Undergraduates, PhD, Junior engineers, and getting-kind-of-crotchety experienced engineers</p>
-<p>[^5]: Thanks to Andy Pavlo, I love that nomenclature</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.3.0 Release</title><link href="https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0" rel="alternate"></link><published>2024-09-27T00:00:00+00:00</published><updated>2024-09-27T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-09-27:/blog/2024/09/27/datafusion-comet-0.3.0< [...]
+<p>[^5]: Thanks to Andy Pavlo, I love that nomenclature</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.3.0 Release</title><link href="https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0" rel="alternate"/><published>2024-09-27T00:00:00+00:00</published><updated>2024-09-27T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-09-27:/blog/2024/09/27/datafusion-comet-0.3.0</id><summary typ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10998,7 +11294,7 @@ project.</p>
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</title><link href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1" rel="alternate"></link><published>2024-09-13T00:00:00+00:00</published [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</title><link href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1" rel="alternate"/><published>2024-09-13T00:00:00+00:00</published><updated>2024-0 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11129,7 +11425,7 @@ along with some of the pitfalls we encountered while implementing them.</p>
<p>[^1]: Benchmarked with AMD Ryzen 7600x (12 core, 24 threads, 32 MiB L3), WD Black SN770 NVMe SSD (5150MB/4950MB seq RW bandwidth)</p>
<p>[^2]: Xiangpeng is a PhD student at the University of Wisconsin-Madison</p>
<p>[^3]: There is also a corresponding <em>BinaryViewArray</em> which is similar except that the data is not constrained to be UTF-8 encoded strings.</p>
-<p>[^4]: We also make sure that offsets do not break a UTF-8 code point, which is <a href="https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/buffer/offset_buffer.rs#L62-L71">cheaply validated</a>.</p></content><category term="blog"></category></entry><entry><title>Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</title><link href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings [...]
+<p>[^4]: We also make sure that offsets do not break a UTF-8 code point, which is <a href="https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/buffer/offset_buffer.rs#L62-L71">cheaply validated</a>.</p></content><category term="blog"/></entry><entry><title>Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</title><link href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2" r [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11238,7 +11534,7 @@ In certain cases, we found that multiple calls to <a href="https://docs.rs/ar
<p>StringView is a big project that has received tremendous community support. Specifically, we would like to thank <a href="https://github.com/tustvold">@tustvold</a>, <a href="https://github.com/ariesdevil">@ariesdevil</a>, <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a>, <a href="https://github.com/ClSlaid">@ClSlaid</a>, <a href="https://github.com/2010YOUY01">@2010YOUY01</a>, <a href="https://github.com/ [...]
<p>As the introduction states, “German Style Strings” is a relatively straightforward research idea that avoid some string copies and accelerates comparisons. However, applying this (great) idea in practice requires a significant investment in careful software engineering. Again, we encourage the research community to continue to help apply research ideas to industrial systems, such as DataFusion, as doing so provides valuable perspectives when evaluating future research questions [...]
<h3 id="footnotes">Footnotes<a class="headerlink" href="#footnotes" title="Permanent link">¶</a></h3>
-<p>[^5]: There are additional optimizations possible in this operation that the community is working on, such as <a href="https://github.com/apache/datafusion/issues/7957">https://github.com/apache/datafusion/issues/7957</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.2.0 Release</title><link href="https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0" rel="alternate"></link><published>2024-08-28T0 [...]
+<p>[^5]: There are additional optimizations possible in this operation that the community is working on, such as <a href="https://github.com/apache/datafusion/issues/7957">https://github.com/apache/datafusion/issues/7957</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.2.0 Release</title><link href="https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0" rel="alternate"/><published>2024-08-28T00:00:00+00:00</p [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11327,7 +11623,7 @@ project.</p>
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Python 40.1.0 Released, Significant usability updates</title><link href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0" rel="alternate"></link><published>2024-08-20T00:00:00+00:00</published><updated>2024-08-20T00:00:00+0 [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Python 40.1.0 Released, Significant usability updates</title><link href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0" rel="alternate"/><published>2024-08-20T00:00:00+00:00</published><updated>2024-08-20T00:00:00+00:00</updated><a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11399,8 +11695,8 @@ the available functions and classes to see the breadth of available functionalit
<a href="https://jedi.readthedocs.io/en/latest/">Jedi</a> to perform analysis of python code, provide useful
hints, and identify usage errors. These are major tools in the python user community. With this
release, users can fully use these tools in their workflow.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Enhanced tooltips in an IDE." class="img-fluid" src="/blog/images/python-datafusion-40.0.0/vscode_hover_tooltip.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Enhanced tooltips in an IDE." class="img-fluid" src="/blog/images/python-datafusion-40.0.0/vscode_hover_tooltip.png"/>
<figcaption>
<b>Figure 1</b>: With the enhanced python wrappers, users can see helpful tool tips with
type annotations directly in modern IDEs.
@@ -11408,8 +11704,8 @@ release, users can fully use these tools in their workflow.</p>
</figure>
<p>By having the type annotations, these IDEs can also identify quickly when a user has incorrectly
used a function's arguments as shown in Figure 2.</p>
-<figure style="text-align: center;">
-<img alt="Fig 2: Error checking in static analysis" class="img-fluid" src="/blog/images/python-datafusion-40.0.0/pylance_error_checking.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 2: Error checking in static analysis" class="img-fluid" src="/blog/images/python-datafusion-40.0.0/pylance_error_checking.png"/>
<figcaption>
<b>Figure 2</b>: Modern Python language servers can perform static analysis and quickly find
errors in the arguments to functions.
@@ -11499,7 +11795,7 @@ page.</p>
<li>
<p>Try out the project and provide feedback, file issues, and contribute code.</p>
</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache DataFusion 40.0.0 Released</title><link href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0" rel="alternate"></link><published>2024-07-24T00:00:00+00:00</published><updated>2024-07-24T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-07-24:/blog/2024/07/24/datafusion-40.0.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache DataFusion 40.0.0 Released</title><link href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0" rel="alternate"/><published>2024-07-24T00:00:00+00:00</published><updated>2024-07-24T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-07-24:/blog/2024/07/24/datafusion-40.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11826,7 +12122,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
-can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.1.0 Release</title><link href="https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0" rel="alternate"></link><published>2024-07-20T00:00:00+00:00</published><updated>2024-07-20T00:00:00+00:00</updated><author><name>pmc</name></author [...]
+can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.1.0 Release</title><link href="https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0" rel="alternate"/><published>2024-07-20T00:00:00+00:00</published><updated>2024-07-20T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafus [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11942,7 +12238,7 @@ recordings of previous calls.</p>
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Announcing Apache Arrow DataFusion is now Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp" rel="alternate"></link><published>2024-05-07T00:00:00+00:00</published><updated>2024-05-07T00:00:00+00:00</updated><author> [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Announcing Apache Arrow DataFusion is now Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp" rel="alternate"/><published>2024-05-07T00:00:00+00:00</published><updated>2024-05-07T00:00:00+00:00</updated><author><name>pmc</name> [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12021,7 +12317,7 @@ DataFusion</a> website.</p>
us. Get in touch using <a href="https://datafusion.apache.org/contributor-guide/communication.html">Communication Doc</a> and learn how to get involved in the
<a href="https://datafusion.apache.org/contributor-guide/index.html">Contributor Guide</a>. We welcome everyone to try DataFusion on their
own data and projects and let us know how it goes, contribute suggestions,
-documentation, bug reports, or a PR with documentation, tests or code.</p></content><category term="blog"></category></entry><entry><title>Announcing Apache Arrow DataFusion Comet</title><link href="https://datafusion.apache.org/blog/2024/03/06/comet-donation" rel="alternate"></link><published>2024-03-06T00:00:00+00:00</published><updated>2024-03-06T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-03-06:/blog/2024/03/06/comet-donation</id> [...]
+documentation, bug reports, or a PR with documentation, tests or code.</p></content><category term="blog"/></entry><entry><title>Announcing Apache Arrow DataFusion Comet</title><link href="https://datafusion.apache.org/blog/2024/03/06/comet-donation" rel="alternate"/><published>2024-03-06T00:00:00+00:00</published><updated>2024-03-06T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-03-06:/blog/2024/03/06/comet-donation</id><summary type="h [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12071,8 +12367,8 @@ a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/dat
accelerate Spark workloads. It is designed as a drop-in
replacement for Spark's JVM based SQL execution engine and offers significant
performance improvements for some workloads as shown below.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-comet/comet-architecture.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-comet/comet-architecture.png"/>
<figcaption>
<b>Figure 1</b>: With Comet, users interact with the same Spark ecosystem, tools
and APIs such as Spark SQL. Queries still run through Spark's query optimizer and planner.
@@ -12105,7 +12401,7 @@ expect to post another update with more details at that time.</p>
<li>
<p>Try out the project and provide feedback, file issues, and contribute code.</p>
</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</title><link href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0" rel="alternate"></link><published>2024-01-19T00:00:00+00:00</published><updated>2024-01-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-01-19:/blog/2024/01/19/datafusion-34.0.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</title><link href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0" rel="alternate"/><published>2024-01-19T00:00:00+00:00</published><updated>2024-01-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-01-19:/blog/2024/01/19/datafusion-34.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12201,16 +12497,16 @@ more than 2x faster on <a href="https://benchmark.clickhouse.com/">ClickBe
Average runtime on 25.0.0: 7.2s (for the queries that actually ran)
Average runtime on 34.0.0: 3.6s (for the same queries that ran in 25.0.0)
-->
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare-new.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare-new.png"/>
<figcaption>
<b>Figure 1</b>: Performance improvement between <code>25.0.0</code> and <code>34.0.0</code> on ClickBench.
Note that DataFusion <code>25.0.0</code>, could not run several queries due to
unsupported SQL (Q9, Q11, Q12, Q14) or memory requirements (Q33).
</figcaption>
</figure>
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare.png"/>
<figcaption>
<b>Figure 2</b>: Total query runtime for DataFusion <code>34.0.0</code> and DataFusion <code>25.0.0</code>.
</figcaption>
@@ -12372,7 +12668,7 @@ suitable for beginners is <a href="https://github.com/apache/arrow-datafusion
meetings. Timezones are always a challenge for such meetings, but we hope to
have two calls that can work for most attendees. If you are interested
in helping, or just want to say hi, please drop us a note via one of
-the methods listed in our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a>.</p></content><category term="blog"></category></entry><entry><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link href="https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping" rel="alternate"></link><published>2023-08-05T00:00:00+00:00</published><updated>2023-08-05T00:00:00+00:00</update [...]
+the methods listed in our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a>.</p></content><category term="blog"/></entry><entry><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link href="https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping" rel="alternate"/><published>2023-08-05T00:00:00+00:00</published><updated>2023-08-05T00:00:00+00:00</updated><author><name> [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12695,7 +12991,7 @@ allocation using the arrow Row format
<p>[^3]: <code>SELECT COUNT(DISTINCT "SearchPhrase") as num_phrases FROM 'hits.parquet';</code></p>
<p>[^4]: <code>SELECT COUNT(*) FROM (SELECT DISTINCT "UserID", "SearchPhrase" FROM 'hits.parquet')</code></p>
<p>[^5]: Full script at <a href="https://github.com/alamb/datafusion-duckdb-benchmark/blob/main/hash.py">hash.py</a></p>
-<p>[^6]: <a href="https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_%7B%7D.parquet">hits_0.parquet</a>, one of the files from the partitioned ClickBench dataset, which has <code>100,000</code> rows and is 117 MB in size. The entire dataset has <code>100,000,000</code> rows in a single 14 GB Parquet file. The script did not complete on the entire dataset after 40 minutes, and used 212 GB RAM at peak.</p></content><categ [...]
+<p>[^6]: <a href="https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_%7B%7D.parquet">hits_0.parquet</a>, one of the files from the partitioned ClickBench dataset, which has <code>100,000</code> rows and is 117 MB in size. The entire dataset has <code>100,000,000</code> rows in a single 14 GB Parquet file. The script did not complete on the entire dataset after 40 minutes, and used 212 GB RAM at peak.</p></content><categ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12915,7 +13211,7 @@ data and projects and let us know how it goes or contribute a PR with
documentation, tests or code. A list of open issues suitable for
beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
<p>Check out our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a> for more ways to engage with the
-community.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 16.0.0 Project Update</title><link href="https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0" rel="alternate"></link><published>2023-01-19T00:00:00+00:00</published><updated>2023-01-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2023-01-19:/blog/2023/01/19/datafusion-16.0.0</id><summary type="html"><!--
+community.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 16.0.0 Project Update</title><link href="https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0" rel="alternate"/><published>2023-01-19T00:00:00+00:00</published><updated>2023-01-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2023-01-19:/blog/2023/01/19/datafusion-16.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13159,7 +13455,7 @@ ways to engage with the community.</p>
1 yujie.zhang
1 zembunia
1 哇呜哇呜呀咦耶
-</code></pre></content><category term="blog"></category></entry><entry><title>Apache Arrow Ballista 0.9.0 Release</title><link href="https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0" rel="alternate"></link><published>2022-10-28T00:00:00+00:00</published><updated>2022-10-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-10-28:/blog/2022/10/28/ballista-0.9.0</id><summary type="html"><!--
+</code></pre></content><category term="blog"/></entry><entry><title>Apache Arrow Ballista 0.9.0 Release</title><link href="https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0" rel="alternate"/><published>2022-10-28T00:00:00+00:00</published><updated>2022-10-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-10-28:/blog/2022/10/28/ballista-0.9.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13259,7 +13555,7 @@ tuning Ballista.</p>
<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
<p>Ballista has a friendly community and we welcome contributions. A good place to start is to following the instructions
in the <a href="https://arrow.apache.org/ballista/">user guide</a> and try using Ballista with your own SQL queries and ETL pipelines, and file issues
-for any bugs or feature suggestions.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 13.0.0 Project Update</title><link href="https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0" rel="alternate"></link><published>2022-10-25T00:00:00+00:00</published><updated>2022-10-25T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-10-25:/blog/2022/10/25/datafusion-13.0.0</id><summary type="html"><!--
+for any bugs or feature suggestions.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 13.0.0 Project Update</title><link href="https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0" rel="alternate"/><published>2022-10-25T00:00:00+00:00</published><updated>2022-10-25T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-10-25:/blog/2022/10/25/datafusion-13.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13476,7 +13772,7 @@ ways to engage with the community.</p>
1 nvartolomei
1 yourenawo
1 Özgür Akkurt
-</code></pre></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 8.0.0 Release</title><link href="https://datafusion.apache.org/blog/2022/05/16/datafusion-8.0.0" rel="alternate"></link><published>2022-05-16T00:00:00+00:00</published><updated>2022-05-16T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-05-16:/blog/2022/05/16/datafusion-8.0.0</id><summary type="html"><!--
+</code></pre></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 8.0.0 Release</title><link href="https://datafusion.apache.org/blog/2022/05/16/datafusion-8.0.0" rel="alternate"/><published>2022-05-16T00:00:00+00:00</published><updated>2022-05-16T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-05-16:/blog/2022/05/16/datafusion-8.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13666,7 +13962,7 @@ love to have you join us on the journey! You can help by trying out DataFusion o
and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable
for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p></content><category term="blog"></category></entry><entry><title>Introducing Apache Arrow DataFusion Contrib</title><link href="https://datafusion.apache.org/blog/2022/03/21/datafusion-contrib" rel="alternate"></link><published>2022-03-21T00:00:00+00:00</published><updated>2022-03-21T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-03-21:/blog/2022/03/21/datafusion-contrib</id><summary type="html"><!--
+ways to engage with the community.</p></content><category term="blog"/></entry><entry><title>Introducing Apache Arrow DataFusion Contrib</title><link href="https://datafusion.apache.org/blog/2022/03/21/datafusion-contrib" rel="alternate"/><published>2022-03-21T00:00:00+00:00</published><updated>2022-03-21T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-03-21:/blog/2022/03/21/datafusion-contrib</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13820,7 +14116,7 @@ the art query processing, we would love to have you join us on the journey! You
can help by trying out DataFusion on some of your own data and projects and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
<p>The best way to find out about creating new extensions within DataFusion-Contrib is reaching out on the <code>#arrow-rust</code> channel of the Apache Software Foundation <a href="https://join.slack.com/t/the-asf/shared_invite/zt-vlfbf7ch-HkbNHiU_uDlcH_RvaHv9gQ">Slack</a> workspace.</p>
<p>You can also check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more ways to engage with the community.</p>
-<p>Links for each DataFusion-Contrib repository are provided above if you would like to contribute to those.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 7.0.0 Release</title><link href="https://datafusion.apache.org/blog/2022/02/28/datafusion-7.0.0" rel="alternate"></link><published>2022-02-28T00:00:00+00:00</published><updated>2022-02-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022 [...]
+<p>Links for each DataFusion-Contrib repository are provided above if you would like to contribute to those.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 7.0.0 Release</title><link href="https://datafusion.apache.org/blog/2022/02/28/datafusion-7.0.0" rel="alternate"/><published>2022-02-28T00:00:00+00:00</published><updated>2022-02-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-02-28:/blog/202 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13978,7 +14274,7 @@ git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc
the art query processing, we would love to have you join us on the journey! You
can help by trying out DataFusion on some of your own data and projects and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 6.0.0 Release</title><link href="https://datafusion.apache.org/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" rel="alternate"></link><published>2021-11-19T00:00:00+00:00</published><updated>2021-11-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-11-19:/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md</id><summary [...]
+ways to engage with the community.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 6.0.0 Release</title><link href="https://datafusion.apache.org/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" rel="alternate"/><published>2021-11-19T00:00:00+00:00</published><updated>2021-11-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-11-19:/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14134,7 +14430,7 @@ improve the documentation, or contribute to the documentation, tests or code. A
beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p>
<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow Ballista 0.5.0 Release</title><link href="https://datafusion.apache.org/blog/2021/08/18/ballista-0.5.0" rel="alternate"></link><published>2021-08-18T00:00:00+00:00</published><updated>2021-08-18T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/ballista-0.5.0</id><summary type="html"><!--
+ways to engage with the community.</p></content><category term="blog"/></entry><entry><title>Apache Arrow Ballista 0.5.0 Release</title><link href="https://datafusion.apache.org/blog/2021/08/18/ballista-0.5.0" rel="alternate"/><published>2021-08-18T00:00:00+00:00</published><updated>2021-08-18T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/ballista-0.5.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14220,7 +14516,7 @@ scalability as well as improving the documentation.</p>
can help by trying out Ballista on some of your own data and projects and filing bug reports and helping to
improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 5.0.0 Release</title><link href="https://datafusion.apache.org/blog/2021/08/18/datafusion-5.0.0" rel="alternate"></link><published>2021-08-18T00:00:00+00:00</published><updated>2021-08-18T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-08-18:/blog [...]
+and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 5.0.0 Release</title><link href="https://datafusion.apache.org/blog/2021/08/18/datafusion-5.0.0" rel="alternate"/><published>2021-08-18T00:00:00+00:00</published><updated>2021-08-18T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/data [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14337,7 +14633,7 @@ performance of individual TPC-H queries compared to the previous release.</p&
can help by trying out DataFusion on some of your own data and projects and filing bug reports and helping to
improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p></content><category term="blog"></category></entry><entry><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2021/04/12/ballista-donation" rel="alternate"></link><published>2021-04-12T00:00:00+00:00</published><updated>2021-04-12T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org [...]
+and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p></content><category term="blog"/></entry><entry><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2021/04/12/ballista-donation" rel="alternate"/><published>2021-04-12T00:00:00+00:00</published><updated>2021-04-12T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org,2021-04-12:/blo [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14409,7 +14705,7 @@ datasets and file bug reports for any issues that you find. You could also check
<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20component%20%3D%20%22Rust%20-%20Ballista%22">list of issues</a> and have a go at fixing one.</p>
<p>The <a href="https://github.com/apache/arrow/blob/master/rust/README.md#arrow-rust-community">Arrow Rust Community</a>
section of the Rust README provides information on other ways to interact with the Ballista contributors and
-maintainers.</p></content><category term="blog"></category></entry><entry><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="alternate"></link><published>2019-02-04T00:00:00+00:00</published><updated>2019-02-04T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</id><summary type="html"><!--
+maintainers.</p></content><category term="blog"/></entry><entry><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="alternate"/><published>2019-02-04T00:00:00+00:00</published><updated>2019-02-04T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14528,4 +14824,4 @@ while let Some(batch) = results.next().unwrap() {
<li>Creating a Docker image to make it easy to use DataFusion as a standalone query tool for interactive and batch queries</li>
</ul>
<h2 id="contributors-welcome">Contributors Welcome!<a class="headerlink" href="#contributors-welcome" title="Permanent link">¶</a></h2>
-<p>If you are excited about being able to use Rust for data science and would like to contribute to this work then there are many ways to get involved. The simplest way to get started is to try out DataFusion against your own data sources and file bug reports for any issues that you find. You could also check out the current <a href="https://cwiki.apache.org/confluence/display/ARROW/Rust+JIRA+Dashboard">list of issues</a> and have a go at fixing one. You can also join t [...]
\ No newline at end of file
+<p>If you are excited about being able to use Rust for data science and would like to contribute to this work then there are many ways to get involved. The simplest way to get started is to try out DataFusion against your own data sources and file bug reports for any issues that you find. You could also check out the current <a href="https://cwiki.apache.org/confluence/display/ARROW/Rust+JIRA+Dashboard">list of issues</a> and have a go at fixing one. You can also join t [...]
\ No newline at end of file
diff --git a/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.atom.xml b/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.atom.xml
index 2095faf..f1d5e39 100644
--- a/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.atom.xml
+++ b/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Andrew Lamb, Achraf B, and Sean Smith</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-04-10T00:00:00+00:00</updated><subtitle></subtitle><entry><title>tpchgen-rs World’s fastest open source TPC-H data generator, [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Andrew Lamb, Achraf B, and Sean Smith</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2025-04-10T00:00:00+00:00</updated><entry><title>tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust</title><link href [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -16,19 +16,11 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
-development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over …</p></summary><content type="html"><!--
+development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
+faster than any other implementation we know of.</p>
+<p>It is now possible to create …</p></summary><content type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -45,16 +37,6 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
@@ -469,4 +451,4 @@ DataFusion (see <a href="https://github.com/apache/datafusion/issues/14608"&g
<p><em>2</em>: It is possible to embed the dbgen code, which appears to be the approach taken by DuckDB. This approach was tried in GlareDB (<a href="https://github.com/GlareDB/glaredb/pull/3313">GlareDB/glaredb#3313</a>), but ultimately shelved given the amount of effort needed to adapt and isolate the dbgen code.</p>
<p><em>3</em>: It is pretty amazing to imagine the machine required to generate SF300 that had 1.8TB (!!) of RAM</p>
<p><em>4</em>: We tried to <a href="https://github.com/clflushopt/tpchgen-rs/pull/34">use Rayon (see discussion here)</a>, but could not easily keep memory bounded.</p>
-<p><em>5</em>: <code>tpchgen-cli</code> memory usage is a function of the number of threads: each thread needs some buffer space</p></content><category term="blog"></category></entry></feed>
\ No newline at end of file
+<p><em>5</em>: <code>tpchgen-cli</code> memory usage is a function of the number of threads: each thread needs some buffer space</p></content><category term="blog"/></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.rss.xml b/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.rss.xml
index b32fe7a..f3e94b5 100644
--- a/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.rss.xml
+++ b/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - Andrew Lamb, Achraf B, and Sean Smith</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Thu, 10 Apr 2025 00:00:00 +0000</lastBuildDate><item><title>tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust</title><link>https://datafusion.apache.org/blog/2025/04/10/fastest-tpch-generator</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - Andrew Lamb, Achraf B, and Sean Smith</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/andrew-lamb-achraf-b-and-sean-smith.rss.xml" rel="self"/><lastBuildDate>Thu, 10 Apr 2025 00:00:00 +0000</lastBuildDate><item><title>tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust</title><link>http [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -16,16 +16,8 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
-development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb, Achraf B, and Sean Smith</dc:creator><pubDate>Thu, 10 Apr 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-04-10:/blog/2025/04/10/fastest-tpch-generator</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
+faster than any other implementation we know of.</p>
+<p>It is now possible to create …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb, Achraf B, and Sean Smith</dc:creator><pubDate>Thu, 10 Apr 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-04-10:/blog/2025/04/10/fastest-tpch-generator</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/andrew-lamb-influxdata.atom.xml b/blog/feeds/andrew-lamb-influxdata.atom.xml
index c2234c6..c7c2f2d 100644
--- a/blog/feeds/andrew-lamb-influxdata.atom.xml
+++ b/blog/feeds/andrew-lamb-influxdata.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Andrew Lamb (InfluxData)</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/andrew-lamb-influxdata.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-08-15T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on A [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Andrew Lamb (InfluxData)</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/andrew-lamb-influxdata.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2025-08-15T00:00:00+00:00</updated><entry><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</title><link href=" [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -589,4 +589,4 @@ such as point lookups, filters of different selectivity, and aggregations.</p
<p><a id="footnote7"></a><code>7</code>: For example, <a href="https://github.com/zhuqi-lucas">Qi Zhu</a> was able to speed up reads by over 2x
simply by rewriting the Parquet files with Offset Indexes and no compression (see <a href="https://github.com/apache/datafusion/issues/16149#issuecomment-2918761743">issue #16149 comment</a> for details).
There is likely significant additional performance available by using Bloom Filters and resorting the data
-to be clustered in a more optimal way for the queries.</p></content><category term="blog"></category></entry></feed>
\ No newline at end of file
+to be clustered in a more optimal way for the queries.</p></content><category term="blog"/></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/andrew-lamb-influxdata.rss.xml b/blog/feeds/andrew-lamb-influxdata.rss.xml
index 6f2ebea..86dd449 100644
--- a/blog/feeds/andrew-lamb-influxdata.rss.xml
+++ b/blog/feeds/andrew-lamb-influxdata.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - Andrew Lamb (InfluxData)</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Fri, 15 Aug 2025 00:00:00 +0000</lastBuildDate><item><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</title><link>https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - Andrew Lamb (InfluxData)</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/andrew-lamb-influxdata.rss.xml" rel="self"/><lastBuildDate>Fri, 15 Aug 2025 00:00:00 +0000</lastBuildDate><item><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</title><link>https: [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -24,4 +24,4 @@ metadata and is limited to indexing structures provided by the format. In fact,
caching parsed metadata and using custom external indexes along with
Parquet's hierarchical data organization can significantly speed up query
processing.</p>
-<p>In this blog, I describe …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb (InfluxData)</dc:creator><pubDate>Fri, 15 Aug 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-08-15:/blog/2025/08/15/external-parquet-indexes</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+<p>In this blog, I describe …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb (InfluxData)</dc:creator><pubDate>Fri, 15 Aug 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-08-15:/blog/2025/08/15/external-parquet-indexes</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.atom.xml b/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.atom.xml
index c381337..fb64d08 100644
--- a/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.atom.xml
+++ b/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Andrew Lamb, Staff Engineer at InfluxData</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2024-11-18T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion is now the fastest single node eng [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Andrew Lamb, Staff Engineer at InfluxData</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2024-11-18T00:00:00+00:00</updated><entry><title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet f [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -225,4 +225,4 @@ online</a>.</p>
<p>[^2]: Want to try your hand at a custom format for ClickBench fame / glory?: <a href="https://github.com/apache/datafusion/issues/13448">Make DataFusion the fastest engine in ClickBench with custom file format</a></p>
<p>[^3]: We have contributors from North America, South American, Europe, Asia, Africa and Australia</p>
<p>[^4]: Undergraduates, PhD, Junior engineers, and getting-kind-of-crotchety experienced engineers</p>
-<p>[^5]: Thanks to Andy Pavlo, I love that nomenclature</p></content><category term="blog"></category></entry></feed>
\ No newline at end of file
+<p>[^5]: Thanks to Andy Pavlo, I love that nomenclature</p></content><category term="blog"/></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.rss.xml b/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.rss.xml
index 55ec12e..732b9dd 100644
--- a/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.rss.xml
+++ b/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - Andrew Lamb, Staff Engineer at InfluxData</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon, 18 Nov 2024 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</title><link>https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - Andrew Lamb, Staff Engineer at InfluxData</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/andrew-lamb-staff-engineer-at-influxdata.rss.xml" rel="self"/><lastBuildDate>Mon, 18 Nov 2024 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet f [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -22,4 +22,4 @@ limitations under the License.
fastest engine for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">ClickBench</a>. It is faster
than <a href="https://duckdb.org/">DuckDB</a>, <a href="https://clickhouse.com/chdb">chDB</a> and <a href="https://clickhouse.com/">Clickhouse</a> using the same hardware. It also marks
the first time a <a href="https://www.rust-lang.org/">Rust</a>-based engine holds the top spot, which has previously
-been …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb, Staff Engineer at InfluxData</dc:creator><pubDate>Mon, 18 Nov 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2024-11-18:/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+been …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb, Staff Engineer at InfluxData</dc:creator><pubDate>Mon, 18 Nov 2024 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2024-11-18:/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/blog.atom.xml b/blog/feeds/blog.atom.xml
index 1c681e8..da3a1b9 100644
--- a/blog/feeds/blog.atom.xml
+++ b/blog/feeds/blog.atom.xml
@@ -1,5 +1,319 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/blog.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-03-31T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Writing Custom Table Providers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2026/03/31/writing- [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/blog.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-04-18T00:00:00+00:00</updated><entry><title>Apache DataFusion Comet 0.15.0 Release</title><link href="https://datafusion.apache.org/blog/2026/04/18/datafusion-comet-0.15.0" rel="alternate"/><published>2 [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.15.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately four weeks of development …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.15.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately four weeks of development work and is the result of merging 142 PRs from 19
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.15.0.md">change log</a> for more information.</p>
+<h2 id="key-features">Key Features<a class="headerlink" href="#key-features" title="Permanent link">¶</a></h2>
+<h3 id="native-iceberg-reader-enabled-by-default">Native Iceberg Reader Enabled by Default<a class="headerlink" href="#native-iceberg-reader-enabled-by-default" title="Permanent link">¶</a></h3>
+<p>This release marks a major milestone for Iceberg users: <strong>Comet's fully-native Iceberg reader is now enabled by
+default</strong>. Workloads that read Iceberg tables will automatically benefit from native Rust-based scans built on
+iceberg-rust, with no additional configuration required.</p>
+<p>To support this change, the release bundles a broad set of Iceberg-focused improvements:</p>
+<ul>
+<li><strong>Correct classloader handling</strong>: Iceberg classes are now loaded via the thread context classloader, resolving
+ class-loading issues in environments where the executor classloader differs from the application classloader.</li>
+<li><strong>Continuous Iceberg CI</strong>: Iceberg Spark integration tests now run on every PR and push to <code>main</code>, providing
+ continuous validation of the native Iceberg code path. Test diffs for Spark 3.4 were updated to keep the matrix
+ green across supported Spark versions.</li>
+<li><strong>iceberg-rust upgrade</strong>: Comet picks up the latest iceberg-rust, pulling in fixes for Parquet reader edge cases
+ discovered in earlier testing.</li>
+<li><strong>Refreshed documentation</strong>: The Iceberg user guide has been rewritten to reflect current capabilities, and the
+ contributor guide now documents how to run the Iceberg Spark test suites locally.</li>
+</ul>
+<p>Users who need to fall back to the previous behavior can still opt out, but we encourage the community to exercise
+the native reader and report any issues.</p>
+<h3 id="auto-mode-enables-native_datafusion-scan">Auto Mode Enables <code>native_datafusion</code> Scan<a class="headerlink" href="#auto-mode-enables-native_datafusion-scan" title="Permanent link">¶</a></h3>
+<p>The <code>auto</code> scan mode now enables the <code>native_datafusion</code> scan where supported, giving users the benefits of the
+native Parquet reader without having to explicitly opt in. This is part of the ongoing effort to make
+<code>native_datafusion</code> the default Parquet path once the deprecation of <code>native_iceberg_compat</code> completes.</p>
+<h3 id="new-expressions-and-function-support">New Expressions and Function Support<a class="headerlink" href="#new-expressions-and-function-support" title="Permanent link">¶</a></h3>
+<p>This release adds support for the following:</p>
+<ul>
+<li><strong>Date/time functions</strong>: <code>days</code>, <code>hours</code>, <code>date_from_unix_date</code></li>
+<li><strong>String/JSON functions</strong>: native <code>get_json_object</code> with improved performance over the fallback path</li>
+<li><strong>Hash/math functions</strong>: <code>crc32c</code>, <code>bin</code></li>
+<li><strong>Array functions</strong>: <code>sort_array</code></li>
+<li><strong>Window functions</strong>: <code>LEAD</code> and <code>LAG</code> with <code>IGNORE NULLS</code></li>
+<li><strong>Aggregates</strong>: SQL <code>FILTER (WHERE ...)</code> clauses now execute natively; <code>Corr</code> aggregate enabled</li>
+</ul>
+<h3 id="expanded-metrics-and-observability">Expanded Metrics and Observability<a class="headerlink" href="#expanded-metrics-and-observability" title="Permanent link">¶</a></h3>
+<p>Comet metrics can now be exposed through Spark's external monitoring system, making it easier to integrate Comet
+execution statistics with existing observability dashboards. Native DataFusion scans also now report accurate
+<code>filesScanned</code> and <code>bytesScanned</code> input metrics, matching Spark's native Parquet scan reporting.</p>
+<h2 id="performance-improvements">Performance Improvements<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p>Performance was a major theme of this release, with a series of targeted optimizations across the shuffle, scan,
+and execution layers:</p>
+<ul>
+<li><strong>Shuffle read path</strong>: The native shuffle reader no longer uses FFI on the read side, removing a per-batch cost
+ that was particularly visible in shuffle-heavy queries.</li>
+<li><strong>Broadcast exchanges</strong>: Batches are now coalesced before broadcasting, reducing the number of small batches
+ crossing the JVM/native boundary.</li>
+<li><strong>Columnar-to-row (C2R)</strong>: Native C2R conversion is now exercised for a broader set of query shapes.</li>
+<li><strong>FFI-safe operators</strong>: More operators are marked as FFI-safe, avoiding unnecessary deep copies when crossing
+ the JVM/native boundary.</li>
+<li><strong>Shared memory pools</strong>: Unified memory pools are now shared across native execution contexts within a Spark
+ task, improving memory accounting and reducing fragmentation.</li>
+<li><strong>Object store caching</strong>: Object stores and bucket region lookups are cached, dramatically reducing DNS query
+ volume on workloads that open many files.</li>
+<li><strong><code>get_ranges</code> performance</strong>: Picked up an upstream <code>opendal</code> fix that restores fast range reads from object
+ storage.</li>
+<li><strong>Spill I/O</strong>: Removed a redundant <code>BufReader</code> wrapper when copying spill files to shuffle output.</li>
+</ul>
+<p>Together, these changes reduce CPU and memory overhead for shuffle-heavy, broadcast-heavy, and
+object-storage-bound workloads.</p>
+<h2 id="stability-and-correctness">Stability and Correctness<a class="headerlink" href="#stability-and-correctness" title="Permanent link">¶</a></h2>
+<p>A significant portion of this release is dedicated to stability and Spark compatibility. Highlights include:</p>
+<ul>
+<li><strong>Cast string to timestamp</strong>: Multiple fixes for UTC timestamps, timezone handling, special formats
+ (<code>epoch</code>, <code>now</code>, etc.), and compatibility with Spark's semantics.</li>
+<li><strong>Cast decimal to string</strong>: Added legacy mode handling to match Spark's output formatting.</li>
+<li><strong>String to decimal</strong>: Support for full-width characters, null characters, and negative scale.</li>
+<li><strong>Decimal arithmetic</strong>: Fixes for decimal division and additional test coverage for ANSI overflow handling,
+ including scalar decimal overflow.</li>
+<li><strong>Array expressions</strong>: Corrected <code>GetArrayItem</code> null handling for dynamic indices; <code>array_append</code> return type
+ fixed and marked <code>Compatible</code>; audited <code>array_insert</code> for correctness; <code>array_compact</code> marked <code>Compatible</code>;
+ array-to-array cast enabled.</li>
+<li><strong>DateTrunc/TimestampTrunc</strong>: Fixed native crashes when the input is a literal.</li>
+<li><strong>Ambiguous local times</strong>: Correct handling of ambiguous and non-existent local times across DST transitions.</li>
+<li><strong>Case-insensitive Parquet fields</strong>: <code>native_datafusion</code> now correctly detects duplicate/ambiguous fields in
+ case-insensitive mode and falls back where appropriate.</li>
+<li><strong>Shuffle planning</strong>: Shuffle fallback decisions are now "sticky" across planning passes, and Comet columnar
+ shuffle is skipped for stages containing DPP scans to avoid mismatched partitioning.</li>
+<li><strong>Error propagation</strong>: Native error messages are now propagated through <code>SparkException</code> even when the
+ <code>errorClass</code> is empty, and file-not-found errors flow through the standard Spark error JSON path.</li>
+<li><strong>Trigonometric compatibility</strong>: <code>tan</code> and <code>atan2</code> are now Spark-compatible.</li>
+</ul>
+<h2 id="dependency-upgrades">Dependency Upgrades<a class="headerlink" href="#dependency-upgrades" title="Permanent link">¶</a></h2>
+<p>This release upgrades to <strong>DataFusion 53.1</strong> and <strong>Arrow 58.1</strong>, and picks up the latest <code>iceberg-rust</code> release
+with additional reader fixes. The <code>jni</code> crate was upgraded to 0.22.4.</p>
+<h2 id="deprecations-and-removals">Deprecations and Removals<a class="headerlink" href="#deprecations-and-removals" title="Permanent link">¶</a></h2>
+<p>The <code>SupportsComet</code> interface has been removed, along with the Java-based Iceberg integration path (which is
+fully superseded by the native Iceberg reader). The <code>native_iceberg_compat</code> scan remains deprecated and is
+expected to be removed in a future release in favor of <code>native_datafusion</code>.</p>
+<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
+<p>Supported platforms include Spark 3.4.3, 3.5.4–3.5.8, and Spark 4.0.x with various JDK and Scala combinations.</p>
+<p>The community encourages users to test Comet with existing Spark and Iceberg workloads and welcomes contributions
+to ongoing development.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 53.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/04/02/datafusion-53.0.0" rel="alternate"/><published>2026-04-02T00:00:00+00:00</published><updated>2026-04-02T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2026-04-02:/blog/2026/04/02/datafusion-53.0.0</id><summary type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/53.0.0">DataFusion 53.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0/">DataFusion 52.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md#credits">114 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p><img alt="Performance over time" class="img-fluid" src="/blog/images/datafusion-53.0.0/performance_over_time_clickbench.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: Average …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are proud to announce the release of <a href="https://crates.io/crates/datafusion/53.0.0">DataFusion 53.0.0</a>. This post highlights
+some of the major improvements since <a href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0/">DataFusion 52.0.0</a>. The complete list of
+changes is available in the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a>. Thanks to the <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md#credits">114 contributors</a> for
+making this release possible.</p>
+<h2 id="performance-improvements">Performance Improvements 🚀<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h2>
+<p><img alt="Performance over time" class="img-fluid" src="/blog/images/datafusion-53.0.0/performance_over_time_clickbench.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: Average and median normalized execution times for DataFusion 53.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+<a href="https://alamb.github.io/datafusion-benchmarking/">DataFusion Benchmarking Page</a>
+for more details.</p>
+<p>DataFusion 53 continues the project-wide focus on performance. This release
+reduces planning overhead, skips more unnecessary I/O, and pushes more work
+into earlier and cheaper stages of execution.</p>
+<h3 id="limit-aware-parquet-row-group-pruning"><code>LIMIT</code>-Aware Parquet Row Group Pruning<a class="headerlink" href="#limit-aware-parquet-row-group-pruning" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 includes a new optimization that makes Parquet pruning aware of
+<code>LIMIT</code>. This optimization is described in full in <a href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning/">limit pruning blog post</a>. If
+DataFusion can prove that an entire row group matches the predicate, and those
+fully matching row groups contain enough rows to satisfy the <code>LIMIT</code>, partially
+matching row groups are skipped entirely.</p>
+<figure>
+<img alt="Pruning pipeline with limit pruning highlighted" class="img-fluid" src="/blog/images/limit-pruning/pruning-pipeline.svg" width="80%"/>
+<figcaption><b>Figure 2</b>: Limit pruning is inserted between row group and page index pruning.</figcaption>
+</figure>
+<p>Thanks to <a href="https://github.com/xudong963">@xudong963</a> for implementing this feature. Related PRs: <a href="https://github.com/apache/datafusion/pull/18868">#18868</a></p>
+<h3 id="improved-filter-pushdown">Improved Filter Pushdown<a class="headerlink" href="#improved-filter-pushdown" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 pushes filters down through more join types and through <code>UnionExec</code>,
+and expands support for pushing down <a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters">dynamic filters</a>. More
+pushdown means fewer rows flow into joins, repartitions, and later operators,
+which reduces CPU, memory, and I/O.</p>
+<p>For example:</p>
+<pre><code class="language-sql">SELECT *
+FROM (
+ SELECT *
+ FROM t1
+ LEFT ANTI JOIN t2 ON t1.k = t2.k
+) a
+JOIN t1 b ON a.k = b.k
+WHERE b.v = 1;
+</code></pre>
+<p>Now DataFusion can often transform the physical plan so filters and
+<a href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters">dynamic filters</a> are pushed deeper into the plan, even through subqueries and
+nested joins. In this example, the filter on <code>b.v</code> helps produce dynamic filters
+that can be pushed into both sides of the nested anti join.</p>
+<figure>
+<img alt="Before and after diagram of dynamic filter pushdown through a subquery with nested joins" class="img-fluid" src="/blog/images/datafusion-53.0.0/join-filter-pushdown.svg" width="100%"/>
+<figcaption><b>Figure 3</b>: DataFusion 53 pushes dynamic filters through subqueries and into both sides of nested joins.</figcaption>
+</figure>
+<p>Thanks to <a href="https://github.com/nuno-faria">@nuno-faria</a>, <a href="https://github.com/haohuaijin">@haohuaijin</a>, and <a href="https://github.com/jackkleeman">@jackkleeman</a> for
+driving this work. Related PRs: <a href="https://github.com/apache/datafusion/pull/19918">#19918</a>, <a href="https://github.com/apache/datafusion/pull/20145">#20145</a>, <a href="https://github.com/apache/datafusion/pull/20192">#20192</a></p>
+<h3 id="faster-query-planning">Faster Query Planning<a class="headerlink" href="#faster-query-planning" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 improves query planning performance by making immutable pieces of
+execution plans cheaper to clone. This helps applications that need extremely
+low latency, plan many or complex queries, or use prepared statements or
+parameterized queries. In some benchmarks, overall execution time drops from
+roughly 4-5 ms to about 100 us.</p>
+<p>Thanks to <a href="https://github.com/askalt">@askalt</a> for leading this work. Related PRs: <a href="https://github.com/apache/datafusion/pull/19792">#19792</a>, <a href="https://github.com/apache/datafusion/pull/19893">#19893</a></p>
+<h3 id="faster-functions">Faster Functions<a class="headerlink" href="#faster-functions" title="Permanent link">¶</a></h3>
+<p>DataFusion includes <a href="https://datafusion.apache.org/user-guide/sql/scalar_functions.html">235 built-in functions</a>. Improving the performance of these
+functions benefits a wide range of workloads. This release improves the performance of 42 of those
+functions, such as <a href="https://github.com/apache/datafusion/pull/20295">strpos</a>, <a href="https://github.com/apache/datafusion/pull/20344">replace</a>, <a href="https://github.com/apache/datafusion/pull/20317">concat</a>, <a href="https://github.com/apache/datafusion/pull/20305">translate</a>, <a href="https://github.com/apache/datafusion/pull/20374">array_has</a>,
+<a href="https://github.com/apache/datafusion/pull/20504">array_agg</a>, <a href="https://github.com/apache/datafusion/pull/19980">left</a>, <a href="https://github.com/apache/datafusion/pull/20069">right</a>, and <a href="https://github.com/apache/datafusion/pull/20097">case_when</a>.</p>
+<p>Thanks to the contributors who drove this work, especially <a href="https://github.com/neilconway">@neilconway</a>,
+<a href="https://github.com/theirix">@theirix</a>, <a href="https://github.com/lyne7-sc">@lyne7-sc</a>, <a href="https://github.com/kumarUjjawal">@kumarUjjawal</a>, <a href="https://github.com/pepijnve">@pepijnve</a>, <a href="https://github.com/zhangxffff">@zhangxffff</a>, and
+<a href="https://github.com/UBarney">@UBarney</a>.</p>
+<h3 id="nested-field-pushdown">Nested Field Pushdown<a class="headerlink" href="#nested-field-pushdown" title="Permanent link">¶</a></h3>
+<p>DataFusion 53 pushes expressions such as <code>get_field</code> down the plan and into data
+sources. This is especially important for nested data such as structs in
+Parquet files. Instead of reading an entire struct column and then extracting
+the field of interest, DataFusion 53 pushes the field extraction into the scan.</p>
+<p>For example, the following query reads a struct column <code>s</code> and extracts the
+<code>label</code> field for rows where the <code>value</code> field is greater than 150:</p>
+<pre><code class="language-sql">SELECT id, s['label']
+FROM t
+WHERE s['value'] &gt; 150;
+</code></pre>
+<figure>
+<img alt="Before and after diagram of field access pushdown into a data source" class="img-fluid" src="/blog/images/datafusion-53.0.0/field-access-pushdown.svg" width="80%"/>
+<figcaption><b>Figure 4</b>: DataFusion 53 pushes field-access expressions closer to the scan.</figcaption>
+</figure>
+<p>Special thanks to <a href="https://github.com/adriangb">@adriangb</a> for designing and implementing this optimizer
+work. Related PRs: <a href="https://github.com/apache/datafusion/pull/20065">#20065</a>, <a href="https://github.com/apache/datafusion/pull/20117">#20117</a>, <a href="https://github.com/apache/datafusion/pull/20239">#20239</a></p>
+<h2 id="new-features">New Features ✨<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h2>
+<ul>
+<li>
+<p><strong>JSON Array File Support</strong>: DataFusion 53 can now read JSON arrays such as
+ <code>[{...}, {...}]</code> directly as multiple rows, including streaming inputs from
+ object stores. Thanks to <a href="https://github.com/zhuqi-lucas">@zhuqi-lucas</a> for implementing this feature.
+ Related PRs: <a href="https://github.com/apache/datafusion/pull/19924">#19924</a></p>
+</li>
+<li>
+<p><strong>Support for <code>:</code> operator</strong>: DataFusion can plan queries such as
+ <code>SELECT payload:'user_id' FROM events;</code>, enabling better <a href="https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/">Parquet Variant</a>
+ support via <a href="https://github.com/datafusion-contrib/datafusion-variant">datafusion-variant</a>. Thanks to <a href="https://github.com/Samyak2">@Samyak2</a>. Related PRs:
+ <a href="https://github.com/apache/datafusion/pull/20717">#20717</a></p>
+</li>
+<li>
+<p><strong>New SQL</strong>: DataFusion supports additional set-comparison subqueries, null-aware
+ anti join, and deletion predicates. Thanks to <a href="https://github.com/waynexia">@waynexia</a>, <a href="https://github.com/viirya">@viirya</a>, and
+ <a href="https://github.com/askalt">@askalt</a> for key contributions in this area. Related PRs: <a href="https://github.com/apache/datafusion/pull/19109">#19109</a>,
+ <a href="https://github.com/apache/datafusion/pull/19635">#19635</a>, <a href="https://github.com/apache/datafusion/pull/20137">#20137</a></p>
+</li>
+<li>
+<p><strong>Spark-Compatible Functions</strong>: This release includes almost 20 new or improved
+ Spark-compatible functions and behaviors in the <a href="https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html">datafusion-spark crate</a>.
+ It includes functions such as <a href="https://github.com/apache/datafusion/pull/19699">collect_list</a>, <a href="https://github.com/apache/datafusion/pull/19845">date_diff</a>,
+ <a href="https://github.com/apache/datafusion/pull/19880">from_utc_timestamp</a>, <a href="https://github.com/apache/datafusion/pull/20412">json_tuple</a>, <a href="https://github.com/apache/datafusion/pull/20440">arrays_zip</a>, <a href="https://github.com/apache/datafusion/pull/20479">bin</a>, and <a href="https://github.com/apache/datafusion/pull/20685">array_contains</a>.
+ Thanks to the contributors who drove this work, especially <a href="https://github.com/cht42">@cht42</a>,
+ <a href="https://github.com/CuteChuanChuan">@CuteChuanChuan</a>, <a href="https://github.com/SubhamSinghal">@SubhamSinghal</a>, <a href="https://github.com/kazantsev-maksim">@kazantsev-maksim</a>, <a href="https://github.com/unknowntpo">@unknowntpo</a>,
+ <a href="https://github.com/aryan-212">@aryan-212</a>, <a href="https://github.com/hsiang-c">@hsiang-c</a>, and <a href="https://github.com/davidlghellin">@davidlghellin</a>.</p>
+</li>
+</ul>
+<h2 id="stability-and-release-engineering">Stability and Release Engineering 🦺<a class="headerlink" href="#stability-and-release-engineering" title="Permanent link">¶</a></h2>
+<p>The community spent significant time this release cycle stabilizing the release
+branch and improving the release process. While such improvements are not as
+headline-friendly as new features, they are highly important for real
+deployments. We are discussing ways to improve the process on <a href="https://github.com/apache/datafusion/issues/21034">#21034</a> and would
+welcome suggestions and contributions to help with release engineering work in
+the future.</p>
+<p>Thanks to <a href="https://github.com/comphead">@comphead</a> for running this release, and to <a href="https://github.com/jonathanc-n">@jonathanc-n</a>, <a href="https://github.com/alamb">@alamb</a>,
+<a href="https://github.com/xanderbailey">@xanderbailey</a>, <a href="https://github.com/haohuaijin">@haohuaijin</a>, <a href="https://github.com/friendlymatthew">@friendlymatthew</a>, <a href="https://github.com/fwojciec">@fwojciec</a>,
+<a href="https://github.com/Kontinuation">@Kontinuation</a>, <a href="https://github.com/nathanb9">@nathanb9</a>, and many others who helped stabilize the release
+branch.</p>
+<h2 id="upgrade-notes">Upgrade Notes<a class="headerlink" href="#upgrade-notes" title="Permanent link">¶</a></h2>
+<p>DataFusion 53 includes some breaking changes, including updates to the SQL
+parser, optimizer behavior, and some physical-plan APIs. Please see the <a href="https://datafusion.apache.org/library-user-guide/upgrading/index.html">upgrade
+guide</a> and <a href="https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md">changelog</a> for the full details before upgrading.</p>
+<h2 id="known-issues">Known Issues<a class="headerlink" href="#known-issues" title="Permanent link">¶</a></h2>
+<p>A small number of issues were discovered after the 53.0.0 release,
+and we expect to publish DataFusion 53.1.0 soon. See the <a href="https://github.com/apache/datafusion/issues/21079">53.1.0 release tracking
+issue</a> for the latest status.</p>
+<h2 id="thank-you">Thank You<a class="headerlink" href="#thank-you" title="Permanent link">¶</a></h2>
+<p>Thank you to everyone in the DataFusion community who contributed code, reviews,
+testing, bug reports, documentation, and release engineering work for 53.0.0.
+This release contains direct contributions from 114 different people, and we are
+grateful for the time and effort that everyone put in to make it happen.</p></content><category term="blog"/></entry><entry><title>Writing Custom Table Providers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2026/03/31/writing-table-providers" rel="alternate"/><published>2026-03-31T00:00:00+00:00</published><updated>2026-03-31T00:00:00+00:00</updated><author><name>Tim Saucer (rerun.io)</name></author><id>tag:datafusion.apache.org,2026-03-31:/blog/2026/0 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -826,7 +1140,7 @@ can find out how to reach us on the <a href="https://datafusion.apache.org/co
contains working examples including custom table providers</li>
</ul>
<hr/>
-<p><em>Note: Portions of this blog post were written with the assistance of an AI agent.</em></p></content><category term="blog"></category></entry><entry><title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</title><link href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="alternate"></link><published>2026-03-20T00:00:00+00:00</published><updated>2026-03-20T00:00:00+00:00</updated><author><name>xudong</nam [...]
+<p><em>Note: Portions of this blog post were written with the assistance of an AI agent.</em></p></content><category term="blog"/></entry><entry><title>Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack</title><link href="https://datafusion.apache.org/blog/2026/03/20/limit-pruning" rel="alternate"/><published>2026-03-20T00:00:00+00:00</published><updated>2026-03-20T00:00:00+00:00</updated><author><name>xudong</name></author><id>t [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1137,7 +1451,7 @@ limit_pruned_row_groups=3 total → 1 matched
<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries, and machine learning and streaming applications.</p>
<p>DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone.</p>
<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
-<p>If you are interested in contributing, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a [...]
+<p>If you are interested in contributing, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1238,7 +1552,7 @@ that the <code>native_iceberg_compat</code> scan is now deprecated a
<p>This release upgrades to DataFusion 52.3, Arrow 57.3, and iceberg-rust 0.9.0. Published binaries now target
x86-64-v3 and neoverse-n1 CPU architectures for improved performance on modern hardware.</p>
<p>Supported platforms include Spark 3.4.3, 3.5.4-3.5.8, and Spark 4.0.x with various JDK and Scala combinations.</p>
-<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"></category></entry><entry><title>Optimizing SQL CASE Expression Evaluation</title><link href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="alternate"></link><published>2026-02-02T00:00:00+00:00</published><updated>2026-02-02T00:00:00+00:00</updated><author><name>Pepijn Van Eeckhoudt</name></a [...]
+<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"/></entry><entry><title>Optimizing SQL CASE Expression Evaluation</title><link href="https://datafusion.apache.org/blog/2026/02/02/datafusion_case" rel="alternate"/><published>2026-02-02T00:00:00+00:00</published><updated>2026-02-02T00:00:00+00:00</updated><author><name>Pepijn Van Eeckhoudt</name></author><id>tag:da [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1633,7 +1947,7 @@ With DataFusion, we can all build on top of a shared foundation and focus on wha
<p>DataFusion is not a project built or driven by a single person, company, or foundation.
Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.</p>
<p>If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code.
-A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.13.0 Release</title><link href="https://datafusion.a [...]
+A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.13.0 Release</title><link href="https://datafusion.apache.org/ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1749,7 +2063,7 @@ spark.comet.parquet.write.enabled=true
<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
<p>This release upgrades to DataFusion 51, Arrow 57, and the latest iceberg-rust. The minimum supported Rust version is now 1.88.</p>
<p>Supported platforms include Spark 3.4.3, 3.5.4-3.5.7, and Spark 4.0.x with various JDK and Scala combinations.</p>
-<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="alternate"></link><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusio [...]
+<p>The community encourages users to test Comet with existing Spark workloads and welcomes contributions to ongoing development.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 52.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0" rel="alternate"/><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,202 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -1988,7 +2302,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="alternate"></link><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>Ge [...]
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https://datafusion.apache.org/blog/2026/01/12/extending-sql" rel="alternate"/><published>2026-01-12T00:00:00+00:00</published><updated>2026-01-12T00:00:00+00:00</updated><author><name>Geoffrey Claude (D [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2272,7 +2586,7 @@ println!("{}", df.logical_plan().display_indent());
<li><strong>Try it out</strong>: Implement one of the extension points and share your experience</li>
<li><strong>File issues or join the conversation</strong>: <a href="https://github.com/apache/datafusion/">GitHub</a> for bugs and feature requests, <a href="https://datafusion.apache.org/contributor-guide/communication.html">Slack or Discord</a> for discussion</li>
</ul>
-<!-- Reference links --></content><category term="blog"></category></entry><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="alternate"></link><published>2025-12-15T00:00:00+00:00</published><updated>2025-12-15T00:00:00+00:00</updated><author><name>Gene Bordegaray</name></author><id>tag:datafusion.apache.org,2025-12-15:/blog/202 [...]
+<!-- Reference links --></content><category term="blog"/></entry><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions" rel="alternate"/><published>2025-12-15T00:00:00+00:00</published><updated>2025-12-15T00:00:00+00:00</updated><author><name>Gene Bordegaray</name></author><id>tag:datafusion.apache.org,2025-12-15:/blog/2025/12/15/avoid-co [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2290,8 +2604,8 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div></summary><content type="html"><!--
{% comment %}
@@ -2311,16 +2625,16 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is exposed by a user-friendly interface and query languages (typically a dialect of SQL).
<br/><br/>
Starting a journey learning about database internals can be daunting. With so many topics that are whole PhD degrees themselves, finding a place to start is difficult. In this blog post, I will share my early journey in the database world and a quick lesson on one of the first topics I dove into. If you are new to the space, this post will help you get your first foot into the database world, and if you are already a veteran, you may still learn something new.
</div>
-<div style="flex: 0 0 40%; text-align: center;">
-<img alt="Database System Components" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/database_system_diagram.png" width="100%"/>
+<div class="col-md-5 text-center">
+<img alt="Database System Components" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/database_system_diagram.png"/>
</div>
</div>
<hr/>
@@ -2361,29 +2675,29 @@ DataFusion achieves multi-core parallelism through the use of "exchange operator
<h3 id="what-is-repartitioning">What is Repartitioning?<a class="headerlink" href="#what-is-repartitioning" title="Permanent link">&para;</a></h3>
<p>Partitioning is a "divide-and-conquer" approach to executing a query. Each partition is a subset of the data that is being processed on a single core. Repartitioning is an operation that redistributes data across different partitions to balance workloads, reduce data skew, and increase parallelism. Two repartitioning methods are used in DataFusion: round-robin and hash.</p>
<h4 id="round-robin-repartitioning"><strong>Round-Robin Repartitioning</strong><a class="headerlink" href="#round-robin-repartitioning" title="Permanent link">&para;</a></h4>
-<div style="display: flex; align-items: top; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-start mb-3">
+<div class="col-md-9">
Round-robin repartitioning is the simplest partitioning strategy. Incoming data is processed in batches (chunks of rows), and these batches are distributed across partitions cyclically or sequentially, with each new batch assigned to the next available partition.
<br/><br/>
Round-robin repartitioning is useful when the data grouping isn't known or when aiming for an even distribution across partitions. Because it simply assigns batches in order without inspecting their contents, it is a low-overhead way to increase parallelism for downstream operations.
</div>
-<div style="flex: 0 0 25%; text-align: center;">
-<img alt="Round-Robin Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/round_robin_repartitioning.png" width="100%"/>
+<div class="col-md-3 text-center">
+<img alt="Round-Robin Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/round_robin_repartitioning.png"/>
</div>
</div>
<h4 id="hash-repartitioning"><strong>Hash Repartitioning</strong><a class="headerlink" href="#hash-repartitioning" title="Permanent link">&para;</a></h4>
-<div style="display: flex; align-items: top; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-start mb-3">
+<div class="col-md-9">
Hash repartitioning distributes data based on a hash function applied to one or more columns, called the partitioning key. Rows with the same hash value are placed in the same partition.
<br/><br/>
Hash repartitioning is useful when working with grouped data. Imagine you have a database containing information on company sales, and you are looking to find the total revenue each store produced. Hash repartitioning would make this query much more efficient. Rather than iterating over the data on a single thread and keeping a running sum for each store, it would be better to hash repartition on the store column and have multiple threads calculate individual store sales.
</div>
-<div style="flex: 0 0 25%; text-align: center;">
-<img alt="Hash Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/hash_repartitioning.png" width="100%"/>
+<div class="col-md-3 text-center">
+<img alt="Hash Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/hash_repartitioning.png"/>
</div>
</div>
<p>Note, the benefit of hash opposed to round-robin partitioning in this scenario. Hash repartitioning consolidates all rows with the same store value in distinct partitions. Because of this property we can compute the complete results for each store in parallel and merge them to get the final outcome. This parallel processing wouldn&rsquo;t be possible with only round-robin partitioning as the same store value may be spread across multiple partitions, making the aggregation re [...]
@@ -2538,7 +2852,7 @@ Hash repartitioning is useful when working with grouped data. Imagine you have a
<p>I hope you gained something from my experience and have fun learning about databases.</p>
<hr/>
<h2 id="acknowledgements"><strong>Acknowledgements</strong><a class="headerlink" href="#acknowledgements" title="Permanent link">&para;</a></h2>
-<p>Thank you to <a href="https://github.com/NGA-TRAN">Nga Tran</a> for continuous mentorship and guidance, the DataFusion community, specifically <a href="https://github.com/alamb">Andrew Lamb</a>, for lending me support throughout my work, and Datadog for providing the opportunity to work on such interesting systems.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.12.0 Release</title><link href="https://da [...]
+<p>Thank you to <a href="https://github.com/NGA-TRAN">Nga Tran</a> for continuous mentorship and guidance, the DataFusion community, specifically <a href="https://github.com/alamb">Andrew Lamb</a>, for lending me support throughout my work, and Datadog for providing the opportunity to work on such interesting systems.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.12.0 Release</title><link href="https://datafusion.a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2660,7 +2974,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 51.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0" rel="alternate"></link><published>2025-11-25T00:00:00+00:00</published><updated>2025-11-25T00:00:00+00:00</updated><author><name>pmc</name></autho [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 51.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0" rel="alternate"/><published>2025-11-25T00:00:00+00:00</published><updated>2025-11-25T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafu [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -2886,7 +3200,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.11.0 Release</title><link href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0" rel="alternate"></link><published>2025-10-21T00:00:00+00:00</published><updated>2025-10-21T00:00:00+00:00</updated><author><name>pmc</name></ [...]
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.11.0 Release</title><link href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0" rel="alternate"/><published>2025-10-21T00:00:00+00:00</published><updated>2025-10-21T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:d [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -3033,7 +3347,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 50.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0" rel="alternate"></link><published>2025-09-29T00:00:00+00:00</published><updated>2025-09-29T00:00:00+00:00</updated><author><name>pmc</name></autho [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 50.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0" rel="alternate"/><published>2025-09-29T00:00:00+00:00</published><updated>2025-09-29T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafu [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -3373,7 +3687,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Implementing User Defined Types and Custom Metadata in DataFusion</title><link href="https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata" rel="alternate"></link><published>2025-09-21T00:00:00+00:00</published><updated>2025-09-21T00:00:00+00:00</upd [...]
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Implementing User Defined Types and Custom Metadata in DataFusion</title><link href="https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata" rel="alternate"/><published>2025-09-21T00:00:00+00:00</published><updated>2025-09-21T00:00:00+00:00</updated><author><na [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -3643,7 +3957,7 @@ us and help the project.</p>
<li>Try out the project and provide feedback, file issues, and contribute code.</li>
<li>Work on a <a href="https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">good first issue</a>.</li>
<li>Reach out to us via the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.10.0 Release</title><link href="https://datafusion.apache.org/blog/2025/09/16/datafusion-comet-0.10.0" rel="alternate"></link><published>2025-09-16T00:00:00+00:00</published><updated>2025-09-16T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-09-16:/blog/2025/09/16/datafusion-comet-0.10.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.10.0 Release</title><link href="https://datafusion.apache.org/blog/2025/09/16/datafusion-comet-0.10.0" rel="alternate"/><published>2025-09-16T00:00:00+00:00</published><updated>2025-09-16T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-09-16:/blog/2025/09/16/datafusion-comet-0.10.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -3756,7 +4070,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</title><link href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters" rel="alternate"></link><published>2025-09-10T00:00:00+00:00</published><updated>2025-09 [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries</title><link href="https://datafusion.apache.org/blog/2025/09/10/dynamic-filters" rel="alternate"/><published>2025-09-10T00:00:00+00:00</published><updated>2025-09-10T00:00:00+00: [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -4329,7 +4643,7 @@ LIMIT 10;
<td style="text-align: right;">0.602</td>
</tr>
</tbody>
-</table></content><category term="blog"></category></entry><entry><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</title><link href="https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes" rel="alternate"></link><published>2025-08-15T00:00:00+00:00</published><updated>2025-08-15T00:00:00+00:00</updated><author><name>Andrew Lamb (InfluxData)</name></author><id>tag:datafusion.apache.org,2025-08-15:/blog/2025/ [...]
+</table></content><category term="blog"/></entry><entry><title>Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet</title><link href="https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes" rel="alternate"/><published>2025-08-15T00:00:00+00:00</published><updated>2025-08-15T00:00:00+00:00</updated><author><name>Andrew Lamb (InfluxData)</name></author><id>tag:datafusion.apache.org,2025-08-15:/blog/2025/08/15/external-p [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -4919,7 +5233,7 @@ such as point lookups, filters of different selectivity, and aggregations.</p
<p><a id="footnote7"></a><code>7</code>: For example, <a href="https://github.com/zhuqi-lucas">Qi Zhu</a> was able to speed up reads by over 2x
simply by rewriting the Parquet files with Offset Indexes and no compression (see <a href="https://github.com/apache/datafusion/issues/16149#issuecomment-2918761743">issue #16149 comment</a> for details).
There is likely significant additional performance available by using Bloom Filters and resorting the data
-to be clustered in a more optimal way for the queries.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 49.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0" rel="alternate"></link><published>2025-07-28T00:00:00+00:00</published><updated>2025-07-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-07-28:/blog/2025/07/28/datafusion-49.0.0</id><summary type="ht [...]
+to be clustered in a more optimal way for the queries.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 49.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/07/28/datafusion-49.0.0" rel="alternate"/><published>2025-07-28T00:00:00+00:00</published><updated>2025-07-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-07-28:/blog/2025/07/28/datafusion-49.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -5227,7 +5541,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests, or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>, and you
-can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 48.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0" rel="alternate"></link><published>2025-07-16T00:00:00+00:00</published><updated>2025-07-16T00:00:00+00:00</updated><author><name>PMC</name></author><id> [...]
+can find out how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion 48.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/07/16/datafusion-48.0.0" rel="alternate"/><published>2025-07-16T00:00:00+00:00</published><updated>2025-07-16T00:00:00+00:00</updated><author><name>PMC</name></author><id>tag:datafusion.a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -5423,7 +5737,7 @@ community discussions – whether you have questions, want to share how you’re
contribute, we’d love to hear from you. A list of open issues suitable for beginners
is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p>
-<p>Happy querying!</p></content><category term="blog"></category></entry><entry><title>Embedding User-Defined Indexes in Apache Parquet Files</title><link href="https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes" rel="alternate"></link><published>2025-07-14T00:00:00+00:00</published><updated>2025-07-14T00:00:00+00:00</updated><author><name>Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</name></author><id>tag: [...]
+<p>Happy querying!</p></content><category term="blog"/></entry><entry><title>Embedding User-Defined Indexes in Apache Parquet Files</title><link href="https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes" rel="alternate"/><published>2025-07-14T00:00:00+00:00</published><updated>2025-07-14T00:00:00+00:00</updated><author><name>Qi Zhu (Cloudera), Jigao Luo (Systems Group at TU Darmstadt), and Andrew Lamb (InfluxData)</name></author><id>tag:datafusion.apach [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -5556,24 +5870,24 @@ limitations under the License.
<hr/>
<p>A <strong>distinct value index</strong> stores the unique values of a specific column. This type of index is effective for columns with a small number of distinct values and can be used to quickly skip files that do not match the query. These indexes are popular in several engines, such as the <a href="https://clickhouse.com/docs/optimize/skipping-indexes#set">"set" Skip Index in ClickHouse</a> and the <a href="https://docs.influxdata.com/influxdb3/ent [...]
<p>For example, if the files contain a column named <code>Category</code> like this:</p>
-<table class="table" style="border-collapse:collapse;">
+<table class="table">
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><b><code>Category</code></b></td>
+<td><b><code>Category</code></b></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>foo</code></td>
+<td><code>foo</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>bar</code></td>
+<td><code>bar</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>...</code></td>
+<td><code>...</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>baz</code></td>
+<td><code>baz</code></td>
</tr>
<tr>
-<td style="border:1px solid #888;padding:2px 6px;"><code>foo</code></td>
+<td><code>foo</code></td>
</tr>
</table>
<p>The distinct value index will contain the values <code>foo</code>, <code>bar</code>, and <code>baz</code>. In contrast, traditional min/max statistics would store only the minimum (<code>bar</code>) and maximum (<code>foo</code>) values, so a query like</p>
@@ -5913,7 +6227,7 @@ it out, we would love for you to join us.</p>
<p><a id="footnote3"></a><code>3</code>: <a href="https://dl.gi.de/items/2a8571f8-0ef2-481c-8ee9-05f82ee258c8">Seamless Integration of Parquet Files into Data Processing. / Rey, Alice; Freitag, Michael; Neumann, Thomas. / BTW 2023</a></p>
<p><a id="footnote4"></a><code>4</code>: For more information about external indexes, see <a href="https://www.youtube.com/watch?v=74YsJT1-Rdk">this talk</a> and the <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/parquet_index.rs">parquet_index.rs</a> and <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_parquet_index.rs">advanced_parquet_index.rs& [...]
<p><a id="footnote5"></a><code>5</code>: For information about rewriting files to optimize for specific queries, such as resorting, repartitioning, and tuning data page and row group sizes, see <a href="https://github.com/XiangpengHao/liquid-cache/issues/227">XiangpengHao/liquid‑cache#227</a> and the conversation between <a href="https://github.com/JigaoLuo">JigaoLuo</a> and <a href="https://github.com/XiangpengHao">XiangpengHao [...]
-<p><a id="footnote6"></a><code>6</code>: An index can also be stored inline in the key-value metadata. This approach is simple to implement and ensures the index is available once the footer is read, without additional I/O. However, it requires the index to be serialized as a UTF-8 string, which may be less efficient and increases the size of the footer metadata, impacting all Parquet readers, even those that ignore the index.</p></content><category te [...]
+<p><a id="footnote6"></a><code>6</code>: An index can also be stored inline in the key-value metadata. This approach is simple to implement and ensures the index is available once the footer is read, without additional I/O. However, it requires the index to be serialized as a UTF-8 string, which may be less efficient and increases the size of the footer metadata, impacting all Parquet readers, even those that ignore the index.</p></content><category te [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6097,7 +6411,7 @@ the SQL <code>varchar</code> columns will be mapped to the <a hre
logging, or metrics) across thread boundaries without depending on any specific instrumentation library.
You can use the <a href="https://docs.rs/datafusion/latest/datafusion/common/runtime/trait.JoinSetTracer.html">JoinSetTracer</a> API to instrument DataFusion plans with your own tracing or logging libraries, or
use pre-integrated community crates such as the <a href="https://github.com/datafusion-contrib/datafusion-tracing">datafusion-tracing</a> crate.</p>
-<div style="text-align: center;">
+<div class="text-center">
<a href="https://github.com/datafusion-contrib/datafusion-tracing">
<img alt="DataFusion telemetry project logo" class="img-fluid" src="/blog/images/datafusion-47.0.0/datafusion-telemetry.png" width="50%"/>
</a>
@@ -6155,7 +6469,7 @@ community discussions – whether you have questions, want to share how you’re
contribute, we’d love to hear from you. A list of open issues suitable for beginners
is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p>
-<p>Happy querying!</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.9.0 Release</title><link href="https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0" rel="alternate"></link><published>2025-07-01T00:00:00+00:00</published><updated>2025-07-01T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-07-01:/blog/2025/07/01/datafusion-comet-0.9.0</id><summary type="html"><!--
+<p>Happy querying!</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.9.0 Release</title><link href="https://datafusion.apache.org/blog/2025/07/01/datafusion-comet-0.9.0" rel="alternate"/><published>2025-07-01T00:00:00+00:00</published><updated>2025-07-01T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2025-07-01:/blog/2025/07/01/datafusion-comet-0.9.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6302,7 +6616,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</title><link href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="alternate"></link><published>2025-06-30T00:00:00+00:00</published><updated>2025-06-30T00:00:00+00:00</updat [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Using Rust async for Query Execution and Cancelling Long-Running Queries</title><link href="https://datafusion.apache.org/blog/2025/06/30/cancellation" rel="alternate"/><published>2025-06-30T00:00:00+00:00</published><updated>2025-06-30T00:00:00+00:00</updated><author><name [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6621,7 +6935,7 @@ The task will then yield, after which the Tokio runtime resets the counter.</
</figure>
<p>If we assume a task budget of 1 unit, each time Tokio schedules the task would result in the following sequence of function calls.</p>
<figure>
-<img alt="Sequence diagram showing how the tokio task budget is used and reset." class="img-fluid" src="/blog/images/task-cancellation/tokio_budget.png" style="width: 100%; max-width: 100%"/>
+<img alt="Sequence diagram showing how the tokio task budget is used and reset." class="img-fluid" src="/blog/images/task-cancellation/tokio_budget.png"/>
<figcaption>Tokio task budget system, assuming the task budget is set to 1, for the plan above.</figcaption>
</figure>
<p>The aggregation stream would try to poll the merge stream in a loop.
@@ -6709,7 +7023,7 @@ components, rather than as a single tightly integrated system.</p>
<p>The <a href="https://datafusion.apache.org/contributor-guide/communication.html">DataFusion community</a> is always looking for new contributors to help
improve the project. If you are interested in learning more about how query
execution works, help document or improve the DataFusion codebase, or just try
-it out, we would love for you to join us.</p></content><category term="blog"></category></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one" rel="alternate"></link><published>2025-06-15T00:00:00+00:00</published><updated>2025-06-15T00:00:00+00:00</updated><author><name>alamb, akurmustafa</name></author><id>tag:datafusion.apache.org [...]
+it out, we would love for you to join us.</p></content><category term="blog"/></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one" rel="alternate"/><published>2025-06-15T00:00:00+00:00</published><updated>2025-06-15T00:00:00+00:00</updated><author><name>alamb, akurmustafa</name></author><id>tag:datafusion.apache.org,2025-06-15:/blo [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -6927,7 +7241,7 @@ optimizations</a>.</p>
<p><a id="footnote3"></a><sup>[3]</sup> <a href="https://www.vldb.org/pvldb/vol17/p1350-justen.pdf">https://www.vldb.org/pvldb/vol17/p1350-justen.pdf</a></p>
<p><a id="footnote4"></a><sup>[4]</sup> <a href="https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/24101">https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/24101</a>, <a href="https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/22111">https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/22111</a>, <a href="https://www.dagstuhl.de/en/seminars/seminar-calendar/sem [...]
<p><a id="footnote5"></a><sup>[5]</sup> And thus in academic classes, by the time you get around to an optimizer the semester is over and everyone is ready for the semester to be done. Once industrial systems mature to the point where the optimizer is a bottleneck, the shiny new-ness of the<a href="https://en.wikipedia.org/wiki/Gartner_hype_cycle"> hype cycle</a> has worn off and it is likely in the trough of disappointment.</p>
-<p><a id="footnote6"></a><sup>[6]</sup> Often systems will classify these passes into different categories, but I am simplifying here</p></content><category term="blog"></category></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="alternate"></link><published>2025-06-15T00:00:00+00:00</publi [...]
+<p><a id="footnote6"></a><sup>[6]</sup> Often systems will classify these passes into different categories, but I am simplifying here</p></content><category term="blog"/></entry><entry><title>Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two" rel="alternate"/><published>2025-06-15T00:00:00+00:00</published><updated>20 [...]
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -7310,7 +7624,7 @@ community</a>. We welcome first time contributors as well as long time par
to the fun of building a database together.</p>
<h2 id="notes">Notes<a class="headerlink" href="#notes" title="Permanent link">¶</a></h2>
<p><a id="footnote7"></a><sup>[7]</sup> See <a href="https://btw-2015.informatik.uni-hamburg.de/res/proceedings/Hauptband/Wiss/Neumann-Unnesting_Arbitrary_Querie.pdf">Unnesting Arbitrary Queries</a> from Neumann and Kemper for a more academic treatment.</p>
-<p><a id="footnote8"></a><sup>[8]</sup> One of my favorite terms I learned from Andy Pavlo’s CMU online lectures</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.8.0 Release</title><link href="https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0" rel="alternate"></link><published>2025-05-06T00:00:00+00:00</published><updated>2025-05-06T00:00:00+00:00</updated><author><name>pmc</name></author> [...]
+<p><a id="footnote8"></a><sup>[8]</sup> One of my favorite terms I learned from Andy Pavlo’s CMU online lectures</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.8.0 Release</title><link href="https://datafusion.apache.org/blog/2025/05/06/datafusion-comet-0.8.0" rel="alternate"/><published>2025-05-06T00:00:00+00:00</published><updated>2025-05-06T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusi [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -7397,7 +7711,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>User defined Window Functions in DataFusion</title><link href="https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions" rel="alternate"></link><published>2025-04-19T00:00:00+00:00</published><updated>2025-04-19T00:00:00+00:00</updated><author>< [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>User defined Window Functions in DataFusion</title><link href="https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions" rel="alternate"/><published>2025-04-19T00:00:00+00:00</published><updated>2025-04-19T00:00:00+00:00</updated><author><name>Aditya Sing [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -7750,7 +8064,7 @@ giving it a try. This post was designed to make it easier for new users to work
<p>When it comes to designing UDFs, I strongly recommend reviewing the
<a href="https://datafusion.apache.org/library-user-guide/adding-udfs.html">Window functions</a> documentation.</p>
<p>A heartfelt thank you to <a href="https://github.com/alamb">@alamb</a> and <a href="https://github.com/andygrove">@andygrove</a> for their invaluable reviews and thoughtful feedback—they’ve been instrumental in shaping this post.</p>
-<p>The Apache Arrow and Apache DataFusion communities are vibrant, welcoming, and full of passionate developers building something truly powerful. If you’re excited about high-performance analytics and want to be part of an open-source journey, I highly encourage you to explore the <a href="(https://datafusion.apache.org/)">official documentation</a> and dive into one of the many <a href="https://github.com/apache/datafusion/issues">open issues</a>. There’s [...]
+<p>The Apache Arrow and Apache DataFusion communities are vibrant, welcoming, and full of passionate developers building something truly powerful. If you’re excited about high-performance analytics and want to be part of an open-source journey, I highly encourage you to explore the <a href="(https://datafusion.apache.org/)">official documentation</a> and dive into one of the many <a href="https://github.com/apache/datafusion/issues">open issues</a>. There’s [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -7767,19 +8081,11 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
-development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over …</p></summary><content type="html"><!--
+development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
+faster than any other implementation we know of.</p>
+<p>It is now possible to create …</p></summary><content type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -7796,16 +8102,6 @@ limitations under the License.
{% endcomment %}x
-->
-<style>
-/* Table borders */
-table, th, td {
- border: 1px solid black;
- border-collapse: collapse;
-}
-th, td {
- padding: 3px;
-}
-</style>
<p><strong>TLDR: TPC-H SF=100 in 1min using tpchgen-rs vs 30min+ with dbgen</strong>.</p>
<p>3 members of the <a href="https://datafusion.apache.org/">Apache DataFusion</a> community used Rust and open source
development to build <a href="https://github.com/clflushopt/tpchgen-rs">tpchgen-rs</a>, a fully open TPC-H data generator over 20x
@@ -8220,7 +8516,7 @@ DataFusion (see <a href="https://github.com/apache/datafusion/issues/14608"&g
<p><em>2</em>: It is possible to embed the dbgen code, which appears to be the approach taken by DuckDB. This approach was tried in GlareDB (<a href="https://github.com/GlareDB/glaredb/pull/3313">GlareDB/glaredb#3313</a>), but ultimately shelved given the amount of effort needed to adapt and isolate the dbgen code.</p>
<p><em>3</em>: It is pretty amazing to imagine the machine required to generate SF300 that had 1.8TB (!!) of RAM</p>
<p><em>4</em>: We tried to <a href="https://github.com/clflushopt/tpchgen-rs/pull/34">use Rayon (see discussion here)</a>, but could not easily keep memory bounded.</p>
-<p><em>5</em>: <code>tpchgen-cli</code> memory usage is a function of the number of threads: each thread needs some buffer space</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Python 46.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/03/30/datafusion-python-46.0.0" rel="alternate"></link><published>2025-03-30T00:00:00+00:00</published><updated>2025-03-30T00:00:00+00:00</updated><author><na [...]
+<p><em>5</em>: <code>tpchgen-cli</code> memory usage is a function of the number of threads: each thread needs some buffer space</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Python 46.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/03/30/datafusion-python-46.0.0" rel="alternate"/><published>2025-03-30T00:00:00+00:00</published><updated>2025-03-30T00:00:00+00:00</updated><author><name>timsaucer</na [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -8355,8 +8651,8 @@ for the user to click on it to expand the data.</li>
</ul>
<p>In the below view you can see an example of some of these features such as the
expandable text and scroll bars.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Example html rendering in a jupyter notebook." class="img-fluid" src="/blog/images/python-datafusion-46.0.0/html_rendering.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Example html rendering in a jupyter notebook." class="img-fluid" src="/blog/images/python-datafusion-46.0.0/html_rendering.png"/>
<figcaption>
<b>Figure 1</b>: With the html rendering enhancements, tables are more easily
viewable in jupyter notebooks.
@@ -8435,7 +8731,7 @@ to have you join us and help the project.</p>
<li>
<p>Join us on <a href="https://s.apache.org/slack-invite">ASF Slack</a> or the <a href="https://discord.gg/Qw5gKqHxUM">Arrow Rust Discord Server</a>.</p>
</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache DataFusion 46.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0" rel="alternate"></link><published>2025-03-24T00:00:00+00:00</published><updated>2025-03-24T00:00:00+00:00</updated><author><name>Oznur Hanci and Berkay Sahin on behalf of the PMC</name></author><id>tag:datafusion.apache.org,2025-03-24:/blog/2025/03/24/datafusion-46.0.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache DataFusion 46.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0" rel="alternate"/><published>2025-03-24T00:00:00+00:00</published><updated>2025-03-24T00:00:00+00:00</updated><author><name>Oznur Hanci and Berkay Sahin on behalf of the PMC</name></author><id>tag:datafusion.apache.org,2025-03-24:/blog/2025/03/24/datafusion-46.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -8539,7 +8835,7 @@ SELECT col2, col1 FROM t2;
<p>Upgrading to 46.0.0 should be straightforward for most users, but do review the <a href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade Guide for DataFusion 46.0.0</a> for detailed steps and code changes. The upgrade guide covers the breaking changes mentioned (like replacing old exec nodes with <code>DataSourceExec</code>, updating UDF invocation to <code>invoke_with_args</code>, etc.) and provides code snippets to h [...]
<h2 id="get-involved">Get Involved<a class="headerlink" href="#get-involved" title="Permanent link">¶</a></h2>
<p>Apache DataFusion is an open-source project, and we welcome involvement from anyone interested. Now is a great time to take 46.0.0 for a spin: try it out on your workloads, and let us know if you encounter any issues or have suggestions. You can report bugs or request features on our GitHub issue tracker, or better yet, submit a pull request. Join our community discussions – whether you have questions, want to share how you’re using DataFusion, or are looking to contribute, we’d [...]
-<p>Happy querying!</p></content><category term="blog"></category></entry><entry><title>Efficient Filter Pushdown in Parquet</title><link href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown" rel="alternate"></link><published>2025-03-21T00:00:00+00:00</published><updated>2025-03-21T00:00:00+00:00</updated><author><name>Xiangpeng Hao</name></author><id>tag:datafusion.apache.org,2025-03-21:/blog/2025/03/21/parquet-pushdown</id><summary type="html"><style>
+<p>Happy querying!</p></content><category term="blog"/></entry><entry><title>Efficient Filter Pushdown in Parquet</title><link href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown" rel="alternate"/><published>2025-03-21T00:00:00+00:00</published><updated>2025-03-21T00:00:00+00:00</updated><author><name>Xiangpeng Hao</name></author><id>tag:datafusion.apache.org,2025-03-21:/blog/2025/03/21/parquet-pushdown</id><summary type="html"><style>
figure {
margin: 20px 0;
}
@@ -8829,7 +9125,7 @@ and queries that get slower are likely due to noise.</p>
<p>Despite being simple in theory, filter pushdown in Parquet is non-trivial to implement.
It requires understanding both the Parquet format and reader implementation details.
The challenge lies in efficiently navigating through the dynamics of decoding, filter evaluation, and memory management.</p>
-<p>If you are interested in this level of optimization and want to help test, document and implement this type of optimization, come find us in the <a href="https://datafusion.apache.org/contributor-guide/communication.html">DataFusion Community</a>. We would love to have you. </p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.7.0 Release</title><link href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0 [...]
+<p>If you are interested in this level of optimization and want to help test, document and implement this type of optimization, come find us in the <a href="https://datafusion.apache.org/contributor-guide/communication.html">DataFusion Community</a>. We would love to have you. </p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.7.0 Release</title><link href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0" rel= [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -8931,7 +9227,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="alternate"></link><published>2025-03-20T00:00:00+00:00</published><updated>2025-03-20T00:00:00+00:00</updated><author><name [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="alternate"/><published>2025-03-20T00:00:00+00:00</published><updated>2025-03-20T00:00:00+00:00</updated><author><name>Xiangpeng Hao</ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9019,7 +9315,7 @@ The last step is to <a href="https://github.com/apache/datafusion/blob/31701b
<p>We are working to remove the <a href="https://github.com/apache/arrow-rs/issues/5523#issuecomment-2429470872">remaining performance issues</a> and enable it by default, which we will discuss in the next blog post.</p>
<h3 id="conclusion">Conclusion<a class="headerlink" href="#conclusion" title="Permanent link">¶</a></h3>
<p>DataFusion employs a multi-step approach to Parquet pruning, from column projection to row group stats, page stats, and potentially row-level filtering.
-Each step may reduce the amount of data to be read and processed, significantly improving query performance.</p></content><category term="blog"></category></entry><entry><title>Using Ordering for Better Plans in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="alternate"></link><published>2025-03-11T00:00:00+00:00</published><updated>2025-03-11T00:00:00+00:00</updated><author><name>Mustafa Akur, Andrew Lamb</name></author><i [...]
+Each step may reduce the amount of data to be read and processed, significantly improving query performance.</p></content><category term="blog"/></entry><entry><title>Using Ordering for Better Plans in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="alternate"/><published>2025-03-11T00:00:00+00:00</published><updated>2025-03-11T00:00:00+00:00</updated><author><name>Mustafa Akur, Andrew Lamb</name></author><id>tag:datafusion [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9041,7 +9337,7 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></summary><content type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -9064,7 +9360,7 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
</blockquote>
<p>There are various use cases where this type of analysis can be useful such as the following examples.</p>
@@ -9145,7 +9441,7 @@ Being logically streamable does not guarantee that a query will execute in a str
</tr>
</table>
<p><br/></p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>How can a table have multiple orderings?</strong> At first glance it may seem counterintuitive for a table to have more than one valid ordering. However, during query execution such scenarios can arise.
For example consider the following query:
@@ -9206,7 +9502,7 @@ DataFusion leverages these functional dependencies to infer new orderings as dat
<li>Equivalent Expression Groups (will be explained shortly)</li>
<li>Succinct Valid Orderings (will be explained shortly)</li>
</ul>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Note:</strong> These properties are implemented in the <code>EquivalenceProperties</code> structure in <code>DataFusion</code>, please see the <a href="https://github.com/apache/datafusion/blob/f47ea73b87eec4af044f9b9923baf042682615b2/datafusion/physical-expr/src/equivalence/properties/mod.rs#L134" target="_blank">source</a> for more details<br/>
</blockquote>
<p>These properties allow us to analyze whether the ordering requirement is satisfied by the data already.</p>
@@ -9216,7 +9512,7 @@ DataFusion leverages these functional dependencies to infer new orderings as dat
<ul>
<li>Columns <code>hostname</code> and <code>currency</code> are constant because every row in the table has the same value (<code>'app.example.com'</code> for <code>hostname</code>, and <code>'USD'</code> for <code>currency</code>) for these columns.</li>
</ul>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Note:</strong> Constant expressions can arise during query execution. For example, in following query:<br/>
<code>SELECT hostname FROM logs</code><br/><code>WHERE hostname='app.example.com'</code> <br/>
after filtering is done, for subsequent operators the <code>hostname</code> column will be constant.
@@ -9224,7 +9520,7 @@ DataFusion leverages these functional dependencies to infer new orderings as dat
<h3 id="2-equivalent-expression-groups">2. Equivalent Expression Groups<a class="headerlink" href="#2-equivalent-expression-groups" title="Permanent link">¶</a></h3>
<p>Equivalent expression groups are expressions that always hold the same value across rows. These expressions can be thought of as clones of each other and may arise from operations like <code>Filter</code>, <code>Join</code>, or <code>Projection</code>.</p>
<p>In the example table, the expressions <code>price</code> and <code>price_cloned</code> form one equivalence group, and <code>time</code> and <code>time_cloned</code> form another equivalence group.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Note:</strong> Equivalent expression groups can arise during the query execution. For example, in the following query:<br/>
<code>SELECT time, time as time_cloned FROM logs</code> <br/>
after the projection is done, for subsequent operators <code>time</code> and <code>time_cloned</code> will form an equivalence group. As another example, in the following query:<br/>
@@ -9292,7 +9588,7 @@ Following third and fourth constraints for the simplified table, the succinct va
<code>[amount ASC, price ASC]</code>, <br/>
<code>[time_bin ASC]</code>,<br/>
<code>[time ASC]</code> </p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<p><strong>How can DataFusion find orderings?</strong></p>
DataFusion's <code>CREATE EXTERNAL TABLE</code> has a <code>WITH ORDER</code> clause (see <a href="https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table">docs</a>) to specify the known orderings of the table during table creation. For example the following query:<br/>
<pre><code>
@@ -9412,7 +9708,7 @@ We go back to step 4 until the ordering requirement list is exhausted or its len
<p id="footnote4"><sup>[4]</sup>This means that, if ordering <code>[amount ASC, price ASC]</code> is a valid ordering for the table. We shouldn't enlist <code>[amount ASC]</code> as valid ordering. Validity of it can be deduced from the ordering: <code>[amount ASC, price ASC]</code>
<p id="footnote5"><sup>[5]</sup>Leading ordering requirement is the first ordering requirement in the list of lexicographic ordering requirement expression. As an example for the requirement: <code>[amount ASC, time_bin ASC, prices ASC, time ASC]</code>, leading ordering requirement is: <code>amount ASC</code>.</p>
<p id="footnote6"><sup>[6]</sup>Leading valid orderings are the first ordering for each valid ordering list in the table. As an example, for the valid orderings: <code>[amount ASC, prices ASC], [time_bin ASC], [time ASC]</code>, leading valid orderings will be: <code>amount ASC, time_bin ASC, time ASC</code>. </p>
-<p id="optimal"><sup>*</sup>Best depends on the use case, <code>DataFusion</code> has many various flags to communicate what user thinks the best plan is (e.g. streamable, fastest, lowest memory, etc.). See <a href="https://datafusion.apache.org/user-guide/configs.html" target="_blank">configurations</a> for detail.</p></p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 45.0.0 Released</title><link h [...]
+<p id="optimal"><sup>*</sup>Best depends on the use case, <code>DataFusion</code> has many various flags to communicate what user thinks the best plan is (e.g. streamable, fastest, lowest memory, etc.). See <a href="https://datafusion.apache.org/user-guide/configs.html" target="_blank">configurations</a> for detail.</p></p></content><category term="blog"/></entry><entry><title>Apache DataFusion 45.0.0 Released</title><link href="https [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9602,7 +9898,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
-can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="alternate"></link><published>2025-02-17T00:00:00+00:00</published><updated>2025-02-17T00:00:00+00:00</updated><author><name>pmc</name></author [...]
+can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="alternate"/><published>2025-02-17T00:00:00+00:00</published><updated>2025-02-17T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafus [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9682,7 +9978,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Ballista 43.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0" rel="alternate"></link><published>2025-02-02T00:00:00+00:00</published><updated>2025-02-02T00:00:00+00:00</updated><author><name [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Ballista 43.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0" rel="alternate"/><published>2025-02-02T00:00:00+00:00</published><updated>2025-02-02T00:00:00+00:00</updated><author><name>milenkovicm</na [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9774,7 +10070,7 @@ async fn main() -&gt; datafusion::error::Result&lt;()&gt; {
<h2 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">¶</a></h2>
<p>Moving forward, Ballista will adopt the same release cadence as DataFusion, providing synchronized updates across the ecosystem.
Currently, there is no established long-term roadmap for Ballista. A plan will be formulated in the coming months based on community feedback and the availability of additional maintainers.</p>
-<p>In the short term, development efforts will concentrate on closing the feature gap between DataFusion and Ballista. Key priorities include implementing support for <code>INSERT INTO</code>, enabling table <code>URL</code> functionality, and achieving deeper integration with the Python ecosystem.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.5.0 Release</title><link href="https://datafusion.apache.org/b [...]
+<p>In the short term, development efforts will concentrate on closing the feature gap between DataFusion and Ballista. Key priorities include implementing support for <code>INSERT INTO</code>, enabling table <code>URL</code> functionality, and achieving deeper integration with the Python ecosystem.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.5.0 Release</title><link href="https://datafusion.apache.org/blog/2025/0 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -9869,7 +10165,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Python 43.1.0 Released</title><link href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0" rel="alternate"></link><published>2024-12-14T00:00:00+00:00</published><updated>2024-12-14T00:00:00+00:00</updated><author><name>tim [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Python 43.1.0 Released</title><link href="https://datafusion.apache.org/blog/2024/12/14/datafusion-python-43.1.0" rel="alternate"/><published>2024-12-14T00:00:00+00:00</published><updated>2024-12-14T00:00:00+00:00</updated><author><name>timsaucer</name></a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10022,7 +10318,7 @@ page.</p>
<li>
<p>Try out the project and provide feedback, file issues, and contribute code.</p>
</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.4.0 Release</title><link href="https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0" rel="alternate"></link><published>2024-11-20T00:00:00+00:00</published><updated>2024-11-20T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-11-20:/blog/2024/11/20/datafusion-comet-0.4.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.4.0 Release</title><link href="https://datafusion.apache.org/blog/2024/11/20/datafusion-comet-0.4.0" rel="alternate"/><published>2024-11-20T00:00:00+00:00</published><updated>2024-11-20T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-11-20:/blog/2024/11/20/datafusion-comet-0.4.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10122,7 +10418,7 @@ project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIA
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Comparing approaches to User Defined Functions in Apache DataFusion using Python</title><link href="https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons" rel="alternate"></link><published>2024-11-19T00:00:00+00:00</published><updated>202 [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Comparing approaches to User Defined Functions in Apache DataFusion using Python</title><link href="https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons" rel="alternate"/><published>2024-11-19T00:00:00+00:00</published><updated>2024-11-19T00:00:00 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10686,7 +10982,7 @@ for their helpful reviews and feedback.</p>
<p>Lastly, the Apache Arrow and DataFusion community is an active group of very helpful people working
to make a great tool. If you want to get involved, please take a look at the
<a href="https://datafusion.apache.org/python/">online documentation</a> and jump in to help with one of the
-<a href="https://github.com/apache/datafusion-python/issues">open issues</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</title><link href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" rel="alternate"></link><published>2024-11-18T00:00:00+00:00</published><updated>2024-11-18T00:00:00+00:00</updated><author [...]
+<a href="https://github.com/apache/datafusion-python/issues">open issues</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</title><link href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" rel="alternate"/><published>2024-11-18T00:00:00+00:00</published><updated>2024-11-18T00:00:00+00:00</updated><author><name>Andrew La [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10912,7 +11208,7 @@ online</a>.</p>
<p>[^2]: Want to try your hand at a custom format for ClickBench fame / glory?: <a href="https://github.com/apache/datafusion/issues/13448">Make DataFusion the fastest engine in ClickBench with custom file format</a></p>
<p>[^3]: We have contributors from North America, South American, Europe, Asia, Africa and Australia</p>
<p>[^4]: Undergraduates, PhD, Junior engineers, and getting-kind-of-crotchety experienced engineers</p>
-<p>[^5]: Thanks to Andy Pavlo, I love that nomenclature</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.3.0 Release</title><link href="https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0" rel="alternate"></link><published>2024-09-27T00:00:00+00:00</published><updated>2024-09-27T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-09-27:/blog/2024/09/27/datafusion-comet-0.3.0< [...]
+<p>[^5]: Thanks to Andy Pavlo, I love that nomenclature</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.3.0 Release</title><link href="https://datafusion.apache.org/blog/2024/09/27/datafusion-comet-0.3.0" rel="alternate"/><published>2024-09-27T00:00:00+00:00</published><updated>2024-09-27T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-09-27:/blog/2024/09/27/datafusion-comet-0.3.0</id><summary typ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -10998,7 +11294,7 @@ project.</p>
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</title><link href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1" rel="alternate"></link><published>2024-09-13T00:00:00+00:00</published [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</title><link href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1" rel="alternate"/><published>2024-09-13T00:00:00+00:00</published><updated>2024-0 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11129,7 +11425,7 @@ along with some of the pitfalls we encountered while implementing them.</p>
<p>[^1]: Benchmarked with AMD Ryzen 7600x (12 core, 24 threads, 32 MiB L3), WD Black SN770 NVMe SSD (5150MB/4950MB seq RW bandwidth)</p>
<p>[^2]: Xiangpeng is a PhD student at the University of Wisconsin-Madison</p>
<p>[^3]: There is also a corresponding <em>BinaryViewArray</em> which is similar except that the data is not constrained to be UTF-8 encoded strings.</p>
-<p>[^4]: We also make sure that offsets do not break a UTF-8 code point, which is <a href="https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/buffer/offset_buffer.rs#L62-L71">cheaply validated</a>.</p></content><category term="blog"></category></entry><entry><title>Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</title><link href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings [...]
+<p>[^4]: We also make sure that offsets do not break a UTF-8 code point, which is <a href="https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/buffer/offset_buffer.rs#L62-L71">cheaply validated</a>.</p></content><category term="blog"/></entry><entry><title>Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</title><link href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2" r [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11238,7 +11534,7 @@ In certain cases, we found that multiple calls to <a href="https://docs.rs/ar
<p>StringView is a big project that has received tremendous community support. Specifically, we would like to thank <a href="https://github.com/tustvold">@tustvold</a>, <a href="https://github.com/ariesdevil">@ariesdevil</a>, <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a>, <a href="https://github.com/ClSlaid">@ClSlaid</a>, <a href="https://github.com/2010YOUY01">@2010YOUY01</a>, <a href="https://github.com/ [...]
<p>As the introduction states, “German Style Strings” is a relatively straightforward research idea that avoid some string copies and accelerates comparisons. However, applying this (great) idea in practice requires a significant investment in careful software engineering. Again, we encourage the research community to continue to help apply research ideas to industrial systems, such as DataFusion, as doing so provides valuable perspectives when evaluating future research questions [...]
<h3 id="footnotes">Footnotes<a class="headerlink" href="#footnotes" title="Permanent link">¶</a></h3>
-<p>[^5]: There are additional optimizations possible in this operation that the community is working on, such as <a href="https://github.com/apache/datafusion/issues/7957">https://github.com/apache/datafusion/issues/7957</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.2.0 Release</title><link href="https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0" rel="alternate"></link><published>2024-08-28T0 [...]
+<p>[^5]: There are additional optimizations possible in this operation that the community is working on, such as <a href="https://github.com/apache/datafusion/issues/7957">https://github.com/apache/datafusion/issues/7957</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.2.0 Release</title><link href="https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0" rel="alternate"/><published>2024-08-28T00:00:00+00:00</p [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11327,7 +11623,7 @@ project.</p>
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Python 40.1.0 Released, Significant usability updates</title><link href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0" rel="alternate"></link><published>2024-08-20T00:00:00+00:00</published><updated>2024-08-20T00:00:00+0 [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Python 40.1.0 Released, Significant usability updates</title><link href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0" rel="alternate"/><published>2024-08-20T00:00:00+00:00</published><updated>2024-08-20T00:00:00+00:00</updated><a [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11399,8 +11695,8 @@ the available functions and classes to see the breadth of available functionalit
<a href="https://jedi.readthedocs.io/en/latest/">Jedi</a> to perform analysis of python code, provide useful
hints, and identify usage errors. These are major tools in the python user community. With this
release, users can fully use these tools in their workflow.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Enhanced tooltips in an IDE." class="img-fluid" src="/blog/images/python-datafusion-40.0.0/vscode_hover_tooltip.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Enhanced tooltips in an IDE." class="img-fluid" src="/blog/images/python-datafusion-40.0.0/vscode_hover_tooltip.png"/>
<figcaption>
<b>Figure 1</b>: With the enhanced python wrappers, users can see helpful tool tips with
type annotations directly in modern IDEs.
@@ -11408,8 +11704,8 @@ release, users can fully use these tools in their workflow.</p>
</figure>
<p>By having the type annotations, these IDEs can also identify quickly when a user has incorrectly
used a function's arguments as shown in Figure 2.</p>
-<figure style="text-align: center;">
-<img alt="Fig 2: Error checking in static analysis" class="img-fluid" src="/blog/images/python-datafusion-40.0.0/pylance_error_checking.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 2: Error checking in static analysis" class="img-fluid" src="/blog/images/python-datafusion-40.0.0/pylance_error_checking.png"/>
<figcaption>
<b>Figure 2</b>: Modern Python language servers can perform static analysis and quickly find
errors in the arguments to functions.
@@ -11499,7 +11795,7 @@ page.</p>
<li>
<p>Try out the project and provide feedback, file issues, and contribute code.</p>
</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache DataFusion 40.0.0 Released</title><link href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0" rel="alternate"></link><published>2024-07-24T00:00:00+00:00</published><updated>2024-07-24T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-07-24:/blog/2024/07/24/datafusion-40.0.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache DataFusion 40.0.0 Released</title><link href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0" rel="alternate"/><published>2024-07-24T00:00:00+00:00</published><updated>2024-07-24T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-07-24:/blog/2024/07/24/datafusion-40.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11826,7 +12122,7 @@ build a shared technology that none of us could have built alone.</p>
DataFusion on some of your own data and projects and let us know how it goes,
contribute suggestions, documentation, bug reports, or a PR with documentation,
tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
-can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.1.0 Release</title><link href="https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0" rel="alternate"></link><published>2024-07-20T00:00:00+00:00</published><updated>2024-07-20T00:00:00+00:00</updated><author><name>pmc</name></author [...]
+can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"/></entry><entry><title>Apache DataFusion Comet 0.1.0 Release</title><link href="https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0" rel="alternate"/><published>2024-07-20T00:00:00+00:00</published><updated>2024-07-20T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafus [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -11942,7 +12238,7 @@ recordings of previous calls.</p>
<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Announcing Apache Arrow DataFusion is now Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp" rel="alternate"></link><published>2024-05-07T00:00:00+00:00</published><updated>2024-05-07T00:00:00+00:00</updated><author> [...]
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"/></entry><entry><title>Announcing Apache Arrow DataFusion is now Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp" rel="alternate"/><published>2024-05-07T00:00:00+00:00</published><updated>2024-05-07T00:00:00+00:00</updated><author><name>pmc</name> [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12021,7 +12317,7 @@ DataFusion</a> website.</p>
us. Get in touch using <a href="https://datafusion.apache.org/contributor-guide/communication.html">Communication Doc</a> and learn how to get involved in the
<a href="https://datafusion.apache.org/contributor-guide/index.html">Contributor Guide</a>. We welcome everyone to try DataFusion on their
own data and projects and let us know how it goes, contribute suggestions,
-documentation, bug reports, or a PR with documentation, tests or code.</p></content><category term="blog"></category></entry><entry><title>Announcing Apache Arrow DataFusion Comet</title><link href="https://datafusion.apache.org/blog/2024/03/06/comet-donation" rel="alternate"></link><published>2024-03-06T00:00:00+00:00</published><updated>2024-03-06T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-03-06:/blog/2024/03/06/comet-donation</id> [...]
+documentation, bug reports, or a PR with documentation, tests or code.</p></content><category term="blog"/></entry><entry><title>Announcing Apache Arrow DataFusion Comet</title><link href="https://datafusion.apache.org/blog/2024/03/06/comet-donation" rel="alternate"/><published>2024-03-06T00:00:00+00:00</published><updated>2024-03-06T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-03-06:/blog/2024/03/06/comet-donation</id><summary type="h [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12071,8 +12367,8 @@ a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/dat
accelerate Spark workloads. It is designed as a drop-in
replacement for Spark's JVM based SQL execution engine and offers significant
performance improvements for some workloads as shown below.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-comet/comet-architecture.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-comet/comet-architecture.png"/>
<figcaption>
<b>Figure 1</b>: With Comet, users interact with the same Spark ecosystem, tools
and APIs such as Spark SQL. Queries still run through Spark's query optimizer and planner.
@@ -12105,7 +12401,7 @@ expect to post another update with more details at that time.</p>
<li>
<p>Try out the project and provide feedback, file issues, and contribute code.</p>
</li>
-</ul></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</title><link href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0" rel="alternate"></link><published>2024-01-19T00:00:00+00:00</published><updated>2024-01-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-01-19:/blog/2024/01/19/datafusion-34.0.0</id><summary type="html"><!--
+</ul></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</title><link href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0" rel="alternate"/><published>2024-01-19T00:00:00+00:00</published><updated>2024-01-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2024-01-19:/blog/2024/01/19/datafusion-34.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12201,16 +12497,16 @@ more than 2x faster on <a href="https://benchmark.clickhouse.com/">ClickBe
Average runtime on 25.0.0: 7.2s (for the queries that actually ran)
Average runtime on 34.0.0: 3.6s (for the same queries that ran in 25.0.0)
-->
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare-new.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare-new.png"/>
<figcaption>
<b>Figure 1</b>: Performance improvement between <code>25.0.0</code> and <code>34.0.0</code> on ClickBench.
Note that DataFusion <code>25.0.0</code>, could not run several queries due to
unsupported SQL (Q9, Q11, Q12, Q14) or memory requirements (Q33).
</figcaption>
</figure>
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare.png" width="100%"/>
+<figure class="text-center">
+<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare.png"/>
<figcaption>
<b>Figure 2</b>: Total query runtime for DataFusion <code>34.0.0</code> and DataFusion <code>25.0.0</code>.
</figcaption>
@@ -12372,7 +12668,7 @@ suitable for beginners is <a href="https://github.com/apache/arrow-datafusion
meetings. Timezones are always a challenge for such meetings, but we hope to
have two calls that can work for most attendees. If you are interested
in helping, or just want to say hi, please drop us a note via one of
-the methods listed in our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a>.</p></content><category term="blog"></category></entry><entry><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link href="https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping" rel="alternate"></link><published>2023-08-05T00:00:00+00:00</published><updated>2023-08-05T00:00:00+00:00</update [...]
+the methods listed in our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a>.</p></content><category term="blog"/></entry><entry><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link href="https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping" rel="alternate"/><published>2023-08-05T00:00:00+00:00</published><updated>2023-08-05T00:00:00+00:00</updated><author><name> [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12695,7 +12991,7 @@ allocation using the arrow Row format
<p>[^3]: <code>SELECT COUNT(DISTINCT "SearchPhrase") as num_phrases FROM 'hits.parquet';</code></p>
<p>[^4]: <code>SELECT COUNT(*) FROM (SELECT DISTINCT "UserID", "SearchPhrase" FROM 'hits.parquet')</code></p>
<p>[^5]: Full script at <a href="https://github.com/alamb/datafusion-duckdb-benchmark/blob/main/hash.py">hash.py</a></p>
-<p>[^6]: <a href="https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_%7B%7D.parquet">hits_0.parquet</a>, one of the files from the partitioned ClickBench dataset, which has <code>100,000</code> rows and is 117 MB in size. The entire dataset has <code>100,000,000</code> rows in a single 14 GB Parquet file. The script did not complete on the entire dataset after 40 minutes, and used 212 GB RAM at peak.</p></content><categ [...]
+<p>[^6]: <a href="https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_%7B%7D.parquet">hits_0.parquet</a>, one of the files from the partitioned ClickBench dataset, which has <code>100,000</code> rows and is 117 MB in size. The entire dataset has <code>100,000,000</code> rows in a single 14 GB Parquet file. The script did not complete on the entire dataset after 40 minutes, and used 212 GB RAM at peak.</p></content><categ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -12915,7 +13211,7 @@ data and projects and let us know how it goes or contribute a PR with
documentation, tests or code. A list of open issues suitable for
beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
<p>Check out our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a> for more ways to engage with the
-community.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 16.0.0 Project Update</title><link href="https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0" rel="alternate"></link><published>2023-01-19T00:00:00+00:00</published><updated>2023-01-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2023-01-19:/blog/2023/01/19/datafusion-16.0.0</id><summary type="html"><!--
+community.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 16.0.0 Project Update</title><link href="https://datafusion.apache.org/blog/2023/01/19/datafusion-16.0.0" rel="alternate"/><published>2023-01-19T00:00:00+00:00</published><updated>2023-01-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2023-01-19:/blog/2023/01/19/datafusion-16.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13159,7 +13455,7 @@ ways to engage with the community.</p>
1 yujie.zhang
1 zembunia
1 哇呜哇呜呀咦耶
-</code></pre></content><category term="blog"></category></entry><entry><title>Apache Arrow Ballista 0.9.0 Release</title><link href="https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0" rel="alternate"></link><published>2022-10-28T00:00:00+00:00</published><updated>2022-10-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-10-28:/blog/2022/10/28/ballista-0.9.0</id><summary type="html"><!--
+</code></pre></content><category term="blog"/></entry><entry><title>Apache Arrow Ballista 0.9.0 Release</title><link href="https://datafusion.apache.org/blog/2022/10/28/ballista-0.9.0" rel="alternate"/><published>2022-10-28T00:00:00+00:00</published><updated>2022-10-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-10-28:/blog/2022/10/28/ballista-0.9.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13259,7 +13555,7 @@ tuning Ballista.</p>
<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
<p>Ballista has a friendly community and we welcome contributions. A good place to start is to following the instructions
in the <a href="https://arrow.apache.org/ballista/">user guide</a> and try using Ballista with your own SQL queries and ETL pipelines, and file issues
-for any bugs or feature suggestions.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 13.0.0 Project Update</title><link href="https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0" rel="alternate"></link><published>2022-10-25T00:00:00+00:00</published><updated>2022-10-25T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-10-25:/blog/2022/10/25/datafusion-13.0.0</id><summary type="html"><!--
+for any bugs or feature suggestions.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 13.0.0 Project Update</title><link href="https://datafusion.apache.org/blog/2022/10/25/datafusion-13.0.0" rel="alternate"/><published>2022-10-25T00:00:00+00:00</published><updated>2022-10-25T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-10-25:/blog/2022/10/25/datafusion-13.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13476,7 +13772,7 @@ ways to engage with the community.</p>
1 nvartolomei
1 yourenawo
1 Özgür Akkurt
-</code></pre></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 8.0.0 Release</title><link href="https://datafusion.apache.org/blog/2022/05/16/datafusion-8.0.0" rel="alternate"></link><published>2022-05-16T00:00:00+00:00</published><updated>2022-05-16T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-05-16:/blog/2022/05/16/datafusion-8.0.0</id><summary type="html"><!--
+</code></pre></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 8.0.0 Release</title><link href="https://datafusion.apache.org/blog/2022/05/16/datafusion-8.0.0" rel="alternate"/><published>2022-05-16T00:00:00+00:00</published><updated>2022-05-16T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-05-16:/blog/2022/05/16/datafusion-8.0.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13666,7 +13962,7 @@ love to have you join us on the journey! You can help by trying out DataFusion o
and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable
for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p></content><category term="blog"></category></entry><entry><title>Introducing Apache Arrow DataFusion Contrib</title><link href="https://datafusion.apache.org/blog/2022/03/21/datafusion-contrib" rel="alternate"></link><published>2022-03-21T00:00:00+00:00</published><updated>2022-03-21T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-03-21:/blog/2022/03/21/datafusion-contrib</id><summary type="html"><!--
+ways to engage with the community.</p></content><category term="blog"/></entry><entry><title>Introducing Apache Arrow DataFusion Contrib</title><link href="https://datafusion.apache.org/blog/2022/03/21/datafusion-contrib" rel="alternate"/><published>2022-03-21T00:00:00+00:00</published><updated>2022-03-21T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-03-21:/blog/2022/03/21/datafusion-contrib</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13820,7 +14116,7 @@ the art query processing, we would love to have you join us on the journey! You
can help by trying out DataFusion on some of your own data and projects and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
<p>The best way to find out about creating new extensions within DataFusion-Contrib is reaching out on the <code>#arrow-rust</code> channel of the Apache Software Foundation <a href="https://join.slack.com/t/the-asf/shared_invite/zt-vlfbf7ch-HkbNHiU_uDlcH_RvaHv9gQ">Slack</a> workspace.</p>
<p>You can also check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more ways to engage with the community.</p>
-<p>Links for each DataFusion-Contrib repository are provided above if you would like to contribute to those.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 7.0.0 Release</title><link href="https://datafusion.apache.org/blog/2022/02/28/datafusion-7.0.0" rel="alternate"></link><published>2022-02-28T00:00:00+00:00</published><updated>2022-02-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022 [...]
+<p>Links for each DataFusion-Contrib repository are provided above if you would like to contribute to those.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 7.0.0 Release</title><link href="https://datafusion.apache.org/blog/2022/02/28/datafusion-7.0.0" rel="alternate"/><published>2022-02-28T00:00:00+00:00</published><updated>2022-02-28T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2022-02-28:/blog/202 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -13978,7 +14274,7 @@ git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc
the art query processing, we would love to have you join us on the journey! You
can help by trying out DataFusion on some of your own data and projects and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 6.0.0 Release</title><link href="https://datafusion.apache.org/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" rel="alternate"></link><published>2021-11-19T00:00:00+00:00</published><updated>2021-11-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-11-19:/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md</id><summary [...]
+ways to engage with the community.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 6.0.0 Release</title><link href="https://datafusion.apache.org/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" rel="alternate"/><published>2021-11-19T00:00:00+00:00</published><updated>2021-11-19T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-11-19:/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14134,7 +14430,7 @@ improve the documentation, or contribute to the documentation, tests or code. A
beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p>
<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow Ballista 0.5.0 Release</title><link href="https://datafusion.apache.org/blog/2021/08/18/ballista-0.5.0" rel="alternate"></link><published>2021-08-18T00:00:00+00:00</published><updated>2021-08-18T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/ballista-0.5.0</id><summary type="html"><!--
+ways to engage with the community.</p></content><category term="blog"/></entry><entry><title>Apache Arrow Ballista 0.5.0 Release</title><link href="https://datafusion.apache.org/blog/2021/08/18/ballista-0.5.0" rel="alternate"/><published>2021-08-18T00:00:00+00:00</published><updated>2021-08-18T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/ballista-0.5.0</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14220,7 +14516,7 @@ scalability as well as improving the documentation.</p>
can help by trying out Ballista on some of your own data and projects and filing bug reports and helping to
improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p></content><category term="blog"></category></entry><entry><title>Apache Arrow DataFusion 5.0.0 Release</title><link href="https://datafusion.apache.org/blog/2021/08/18/datafusion-5.0.0" rel="alternate"></link><published>2021-08-18T00:00:00+00:00</published><updated>2021-08-18T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-08-18:/blog [...]
+and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p></content><category term="blog"/></entry><entry><title>Apache Arrow DataFusion 5.0.0 Release</title><link href="https://datafusion.apache.org/blog/2021/08/18/datafusion-5.0.0" rel="alternate"/><published>2021-08-18T00:00:00+00:00</published><updated>2021-08-18T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2021-08-18:/blog/2021/08/18/data [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14337,7 +14633,7 @@ performance of individual TPC-H queries compared to the previous release.</p&
can help by trying out DataFusion on some of your own data and projects and filing bug reports and helping to
improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p></content><category term="blog"></category></entry><entry><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2021/04/12/ballista-donation" rel="alternate"></link><published>2021-04-12T00:00:00+00:00</published><updated>2021-04-12T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org [...]
+and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p></content><category term="blog"/></entry><entry><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2021/04/12/ballista-donation" rel="alternate"/><published>2021-04-12T00:00:00+00:00</published><updated>2021-04-12T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org,2021-04-12:/blo [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14409,7 +14705,7 @@ datasets and file bug reports for any issues that you find. You could also check
<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20component%20%3D%20%22Rust%20-%20Ballista%22">list of issues</a> and have a go at fixing one.</p>
<p>The <a href="https://github.com/apache/arrow/blob/master/rust/README.md#arrow-rust-community">Arrow Rust Community</a>
section of the Rust README provides information on other ways to interact with the Ballista contributors and
-maintainers.</p></content><category term="blog"></category></entry><entry><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="alternate"></link><published>2019-02-04T00:00:00+00:00</published><updated>2019-02-04T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</id><summary type="html"><!--
+maintainers.</p></content><category term="blog"/></entry><entry><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link href="https://datafusion.apache.org/blog/2019/02/04/datafusion-donation" rel="alternate"/><published>2019-02-04T00:00:00+00:00</published><updated>2019-02-04T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.apache.org,2019-02-04:/blog/2019/02/04/datafusion-donation</id><summary type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -14528,4 +14824,4 @@ while let Some(batch) = results.next().unwrap() {
<li>Creating a Docker image to make it easy to use DataFusion as a standalone query tool for interactive and batch queries</li>
</ul>
<h2 id="contributors-welcome">Contributors Welcome!<a class="headerlink" href="#contributors-welcome" title="Permanent link">¶</a></h2>
-<p>If you are excited about being able to use Rust for data science and would like to contribute to this work then there are many ways to get involved. The simplest way to get started is to try out DataFusion against your own data sources and file bug reports for any issues that you find. You could also check out the current <a href="https://cwiki.apache.org/confluence/display/ARROW/Rust+JIRA+Dashboard">list of issues</a> and have a go at fixing one. You can also join t [...]
\ No newline at end of file
+<p>If you are excited about being able to use Rust for data science and would like to contribute to this work then there are many ways to get involved. The simplest way to get started is to try out DataFusion against your own data sources and file bug reports for any issues that you find. You could also check out the current <a href="https://cwiki.apache.org/confluence/display/ARROW/Rust+JIRA+Dashboard">list of issues</a> and have a go at fixing one. You can also join t [...]
\ No newline at end of file
diff --git a/blog/feeds/gene-bordegaray.atom.xml b/blog/feeds/gene-bordegaray.atom.xml
index 93e0208..6a845b2 100644
--- a/blog/feeds/gene-bordegaray.atom.xml
+++ b/blog/feeds/gene-bordegaray.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Gene Bordegaray</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/gene-bordegaray.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-12-15T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Gene Bordegaray</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/gene-bordegaray.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2025-12-15T00:00:00+00:00</updated><entry><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link href="https://datafusion.apache.o [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -17,8 +17,8 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div></summary><content type="html"><!--
{% comment %}
@@ -38,16 +38,16 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is exposed by a user-friendly interface and query languages (typically a dialect of SQL).
<br/><br/>
Starting a journey learning about database internals can be daunting. With so many topics that are whole PhD degrees themselves, finding a place to start is difficult. In this blog post, I will share my early journey in the database world and a quick lesson on one of the first topics I dove into. If you are new to the space, this post will help you get your first foot into the database world, and if you are already a veteran, you may still learn something new.
</div>
-<div style="flex: 0 0 40%; text-align: center;">
-<img alt="Database System Components" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/database_system_diagram.png" width="100%"/>
+<div class="col-md-5 text-center">
+<img alt="Database System Components" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/database_system_diagram.png"/>
</div>
</div>
<hr/>
@@ -88,29 +88,29 @@ DataFusion achieves multi-core parallelism through the use of "exchange operator
<h3 id="what-is-repartitioning">What is Repartitioning?<a class="headerlink" href="#what-is-repartitioning" title="Permanent link">&para;</a></h3>
<p>Partitioning is a "divide-and-conquer" approach to executing a query. Each partition is a subset of the data that is being processed on a single core. Repartitioning is an operation that redistributes data across different partitions to balance workloads, reduce data skew, and increase parallelism. Two repartitioning methods are used in DataFusion: round-robin and hash.</p>
<h4 id="round-robin-repartitioning"><strong>Round-Robin Repartitioning</strong><a class="headerlink" href="#round-robin-repartitioning" title="Permanent link">&para;</a></h4>
-<div style="display: flex; align-items: top; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-start mb-3">
+<div class="col-md-9">
Round-robin repartitioning is the simplest partitioning strategy. Incoming data is processed in batches (chunks of rows), and these batches are distributed across partitions cyclically or sequentially, with each new batch assigned to the next available partition.
<br/><br/>
Round-robin repartitioning is useful when the data grouping isn't known or when aiming for an even distribution across partitions. Because it simply assigns batches in order without inspecting their contents, it is a low-overhead way to increase parallelism for downstream operations.
</div>
-<div style="flex: 0 0 25%; text-align: center;">
-<img alt="Round-Robin Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/round_robin_repartitioning.png" width="100%"/>
+<div class="col-md-3 text-center">
+<img alt="Round-Robin Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/round_robin_repartitioning.png"/>
</div>
</div>
<h4 id="hash-repartitioning"><strong>Hash Repartitioning</strong><a class="headerlink" href="#hash-repartitioning" title="Permanent link">&para;</a></h4>
-<div style="display: flex; align-items: top; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-start mb-3">
+<div class="col-md-9">
Hash repartitioning distributes data based on a hash function applied to one or more columns, called the partitioning key. Rows with the same hash value are placed in the same partition.
<br/><br/>
Hash repartitioning is useful when working with grouped data. Imagine you have a database containing information on company sales, and you are looking to find the total revenue each store produced. Hash repartitioning would make this query much more efficient. Rather than iterating over the data on a single thread and keeping a running sum for each store, it would be better to hash repartition on the store column and have multiple threads calculate individual store sales.
</div>
-<div style="flex: 0 0 25%; text-align: center;">
-<img alt="Hash Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/hash_repartitioning.png" width="100%"/>
+<div class="col-md-3 text-center">
+<img alt="Hash Repartitioning" class="img-fluid" src="/blog/images/avoid-consecutive-repartitions/hash_repartitioning.png"/>
</div>
</div>
<p>Note, the benefit of hash opposed to round-robin partitioning in this scenario. Hash repartitioning consolidates all rows with the same store value in distinct partitions. Because of this property we can compute the complete results for each store in parallel and merge them to get the final outcome. This parallel processing wouldn&rsquo;t be possible with only round-robin partitioning as the same store value may be spread across multiple partitions, making the aggregation re [...]
@@ -265,4 +265,4 @@ Hash repartitioning is useful when working with grouped data. Imagine you have a
<p>I hope you gained something from my experience and have fun learning about databases.</p>
<hr/>
<h2 id="acknowledgements"><strong>Acknowledgements</strong><a class="headerlink" href="#acknowledgements" title="Permanent link">&para;</a></h2>
-<p>Thank you to <a href="https://github.com/NGA-TRAN">Nga Tran</a> for continuous mentorship and guidance, the DataFusion community, specifically <a href="https://github.com/alamb">Andrew Lamb</a>, for lending me support throughout my work, and Datadog for providing the opportunity to work on such interesting systems.</p></content><category term="blog"></category></entry></feed>
\ No newline at end of file
+<p>Thank you to <a href="https://github.com/NGA-TRAN">Nga Tran</a> for continuous mentorship and guidance, the DataFusion community, specifically <a href="https://github.com/alamb">Andrew Lamb</a>, for lending me support throughout my work, and Datadog for providing the opportunity to work on such interesting systems.</p></content><category term="blog"/></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/gene-bordegaray.rss.xml b/blog/feeds/gene-bordegaray.rss.xml
index 6a87e03..ab0da9e 100644
--- a/blog/feeds/gene-bordegaray.rss.xml
+++ b/blog/feeds/gene-bordegaray.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - Gene Bordegaray</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon, 15 Dec 2025 00:00:00 +0000</lastBuildDate><item><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link>https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - Gene Bordegaray</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/gene-bordegaray.rss.xml" rel="self"/><lastBuildDate>Mon, 15 Dec 2025 00:00:00 +0000</lastBuildDate><item><title>Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution</title><link>https://datafusion.apache.org/blo [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -17,7 +17,7 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
-<div style="display: flex; align-items: center; gap: 20px; margin-bottom: 20px;">
-<div style="flex: 1;">
+<div class="row align-items-center mb-3">
+<div class="col-md-7">
-Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gene Bordegaray</dc:creator><pubDate>Mon, 15 Dec 2025 00:00 [...]
\ No newline at end of file
+Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is …</div></div></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gene Bordegaray</dc:creator><pubDate>Mon, 15 Dec 2025 00:00 [...]
\ No newline at end of file
diff --git a/blog/feeds/geoffrey-claude-datadog.atom.xml b/blog/feeds/geoffrey-claude-datadog.atom.xml
index 1bf9465..d8e04d7 100644
--- a/blog/feeds/geoffrey-claude-datadog.atom.xml
+++ b/blog/feeds/geoffrey-claude-datadog.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Geoffrey Claude (Datadog)</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/geoffrey-claude-datadog.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-01-12T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https:/ [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Geoffrey Claude (Datadog)</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/geoffrey-claude-datadog.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-01-12T00:00:00+00:00</updated><entry><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https://datafusion.apache.org/blog/2026/ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -283,4 +283,4 @@ println!("{}", df.logical_plan().display_indent());
<li><strong>Try it out</strong>: Implement one of the extension points and share your experience</li>
<li><strong>File issues or join the conversation</strong>: <a href="https://github.com/apache/datafusion/">GitHub</a> for bugs and feature requests, <a href="https://datafusion.apache.org/contributor-guide/communication.html">Slack or Discord</a> for discussion</li>
</ul>
-<!-- Reference links --></content><category term="blog"></category></entry></feed>
\ No newline at end of file
+<!-- Reference links --></content><category term="blog"/></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/geoffrey-claude-datadog.rss.xml b/blog/feeds/geoffrey-claude-datadog.rss.xml
index 2ec017c..791b9d7 100644
--- a/blog/feeds/geoffrey-claude-datadog.rss.xml
+++ b/blog/feeds/geoffrey-claude-datadog.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - Geoffrey Claude (Datadog)</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon, 12 Jan 2026 00:00:00 +0000</lastBuildDate><item><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link>https://datafusion.apache.org/blog/2026/01/12/extending-sql</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - Geoffrey Claude (Datadog)</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/geoffrey-claude-datadog.rss.xml" rel="self"/><lastBuildDate>Mon, 12 Jan 2026 00:00:00 +0000</lastBuildDate><item><title>Extending SQL in DataFusion: from ->> to TABLESAMPLE</title><link>https://datafusion.apache.org/blog/2026/01/12/ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -19,4 +19,4 @@ limitations under the License.
-->
<p>If you embed <a href="https://datafusion.apache.org/">DataFusion</a> in your product, your users will eventually run SQL that DataFusion does not recognize. Not because the query is unreasonable, but because SQL in practice includes many dialects and system-specific statements.</p>
-<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Geoffrey Claude (Datadog)</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/extending-sql</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+<p>Suppose you store data as Parquet files on S3 and want users to attach an …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Geoffrey Claude (Datadog)</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/extending-sql</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/milenkovicm.atom.xml b/blog/feeds/milenkovicm.atom.xml
index 057813c..dc8ac84 100644
--- a/blog/feeds/milenkovicm.atom.xml
+++ b/blog/feeds/milenkovicm.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - milenkovicm</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/milenkovicm.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-02-02T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion Ballista 43.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/02/02/dat [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - milenkovicm</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/milenkovicm.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2025-02-02T00:00:00+00:00</updated><entry><title>Apache DataFusion Ballista 43.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0" rel="alt [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -91,4 +91,4 @@ async fn main() -&gt; datafusion::error::Result&lt;()&gt; {
<h2 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">¶</a></h2>
<p>Moving forward, Ballista will adopt the same release cadence as DataFusion, providing synchronized updates across the ecosystem.
Currently, there is no established long-term roadmap for Ballista. A plan will be formulated in the coming months based on community feedback and the availability of additional maintainers.</p>
-<p>In the short term, development efforts will concentrate on closing the feature gap between DataFusion and Ballista. Key priorities include implementing support for <code>INSERT INTO</code>, enabling table <code>URL</code> functionality, and achieving deeper integration with the Python ecosystem.</p></content><category term="blog"></category></entry></feed>
\ No newline at end of file
+<p>In the short term, development efforts will concentrate on closing the feature gap between DataFusion and Ballista. Key priorities include implementing support for <code>INSERT INTO</code>, enabling table <code>URL</code> functionality, and achieving deeper integration with the Python ecosystem.</p></content><category term="blog"/></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/milenkovicm.rss.xml b/blog/feeds/milenkovicm.rss.xml
index 4c6f0f0..6329cac 100644
--- a/blog/feeds/milenkovicm.rss.xml
+++ b/blog/feeds/milenkovicm.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - milenkovicm</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Sun, 02 Feb 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Ballista 43.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0</link><description><!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - milenkovicm</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/milenkovicm.rss.xml" rel="self"/><lastBuildDate>Sun, 02 Feb 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Ballista 43.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/02/02/datafusion-ballista-43.0.0</link><descript [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -21,4 +21,4 @@ limitations under the License.
<p>We are pleased to announce version <a href="https://github.com/apache/datafusion-ballista/blob/main/CHANGELOG.md#4300-2025-01-07">43.0.0</a> of the <a href="https://datafusion.apache.org/ballista/">DataFusion Ballista</a>. Ballista allows existing <a href="https://datafusion.apache.org">DataFusion</a> applications to be scaled out on a cluster for use cases that are not practical to run on a single node.</p>
<h2 id="highlights-of-this-release">Highlights of this release<a class="headerlink" href="#highlights-of-this-release" title="Permanent link">¶</a></h2>
<h3 id="seamless-integration-with-datafusion">Seamless Integration with DataFusion<a class="headerlink" href="#seamless-integration-with-datafusion" title="Permanent link">¶</a></h3>
-<p>The primary objective of …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">milenkovicm</dc:creator><pubDate>Sun, 02 Feb 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-02-02:/blog/2025/02/02/datafusion-ballista-43.0.0</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
+<p>The primary objective of …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">milenkovicm</dc:creator><pubDate>Sun, 02 Feb 2025 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2025-02-02:/blog/2025/02/02/datafusion-ballista-43.0.0</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/mustafa-akur-andrew-lamb.atom.xml b/blog/feeds/mustafa-akur-andrew-lamb.atom.xml
index b0613c0..d7de207 100644
--- a/blog/feeds/mustafa-akur-andrew-lamb.atom.xml
+++ b/blog/feeds/mustafa-akur-andrew-lamb.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Mustafa Akur, Andrew Lamb</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/mustafa-akur-andrew-lamb.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-11T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Using Ordering for Better Plans in Apache DataFusion</title><link href="https://data [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Mustafa Akur, Andrew Lamb</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/mustafa-akur-andrew-lamb.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2025-03-11T00:00:00+00:00</updated><entry><title>Using Ordering for Better Plans in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/03/11 [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -21,7 +21,7 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></summary><content type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -44,7 +44,7 @@ limitations under the License.
<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
</blockquote>
<p>There are various use cases where this type of analysis can be useful such as the following examples.</p>
@@ -125,7 +125,7 @@ Being logically streamable does not guarantee that a query will execute in a str
</tr>
</table>
<p><br/></p>
-<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
... 1956 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org