You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/03/20 19:50:20 UTC
(datafusion-site) branch asf-staging updated: Commit build products
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new ebdc943 Commit build products
ebdc943 is described below
commit ebdc94389b7dd3e721bdc4a71acb2e09ea5c40c8
Author: Build Pelican (action) <pr...@infra.apache.org>
AuthorDate: Thu Mar 20 19:50:16 2025 +0000
Commit build products
---
blog/2025/02/20/datafusion-45.0.0/index.html | 226 ++++++
blog/2025/03/11/ordering-analysis/index.html | 431 ++++++++++++
blog/2025/03/20/datafusion-comet-0.7.0/index.html | 139 ++++
blog/2025/03/20/parquet-pruning/index.html | 126 ++++
blog/author/mustafa-akur-andrew-lamb.html | 110 +++
blog/author/pmc.html | 82 +++
blog/author/xiangpeng-hao.html | 109 +++
blog/category/blog.html | 163 +++++
blog/feed.xml | 89 ++-
blog/feeds/all-en.atom.xml | 767 ++++++++++++++++++++-
blog/feeds/blog.atom.xml | 767 ++++++++++++++++++++-
blog/feeds/mustafa-akur-andrew-lamb.atom.xml | 394 +++++++++++
blog/feeds/mustafa-akur-andrew-lamb.rss.xml | 24 +
blog/feeds/pmc.atom.xml | 289 +++++++-
blog/feeds/pmc.rss.xml | 46 +-
blog/feeds/xiangpeng-hao.atom.xml | 88 +++
blog/feeds/xiangpeng-hao.rss.xml | 23 +
.../datafusion-45.0.0/performance_over_time.png | Bin 0 -> 52136 bytes
.../images/ordering_analysis/query_window_plan.png | Bin 0 -> 189377 bytes
blog/images/parquet-pruning/filter-pushdown.jpg | Bin 0 -> 376764 bytes
blog/images/parquet-pruning/read-parquet.jpg | Bin 0 -> 284097 bytes
blog/index.html | 163 +++++
22 files changed, 4031 insertions(+), 5 deletions(-)
diff --git a/blog/2025/02/20/datafusion-45.0.0/index.html b/blog/2025/02/20/datafusion-45.0.0/index.html
new file mode 100644
index 0000000..cdf1377
--- /dev/null
+++ b/blog/2025/02/20/datafusion-45.0.0/index.html
@@ -0,0 +1,226 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion 45.0.0 Released - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+<!-- page contents -->
+<div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+ <h1>
+ Apache DataFusion 45.0.0 Released
+ </h1>
+ <p>Posted on: Thu 20 February 2025 by pmc</p>
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development for DataFusion!</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that
+uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
+create new, fast data centric systems such as databases, dataframe libraries,
+machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion’s primary design
+goal</a> is to accelerate the creation of other data centric systems, it has a
+reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe library</a>,
+<a href="https://datafusion.apache.org/python/">python library</a> and <a href="https://datafusion.apache.org/user-guide/cli/">command line SQL tool</a>.</p>
+<p>DataFusion's core thesis is that as a community, together we can build much more
+advanced technology than any of us as individuals or companies could do alone.
+Without DataFusion, highly performant vectorized query engines would remain
+the domain of a few large companies and world-class research institutions.
+With DataFusion, we can all build on top of a shared foundation, and focus on
+what makes our projects unique.</p>
+<h2>Community Growth 📈</h2>
+<p>In the last 6 months, between <code>40.0.0</code> and <code>45.0.0</code>, our community continues to
+grow in new and exciting ways.</p>
+<ol>
+<li>We added several PMC members and new committers: <a href="https://github.com/jayzhan211">@jayzhan211</a> and <a href="https://github.com/jonahgao">@jonahgao</a> joined the PMC,
+ <a href="https://github.com/2010YOUY01">@2010YOUY01</a>, <a href="https://github.com/rachelint">@rachelint</a>, <a href="https://github.com/findepi/">@findpi</a>, <a href="https://github.com/iffyio">@iffyio</a>, <a href="https://github.com/goldmedal">@goldmedal</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/Michael-J-Ward">@Michael-J-Ward</a> and <a href="https://github.com/korowa">@korowa</a>
+ joined as committers. See the <a href="https://lists.apache.org/list.html?dev@datafusion.apache.org">mailing list</a> for more details.</li>
+<li>In the <a href="https://github.com/apache/arrow-datafusion">core DataFusion repo</a> alone we reviewed and accepted almost 1600 PRs from 206 different
+ committers, created over 1100 issues and closed 751 of them 🚀. All changes are listed in the detailed
+ <a href="https://github.com/apache/datafusion/tree/main/dev/changelog">changelogs</a>.</li>
+<li>DataFusion focused meetups happened in multiple cities around the world: <a href="https://github.com/apache/datafusion/discussions/10341#discussioncomment-10110273">Hangzhou</a>, <a href="https://github.com/apache/datafusion/discussions/11431">Belgrade</a>, <a href="https://github.com/apache/datafusion/discussions/11213">New York</a>,
+ <a href="https://github.com/apache/datafusion/discussions/10348">Seattle</a>, <a href="https://github.com/apache/datafusion/discussions/12894">Chicago</a>, <a href="https://github.com/apache/datafusion/discussions/13165">Boston</a> and <a href="https://github.com/apache/datafusion/discussions/12988">Amsterdam</a> as well as a Rust NYC meetup in NYC focused on DataFusion.</li>
+</ol>
+<!--
+$ git log --pretty=oneline 40.0.0..45.0.0 . | wc -l
+ 1532 (up from 1453)
+
+$ git shortlog -sn 40.0.0..45.0.0 . | wc -l
+ 206 (up from 182)
+
+https://crates.io/crates/datafusion/45.0.0
+DataFusion 45 released Feb 7, 2025
+
+https://crates.io/crates/datafusion/40.0.0
+DataFusion 40 released July 12, 2024
+
+Issues created in this time: 375 open, 751 closed (from 321 open, 781 closed)
+https://github.com/apache/datafusion/issues?q=is%3Aissue+created%3A2024-07-12..2025-02-07
+
+Issues closed: 956 (up from 911)
+https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2024-07-12..2025-02-07
+
+PRs merged in this time 1597 (up from 1490)
+https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2024-07-12..2025-02-07
+
+-->
+<p>DataFusion has put in an application to be part of <a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> with a
+<a href="https://github.com/apache/datafusion/issues/14478">number of ideas</a> for projects with mentors already selected. Additionally, <a href="https://github.com/apache/datafusion/issues/14373">some ideas</a> on
+how to make DataFusion an ideal selection for university database projects such as the
+<a href="https://15445.courses.cs.cmu.edu/spring2025/">CMU database classes</a> have been put forward.</p>
+<p>In addition, DataFusion has been appearing publicly more and more, both online and offline. Here are some highlights:</p>
+<ol>
+<li>A <a href="https://uwheel.rs/post/datafusion_uwheel/">demonstration of how uwheel</a> is integrated into DataFusion</li>
+<li>Integrating StringView into DataFusion - <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">part 1</a> and <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">part 2</a></li>
+<li><a href="https://techontherocks.show/3">Building streams</a> with DataFusion</li>
+<li><a href="https://blog.haoxp.xyz/posts/caching-datafusion">Caching in DataFusion</a>: Don't read twice</li>
+<li><a href="https://blog.haoxp.xyz/posts/parquet-to-arrow/">Parquet pruning in DataFusion</a>: Read no more than you need</li>
+<li>DataFusion is one of <a href="https://www.crn.com/news/software/2024/the-10-coolest-open-source-software-tools-of-2024?page=3">The 10 coolest open source software tools</a></li>
+<li><a href="https://www.denormalized.io/blog/building-databases">Building databases over a weekend</a></li>
+</ol>
+<h2>Improved Performance 🚀</h2>
+<p>DataFusion hit a milestone in its development by becoming <a href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/">the fastest single node engine</a>
+for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">clickbench</a> benchmark for the 43.0.0 release. A <a href="https://github.com/apache/datafusion/issues/12821">lot
+of work</a> went into making this happen! While other engines have subsequently gotten faster,
+displacing DataFusion from the top spot, DataFusion still remains near the top and we <a href="https://github.com/apache/datafusion/issues/14586">are planning
+more improvements</a>.</p>
+<p><img alt="ClickBench performance results over time for DataFusion" class="img-responsive" src="/blog/images/datafusion-45.0.0/performance_over_time.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: ClickBench performance improved over 33% between DataFusion 33
+(released Nov. 2023) and DataFusion 45 (released Feb. 2025). </p>
+<p>The task of <a href="https://github.com/apache/datafusion/issues/10918">integrating</a> the new <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html">Arrow StringView</a> which significantly improves performance
+for workloads that scan, filter and group by variable length string and binary data was completed
+and enabled by default in the past 6 months. The improvement is especially pronounced for Parquet
+files due to <a href="https://github.com/apache/arrow-rs/issues/5530">upstream work in the parquet reader</a>. Kudos to <a href="https://github.com/XiangpengHong">@XiangpengHong</a>, <a href="https://github.com/AriesDevil">@AriesDevil</a>,
+<a href="https://github.com/PsiACE">@PsiACE</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/a10y">@a10y</a>, and <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a> for driving this project.</p>
+<h2>Improved Quality 📋</h2>
+<p>DataFusion continues to improve overall in quality. In addition to ongoing bug
+fixes, one of the most exciting improvements in the last 6 months was the addition of the
+<a href="https://github.com/apache/datafusion/pull/13936">SQLite sqllogictest suite</a> thanks to <a href="https://github.com/Omega359">@Omega359</a>. These tests run over 5 million
+sql statements on every push to the main branch.</p>
+<p>Support for <a href="https://github.com/apache/datafusion/pull/13651">explicitly checking logical plan invariants</a> was added by <a href="https://github.com/wiedld">@wiedld</a> which
+can help catch implicit changes that might cause problems during upgrades.</p>
+<p>We have also started other quality initiatives to make it <a href="https://github.com/apache/datafusion/issues/13525">easier to use DataFusion</a>
+based on <a href="https://glaredb.com/">GlareDB</a>'s experience along with more <a href="https://github.com/apache/datafusion/issues/13661">extensive prerelease testing</a>. </p>
+<h2>Improved Documentation 📚</h2>
+<p>We continue to improve the documentation to make it easier to get started using DataFusion.
+During the last 6 months two projects were initiated to migrate the function documentation
+from strictly static markdown files. First, <a href="https://github.com/apache/datafusion/pull/12668">@Omega359</a> to allow function
+documentation to be generated from code and <a href="https://github.com/jonathanc-n">@jonathanc-n</a> and others helped with the migration,
+then <a href="https://github.com/comphead">@comphead</a> lead a project to <a href="https://github.com/apache/datafusion/pull/12822">create a doc macro</a> to allow for an even easier way to write
+function documentation. A special thanks to <a href="https://github.com/Chen-Yuan-Lai">@Chen-Yuan-Lai</a> for migrating many functions to
+the new syntax.</p>
+<p>Additionally, the <a href="https://github.com/apache/datafusion/pull/13877">examples</a> were <a href="https://github.com/apache/datafusion/pull/13905">refactored</a> and <a href="https://github.com/apache/datafusion/pull/13950">cleaned up</a> to improve their usefulness.</p>
+<h2>New Features ✨</h2>
+<p>There are too many new features in the last 6 months to list them all, but here
+are some highlights:</p>
+<h3>Functions</h3>
+<ul>
+<li>Uniform Window Functions: <code>BuiltInWindowFunctions</code> was removed and all now use UDFs (<a href="https://github.com/jcsherin">@jcsherin</a>)</li>
+<li>Uniform Aggregate Functions: <code>BuiltInAggregateFunctions</code> was removed and all now use UDFs</li>
+<li>As mentioned above function documentation was extracted from the markdown files</li>
+<li>Some new functions and sql support were added including '<a href="https://github.com/apache/datafusion/pull/13799">show functions</a>', '<a href="https://github.com/apache/datafusion/pull/11347">to_local_time</a>',
+ '<a href="https://github.com/apache/datafusion/pull/12970">regexp_count</a>', '<a href="https://github.com/apache/datafusion/pull/11969">map_extract</a>', '<a href="https://github.com/apache/datafusion/pull/12211">array_distance</a>', '<a href="https://github.com/apache/datafusion/pull/12329">array_any_value</a>', '<a href="https://github.com/apache/datafusion/pull/12474">greatest</a>',
+ '<a href="https://github.com/apache/datafusion/pull/13786">least</a>', '<a href="https://github.com/apache/datafusion/pull/14217">arrays_overlap</a>'</li>
+</ul>
+<h3>FFI</h3>
+<ul>
+<li>Foreign Function Interface work has started. This should allow for
+ <a href="https://github.com/apache/datafusion/pull/12920">using table providers</a> across languages and versions of DataFusion. This
+ is especially pertinent for integration with <a href="https://delta-io.github.io/delta-rs/">delta-rs</a> and other table formats.</li>
+</ul>
+<h3>Materialized Views</h3>
+<ul>
+<li><a href="https://github.com/suremarc">@suremarc</a> has added a <a href="https://github.com/datafusion-contrib/datafusion-materialized-views">materialized view implementation</a> in datafusion-contrib 🚀</li>
+</ul>
+<h3>Substrait</h3>
+<ul>
+<li>A lot of work was put into improving and enhancing substrait support (<a href="https://github.com/Blizzara">@Blizzara</a>, <a href="https://github.com/westonpace">@westonpace</a>, <a href="https://github.com/tokoko">@tokoko</a>, <a href="https://github.com/vbarua">@vbarua</a>, <a href="https://github.com/LatrecheYasser">@LatrecheYasser</a>, <a href="https://github.com/notfilippo">@notfilippo</a> and others)</li>
+</ul>
+<h2>Looking Ahead: The Next Six Months ðŸ”</h2>
+<p>One of the long term goals of <a href="https://github.com/alamb">@alamb</a>, DataFusion's PMC chair, has been to have
+<a href="https://www.influxdata.com/blog/datafusion-2025-influxdb/">1000 DataFusion based projects</a>. This may be the year that happens!</p>
+<p>The community has been <a href="https://github.com/apache/datafusion/issues/14580">discussing what we will work on in the next six months</a>.
+Some major initiatives are likely to be:</p>
+<ol>
+<li><em>Performance</em>: A <a href="https://github.com/apache/datafusion/issues/14482">number of items have been identified</a> as areas that could use additional work</li>
+<li><em>Memory usage</em>: Tracking and improving memory usage, statistics and spilling to disk </li>
+<li><em><a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> (GSOC)</em>: DataFusion is hopefully selected as a project and we start accepting and supporting student projects </li>
+<li><em>FFI</em>: Extending the FFI implementation to support to all types of UDF's and SessionContext</li>
+<li><em>Spark Functions</em>: A <a href="https://github.com/apache/datafusion/issues/5600">proposal has been made to add a crate</a> covering spark compatible builtin functions </li>
+</ol>
+<h2>How to Get Involved</h2>
+<p>DataFusion is not a project built or driven by a single person, company, or
+foundation. Rather, our community of users and contributors work together to
+build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us we would love to have you. You can try out
+DataFusion on some of your own data and projects and let us know how it goes,
+contribute suggestions, documentation, bug reports, or a PR with documentation,
+tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
+can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p>
+ </div>
+ </div>
+ </div>
+ <!-- footer -->
+ <div class="row">
+ <div class="large-12 medium-12 columns">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2025, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/2025/03/11/ordering-analysis/index.html b/blog/2025/03/11/ordering-analysis/index.html
new file mode 100644
index 0000000..c829a2c
--- /dev/null
+++ b/blog/2025/03/11/ordering-analysis/index.html
@@ -0,0 +1,431 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Using Ordering for Better Plans in Apache DataFusion - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+<!-- page contents -->
+<div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+ <h1>
+ Using Ordering for Better Plans in Apache DataFusion
+ </h1>
+ <p>Posted on: Tue 11 March 2025 by Mustafa Akur, Andrew Lamb</p>
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
+</blockquote>
+<p>There are various use cases where this type of analysis can be useful such as the following examples.</p>
+<h3>Removing Unnecessary Sorts</h3>
+<p>Imagine a user wants to execute the following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">log_line</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="k">time</span> <span class="k">ASC</span> <span class="k">limit</span> <span class="mi">10</span>
+</code></pre></div>
+<p>If we don't know anything about the <code>telemetry</code> table we need to sort it by <code>time ASC</code> and then retrieve the first 10 rows to get the correct result. However, if the table is already ordered by <code>time ASC</code>, we can simply retrieve the first 10 rows. This approach executes much faster and uses less memory compared to resorting the entire table, even when the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/struct.TopK.html">TopK</a> ope [...]
+<p>In order to avoid the sort the query optimizer must determine the data is already sorted. For simple queries the analysis is straightforward however it gets complicated fast. For example, what if your data is sorted by <code>[hostname, time ASC]</code> and your query is</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">log_line</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">WHERE</span> <span class="n">hostname</span> <span class="o">=</span> <span class="s1">'app.example.com'</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="k">time</span> <span class="k">ASC</span><span class="p">;</span>
+</code></pre></div>
+<p>In this case a sort still isn't needed, but the analysis must reason about the sortedness of the stream when it knows <code>hostname</code> has a single value.</p>
+<h3>Optimized Operator Implementations</h3>
+<p>As another use case, some operators can utilize the ordering information to change its underlying algorithm to execute more efficiently. Consider the following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">log_line</span><span class="p">)</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="n">hostname</span><span class="p">;</span>
+</code></pre></div>
+<p>Most analytic systems, including DataFusion, by default implement such a query using a hash table keyed on values of <code>hostname</code> to store the counts. However, if the <code>telemetry</code> table is sorted by <code>hostname</code>, there are much more efficient algorithms for grouping on <code>hostname</code> values than hashing every value and storing it in memory. However, the more efficient algorithm can only be used when the input is sorted correctly. To see this in prac [...]
+<h3>Streaming-Friendly Execution</h3>
+<p>Stream processing aims to produce results immediately as they become available ensuring minimal latency for real-time workloads. However, some operators need to consume all input data before producing any output. Consider the <code>Sort</code> operation: before it can start generating output, the algorithm must first process all input data. As a result, data flow halts whenever such an operator is encountered until all input is consumed. When a physical query plan contains such an ope [...]
+<p>For a query to be executed in a streaming fashion we need to satisfy 2 conditions:</p>
+<p><strong>Logically Streamable</strong><br/>
+It should be possible to generate what user wants in streaming fashion. Consider following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">SUM</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
+<span class="k">FROM</span> <span class="n">orders</span>
+</code></pre></div>
+<p>Here, the user wants to compute the sum of all amounts in the orders table. By the nature of the query this requires scanning the entire table to generate a result making it impossible to execute in a streaming fashion.</p>
+<p><strong>Streaming Aware Planner</strong><br/>
+Being logically streamable does not guarantee that a query will execute in a streaming fashion. SQL is a declarative language, meaning it specifies 'WHAT' user wants. It is up to the planner 'HOW' to generate the result. In most cases there are many ways to compute the correct result for a given query. The query planner is responsible for choosing "a way" (ideally the best<sup id="optimal1"><a href="#optimal">*</a></sup> one) among the all alternatives to generate what user asks for. If [...]
+<h2>Analysis</h2>
+<p>Let's start by creating an example table that we will refer throughout the post. This table models the input data of an operator for the analysis:</p>
+<h3>Example Virtual Table</h3>
+<style>
+ table {
+ border-collapse: collapse;
+ width: 80%;
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+ }
+ th, td {
+ padding: 12px 16px;
+ text-align: left;
+ border-bottom: 1px solid #e0e0e0;
+ }
+ th {
+ background-color: #f9f9f9;
+ font-weight: 600;
+ }
+ tr:hover {
+ background-color: #f1f1f1;
+ }
+</style>
+<table>
+<tr>
+<th>amount</th> <th>price</th> <th>hostname</th><th>currency</th><th>time_bin</th> <th>time</th> <th>price_cloned</th> <th>time_cloned</th>
+</tr>
+<tr>
+<td>12</td> <td>25</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:01:30</td> <td>25</td> <td>08:01:30</td>
+</tr>
+<tr>
+<td>12</td> <td>26</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:11:30</td> <td>26</td> <td>08:11:30</td>
+</tr>
+<tr>
+<td>15</td> <td>30</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:41:30</td> <td>30</td> <td>08:41:30</td>
+</tr>
+<tr>
+<td>15</td> <td>32</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:55:15</td> <td>32</td> <td>08:55:15</td>
+</tr>
+<tr>
+<td>15</td> <td>35</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:10:23</td> <td>35</td> <td>09:10:23</td>
+</tr>
+<tr>
+<td>20</td> <td>18</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:20:33</td> <td>18</td> <td>09:20:33</td>
+</tr>
+<tr>
+<td>20</td> <td>22</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:40:15</td> <td>22</td> <td>09:40:15</td>
+</tr>
+</table>
+<p><br/></p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>How can a table have multiple orderings?</strong> At first glance it may seem counterintuitive for a table to have more than one valid ordering. However, during query execution such scenarios can arise.
+
+For example consider the following query:
+
+
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">time</span><span class="p">,</span> <span class="n">date_bin</span><span class="p">(</span><span class="s1">'1 hour'</span><span class="p">,</span> <span class="k">time</span><span class="p">,</span> <span class="s1">'1970-01-01'</span><span class="p">)</span> <span class="k">as</span> <span class="n">time_bin</span>
+<span class="k">FROM</span> <span class="k">table</span><span class="p">;</span>
+</code></pre></div>
+
+
+If we know that the table is ordered by <code>time ASC</code> we can infer that <code>time_bin ASC</code> is also a valid ordering. This is because the <code>date_bin</code> function is monotonic, meaning it preserves the order of its input.
+
+DataFusion leverages these functional dependencies to infer new orderings as data flows through different query operators. For details on the implementation see the <a ,="" href="https://github.com/apache/datafusion/blob/main/datafusion/common/src/functional_dependencies.rs" target="_blank">source</a> code.
+</blockquote>
+<p>By inspection, you can see this table is sorted by the <code>amount</code> column, but It is also sorted by <code>time</code> and <code>time_bin</code> as well as the compound <code>(time_bin, amount)</code> and many other variations. While this example is an extreme case, real-world data often has multiple sort orders. </p>
+<p>A naive approach for analyzing whether the ordering requirement of an operator is satisfied by its input would be: </p>
+<ul>
+<li>Store all the valid ordering expressions that the tables satisfies </li>
+<li>Check whether the ordering requirement by the operator is among valid orderings. </li>
+</ul>
+<p>This naive algorithm works and correct. However, listing all valid orderings can be quite lengthy and is of exponential complexity as the number of orderings grows. For the example table here is a (small) subset of the valid orderings:</p>
+<p><code>[amount ASC]</code><br/>
+<code>[amount ASC, price ASC]</code><br/>
+<code>[amount ASC, price_cloned ASC]</code><br/>
+<code>[hostname ASC, amount ASC, price_cloned ASC]</code><br/>
+<code>[amount ASC, hostname ASC, price_cloned ASC]</code><br/>
+<code>[amount ASC, price_cloned ASC, hostname ASC]</code><br/>
+.<br/>
+.<br/>
+. </p>
+<p>As can be seen from the listing above storing all valid orderings is wasteful and contains significant redundancy. Here are some observations which suggest that we can do much better:</p>
+<ul>
+<li>
+<p>Storing a prefix of another valid ordering is redundant. If the table satisfies the lexicographic ordering<sup id="fn1"><a href="#footnote1">1</a></sup>: <code>[amount ASC, price ASC]</code>, it already satisfies ordering <code>[amount ASC]</code> trivially. Hence, once we store <code>[amount ASC, price ASC]</code> storing <code>[amount ASC]</code> is redundant.</p>
+</li>
+<li>
+<p>Using all columns that are equal to each other in the listings is redundant. If we know the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by <code>[amount ASC, price_cloned ASC]</code> since <code>price</code> and <code>price_cloned</code> are copy of each other. It is enough to use just one expression among the expressions that exact copy of each other.</p>
+</li>
+<li>
+<p>Constant expressions can be inserted anywhere in a valid ordering with an arbitrary direction (e.g. <code>ASC</code>, <code>DESC</code>). Hence, if the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by: <br/>
+<code>[hostname ASC, amount ASC, price ASC]</code>,<br/>
+<code>[hostname DESC, amount ASC, price ASC]</code>,<br/>
+<code>[amount ASC, hostname ASC, price ASC]</code>,<br/>
+ .<br/>
+ . </p>
+</li>
+</ul>
+<p>This is clearly redundant. For this reason, it is better to avoid explicitly encoding constant expressions in valid sort orders.</p>
+<p>In summary,</p>
+<ul>
+<li>We should store only the longest lexicographic ordering (shouldn't use any prefix of it)</li>
+<li>Using expressions that are exact copies of each other is redundant.</li>
+<li>Ordering expressions shouldn't contain any constant expression.</li>
+</ul>
+<h2>Key Concepts for Analyzing Orderings</h2>
+<p>To solve the shortcomings above DataFusion needs to track of following properties for the table:</p>
+<ul>
+<li>Constant Expressions </li>
+<li>Equivalent Expression Groups (will be explained shortly)</li>
+<li>Succinct Valid Orderings (will be explained shortly)</li>
+</ul>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> These properties are implemented in the <code>EquivalenceProperties</code> structure in <code>DataFusion</code>, please see the <a href="https://github.com/apache/datafusion/blob/f47ea73b87eec4af044f9b9923baf042682615b2/datafusion/physical-expr/src/equivalence/properties/mod.rs#L134" target="_blank">source</a> for more details<br/>
+</blockquote>
+<p>These properties allow us to analyze whether the ordering requirement is satisfied by the data already.</p>
+<h3>1. Constant Expressions</h3>
+<p>Constant expressions are those where each row in the expression has the same value across all rows. Although constant expressions may seem odd in a table they can arise after operations like <code>Filter</code> or <code>Join</code> occur. </p>
+<p>For instance in the example table:</p>
+<ul>
+<li>Columns <code>hostname</code> and <code>currency</code> are constant because every row in the table has the same value (<code>'app.example.com'</code> for <code>hostname</code>, and <code>'USD'</code> for <code>currency</code>) for these columns.</li>
+</ul>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> Constant expressions can arise during query execution. For example, in following query:<br/>
+<code>SELECT hostname FROM logs</code><br/><code>WHERE hostname='app.example.com'</code> <br/>
+ after filtering is done, for subsequent operators the <code>hostname</code> column will be constant.
+</blockquote>
+<h3>2. Equivalent Expression Groups</h3>
+<p>Equivalent expression groups are expressions that always hold the same value across rows. These expressions can be thought of as clones of each other and may arise from operations like <code>Filter</code>, <code>Join</code>, or <code>Projection</code>.</p>
+<p>In the example table, the expressions <code>price</code> and <code>price_cloned</code> form one equivalence group, and <code>time</code> and <code>time_cloned</code> form another equivalence group.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> Equivalent expression groups can arise during the query execution. For example, in the following query:<br/>
+<code>SELECT time, time as time_cloned FROM logs</code> <br/>
+ after the projection is done, for subsequent operators <code>time</code> and <code>time_cloned</code> will form an equivalence group. As another example, in the following query:<br/>
+<code>SELECT employees.id, employees.name, departments.department_name</code>
+<code>FROM employees</code>
+<code>JOIN departments ON employees.department_id = departments.id;</code> <br/>
+after joining, <code>employees.department_id</code> and <code>departments.id</code> will form an equivalence group.
+</blockquote>
+<h3>3. Succinct Encoding of Valid Orderings</h3>
+<p>Valid orderings are the orderings that the table already satisfies. However, naively listing them requires exponential space as the number of columns grows as discussed before. Instead, we list all valid orderings after following constraints are applied:</p>
+<ul>
+<li>Do not use any constant expressions in the valid ordering construction</li>
+<li>Use only one entry (by convention the first entry) in the equivalent expression group.</li>
+<li>Lexicographic ordering shouldn't contain any leading ordering<sup id="fn2"><a href="#footnote2">2</a></sup>except the first position <sup id="fn3"><a href="#footnote3">3</a></sup>.</li>
+<li>Do not use any prefix of a valid lexicographic ordering<sup id="fn4"><a href="#footnote4">4</a></sup>.</li>
+</ul>
+<p>After applying the first and second constraint, the example table simplifies to </p>
+<style>
+ table {
+ border-collapse: collapse;
+ width: 80%;
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+ }
+ th, td {
+ padding: 12px 16px;
+ text-align: left;
+ border-bottom: 1px solid #e0e0e0;
+ }
+ th {
+ background-color: #f9f9f9;
+ font-weight: 600;
+ }
+ tr:hover {
+ background-color: #f1f1f1;
+ }
+</style>
+<table>
+<tr>
+<th>amount</th> <th>price</th><th>time_bin</th> <th>time</th>
+</tr>
+<tr>
+<td>12</td> <td>25</td><td>08:00:00</td> <td>08:01:30</td>
+</tr>
+<tr>
+<td>12</td> <td>26</td><td>08:00:00</td> <td>08:11:30</td>
+</tr>
+<tr>
+<td>15</td> <td>30</td><td>08:00:00</td> <td>08:41:30</td>
+</tr>
+<tr>
+<td>15</td> <td>32</td><td>08:00:00</td> <td>08:55:15</td>
+</tr>
+<tr>
+<td>15</td> <td>35</td><td>09:00:00</td> <td>09:10:23</td>
+</tr>
+<tr>
+<td>20</td> <td>18</td><td>09:00:00</td> <td>09:20:33</td>
+</tr>
+<tr>
+<td>20</td> <td>22</td><td>09:00:00</td> <td>09:40:15</td>
+</tr>
+</table>
+<p><br/>
+Following third and fourth constraints for the simplified table, the succinct valid orderings are:<br/>
+<code>[amount ASC, price ASC]</code>, <br/>
+<code>[time_bin ASC]</code>,<br/>
+<code>[time ASC]</code> </p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<p><strong>How can DataFusion find orderings?</strong></p>
+DataFusion's <code>CREATE EXTERNAL TABLE</code> has a <code>WITH ORDER</code> clause (see <a href="https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table">docs</a>) to specify the known orderings of the table during table creation. For example the following query:<br/>
+<pre><code>
+CREATE EXTERNAL TABLE source (
+ amount INT NOT NULL,
+ price DOUBLE NOT NULL,
+ time TIMESTAMP NOT NULL,
+ ...
+)
+STORED AS CSV
+WITH ORDER (time ASC)
+WITH ORDER (amount ASC, price ASC)
+LOCATION '/path/to/FILE_NAME.csv'
+OPTIONS ('has_header' 'true');
+</code></pre>
+communicates that <code>source</code> table has the orderings: <code>[time ASC]</code> and <code>[amount ASC, price ASC]</code>.<br/>
+When orderings are communicated from the source, DataFusion tracks the orderings through each operator while optimizing the plan.<br/>
+<ul>
+<li>add new orderings (such as when "date_bin" function is applied to the "time" column)</li>
+<li>Remove orderings, if operation doesn't preserve the ordering of the data at its input</li>
+<li>Update equivalent groups</li>
+<li>Update constant expressions</li>
+</ul>
+
+Figure 1 shows an example how DataFusion generates an efficient plan for the query:
+<pre><code>
+SELECT
+ row_number() OVER (ORDER BY time) as rn,
+ time
+FROM events
+ORDER BY rn, time
+</code></pre>
+using the orderings of the query intermediates.<br/>
+<br/>
+<figure>
+<img alt="Window Query Datafusion Optimization" class="img-responsive" src="/blog/images/ordering_analysis/query_window_plan.png" width="80%"/>
+<figcaption><strong>Figure 1:</strong> DataFusion analyzes orderings of the sources and query intermediates to generate efficient plans</figcaption>
+</figure>
+</blockquote>
+<h3>Table Properties</h3>
+<p>In summary, for the example table, the following properties correctly describe the sort properties:</p>
+<ul>
+<li><strong>Constant Expressions</strong> = <code>hostname, currency</code> </li>
+<li><strong>Equivalent Expression Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code> </li>
+<li><strong>Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code> </li>
+</ul>
+<h3>Algorithm for Analyzing Ordering Requirements</h3>
+<p>After deriving these properties for the data, following algorithm can be used to check whether an ordering requirement is satisfied by the table:</p>
+<ol>
+<li><strong>Prune constant expressions</strong>: Remove any constant expressions from the ordering requirement.</li>
+<li><strong>Normalize the requirement</strong>: Replace each expression in the ordering requirement with the first entry from its equivalence group.</li>
+<li><strong>De-duplicate expressions</strong>: If an expression appears more than once, remove duplicates, keeping only the first occurrence.</li>
+<li><strong>Match leading orderings</strong>: Check whether the leading ordering requirement<sup id="fn5"><a href="#footnote5">5</a></sup> matches the leading valid orderings<sup id="fn6"><a href="#footnote6">6</a></sup> of table. If so:<ul>
+<li>Remove the leading ordering requirement from the ordering requirement </li>
+<li>Remove the matching leading valid ordering from the valid orderings of table. </li>
+</ul>
+</li>
+<li><strong>Iterate through the remaining expressions</strong>: Go back to step 4 until ordering requirement is empty or leading ordering requirement is not found among the leading valid orderings of table.</li>
+</ol>
+<p>If, at the end of the procedure above, the ordering requirement is an empty list, we can conclude that the requirement is satisfied by the table.</p>
+<h3>Example Walkthrough</h3>
+<p>Let's say the user provided a query such as the following:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="k">table</span>
+<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">hostname</span> <span class="k">DESC</span><span class="p">,</span> <span class="n">amount</span> <span class="k">ASC</span><span class="p">,</span> <span class="n">time_bin</span> <span class="k">ASC</span><span class="p">,</span> <span class="n">price_cloned</span> <span class="k">ASC</span><span class="p">,</span> <span class="k">time</span> <span class="k">ASC</span><span class="p">,</span> <span class="n">currenc [...]
+</code></pre></div>
+<p>And the input has the same properties explained above</p>
+<ul>
+<li><strong>Constant Expressions</strong> = <code>hostname, currency</code></li>
+<li><strong>Equivalent Expressions Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code></li>
+<li><strong>Succinct Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code></li>
+</ul>
+<p>In order to remove a sort the optimizer must check if the ordering requirement <code>[hostname DESC, amount ASC, time_bin ASC, price_cloned ASC, time ASC, currency ASC, price DESC]</code> is satisfied by the properties.</p>
+<h3>Algorithm Steps</h3>
+<ol>
+<li>
+<p><strong>Prune constant expressions</strong>:<br/>
+ Remove <code>hostname</code> and <code>currency</code> from the requirement. The requirement becomes:<br/>
+<code>[amount ASC, time_bin ASC, price_cloned ASC, time ASC, price DESC]</code>.</p>
+</li>
+<li>
+<p><strong>Normalize using equivalent groups</strong>:<br/>
+ Replace <code>price_cloned</code> with <code>price</code> and <code>time_cloned</code> with <code>time</code>. The requirement becomes:<br/>
+<code>[amount ASC, time_bin ASC, price ASC, time ASC, price DESC]</code>.</p>
+</li>
+<li>
+<p><strong>De-duplicate expressions</strong>:<br/>
+ Since <code>price</code> appears twice, we simplify the requirement to:<br/>
+<code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> (keeping the first occurrence from the left side).</p>
+</li>
+<li>
+<p><strong>Match leading orderings</strong>:<br/>
+ Check if leading ordering requirement <code>amount ASC</code> is among the leading valid orderings: <code>amount ASC, time_bin ASC, time ASC</code>. Since this is the case, we remove <code>amount ASC</code> from both the ordering requirement and the valid orderings of the table.</p>
+</li>
+<li><strong>Iterate through the remaining expressions</strong>:
+Now, the problem is converted from<br/>
+<em>"whether the requirement: <code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
+into<br/>
+<em>"whether the requirement: <code>[time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
+We go back to step 4 until the ordering requirement list is exhausted or its length no longer decreases.</li>
+</ol>
+<p>At the end of stages above we end up with an empty ordering requirement list. Given this, we can conclude that the table satisfies the ordering requirement and thus no sort is required. </p>
+<h2>Conclusion</h2>
+<p>In this post, we described the conditions under which an ordering requirement is satisfied based on the properties of a table. We introduced key concepts such as constant expressions, equivalence groups, and valid orderings, and used them to determine whether a given ordering requirement are satisfied by an input table.</p>
+<p>This analysis plays a crucial role in:</p>
+<ul>
+<li>Choosing more efficient algorithm variants</li>
+<li>Generating streaming-friendly plans</li>
+</ul>
+<p>The <code>DataFusion</code> query engine employs this analysis (and many more) during its planning stage to ensure correct and efficient query execution. We <a href="https://datafusion.apache.org/contributor-guide/index.html">welcome you</a> to come and join the project.</p>
+<h2>Appendix</h2>
+<!--
+<p id="footnote1"><sup>[1]</sup>The ordering requirement refers to the condition that input data must be sorted in a certain way for a specific operator to function as intended.</p>
+-->
+<p id="footnote1"><sup>[1]</sup>Lexicographic order is a way of ordering sequences (like strings, list of expressions) based on the order of their components, similar to how words are ordered in a dictionary. It compares each element of the sequences one by one, from left to right.</p>
+<p id="footnote2"><sup>[2]</sup>Leading ordering is the first ordering in a lexicographic ordering list. As an example, for the ordering: <code>[amount ASC, price ASC]</code>, leading ordering will be: <code>amount ASC</code>. </p>
+<p id="footnote3"><sup>[3]</sup>This means that, if we know that <code>[amount ASC]</code> and <code>[time ASC]</code> are both valid orderings for the table. We shouldn't enlist <code>[amount ASC, time ASC]</code> or <code>[time ASC, amount ASC]</code> as valid orderings. These orderings can be deduced if we know that table satisfies the ordering <code>[amount ASC]</code> and <code>[time ASC]</code>.</p>
+<p id="footnote4"><sup>[4]</sup>This means that, if ordering <code>[amount ASC, price ASC]</code> is a valid ordering for the table. We shouldn't enlist <code>[amount ASC]</code> as valid ordering. Validity of it can be deduced from the ordering: <code>[amount ASC, price ASC]</code>
+<p id="footnote5"><sup>[5]</sup>Leading ordering requirement is the first ordering requirement in the list of lexicographic ordering requirement expression. As an example for the requirement: <code>[amount ASC, time_bin ASC, prices ASC, time ASC]</code>, leading ordering requirement is: <code>amount ASC</code>.</p>
+<p id="footnote6"><sup>[6]</sup>Leading valid orderings are the first ordering for each valid ordering list in the table. As an example, for the valid orderings: <code>[amount ASC, prices ASC], [time_bin ASC], [time ASC]</code>, leading valid orderings will be: <code>amount ASC, time_bin ASC, time ASC</code>. </p>
+<p id="optimal"><sup>*</sup>Best depends on the use case, <code>DataFusion</code> has many various flags to communicate what user thinks the best plan is (e.g. streamable, fastest, lowest memory, etc.). See <a href="https://datafusion.apache.org/user-guide/configs.html" target="_blank">configurations</a> for detail.</p></p>
+ </div>
+ </div>
+ </div>
+ <!-- footer -->
+ <div class="row">
+ <div class="large-12 medium-12 columns">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2025, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/2025/03/20/datafusion-comet-0.7.0/index.html b/blog/2025/03/20/datafusion-comet-0.7.0/index.html
new file mode 100644
index 0000000..f3949d7
--- /dev/null
+++ b/blog/2025/03/20/datafusion-comet-0.7.0/index.html
@@ -0,0 +1,139 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion Comet 0.7.0 Release - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+<!-- page contents -->
+<div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+ <h1>
+ Apache DataFusion Comet 0.7.0 Release
+ </h1>
+ <p>Posted on: Thu 20 March 2025 by pmc</p>
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
+expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
+to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
+<p>This release covers approximately four weeks of development work and is the result of merging 46 PRs from 11
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.7.0.md">change log</a> for more information.</p>
+<h2>Release Highlights</h2>
+<h3>Performance</h3>
+<p>Comet 0.7.0 has improved performance compared to the previous release due to improvements in the native shuffle
+implementation and performance improvements in DataFusion 46.</p>
+<p>For single-node TPC-H at 100 GB, Comet now delivers a <strong>2.2x speedup</strong> compared to Spark using the same CPU and RAM. Even
+with <strong>half the resources</strong>, Comet still provides a measurable performance improvement.</p>
+<p><img alt="Chart showing TPC-H benchmark results for Comet 0.7.0" class="img-responsive" src="/blog/images/comet-0.7.0/performance.png" width="100%"/></p>
+<p><em>These benchmarks were performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and data
+stored locally in Parquet format on NVMe storage. Spark was running in Kubernetes with hard memory limits.</em></p>
+<h2>Shuffle Improvements</h2>
+<p>There are several improvements to shuffle in this release:</p>
+<ul>
+<li>When running in off-heap mode (which is the recommended approach), Comet was using the wrong memory allocator
+ implementation for some types of shuffle operation, which could result in OOM rather than spilling to disk.</li>
+<li>The number of spill files is drastically reduced. In previous releases, each instance of ShuffleMapTask could
+ potentially create a new spill file for each output partition each time that spill was invoked. Comet now creates
+ a maximum of one spill file per output partition per instance of ShuffleMapTask, which is appended to in subsequent
+ spills.</li>
+<li>There was a flaw in the memory accounting which resulted in Comet requesting approximately twice the amount of
+ memory that was needed, resulting in premature spilling. This is now resolved.</li>
+<li>The metric for number of spilled bytes is now accurate. It was previously reporting invalid information.</li>
+</ul>
+<h2>Improved Hash Join Performance</h2>
+<p>When using the <code>spark.comet.exec.replaceSortMergeJoin</code> setting to replace sort-merge joins with hash joins, Comet
+will now do a better job of picking the optimal build side (thanks to [@hayman42] for suggesting this, and thanks to the
+[Apache Gluten(incubating)] project for the inspiration in implementing this feature).</p>
+<p><a href="https://github.com/hayman42">@hayman42</a></p>
+<h2>Experimental Support for DataFusion’s DataSourceExec</h2>
+<p>It is now possible to configure Comet to use DataFusion’s <code>DataSourceExec</code> instead of Comet’s current Parquet reader.
+Support should still be considered experimental, but most of Comet’s unit tests are now passing with the new reader.
+Known issues include handling of <code>INT96</code> timestamps and negative bytes and shorts.</p>
+<p>To enable DataFusion’s <code>DataSourceExec</code>, either set <code>spark.comet.scan.impl=native_datafusion</code> or set the environment
+variable <code>COMET_PARQUET_SCAN_IMPL=native_datafusion</code>.</p>
+<h2>Complex Type Support</h2>
+<p>With DataFusion’s <code>DataSourceExec</code> enabled, there is now some early support for reading structs from Parquet. This is
+largely untested and we would welcome additional testing from the community to help determine what is and isn’t working,
+as well as contributions to improve support for structs and other complex types. The tracking issue is
+https://github.com/apache/datafusion-comet/issues/1043.</p>
+<h2>Updates to supported Spark versions</h2>
+<ul>
+<li>Comet 0.7.0 is now tested against Spark 3.5.4 rather than 3.5.1</li>
+<li>This will be the last Comet release to support Spark 3.3.x</li>
+</ul>
+<h2>Improved Tuning Guide</h2>
+<p>The <a href="https://datafusion.apache.org/comet/user-guide/tuning.html">Comet Tuning Guide</a> has been improved and now provides guidance on determining how much memory to allocate to
+Comet.</p>
+<h2>Getting Involved</h2>
+<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
+project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
+<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
+performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
+Comet.</p>
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
+ </div>
+ </div>
+ </div>
+ <!-- footer -->
+ <div class="row">
+ <div class="large-12 medium-12 columns">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2025, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/2025/03/20/parquet-pruning/index.html b/blog/2025/03/20/parquet-pruning/index.html
new file mode 100644
index 0000000..1953076
--- /dev/null
+++ b/blog/2025/03/20/parquet-pruning/index.html
@@ -0,0 +1,126 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Parquet Pruning in DataFusion: Read Only What Matters - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+<!-- page contents -->
+<div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+ <h1>
+ Parquet Pruning in DataFusion: Read Only What Matters
+ </h1>
+ <p>Posted on: Thu 20 March 2025 by Xiangpeng Hao</p>
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> implements advanced Parquet pruning techniques to effectively read only the data that matters for a given query.</p>
+<p>Achieving high performance adds complexity.
+This post provides an overview of the techniques used in DataFusion to selectively read Parquet files.</p>
+<h3>The pipeline</h3>
+<p>The diagram below illustrates the <a href="https://docs.rs/datafusion/46.0.0/datafusion/datasource/physical_plan/parquet/source/struct.ParquetSource.html">Parquet reading pipeline</a> in DataFusion, highlighting how data flows through various pruning stages before being converted to Arrow format:</p>
+<p><img alt="Parquet pruning pipeline in DataFusion" class="img-responsive" src="/blog/images/parquet-pruning/read-parquet.jpg" width="100%"/></p>
+<h4>Background: Parquet file structure</h4>
+<p>As shown in the figure above, each Parquet file has multiple row groups. Each row group contains a set of columns, and each column contains a set of pages.</p>
+<p>Pages are the smallest units of data in Parquet files and typically contain compressed and encoded values for a specific column. This hierarchical structure enables efficient columnar access and forms the foundation for the pruning techniques we'll discuss.</p>
+<p>Check out <a href="https://www.influxdata.com/blog/querying-parquet-millisecond-latency/">Querying Parquet with Millisecond Latency</a> for more details on the Parquet file structure.</p>
+<h4>1. Read metadata</h4>
+<p>DataFusion first reads the <a href="https://parquet.apache.org/docs/file-format/metadata/">Parquet metadata</a> to understand the data in the file.
+Metadata often includes data schema, the exact location of each row group and column chunk, and their corresponding statistics (e.g., min/max values).
+It also optionally includes <a href="https://parquet.apache.org/docs/file-format/pageindex/">page-level stats</a> and <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>.
+This information is used to prune the file before reading the actual data.</p>
+<p><a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L118">Fetching metadata</a> requires up to two network requests: one to read the footer size from the end of the file, and another to read the footer itself. </p>
+<p><a href="https://www.influxdata.com/blog/how-good-parquet-wide-tables/">Decoding metadata</a> is generally fast since it only requires parsing a small amount of data. However, for tables with hundreds or thousands of columns, the metadata can become quite large and decoding it can become a bottleneck. This is particularly noticeable when scanning many small files.</p>
+<p>Reading metadata is latency-critical, so DataFusion allows users to cache metadata through the <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L39">ParquetFileReaderFactory</a> trait.</p>
+<h4>2. Prune by projection</h4>
+<p>The simplest yet perhaps most effective pruning is to read only the columns that are needed.
+This is because queries usually don't select all columns, e.g., <code>SELECT a FROM table</code> only reads column <code>a</code>.
+As a <strong>columnar</strong> format, Parquet allows DataFusion to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/mod.rs#L778">only read</a> the <strong>columns</strong> that are needed.</p>
+<p>This projection pruning happens at the column level and can dramatically reduce I/O when working with wide tables where queries typically access only a small subset of columns.</p>
+<h4>3. Prune by row group stats and Bloom filters</h4>
+<p>Each row group has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/physical_optimizer/pruning.rs#L81">basic stats</a> like min/max values for each column.
+DataFusion applies the query predicates to these stats to prune row groups, e.g., <code>SELECT * FROM table WHERE a > 10</code> will only read row groups where <code>a</code> has a max value greater than 10.</p>
+<p>Sometimes min/max stats are too simple to prune effectively, so Parquet also supports <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>. DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L202">uses Bloom filters when available</a>.</p>
+<p>Bloom filters are particularly effective for equality predicates (<code>WHERE a = 10</code>) and can significantly reduce the number of row groups that need to be read for point queries or queries with highly selective predicates.</p>
+<h4>4. Prune by page stats</h4>
+<p>Parquet optionally supports <a href="https://github.com/apache/parquet-format/blob/master/PageIndex.md">page-level stats</a> -- similar to row group stats but more fine-grained.
+DataFusion implements <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L219">page pruning</a> when the stats are present.</p>
+<p>Page-level pruning provides an additional layer of filtering after row group pruning. It allows DataFusion to skip individual pages within a row group, further reducing the amount of data that needs to be read and decoded.</p>
+<h4>5. Read from storage</h4>
+<p>Now we (hopefully) have pruned the Parquet file into small ranges of bytes, i.e., the <a href="https://github.com/apache/datafusion/blob/76a7789ace33ced54c973fa0d5fc9d1866e1bf19/datafusion/datasource-parquet/src/access_plan.rs#L86">Access Plan</a>.
+The last step is to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L103">make requests</a> to fetch those bytes and decode them into Arrow RecordBatch. </p>
+<h3>Preview of coming attractions: filter pushdown</h3>
+<p>So far we have discussed techniques that prune the Parquet file using only the metadata, i.e., before reading the actual data.</p>
+<p>Filter pushdown, also known as predicate pushdown or late materialization, is a technique that prunes data during scanning, with filters being generated and applied in the Parquet reader.</p>
+<p><img alt="Filter pushdown in DataFusion" class="img-responsive" src="/blog/images/parquet-pruning/filter-pushdown.jpg" width="100%"/></p>
+<p>Unlike metadata-based pruning which works at the row group or page level, filter pushdown operates at the row level, allowing DataFusion to filter out individual rows that don't match the query predicates during the decoding process.</p>
+<p>DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/row_filter.rs#L154">implements filter pushdown</a> but has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/common/src/config.rs#L382">not enabled it by default</a> due to <a href="https://github.com/apache/datafusion/issues/3463">some performance regressions</a>.</p>
+<p>We are working to remove the <a href="https://github.com/apache/arrow-rs/issues/5523#issuecomment-2429470872">remaining performance issues</a> and enable it by default, which we will discuss in the next blog post.</p>
+<h3>Conclusion</h3>
+<p>DataFusion employs a multi-step approach to Parquet pruning, from column projection to row group stats, page stats, and potentially row-level filtering.
+Each step may reduce the amount of data to be read and processed, significantly improving query performance.</p>
+ </div>
+ </div>
+ </div>
+ <!-- footer -->
+ <div class="row">
+ <div class="large-12 medium-12 columns">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2025, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/author/mustafa-akur-andrew-lamb.html b/blog/author/mustafa-akur-andrew-lamb.html
new file mode 100644
index 0000000..4787452
--- /dev/null
+++ b/blog/author/mustafa-akur-andrew-lamb.html
@@ -0,0 +1,110 @@
+ <!doctype html>
+ <html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> <link href="/blog/css/blog_index.css" rel="stylesheet">
+ </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+ <div>
+
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+ <div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+<div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+
+ <h3>Welcome to the Apache DataFusion Blog!</h3>
+ <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
+
+
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/03/11/ordering-analysis">Using Ordering for Better Plans in Apache DataFusion</a></h1>
+ <p>Posted on: Tue 11 March 2025 by Mustafa Akur, Andrew Lamb</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/03/11/ordering-analysis" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+
+ </div>
+ </div>
+</div> </div>
+ </div>
+ </div>
+
+ <!-- footer -->
+ <div class="row">
+ <div class="large-12 medium-12 columns">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2025, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </div>
+ </main>
+ </body>
+ </html>
diff --git a/blog/author/pmc.html b/blog/author/pmc.html
index 1c09344..dc365a8 100644
--- a/blog/author/pmc.html
+++ b/blog/author/pmc.html
@@ -47,6 +47,88 @@
<p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/03/20/datafusion-comet-0.7.0">Apache DataFusion Comet 0.7.0 Release</a></h1>
+ <p>Posted on: Thu 20 March 2025 by pmc</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/03/20/datafusion-comet-0.7.0" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/02/20/datafusion-45.0.0">Apache DataFusion 45.0.0 Released</a></h1>
+ <p>Posted on: Thu 20 February 2025 by pmc</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/02/20/datafusion-45.0.0" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
<!-- Post -->
<div class="row">
<div class="callout">
diff --git a/blog/author/xiangpeng-hao.html b/blog/author/xiangpeng-hao.html
new file mode 100644
index 0000000..f2303b8
--- /dev/null
+++ b/blog/author/xiangpeng-hao.html
@@ -0,0 +1,109 @@
+ <!doctype html>
+ <html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> <link href="/blog/css/blog_index.css" rel="stylesheet">
+ </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+ <div>
+
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+ <div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+<div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+
+ <h3>Welcome to the Apache DataFusion Blog!</h3>
+ <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
+
+
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/03/20/parquet-pruning">Parquet Pruning in DataFusion: Read Only What Matters</a></h1>
+ <p>Posted on: Thu 20 March 2025 by Xiangpeng Hao</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/03/20/parquet-pruning" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+
+ </div>
+ </div>
+</div> </div>
+ </div>
+ </div>
+
+ <!-- footer -->
+ <div class="row">
+ <div class="large-12 medium-12 columns">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2025, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </div>
+ </main>
+ </body>
+ </html>
diff --git a/blog/category/blog.html b/blog/category/blog.html
index 739cf1c..56f617b 100644
--- a/blog/category/blog.html
+++ b/blog/category/blog.html
@@ -47,6 +47,169 @@
<p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/03/20/datafusion-comet-0.7.0">Apache DataFusion Comet 0.7.0 Release</a></h1>
+ <p>Posted on: Thu 20 March 2025 by pmc</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/03/20/datafusion-comet-0.7.0" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/03/20/parquet-pruning">Parquet Pruning in DataFusion: Read Only What Matters</a></h1>
+ <p>Posted on: Thu 20 March 2025 by Xiangpeng Hao</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/03/20/parquet-pruning" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/03/11/ordering-analysis">Using Ordering for Better Plans in Apache DataFusion</a></h1>
+ <p>Posted on: Tue 11 March 2025 by Mustafa Akur, Andrew Lamb</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/03/11/ordering-analysis" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/02/20/datafusion-45.0.0">Apache DataFusion 45.0.0 Released</a></h1>
+ <p>Posted on: Thu 20 February 2025 by pmc</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/02/20/datafusion-45.0.0" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
<!-- Post -->
<div class="row">
<div class="callout">
diff --git a/blog/feed.xml b/blog/feed.xml
index d9f2c7b..fccf433 100644
--- a/blog/feed.xml
+++ b/blog/feed.xml
@@ -1,5 +1,92 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon, 17 Feb 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet 0.6.0 Release</title><link>https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0</link><description><!--
+<rss version="2.0"><channel><title>Apache DataFusion Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Thu, 20 Mar 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet 0.7.0 Release</title><link>https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 20 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-20:/blog/2025/03/20/datafusion-comet-0.7.0</guid><category>blog</category></item><item><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link>https://datafusion.apache.org/blog/2025/03/20/parquet-pruning</link><d [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao</dc:creator><pubDate>Thu, 20 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-20:/blog/2025/03/20/parquet-pruning</guid><category>blog</category></item><item><title>Using Ordering for Better Plans in Apache DataFusion</title><link>https://datafusion.apache.org/blog/202 [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Mustafa Akur, Andrew Lamb</dc:creator><pubDate>Tue, 11 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-11:/blog/2025/03/11/ordering-analysis</guid><category>blog</category></item><item><title>Apache DataFusion 45.0.0 Released</title><link>https: [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 20 Feb 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-02-20:/blog/2025/02/20/datafusion-45.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.6.0 Release</title><link>https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/all-en.atom.xml b/blog/feeds/all-en.atom.xml
index 974c2b8..9ab59a2 100644
--- a/blog/feeds/all-en.atom.xml
+++ b/blog/feeds/all-en.atom.xml
@@ -1,5 +1,770 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-02-17T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-20T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion Comet 0.7.0 Release</title><link href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0" rel [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
+expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
+to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
+<p>This release covers approximately four weeks of development work and is the result of merging 46 PRs from 11
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.7.0.md">change log</a> for more information.</p>
+<h2>Release Highlights</h2>
+<h3>Performance</h3>
+<p>Comet 0.7.0 has improved performance compared to the previous release due to improvements in the native shuffle
+implementation and performance improvements in DataFusion 46.</p>
+<p>For single-node TPC-H at 100 GB, Comet now delivers a <strong>2.2x speedup</strong> compared to Spark using the same CPU and RAM. Even
+with <strong>half the resources</strong>, Comet still provides a measurable performance improvement.</p>
+<p><img alt="Chart showing TPC-H benchmark results for Comet 0.7.0" class="img-responsive" src="/blog/images/comet-0.7.0/performance.png" width="100%"/></p>
+<p><em>These benchmarks were performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and data
+stored locally in Parquet format on NVMe storage. Spark was running in Kubernetes with hard memory limits.</em></p>
+<h2>Shuffle Improvements</h2>
+<p>There are several improvements to shuffle in this release:</p>
+<ul>
+<li>When running in off-heap mode (which is the recommended approach), Comet was using the wrong memory allocator
+ implementation for some types of shuffle operation, which could result in OOM rather than spilling to disk.</li>
+<li>The number of spill files is drastically reduced. In previous releases, each instance of ShuffleMapTask could
+ potentially create a new spill file for each output partition each time that spill was invoked. Comet now creates
+ a maximum of one spill file per output partition per instance of ShuffleMapTask, which is appended to in subsequent
+ spills.</li>
+<li>There was a flaw in the memory accounting which resulted in Comet requesting approximately twice the amount of
+ memory that was needed, resulting in premature spilling. This is now resolved.</li>
+<li>The metric for number of spilled bytes is now accurate. It was previously reporting invalid information.</li>
+</ul>
+<h2>Improved Hash Join Performance</h2>
+<p>When using the <code>spark.comet.exec.replaceSortMergeJoin</code> setting to replace sort-merge joins with hash joins, Comet
+will now do a better job of picking the optimal build side (thanks to [@hayman42] for suggesting this, and thanks to the
+[Apache Gluten(incubating)] project for the inspiration in implementing this feature).</p>
+<p><a href="https://github.com/hayman42">@hayman42</a></p>
+<h2>Experimental Support for DataFusion&rsquo;s DataSourceExec</h2>
+<p>It is now possible to configure Comet to use DataFusion&rsquo;s <code>DataSourceExec</code> instead of Comet&rsquo;s current Parquet reader.
+Support should still be considered experimental, but most of Comet&rsquo;s unit tests are now passing with the new reader.
+Known issues include handling of <code>INT96</code> timestamps and negative bytes and shorts.</p>
+<p>To enable DataFusion&rsquo;s <code>DataSourceExec</code>, either set <code>spark.comet.scan.impl=native_datafusion</code> or set the environment
+variable <code>COMET_PARQUET_SCAN_IMPL=native_datafusion</code>.</p>
+<h2>Complex Type Support</h2>
+<p>With DataFusion&rsquo;s <code>DataSourceExec</code> enabled, there is now some early support for reading structs from Parquet. This is
+largely untested and we would welcome additional testing from the community to help determine what is and isn&rsquo;t working,
+as well as contributions to improve support for structs and other complex types. The tracking issue is
+https://github.com/apache/datafusion-comet/issues/1043.</p>
+<h2>Updates to supported Spark versions</h2>
+<ul>
+<li>Comet 0.7.0 is now tested against Spark 3.5.4 rather than 3.5.1</li>
+<li>This will be the last Comet release to support Spark 3.3.x</li>
+</ul>
+<h2>Improved Tuning Guide</h2>
+<p>The <a href="https://datafusion.apache.org/comet/user-guide/tuning.html">Comet Tuning Guide</a> has been improved and now provides guidance on determining how much memory to allocate to
+Comet.</p>
+<h2>Getting Involved</h2>
+<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
+project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
+<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
+performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
+Comet.</p>
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="alternate"></link><published>2025-03-20T00:00:00+00:00</published><updated>2025-03-20T00:00:00+00:00</updated><author><name [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> implements advanced Parquet pruning techniques to effectively read only the data that matters for a given query.</p>
+<p>Achieving high performance adds complexity.
+This post provides an overview of the techniques used in DataFusion to selectively read Parquet files.</p>
+<h3>The pipeline</h3>
+<p>The diagram below illustrates the <a href="https://docs.rs/datafusion/46.0.0/datafusion/datasource/physical_plan/parquet/source/struct.ParquetSource.html">Parquet reading pipeline</a> in DataFusion, highlighting how data flows through various pruning stages before being converted to Arrow format:</p>
+<p><img alt="Parquet pruning pipeline in DataFusion" class="img-responsive" src="/blog/images/parquet-pruning/read-parquet.jpg" width="100%"/></p>
+<h4>Background: Parquet file structure</h4>
+<p>As shown in the figure above, each Parquet file has multiple row groups. Each row group contains a set of columns, and each column contains a set of pages.</p>
+<p>Pages are the smallest units of data in Parquet files and typically contain compressed and encoded values for a specific column. This hierarchical structure enables efficient columnar access and forms the foundation for the pruning techniques we'll discuss.</p>
+<p>Check out <a href="https://www.influxdata.com/blog/querying-parquet-millisecond-latency/">Querying Parquet with Millisecond Latency</a> for more details on the Parquet file structure.</p>
+<h4>1. Read metadata</h4>
+<p>DataFusion first reads the <a href="https://parquet.apache.org/docs/file-format/metadata/">Parquet metadata</a> to understand the data in the file.
+Metadata often includes data schema, the exact location of each row group and column chunk, and their corresponding statistics (e.g., min/max values).
+It also optionally includes <a href="https://parquet.apache.org/docs/file-format/pageindex/">page-level stats</a> and <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>.
+This information is used to prune the file before reading the actual data.</p>
+<p><a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L118">Fetching metadata</a> requires up to two network requests: one to read the footer size from the end of the file, and another to read the footer itself. </p>
+<p><a href="https://www.influxdata.com/blog/how-good-parquet-wide-tables/">Decoding metadata</a> is generally fast since it only requires parsing a small amount of data. However, for tables with hundreds or thousands of columns, the metadata can become quite large and decoding it can become a bottleneck. This is particularly noticeable when scanning many small files.</p>
+<p>Reading metadata is latency-critical, so DataFusion allows users to cache metadata through the <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L39">ParquetFileReaderFactory</a> trait.</p>
+<h4>2. Prune by projection</h4>
+<p>The simplest yet perhaps most effective pruning is to read only the columns that are needed.
+This is because queries usually don't select all columns, e.g., <code>SELECT a FROM table</code> only reads column <code>a</code>.
+As a <strong>columnar</strong> format, Parquet allows DataFusion to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/mod.rs#L778">only read</a> the <strong>columns</strong> that are needed.</p>
+<p>This projection pruning happens at the column level and can dramatically reduce I/O when working with wide tables where queries typically access only a small subset of columns.</p>
+<h4>3. Prune by row group stats and Bloom filters</h4>
+<p>Each row group has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/physical_optimizer/pruning.rs#L81">basic stats</a> like min/max values for each column.
+DataFusion applies the query predicates to these stats to prune row groups, e.g., <code>SELECT * FROM table WHERE a &gt; 10</code> will only read row groups where <code>a</code> has a max value greater than 10.</p>
+<p>Sometimes min/max stats are too simple to prune effectively, so Parquet also supports <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>. DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L202">uses Bloom filters when available</a>.</p>
+<p>Bloom filters are particularly effective for equality predicates (<code>WHERE a = 10</code>) and can significantly reduce the number of row groups that need to be read for point queries or queries with highly selective predicates.</p>
+<h4>4. Prune by page stats</h4>
+<p>Parquet optionally supports <a href="https://github.com/apache/parquet-format/blob/master/PageIndex.md">page-level stats</a> -- similar to row group stats but more fine-grained.
+DataFusion implements <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L219">page pruning</a> when the stats are present.</p>
+<p>Page-level pruning provides an additional layer of filtering after row group pruning. It allows DataFusion to skip individual pages within a row group, further reducing the amount of data that needs to be read and decoded.</p>
+<h4>5. Read from storage</h4>
+<p>Now we (hopefully) have pruned the Parquet file into small ranges of bytes, i.e., the <a href="https://github.com/apache/datafusion/blob/76a7789ace33ced54c973fa0d5fc9d1866e1bf19/datafusion/datasource-parquet/src/access_plan.rs#L86">Access Plan</a>.
+The last step is to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L103">make requests</a> to fetch those bytes and decode them into Arrow RecordBatch. </p>
+<h3>Preview of coming attractions: filter pushdown</h3>
+<p>So far we have discussed techniques that prune the Parquet file using only the metadata, i.e., before reading the actual data.</p>
+<p>Filter pushdown, also known as predicate pushdown or late materialization, is a technique that prunes data during scanning, with filters being generated and applied in the Parquet reader.</p>
+<p><img alt="Filter pushdown in DataFusion" class="img-responsive" src="/blog/images/parquet-pruning/filter-pushdown.jpg" width="100%"/></p>
+<p>Unlike metadata-based pruning which works at the row group or page level, filter pushdown operates at the row level, allowing DataFusion to filter out individual rows that don't match the query predicates during the decoding process.</p>
+<p>DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/row_filter.rs#L154">implements filter pushdown</a> but has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/common/src/config.rs#L382">not enabled it by default</a> due to <a href="https://github.com/apache/datafusion/issues/3463">some perform [...]
+<p>We are working to remove the <a href="https://github.com/apache/arrow-rs/issues/5523#issuecomment-2429470872">remaining performance issues</a> and enable it by default, which we will discuss in the next blog post.</p>
+<h3>Conclusion</h3>
+<p>DataFusion employs a multi-step approach to Parquet pruning, from column projection to row group stats, page stats, and potentially row-level filtering.
+Each step may reduce the amount of data to be read and processed, significantly improving query performance.</p></content><category term="blog"></category></entry><entry><title>Using Ordering for Better Plans in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="alternate"></link><published>2025-03-11T00:00:00+00:00</published><updated>2025-03-11T00:00:00+00:00</updated><author><name>Mustafa Akur, Andrew Lamb</name></author><i [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
+</blockquote>
+<p>There are various use cases where this type of analysis can be useful such as the following examples.</p>
+<h3>Removing Unnecessary Sorts</h3>
+<p>Imagine a user wants to execute the following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">log_line</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="k">time</span> <span class="k">ASC</span> <span class="k">limit</span> <span class="mi">10</span>
+</code></pre></div>
+<p>If we don't know anything about the <code>telemetry</code> table we need to sort it by <code>time ASC</code> and then retrieve the first 10 rows to get the correct result. However, if the table is already ordered by <code>time ASC</code>, we can simply retrieve the first 10 rows. This approach executes much faster and uses less memory compared to resorting the entire table, even when the <a href="https://docs.rs/datafusion/latest/datafusion [...]
+<p>In order to avoid the sort the query optimizer must determine the data is already sorted. For simple queries the analysis is straightforward however it gets complicated fast. For example, what if your data is sorted by <code>[hostname, time ASC]</code> and your query is</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">log_line</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">WHERE</span> <span class="n">hostname</span> <span class="o">=</span> <span class="s1">'app.example.com'</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="k">time</span> <span class="k">ASC</span><span class="p">;</span>
+</code></pre></div>
+<p>In this case a sort still isn't needed, but the analysis must reason about the sortedness of the stream when it knows <code>hostname</code> has a single value.</p>
+<h3>Optimized Operator Implementations</h3>
+<p>As another use case, some operators can utilize the ordering information to change its underlying algorithm to execute more efficiently. Consider the following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">log_line</span><span class="p">)</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="n">hostname</span><span class="p">;</span>
+</code></pre></div>
+<p>Most analytic systems, including DataFusion, by default implement such a query using a hash table keyed on values of <code>hostname</code> to store the counts. However, if the <code>telemetry</code> table is sorted by <code>hostname</code>, there are much more efficient algorithms for grouping on <code>hostname</code> values than hashing every value and storing it in memory. However, the more efficient algorithm can only be used w [...]
+<h3>Streaming-Friendly Execution</h3>
+<p>Stream processing aims to produce results immediately as they become available ensuring minimal latency for real-time workloads. However, some operators need to consume all input data before producing any output. Consider the <code>Sort</code> operation: before it can start generating output, the algorithm must first process all input data. As a result, data flow halts whenever such an operator is encountered until all input is consumed. When a physical query plan co [...]
+<p>For a query to be executed in a streaming fashion we need to satisfy 2 conditions:</p>
+<p><strong>Logically Streamable</strong><br/>
+It should be possible to generate what user wants in streaming fashion. Consider following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">SUM</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
+<span class="k">FROM</span> <span class="n">orders</span>
+</code></pre></div>
+<p>Here, the user wants to compute the sum of all amounts in the orders table. By the nature of the query this requires scanning the entire table to generate a result making it impossible to execute in a streaming fashion.</p>
+<p><strong>Streaming Aware Planner</strong><br/>
+Being logically streamable does not guarantee that a query will execute in a streaming fashion. SQL is a declarative language, meaning it specifies 'WHAT' user wants. It is up to the planner 'HOW' to generate the result. In most cases there are many ways to compute the correct result for a given query. The query planner is responsible for choosing "a way" (ideally the best<sup id="optimal1"><a href="#optimal">*</a></sup> one) among the all alternatives to generate [...]
+<h2>Analysis</h2>
+<p>Let's start by creating an example table that we will refer throughout the post. This table models the input data of an operator for the analysis:</p>
+<h3>Example Virtual Table</h3>
+<style>
+ table {
+ border-collapse: collapse;
+ width: 80%;
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+ }
+ th, td {
+ padding: 12px 16px;
+ text-align: left;
+ border-bottom: 1px solid #e0e0e0;
+ }
+ th {
+ background-color: #f9f9f9;
+ font-weight: 600;
+ }
+ tr:hover {
+ background-color: #f1f1f1;
+ }
+</style>
+<table>
+<tr>
+<th>amount</th> <th>price</th> <th>hostname</th><th>currency</th><th>time_bin</th> <th>time</th> <th>price_cloned</th> <th>time_cloned</th>
+</tr>
+<tr>
+<td>12</td> <td>25</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:01:30</td> <td>25</td> <td>08:01:30</td>
+</tr>
+<tr>
+<td>12</td> <td>26</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:11:30</td> <td>26</td> <td>08:11:30</td>
+</tr>
+<tr>
+<td>15</td> <td>30</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:41:30</td> <td>30</td> <td>08:41:30</td>
+</tr>
+<tr>
+<td>15</td> <td>32</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:55:15</td> <td>32</td> <td>08:55:15</td>
+</tr>
+<tr>
+<td>15</td> <td>35</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:10:23</td> <td>35</td> <td>09:10:23</td>
+</tr>
+<tr>
+<td>20</td> <td>18</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:20:33</td> <td>18</td> <td>09:20:33</td>
+</tr>
+<tr>
+<td>20</td> <td>22</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:40:15</td> <td>22</td> <td>09:40:15</td>
+</tr>
+</table>
+<p><br/></p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>How can a table have multiple orderings?</strong> At first glance it may seem counterintuitive for a table to have more than one valid ordering. However, during query execution such scenarios can arise.
+
+For example consider the following query:
+
+
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">time</span><span class="p">,</span> <span class="n">date_bin</span><span class="p">(</span><span class="s1">'1 hour'</span><span class="p">,</span> <span class="k">time</span><span class="p">,</span> <span class="s1">'1970-01-01'</span><span cla [...]
+<span class="k">FROM</span> <span class="k">table</span><span class="p">;</span>
+</code></pre></div>
+
+
+If we know that the table is ordered by <code>time ASC</code> we can infer that <code>time_bin ASC</code> is also a valid ordering. This is because the <code>date_bin</code> function is monotonic, meaning it preserves the order of its input.
+
+DataFusion leverages these functional dependencies to infer new orderings as data flows through different query operators. For details on the implementation see the <a ,="" href="https://github.com/apache/datafusion/blob/main/datafusion/common/src/functional_dependencies.rs" target="_blank">source</a> code.
+</blockquote>
+<p>By inspection, you can see this table is sorted by the <code>amount</code> column, but It is also sorted by <code>time</code> and <code>time_bin</code> as well as the compound <code>(time_bin, amount)</code> and many other variations. While this example is an extreme case, real-world data often has multiple sort orders. </p>
+<p>A naive approach for analyzing whether the ordering requirement of an operator is satisfied by its input would be: </p>
+<ul>
+<li>Store all the valid ordering expressions that the tables satisfies </li>
+<li>Check whether the ordering requirement by the operator is among valid orderings. </li>
+</ul>
+<p>This naive algorithm works and correct. However, listing all valid orderings can be quite lengthy and is of exponential complexity as the number of orderings grows. For the example table here is a (small) subset of the valid orderings:</p>
+<p><code>[amount ASC]</code><br/>
+<code>[amount ASC, price ASC]</code><br/>
+<code>[amount ASC, price_cloned ASC]</code><br/>
+<code>[hostname ASC, amount ASC, price_cloned ASC]</code><br/>
+<code>[amount ASC, hostname ASC, price_cloned ASC]</code><br/>
+<code>[amount ASC, price_cloned ASC, hostname ASC]</code><br/>
+.<br/>
+.<br/>
+. </p>
+<p>As can be seen from the listing above storing all valid orderings is wasteful and contains significant redundancy. Here are some observations which suggest that we can do much better:</p>
+<ul>
+<li>
+<p>Storing a prefix of another valid ordering is redundant. If the table satisfies the lexicographic ordering<sup id="fn1"><a href="#footnote1">1</a></sup>: <code>[amount ASC, price ASC]</code>, it already satisfies ordering <code>[amount ASC]</code> trivially. Hence, once we store <code>[amount ASC, price ASC]</code> storing <code>[amount ASC]</code> is redundant.</p>
+</li>
+<li>
+<p>Using all columns that are equal to each other in the listings is redundant. If we know the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by <code>[amount ASC, price_cloned ASC]</code> since <code>price</code> and <code>price_cloned</code> are copy of each other. It is enough to use just one expression among the expressions that exact copy of each other.</p>
+</li>
+<li>
+<p>Constant expressions can be inserted anywhere in a valid ordering with an arbitrary direction (e.g. <code>ASC</code>, <code>DESC</code>). Hence, if the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by: <br/>
+<code>[hostname ASC, amount ASC, price ASC]</code>,<br/>
+<code>[hostname DESC, amount ASC, price ASC]</code>,<br/>
+<code>[amount ASC, hostname ASC, price ASC]</code>,<br/>
+ .<br/>
+ . </p>
+</li>
+</ul>
+<p>This is clearly redundant. For this reason, it is better to avoid explicitly encoding constant expressions in valid sort orders.</p>
+<p>In summary,</p>
+<ul>
+<li>We should store only the longest lexicographic ordering (shouldn't use any prefix of it)</li>
+<li>Using expressions that are exact copies of each other is redundant.</li>
+<li>Ordering expressions shouldn't contain any constant expression.</li>
+</ul>
+<h2>Key Concepts for Analyzing Orderings</h2>
+<p>To solve the shortcomings above DataFusion needs to track of following properties for the table:</p>
+<ul>
+<li>Constant Expressions </li>
+<li>Equivalent Expression Groups (will be explained shortly)</li>
+<li>Succinct Valid Orderings (will be explained shortly)</li>
+</ul>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> These properties are implemented in the <code>EquivalenceProperties</code> structure in <code>DataFusion</code>, please see the <a href="https://github.com/apache/datafusion/blob/f47ea73b87eec4af044f9b9923baf042682615b2/datafusion/physical-expr/src/equivalence/properties/mod.rs#L134" target="_blank">source</a> for more details<br/>
+</blockquote>
+<p>These properties allow us to analyze whether the ordering requirement is satisfied by the data already.</p>
+<h3>1. Constant Expressions</h3>
+<p>Constant expressions are those where each row in the expression has the same value across all rows. Although constant expressions may seem odd in a table they can arise after operations like <code>Filter</code> or <code>Join</code> occur. </p>
+<p>For instance in the example table:</p>
+<ul>
+<li>Columns <code>hostname</code> and <code>currency</code> are constant because every row in the table has the same value (<code>'app.example.com'</code> for <code>hostname</code>, and <code>'USD'</code> for <code>currency</code>) for these columns.</li>
+</ul>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> Constant expressions can arise during query execution. For example, in following query:<br/>
+<code>SELECT hostname FROM logs</code><br/><code>WHERE hostname='app.example.com'</code> <br/>
+ after filtering is done, for subsequent operators the <code>hostname</code> column will be constant.
+</blockquote>
+<h3>2. Equivalent Expression Groups</h3>
+<p>Equivalent expression groups are expressions that always hold the same value across rows. These expressions can be thought of as clones of each other and may arise from operations like <code>Filter</code>, <code>Join</code>, or <code>Projection</code>.</p>
+<p>In the example table, the expressions <code>price</code> and <code>price_cloned</code> form one equivalence group, and <code>time</code> and <code>time_cloned</code> form another equivalence group.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> Equivalent expression groups can arise during the query execution. For example, in the following query:<br/>
+<code>SELECT time, time as time_cloned FROM logs</code> <br/>
+ after the projection is done, for subsequent operators <code>time</code> and <code>time_cloned</code> will form an equivalence group. As another example, in the following query:<br/>
+<code>SELECT employees.id, employees.name, departments.department_name</code>
+<code>FROM employees</code>
+<code>JOIN departments ON employees.department_id = departments.id;</code> <br/>
+after joining, <code>employees.department_id</code> and <code>departments.id</code> will form an equivalence group.
+</blockquote>
+<h3>3. Succinct Encoding of Valid Orderings</h3>
+<p>Valid orderings are the orderings that the table already satisfies. However, naively listing them requires exponential space as the number of columns grows as discussed before. Instead, we list all valid orderings after following constraints are applied:</p>
+<ul>
+<li>Do not use any constant expressions in the valid ordering construction</li>
+<li>Use only one entry (by convention the first entry) in the equivalent expression group.</li>
+<li>Lexicographic ordering shouldn't contain any leading ordering<sup id="fn2"><a href="#footnote2">2</a></sup>except the first position <sup id="fn3"><a href="#footnote3">3</a></sup>.</li>
+<li>Do not use any prefix of a valid lexicographic ordering<sup id="fn4"><a href="#footnote4">4</a></sup>.</li>
+</ul>
+<p>After applying the first and second constraint, the example table simplifies to </p>
+<style>
+ table {
+ border-collapse: collapse;
+ width: 80%;
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+ }
+ th, td {
+ padding: 12px 16px;
+ text-align: left;
+ border-bottom: 1px solid #e0e0e0;
+ }
+ th {
+ background-color: #f9f9f9;
+ font-weight: 600;
+ }
+ tr:hover {
+ background-color: #f1f1f1;
+ }
+</style>
+<table>
+<tr>
+<th>amount</th> <th>price</th><th>time_bin</th> <th>time</th>
+</tr>
+<tr>
+<td>12</td> <td>25</td><td>08:00:00</td> <td>08:01:30</td>
+</tr>
+<tr>
+<td>12</td> <td>26</td><td>08:00:00</td> <td>08:11:30</td>
+</tr>
+<tr>
+<td>15</td> <td>30</td><td>08:00:00</td> <td>08:41:30</td>
+</tr>
+<tr>
+<td>15</td> <td>32</td><td>08:00:00</td> <td>08:55:15</td>
+</tr>
+<tr>
+<td>15</td> <td>35</td><td>09:00:00</td> <td>09:10:23</td>
+</tr>
+<tr>
+<td>20</td> <td>18</td><td>09:00:00</td> <td>09:20:33</td>
+</tr>
+<tr>
+<td>20</td> <td>22</td><td>09:00:00</td> <td>09:40:15</td>
+</tr>
+</table>
+<p><br/>
+Following third and fourth constraints for the simplified table, the succinct valid orderings are:<br/>
+<code>[amount ASC, price ASC]</code>, <br/>
+<code>[time_bin ASC]</code>,<br/>
+<code>[time ASC]</code> </p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<p><strong>How can DataFusion find orderings?</strong></p>
+DataFusion's <code>CREATE EXTERNAL TABLE</code> has a <code>WITH ORDER</code> clause (see <a href="https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table">docs</a>) to specify the known orderings of the table during table creation. For example the following query:<br/>
+<pre><code>
+CREATE EXTERNAL TABLE source (
+ amount INT NOT NULL,
+ price DOUBLE NOT NULL,
+ time TIMESTAMP NOT NULL,
+ ...
+)
+STORED AS CSV
+WITH ORDER (time ASC)
+WITH ORDER (amount ASC, price ASC)
+LOCATION '/path/to/FILE_NAME.csv'
+OPTIONS ('has_header' 'true');
+</code></pre>
+communicates that <code>source</code> table has the orderings: <code>[time ASC]</code> and <code>[amount ASC, price ASC]</code>.<br/>
+When orderings are communicated from the source, DataFusion tracks the orderings through each operator while optimizing the plan.<br/>
+<ul>
+<li>add new orderings (such as when "date_bin" function is applied to the "time" column)</li>
+<li>Remove orderings, if operation doesn't preserve the ordering of the data at its input</li>
+<li>Update equivalent groups</li>
+<li>Update constant expressions</li>
+</ul>
+
+Figure 1 shows an example how DataFusion generates an efficient plan for the query:
+<pre><code>
+SELECT
+ row_number() OVER (ORDER BY time) as rn,
+ time
+FROM events
+ORDER BY rn, time
+</code></pre>
+using the orderings of the query intermediates.<br/>
+<br/>
+<figure>
+<img alt="Window Query Datafusion Optimization" class="img-responsive" src="/blog/images/ordering_analysis/query_window_plan.png" width="80%"/>
+<figcaption><strong>Figure 1:</strong> DataFusion analyzes orderings of the sources and query intermediates to generate efficient plans</figcaption>
+</figure>
+</blockquote>
+<h3>Table Properties</h3>
+<p>In summary, for the example table, the following properties correctly describe the sort properties:</p>
+<ul>
+<li><strong>Constant Expressions</strong> = <code>hostname, currency</code> </li>
+<li><strong>Equivalent Expression Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code> </li>
+<li><strong>Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code> </li>
+</ul>
+<h3>Algorithm for Analyzing Ordering Requirements</h3>
+<p>After deriving these properties for the data, following algorithm can be used to check whether an ordering requirement is satisfied by the table:</p>
+<ol>
+<li><strong>Prune constant expressions</strong>: Remove any constant expressions from the ordering requirement.</li>
+<li><strong>Normalize the requirement</strong>: Replace each expression in the ordering requirement with the first entry from its equivalence group.</li>
+<li><strong>De-duplicate expressions</strong>: If an expression appears more than once, remove duplicates, keeping only the first occurrence.</li>
+<li><strong>Match leading orderings</strong>: Check whether the leading ordering requirement<sup id="fn5"><a href="#footnote5">5</a></sup> matches the leading valid orderings<sup id="fn6"><a href="#footnote6">6</a></sup> of table. If so:<ul>
+<li>Remove the leading ordering requirement from the ordering requirement </li>
+<li>Remove the matching leading valid ordering from the valid orderings of table. </li>
+</ul>
+</li>
+<li><strong>Iterate through the remaining expressions</strong>: Go back to step 4 until ordering requirement is empty or leading ordering requirement is not found among the leading valid orderings of table.</li>
+</ol>
+<p>If, at the end of the procedure above, the ordering requirement is an empty list, we can conclude that the requirement is satisfied by the table.</p>
+<h3>Example Walkthrough</h3>
+<p>Let's say the user provided a query such as the following:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="k">table</span>
+<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">hostname</span> <span class="k">DESC</span><span class="p">,</span> <span class="n">amount</span> <span class="k">ASC</span><span class="p">,</span> <span class="n">time_bin</span> <span class="k">ASC</span><span class="p">,</span> <span class="n">price_cloned</span> <span [...]
+</code></pre></div>
+<p>And the input has the same properties explained above</p>
+<ul>
+<li><strong>Constant Expressions</strong> = <code>hostname, currency</code></li>
+<li><strong>Equivalent Expressions Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code></li>
+<li><strong>Succinct Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code></li>
+</ul>
+<p>In order to remove a sort the optimizer must check if the ordering requirement <code>[hostname DESC, amount ASC, time_bin ASC, price_cloned ASC, time ASC, currency ASC, price DESC]</code> is satisfied by the properties.</p>
+<h3>Algorithm Steps</h3>
+<ol>
+<li>
+<p><strong>Prune constant expressions</strong>:<br/>
+ Remove <code>hostname</code> and <code>currency</code> from the requirement. The requirement becomes:<br/>
+<code>[amount ASC, time_bin ASC, price_cloned ASC, time ASC, price DESC]</code>.</p>
+</li>
+<li>
+<p><strong>Normalize using equivalent groups</strong>:<br/>
+ Replace <code>price_cloned</code> with <code>price</code> and <code>time_cloned</code> with <code>time</code>. The requirement becomes:<br/>
+<code>[amount ASC, time_bin ASC, price ASC, time ASC, price DESC]</code>.</p>
+</li>
+<li>
+<p><strong>De-duplicate expressions</strong>:<br/>
+ Since <code>price</code> appears twice, we simplify the requirement to:<br/>
+<code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> (keeping the first occurrence from the left side).</p>
+</li>
+<li>
+<p><strong>Match leading orderings</strong>:<br/>
+ Check if leading ordering requirement <code>amount ASC</code> is among the leading valid orderings: <code>amount ASC, time_bin ASC, time ASC</code>. Since this is the case, we remove <code>amount ASC</code> from both the ordering requirement and the valid orderings of the table.</p>
+</li>
+<li><strong>Iterate through the remaining expressions</strong>:
+Now, the problem is converted from<br/>
+<em>"whether the requirement: <code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
+into<br/>
+<em>"whether the requirement: <code>[time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
+We go back to step 4 until the ordering requirement list is exhausted or its length no longer decreases.</li>
+</ol>
+<p>At the end of stages above we end up with an empty ordering requirement list. Given this, we can conclude that the table satisfies the ordering requirement and thus no sort is required. </p>
+<h2>Conclusion</h2>
+<p>In this post, we described the conditions under which an ordering requirement is satisfied based on the properties of a table. We introduced key concepts such as constant expressions, equivalence groups, and valid orderings, and used them to determine whether a given ordering requirement are satisfied by an input table.</p>
+<p>This analysis plays a crucial role in:</p>
+<ul>
+<li>Choosing more efficient algorithm variants</li>
+<li>Generating streaming-friendly plans</li>
+</ul>
+<p>The <code>DataFusion</code> query engine employs this analysis (and many more) during its planning stage to ensure correct and efficient query execution. We <a href="https://datafusion.apache.org/contributor-guide/index.html">welcome you</a> to come and join the project.</p>
+<h2>Appendix</h2>
+<!--
+<p id="footnote1"><sup>[1]</sup>The ordering requirement refers to the condition that input data must be sorted in a certain way for a specific operator to function as intended.</p>
+-->
+<p id="footnote1"><sup>[1]</sup>Lexicographic order is a way of ordering sequences (like strings, list of expressions) based on the order of their components, similar to how words are ordered in a dictionary. It compares each element of the sequences one by one, from left to right.</p>
+<p id="footnote2"><sup>[2]</sup>Leading ordering is the first ordering in a lexicographic ordering list. As an example, for the ordering: <code>[amount ASC, price ASC]</code>, leading ordering will be: <code>amount ASC</code>. </p>
+<p id="footnote3"><sup>[3]</sup>This means that, if we know that <code>[amount ASC]</code> and <code>[time ASC]</code> are both valid orderings for the table. We shouldn't enlist <code>[amount ASC, time ASC]</code> or <code>[time ASC, amount ASC]</code> as valid orderings. These orderings can be deduced if we know that table satisfies the ordering <code>[amount ASC]</code> and <code>[time ASC]</code> [...]
+<p id="footnote4"><sup>[4]</sup>This means that, if ordering <code>[amount ASC, price ASC]</code> is a valid ordering for the table. We shouldn't enlist <code>[amount ASC]</code> as valid ordering. Validity of it can be deduced from the ordering: <code>[amount ASC, price ASC]</code>
+<p id="footnote5"><sup>[5]</sup>Leading ordering requirement is the first ordering requirement in the list of lexicographic ordering requirement expression. As an example for the requirement: <code>[amount ASC, time_bin ASC, prices ASC, time ASC]</code>, leading ordering requirement is: <code>amount ASC</code>.</p>
+<p id="footnote6"><sup>[6]</sup>Leading valid orderings are the first ordering for each valid ordering list in the table. As an example, for the valid orderings: <code>[amount ASC, prices ASC], [time_bin ASC], [time ASC]</code>, leading valid orderings will be: <code>amount ASC, time_bin ASC, time ASC</code>. </p>
+<p id="optimal"><sup>*</sup>Best depends on the use case, <code>DataFusion</code> has many various flags to communicate what user thinks the best plan is (e.g. streamable, fastest, lowest memory, etc.). See <a href="https://datafusion.apache.org/user-guide/configs.html" target="_blank">configurations</a> for detail.</p></p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 45.0.0 Released</title><link h [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development for DataFusion!</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that
+uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
+create new, fast data centric systems such as databases, dataframe libraries,
+machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion&rsquo;s primary design
+goal</a> is to accelerate the creation of other data centric systems, it has a
+reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe library</a>,
+<a href="https://datafusion.apache.org/python/">python library</a> and <a href="https://datafusion.apache.org/user-guide/cli/">command line SQL tool</a>.</p>
+<p>DataFusion's core thesis is that as a community, together we can build much more
+advanced technology than any of us as individuals or companies could do alone.
+Without DataFusion, highly performant vectorized query engines would remain
+the domain of a few large companies and world-class research institutions.
+With DataFusion, we can all build on top of a shared foundation, and focus on
+what makes our projects unique.</p>
+<h2>Community Growth 📈</h2>
+<p>In the last 6 months, between <code>40.0.0</code> and <code>45.0.0</code>, our community continues to
+grow in new and exciting ways.</p>
+<ol>
+<li>We added several PMC members and new committers: <a href="https://github.com/jayzhan211">@jayzhan211</a> and <a href="https://github.com/jonahgao">@jonahgao</a> joined the PMC,
+ <a href="https://github.com/2010YOUY01">@2010YOUY01</a>, <a href="https://github.com/rachelint">@rachelint</a>, <a href="https://github.com/findepi/">@findpi</a>, <a href="https://github.com/iffyio">@iffyio</a>, <a href="https://github.com/goldmedal">@goldmedal</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/Michael-J-Ward">@Michael-J-Ward</a> and <a href="https [...]
+ joined as committers. See the <a href="https://lists.apache.org/list.html?dev@datafusion.apache.org">mailing list</a> for more details.</li>
+<li>In the <a href="https://github.com/apache/arrow-datafusion">core DataFusion repo</a> alone we reviewed and accepted almost 1600 PRs from 206 different
+ committers, created over 1100 issues and closed 751 of them 🚀. All changes are listed in the detailed
+ <a href="https://github.com/apache/datafusion/tree/main/dev/changelog">changelogs</a>.</li>
+<li>DataFusion focused meetups happened in multiple cities around the world: <a href="https://github.com/apache/datafusion/discussions/10341#discussioncomment-10110273">Hangzhou</a>, <a href="https://github.com/apache/datafusion/discussions/11431">Belgrade</a>, <a href="https://github.com/apache/datafusion/discussions/11213">New York</a>,
+ <a href="https://github.com/apache/datafusion/discussions/10348">Seattle</a>, <a href="https://github.com/apache/datafusion/discussions/12894">Chicago</a>, <a href="https://github.com/apache/datafusion/discussions/13165">Boston</a> and <a href="https://github.com/apache/datafusion/discussions/12988">Amsterdam</a> as well as a Rust NYC meetup in NYC focused on DataFusion.</li>
+</ol>
+<!--
+$ git log --pretty=oneline 40.0.0..45.0.0 . | wc -l
+ 1532 (up from 1453)
+
+$ git shortlog -sn 40.0.0..45.0.0 . | wc -l
+ 206 (up from 182)
+
+https://crates.io/crates/datafusion/45.0.0
+DataFusion 45 released Feb 7, 2025
+
+https://crates.io/crates/datafusion/40.0.0
+DataFusion 40 released July 12, 2024
+
+Issues created in this time: 375 open, 751 closed (from 321 open, 781 closed)
+https://github.com/apache/datafusion/issues?q=is%3Aissue+created%3A2024-07-12..2025-02-07
+
+Issues closed: 956 (up from 911)
+https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2024-07-12..2025-02-07
+
+PRs merged in this time 1597 (up from 1490)
+https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2024-07-12..2025-02-07
+
+-->
+<p>DataFusion has put in an application to be part of <a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> with a
+<a href="https://github.com/apache/datafusion/issues/14478">number of ideas</a> for projects with mentors already selected. Additionally, <a href="https://github.com/apache/datafusion/issues/14373">some ideas</a> on
+how to make DataFusion an ideal selection for university database projects such as the
+<a href="https://15445.courses.cs.cmu.edu/spring2025/">CMU database classes</a> have been put forward.</p>
+<p>In addition, DataFusion has been appearing publicly more and more, both online and offline. Here are some highlights:</p>
+<ol>
+<li>A <a href="https://uwheel.rs/post/datafusion_uwheel/">demonstration of how uwheel</a> is integrated into DataFusion</li>
+<li>Integrating StringView into DataFusion - <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">part 1</a> and <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">part 2</a></li>
+<li><a href="https://techontherocks.show/3">Building streams</a> with DataFusion</li>
+<li><a href="https://blog.haoxp.xyz/posts/caching-datafusion">Caching in DataFusion</a>: Don't read twice</li>
+<li><a href="https://blog.haoxp.xyz/posts/parquet-to-arrow/">Parquet pruning in DataFusion</a>: Read no more than you need</li>
+<li>DataFusion is one of <a href="https://www.crn.com/news/software/2024/the-10-coolest-open-source-software-tools-of-2024?page=3">The 10 coolest open source software tools</a></li>
+<li><a href="https://www.denormalized.io/blog/building-databases">Building databases over a weekend</a></li>
+</ol>
+<h2>Improved Performance 🚀</h2>
+<p>DataFusion hit a milestone in its development by becoming <a href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/">the fastest single node engine</a>
+for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">clickbench</a> benchmark for the 43.0.0 release. A <a href="https://github.com/apache/datafusion/issues/12821">lot
+of work</a> went into making this happen! While other engines have subsequently gotten faster,
+displacing DataFusion from the top spot, DataFusion still remains near the top and we <a href="https://github.com/apache/datafusion/issues/14586">are planning
+more improvements</a>.</p>
+<p><img alt="ClickBench performance results over time for DataFusion" class="img-responsive" src="/blog/images/datafusion-45.0.0/performance_over_time.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: ClickBench performance improved over 33% between DataFusion 33
+(released Nov. 2023) and DataFusion 45 (released Feb. 2025). </p>
+<p>The task of <a href="https://github.com/apache/datafusion/issues/10918">integrating</a> the new <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html">Arrow StringView</a> which significantly improves performance
+for workloads that scan, filter and group by variable length string and binary data was completed
+and enabled by default in the past 6 months. The improvement is especially pronounced for Parquet
+files due to <a href="https://github.com/apache/arrow-rs/issues/5530">upstream work in the parquet reader</a>. Kudos to <a href="https://github.com/XiangpengHong">@XiangpengHong</a>, <a href="https://github.com/AriesDevil">@AriesDevil</a>,
+<a href="https://github.com/PsiACE">@PsiACE</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/a10y">@a10y</a>, and <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a> for driving this project.</p>
+<h2>Improved Quality 📋</h2>
+<p>DataFusion continues to improve overall in quality. In addition to ongoing bug
+fixes, one of the most exciting improvements in the last 6 months was the addition of the
+<a href="https://github.com/apache/datafusion/pull/13936">SQLite sqllogictest suite</a> thanks to <a href="https://github.com/Omega359">@Omega359</a>. These tests run over 5 million
+sql statements on every push to the main branch.</p>
+<p>Support for <a href="https://github.com/apache/datafusion/pull/13651">explicitly checking logical plan invariants</a> was added by <a href="https://github.com/wiedld">@wiedld</a> which
+can help catch implicit changes that might cause problems during upgrades.</p>
+<p>We have also started other quality initiatives to make it <a href="https://github.com/apache/datafusion/issues/13525">easier to use DataFusion</a>
+based on <a href="https://glaredb.com/">GlareDB</a>'s experience along with more <a href="https://github.com/apache/datafusion/issues/13661">extensive prerelease testing</a>. </p>
+<h2>Improved Documentation 📚</h2>
+<p>We continue to improve the documentation to make it easier to get started using DataFusion.
+During the last 6 months two projects were initiated to migrate the function documentation
+from strictly static markdown files. First, <a href="https://github.com/apache/datafusion/pull/12668">@Omega359</a> to allow function
+documentation to be generated from code and <a href="https://github.com/jonathanc-n">@jonathanc-n</a> and others helped with the migration,
+then <a href="https://github.com/comphead">@comphead</a> lead a project to <a href="https://github.com/apache/datafusion/pull/12822">create a doc macro</a> to allow for an even easier way to write
+function documentation. A special thanks to <a href="https://github.com/Chen-Yuan-Lai">@Chen-Yuan-Lai</a> for migrating many functions to
+the new syntax.</p>
+<p>Additionally, the <a href="https://github.com/apache/datafusion/pull/13877">examples</a> were <a href="https://github.com/apache/datafusion/pull/13905">refactored</a> and <a href="https://github.com/apache/datafusion/pull/13950">cleaned up</a> to improve their usefulness.</p>
+<h2>New Features ✨</h2>
+<p>There are too many new features in the last 6 months to list them all, but here
+are some highlights:</p>
+<h3>Functions</h3>
+<ul>
+<li>Uniform Window Functions: <code>BuiltInWindowFunctions</code> was removed and all now use UDFs (<a href="https://github.com/jcsherin">@jcsherin</a>)</li>
+<li>Uniform Aggregate Functions: <code>BuiltInAggregateFunctions</code> was removed and all now use UDFs</li>
+<li>As mentioned above function documentation was extracted from the markdown files</li>
+<li>Some new functions and sql support were added including '<a href="https://github.com/apache/datafusion/pull/13799">show functions</a>', '<a href="https://github.com/apache/datafusion/pull/11347">to_local_time</a>',
+ '<a href="https://github.com/apache/datafusion/pull/12970">regexp_count</a>', '<a href="https://github.com/apache/datafusion/pull/11969">map_extract</a>', '<a href="https://github.com/apache/datafusion/pull/12211">array_distance</a>', '<a href="https://github.com/apache/datafusion/pull/12329">array_any_value</a>', '<a href="https://github.com/apache/datafusion/pull/12474">greatest</a>',
+ '<a href="https://github.com/apache/datafusion/pull/13786">least</a>', '<a href="https://github.com/apache/datafusion/pull/14217">arrays_overlap</a>'</li>
+</ul>
+<h3>FFI</h3>
+<ul>
+<li>Foreign Function Interface work has started. This should allow for
+ <a href="https://github.com/apache/datafusion/pull/12920">using table providers</a> across languages and versions of DataFusion. This
+ is especially pertinent for integration with <a href="https://delta-io.github.io/delta-rs/">delta-rs</a> and other table formats.</li>
+</ul>
+<h3>Materialized Views</h3>
+<ul>
+<li><a href="https://github.com/suremarc">@suremarc</a> has added a <a href="https://github.com/datafusion-contrib/datafusion-materialized-views">materialized view implementation</a> in datafusion-contrib 🚀</li>
+</ul>
+<h3>Substrait</h3>
+<ul>
+<li>A lot of work was put into improving and enhancing substrait support (<a href="https://github.com/Blizzara">@Blizzara</a>, <a href="https://github.com/westonpace">@westonpace</a>, <a href="https://github.com/tokoko">@tokoko</a>, <a href="https://github.com/vbarua">@vbarua</a>, <a href="https://github.com/LatrecheYasser">@LatrecheYasser</a>, <a href="https://github.com/notfilippo">@notfilippo</a> and others) [...]
+</ul>
+<h2>Looking Ahead: The Next Six Months ðŸ”</h2>
+<p>One of the long term goals of <a href="https://github.com/alamb">@alamb</a>, DataFusion's PMC chair, has been to have
+<a href="https://www.influxdata.com/blog/datafusion-2025-influxdb/">1000 DataFusion based projects</a>. This may be the year that happens!</p>
+<p>The community has been <a href="https://github.com/apache/datafusion/issues/14580">discussing what we will work on in the next six months</a>.
+Some major initiatives are likely to be:</p>
+<ol>
+<li><em>Performance</em>: A <a href="https://github.com/apache/datafusion/issues/14482">number of items have been identified</a> as areas that could use additional work</li>
+<li><em>Memory usage</em>: Tracking and improving memory usage, statistics and spilling to disk </li>
+<li><em><a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> (GSOC)</em>: DataFusion is hopefully selected as a project and we start accepting and supporting student projects </li>
+<li><em>FFI</em>: Extending the FFI implementation to support to all types of UDF's and SessionContext</li>
+<li><em>Spark Functions</em>: A <a href="https://github.com/apache/datafusion/issues/5600">proposal has been made to add a crate</a> covering spark compatible builtin functions </li>
+</ol>
+<h2>How to Get Involved</h2>
+<p>DataFusion is not a project built or driven by a single person, company, or
+foundation. Rather, our community of users and contributors work together to
+build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us we would love to have you. You can try out
+DataFusion on some of your own data and projects and let us know how it goes,
+contribute suggestions, documentation, bug reports, or a PR with documentation,
+tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
+can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="alternate"></link><published>2025-02-17T00:00:00+00:00</published><updated>2025-02-17T00:00:00+00:00</updated><author><name>pmc</name></author [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/blog.atom.xml b/blog/feeds/blog.atom.xml
index d980ab4..a7a1e4f 100644
--- a/blog/feeds/blog.atom.xml
+++ b/blog/feeds/blog.atom.xml
@@ -1,5 +1,770 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/blog.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-02-17T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0 [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/blog.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-20T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion Comet 0.7.0 Release</title><link href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0 [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
+expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
+to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
+<p>This release covers approximately four weeks of development work and is the result of merging 46 PRs from 11
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.7.0.md">change log</a> for more information.</p>
+<h2>Release Highlights</h2>
+<h3>Performance</h3>
+<p>Comet 0.7.0 has improved performance compared to the previous release due to improvements in the native shuffle
+implementation and performance improvements in DataFusion 46.</p>
+<p>For single-node TPC-H at 100 GB, Comet now delivers a <strong>2.2x speedup</strong> compared to Spark using the same CPU and RAM. Even
+with <strong>half the resources</strong>, Comet still provides a measurable performance improvement.</p>
+<p><img alt="Chart showing TPC-H benchmark results for Comet 0.7.0" class="img-responsive" src="/blog/images/comet-0.7.0/performance.png" width="100%"/></p>
+<p><em>These benchmarks were performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and data
+stored locally in Parquet format on NVMe storage. Spark was running in Kubernetes with hard memory limits.</em></p>
+<h2>Shuffle Improvements</h2>
+<p>There are several improvements to shuffle in this release:</p>
+<ul>
+<li>When running in off-heap mode (which is the recommended approach), Comet was using the wrong memory allocator
+ implementation for some types of shuffle operation, which could result in OOM rather than spilling to disk.</li>
+<li>The number of spill files is drastically reduced. In previous releases, each instance of ShuffleMapTask could
+ potentially create a new spill file for each output partition each time that spill was invoked. Comet now creates
+ a maximum of one spill file per output partition per instance of ShuffleMapTask, which is appended to in subsequent
+ spills.</li>
+<li>There was a flaw in the memory accounting which resulted in Comet requesting approximately twice the amount of
+ memory that was needed, resulting in premature spilling. This is now resolved.</li>
+<li>The metric for number of spilled bytes is now accurate. It was previously reporting invalid information.</li>
+</ul>
+<h2>Improved Hash Join Performance</h2>
+<p>When using the <code>spark.comet.exec.replaceSortMergeJoin</code> setting to replace sort-merge joins with hash joins, Comet
+will now do a better job of picking the optimal build side (thanks to [@hayman42] for suggesting this, and thanks to the
+[Apache Gluten(incubating)] project for the inspiration in implementing this feature).</p>
+<p><a href="https://github.com/hayman42">@hayman42</a></p>
+<h2>Experimental Support for DataFusion&rsquo;s DataSourceExec</h2>
+<p>It is now possible to configure Comet to use DataFusion&rsquo;s <code>DataSourceExec</code> instead of Comet&rsquo;s current Parquet reader.
+Support should still be considered experimental, but most of Comet&rsquo;s unit tests are now passing with the new reader.
+Known issues include handling of <code>INT96</code> timestamps and negative bytes and shorts.</p>
+<p>To enable DataFusion&rsquo;s <code>DataSourceExec</code>, either set <code>spark.comet.scan.impl=native_datafusion</code> or set the environment
+variable <code>COMET_PARQUET_SCAN_IMPL=native_datafusion</code>.</p>
+<h2>Complex Type Support</h2>
+<p>With DataFusion&rsquo;s <code>DataSourceExec</code> enabled, there is now some early support for reading structs from Parquet. This is
+largely untested and we would welcome additional testing from the community to help determine what is and isn&rsquo;t working,
+as well as contributions to improve support for structs and other complex types. The tracking issue is
+https://github.com/apache/datafusion-comet/issues/1043.</p>
+<h2>Updates to supported Spark versions</h2>
+<ul>
+<li>Comet 0.7.0 is now tested against Spark 3.5.4 rather than 3.5.1</li>
+<li>This will be the last Comet release to support Spark 3.3.x</li>
+</ul>
+<h2>Improved Tuning Guide</h2>
+<p>The <a href="https://datafusion.apache.org/comet/user-guide/tuning.html">Comet Tuning Guide</a> has been improved and now provides guidance on determining how much memory to allocate to
+Comet.</p>
+<h2>Getting Involved</h2>
+<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
+project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
+<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
+performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
+Comet.</p>
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning" rel="alternate"></link><published>2025-03-20T00:00:00+00:00</published><updated>2025-03-20T00:00:00+00:00</updated><author><name [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> implements advanced Parquet pruning techniques to effectively read only the data that matters for a given query.</p>
+<p>Achieving high performance adds complexity.
+This post provides an overview of the techniques used in DataFusion to selectively read Parquet files.</p>
+<h3>The pipeline</h3>
+<p>The diagram below illustrates the <a href="https://docs.rs/datafusion/46.0.0/datafusion/datasource/physical_plan/parquet/source/struct.ParquetSource.html">Parquet reading pipeline</a> in DataFusion, highlighting how data flows through various pruning stages before being converted to Arrow format:</p>
+<p><img alt="Parquet pruning pipeline in DataFusion" class="img-responsive" src="/blog/images/parquet-pruning/read-parquet.jpg" width="100%"/></p>
+<h4>Background: Parquet file structure</h4>
+<p>As shown in the figure above, each Parquet file has multiple row groups. Each row group contains a set of columns, and each column contains a set of pages.</p>
+<p>Pages are the smallest units of data in Parquet files and typically contain compressed and encoded values for a specific column. This hierarchical structure enables efficient columnar access and forms the foundation for the pruning techniques we'll discuss.</p>
+<p>Check out <a href="https://www.influxdata.com/blog/querying-parquet-millisecond-latency/">Querying Parquet with Millisecond Latency</a> for more details on the Parquet file structure.</p>
+<h4>1. Read metadata</h4>
+<p>DataFusion first reads the <a href="https://parquet.apache.org/docs/file-format/metadata/">Parquet metadata</a> to understand the data in the file.
+Metadata often includes data schema, the exact location of each row group and column chunk, and their corresponding statistics (e.g., min/max values).
+It also optionally includes <a href="https://parquet.apache.org/docs/file-format/pageindex/">page-level stats</a> and <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>.
+This information is used to prune the file before reading the actual data.</p>
+<p><a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L118">Fetching metadata</a> requires up to two network requests: one to read the footer size from the end of the file, and another to read the footer itself. </p>
+<p><a href="https://www.influxdata.com/blog/how-good-parquet-wide-tables/">Decoding metadata</a> is generally fast since it only requires parsing a small amount of data. However, for tables with hundreds or thousands of columns, the metadata can become quite large and decoding it can become a bottleneck. This is particularly noticeable when scanning many small files.</p>
+<p>Reading metadata is latency-critical, so DataFusion allows users to cache metadata through the <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L39">ParquetFileReaderFactory</a> trait.</p>
+<h4>2. Prune by projection</h4>
+<p>The simplest yet perhaps most effective pruning is to read only the columns that are needed.
+This is because queries usually don't select all columns, e.g., <code>SELECT a FROM table</code> only reads column <code>a</code>.
+As a <strong>columnar</strong> format, Parquet allows DataFusion to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/mod.rs#L778">only read</a> the <strong>columns</strong> that are needed.</p>
+<p>This projection pruning happens at the column level and can dramatically reduce I/O when working with wide tables where queries typically access only a small subset of columns.</p>
+<h4>3. Prune by row group stats and Bloom filters</h4>
+<p>Each row group has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/physical_optimizer/pruning.rs#L81">basic stats</a> like min/max values for each column.
+DataFusion applies the query predicates to these stats to prune row groups, e.g., <code>SELECT * FROM table WHERE a &gt; 10</code> will only read row groups where <code>a</code> has a max value greater than 10.</p>
+<p>Sometimes min/max stats are too simple to prune effectively, so Parquet also supports <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>. DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L202">uses Bloom filters when available</a>.</p>
+<p>Bloom filters are particularly effective for equality predicates (<code>WHERE a = 10</code>) and can significantly reduce the number of row groups that need to be read for point queries or queries with highly selective predicates.</p>
+<h4>4. Prune by page stats</h4>
+<p>Parquet optionally supports <a href="https://github.com/apache/parquet-format/blob/master/PageIndex.md">page-level stats</a> -- similar to row group stats but more fine-grained.
+DataFusion implements <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L219">page pruning</a> when the stats are present.</p>
+<p>Page-level pruning provides an additional layer of filtering after row group pruning. It allows DataFusion to skip individual pages within a row group, further reducing the amount of data that needs to be read and decoded.</p>
+<h4>5. Read from storage</h4>
+<p>Now we (hopefully) have pruned the Parquet file into small ranges of bytes, i.e., the <a href="https://github.com/apache/datafusion/blob/76a7789ace33ced54c973fa0d5fc9d1866e1bf19/datafusion/datasource-parquet/src/access_plan.rs#L86">Access Plan</a>.
+The last step is to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L103">make requests</a> to fetch those bytes and decode them into Arrow RecordBatch. </p>
+<h3>Preview of coming attractions: filter pushdown</h3>
+<p>So far we have discussed techniques that prune the Parquet file using only the metadata, i.e., before reading the actual data.</p>
+<p>Filter pushdown, also known as predicate pushdown or late materialization, is a technique that prunes data during scanning, with filters being generated and applied in the Parquet reader.</p>
+<p><img alt="Filter pushdown in DataFusion" class="img-responsive" src="/blog/images/parquet-pruning/filter-pushdown.jpg" width="100%"/></p>
+<p>Unlike metadata-based pruning which works at the row group or page level, filter pushdown operates at the row level, allowing DataFusion to filter out individual rows that don't match the query predicates during the decoding process.</p>
+<p>DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/row_filter.rs#L154">implements filter pushdown</a> but has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/common/src/config.rs#L382">not enabled it by default</a> due to <a href="https://github.com/apache/datafusion/issues/3463">some perform [...]
+<p>We are working to remove the <a href="https://github.com/apache/arrow-rs/issues/5523#issuecomment-2429470872">remaining performance issues</a> and enable it by default, which we will discuss in the next blog post.</p>
+<h3>Conclusion</h3>
+<p>DataFusion employs a multi-step approach to Parquet pruning, from column projection to row group stats, page stats, and potentially row-level filtering.
+Each step may reduce the amount of data to be read and processed, significantly improving query performance.</p></content><category term="blog"></category></entry><entry><title>Using Ordering for Better Plans in Apache DataFusion</title><link href="https://datafusion.apache.org/blog/2025/03/11/ordering-analysis" rel="alternate"></link><published>2025-03-11T00:00:00+00:00</published><updated>2025-03-11T00:00:00+00:00</updated><author><name>Mustafa Akur, Andrew Lamb</name></author><i [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
+</blockquote>
+<p>There are various use cases where this type of analysis can be useful such as the following examples.</p>
+<h3>Removing Unnecessary Sorts</h3>
+<p>Imagine a user wants to execute the following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">log_line</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="k">time</span> <span class="k">ASC</span> <span class="k">limit</span> <span class="mi">10</span>
+</code></pre></div>
+<p>If we don't know anything about the <code>telemetry</code> table we need to sort it by <code>time ASC</code> and then retrieve the first 10 rows to get the correct result. However, if the table is already ordered by <code>time ASC</code>, we can simply retrieve the first 10 rows. This approach executes much faster and uses less memory compared to resorting the entire table, even when the <a href="https://docs.rs/datafusion/latest/datafusion [...]
+<p>In order to avoid the sort the query optimizer must determine the data is already sorted. For simple queries the analysis is straightforward however it gets complicated fast. For example, what if your data is sorted by <code>[hostname, time ASC]</code> and your query is</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">log_line</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">WHERE</span> <span class="n">hostname</span> <span class="o">=</span> <span class="s1">'app.example.com'</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="k">time</span> <span class="k">ASC</span><span class="p">;</span>
+</code></pre></div>
+<p>In this case a sort still isn't needed, but the analysis must reason about the sortedness of the stream when it knows <code>hostname</code> has a single value.</p>
+<h3>Optimized Operator Implementations</h3>
+<p>As another use case, some operators can utilize the ordering information to change its underlying algorithm to execute more efficiently. Consider the following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">log_line</span><span class="p">)</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="n">hostname</span><span class="p">;</span>
+</code></pre></div>
+<p>Most analytic systems, including DataFusion, by default implement such a query using a hash table keyed on values of <code>hostname</code> to store the counts. However, if the <code>telemetry</code> table is sorted by <code>hostname</code>, there are much more efficient algorithms for grouping on <code>hostname</code> values than hashing every value and storing it in memory. However, the more efficient algorithm can only be used w [...]
+<h3>Streaming-Friendly Execution</h3>
+<p>Stream processing aims to produce results immediately as they become available ensuring minimal latency for real-time workloads. However, some operators need to consume all input data before producing any output. Consider the <code>Sort</code> operation: before it can start generating output, the algorithm must first process all input data. As a result, data flow halts whenever such an operator is encountered until all input is consumed. When a physical query plan co [...]
+<p>For a query to be executed in a streaming fashion we need to satisfy 2 conditions:</p>
+<p><strong>Logically Streamable</strong><br/>
+It should be possible to generate what user wants in streaming fashion. Consider following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">SUM</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
+<span class="k">FROM</span> <span class="n">orders</span>
+</code></pre></div>
+<p>Here, the user wants to compute the sum of all amounts in the orders table. By the nature of the query this requires scanning the entire table to generate a result making it impossible to execute in a streaming fashion.</p>
+<p><strong>Streaming Aware Planner</strong><br/>
+Being logically streamable does not guarantee that a query will execute in a streaming fashion. SQL is a declarative language, meaning it specifies 'WHAT' user wants. It is up to the planner 'HOW' to generate the result. In most cases there are many ways to compute the correct result for a given query. The query planner is responsible for choosing "a way" (ideally the best<sup id="optimal1"><a href="#optimal">*</a></sup> one) among the all alternatives to generate [...]
+<h2>Analysis</h2>
+<p>Let's start by creating an example table that we will refer throughout the post. This table models the input data of an operator for the analysis:</p>
+<h3>Example Virtual Table</h3>
+<style>
+ table {
+ border-collapse: collapse;
+ width: 80%;
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+ }
+ th, td {
+ padding: 12px 16px;
+ text-align: left;
+ border-bottom: 1px solid #e0e0e0;
+ }
+ th {
+ background-color: #f9f9f9;
+ font-weight: 600;
+ }
+ tr:hover {
+ background-color: #f1f1f1;
+ }
+</style>
+<table>
+<tr>
+<th>amount</th> <th>price</th> <th>hostname</th><th>currency</th><th>time_bin</th> <th>time</th> <th>price_cloned</th> <th>time_cloned</th>
+</tr>
+<tr>
+<td>12</td> <td>25</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:01:30</td> <td>25</td> <td>08:01:30</td>
+</tr>
+<tr>
+<td>12</td> <td>26</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:11:30</td> <td>26</td> <td>08:11:30</td>
+</tr>
+<tr>
+<td>15</td> <td>30</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:41:30</td> <td>30</td> <td>08:41:30</td>
+</tr>
+<tr>
+<td>15</td> <td>32</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:55:15</td> <td>32</td> <td>08:55:15</td>
+</tr>
+<tr>
+<td>15</td> <td>35</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:10:23</td> <td>35</td> <td>09:10:23</td>
+</tr>
+<tr>
+<td>20</td> <td>18</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:20:33</td> <td>18</td> <td>09:20:33</td>
+</tr>
+<tr>
+<td>20</td> <td>22</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:40:15</td> <td>22</td> <td>09:40:15</td>
+</tr>
+</table>
+<p><br/></p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>How can a table have multiple orderings?</strong> At first glance it may seem counterintuitive for a table to have more than one valid ordering. However, during query execution such scenarios can arise.
+
+For example consider the following query:
+
+
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">time</span><span class="p">,</span> <span class="n">date_bin</span><span class="p">(</span><span class="s1">'1 hour'</span><span class="p">,</span> <span class="k">time</span><span class="p">,</span> <span class="s1">'1970-01-01'</span><span cla [...]
+<span class="k">FROM</span> <span class="k">table</span><span class="p">;</span>
+</code></pre></div>
+
+
+If we know that the table is ordered by <code>time ASC</code> we can infer that <code>time_bin ASC</code> is also a valid ordering. This is because the <code>date_bin</code> function is monotonic, meaning it preserves the order of its input.
+
+DataFusion leverages these functional dependencies to infer new orderings as data flows through different query operators. For details on the implementation see the <a ,="" href="https://github.com/apache/datafusion/blob/main/datafusion/common/src/functional_dependencies.rs" target="_blank">source</a> code.
+</blockquote>
+<p>By inspection, you can see this table is sorted by the <code>amount</code> column, but It is also sorted by <code>time</code> and <code>time_bin</code> as well as the compound <code>(time_bin, amount)</code> and many other variations. While this example is an extreme case, real-world data often has multiple sort orders. </p>
+<p>A naive approach for analyzing whether the ordering requirement of an operator is satisfied by its input would be: </p>
+<ul>
+<li>Store all the valid ordering expressions that the tables satisfies </li>
+<li>Check whether the ordering requirement by the operator is among valid orderings. </li>
+</ul>
+<p>This naive algorithm works and correct. However, listing all valid orderings can be quite lengthy and is of exponential complexity as the number of orderings grows. For the example table here is a (small) subset of the valid orderings:</p>
+<p><code>[amount ASC]</code><br/>
+<code>[amount ASC, price ASC]</code><br/>
+<code>[amount ASC, price_cloned ASC]</code><br/>
+<code>[hostname ASC, amount ASC, price_cloned ASC]</code><br/>
+<code>[amount ASC, hostname ASC, price_cloned ASC]</code><br/>
+<code>[amount ASC, price_cloned ASC, hostname ASC]</code><br/>
+.<br/>
+.<br/>
+. </p>
+<p>As can be seen from the listing above storing all valid orderings is wasteful and contains significant redundancy. Here are some observations which suggest that we can do much better:</p>
+<ul>
+<li>
+<p>Storing a prefix of another valid ordering is redundant. If the table satisfies the lexicographic ordering<sup id="fn1"><a href="#footnote1">1</a></sup>: <code>[amount ASC, price ASC]</code>, it already satisfies ordering <code>[amount ASC]</code> trivially. Hence, once we store <code>[amount ASC, price ASC]</code> storing <code>[amount ASC]</code> is redundant.</p>
+</li>
+<li>
+<p>Using all columns that are equal to each other in the listings is redundant. If we know the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by <code>[amount ASC, price_cloned ASC]</code> since <code>price</code> and <code>price_cloned</code> are copy of each other. It is enough to use just one expression among the expressions that exact copy of each other.</p>
+</li>
+<li>
+<p>Constant expressions can be inserted anywhere in a valid ordering with an arbitrary direction (e.g. <code>ASC</code>, <code>DESC</code>). Hence, if the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by: <br/>
+<code>[hostname ASC, amount ASC, price ASC]</code>,<br/>
+<code>[hostname DESC, amount ASC, price ASC]</code>,<br/>
+<code>[amount ASC, hostname ASC, price ASC]</code>,<br/>
+ .<br/>
+ . </p>
+</li>
+</ul>
+<p>This is clearly redundant. For this reason, it is better to avoid explicitly encoding constant expressions in valid sort orders.</p>
+<p>In summary,</p>
+<ul>
+<li>We should store only the longest lexicographic ordering (shouldn't use any prefix of it)</li>
+<li>Using expressions that are exact copies of each other is redundant.</li>
+<li>Ordering expressions shouldn't contain any constant expression.</li>
+</ul>
+<h2>Key Concepts for Analyzing Orderings</h2>
+<p>To solve the shortcomings above DataFusion needs to track of following properties for the table:</p>
+<ul>
+<li>Constant Expressions </li>
+<li>Equivalent Expression Groups (will be explained shortly)</li>
+<li>Succinct Valid Orderings (will be explained shortly)</li>
+</ul>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> These properties are implemented in the <code>EquivalenceProperties</code> structure in <code>DataFusion</code>, please see the <a href="https://github.com/apache/datafusion/blob/f47ea73b87eec4af044f9b9923baf042682615b2/datafusion/physical-expr/src/equivalence/properties/mod.rs#L134" target="_blank">source</a> for more details<br/>
+</blockquote>
+<p>These properties allow us to analyze whether the ordering requirement is satisfied by the data already.</p>
+<h3>1. Constant Expressions</h3>
+<p>Constant expressions are those where each row in the expression has the same value across all rows. Although constant expressions may seem odd in a table they can arise after operations like <code>Filter</code> or <code>Join</code> occur. </p>
+<p>For instance in the example table:</p>
+<ul>
+<li>Columns <code>hostname</code> and <code>currency</code> are constant because every row in the table has the same value (<code>'app.example.com'</code> for <code>hostname</code>, and <code>'USD'</code> for <code>currency</code>) for these columns.</li>
+</ul>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> Constant expressions can arise during query execution. For example, in following query:<br/>
+<code>SELECT hostname FROM logs</code><br/><code>WHERE hostname='app.example.com'</code> <br/>
+ after filtering is done, for subsequent operators the <code>hostname</code> column will be constant.
+</blockquote>
+<h3>2. Equivalent Expression Groups</h3>
+<p>Equivalent expression groups are expressions that always hold the same value across rows. These expressions can be thought of as clones of each other and may arise from operations like <code>Filter</code>, <code>Join</code>, or <code>Projection</code>.</p>
+<p>In the example table, the expressions <code>price</code> and <code>price_cloned</code> form one equivalence group, and <code>time</code> and <code>time_cloned</code> form another equivalence group.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> Equivalent expression groups can arise during the query execution. For example, in the following query:<br/>
+<code>SELECT time, time as time_cloned FROM logs</code> <br/>
+ after the projection is done, for subsequent operators <code>time</code> and <code>time_cloned</code> will form an equivalence group. As another example, in the following query:<br/>
+<code>SELECT employees.id, employees.name, departments.department_name</code>
+<code>FROM employees</code>
+<code>JOIN departments ON employees.department_id = departments.id;</code> <br/>
+after joining, <code>employees.department_id</code> and <code>departments.id</code> will form an equivalence group.
+</blockquote>
+<h3>3. Succinct Encoding of Valid Orderings</h3>
+<p>Valid orderings are the orderings that the table already satisfies. However, naively listing them requires exponential space as the number of columns grows as discussed before. Instead, we list all valid orderings after following constraints are applied:</p>
+<ul>
+<li>Do not use any constant expressions in the valid ordering construction</li>
+<li>Use only one entry (by convention the first entry) in the equivalent expression group.</li>
+<li>Lexicographic ordering shouldn't contain any leading ordering<sup id="fn2"><a href="#footnote2">2</a></sup>except the first position <sup id="fn3"><a href="#footnote3">3</a></sup>.</li>
+<li>Do not use any prefix of a valid lexicographic ordering<sup id="fn4"><a href="#footnote4">4</a></sup>.</li>
+</ul>
+<p>After applying the first and second constraint, the example table simplifies to </p>
+<style>
+ table {
+ border-collapse: collapse;
+ width: 80%;
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+ }
+ th, td {
+ padding: 12px 16px;
+ text-align: left;
+ border-bottom: 1px solid #e0e0e0;
+ }
+ th {
+ background-color: #f9f9f9;
+ font-weight: 600;
+ }
+ tr:hover {
+ background-color: #f1f1f1;
+ }
+</style>
+<table>
+<tr>
+<th>amount</th> <th>price</th><th>time_bin</th> <th>time</th>
+</tr>
+<tr>
+<td>12</td> <td>25</td><td>08:00:00</td> <td>08:01:30</td>
+</tr>
+<tr>
+<td>12</td> <td>26</td><td>08:00:00</td> <td>08:11:30</td>
+</tr>
+<tr>
+<td>15</td> <td>30</td><td>08:00:00</td> <td>08:41:30</td>
+</tr>
+<tr>
+<td>15</td> <td>32</td><td>08:00:00</td> <td>08:55:15</td>
+</tr>
+<tr>
+<td>15</td> <td>35</td><td>09:00:00</td> <td>09:10:23</td>
+</tr>
+<tr>
+<td>20</td> <td>18</td><td>09:00:00</td> <td>09:20:33</td>
+</tr>
+<tr>
+<td>20</td> <td>22</td><td>09:00:00</td> <td>09:40:15</td>
+</tr>
+</table>
+<p><br/>
+Following third and fourth constraints for the simplified table, the succinct valid orderings are:<br/>
+<code>[amount ASC, price ASC]</code>, <br/>
+<code>[time_bin ASC]</code>,<br/>
+<code>[time ASC]</code> </p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<p><strong>How can DataFusion find orderings?</strong></p>
+DataFusion's <code>CREATE EXTERNAL TABLE</code> has a <code>WITH ORDER</code> clause (see <a href="https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table">docs</a>) to specify the known orderings of the table during table creation. For example the following query:<br/>
+<pre><code>
+CREATE EXTERNAL TABLE source (
+ amount INT NOT NULL,
+ price DOUBLE NOT NULL,
+ time TIMESTAMP NOT NULL,
+ ...
+)
+STORED AS CSV
+WITH ORDER (time ASC)
+WITH ORDER (amount ASC, price ASC)
+LOCATION '/path/to/FILE_NAME.csv'
+OPTIONS ('has_header' 'true');
+</code></pre>
+communicates that <code>source</code> table has the orderings: <code>[time ASC]</code> and <code>[amount ASC, price ASC]</code>.<br/>
+When orderings are communicated from the source, DataFusion tracks the orderings through each operator while optimizing the plan.<br/>
+<ul>
+<li>add new orderings (such as when "date_bin" function is applied to the "time" column)</li>
+<li>Remove orderings, if operation doesn't preserve the ordering of the data at its input</li>
+<li>Update equivalent groups</li>
+<li>Update constant expressions</li>
+</ul>
+
+Figure 1 shows an example how DataFusion generates an efficient plan for the query:
+<pre><code>
+SELECT
+ row_number() OVER (ORDER BY time) as rn,
+ time
+FROM events
+ORDER BY rn, time
+</code></pre>
+using the orderings of the query intermediates.<br/>
+<br/>
+<figure>
+<img alt="Window Query Datafusion Optimization" class="img-responsive" src="/blog/images/ordering_analysis/query_window_plan.png" width="80%"/>
+<figcaption><strong>Figure 1:</strong> DataFusion analyzes orderings of the sources and query intermediates to generate efficient plans</figcaption>
+</figure>
+</blockquote>
+<h3>Table Properties</h3>
+<p>In summary, for the example table, the following properties correctly describe the sort properties:</p>
+<ul>
+<li><strong>Constant Expressions</strong> = <code>hostname, currency</code> </li>
+<li><strong>Equivalent Expression Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code> </li>
+<li><strong>Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code> </li>
+</ul>
+<h3>Algorithm for Analyzing Ordering Requirements</h3>
+<p>After deriving these properties for the data, following algorithm can be used to check whether an ordering requirement is satisfied by the table:</p>
+<ol>
+<li><strong>Prune constant expressions</strong>: Remove any constant expressions from the ordering requirement.</li>
+<li><strong>Normalize the requirement</strong>: Replace each expression in the ordering requirement with the first entry from its equivalence group.</li>
+<li><strong>De-duplicate expressions</strong>: If an expression appears more than once, remove duplicates, keeping only the first occurrence.</li>
+<li><strong>Match leading orderings</strong>: Check whether the leading ordering requirement<sup id="fn5"><a href="#footnote5">5</a></sup> matches the leading valid orderings<sup id="fn6"><a href="#footnote6">6</a></sup> of table. If so:<ul>
+<li>Remove the leading ordering requirement from the ordering requirement </li>
+<li>Remove the matching leading valid ordering from the valid orderings of table. </li>
+</ul>
+</li>
+<li><strong>Iterate through the remaining expressions</strong>: Go back to step 4 until ordering requirement is empty or leading ordering requirement is not found among the leading valid orderings of table.</li>
+</ol>
+<p>If, at the end of the procedure above, the ordering requirement is an empty list, we can conclude that the requirement is satisfied by the table.</p>
+<h3>Example Walkthrough</h3>
+<p>Let's say the user provided a query such as the following:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="k">table</span>
+<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">hostname</span> <span class="k">DESC</span><span class="p">,</span> <span class="n">amount</span> <span class="k">ASC</span><span class="p">,</span> <span class="n">time_bin</span> <span class="k">ASC</span><span class="p">,</span> <span class="n">price_cloned</span> <span [...]
+</code></pre></div>
+<p>And the input has the same properties explained above</p>
+<ul>
+<li><strong>Constant Expressions</strong> = <code>hostname, currency</code></li>
+<li><strong>Equivalent Expressions Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code></li>
+<li><strong>Succinct Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code></li>
+</ul>
+<p>In order to remove a sort the optimizer must check if the ordering requirement <code>[hostname DESC, amount ASC, time_bin ASC, price_cloned ASC, time ASC, currency ASC, price DESC]</code> is satisfied by the properties.</p>
+<h3>Algorithm Steps</h3>
+<ol>
+<li>
+<p><strong>Prune constant expressions</strong>:<br/>
+ Remove <code>hostname</code> and <code>currency</code> from the requirement. The requirement becomes:<br/>
+<code>[amount ASC, time_bin ASC, price_cloned ASC, time ASC, price DESC]</code>.</p>
+</li>
+<li>
+<p><strong>Normalize using equivalent groups</strong>:<br/>
+ Replace <code>price_cloned</code> with <code>price</code> and <code>time_cloned</code> with <code>time</code>. The requirement becomes:<br/>
+<code>[amount ASC, time_bin ASC, price ASC, time ASC, price DESC]</code>.</p>
+</li>
+<li>
+<p><strong>De-duplicate expressions</strong>:<br/>
+ Since <code>price</code> appears twice, we simplify the requirement to:<br/>
+<code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> (keeping the first occurrence from the left side).</p>
+</li>
+<li>
+<p><strong>Match leading orderings</strong>:<br/>
+ Check if leading ordering requirement <code>amount ASC</code> is among the leading valid orderings: <code>amount ASC, time_bin ASC, time ASC</code>. Since this is the case, we remove <code>amount ASC</code> from both the ordering requirement and the valid orderings of the table.</p>
+</li>
+<li><strong>Iterate through the remaining expressions</strong>:
+Now, the problem is converted from<br/>
+<em>"whether the requirement: <code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
+into<br/>
+<em>"whether the requirement: <code>[time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
+We go back to step 4 until the ordering requirement list is exhausted or its length no longer decreases.</li>
+</ol>
+<p>At the end of stages above we end up with an empty ordering requirement list. Given this, we can conclude that the table satisfies the ordering requirement and thus no sort is required. </p>
+<h2>Conclusion</h2>
+<p>In this post, we described the conditions under which an ordering requirement is satisfied based on the properties of a table. We introduced key concepts such as constant expressions, equivalence groups, and valid orderings, and used them to determine whether a given ordering requirement are satisfied by an input table.</p>
+<p>This analysis plays a crucial role in:</p>
+<ul>
+<li>Choosing more efficient algorithm variants</li>
+<li>Generating streaming-friendly plans</li>
+</ul>
+<p>The <code>DataFusion</code> query engine employs this analysis (and many more) during its planning stage to ensure correct and efficient query execution. We <a href="https://datafusion.apache.org/contributor-guide/index.html">welcome you</a> to come and join the project.</p>
+<h2>Appendix</h2>
+<!--
+<p id="footnote1"><sup>[1]</sup>The ordering requirement refers to the condition that input data must be sorted in a certain way for a specific operator to function as intended.</p>
+-->
+<p id="footnote1"><sup>[1]</sup>Lexicographic order is a way of ordering sequences (like strings, list of expressions) based on the order of their components, similar to how words are ordered in a dictionary. It compares each element of the sequences one by one, from left to right.</p>
+<p id="footnote2"><sup>[2]</sup>Leading ordering is the first ordering in a lexicographic ordering list. As an example, for the ordering: <code>[amount ASC, price ASC]</code>, leading ordering will be: <code>amount ASC</code>. </p>
+<p id="footnote3"><sup>[3]</sup>This means that, if we know that <code>[amount ASC]</code> and <code>[time ASC]</code> are both valid orderings for the table. We shouldn't enlist <code>[amount ASC, time ASC]</code> or <code>[time ASC, amount ASC]</code> as valid orderings. These orderings can be deduced if we know that table satisfies the ordering <code>[amount ASC]</code> and <code>[time ASC]</code> [...]
+<p id="footnote4"><sup>[4]</sup>This means that, if ordering <code>[amount ASC, price ASC]</code> is a valid ordering for the table. We shouldn't enlist <code>[amount ASC]</code> as valid ordering. Validity of it can be deduced from the ordering: <code>[amount ASC, price ASC]</code>
+<p id="footnote5"><sup>[5]</sup>Leading ordering requirement is the first ordering requirement in the list of lexicographic ordering requirement expression. As an example for the requirement: <code>[amount ASC, time_bin ASC, prices ASC, time ASC]</code>, leading ordering requirement is: <code>amount ASC</code>.</p>
+<p id="footnote6"><sup>[6]</sup>Leading valid orderings are the first ordering for each valid ordering list in the table. As an example, for the valid orderings: <code>[amount ASC, prices ASC], [time_bin ASC], [time ASC]</code>, leading valid orderings will be: <code>amount ASC, time_bin ASC, time ASC</code>. </p>
+<p id="optimal"><sup>*</sup>Best depends on the use case, <code>DataFusion</code> has many various flags to communicate what user thinks the best plan is (e.g. streamable, fastest, lowest memory, etc.). See <a href="https://datafusion.apache.org/user-guide/configs.html" target="_blank">configurations</a> for detail.</p></p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 45.0.0 Released</title><link h [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development for DataFusion!</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that
+uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
+create new, fast data centric systems such as databases, dataframe libraries,
+machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion&rsquo;s primary design
+goal</a> is to accelerate the creation of other data centric systems, it has a
+reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe library</a>,
+<a href="https://datafusion.apache.org/python/">python library</a> and <a href="https://datafusion.apache.org/user-guide/cli/">command line SQL tool</a>.</p>
+<p>DataFusion's core thesis is that as a community, together we can build much more
+advanced technology than any of us as individuals or companies could do alone.
+Without DataFusion, highly performant vectorized query engines would remain
+the domain of a few large companies and world-class research institutions.
+With DataFusion, we can all build on top of a shared foundation, and focus on
+what makes our projects unique.</p>
+<h2>Community Growth 📈</h2>
+<p>In the last 6 months, between <code>40.0.0</code> and <code>45.0.0</code>, our community continues to
+grow in new and exciting ways.</p>
+<ol>
+<li>We added several PMC members and new committers: <a href="https://github.com/jayzhan211">@jayzhan211</a> and <a href="https://github.com/jonahgao">@jonahgao</a> joined the PMC,
+ <a href="https://github.com/2010YOUY01">@2010YOUY01</a>, <a href="https://github.com/rachelint">@rachelint</a>, <a href="https://github.com/findepi/">@findpi</a>, <a href="https://github.com/iffyio">@iffyio</a>, <a href="https://github.com/goldmedal">@goldmedal</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/Michael-J-Ward">@Michael-J-Ward</a> and <a href="https [...]
+ joined as committers. See the <a href="https://lists.apache.org/list.html?dev@datafusion.apache.org">mailing list</a> for more details.</li>
+<li>In the <a href="https://github.com/apache/arrow-datafusion">core DataFusion repo</a> alone we reviewed and accepted almost 1600 PRs from 206 different
+ committers, created over 1100 issues and closed 751 of them 🚀. All changes are listed in the detailed
+ <a href="https://github.com/apache/datafusion/tree/main/dev/changelog">changelogs</a>.</li>
+<li>DataFusion focused meetups happened in multiple cities around the world: <a href="https://github.com/apache/datafusion/discussions/10341#discussioncomment-10110273">Hangzhou</a>, <a href="https://github.com/apache/datafusion/discussions/11431">Belgrade</a>, <a href="https://github.com/apache/datafusion/discussions/11213">New York</a>,
+ <a href="https://github.com/apache/datafusion/discussions/10348">Seattle</a>, <a href="https://github.com/apache/datafusion/discussions/12894">Chicago</a>, <a href="https://github.com/apache/datafusion/discussions/13165">Boston</a> and <a href="https://github.com/apache/datafusion/discussions/12988">Amsterdam</a> as well as a Rust NYC meetup in NYC focused on DataFusion.</li>
+</ol>
+<!--
+$ git log --pretty=oneline 40.0.0..45.0.0 . | wc -l
+ 1532 (up from 1453)
+
+$ git shortlog -sn 40.0.0..45.0.0 . | wc -l
+ 206 (up from 182)
+
+https://crates.io/crates/datafusion/45.0.0
+DataFusion 45 released Feb 7, 2025
+
+https://crates.io/crates/datafusion/40.0.0
+DataFusion 40 released July 12, 2024
+
+Issues created in this time: 375 open, 751 closed (from 321 open, 781 closed)
+https://github.com/apache/datafusion/issues?q=is%3Aissue+created%3A2024-07-12..2025-02-07
+
+Issues closed: 956 (up from 911)
+https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2024-07-12..2025-02-07
+
+PRs merged in this time 1597 (up from 1490)
+https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2024-07-12..2025-02-07
+
+-->
+<p>DataFusion has put in an application to be part of <a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> with a
+<a href="https://github.com/apache/datafusion/issues/14478">number of ideas</a> for projects with mentors already selected. Additionally, <a href="https://github.com/apache/datafusion/issues/14373">some ideas</a> on
+how to make DataFusion an ideal selection for university database projects such as the
+<a href="https://15445.courses.cs.cmu.edu/spring2025/">CMU database classes</a> have been put forward.</p>
+<p>In addition, DataFusion has been appearing publicly more and more, both online and offline. Here are some highlights:</p>
+<ol>
+<li>A <a href="https://uwheel.rs/post/datafusion_uwheel/">demonstration of how uwheel</a> is integrated into DataFusion</li>
+<li>Integrating StringView into DataFusion - <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">part 1</a> and <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">part 2</a></li>
+<li><a href="https://techontherocks.show/3">Building streams</a> with DataFusion</li>
+<li><a href="https://blog.haoxp.xyz/posts/caching-datafusion">Caching in DataFusion</a>: Don't read twice</li>
+<li><a href="https://blog.haoxp.xyz/posts/parquet-to-arrow/">Parquet pruning in DataFusion</a>: Read no more than you need</li>
+<li>DataFusion is one of <a href="https://www.crn.com/news/software/2024/the-10-coolest-open-source-software-tools-of-2024?page=3">The 10 coolest open source software tools</a></li>
+<li><a href="https://www.denormalized.io/blog/building-databases">Building databases over a weekend</a></li>
+</ol>
+<h2>Improved Performance 🚀</h2>
+<p>DataFusion hit a milestone in its development by becoming <a href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/">the fastest single node engine</a>
+for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">clickbench</a> benchmark for the 43.0.0 release. A <a href="https://github.com/apache/datafusion/issues/12821">lot
+of work</a> went into making this happen! While other engines have subsequently gotten faster,
+displacing DataFusion from the top spot, DataFusion still remains near the top and we <a href="https://github.com/apache/datafusion/issues/14586">are planning
+more improvements</a>.</p>
+<p><img alt="ClickBench performance results over time for DataFusion" class="img-responsive" src="/blog/images/datafusion-45.0.0/performance_over_time.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: ClickBench performance improved over 33% between DataFusion 33
+(released Nov. 2023) and DataFusion 45 (released Feb. 2025). </p>
+<p>The task of <a href="https://github.com/apache/datafusion/issues/10918">integrating</a> the new <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html">Arrow StringView</a> which significantly improves performance
+for workloads that scan, filter and group by variable length string and binary data was completed
+and enabled by default in the past 6 months. The improvement is especially pronounced for Parquet
+files due to <a href="https://github.com/apache/arrow-rs/issues/5530">upstream work in the parquet reader</a>. Kudos to <a href="https://github.com/XiangpengHong">@XiangpengHong</a>, <a href="https://github.com/AriesDevil">@AriesDevil</a>,
+<a href="https://github.com/PsiACE">@PsiACE</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/a10y">@a10y</a>, and <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a> for driving this project.</p>
+<h2>Improved Quality 📋</h2>
+<p>DataFusion continues to improve overall in quality. In addition to ongoing bug
+fixes, one of the most exciting improvements in the last 6 months was the addition of the
+<a href="https://github.com/apache/datafusion/pull/13936">SQLite sqllogictest suite</a> thanks to <a href="https://github.com/Omega359">@Omega359</a>. These tests run over 5 million
+sql statements on every push to the main branch.</p>
+<p>Support for <a href="https://github.com/apache/datafusion/pull/13651">explicitly checking logical plan invariants</a> was added by <a href="https://github.com/wiedld">@wiedld</a> which
+can help catch implicit changes that might cause problems during upgrades.</p>
+<p>We have also started other quality initiatives to make it <a href="https://github.com/apache/datafusion/issues/13525">easier to use DataFusion</a>
+based on <a href="https://glaredb.com/">GlareDB</a>'s experience along with more <a href="https://github.com/apache/datafusion/issues/13661">extensive prerelease testing</a>. </p>
+<h2>Improved Documentation 📚</h2>
+<p>We continue to improve the documentation to make it easier to get started using DataFusion.
+During the last 6 months two projects were initiated to migrate the function documentation
+from strictly static markdown files. First, <a href="https://github.com/apache/datafusion/pull/12668">@Omega359</a> to allow function
+documentation to be generated from code and <a href="https://github.com/jonathanc-n">@jonathanc-n</a> and others helped with the migration,
+then <a href="https://github.com/comphead">@comphead</a> lead a project to <a href="https://github.com/apache/datafusion/pull/12822">create a doc macro</a> to allow for an even easier way to write
+function documentation. A special thanks to <a href="https://github.com/Chen-Yuan-Lai">@Chen-Yuan-Lai</a> for migrating many functions to
+the new syntax.</p>
+<p>Additionally, the <a href="https://github.com/apache/datafusion/pull/13877">examples</a> were <a href="https://github.com/apache/datafusion/pull/13905">refactored</a> and <a href="https://github.com/apache/datafusion/pull/13950">cleaned up</a> to improve their usefulness.</p>
+<h2>New Features ✨</h2>
+<p>There are too many new features in the last 6 months to list them all, but here
+are some highlights:</p>
+<h3>Functions</h3>
+<ul>
+<li>Uniform Window Functions: <code>BuiltInWindowFunctions</code> was removed and all now use UDFs (<a href="https://github.com/jcsherin">@jcsherin</a>)</li>
+<li>Uniform Aggregate Functions: <code>BuiltInAggregateFunctions</code> was removed and all now use UDFs</li>
+<li>As mentioned above function documentation was extracted from the markdown files</li>
+<li>Some new functions and sql support were added including '<a href="https://github.com/apache/datafusion/pull/13799">show functions</a>', '<a href="https://github.com/apache/datafusion/pull/11347">to_local_time</a>',
+ '<a href="https://github.com/apache/datafusion/pull/12970">regexp_count</a>', '<a href="https://github.com/apache/datafusion/pull/11969">map_extract</a>', '<a href="https://github.com/apache/datafusion/pull/12211">array_distance</a>', '<a href="https://github.com/apache/datafusion/pull/12329">array_any_value</a>', '<a href="https://github.com/apache/datafusion/pull/12474">greatest</a>',
+ '<a href="https://github.com/apache/datafusion/pull/13786">least</a>', '<a href="https://github.com/apache/datafusion/pull/14217">arrays_overlap</a>'</li>
+</ul>
+<h3>FFI</h3>
+<ul>
+<li>Foreign Function Interface work has started. This should allow for
+ <a href="https://github.com/apache/datafusion/pull/12920">using table providers</a> across languages and versions of DataFusion. This
+ is especially pertinent for integration with <a href="https://delta-io.github.io/delta-rs/">delta-rs</a> and other table formats.</li>
+</ul>
+<h3>Materialized Views</h3>
+<ul>
+<li><a href="https://github.com/suremarc">@suremarc</a> has added a <a href="https://github.com/datafusion-contrib/datafusion-materialized-views">materialized view implementation</a> in datafusion-contrib 🚀</li>
+</ul>
+<h3>Substrait</h3>
+<ul>
+<li>A lot of work was put into improving and enhancing substrait support (<a href="https://github.com/Blizzara">@Blizzara</a>, <a href="https://github.com/westonpace">@westonpace</a>, <a href="https://github.com/tokoko">@tokoko</a>, <a href="https://github.com/vbarua">@vbarua</a>, <a href="https://github.com/LatrecheYasser">@LatrecheYasser</a>, <a href="https://github.com/notfilippo">@notfilippo</a> and others) [...]
+</ul>
+<h2>Looking Ahead: The Next Six Months ðŸ”</h2>
+<p>One of the long term goals of <a href="https://github.com/alamb">@alamb</a>, DataFusion's PMC chair, has been to have
+<a href="https://www.influxdata.com/blog/datafusion-2025-influxdb/">1000 DataFusion based projects</a>. This may be the year that happens!</p>
+<p>The community has been <a href="https://github.com/apache/datafusion/issues/14580">discussing what we will work on in the next six months</a>.
+Some major initiatives are likely to be:</p>
+<ol>
+<li><em>Performance</em>: A <a href="https://github.com/apache/datafusion/issues/14482">number of items have been identified</a> as areas that could use additional work</li>
+<li><em>Memory usage</em>: Tracking and improving memory usage, statistics and spilling to disk </li>
+<li><em><a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> (GSOC)</em>: DataFusion is hopefully selected as a project and we start accepting and supporting student projects </li>
+<li><em>FFI</em>: Extending the FFI implementation to support to all types of UDF's and SessionContext</li>
+<li><em>Spark Functions</em>: A <a href="https://github.com/apache/datafusion/issues/5600">proposal has been made to add a crate</a> covering spark compatible builtin functions </li>
+</ol>
+<h2>How to Get Involved</h2>
+<p>DataFusion is not a project built or driven by a single person, company, or
+foundation. Rather, our community of users and contributors work together to
+build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us we would love to have you. You can try out
+DataFusion on some of your own data and projects and let us know how it goes,
+contribute suggestions, documentation, bug reports, or a PR with documentation,
+tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
+can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="alternate"></link><published>2025-02-17T00:00:00+00:00</published><updated>2025-02-17T00:00:00+00:00</updated><author><name>pmc</name></author [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/mustafa-akur-andrew-lamb.atom.xml b/blog/feeds/mustafa-akur-andrew-lamb.atom.xml
new file mode 100644
index 0000000..d9ee35a
--- /dev/null
+++ b/blog/feeds/mustafa-akur-andrew-lamb.atom.xml
@@ -0,0 +1,394 @@
+<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Mustafa Akur, Andrew Lamb</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/mustafa-akur-andrew-lamb.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-11T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Using Ordering for Better Plans in Apache DataFusion</title><link href="https://data [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
+</blockquote>
+<p>There are various use cases where this type of analysis can be useful such as the following examples.</p>
+<h3>Removing Unnecessary Sorts</h3>
+<p>Imagine a user wants to execute the following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">log_line</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="k">time</span> <span class="k">ASC</span> <span class="k">limit</span> <span class="mi">10</span>
+</code></pre></div>
+<p>If we don't know anything about the <code>telemetry</code> table we need to sort it by <code>time ASC</code> and then retrieve the first 10 rows to get the correct result. However, if the table is already ordered by <code>time ASC</code>, we can simply retrieve the first 10 rows. This approach executes much faster and uses less memory compared to resorting the entire table, even when the <a href="https://docs.rs/datafusion/latest/datafusion [...]
+<p>In order to avoid the sort the query optimizer must determine the data is already sorted. For simple queries the analysis is straightforward however it gets complicated fast. For example, what if your data is sorted by <code>[hostname, time ASC]</code> and your query is</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">log_line</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">WHERE</span> <span class="n">hostname</span> <span class="o">=</span> <span class="s1">'app.example.com'</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="k">time</span> <span class="k">ASC</span><span class="p">;</span>
+</code></pre></div>
+<p>In this case a sort still isn't needed, but the analysis must reason about the sortedness of the stream when it knows <code>hostname</code> has a single value.</p>
+<h3>Optimized Operator Implementations</h3>
+<p>As another use case, some operators can utilize the ordering information to change its underlying algorithm to execute more efficiently. Consider the following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">log_line</span><span class="p">)</span>
+<span class="k">FROM</span> <span class="n">telemetry</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="n">hostname</span><span class="p">;</span>
+</code></pre></div>
+<p>Most analytic systems, including DataFusion, by default implement such a query using a hash table keyed on values of <code>hostname</code> to store the counts. However, if the <code>telemetry</code> table is sorted by <code>hostname</code>, there are much more efficient algorithms for grouping on <code>hostname</code> values than hashing every value and storing it in memory. However, the more efficient algorithm can only be used w [...]
+<h3>Streaming-Friendly Execution</h3>
+<p>Stream processing aims to produce results immediately as they become available ensuring minimal latency for real-time workloads. However, some operators need to consume all input data before producing any output. Consider the <code>Sort</code> operation: before it can start generating output, the algorithm must first process all input data. As a result, data flow halts whenever such an operator is encountered until all input is consumed. When a physical query plan co [...]
+<p>For a query to be executed in a streaming fashion we need to satisfy 2 conditions:</p>
+<p><strong>Logically Streamable</strong><br/>
+It should be possible to generate what user wants in streaming fashion. Consider following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">SUM</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
+<span class="k">FROM</span> <span class="n">orders</span>
+</code></pre></div>
+<p>Here, the user wants to compute the sum of all amounts in the orders table. By the nature of the query this requires scanning the entire table to generate a result making it impossible to execute in a streaming fashion.</p>
+<p><strong>Streaming Aware Planner</strong><br/>
+Being logically streamable does not guarantee that a query will execute in a streaming fashion. SQL is a declarative language, meaning it specifies 'WHAT' user wants. It is up to the planner 'HOW' to generate the result. In most cases there are many ways to compute the correct result for a given query. The query planner is responsible for choosing "a way" (ideally the best<sup id="optimal1"><a href="#optimal">*</a></sup> one) among the all alternatives to generate [...]
+<h2>Analysis</h2>
+<p>Let's start by creating an example table that we will refer throughout the post. This table models the input data of an operator for the analysis:</p>
+<h3>Example Virtual Table</h3>
+<style>
+ table {
+ border-collapse: collapse;
+ width: 80%;
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+ }
+ th, td {
+ padding: 12px 16px;
+ text-align: left;
+ border-bottom: 1px solid #e0e0e0;
+ }
+ th {
+ background-color: #f9f9f9;
+ font-weight: 600;
+ }
+ tr:hover {
+ background-color: #f1f1f1;
+ }
+</style>
+<table>
+<tr>
+<th>amount</th> <th>price</th> <th>hostname</th><th>currency</th><th>time_bin</th> <th>time</th> <th>price_cloned</th> <th>time_cloned</th>
+</tr>
+<tr>
+<td>12</td> <td>25</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:01:30</td> <td>25</td> <td>08:01:30</td>
+</tr>
+<tr>
+<td>12</td> <td>26</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:11:30</td> <td>26</td> <td>08:11:30</td>
+</tr>
+<tr>
+<td>15</td> <td>30</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:41:30</td> <td>30</td> <td>08:41:30</td>
+</tr>
+<tr>
+<td>15</td> <td>32</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:55:15</td> <td>32</td> <td>08:55:15</td>
+</tr>
+<tr>
+<td>15</td> <td>35</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:10:23</td> <td>35</td> <td>09:10:23</td>
+</tr>
+<tr>
+<td>20</td> <td>18</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:20:33</td> <td>18</td> <td>09:20:33</td>
+</tr>
+<tr>
+<td>20</td> <td>22</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:40:15</td> <td>22</td> <td>09:40:15</td>
+</tr>
+</table>
+<p><br/></p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>How can a table have multiple orderings?</strong> At first glance it may seem counterintuitive for a table to have more than one valid ordering. However, during query execution such scenarios can arise.
+
+For example consider the following query:
+
+
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="k">time</span><span class="p">,</span> <span class="n">date_bin</span><span class="p">(</span><span class="s1">'1 hour'</span><span class="p">,</span> <span class="k">time</span><span class="p">,</span> <span class="s1">'1970-01-01'</span><span cla [...]
+<span class="k">FROM</span> <span class="k">table</span><span class="p">;</span>
+</code></pre></div>
+
+
+If we know that the table is ordered by <code>time ASC</code> we can infer that <code>time_bin ASC</code> is also a valid ordering. This is because the <code>date_bin</code> function is monotonic, meaning it preserves the order of its input.
+
+DataFusion leverages these functional dependencies to infer new orderings as data flows through different query operators. For details on the implementation see the <a ,="" href="https://github.com/apache/datafusion/blob/main/datafusion/common/src/functional_dependencies.rs" target="_blank">source</a> code.
+</blockquote>
+<p>By inspection, you can see this table is sorted by the <code>amount</code> column, but It is also sorted by <code>time</code> and <code>time_bin</code> as well as the compound <code>(time_bin, amount)</code> and many other variations. While this example is an extreme case, real-world data often has multiple sort orders. </p>
+<p>A naive approach for analyzing whether the ordering requirement of an operator is satisfied by its input would be: </p>
+<ul>
+<li>Store all the valid ordering expressions that the tables satisfies </li>
+<li>Check whether the ordering requirement by the operator is among valid orderings. </li>
+</ul>
+<p>This naive algorithm works and correct. However, listing all valid orderings can be quite lengthy and is of exponential complexity as the number of orderings grows. For the example table here is a (small) subset of the valid orderings:</p>
+<p><code>[amount ASC]</code><br/>
+<code>[amount ASC, price ASC]</code><br/>
+<code>[amount ASC, price_cloned ASC]</code><br/>
+<code>[hostname ASC, amount ASC, price_cloned ASC]</code><br/>
+<code>[amount ASC, hostname ASC, price_cloned ASC]</code><br/>
+<code>[amount ASC, price_cloned ASC, hostname ASC]</code><br/>
+.<br/>
+.<br/>
+. </p>
+<p>As can be seen from the listing above storing all valid orderings is wasteful and contains significant redundancy. Here are some observations which suggest that we can do much better:</p>
+<ul>
+<li>
+<p>Storing a prefix of another valid ordering is redundant. If the table satisfies the lexicographic ordering<sup id="fn1"><a href="#footnote1">1</a></sup>: <code>[amount ASC, price ASC]</code>, it already satisfies ordering <code>[amount ASC]</code> trivially. Hence, once we store <code>[amount ASC, price ASC]</code> storing <code>[amount ASC]</code> is redundant.</p>
+</li>
+<li>
+<p>Using all columns that are equal to each other in the listings is redundant. If we know the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by <code>[amount ASC, price_cloned ASC]</code> since <code>price</code> and <code>price_cloned</code> are copy of each other. It is enough to use just one expression among the expressions that exact copy of each other.</p>
+</li>
+<li>
+<p>Constant expressions can be inserted anywhere in a valid ordering with an arbitrary direction (e.g. <code>ASC</code>, <code>DESC</code>). Hence, if the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by: <br/>
+<code>[hostname ASC, amount ASC, price ASC]</code>,<br/>
+<code>[hostname DESC, amount ASC, price ASC]</code>,<br/>
+<code>[amount ASC, hostname ASC, price ASC]</code>,<br/>
+ .<br/>
+ . </p>
+</li>
+</ul>
+<p>This is clearly redundant. For this reason, it is better to avoid explicitly encoding constant expressions in valid sort orders.</p>
+<p>In summary,</p>
+<ul>
+<li>We should store only the longest lexicographic ordering (shouldn't use any prefix of it)</li>
+<li>Using expressions that are exact copies of each other is redundant.</li>
+<li>Ordering expressions shouldn't contain any constant expression.</li>
+</ul>
+<h2>Key Concepts for Analyzing Orderings</h2>
+<p>To solve the shortcomings above DataFusion needs to track of following properties for the table:</p>
+<ul>
+<li>Constant Expressions </li>
+<li>Equivalent Expression Groups (will be explained shortly)</li>
+<li>Succinct Valid Orderings (will be explained shortly)</li>
+</ul>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> These properties are implemented in the <code>EquivalenceProperties</code> structure in <code>DataFusion</code>, please see the <a href="https://github.com/apache/datafusion/blob/f47ea73b87eec4af044f9b9923baf042682615b2/datafusion/physical-expr/src/equivalence/properties/mod.rs#L134" target="_blank">source</a> for more details<br/>
+</blockquote>
+<p>These properties allow us to analyze whether the ordering requirement is satisfied by the data already.</p>
+<h3>1. Constant Expressions</h3>
+<p>Constant expressions are those where each row in the expression has the same value across all rows. Although constant expressions may seem odd in a table they can arise after operations like <code>Filter</code> or <code>Join</code> occur. </p>
+<p>For instance in the example table:</p>
+<ul>
+<li>Columns <code>hostname</code> and <code>currency</code> are constant because every row in the table has the same value (<code>'app.example.com'</code> for <code>hostname</code>, and <code>'USD'</code> for <code>currency</code>) for these columns.</li>
+</ul>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> Constant expressions can arise during query execution. For example, in following query:<br/>
+<code>SELECT hostname FROM logs</code><br/><code>WHERE hostname='app.example.com'</code> <br/>
+ after filtering is done, for subsequent operators the <code>hostname</code> column will be constant.
+</blockquote>
+<h3>2. Equivalent Expression Groups</h3>
+<p>Equivalent expression groups are expressions that always hold the same value across rows. These expressions can be thought of as clones of each other and may arise from operations like <code>Filter</code>, <code>Join</code>, or <code>Projection</code>.</p>
+<p>In the example table, the expressions <code>price</code> and <code>price_cloned</code> form one equivalence group, and <code>time</code> and <code>time_cloned</code> form another equivalence group.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Note:</strong> Equivalent expression groups can arise during the query execution. For example, in the following query:<br/>
+<code>SELECT time, time as time_cloned FROM logs</code> <br/>
+ after the projection is done, for subsequent operators <code>time</code> and <code>time_cloned</code> will form an equivalence group. As another example, in the following query:<br/>
+<code>SELECT employees.id, employees.name, departments.department_name</code>
+<code>FROM employees</code>
+<code>JOIN departments ON employees.department_id = departments.id;</code> <br/>
+after joining, <code>employees.department_id</code> and <code>departments.id</code> will form an equivalence group.
+</blockquote>
+<h3>3. Succinct Encoding of Valid Orderings</h3>
+<p>Valid orderings are the orderings that the table already satisfies. However, naively listing them requires exponential space as the number of columns grows as discussed before. Instead, we list all valid orderings after following constraints are applied:</p>
+<ul>
+<li>Do not use any constant expressions in the valid ordering construction</li>
+<li>Use only one entry (by convention the first entry) in the equivalent expression group.</li>
+<li>Lexicographic ordering shouldn't contain any leading ordering<sup id="fn2"><a href="#footnote2">2</a></sup>except the first position <sup id="fn3"><a href="#footnote3">3</a></sup>.</li>
+<li>Do not use any prefix of a valid lexicographic ordering<sup id="fn4"><a href="#footnote4">4</a></sup>.</li>
+</ul>
+<p>After applying the first and second constraint, the example table simplifies to </p>
+<style>
+ table {
+ border-collapse: collapse;
+ width: 80%;
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+ }
+ th, td {
+ padding: 12px 16px;
+ text-align: left;
+ border-bottom: 1px solid #e0e0e0;
+ }
+ th {
+ background-color: #f9f9f9;
+ font-weight: 600;
+ }
+ tr:hover {
+ background-color: #f1f1f1;
+ }
+</style>
+<table>
+<tr>
+<th>amount</th> <th>price</th><th>time_bin</th> <th>time</th>
+</tr>
+<tr>
+<td>12</td> <td>25</td><td>08:00:00</td> <td>08:01:30</td>
+</tr>
+<tr>
+<td>12</td> <td>26</td><td>08:00:00</td> <td>08:11:30</td>
+</tr>
+<tr>
+<td>15</td> <td>30</td><td>08:00:00</td> <td>08:41:30</td>
+</tr>
+<tr>
+<td>15</td> <td>32</td><td>08:00:00</td> <td>08:55:15</td>
+</tr>
+<tr>
+<td>15</td> <td>35</td><td>09:00:00</td> <td>09:10:23</td>
+</tr>
+<tr>
+<td>20</td> <td>18</td><td>09:00:00</td> <td>09:20:33</td>
+</tr>
+<tr>
+<td>20</td> <td>22</td><td>09:00:00</td> <td>09:40:15</td>
+</tr>
+</table>
+<p><br/>
+Following third and fourth constraints for the simplified table, the succinct valid orderings are:<br/>
+<code>[amount ASC, price ASC]</code>, <br/>
+<code>[time_bin ASC]</code>,<br/>
+<code>[time ASC]</code> </p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<p><strong>How can DataFusion find orderings?</strong></p>
+DataFusion's <code>CREATE EXTERNAL TABLE</code> has a <code>WITH ORDER</code> clause (see <a href="https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table">docs</a>) to specify the known orderings of the table during table creation. For example the following query:<br/>
+<pre><code>
+CREATE EXTERNAL TABLE source (
+ amount INT NOT NULL,
+ price DOUBLE NOT NULL,
+ time TIMESTAMP NOT NULL,
+ ...
+)
+STORED AS CSV
+WITH ORDER (time ASC)
+WITH ORDER (amount ASC, price ASC)
+LOCATION '/path/to/FILE_NAME.csv'
+OPTIONS ('has_header' 'true');
+</code></pre>
+communicates that <code>source</code> table has the orderings: <code>[time ASC]</code> and <code>[amount ASC, price ASC]</code>.<br/>
+When orderings are communicated from the source, DataFusion tracks the orderings through each operator while optimizing the plan.<br/>
+<ul>
+<li>add new orderings (such as when "date_bin" function is applied to the "time" column)</li>
+<li>Remove orderings, if operation doesn't preserve the ordering of the data at its input</li>
+<li>Update equivalent groups</li>
+<li>Update constant expressions</li>
+</ul>
+
+Figure 1 shows an example how DataFusion generates an efficient plan for the query:
+<pre><code>
+SELECT
+ row_number() OVER (ORDER BY time) as rn,
+ time
+FROM events
+ORDER BY rn, time
+</code></pre>
+using the orderings of the query intermediates.<br/>
+<br/>
+<figure>
+<img alt="Window Query Datafusion Optimization" class="img-responsive" src="/blog/images/ordering_analysis/query_window_plan.png" width="80%"/>
+<figcaption><strong>Figure 1:</strong> DataFusion analyzes orderings of the sources and query intermediates to generate efficient plans</figcaption>
+</figure>
+</blockquote>
+<h3>Table Properties</h3>
+<p>In summary, for the example table, the following properties correctly describe the sort properties:</p>
+<ul>
+<li><strong>Constant Expressions</strong> = <code>hostname, currency</code> </li>
+<li><strong>Equivalent Expression Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code> </li>
+<li><strong>Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code> </li>
+</ul>
+<h3>Algorithm for Analyzing Ordering Requirements</h3>
+<p>After deriving these properties for the data, following algorithm can be used to check whether an ordering requirement is satisfied by the table:</p>
+<ol>
+<li><strong>Prune constant expressions</strong>: Remove any constant expressions from the ordering requirement.</li>
+<li><strong>Normalize the requirement</strong>: Replace each expression in the ordering requirement with the first entry from its equivalence group.</li>
+<li><strong>De-duplicate expressions</strong>: If an expression appears more than once, remove duplicates, keeping only the first occurrence.</li>
+<li><strong>Match leading orderings</strong>: Check whether the leading ordering requirement<sup id="fn5"><a href="#footnote5">5</a></sup> matches the leading valid orderings<sup id="fn6"><a href="#footnote6">6</a></sup> of table. If so:<ul>
+<li>Remove the leading ordering requirement from the ordering requirement </li>
+<li>Remove the matching leading valid ordering from the valid orderings of table. </li>
+</ul>
+</li>
+<li><strong>Iterate through the remaining expressions</strong>: Go back to step 4 until ordering requirement is empty or leading ordering requirement is not found among the leading valid orderings of table.</li>
+</ol>
+<p>If, at the end of the procedure above, the ordering requirement is an empty list, we can conclude that the requirement is satisfied by the table.</p>
+<h3>Example Walkthrough</h3>
+<p>Let's say the user provided a query such as the following:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="k">table</span>
+<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">hostname</span> <span class="k">DESC</span><span class="p">,</span> <span class="n">amount</span> <span class="k">ASC</span><span class="p">,</span> <span class="n">time_bin</span> <span class="k">ASC</span><span class="p">,</span> <span class="n">price_cloned</span> <span [...]
+</code></pre></div>
+<p>And the input has the same properties explained above</p>
+<ul>
+<li><strong>Constant Expressions</strong> = <code>hostname, currency</code></li>
+<li><strong>Equivalent Expressions Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code></li>
+<li><strong>Succinct Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code></li>
+</ul>
+<p>In order to remove a sort the optimizer must check if the ordering requirement <code>[hostname DESC, amount ASC, time_bin ASC, price_cloned ASC, time ASC, currency ASC, price DESC]</code> is satisfied by the properties.</p>
+<h3>Algorithm Steps</h3>
+<ol>
+<li>
+<p><strong>Prune constant expressions</strong>:<br/>
+ Remove <code>hostname</code> and <code>currency</code> from the requirement. The requirement becomes:<br/>
+<code>[amount ASC, time_bin ASC, price_cloned ASC, time ASC, price DESC]</code>.</p>
+</li>
+<li>
+<p><strong>Normalize using equivalent groups</strong>:<br/>
+ Replace <code>price_cloned</code> with <code>price</code> and <code>time_cloned</code> with <code>time</code>. The requirement becomes:<br/>
+<code>[amount ASC, time_bin ASC, price ASC, time ASC, price DESC]</code>.</p>
+</li>
+<li>
+<p><strong>De-duplicate expressions</strong>:<br/>
+ Since <code>price</code> appears twice, we simplify the requirement to:<br/>
+<code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> (keeping the first occurrence from the left side).</p>
+</li>
+<li>
+<p><strong>Match leading orderings</strong>:<br/>
+ Check if leading ordering requirement <code>amount ASC</code> is among the leading valid orderings: <code>amount ASC, time_bin ASC, time ASC</code>. Since this is the case, we remove <code>amount ASC</code> from both the ordering requirement and the valid orderings of the table.</p>
+</li>
+<li><strong>Iterate through the remaining expressions</strong>:
+Now, the problem is converted from<br/>
+<em>"whether the requirement: <code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
+into<br/>
+<em>"whether the requirement: <code>[time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
+We go back to step 4 until the ordering requirement list is exhausted or its length no longer decreases.</li>
+</ol>
+<p>At the end of stages above we end up with an empty ordering requirement list. Given this, we can conclude that the table satisfies the ordering requirement and thus no sort is required. </p>
+<h2>Conclusion</h2>
+<p>In this post, we described the conditions under which an ordering requirement is satisfied based on the properties of a table. We introduced key concepts such as constant expressions, equivalence groups, and valid orderings, and used them to determine whether a given ordering requirement are satisfied by an input table.</p>
+<p>This analysis plays a crucial role in:</p>
+<ul>
+<li>Choosing more efficient algorithm variants</li>
+<li>Generating streaming-friendly plans</li>
+</ul>
+<p>The <code>DataFusion</code> query engine employs this analysis (and many more) during its planning stage to ensure correct and efficient query execution. We <a href="https://datafusion.apache.org/contributor-guide/index.html">welcome you</a> to come and join the project.</p>
+<h2>Appendix</h2>
+<!--
+<p id="footnote1"><sup>[1]</sup>The ordering requirement refers to the condition that input data must be sorted in a certain way for a specific operator to function as intended.</p>
+-->
+<p id="footnote1"><sup>[1]</sup>Lexicographic order is a way of ordering sequences (like strings, list of expressions) based on the order of their components, similar to how words are ordered in a dictionary. It compares each element of the sequences one by one, from left to right.</p>
+<p id="footnote2"><sup>[2]</sup>Leading ordering is the first ordering in a lexicographic ordering list. As an example, for the ordering: <code>[amount ASC, price ASC]</code>, leading ordering will be: <code>amount ASC</code>. </p>
+<p id="footnote3"><sup>[3]</sup>This means that, if we know that <code>[amount ASC]</code> and <code>[time ASC]</code> are both valid orderings for the table. We shouldn't enlist <code>[amount ASC, time ASC]</code> or <code>[time ASC, amount ASC]</code> as valid orderings. These orderings can be deduced if we know that table satisfies the ordering <code>[amount ASC]</code> and <code>[time ASC]</code> [...]
+<p id="footnote4"><sup>[4]</sup>This means that, if ordering <code>[amount ASC, price ASC]</code> is a valid ordering for the table. We shouldn't enlist <code>[amount ASC]</code> as valid ordering. Validity of it can be deduced from the ordering: <code>[amount ASC, price ASC]</code>
+<p id="footnote5"><sup>[5]</sup>Leading ordering requirement is the first ordering requirement in the list of lexicographic ordering requirement expression. As an example for the requirement: <code>[amount ASC, time_bin ASC, prices ASC, time ASC]</code>, leading ordering requirement is: <code>amount ASC</code>.</p>
+<p id="footnote6"><sup>[6]</sup>Leading valid orderings are the first ordering for each valid ordering list in the table. As an example, for the valid orderings: <code>[amount ASC, prices ASC], [time_bin ASC], [time ASC]</code>, leading valid orderings will be: <code>amount ASC, time_bin ASC, time ASC</code>. </p>
+<p id="optimal"><sup>*</sup>Best depends on the use case, <code>DataFusion</code> has many various flags to communicate what user thinks the best plan is (e.g. streamable, fastest, lowest memory, etc.). See <a href="https://datafusion.apache.org/user-guide/configs.html" target="_blank">configurations</a> for detail.</p></p></content><category term="blog"></category></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/mustafa-akur-andrew-lamb.rss.xml b/blog/feeds/mustafa-akur-andrew-lamb.rss.xml
new file mode 100644
index 0000000..df5536d
--- /dev/null
+++ b/blog/feeds/mustafa-akur-andrew-lamb.rss.xml
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="utf-8"?>
+<rss version="2.0"><channel><title>Apache DataFusion Blog - Mustafa Akur, Andrew Lamb</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Tue, 11 Mar 2025 00:00:00 +0000</lastBuildDate><item><title>Using Ordering for Better Plans in Apache DataFusion</title><link>https://datafusion.apache.org/blog/2025/03/11/ordering-analysis</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Mustafa Akur, Andrew Lamb</dc:creator><pubDate>Tue, 11 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-11:/blog/2025/03/11/ordering-analysis</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/feeds/pmc.atom.xml b/blog/feeds/pmc.atom.xml
index 2f91701..3ca7d1c 100644
--- a/blog/feeds/pmc.atom.xml
+++ b/blog/feeds/pmc.atom.xml
@@ -1,5 +1,292 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - pmc</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-02-17T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - pmc</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-20T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion Comet 0.7.0 Release</title><link href="https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0" [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
+expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
+to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
+<p>This release covers approximately four weeks of development work and is the result of merging 46 PRs from 11
+contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.7.0.md">change log</a> for more information.</p>
+<h2>Release Highlights</h2>
+<h3>Performance</h3>
+<p>Comet 0.7.0 has improved performance compared to the previous release due to improvements in the native shuffle
+implementation and performance improvements in DataFusion 46.</p>
+<p>For single-node TPC-H at 100 GB, Comet now delivers a <strong>2.2x speedup</strong> compared to Spark using the same CPU and RAM. Even
+with <strong>half the resources</strong>, Comet still provides a measurable performance improvement.</p>
+<p><img alt="Chart showing TPC-H benchmark results for Comet 0.7.0" class="img-responsive" src="/blog/images/comet-0.7.0/performance.png" width="100%"/></p>
+<p><em>These benchmarks were performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and data
+stored locally in Parquet format on NVMe storage. Spark was running in Kubernetes with hard memory limits.</em></p>
+<h2>Shuffle Improvements</h2>
+<p>There are several improvements to shuffle in this release:</p>
+<ul>
+<li>When running in off-heap mode (which is the recommended approach), Comet was using the wrong memory allocator
+ implementation for some types of shuffle operation, which could result in OOM rather than spilling to disk.</li>
+<li>The number of spill files is drastically reduced. In previous releases, each instance of ShuffleMapTask could
+ potentially create a new spill file for each output partition each time that spill was invoked. Comet now creates
+ a maximum of one spill file per output partition per instance of ShuffleMapTask, which is appended to in subsequent
+ spills.</li>
+<li>There was a flaw in the memory accounting which resulted in Comet requesting approximately twice the amount of
+ memory that was needed, resulting in premature spilling. This is now resolved.</li>
+<li>The metric for number of spilled bytes is now accurate. It was previously reporting invalid information.</li>
+</ul>
+<h2>Improved Hash Join Performance</h2>
+<p>When using the <code>spark.comet.exec.replaceSortMergeJoin</code> setting to replace sort-merge joins with hash joins, Comet
+will now do a better job of picking the optimal build side (thanks to [@hayman42] for suggesting this, and thanks to the
+[Apache Gluten(incubating)] project for the inspiration in implementing this feature).</p>
+<p><a href="https://github.com/hayman42">@hayman42</a></p>
+<h2>Experimental Support for DataFusion&rsquo;s DataSourceExec</h2>
+<p>It is now possible to configure Comet to use DataFusion&rsquo;s <code>DataSourceExec</code> instead of Comet&rsquo;s current Parquet reader.
+Support should still be considered experimental, but most of Comet&rsquo;s unit tests are now passing with the new reader.
+Known issues include handling of <code>INT96</code> timestamps and negative bytes and shorts.</p>
+<p>To enable DataFusion&rsquo;s <code>DataSourceExec</code>, either set <code>spark.comet.scan.impl=native_datafusion</code> or set the environment
+variable <code>COMET_PARQUET_SCAN_IMPL=native_datafusion</code>.</p>
+<h2>Complex Type Support</h2>
+<p>With DataFusion&rsquo;s <code>DataSourceExec</code> enabled, there is now some early support for reading structs from Parquet. This is
+largely untested and we would welcome additional testing from the community to help determine what is and isn&rsquo;t working,
+as well as contributions to improve support for structs and other complex types. The tracking issue is
+https://github.com/apache/datafusion-comet/issues/1043.</p>
+<h2>Updates to supported Spark versions</h2>
+<ul>
+<li>Comet 0.7.0 is now tested against Spark 3.5.4 rather than 3.5.1</li>
+<li>This will be the last Comet release to support Spark 3.3.x</li>
+</ul>
+<h2>Improved Tuning Guide</h2>
+<p>The <a href="https://datafusion.apache.org/comet/user-guide/tuning.html">Comet Tuning Guide</a> has been improved and now provides guidance on determining how much memory to allocate to
+Comet.</p>
+<h2>Getting Involved</h2>
+<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
+project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
+<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
+performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
+Comet.</p>
+<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion 45.0.0 Released</title><link href="https://datafusion.apache.org/blog/2025/02/20/datafusion-45.0.0" rel="alternate"></link><published>2025-02-20T00:00:00+00:00</published><updated>2025-02-20T00:00:00+00:00</updated><author><name>pmc</name></autho [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development for DataFusion!</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that
+uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
+create new, fast data centric systems such as databases, dataframe libraries,
+machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion&rsquo;s primary design
+goal</a> is to accelerate the creation of other data centric systems, it has a
+reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe library</a>,
+<a href="https://datafusion.apache.org/python/">python library</a> and <a href="https://datafusion.apache.org/user-guide/cli/">command line SQL tool</a>.</p>
+<p>DataFusion's core thesis is that as a community, together we can build much more
+advanced technology than any of us as individuals or companies could do alone.
+Without DataFusion, highly performant vectorized query engines would remain
+the domain of a few large companies and world-class research institutions.
+With DataFusion, we can all build on top of a shared foundation, and focus on
+what makes our projects unique.</p>
+<h2>Community Growth 📈</h2>
+<p>In the last 6 months, between <code>40.0.0</code> and <code>45.0.0</code>, our community continues to
+grow in new and exciting ways.</p>
+<ol>
+<li>We added several PMC members and new committers: <a href="https://github.com/jayzhan211">@jayzhan211</a> and <a href="https://github.com/jonahgao">@jonahgao</a> joined the PMC,
+ <a href="https://github.com/2010YOUY01">@2010YOUY01</a>, <a href="https://github.com/rachelint">@rachelint</a>, <a href="https://github.com/findepi/">@findpi</a>, <a href="https://github.com/iffyio">@iffyio</a>, <a href="https://github.com/goldmedal">@goldmedal</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/Michael-J-Ward">@Michael-J-Ward</a> and <a href="https [...]
+ joined as committers. See the <a href="https://lists.apache.org/list.html?dev@datafusion.apache.org">mailing list</a> for more details.</li>
+<li>In the <a href="https://github.com/apache/arrow-datafusion">core DataFusion repo</a> alone we reviewed and accepted almost 1600 PRs from 206 different
+ committers, created over 1100 issues and closed 751 of them 🚀. All changes are listed in the detailed
+ <a href="https://github.com/apache/datafusion/tree/main/dev/changelog">changelogs</a>.</li>
+<li>DataFusion focused meetups happened in multiple cities around the world: <a href="https://github.com/apache/datafusion/discussions/10341#discussioncomment-10110273">Hangzhou</a>, <a href="https://github.com/apache/datafusion/discussions/11431">Belgrade</a>, <a href="https://github.com/apache/datafusion/discussions/11213">New York</a>,
+ <a href="https://github.com/apache/datafusion/discussions/10348">Seattle</a>, <a href="https://github.com/apache/datafusion/discussions/12894">Chicago</a>, <a href="https://github.com/apache/datafusion/discussions/13165">Boston</a> and <a href="https://github.com/apache/datafusion/discussions/12988">Amsterdam</a> as well as a Rust NYC meetup in NYC focused on DataFusion.</li>
+</ol>
+<!--
+$ git log --pretty=oneline 40.0.0..45.0.0 . | wc -l
+ 1532 (up from 1453)
+
+$ git shortlog -sn 40.0.0..45.0.0 . | wc -l
+ 206 (up from 182)
+
+https://crates.io/crates/datafusion/45.0.0
+DataFusion 45 released Feb 7, 2025
+
+https://crates.io/crates/datafusion/40.0.0
+DataFusion 40 released July 12, 2024
+
+Issues created in this time: 375 open, 751 closed (from 321 open, 781 closed)
+https://github.com/apache/datafusion/issues?q=is%3Aissue+created%3A2024-07-12..2025-02-07
+
+Issues closed: 956 (up from 911)
+https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2024-07-12..2025-02-07
+
+PRs merged in this time 1597 (up from 1490)
+https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2024-07-12..2025-02-07
+
+-->
+<p>DataFusion has put in an application to be part of <a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> with a
+<a href="https://github.com/apache/datafusion/issues/14478">number of ideas</a> for projects with mentors already selected. Additionally, <a href="https://github.com/apache/datafusion/issues/14373">some ideas</a> on
+how to make DataFusion an ideal selection for university database projects such as the
+<a href="https://15445.courses.cs.cmu.edu/spring2025/">CMU database classes</a> have been put forward.</p>
+<p>In addition, DataFusion has been appearing publicly more and more, both online and offline. Here are some highlights:</p>
+<ol>
+<li>A <a href="https://uwheel.rs/post/datafusion_uwheel/">demonstration of how uwheel</a> is integrated into DataFusion</li>
+<li>Integrating StringView into DataFusion - <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">part 1</a> and <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">part 2</a></li>
+<li><a href="https://techontherocks.show/3">Building streams</a> with DataFusion</li>
+<li><a href="https://blog.haoxp.xyz/posts/caching-datafusion">Caching in DataFusion</a>: Don't read twice</li>
+<li><a href="https://blog.haoxp.xyz/posts/parquet-to-arrow/">Parquet pruning in DataFusion</a>: Read no more than you need</li>
+<li>DataFusion is one of <a href="https://www.crn.com/news/software/2024/the-10-coolest-open-source-software-tools-of-2024?page=3">The 10 coolest open source software tools</a></li>
+<li><a href="https://www.denormalized.io/blog/building-databases">Building databases over a weekend</a></li>
+</ol>
+<h2>Improved Performance 🚀</h2>
+<p>DataFusion hit a milestone in its development by becoming <a href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/">the fastest single node engine</a>
+for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">clickbench</a> benchmark for the 43.0.0 release. A <a href="https://github.com/apache/datafusion/issues/12821">lot
+of work</a> went into making this happen! While other engines have subsequently gotten faster,
+displacing DataFusion from the top spot, DataFusion still remains near the top and we <a href="https://github.com/apache/datafusion/issues/14586">are planning
+more improvements</a>.</p>
+<p><img alt="ClickBench performance results over time for DataFusion" class="img-responsive" src="/blog/images/datafusion-45.0.0/performance_over_time.png" width="100%"/></p>
+<p><strong>Figure 1</strong>: ClickBench performance improved over 33% between DataFusion 33
+(released Nov. 2023) and DataFusion 45 (released Feb. 2025). </p>
+<p>The task of <a href="https://github.com/apache/datafusion/issues/10918">integrating</a> the new <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html">Arrow StringView</a> which significantly improves performance
+for workloads that scan, filter and group by variable length string and binary data was completed
+and enabled by default in the past 6 months. The improvement is especially pronounced for Parquet
+files due to <a href="https://github.com/apache/arrow-rs/issues/5530">upstream work in the parquet reader</a>. Kudos to <a href="https://github.com/XiangpengHong">@XiangpengHong</a>, <a href="https://github.com/AriesDevil">@AriesDevil</a>,
+<a href="https://github.com/PsiACE">@PsiACE</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/a10y">@a10y</a>, and <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a> for driving this project.</p>
+<h2>Improved Quality 📋</h2>
+<p>DataFusion continues to improve overall in quality. In addition to ongoing bug
+fixes, one of the most exciting improvements in the last 6 months was the addition of the
+<a href="https://github.com/apache/datafusion/pull/13936">SQLite sqllogictest suite</a> thanks to <a href="https://github.com/Omega359">@Omega359</a>. These tests run over 5 million
+sql statements on every push to the main branch.</p>
+<p>Support for <a href="https://github.com/apache/datafusion/pull/13651">explicitly checking logical plan invariants</a> was added by <a href="https://github.com/wiedld">@wiedld</a> which
+can help catch implicit changes that might cause problems during upgrades.</p>
+<p>We have also started other quality initiatives to make it <a href="https://github.com/apache/datafusion/issues/13525">easier to use DataFusion</a>
+based on <a href="https://glaredb.com/">GlareDB</a>'s experience along with more <a href="https://github.com/apache/datafusion/issues/13661">extensive prerelease testing</a>. </p>
+<h2>Improved Documentation 📚</h2>
+<p>We continue to improve the documentation to make it easier to get started using DataFusion.
+During the last 6 months two projects were initiated to migrate the function documentation
+from strictly static markdown files. First, <a href="https://github.com/apache/datafusion/pull/12668">@Omega359</a> to allow function
+documentation to be generated from code and <a href="https://github.com/jonathanc-n">@jonathanc-n</a> and others helped with the migration,
+then <a href="https://github.com/comphead">@comphead</a> lead a project to <a href="https://github.com/apache/datafusion/pull/12822">create a doc macro</a> to allow for an even easier way to write
+function documentation. A special thanks to <a href="https://github.com/Chen-Yuan-Lai">@Chen-Yuan-Lai</a> for migrating many functions to
+the new syntax.</p>
+<p>Additionally, the <a href="https://github.com/apache/datafusion/pull/13877">examples</a> were <a href="https://github.com/apache/datafusion/pull/13905">refactored</a> and <a href="https://github.com/apache/datafusion/pull/13950">cleaned up</a> to improve their usefulness.</p>
+<h2>New Features ✨</h2>
+<p>There are too many new features in the last 6 months to list them all, but here
+are some highlights:</p>
+<h3>Functions</h3>
+<ul>
+<li>Uniform Window Functions: <code>BuiltInWindowFunctions</code> was removed and all now use UDFs (<a href="https://github.com/jcsherin">@jcsherin</a>)</li>
+<li>Uniform Aggregate Functions: <code>BuiltInAggregateFunctions</code> was removed and all now use UDFs</li>
+<li>As mentioned above function documentation was extracted from the markdown files</li>
+<li>Some new functions and sql support were added including '<a href="https://github.com/apache/datafusion/pull/13799">show functions</a>', '<a href="https://github.com/apache/datafusion/pull/11347">to_local_time</a>',
+ '<a href="https://github.com/apache/datafusion/pull/12970">regexp_count</a>', '<a href="https://github.com/apache/datafusion/pull/11969">map_extract</a>', '<a href="https://github.com/apache/datafusion/pull/12211">array_distance</a>', '<a href="https://github.com/apache/datafusion/pull/12329">array_any_value</a>', '<a href="https://github.com/apache/datafusion/pull/12474">greatest</a>',
+ '<a href="https://github.com/apache/datafusion/pull/13786">least</a>', '<a href="https://github.com/apache/datafusion/pull/14217">arrays_overlap</a>'</li>
+</ul>
+<h3>FFI</h3>
+<ul>
+<li>Foreign Function Interface work has started. This should allow for
+ <a href="https://github.com/apache/datafusion/pull/12920">using table providers</a> across languages and versions of DataFusion. This
+ is especially pertinent for integration with <a href="https://delta-io.github.io/delta-rs/">delta-rs</a> and other table formats.</li>
+</ul>
+<h3>Materialized Views</h3>
+<ul>
+<li><a href="https://github.com/suremarc">@suremarc</a> has added a <a href="https://github.com/datafusion-contrib/datafusion-materialized-views">materialized view implementation</a> in datafusion-contrib 🚀</li>
+</ul>
+<h3>Substrait</h3>
+<ul>
+<li>A lot of work was put into improving and enhancing substrait support (<a href="https://github.com/Blizzara">@Blizzara</a>, <a href="https://github.com/westonpace">@westonpace</a>, <a href="https://github.com/tokoko">@tokoko</a>, <a href="https://github.com/vbarua">@vbarua</a>, <a href="https://github.com/LatrecheYasser">@LatrecheYasser</a>, <a href="https://github.com/notfilippo">@notfilippo</a> and others) [...]
+</ul>
+<h2>Looking Ahead: The Next Six Months ðŸ”</h2>
+<p>One of the long term goals of <a href="https://github.com/alamb">@alamb</a>, DataFusion's PMC chair, has been to have
+<a href="https://www.influxdata.com/blog/datafusion-2025-influxdb/">1000 DataFusion based projects</a>. This may be the year that happens!</p>
+<p>The community has been <a href="https://github.com/apache/datafusion/issues/14580">discussing what we will work on in the next six months</a>.
+Some major initiatives are likely to be:</p>
+<ol>
+<li><em>Performance</em>: A <a href="https://github.com/apache/datafusion/issues/14482">number of items have been identified</a> as areas that could use additional work</li>
+<li><em>Memory usage</em>: Tracking and improving memory usage, statistics and spilling to disk </li>
+<li><em><a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> (GSOC)</em>: DataFusion is hopefully selected as a project and we start accepting and supporting student projects </li>
+<li><em>FFI</em>: Extending the FFI implementation to support to all types of UDF's and SessionContext</li>
+<li><em>Spark Functions</em>: A <a href="https://github.com/apache/datafusion/issues/5600">proposal has been made to add a crate</a> covering spark compatible builtin functions </li>
+</ol>
+<h2>How to Get Involved</h2>
+<p>DataFusion is not a project built or driven by a single person, company, or
+foundation. Rather, our community of users and contributors work together to
+build a shared technology that none of us could have built alone.</p>
+<p>If you are interested in joining us we would love to have you. You can try out
+DataFusion on some of your own data and projects and let us know how it goes,
+contribute suggestions, documentation, bug reports, or a PR with documentation,
+tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
+can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p></content><category term="blog"></category></entry><entry><title>Apache DataFusion Comet 0.6.0 Release</title><link href="https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0" rel="alternate"></link><published>2025-02-17T00:00:00+00:00</published><updated>2025-02-17T00:00:00+00:00</updated><author><name>pmc</name></author [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/pmc.rss.xml b/blog/feeds/pmc.rss.xml
index a0ed98c..13aca78 100644
--- a/blog/feeds/pmc.rss.xml
+++ b/blog/feeds/pmc.rss.xml
@@ -1,5 +1,49 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - pmc</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon, 17 Feb 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet 0.6.0 Release</title><link>https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0</link><description><!--
+<rss version="2.0"><channel><title>Apache DataFusion Blog - pmc</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Thu, 20 Mar 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet 0.7.0 Release</title><link>https://datafusion.apache.org/blog/2025/03/20/datafusion-comet-0.7.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 20 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-20:/blog/2025/03/20/datafusion-comet-0.7.0</guid><category>blog</category></item><item><title>Apache DataFusion 45.0.0 Released</title><link>https://datafusion.apache.org/blog/2025/02/20/datafusion-45.0.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 20 Feb 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-02-20:/blog/2025/02/20/datafusion-45.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.6.0 Release</title><link>https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/xiangpeng-hao.atom.xml b/blog/feeds/xiangpeng-hao.atom.xml
new file mode 100644
index 0000000..8d6ba2f
--- /dev/null
+++ b/blog/feeds/xiangpeng-hao.atom.xml
@@ -0,0 +1,88 @@
+<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - Xiangpeng Hao</title><link href="https://datafusion.apache.org/blog/" rel="alternate"></link><link href="https://datafusion.apache.org/blog/feeds/xiangpeng-hao.atom.xml" rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-20T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link href="https://datafusion.apache.org/blog [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> implements advanced Parquet pruning techniques to effectively read only the data that matters for a given query.</p>
+<p>Achieving high performance adds complexity.
+This post provides an overview of the techniques used in DataFusion to selectively read Parquet files.</p>
+<h3>The pipeline</h3>
+<p>The diagram below illustrates the <a href="https://docs.rs/datafusion/46.0.0/datafusion/datasource/physical_plan/parquet/source/struct.ParquetSource.html">Parquet reading pipeline</a> in DataFusion, highlighting how data flows through various pruning stages before being converted to Arrow format:</p>
+<p><img alt="Parquet pruning pipeline in DataFusion" class="img-responsive" src="/blog/images/parquet-pruning/read-parquet.jpg" width="100%"/></p>
+<h4>Background: Parquet file structure</h4>
+<p>As shown in the figure above, each Parquet file has multiple row groups. Each row group contains a set of columns, and each column contains a set of pages.</p>
+<p>Pages are the smallest units of data in Parquet files and typically contain compressed and encoded values for a specific column. This hierarchical structure enables efficient columnar access and forms the foundation for the pruning techniques we'll discuss.</p>
+<p>Check out <a href="https://www.influxdata.com/blog/querying-parquet-millisecond-latency/">Querying Parquet with Millisecond Latency</a> for more details on the Parquet file structure.</p>
+<h4>1. Read metadata</h4>
+<p>DataFusion first reads the <a href="https://parquet.apache.org/docs/file-format/metadata/">Parquet metadata</a> to understand the data in the file.
+Metadata often includes data schema, the exact location of each row group and column chunk, and their corresponding statistics (e.g., min/max values).
+It also optionally includes <a href="https://parquet.apache.org/docs/file-format/pageindex/">page-level stats</a> and <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>.
+This information is used to prune the file before reading the actual data.</p>
+<p><a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L118">Fetching metadata</a> requires up to two network requests: one to read the footer size from the end of the file, and another to read the footer itself. </p>
+<p><a href="https://www.influxdata.com/blog/how-good-parquet-wide-tables/">Decoding metadata</a> is generally fast since it only requires parsing a small amount of data. However, for tables with hundreds or thousands of columns, the metadata can become quite large and decoding it can become a bottleneck. This is particularly noticeable when scanning many small files.</p>
+<p>Reading metadata is latency-critical, so DataFusion allows users to cache metadata through the <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L39">ParquetFileReaderFactory</a> trait.</p>
+<h4>2. Prune by projection</h4>
+<p>The simplest yet perhaps most effective pruning is to read only the columns that are needed.
+This is because queries usually don't select all columns, e.g., <code>SELECT a FROM table</code> only reads column <code>a</code>.
+As a <strong>columnar</strong> format, Parquet allows DataFusion to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/mod.rs#L778">only read</a> the <strong>columns</strong> that are needed.</p>
+<p>This projection pruning happens at the column level and can dramatically reduce I/O when working with wide tables where queries typically access only a small subset of columns.</p>
+<h4>3. Prune by row group stats and Bloom filters</h4>
+<p>Each row group has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/physical_optimizer/pruning.rs#L81">basic stats</a> like min/max values for each column.
+DataFusion applies the query predicates to these stats to prune row groups, e.g., <code>SELECT * FROM table WHERE a &gt; 10</code> will only read row groups where <code>a</code> has a max value greater than 10.</p>
+<p>Sometimes min/max stats are too simple to prune effectively, so Parquet also supports <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>. DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L202">uses Bloom filters when available</a>.</p>
+<p>Bloom filters are particularly effective for equality predicates (<code>WHERE a = 10</code>) and can significantly reduce the number of row groups that need to be read for point queries or queries with highly selective predicates.</p>
+<h4>4. Prune by page stats</h4>
+<p>Parquet optionally supports <a href="https://github.com/apache/parquet-format/blob/master/PageIndex.md">page-level stats</a> -- similar to row group stats but more fine-grained.
+DataFusion implements <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L219">page pruning</a> when the stats are present.</p>
+<p>Page-level pruning provides an additional layer of filtering after row group pruning. It allows DataFusion to skip individual pages within a row group, further reducing the amount of data that needs to be read and decoded.</p>
+<h4>5. Read from storage</h4>
+<p>Now we (hopefully) have pruned the Parquet file into small ranges of bytes, i.e., the <a href="https://github.com/apache/datafusion/blob/76a7789ace33ced54c973fa0d5fc9d1866e1bf19/datafusion/datasource-parquet/src/access_plan.rs#L86">Access Plan</a>.
+The last step is to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L103">make requests</a> to fetch those bytes and decode them into Arrow RecordBatch. </p>
+<h3>Preview of coming attractions: filter pushdown</h3>
+<p>So far we have discussed techniques that prune the Parquet file using only the metadata, i.e., before reading the actual data.</p>
+<p>Filter pushdown, also known as predicate pushdown or late materialization, is a technique that prunes data during scanning, with filters being generated and applied in the Parquet reader.</p>
+<p><img alt="Filter pushdown in DataFusion" class="img-responsive" src="/blog/images/parquet-pruning/filter-pushdown.jpg" width="100%"/></p>
+<p>Unlike metadata-based pruning which works at the row group or page level, filter pushdown operates at the row level, allowing DataFusion to filter out individual rows that don't match the query predicates during the decoding process.</p>
+<p>DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/row_filter.rs#L154">implements filter pushdown</a> but has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/common/src/config.rs#L382">not enabled it by default</a> due to <a href="https://github.com/apache/datafusion/issues/3463">some perform [...]
+<p>We are working to remove the <a href="https://github.com/apache/arrow-rs/issues/5523#issuecomment-2429470872">remaining performance issues</a> and enable it by default, which we will discuss in the next blog post.</p>
+<h3>Conclusion</h3>
+<p>DataFusion employs a multi-step approach to Parquet pruning, from column projection to row group stats, page stats, and potentially row-level filtering.
+Each step may reduce the amount of data to be read and processed, significantly improving query performance.</p></content><category term="blog"></category></entry></feed>
\ No newline at end of file
diff --git a/blog/feeds/xiangpeng-hao.rss.xml b/blog/feeds/xiangpeng-hao.rss.xml
new file mode 100644
index 0000000..d48cdfa
--- /dev/null
+++ b/blog/feeds/xiangpeng-hao.rss.xml
@@ -0,0 +1,23 @@
+<?xml version="1.0" encoding="utf-8"?>
+<rss version="2.0"><channel><title>Apache DataFusion Blog - Xiangpeng Hao</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Thu, 20 Mar 2025 00:00:00 +0000</lastBuildDate><item><title>Parquet Pruning in DataFusion: Read Only What Matters</title><link>https://datafusion.apache.org/blog/2025/03/20/parquet-pruning</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao</dc:creator><pubDate>Thu, 20 Mar 2025 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.apache.org,2025-03-20:/blog/2025/03/20/parquet-pruning</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/blog/images/datafusion-45.0.0/performance_over_time.png b/blog/images/datafusion-45.0.0/performance_over_time.png
new file mode 100644
index 0000000..dc899fa
Binary files /dev/null and b/blog/images/datafusion-45.0.0/performance_over_time.png differ
diff --git a/blog/images/ordering_analysis/query_window_plan.png b/blog/images/ordering_analysis/query_window_plan.png
new file mode 100644
index 0000000..ca30d22
Binary files /dev/null and b/blog/images/ordering_analysis/query_window_plan.png differ
diff --git a/blog/images/parquet-pruning/filter-pushdown.jpg b/blog/images/parquet-pruning/filter-pushdown.jpg
new file mode 100644
index 0000000..23e16cf
Binary files /dev/null and b/blog/images/parquet-pruning/filter-pushdown.jpg differ
diff --git a/blog/images/parquet-pruning/read-parquet.jpg b/blog/images/parquet-pruning/read-parquet.jpg
new file mode 100644
index 0000000..22b54c4
Binary files /dev/null and b/blog/images/parquet-pruning/read-parquet.jpg differ
diff --git a/blog/index.html b/blog/index.html
index befc312..9fecc28 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -44,6 +44,169 @@
<p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/03/20/datafusion-comet-0.7.0">Apache DataFusion Comet 0.7.0 Release</a></h1>
+ <p>Posted on: Thu 20 March 2025 by pmc</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>Comet runs on commodity hardware and aims to …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/03/20/datafusion-comet-0.7.0" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/03/20/parquet-pruning">Parquet Pruning in DataFusion: Read Only What Matters</a></h1>
+ <p>Posted on: Thu 20 March 2025 by Xiangpeng Hao</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
+<hr/>
+<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
+<p><a href="https://datafusion.apache.org/">Apache DataFusion …</a></p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/03/20/parquet-pruning" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/03/11/ordering-analysis">Using Ordering for Better Plans in Apache DataFusion</a></h1>
+ <p>Posted on: Tue 11 March 2025 by Mustafa Akur, Andrew Lamb</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
+<blockquote style="border-left: 4px solid #007bff; padding: 10px; background-color: #f8f9fa;">
+<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator …</blockquote></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/03/11/ordering-analysis" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2025/02/20/datafusion-45.0.0">Apache DataFusion 45.0.0 Released</a></h1>
+ <p>Posted on: Thu 20 February 2025 by pmc</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
+<h2>Introduction</h2>
+<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
+many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
+what the community is thinking about in the next 6 months. It has been an exciting
+period of development …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a href="/blog/2025/02/20/datafusion-45.0.0" class="button medium">Continue Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
<!-- Post -->
<div class="row">
<div class="callout">
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org