You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ti...@apache.org on 2024/12/06 23:24:25 UTC
(datafusion-site) branch asf-staging updated: Remove old output dir and set a redirect
This is an automated email from the ASF dual-hosted git repository.
timsaucer pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new 5a0b1dd Remove old output dir and set a redirect
5a0b1dd is described below
commit 5a0b1dde9bf194f8b82aa62d3bc2608fda4197ae
Author: Tim Saucer <ti...@maymobility.com>
AuthorDate: Fri Dec 6 18:24:15 2024 -0500
Remove old output dir and set a redirect
---
index.html | 11 +
output/2019/02/04/datafusion-donation/index.html | 159 -
output/2021/04/12/ballista-donation/index.html | 110 -
output/2021/08/18/ballista-0.5.0/index.html | 124 -
output/2021/08/18/datafusion-5.0.0/index.html | 155 -
.../11/19/2021-11-8-datafusion-6.0.0.md/index.html | 193 -
output/2022/02/28/datafusion-7.0.0/index.html | 199 -
output/2022/03/21/datafusion-contrib/index.html | 195 -
output/2022/05/16/datafusion-8.0.0/index.html | 227 -
output/2022/10/25/datafusion-13.0.0/index.html | 256 -
output/2022/10/28/ballista-0.9.0/index.html | 136 -
output/2023/01/19/datafusion-16.0.0/index.html | 277 -
output/2023/06/24/datafusion-25.0.0/index.html | 255 -
.../2023/08/05/datafusion_fast_grouping/index.html | 357 -
output/2024/01/19/datafusion-34.0.0/index.html | 304 -
output/2024/03/06/comet-donation/index.html | 120 -
output/2024/05/07/datafusion-tlp/index.html | 114 -
.../2024/07/20/datafusion-comet-0.1.0/index.html | 154 -
output/2024/07/24/datafusion-40.0.0/index.html | 363 -
.../2024/08/20/python-datafusion-40.0.0/index.html | 210 -
.../2024/08/28/datafusion-comet-0.2.0/index.html | 127 -
.../index.html | 171 -
.../index.html | 148 -
.../2024/09/27/datafusion-comet-0.3.0/index.html | 124 -
.../index.html | 263 -
.../datafusion-python-udf-comparisons/index.html | 601 -
.../2024/11/20/datafusion-comet-0.4.0/index.html | 138 -
output/about.html | 64 -
output/author/agrove.html | 147 -
output/author/alamb-dandandan-tustvold.html | 111 -
.../andrew-lamb-staff-engineer-at-influxdata.html | 110 -
output/author/pmc.html | 810 -
output/author/timsaucer.html | 150 -
output/author/xiangpeng-hao-andrew-lamb.html | 146 -
output/category/blog.html | 1129 --
output/css/app.css | 0
output/css/blog_index.css | 16 -
output/css/bootstrap.min.css | 6 -
output/css/font-awesome.min.css | 4 -
output/css/fontawesome.all.min.css | 6 -
output/css/headerlink.css | 16 -
output/css/webfonts/fa-brands-400.ttf | Bin 186112 -> 0 bytes
output/css/webfonts/fa-brands-400.woff2 | Bin 107460 -> 0 bytes
output/css/webfonts/fa-regular-400.ttf | Bin 62048 -> 0 bytes
output/css/webfonts/fa-regular-400.woff2 | Bin 25096 -> 0 bytes
output/css/webfonts/fa-solid-900.ttf | Bin 397728 -> 0 bytes
output/css/webfonts/fa-solid-900.woff2 | Bin 150472 -> 0 bytes
output/css/webfonts/fa-v4compatibility.ttf | Bin 10136 -> 0 bytes
output/css/webfonts/fa-v4compatibility.woff2 | Bin 4584 -> 0 bytes
output/favicon.ico | Bin 8894 -> 0 bytes
output/feed.xml | 568 -
output/feeds/agrove.atom.xml | 189 -
output/feeds/agrove.rss.xml | 42 -
output/feeds/alamb-dandandan-tustvold.atom.xml | 321 -
output/feeds/alamb-dandandan-tustvold.rss.xml | 25 -
output/feeds/all-en.atom.xml | 4462 -----
...drew-lamb-staff-engineer-at-influxdata.atom.xml | 226 -
...ndrew-lamb-staff-engineer-at-influxdata.rss.xml | 24 -
output/feeds/blog.atom.xml | 4462 -----
output/feeds/pmc.atom.xml | 2764 ---
output/feeds/pmc.rss.xml | 401 -
output/feeds/timsaucer.atom.xml | 734 -
output/feeds/timsaucer.rss.xml | 45 -
output/feeds/xiangpeng-hao-andrew-lamb.atom.xml | 238 -
output/feeds/xiangpeng-hao-andrew-lamb.rss.xml | 41 -
output/highlight/default.min.css | 9 -
output/highlight/highlight.js | 17194 -------------------
output/images/2021-08-18-datafusion500perf.png | Bin 21245 -> 0 bytes
output/images/2x_bgwhite_original.png | Bin 60451 -> 0 bytes
.../column-based-storage.png | Bin 47390 -> 0 bytes
.../clickbench-datafusion-43/perf-over-time.png | Bin 62185 -> 0 bytes
output/images/clickbench-datafusion-43/perf.png | Bin 57326 -> 0 bytes
.../clickbench-datafusion-43/row-based-storage.png | Bin 45024 -> 0 bytes
.../skipping-partial-aggregation.png | Bin 133362 -> 0 bytes
.../clickbench-datafusion-43/string-view-take.png | Bin 66566 -> 0 bytes
output/images/comet-0.1.0/tpch_allqueries.png | Bin 26393 -> 0 bytes
output/images/comet-0.1.0/tpch_queries_compare.png | Bin 29269 -> 0 bytes
output/images/comet-0.1.0/tpch_queries_speedup.png | Bin 42444 -> 0 bytes
output/images/comet-0.2.0/tpcds_allqueries.png | Bin 24925 -> 0 bytes
output/images/comet-0.2.0/tpch_allqueries.png | Bin 25972 -> 0 bytes
output/images/datafusion-34.0.0/compare-new.png | Bin 176796 -> 0 bytes
output/images/datafusion-34.0.0/compare.png | Bin 149835 -> 0 bytes
.../datafusion-40.0.0/improved-planning-time.png | Bin 47905 -> 0 bytes
.../images/datafusion-comet/comet-architecture.png | Bin 30027 -> 0 bytes
output/images/datafusion_fast_grouping/full.png | Bin 50146 -> 0 bytes
output/images/datafusion_fast_grouping/summary.png | Bin 52822 -> 0 bytes
output/images/logo_original4x.png | Bin 119116 -> 0 bytes
.../pylance_error_checking.png | Bin 39119 -> 0 bytes
.../vscode_hover_tooltip.png | Bin 87320 -> 0 bytes
.../images/string-view-1/figure1-performance.png | Bin 153348 -> 0 bytes
.../images/string-view-1/figure2-string-view.png | Bin 197178 -> 0 bytes
output/images/string-view-1/figure4-copying.png | Bin 115588 -> 0 bytes
.../string-view-1/figure5-loading-strings.png | Bin 68999 -> 0 bytes
.../string-view-1/figure6-utf8-validation.png | Bin 187490 -> 0 bytes
output/images/string-view-1/figure7-end-to-end.png | Bin 64736 -> 0 bytes
.../string-view-2/figure1-zero-copy-take.png | Bin 95659 -> 0 bytes
.../images/string-view-2/figure2-filter-time.png | Bin 52437 -> 0 bytes
output/index.html | 1123 --
output/js/bootstrap.bundle.min.js | 7 -
output/js/bootstrap.bundle.min.js.map | 1 -
100 files changed, 11 insertions(+), 41071 deletions(-)
diff --git a/index.html b/index.html
new file mode 100644
index 0000000..a642ee0
--- /dev/null
+++ b/index.html
@@ -0,0 +1,11 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+<head>
+ <meta http-equiv="refresh" content="0; URL=./blog" />
+</head>
+<body class="d-flex flex-column h-100">
+<main class="flex-shrink-0">
+ You should be redirected <a href="blog">here</a>.
+</main>
+</body>
+</html>
diff --git a/output/2019/02/04/datafusion-donation/index.html b/output/2019/02/04/datafusion-donation/index.html
deleted file mode 100644
index a8bdcde..0000000
--- a/output/2019/02/04/datafusion-donation/index.html
+++ /dev/null
@@ -1,159 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>DataFusion: A Rust-native Query Engine for Apache Arrow - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- DataFusion: A Rust-native Query Engine for Apache Arrow
- </h1>
- <p>Posted on: Mon 04 February 2019 by agrove</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support SQL queries against iterators of RecordBatch and has support for CSV files. There are plans to <a href="https://issues.apache.org/jira/browse/ARROW-4466">add support for Parquet files</a>.</p>
-<p>SQL support is limited to projection (<code>SELECT</code>), selection (<code>WHERE</code>), and simple aggregates (<code>MIN</code>, <code>MAX</code>, <code>SUM</code>) with an optional <code>GROUP BY</code> clause.</p>
-<p>Supported expressions are identifiers, literals, simple math operations (<code>+</code>, <code>-</code>, <code>*</code>, <code>/</code>), binary expressions (<code>AND</code>, <code>OR</code>), equality and comparison operators (<code>=</code>, <code>!=</code>, <code><</code>, <code><=</code>, <code>>=</code>, <code>></code>), and <code>CAST(expr AS type)</code>.</p>
-<h2>Example</h2>
-<p>The following example demonstrates running a simple aggregate SQL query against a CSV file.</p>
-<div class="codehilite"><pre><span></span><code><span class="c1">// create execution context</span>
-<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">ctx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ExecutionContext</span>::<span class="n">new</span><span class="p">();</span><span class="w"></span>
-
-<span class="c1">// define schema for data source (csv file)</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Arc</span>::<span class="n">new</span><span class="p">(</span><span class="n">Schema</span>::<span class="n">new</span><span class="p">(</span><span class="n">vec</span><span class="o">!</span><span class="p">[</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c1"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c2"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt32</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c3"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int8</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c4"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int16</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c5"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int32</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c6"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int64</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c7"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt8</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c8"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt16</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c9"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt32</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c10"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt64</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c11"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float32</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c12"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float64</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c13"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"></span>
-<span class="p">]));</span><span class="w"></span>
-
-<span class="c1">// register csv file with the execution context</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">csv_datasource</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">CsvDataSource</span>::<span class="n">new</span><span class="p">(</span><span class="s">"test/data/aggregate_test_100.csv"</span><span class="p">,</span><span class="w"> </span><span class="n">schema</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span><span class="w"> </span><span class="mi">1024</span><span class="p">);</span><span class="w"></span>
-<span class="n">ctx</span><span class="p">.</span><span class="n">register_datasource</span><span class="p">(</span><span class="s">"aggregate_test_100"</span><span class="p">,</span><span class="w"> </span><span class="n">Rc</span>::<span class="n">new</span><span class="p">(</span><span class="n">RefCell</span>::<span class="n">new</span><span class="p">(</span><span class="n">csv_datasource</span><span class="p">)));</span><span class="w"></span>
-
-<span class="kd">let</span><span class="w"> </span><span class="n">sql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"SELECT c1, MIN(c12), MAX(c12) FROM aggregate_test_100 WHERE c11 > 0.1 AND c11 < 0.9 GROUP BY c1"</span><span class="p">;</span><span class="w"></span>
-
-<span class="c1">// execute the query</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">relation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ctx</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="o">&</span><span class="n">sql</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
-<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">results</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">relation</span><span class="p">.</span><span class="n">borrow_mut</span><span class="p">();</span><span class="w"></span>
-
-<span class="c1">// iterate over the results</span>
-<span class="k">while</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">batch</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">results</span><span class="p">.</span><span class="n">next</span><span class="p">().</span><span class="n">unwrap</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><spa [...]
-<span class="w"> </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="w"></span>
-<span class="w"> </span><span class="s">"RecordBatch has {} rows and {} columns"</span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">batch</span><span class="p">.</span><span class="n">num_rows</span><span class="p">(),</span><span class="w"></span>
-<span class="w"> </span><span class="n">batch</span><span class="p">.</span><span class="n">num_columns</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">);</span><span class="w"></span>
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">c1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">batch</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">as_any</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">downcast_ref</span>::<span class="o"><</span><span class="n">BinaryArray</span><span class="o">></span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">min</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">batch</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">column</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">as_any</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">downcast_ref</span>::<span class="o"><</span><span class="n">Float64Array</span><span class="o">></span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">max</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">batch</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">column</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">as_any</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">downcast_ref</span>::<span class="o"><</span><span class="n">Float64Array</span><span class="o">></span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
-
-<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">0</span><span class="o">..</span><span class="n">batch</span><span class="p">.</span><span class="n">num_rows</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">c1_value</span>: <span class="nb">String</span> <span class="o">=</span><span class="w"> </span><span class="nb">String</span>::<span class="n">from_utf8</span><span class="p">(</span><span class="n">c1</span><span class="p">.</span><span class="n">value</span><span class="p">(</span><span class="n">i</span><span class="p">).</span><span class="n">to_vec</span><span class="p">()).</span><spa [...]
-<span class="w"> </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{}, Min: {}, Max: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">c1_value</span><span class="p">,</span><span class="w"> </span><span class="n">min</span><span class="p">.</span><span class="n">value</span><span class="p">(</span><span class="n">i</span><span class="p">),</span><span class="w"> </span><span class="n">max</span><span clas [...]
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-</code></pre></div>
-<h2>Roadmap</h2>
-<p>The roadmap for DataFusion will depend on interest from the Rust community, but here are some of the short term items that are planned:</p>
-<ul>
-<li>Extending test coverage of the existing functionality</li>
-<li>Adding support for Parquet data sources</li>
-<li>Implementing more SQL features such as <code>JOIN</code>, <code>ORDER BY</code> and <code>LIMIT</code></li>
-<li>Implement a DataFrame API as an alternative to SQL</li>
-<li>Adding support for partitioning and parallel query execution using Rust's async and await functionality</li>
-<li>Creating a Docker image to make it easy to use DataFusion as a standalone query tool for interactive and batch queries</li>
-</ul>
-<h2>Contributors Welcome!</h2>
-<p>If you are excited about being able to use Rust for data science and would like to contribute to this work then there are many ways to get involved. The simplest way to get started is to try out DataFusion against your own data sources and file bug reports for any issues that you find. You could also check out the current <a href="https://cwiki.apache.org/confluence/display/ARROW/Rust+JIRA+Dashboard">list of issues</a> and have a go at fixing one. You can also join the <a href="http:/ [...]
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2021/04/12/ballista-donation/index.html b/output/2021/04/12/ballista-donation/index.html
deleted file mode 100644
index c689f77..0000000
--- a/output/2021/04/12/ballista-donation/index.html
+++ /dev/null
@@ -1,110 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Ballista: A Distributed Scheduler for Apache Arrow - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Ballista: A Distributed Scheduler for Apache Arrow
- </h1>
- <p>Posted on: Mon 12 April 2021 by agrove</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
-to the Apache Arrow project. </p>
-<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported as
-first-class citizens without paying a penalty for serialization costs.</p>
-<p>The foundational technologies in Ballista are:</p>
-<ul>
-<li><a href="https://arrow.apache.org/">Apache Arrow</a> memory model and compute kernels for efficient processing of data.</li>
-<li><a href="https://github.com/apache/arrow-datafusion">Apache Arrow DataFusion</a> query planning and
- execution framework, extended by Ballista to provide distributed planning and execution.</li>
-<li><a href="https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/">Apache Arrow Flight Protocol</a> for efficient
- data transfer between processes.</li>
-<li><a href="https://developers.google.com/protocol-buffers">Google Protocol Buffers</a> for serializing query plans.</li>
-<li><a href="https://www.docker.com/">Docker</a> for packaging up executors along with user-defined code.</li>
-</ul>
-<p>Ballista can be deployed as a standalone cluster and also supports <a href="https://kubernetes.io/">Kubernetes</a>. In either
-case, the scheduler can be configured to use <a href="https://etcd.io/">etcd</a> as a backing store to (eventually) provide
-redundancy in the case of a scheduler failing.</p>
-<h2>Status</h2>
-<p>The Ballista project is at an early stage of development. However, it is capable of running complex analytics queries
-in a distributed cluster with reasonable performance (comparable to more established distributed query frameworks).</p>
-<p>One of the benefits of Ballista being part of the Arrow codebase is that there is now an opportunity to push parts of
-the scheduler down to DataFusion so that is possible to seamlessly scale across cores in DataFusion, and across nodes
-in Ballista, using the same unified query scheduler.</p>
-<h2>Contributors Welcome!</h2>
-<p>If you are excited about being able to use Rust for distributed compute and ETL and would like to contribute to this
-work then there are many ways to get involved. The simplest way to get started is to try out Ballista against your own
-datasets and file bug reports for any issues that you find. You could also check out the current
-<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20component%20%3D%20%22Rust%20-%20Ballista%22">list of issues</a> and have a go at fixing one.</p>
-<p>The <a href="https://github.com/apache/arrow/blob/master/rust/README.md#arrow-rust-community">Arrow Rust Community</a>
-section of the Rust README provides information on other ways to interact with the Ballista contributors and
-maintainers.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2021/08/18/ballista-0.5.0/index.html b/output/2021/08/18/ballista-0.5.0/index.html
deleted file mode 100644
index 95f7054..0000000
--- a/output/2021/08/18/ballista-0.5.0/index.html
+++ /dev/null
@@ -1,124 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow Ballista 0.5.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow Ballista 0.5.0 Release
- </h1>
- <p>Posted on: Wed 18 August 2021 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>Ballista extends DataFusion to provide support for distributed queries. This is the first release of Ballista since
-the project was <a href="https://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to the Apache Arrow project
-and includes 80 commits from 11 contributors.</p>
-<div class="codehilite"><pre><span></span><code><span class="err">git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust/executor ballista/rust/scheduler</span>
-<span class="err"> 27 Andy Grove</span>
-<span class="err"> 15 Jiayu Liu</span>
-<span class="err"> 12 Andrew Lamb</span>
-<span class="err"> 8 Ximo Guanter</span>
-<span class="err"> 6 Daniël Heres</span>
-<span class="err"> 5 QP Hou</span>
-<span class="err"> 2 Jorge Leitao</span>
-<span class="err"> 1 Javier Goday</span>
-<span class="err"> 1 K.I. (Dennis) Jung</span>
-<span class="err"> 1 Mike Seddon</span>
-<span class="err"> 1 sathis</span>
-</code></pre></div>
-<!--
-$ git log --pretty=oneline 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust/executor ballista/rust/scheduler ballista-examples/ | wc -l
-80
--->
-<p>The release notes below are not exhaustive and only expose selected highlights of the release. Many other bug fixes
-and improvements have been made: we refer you to the <a href="https://github.com/apache/arrow-datafusion/blob/5.0.0/ballista/CHANGELOG.md">complete changelog</a>.</p>
-<h1>Performance and Scalability</h1>
-<p>Ballista is now capable of running complex SQL queries at scale and supports scalable distributed joins. We have been
-benchmarking using individual queries from the TPC-H benchmark at scale factors up to 1000 (1 TB). When running against
-CSV files, performance is generally very close to DataFusion, and significantly faster in some cases due to the fact
-that the scheduler limits the number of concurrent tasks that run at any given time. Performance against large Parquet
-datasets is currently non ideal due to some issues (<a href="https://github.com/apache/arrow-datafusion/issues/867">#867</a>,
-<a href="https://github.com/apache/arrow-datafusion/issues/868">#868</a>) that we hope to resolve for the next release. </p>
-<h1>New Features</h1>
-<p>The main new features in this release are:</p>
-<ul>
-<li>Ballista queries can now be executed by calling DataFrame.collect()</li>
-<li>The shuffle mechanism has been re-implemented</li>
-<li>Distributed hash-partitioned joins are now supported</li>
-<li>Keda autoscaling is supported</li>
-</ul>
-<p>To get started with Ballista, refer to the <a href="https://docs.rs/ballista/0.5.0/ballista/">crate documentation</a>.</p>
-<p>Now that the basic functionality is in place, the focus for the next release will be to improve the performance and
-scalability as well as improving the documentation.</p>
-<h1>How to Get Involved</h1>
-<p>If you are interested in contributing to Ballista, we would love to have you! You
-can help by trying out Ballista on some of your own data and projects and filing bug reports and helping to
-improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
-beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2021/08/18/datafusion-5.0.0/index.html b/output/2021/08/18/datafusion-5.0.0/index.html
deleted file mode 100644
index 4e6c5c0..0000000
--- a/output/2021/08/18/datafusion-5.0.0/index.html
+++ /dev/null
@@ -1,155 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 5.0.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow DataFusion 5.0.0 Release
- </h1>
- <p>Posted on: Wed 18 August 2021 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache Arrow team is pleased to announce the DataFusion 5.0.0 release. This covers 4 months of development work
-and includes 211 commits from the following 31 distinct contributors.</p>
-<div class="codehilite"><pre><span></span><code>$ git shortlog -sn <span class="m">4</span>.0.0..5.0.0 datafusion datafusion-cli datafusion-examples
- <span class="m">61</span> Jiayu Liu
- <span class="m">47</span> Andrew Lamb
- <span class="m">27</span> Daniël Heres
- <span class="m">13</span> QP Hou
- <span class="m">13</span> Andy Grove
- <span class="m">4</span> Javier Goday
- <span class="m">4</span> sathis
- <span class="m">3</span> Ruan Pearce-Authers
- <span class="m">3</span> Raphael Taylor-Davies
- <span class="m">3</span> Jorge Leitao
- <span class="m">3</span> Cui Wenzheng
- <span class="m">3</span> Mike Seddon
- <span class="m">3</span> Edd Robinson
- <span class="m">2</span> 思维
- <span class="m">2</span> Liang-Chi Hsieh
- <span class="m">2</span> Michael Lu
- <span class="m">2</span> Parth Sarthy
- <span class="m">2</span> Patrick More
- <span class="m">2</span> Rich
- <span class="m">1</span> Charlie Evans
- <span class="m">1</span> Gang Liao
- <span class="m">1</span> Agata Naomichi
- <span class="m">1</span> Ritchie Vink
- <span class="m">1</span> Evan Chan
- <span class="m">1</span> Ruihang Xia
- <span class="m">1</span> Todd Treece
- <span class="m">1</span> Yichen Wang
- <span class="m">1</span> baishen
- <span class="m">1</span> Nga Tran
- <span class="m">1</span> rdettai
- <span class="m">1</span> Marco Neumann
-</code></pre></div>
-<!--
-$ git log --pretty=oneline 4.0.0..5.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 211
--->
-<p>The release notes below are not exhaustive and only expose selected highlights of the release. Many other bug fixes
-and improvements have been made: we refer you to the complete
-<a href="https://github.com/apache/arrow-datafusion/blob/5.0.0/datafusion/CHANGELOG.md">changelog</a>.</p>
-<h1>Performance</h1>
-<p>There have been numerous performance improvements in this release. The following chart shows the relative
-performance of individual TPC-H queries compared to the previous release.</p>
-<p><i>TPC-H @ scale factor 100, in parquet format. Concurrency 24.</i></p>
-<p><img src="../images/2021-08-18-datafusion500perf.png"/></p>
-<p>We also extended support for more TPC-H queries: q7, q8, q9 and q13 are running successfully in DataFusion 5.0.</p>
-<h1>New Features</h1>
-<ul>
-<li>Initial support for SQL-99 Analytics (WINDOW functions)</li>
-<li>Improved JOIN support: cross join, semi-join, anti join, and fixes to null handling</li>
-<li>Improved EXPLAIN support</li>
-<li>Initial implementation of metrics in the physical plan</li>
-<li>Support for SELECT DISTINCT</li>
-<li>Support for Json and NDJson formatted inputs</li>
-<li>Query column with relations</li>
-<li>Added more datetime related functions: <code>now</code>, <code>date_trunc</code>, <code>to_timestamp_millis</code>, <code>to_timestamp_micros</code>, <code>to_timestamp_seconds</code></li>
-<li>Streaming Dataframe.collect</li>
-<li>Support table column aliases</li>
-<li>Answer count(*), min() and max() queries using only statistics</li>
-<li>Non-equi-join filters in JOIN conditions</li>
-<li>Modulus operation</li>
-<li>Support group by column positions</li>
-<li>Added constant folding query optimizer</li>
-<li>Hash partitioned aggregation</li>
-<li>Added <code>random</code> SQL function</li>
-<li>Implemented count distinct for floats and dictionary types</li>
-<li>Re-exported arrow and parquet crates in Datafusion</li>
-<li>General row group pruning logic that’s agnostic to storage format</li>
-</ul>
-<h1>How to Get Involved</h1>
-<p>If you are interested in contributing to DataFusion, we would love to have you! You
-can help by trying out DataFusion on some of your own data and projects and filing bug reports and helping to
-improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
-beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html b/output/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html
deleted file mode 100644
index 451ec64..0000000
--- a/output/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html
+++ /dev/null
@@ -1,193 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 6.0.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow DataFusion 6.0.0 Release
- </h1>
- <p>Posted on: Fri 19 November 2021 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an embedded
-query engine which leverages the unique features of
-<a href="https://www.rust-lang.org/">Rust</a> and <a href="https://arrow.apache.org/">Apache
-Arrow</a> to provide a system that is high
-performance, easy to connect, easy to embed, and high quality.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers 4 months of development work
-and includes 134 commits from the following 28 distinct contributors.</p>
-<!--
-git log --pretty=oneline 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 134
-
-git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 29
-
- Carlos and xudong963 are same individual
--->
-<div class="codehilite"><pre><span></span><code><span class="err"> 28 Andrew Lamb</span>
-<span class="err"> 26 Jiayu Liu</span>
-<span class="err"> 13 xudong963</span>
-<span class="err"> 9 rdettai</span>
-<span class="err"> 9 QP Hou</span>
-<span class="err"> 6 Matthew Turner</span>
-<span class="err"> 5 Daniël Heres</span>
-<span class="err"> 4 Guillaume Balaine</span>
-<span class="err"> 3 Francis Du</span>
-<span class="err"> 3 Marco Neumann</span>
-<span class="err"> 3 Jon Mease</span>
-<span class="err"> 3 Nga Tran</span>
-<span class="err"> 2 Yijie Shen</span>
-<span class="err"> 2 Ruihang Xia</span>
-<span class="err"> 2 Liang-Chi Hsieh</span>
-<span class="err"> 2 baishen</span>
-<span class="err"> 2 Andy Grove</span>
-<span class="err"> 2 Jason Tianyi Wang</span>
-<span class="err"> 1 Nan Zhu</span>
-<span class="err"> 1 Antoine Wendlinger</span>
-<span class="err"> 1 Krisztián Szűcs</span>
-<span class="err"> 1 Mike Seddon</span>
-<span class="err"> 1 Conner Murphy</span>
-<span class="err"> 1 Patrick More</span>
-<span class="err"> 1 Taehoon Moon</span>
-<span class="err"> 1 Tiphaine Ruy</span>
-<span class="err"> 1 adsharma</span>
-<span class="err"> 1 lichuan6</span>
-</code></pre></div>
-<p>The release notes below are not exhaustive and only expose selected highlights of the release. Many other bug fixes
-and improvements have been made: we refer you to the complete
-<a href="https://github.com/apache/arrow-datafusion/blob/6.0.0/datafusion/CHANGELOG.md">changelog</a>.</p>
-<h1>New Website</h1>
-<p>Befitting a growing project, DataFusion now has its
-<a href="https://arrow.apache.org/datafusion/">own website</a> hosted as part of the
-main <a href="https://arrow.apache.org">Apache Arrow Website</a></p>
-<h1>Roadmap</h1>
-<p>The community worked to gather their thoughts about where we are
-taking DataFusion into a public
-<a href="https://arrow.apache.org/datafusion/specification/roadmap.html">Roadmap</a>
-for the first time</p>
-<h1>New Features</h1>
-<ul>
-<li>Runtime operator metrics collection framework</li>
-<li>Object store abstraction for unified access to local or remote storage</li>
-<li>Hive style table partitioning support, for Parquet, CSV, Avro and Json files</li>
-<li>DataFrame API support for: <code>except</code>, <code>intersect</code>, <code>show</code>, <code>limit</code> and window functions</li>
-<li>SQL</li>
-<li><code>EXPLAIN ANALYZE</code> with runtime metrics</li>
-<li><code>trim ( [ LEADING | TRAILING | BOTH ] [ FROM ] string text [, characters text ] )</code> syntax</li>
-<li>Postgres style regular expression matching operators <code>~</code>, <code>~*</code>, <code>!~</code>, and <code>!~*</code></li>
-<li>SQL set operators <code>UNION</code>, <code>INTERSECT</code>, and <code>EXCEPT</code></li>
-<li><code>cume_dist</code>, <code>percent_rank</code> window functions</li>
-<li><code>digest</code>, <code>blake2s</code>, <code>blake2b</code>, <code>blake3</code> crypto functions</li>
-<li>HyperLogLog based <code>approx_distinct</code></li>
-<li><code>is distinct from</code> and <code>is not distinct from</code></li>
-<li><code>CREATE TABLE AS SELECT</code></li>
-<li>Accessing elements of nested <code>Struct</code> and <code>List</code> columns (e.g. <code>SELECT struct_column['field_name'], array_column[0] FROM ...</code>)</li>
-<li>Boolean expressions in <code>CASE</code> statement</li>
-<li><code>DROP TABLE</code></li>
-<li><code>VALUES</code> List</li>
-<li>Postgres regex match operators</li>
-<li>Support for Avro format</li>
-<li>Support for <code>ScalarValue::Struct</code></li>
-<li>Automatic schema inference for CSV files</li>
-<li>Better interactive editing support in <code>datafusion-cli</code> as well as <code>psql</code> style commands such as <code>\d</code>, <code>\?</code>, and <code>\q</code></li>
-<li>Generic constant evaluation and simplification framework</li>
-<li>Added common subexpression eliminate query plan optimization rule</li>
-<li>Python binding 0.4.0 with all Datafusion 6.0.0 features</li>
-</ul>
-<p>With these new features, we are also now passing TPC-H queries 8, 13 and 21.</p>
-<p>For the full list of new features with their relevant PRs, see the
-<a href="https://github.com/apache/arrow-datafusion/blob/6.0.0/datafusion/CHANGELOG.md">enhancements section</a>
-in the changelog.</p>
-<h1><code>async</code> planning and decoupling file format from table layout</h1>
-<p>Driven by the need to support Hive style table partitioning, @rdettai
-introduced the following design change to the Datafusion core.</p>
-<ul>
-<li>The code for reading specific file formats (<code>Parquet</code>, <code>Avro</code>, <code>CSV</code>, and
-<code>JSON</code>) was separated from the logic that handles grouping sets of
-files into execution partitions.</li>
-<li>The query planning process was made <code>async</code>.</li>
-</ul>
-<p>As a result, we are able to replace the old <code>Parquet</code>, <code>CSV</code> and <code>JSON</code> table
-providers with a single <code>ListingTable</code> table provider.</p>
-<p>This also sets up DataFusion and its plug-in ecosystem to
-supporting a wide range of catalogs and various object store implementations.
-You can read more about this change in the
-<a href="https://docs.google.com/document/d/1Bd4-PLLH-pHj0BquMDsJ6cVr_awnxTuvwNJuWsTHxAQ">design document</a>
-and on the <a href="https://github.com/apache/arrow-datafusion/pull/1010">arrow-datafusion#1010 PR</a>.</p>
-<h1>How to Get Involved</h1>
-<p>If you are interested in contributing to DataFusion, we would love to have you! You
-can help by trying out DataFusion on some of your own data and projects and filing bug reports and helping to
-improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
-beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p>
-<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2022/02/28/datafusion-7.0.0/index.html b/output/2022/02/28/datafusion-7.0.0/index.html
deleted file mode 100644
index d396bf0..0000000
--- a/output/2022/02/28/datafusion-7.0.0/index.html
+++ /dev/null
@@ -1,199 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 7.0.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow DataFusion 7.0.0 Release
- </h1>
- <p>Posted on: Mon 28 February 2022 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth checking out.</p>
-<p>DataFusion's SQL, <code>DataFrame</code>, and manual <code>PlanBuilder</code> API let users access a sophisticated query optimizer and execution engine capable of fast, resource efficient, and parallel execution that takes optimal advantage of todays multicore hardware. Being written in Rust means DataFusion can offer <em>both</em> the safety of dynamic languages as well as the resource efficiency of a compiled language.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 7.0.0 release. This covers 4 months of development work
-and includes 195 commits from the following 37 distinct contributors.</p>
-<!--
-git log --pretty=oneline 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 134
-
-git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 29
-
- Carlos and xudong963 are same individual
--->
-<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="mi">44</span><span class="w"> </span><span class="n">Andrew</span><span class="w"> </span><span class="n">Lamb</span><span class="w"></span>
-<span class="w"> </span><span class="mi">24</span><span class="w"> </span><span class="n">Kun</span><span class="w"> </span><span class="n">Liu</span><span class="w"></span>
-<span class="w"> </span><span class="mi">23</span><span class="w"> </span><span class="n">Jiayu</span><span class="w"> </span><span class="n">Liu</span><span class="w"></span>
-<span class="w"> </span><span class="mi">17</span><span class="w"> </span><span class="n">xudong</span><span class="p">.</span><span class="n">w</span><span class="w"></span>
-<span class="w"> </span><span class="mi">11</span><span class="w"> </span><span class="n">Yijie</span><span class="w"> </span><span class="n">Shen</span><span class="w"></span>
-<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">Matthew</span><span class="w"> </span><span class="n">Turner</span><span class="w"></span>
-<span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">Liang</span><span class="o">-</span><span class="n">Chi</span><span class="w"> </span><span class="n">Hsieh</span><span class="w"></span>
-<span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="n">Lin</span><span class="w"> </span><span class="n">Ma</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">Stephen</span><span class="w"> </span><span class="n">Carman</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">James</span><span class="w"> </span><span class="n">Katz</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">Dmitry</span><span class="w"> </span><span class="n">Patsura</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">QP</span><span class="w"> </span><span class="n">Hou</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">dependabot</span><span class="o">[</span><span class="n">bot</span><span class="o">]</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Remzi</span><span class="w"> </span><span class="n">Yang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Yang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">ic4y</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Daniël</span><span class="w"> </span><span class="n">Heres</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Andy</span><span class="w"> </span><span class="n">Grove</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Raphael</span><span class="w"> </span><span class="n">Taylor</span><span class="o">-</span><span class="n">Davies</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Jason</span><span class="w"> </span><span class="n">Tianyi</span><span class="w"> </span><span class="n">Wang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Dan</span><span class="w"> </span><span class="n">Harris</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Sergey</span><span class="w"> </span><span class="n">Melnychuk</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Nitish</span><span class="w"> </span><span class="n">Tiwari</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Dom</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Eduard</span><span class="w"> </span><span class="n">Karacharov</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Javier</span><span class="w"> </span><span class="n">Goday</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Boaz</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Marko</span><span class="w"> </span><span class="n">Mikulicic</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nf">Max</span><span class="w"> </span><span class="n">Burke</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Carol</span><span class="w"> </span><span class="p">(</span><span class="n">Nichols</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="n">Goulding</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Phillip</span><span class="w"> </span><span class="n">Cloud</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Rich</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Toby</span><span class="w"> </span><span class="n">Hede</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Will</span><span class="w"> </span><span class="n">Jones</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">r</span><span class="mf">.4</span><span class="n">ntix</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">rdettai</span><span class="w"></span>
-</code></pre></div>
-<p>The following section highlights some of the improvements in this release. Of course, many other bug fixes and improvements have also been made and we refer you to the complete <a href="https://github.com/apache/arrow-datafusion/blob/7.0.0/datafusion/CHANGELOG.md">changelog</a> for the full detail.</p>
-<h1>Summary</h1>
-<ul>
-<li>DataFusion Crate</li>
-<li>The DataFusion crate is being split into multiple crates to decrease compilation times and improve the development experience. Initially, <code>datafusion-common</code> (the core DataFusion components) and <code>datafusion-expr</code> (DataFusion expressions, functions, and operators) have been split out. There will be additional splits after the 7.0 release.</li>
-<li>Performance Improvements and Optimizations</li>
-<li>Arrow’s dyn scalar kernels are now used to enable efficient operations on <code>DictionaryArray</code>s <a href="https://github.com/apache/arrow-datafusion/pull/1685">#1685</a></li>
-<li>Switch from <code>std::sync::Mutex</code> to <code>parking_lot::Mutex</code> <a href="https://github.com/apache/arrow-datafusion/pull/1720">#1720</a></li>
-<li>New Features</li>
-<li>Support for memory tracking and spilling to disk<ul>
-<li>MemoryMananger and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
-<li>Out of core sort <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
-<li>New metrics</li>
-<li><code>Gauge</code> and <code>CurrentMemoryUsage</code> <a href="https://github.com/apache/arrow-datafusion/pull/1682">#1682</a></li>
-<li><code>Spill_count</code> and <code>spilled_bytes</code> <a href="https://github.com/apache/arrow-datafusion/pull/1641">#1641</a></li>
-</ul>
-</li>
-<li>New math functions<ul>
-<li><code>Approx_quantile</code> <a href="https://github.com/apache/arrow-datafusion/pull/1539">#1529</a></li>
-<li><code>stddev</code> and <code>variance</code> (sample and population) <a href="https://github.com/apache/arrow-datafusion/pull/1525">#1525</a></li>
-<li><code>corr</code> <a href="https://github.com/apache/arrow-datafusion/pull/1561">#1561</a></li>
-</ul>
-</li>
-<li>Support decimal type <a href="https://github.com/apache/arrow-datafusion/pull/1394">#1394</a><a href="https://github.com/apache/arrow-datafusion/pull/1407">#1407</a><a href="https://github.com/apache/arrow-datafusion/pull/1408">#1408</a><a href="https://github.com/apache/arrow-datafusion/pull/1431">#1431</a><a href="https://github.com/apache/arrow-datafusion/pull/1483">#1483</a><a href="https://github.com/apache/arrow-datafusion/pull/1554">#1554</a><a href="https://github.com/apache/ [...]
-<li>Support for reading Parquet files with evolved schemas <a href="https://github.com/apache/arrow-datafusion/pull/1622">#1622</a><a href="https://github.com/apache/arrow-datafusion/pull/1709">#1709</a></li>
-<li>Support for registering <code>DataFrame</code> as table <a href="https://github.com/apache/arrow-datafusion/pull/1699">#1699</a></li>
-<li>Support for the <code>substring</code> function <a href="https://github.com/apache/arrow-datafusion/pull/1621">#1621</a></li>
-<li>Support <code>array_agg(distinct ...)</code> <a href="https://github.com/apache/arrow-datafusion/pull/1579">#1579</a></li>
-<li>Support <code>sort</code> on unprojected columns <a href="https://github.com/apache/arrow-datafusion/pull/1415">#1415</a></li>
-<li>Additional Integration Points</li>
-<li>A new public Expression simplification API <a href="https://github.com/apache/arrow-datafusion/pull/1717">#1717</a></li>
-<li><a href="https://github.com/datafusion-contrib">DataFusion-Contrib</a></li>
-<li>A new GitHub organization created as a home for both <code>DataFusion</code> extensions and as a testing ground for new features.<ul>
-<li>Extensions</li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-python">DataFusion-Python</a></li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-java">DataFusion-Java</a></li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-hdfs-native">DataFusion-hdsfs-native</a></li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-objectstore-s3">DataFusion-ObjectStore-s3</a></li>
-<li>New Features</li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-streams">DataFusion-Streams</a></li>
-</ul>
-</li>
-<li><a href="https://github.com/jorgecarleitao/arrow2">Arrow2</a></li>
-<li>An <a href="https://github.com/apache/arrow-datafusion/tree/arrow2">Arrow2 Branch</a> has been created. There are ongoing discussions in <a href="https://github.com/apache/arrow-datafusion/issues/1532">DataFusion</a> and <a href="https://github.com/apache/arrow-rs/issues/1176">arrow-rs</a> about migrating <code>DataFusion</code> to <code>Arrow2</code></li>
-</ul>
-<h1>Documentation and Roadmap</h1>
-<p>We are working to consolidate the documentation into the <a href="https://arrow.apache.org/datafusion">official site</a>. You can find more details there on topics such as the <a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL status</a> and a <a href="https://arrow.apache.org/datafusion/user-guide/introduction.html#introduction">user guide</a>. This is also an area we would love to get help from the broader community <a href="https://github.com/apache/arrow [...]
-<p>To provide transparency on DataFusion’s priorities to users and developers a three month roadmap will be published at the beginning of each quarter. This can be found here <a href="https://arrow.apache.org/datafusion/specification/roadmap.html">here</a>.</p>
-<h1>Upcoming Attractions</h1>
-<ul>
-<li>Ballista is gaining momentum, and several groups are now evaluating and contributing to the project.</li>
-<li>Some of the proposed improvements<ul>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/1701">Improvements Overview</a></li>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/1675">Extensibility</a></li>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/1702">File system access</a></li>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/1704">Cluster state</a></li>
-</ul>
-</li>
-<li>Continued improvements for working with limited resources and large datasets</li>
-<li>Memory limited joins<a href="https://github.com/apache/arrow-datafusion/issues/1599">#1599</a></li>
-<li>Sort-merge join<a href="https://github.com/apache/arrow-datafusion/issues/141">#141</a><a href="https://github.com/apache/arrow-datafusion/pull/1776">#1776</a></li>
-<li>Introduce row based bytes representation <a href="https://github.com/apache/arrow-datafusion/pull/1708">#1708</a></li>
-</ul>
-<h1>How to Get Involved</h1>
-<p>If you are interested in contributing to DataFusion, and learning about state of
-the art query processing, we would love to have you join us on the journey! You
-can help by trying out DataFusion on some of your own data and projects and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
-<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2022/03/21/datafusion-contrib/index.html b/output/2022/03/21/datafusion-contrib/index.html
deleted file mode 100644
index 5ba4946..0000000
--- a/output/2022/03/21/datafusion-contrib/index.html
+++ /dev/null
@@ -1,195 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Introducing Apache Arrow DataFusion Contrib - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Introducing Apache Arrow DataFusion Contrib
- </h1>
- <p>Posted on: Mon 21 March 2022 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p>Apache Arrow <a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth checking out. DataFusion's pluggable design makes creating extensions at various points particular easy to build.</p>
-<p>DataFusion's SQL, <code>DataFrame</code>, and manual <code>PlanBuilder</code> API let users access a sophisticated query optimizer and execution engine capable of fast, resource efficient, and parallel execution that takes optimal advantage of todays multicore hardware. Being written in Rust means DataFusion can offer <em>both</em> the safety of dynamic languages as well as the resource efficiency of a compiled language.</p>
-<p>The DataFusion team is pleased to announce the creation of the <a href="https://github.com/datafusion-contrib">DataFusion-Contrib</a> GitHub organization to support and accelerate other projects. While the core DataFusion library remains under Apache governance, the contrib organization provides a more flexible testing ground for new DataFusion features and a home for DataFusion extensions. With this announcement, we are pleased to introduce the following inaugural DataFusion-Contri [...]
-<h2>DataFusion-Python</h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-python">project</a> provides Python bindings to the core Rust implementation of DataFusion, which allows users to:</p>
-<ul>
-<li>Work with familiar SQL or DataFrame APIs to run queries in a safe, multi-threaded environment, returning results in Python</li>
-<li>Create User Defined Functions and User Defined Aggregate Functions for complex operations</li>
-<li>Pay no overhead to copy between Python and underlying Rust execution engine (by way of Apache Arrow arrays)</li>
-</ul>
-<h3>Upcoming enhancements</h3>
-<p>The team is focusing on exposing more features from the underlying Rust implementation of DataFusion and improving documentation.</p>
-<h3>How to install</h3>
-<p>From <code>pip</code></p>
-<div class="codehilite"><pre><span></span><code>pip install datafusion
-</code></pre></div>
-<p>Or</p>
-<div class="codehilite"><pre><span></span><code>python -m pip install datafusion
-</code></pre></div>
-<h2>DataFusion-ObjectStore-S3</h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-objectstore-s3">crate</a> provides an <code>ObjectStore</code> implementation for querying data stored in S3 or S3 compatible storage. This makes it almost as easy to query data that lives on S3 as lives in local files</p>
-<ul>
-<li>Ability to create <code>S3FileSystem</code> to register as part of DataFusion <code>ExecutionContext</code></li>
-<li>Register files or directories stored on S3 with <code>ctx.register_listing_table</code></li>
-</ul>
-<h3>Upcoming enhancements</h3>
-<p>The current priority is adding python bindings for <code>S3FileSystem</code>. After that there will be async improvements as DataFusion adopts more of that functionality and we are looking into S3 Select functionality.</p>
-<h3>How to Install</h3>
-<p>Add the below to your <code>Cargo.toml</code> in your Rust Project with DataFusion.</p>
-<div class="codehilite"><pre><span></span><code><span class="n">datafusion-objectstore-s3</span> <span class="o">=</span> <span class="s">"0.1.0"</span>
-</code></pre></div>
-<h2>DataFusion-Substrait</h2>
-<p><a href="https://substrait.io/">Substrait</a> is an emerging standard that provides a cross-language serialization format for relational algebra (e.g. expressions and query plans).</p>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-substrait">crate</a> provides a Substrait producer and consumer for DataFusion. A producer converts a DataFusion logical plan into a Substrait protobuf and a consumer does the reverse.</p>
-<p>Examples of how to use this crate can be found <a href="https://github.com/datafusion-contrib/datafusion-substrait/blob/main/src/lib.rs">here</a>.</p>
-<h3>Potential Use Cases</h3>
-<ul>
-<li>Replace custom DataFusion protobuf serialization.</li>
-<li>Make it easier to pass query plans over FFI boundaries, such as from Python to Rust</li>
-<li>Allow Apache Calcite query plans to be executed in DataFusion</li>
-</ul>
-<h2>DataFusion-BigTable</h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-bigtable">crate</a> implements <a href="https://cloud.google.com/bigtable">Bigtable</a> as a data source and physical executor for DataFusion queries. It currently supports both UTF-8 string and 64-bit big-endian signed integers in Bigtable. From a SQL perspective it supports both simple and composite row keys with <code>=</code>, <code>IN</code>, and <code>BETWEEN</code> operators as well as projection pushdown. The ph [...]
-<h3>Upcoming Enhancements</h3>
-<ul>
-<li>Predicate pushdown</li>
-<li>Value range</li>
-<li>Value Regex</li>
-<li>Timestamp range</li>
-<li>Multithreaded</li>
-<li>Partition aware execution</li>
-<li>Production ready</li>
-</ul>
-<h3>How to Install</h3>
-<p>Add the below to your <code>Cargo.toml</code> in your Rust Project with DataFusion.</p>
-<div class="codehilite"><pre><span></span><code><span class="n">datafusion-bigtable</span> <span class="o">=</span> <span class="s">"0.1.0"</span>
-</code></pre></div>
-<h2>DataFusion-HDFS</h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-objectstore-hdfs">crate</a> introduces <code>HadoopFileSystem</code> as a remote <code>ObjectStore</code> which provides the ability to query HDFS files. For HDFS access the <a href="https://github.com/yahoNanJing/fs-hdfs">fs-hdfs</a> library is used.</p>
-<h2>DataFusion-Tokomak</h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-tokomak">crate</a> provides an e-graph based DataFusion optimization framework based on the Rust <a href="https://egraphs-good.github.io">egg</a> library. An e-graph is a data structure that powers the equality saturation optimization technique.</p>
-<p>As context, the optimizer framework within DataFusion is currently <a href="https://github.com/apache/arrow-datafusion/issues/1972">under review</a> with the objective of implementing a more strategic long term solution that is more efficient and simpler to develop.</p>
-<p>Some of the benefits of using <code>egg</code> within DataFusion are:</p>
-<ul>
-<li>Implements optimized algorithms that are hard to match with manually written optimization passes</li>
-<li>Makes it easy and less verbose to add optimization rules</li>
-<li>Plugin framework to add more complex optimizations</li>
-<li>Egg does not depend on rule order and can lead to a higher level of optimization by being able to apply multiple rules at the same time until it converges</li>
-<li>Allows for cost-based optimizations</li>
-</ul>
-<p>This is an exciting new area for DataFusion with lots of opportunity for community involvement!</p>
-<h2>DataFusion-Tui</h2>
-<p><a href="https://github.com/datafusion-contrib/datafusion-tui">DataFusion-tui</a> aka <code>dft</code> provides a feature rich terminal application for using DataFusion. It has drawn inspiration and several features from <code>datafusion-cli</code>. In contrast to <code>datafusion-cli</code> the objective of this tool is to provide a light SQL IDE experience for querying data with DataFusion. This includes features such as the following which are currently implemented:</p>
-<ul>
-<li>Tab Management to provide clean and structured organization of DataFusion queries, results, <code>ExecutionContext</code> information, and logs</li>
-<li>SQL Editor<ul>
-<li>Text editor for writing SQL queries</li>
-</ul>
-</li>
-<li>Query History<ul>
-<li>History of executed queries, their execution time, and the number of returned rows</li>
-</ul>
-</li>
-<li><code>ExecutionContext</code> information<ul>
-<li>Expose information on which physical optimizers are used and which <code>ExecutionConfig</code> settings are set</li>
-</ul>
-</li>
-<li>Logs<ul>
-<li>Logs from <code>dft</code>, DataFusion, and any dependent libraries</li>
-</ul>
-</li>
-<li>Support for custom <code>ObjectStore</code>s</li>
-<li>S3</li>
-<li>Preload DDL from <code>~/.datafusionrc</code> to enable having local "database" available at startup</li>
-</ul>
-<h3>Upcoming Enhancements</h3>
-<ul>
-<li>SQL Editor</li>
-<li>Command to write query results to file</li>
-<li>Multiple SQL editor tabs</li>
-<li>Expose more information from <code>ExecutionContext</code></li>
-<li>A help tab that provides information on functions</li>
-<li>Query custom <code>TableProvider</code>s such as <a href="https://github.com/delta-io/delta-rs">DeltaTable</a> or <a href="https://github.com/datafusion-contrib/datafusion-bigtable">BigTable</a></li>
-</ul>
-<h2>DataFusion-Streams</h2>
-<p><a href="https://github.com/datafusion-contrib/datafusion-streams">DataFusion-Stream</a> is a new testing ground for creating a <code>StreamProvider</code> in DataFusion that will enable querying streaming data sources such as Apache Kafka. The implementation for this feature is currently being designed and is under active review. Once the design is finalized the trait and attendant data structures will be added back to the core DataFusion crate.</p>
-<h2>DataFusion-Java</h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-java">project</a> created an initial set of Java bindings to DataFusion. The project is currently in maintenance mode and is looking for maintainers to drive future development.</p>
-<h1>How to Get Involved</h1>
-<p>If you are interested in contributing to DataFusion, and learning about state of
-the art query processing, we would love to have you join us on the journey! You
-can help by trying out DataFusion on some of your own data and projects and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
-<p>The best way to find out about creating new extensions within DataFusion-Contrib is reaching out on the <code>#arrow-rust</code> channel of the Apache Software Foundation <a href="https://join.slack.com/t/the-asf/shared_invite/zt-vlfbf7ch-HkbNHiU_uDlcH_RvaHv9gQ">Slack</a> workspace.</p>
-<p>You can also check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more ways to engage with the community.</p>
-<p>Links for each DataFusion-Contrib repository are provided above if you would like to contribute to those.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2022/05/16/datafusion-8.0.0/index.html b/output/2022/05/16/datafusion-8.0.0/index.html
deleted file mode 100644
index eb3e356..0000000
--- a/output/2022/05/16/datafusion-8.0.0/index.html
+++ /dev/null
@@ -1,227 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 8.0.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow DataFusion 8.0.0 Release
- </h1>
- <p>Posted on: Mon 16 May 2022 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that
-uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>,
-a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth
-checking out.</p>
-<p>DataFusion's SQL, <code>DataFrame</code>, and manual <code>PlanBuilder</code> API let users access a sophisticated query optimizer and
-execution engine capable of fast, resource efficient, and parallel execution that takes optimal advantage of
-today's multicore hardware. Being written in Rust means DataFusion can offer <em>both</em> the safety of a dynamic language and
-the resource efficiency of a compiled language.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 8.0.0 release (and also the release of version 0.7.0 of
-the Ballista subproject). This covers 3 months of development work and includes 279 commits from the following 49
-distinct contributors.</p>
-<!--
-$ git log --pretty=oneline 7.0.0..8.0.0 datafusion datafusion-cli datafusion-examples ballista ballista-cli ballista-examples | wc -l
-279
-
-$ git shortlog -sn 7.0.0..8.0.0 datafusion datafusion-cli datafusion-examples ballista ballista-cli ballista-examples | wc -l
-49
-
-(feynman han, feynman.h, Feynman Han were assumed to be the same person)
--->
-<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="mi">39</span><span class="w"> </span><span class="n">Andy</span><span class="w"> </span><span class="n">Grove</span><span class="w"></span>
-<span class="w"> </span><span class="mi">33</span><span class="w"> </span><span class="n">Andrew</span><span class="w"> </span><span class="n">Lamb</span><span class="w"></span>
-<span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="n">DuRipeng</span><span class="w"></span>
-<span class="w"> </span><span class="mi">20</span><span class="w"> </span><span class="n">Yijie</span><span class="w"> </span><span class="n">Shen</span><span class="w"></span>
-<span class="w"> </span><span class="mi">19</span><span class="w"> </span><span class="n">Yang</span><span class="w"> </span><span class="n">Jiang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">17</span><span class="w"> </span><span class="n">Raphael</span><span class="w"> </span><span class="n">Taylor</span><span class="o">-</span><span class="n">Davies</span><span class="w"></span>
-<span class="w"> </span><span class="mi">11</span><span class="w"> </span><span class="n">Dan</span><span class="w"> </span><span class="n">Harris</span><span class="w"></span>
-<span class="w"> </span><span class="mi">11</span><span class="w"> </span><span class="n">Matthew</span><span class="w"> </span><span class="n">Turner</span><span class="w"></span>
-<span class="w"> </span><span class="mi">11</span><span class="w"> </span><span class="n">yahoNanJing</span><span class="w"></span>
-<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">dependabot</span><span class="o">[</span><span class="n">bot</span><span class="o">]</span><span class="w"></span>
-<span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="n">jakevin</span><span class="w"></span>
-<span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">Kun</span><span class="w"> </span><span class="n">Liu</span><span class="w"></span>
-<span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="n">Jiayu</span><span class="w"> </span><span class="n">Liu</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">Daniël</span><span class="w"> </span><span class="n">Heres</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">mingmwang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">xudong</span><span class="p">.</span><span class="n">w</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Carol</span><span class="w"> </span><span class="p">(</span><span class="n">Nichols</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="n">Goulding</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Dmitry</span><span class="w"> </span><span class="n">Patsura</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Eduard</span><span class="w"> </span><span class="n">Karacharov</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Jeremy</span><span class="w"> </span><span class="n">Dyer</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Kaushik</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Rich</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">comphead</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">gaojun2048</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Feynman</span><span class="w"> </span><span class="n">Han</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Jie</span><span class="w"> </span><span class="n">Han</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Jon</span><span class="w"> </span><span class="n">Mease</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Tim</span><span class="w"> </span><span class="n">Van</span><span class="w"> </span><span class="n">Wassenhove</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Yt</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Zhang</span><span class="w"> </span><span class="n">Li</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">silence</span><span class="o">-</span><span class="n">coding</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Alexander</span><span class="w"> </span><span class="n">Spies</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">George</span><span class="w"> </span><span class="n">Andronchik</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Guillaume</span><span class="w"> </span><span class="n">Balaine</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Hao</span><span class="w"> </span><span class="n">Xin</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Jiacai</span><span class="w"> </span><span class="n">Liu</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Jörn</span><span class="w"> </span><span class="n">Horstmann</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Liang</span><span class="o">-</span><span class="n">Chi</span><span class="w"> </span><span class="n">Hsieh</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nf">Max</span><span class="w"> </span><span class="n">Burke</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">NaincyKumariKnoldus</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Nga</span><span class="w"> </span><span class="k">Tran</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Patrick</span><span class="w"> </span><span class="n">More</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Pierre</span><span class="w"> </span><span class="n">Zemb</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Remzi</span><span class="w"> </span><span class="n">Yang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Sergey</span><span class="w"> </span><span class="n">Melnychuk</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Stephen</span><span class="w"> </span><span class="n">Carman</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">doki</span><span class="w"></span>
-</code></pre></div>
-<p>The following sections highlight some of the changes in this release. Of course, many other bug fixes and
-improvements have been made and we encourage you to check out the
-<a href="https://github.com/apache/arrow-datafusion/blob/8.0.0/datafusion/CHANGELOG.md">changelog</a> for full details.</p>
-<h1>Summary</h1>
-<h2>DDL Support</h2>
-<p>DDL support has been expanded to include the following commands for creating databases, schemas, and views. This
-allows DataFusion to be used more effectively from the CLI.</p>
-<ul>
-<li><code>CREATE DATABASE</code></li>
-<li><code>CREATE VIEW</code></li>
-<li><code>CREATE SCHEMA</code></li>
-<li><code>CREATE EXTERNAL TABLE</code> now supports JSON files, <code>IF NOT EXISTS</code>, and partition columns</li>
-</ul>
-<h2>SQL Support</h2>
-<p>The SQL query planner now supports a number of new SQL features, including:</p>
-<ul>
-<li><em>Subqueries</em>: when used via <code>IN</code>, <code>EXISTS</code>, and as scalars</li>
-<li><em>Grouping Sets</em>: <code>CUBE</code> and <code>ROLLUP</code> grouping sets.</li>
-<li><em>Aggregate functions</em>: <code>approx_percentile</code>, <code>approx_percentile_cont</code>, <code>approx_percentile_cont_with_weight</code>, <code>approx_distinct</code>, <code>approx_median</code> and <code>array</code></li>
-<li><em><code>null</code> literals</em></li>
-<li><em>bitwise operations</em>: for example '<code>|</code>'</li>
-</ul>
-<p>There are also many bug fixes and improvements around normalizing identifiers consistently.</p>
-<p>We continue our tradition of incrementally releasing support for new
-features as they are developed. Thus, while the physical plan may not yet
-support all new features, it gets more complete each release. These
-changes also make DataFusion an increasingly compelling choice for
-projects looking for a SQL parser and query planner that can produce
-optimized logical plans that can be translated to
-their own execution engine.</p>
-<h2>Query Execution & Internals</h2>
-<p>There are several notable improvements and new features in the query execution engine:</p>
-<ul>
-<li>The <code>ExecutionContext</code> has been renamed to <code>SessionContext</code> and now supports multi-tenancy</li>
-<li>The <code>ExecutionPlan</code> trait is no longer <code>async</code></li>
-<li>A new serialization API for serializing plans to bytes (based on protobuf)</li>
-</ul>
-<p>In addition, we have added several foundational features to drive even
-more advanced query processing into DataFusion, focusing on running
-arbitrary queries larger than available memory, and pushing the
-envelope for performance of sorting, grouping, and joining even
-further:</p>
-<ul>
-<li>Morsel-Driven Scheduler based on <a href="https://15721.courses.cs.cmu.edu/spring2016/papers/p743-leis.pdf">"Morsel-Driven Parallelism: A NUMA-Aware Query
- Evaluation Framework for the Many-Core Age"</a></li>
-<li>Consolidated object store implementation and integration with parquet decoding</li>
-<li>Memory Limited Spilling sort operator</li>
-<li>Memory Limited Sort-Merge join operator</li>
-<li>High performance JIT code generation for tuple comparisons</li>
-<li>Memory efficient Row Format</li>
-</ul>
-<h2>Improved file support</h2>
-<p>DataFusion now supports JSON, both for reading and writing. There are also new DataFrame methods for writing query
-results to files in CSV, Parquet, and JSON format.</p>
-<h2>Ballista</h2>
-<p>Ballista continues to mature and now supports a wider range of operators and expressions. There are also improvements
-to the scheduler to support UDFs, and there are some robustness improvements, such as cleaning up work directories
-and persisting session configs to allow schedulers to restart and continue processing in-flight jobs.</p>
-<h2>Upcoming Work</h2>
-<p>Here are some of the initiatives that the community plans on working on prior to the next release.</p>
-<ul>
-<li>There is a <a href="https://docs.google.com/document/d/1jNRbadyStSrV5kifwn0khufAwq6OnzGczG4z8oTQJP4/edit?usp=sharing">proposal to move Ballista to its own top-level arrow-ballista repository</a>
- to decouple DataFusion and Ballista releases and to allow each project to have documentation better targeted at
- its particular audience.</li>
-<li>We plan on increasing the frequency of DataFusion releases, with monthly releases now instead of quarterly. This
- is driven by requests from the increasing number of projects that now depend on DataFusion.</li>
-<li>There is ongoing work to implement new optimizer rules to rewrite queries containing subquery expressions as
- joins, to support a wider range of queries.</li>
-<li>The new scheduler based on morsel-driven execution will continue to evolve in this next release, with work to
- refine IO abstractions to improve performance and integration with the new scheduler.</li>
-<li>Improved performance for Sort, Grouping and Joins</li>
-</ul>
-<h1>How to Get Involved</h1>
-<p>If you are interested in contributing to DataFusion, and learning about state-of-the-art query processing, we would
-love to have you join us on the journey! You can help by trying out DataFusion on some of your own data and projects
-and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable
-for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
-<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2022/10/25/datafusion-13.0.0/index.html b/output/2022/10/25/datafusion-13.0.0/index.html
deleted file mode 100644
index 050f10b..0000000
--- a/output/2022/10/25/datafusion-13.0.0/index.html
+++ /dev/null
@@ -1,256 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 13.0.0 Project Update - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow DataFusion 13.0.0 Project Update
- </h1>
- <p>Posted on: Tue 25 October 2022 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> <a href="https://crates.io/crates/datafusion"><code>13.0.0</code></a> is released, and this blog contains an update on the project for the 5 months since our <a href="https://arrow.apache.org/blog/2022/05/16/datafusion-8.0.0/">last update in May 2022</a>.</p>
-<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL processes, and database systems. You may want to check out DataFusion to extend your Rust project to:</p>
-<ul>
-<li>Support <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a></li>
-<li>Support <a href="https://docs.rs/datafusion/13.0.0/datafusion/dataframe/struct.DataFrame.html">DataFrame API</a></li>
-<li>Support a Domain Specific Query Language</li>
-<li>Easily and quickly read and process Parquet, JSON, Avro or CSV data.</li>
-<li>Read from remote object stores such as AWS S3, Azure Blob Storage, GCP.</li>
-</ul>
-<p>Even though DataFusion is 4 years "young," it has seen significant community growth in the last few months and the momentum continues to accelerate.</p>
-<h1>Background</h1>
-<p>DataFusion is used as the engine in <a href="https://github.com/apache/arrow-datafusion#known-uses">many open source and commercial projects</a> and was one of the early open source projects to provide this capability. 2022 has validated our belief in the need for such a <a href="https://docs.google.com/presentation/d/1iNX_35sWUakee2q3zMFPyHE4IV2nC3lkCK_H6Y2qK84/edit#slide=id.p">"LLVM for database and AI systems"</a><a href="https://www.slideshare.net/AndrewLamb32/20220623-apache-arro [...]
-<p>While Velox and Acero focus on execution engines, DataFusion provides the entire suite of components needed to build most analytic systems, including a SQL frontend, a dataframe API, and extension points for just about everything. Some <a href="https://github.com/apache/arrow-datafusion#known-uses">DataFusion users</a> use a subset of the features such as the frontend (e.g. <a href="https://dask-sql.readthedocs.io/en/latest/">dask-sql</a>) or the execution engine, (e.g. <a href="htt [...]
-<p>One of DataFusion’s advantages is its implementation in <a href="https://www.rust-lang.org/">Rust</a> and thus its easy integration with the broader Rust ecosystem. Rust continues to be a major source of benefit, from the <a href="https://www.influxdata.com/blog/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/">ease of parallelization with the high quality and standardized <code>async</code> ecosystem</a> , as well as its modern dependency management system and wonderful [...]
-<!--While we haven’t invested in the benchmarking ratings game datafusion continues to be quite speedy (todo quantity this, with some evidence) – maybe clickbench?-->
-<!--
-Maybe we can do this un a future post
-# DataFusion in Action
-
-While DataFusion really shines as an embeddable query engine, if you want to try it out and get a feel for its power, you can use the basic[`datafusion-cli`](https://docs.rs/datafusion-cli/13.0.0/datafusion_cli/) tool to get a sense for what is possible to add in your application
-
-(TODO example here of using datafusion-cli to query from local parquet files on disk)
-
-TODO: also mention you can use the same thing to query data from S3
--->
-<h1>Summary</h1>
-<p>We have increased the frequency of DataFusion releases to monthly instead of quarterly. This
-makes it easier for the increasing number of projects that now depend on DataFusion.</p>
-<p>We have also completed the "graduation" of <a href="https://github.com/apache/arrow-ballista">Ballista to its own top-level arrow-ballista repository</a>
-which decouples the two projects and allows each project to move even faster.</p>
-<p>Along with numerous other bug fixes and smaller improvements, here are some of the major advances:</p>
-<h1>Improved Support for Cloud Object Stores</h1>
-<p>DataFusion now supports many major cloud object stores (Amazon S3, Azure Blob Storage, and Google Cloud Storage) "out of the box" via the <a href="https://crates.io/crates/object_store">object_store</a> crate. Using this integration, DataFusion optimizes reading parquet files by reading only the parts of the files that are needed.</p>
-<h2>Advanced SQL</h2>
-<p>DataFusion now supports correlated subqueries, by rewriting them as joins. See the <a href="https://arrow.apache.org/datafusion/user-guide/sql/subqueries.html">Subquery</a> page in the User Guide for more information.</p>
-<p>In addition to numerous other small improvements, the following SQL features are now supported:</p>
-<ul>
-<li><code>ROWS</code>, <code>RANGE</code>, <code>PRECEDING</code> and <code>FOLLOWING</code> in <code>OVER</code> clauses <a href="https://github.com/apache/arrow-datafusion/issues/3570">#3570</a></li>
-<li><code>ROLLUP</code> and <code>CUBE</code> grouping set expressions <a href="https://github.com/apache/arrow-datafusion/issues/2446">#2446</a></li>
-<li><code>SUM DISTINCT</code> aggregate support <a href="https://github.com/apache/arrow-datafusion/issues/2405">#2405</a></li>
-<li><code>IN</code> and <code>NOT IN</code> Subqueries by rewriting them to <code>SEMI</code> / <code>ANTI</code> <a href="https://github.com/apache/arrow-datafusion/issues/2885">#2421</a></li>
-<li>Non equality predicates in <code>ON</code> clause of <code>LEFT</code>, <code>RIGHT,</code>and <code>FULL</code> joins <a href="https://github.com/apache/arrow-datafusion/issues/2591">#2591</a></li>
-<li>Exact <code>MEDIAN</code> <a href="https://github.com/apache/arrow-datafusion/issues/3009">#3009</a></li>
-<li><code>GROUPING SETS</code>/<code>CUBE</code>/<code>ROLLUP</code> <a href="https://github.com/apache/arrow-datafusion/issues/2716">#2716</a></li>
-</ul>
-<h1>More DDL Support</h1>
-<p>Just as it is important to query, it is also important to give users the ability to define their data sources. We have added:</p>
-<ul>
-<li><code>CREATE VIEW</code> <a href="https://github.com/apache/arrow-datafusion/issues/2279">#2279</a></li>
-<li><code>DESCRIBE <table></code> <a href="https://github.com/apache/arrow-datafusion/issues/2642">#2642</a></li>
-<li>Custom / Dynamic table provider factories <a href="https://github.com/apache/arrow-datafusion/issues/3311">#3311</a></li>
-<li><code>SHOW CREATE TABLE</code> for support for views <a href="https://github.com/apache/arrow-datafusion/issues/2830">#2830</a></li>
-</ul>
-<h1>Faster Execution</h1>
-<p>Performance is always an important goal for DataFusion, and there are a number of significant new optimizations such as</p>
-<ul>
-<li>Optimizations of TopK (queries with a <code>LIMIT</code> or <code>OFFSET</code> clause): <a href="https://github.com/apache/arrow-datafusion/issues/3527">#3527</a>, <a href="https://github.com/apache/arrow-datafusion/issues/2521">#2521</a></li>
-<li>Reduce <code>left</code>/<code>right</code>/<code>full</code> joins to <code>inner</code> join <a href="https://github.com/apache/arrow-datafusion/issues/2750">#2750</a></li>
-<li>Convert cross joins to inner joins when possible <a href="https://github.com/apache/arrow-datafusion/issues/3482">#3482</a></li>
-<li>Sort preserving <code>SortMergeJoin</code> <a href="https://github.com/apache/arrow-datafusion/issues/2699">#2699</a></li>
-<li>Improvements in group by and sort performance <a href="https://github.com/apache/arrow-datafusion/issues/2375">#2375</a></li>
-<li>Adaptive <code>regex_replace</code> implementation <a href="https://github.com/apache/arrow-datafusion/issues/3518">#3518</a></li>
-</ul>
-<h1>Optimizer Enhancements</h1>
-<p>Internally the optimizer has been significantly enhanced as well.</p>
-<ul>
-<li>Casting / coercion now happens during logical planning <a href="https://github.com/apache/arrow-datafusion/issues/3396">#3185</a> <a href="https://github.com/apache/arrow-datafusion/issues/3636">#3636</a></li>
-<li>More sophisticated expression analysis and simplification is available</li>
-</ul>
-<h1>Parquet</h1>
-<ul>
-<li>The parquet reader can now read directly from parquet files on remote object storage <a href="https://github.com/apache/arrow-datafusion/issues/2677">#2489</a> <a href="https://github.com/apache/arrow-datafusion/issues/3051">#3051</a></li>
-<li>Experimental support for “predicate pushdown” with late materialization after filtering during the scan (another blog post on this topic is coming soon).</li>
-<li>Support reading directly from AWS S3 and other object stores via <code>datafusion-cli</code> <a href="https://github.com/apache/arrow-datafusion/issues/3631">#3631</a></li>
-</ul>
-<h1>DataType Support</h1>
-<ul>
-<li>Support for <code>TimestampTz</code> <a href="https://github.com/apache/arrow-datafusion/issues/3660">#3660</a></li>
-<li>Expanded support for the <code>Decimal</code> type, including <code>IN</code> list and better built in coercion.</li>
-<li>Expanded support for date/time manipulation such as <code>date_bin</code> built-in function , timestamp <code>+/-</code> interval, <code>TIME</code> literal values <a href="https://github.com/apache/arrow-datafusion/issues/3010">#3010</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3110">#3110</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3034">#3034</a></li>
-<li>Binary operations (<code>AND</code>, <code>XOR</code>, etc): <a href="https://github.com/apache/arrow-datafusion/issues/1619">#3037</a> <a href="https://github.com/apache/arrow-datafusion/issues/3430">#3420</a></li>
-<li><code>IS TRUE/FALSE</code> and <code>IS [NOT] UNKNOWN</code> <a href="https://github.com/apache/arrow-datafusion/issues/3235">#3235</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3246">#3246</a></li>
-</ul>
-<h2>Upcoming Work</h2>
-<p>With the community growing and code accelerating, there is so much great stuff on the horizon. Some features we expect to land in the next few months:</p>
-<ul>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/3462">Complete Parquet Pushdown</a></li>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/3148">Additional date/time support</a></li>
-<li>Cost models, Nested Join Optimizations, analysis framework <a href="https://github.com/apache/arrow-datafusion/issues/128">#128</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3843">#3843</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3845">#3845</a></li>
-</ul>
-<h1>Community Growth</h1>
-<p>The DataFusion 9.0.0 and 13.0.0 releases consists of 433 PRs from 64 distinct contributors. This does not count all the work that goes into our dependencies such as <a href="https://crates.io/crates/arrow">arrow</a>, <a href="https://crates.io/crates/parquet">parquet</a>, and <a href="https://crates.io/crates/object_store">object_store</a>, that much of the same community helps nurture.</p>
-<!--
-$ git log --pretty=oneline 9.0.0..13.0.0 . | wc -l
-433
-
-$ git shortlog -sn 9.0.0..13.0.0 . | wc -l
-65
--->
-<h1>How to Get Involved</h1>
-<p>Kudos to everyone in the community who contributed ideas, discussions, bug reports, documentation and code. It is exciting to be building something so cool together!</p>
-<p>If you are interested in contributing to DataFusion, we would love to
-have you join us on our journey to create the most advanced open
-source query engine. You can try out DataFusion on some of your own
-data and projects and let us know how it goes or contribute a PR with
-documentation, tests or code. A list of open issues suitable for
-beginners is
-<a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
-<p>Check out our <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
-<h2>Appendix: Contributor Shoutout</h2>
-<p>To give a sense of the number of people who contribute to this project regularly, we present for your consideration the following list derived from <code>git shortlog -sn 9.0.0..13.0.0 .</code> Thank you all again!</p>
-<!-- Note: combined kmitchener and Kirk Mitchener -->
-<div class="codehilite"><pre><span></span><code><span class="err"> 87 Andy Grove</span>
-<span class="err"> 71 Andrew Lamb</span>
-<span class="err"> 29 Kun Liu</span>
-<span class="err"> 29 Kirk Mitchener</span>
-<span class="err"> 17 Wei-Ting Kuo</span>
-<span class="err"> 14 Yang Jiang</span>
-<span class="err"> 12 Raphael Taylor-Davies</span>
-<span class="err"> 11 Batuhan Taskaya</span>
-<span class="err"> 10 Brent Gardner</span>
-<span class="err"> 10 Remzi Yang</span>
-<span class="err"> 10 comphead</span>
-<span class="err"> 10 xudong.w</span>
-<span class="err"> 8 AssHero</span>
-<span class="err"> 7 Ruihang Xia</span>
-<span class="err"> 6 Dan Harris</span>
-<span class="err"> 6 Daniël Heres</span>
-<span class="err"> 6 Ian Alexander Joiner</span>
-<span class="err"> 6 Mike Roberts</span>
-<span class="err"> 6 askoa</span>
-<span class="err"> 4 BaymaxHWY</span>
-<span class="err"> 4 gorkem</span>
-<span class="err"> 4 jakevin</span>
-<span class="err"> 3 George Andronchik</span>
-<span class="err"> 3 Sarah Yurick</span>
-<span class="err"> 3 Stuart Carnie</span>
-<span class="err"> 2 Dalton Modlin</span>
-<span class="err"> 2 Dmitry Patsura</span>
-<span class="err"> 2 JasonLi</span>
-<span class="err"> 2 Jon Mease</span>
-<span class="err"> 2 Marco Neumann</span>
-<span class="err"> 2 yahoNanJing</span>
-<span class="err"> 1 Adilet Sarsembayev</span>
-<span class="err"> 1 Ayush Dattagupta</span>
-<span class="err"> 1 Dezhi Wu</span>
-<span class="err"> 1 Dhamotharan Sritharan</span>
-<span class="err"> 1 Eduard Karacharov</span>
-<span class="err"> 1 Francis Du</span>
-<span class="err"> 1 Harbour Zheng</span>
-<span class="err"> 1 Ismaël Mejía</span>
-<span class="err"> 1 Jack Klamer</span>
-<span class="err"> 1 Jeremy Dyer</span>
-<span class="err"> 1 Jiayu Liu</span>
-<span class="err"> 1 Kamil Konior</span>
-<span class="err"> 1 Liang-Chi Hsieh</span>
-<span class="err"> 1 Martin Grigorov</span>
-<span class="err"> 1 Matthijs Brobbel</span>
-<span class="err"> 1 Mehmet Ozan Kabak</span>
-<span class="err"> 1 Metehan Yıldırım</span>
-<span class="err"> 1 Morgan Cassels</span>
-<span class="err"> 1 Nitish Tiwari</span>
-<span class="err"> 1 Renjie Liu</span>
-<span class="err"> 1 Rito Takeuchi</span>
-<span class="err"> 1 Robert Pack</span>
-<span class="err"> 1 Thomas Cameron</span>
-<span class="err"> 1 Vrishabh</span>
-<span class="err"> 1 Xin Hao</span>
-<span class="err"> 1 Yijie Shen</span>
-<span class="err"> 1 byteink</span>
-<span class="err"> 1 kamille</span>
-<span class="err"> 1 mateuszkj</span>
-<span class="err"> 1 nvartolomei</span>
-<span class="err"> 1 yourenawo</span>
-<span class="err"> 1 Özgür Akkurt</span>
-</code></pre></div>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2022/10/28/ballista-0.9.0/index.html b/output/2022/10/28/ballista-0.9.0/index.html
deleted file mode 100644
index 143f777..0000000
--- a/output/2022/10/28/ballista-0.9.0/index.html
+++ /dev/null
@@ -1,136 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow Ballista 0.9.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow Ballista 0.9.0 Release
- </h1>
- <p>Posted on: Fri 28 October 2022 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://github.com/apache/arrow-ballista">Ballista</a> is an Arrow-native distributed SQL query engine implemented in Rust.</p>
-<p>Ballista 0.9.0 is now available and is the most significant release since the project was <a href="http://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to Apache
-Arrow in 2021.</p>
-<p>This release represents 4 weeks of work, with 66 commits from 14 contributors:</p>
-<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="mi">22</span><span class="w"> </span><span class="n">Andy</span><span class="w"> </span><span class="n">Grove</span><span class="w"></span>
-<span class="w"> </span><span class="mi">12</span><span class="w"> </span><span class="n">yahoNanJing</span><span class="w"></span>
-<span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">Daniël</span><span class="w"> </span><span class="n">Heres</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">Brent</span><span class="w"> </span><span class="n">Gardner</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">dependabot</span><span class="o">[</span><span class="n">bot</span><span class="o">]</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">r</span><span class="mf">.4</span><span class="n">ntix</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Stefan</span><span class="w"> </span><span class="n">Stanciulescu</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">mingmwang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Ken</span><span class="w"> </span><span class="n">Suenobu</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Yang</span><span class="w"> </span><span class="n">Jiang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Metehan</span><span class="w"> </span><span class="n">Yıldırım</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Trent</span><span class="w"> </span><span class="n">Feda</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">askoa</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">yangzhong</span><span class="w"></span>
-</code></pre></div>
-<h2>Release Highlights</h2>
-<p>The release notes below are not exhaustive and only expose selected highlights of the release. Many other bug fixes
-and improvements have been made: we refer you to the <a href="https://github.com/apache/arrow-ballista/blob/0.9.0-rc2/ballista/CHANGELOG.md">complete changelog</a>.</p>
-<h3>Support for Cloud Object Stores and Distributed File Systems</h3>
-<p>This is the first release of Ballista to have documented support for querying data from distributed file systems and
-object stores. Currently, S3 and HDFS are supported. Support for Google Cloud Storage and Azure Blob Storage is planned
-for the next release.</p>
-<h3>Flight SQL & JDBC support</h3>
-<p>The Ballista scheduler now implements the <a href="https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/">Flight SQL protocol</a>, enabling any compliant Flight SQL client
-to connect to and run queries against a Ballista cluster.</p>
-<p>The Apache Arrow Flight SQL JDBC driver can be used to connect Business Intelligence tools to a Ballista cluster.</p>
-<h3>Python Bindings</h3>
-<p>It is now possible to connect to a Ballista cluster from Python and execute queries using both the DataFrame and SQL
-interfaces.</p>
-<h3>Scheduler Web User Interface and REST API</h3>
-<p>The scheduler now has a web user interface for monitoring queries. It is also possible to view graphical query plans
-that show how the query was executed, along with metrics.</p>
-<p><img src="../images/2022-10-28-ballista-web-ui.png" width="800"/></p>
-<p>The REST API that powers the user interface can also be accessed directly.</p>
-<h3>Simplified Kubernetes Deployment</h3>
-<p>Ballista now provides a <a href="https://github.com/apache/arrow-ballista/tree/master/helm">Helm chart</a> for simplified Kubernetes deployment.</p>
-<h3>User Guide</h3>
-<p>The user guide is published at <a href="https://arrow.apache.org/ballista/">https://arrow.apache.org/ballista/</a> and provides
-deployment instructions for Docker, Docker Compose, and Kubernetes, as well as references for configuring and
-tuning Ballista.</p>
-<h2>Roadmap</h2>
-<p>The Ballista community is currently focused on the following tasks for the next release:</p>
-<ul>
-<li>Support for Azure Blob Storage and Google Cloud Storage</li>
-<li>Improve benchmark performance by implementing more query optimizations</li>
-<li>Improve scheduler web user interface</li>
-<li>Publish Docker images to GitHub Container Registry</li>
-</ul>
-<p>The detailed list of issues planned for the 0.10.0 release can be found in the <a href="https://github.com/apache/arrow-ballista/issues/361">tracking issue</a>.</p>
-<h2>Getting Involved</h2>
-<p>Ballista has a friendly community and we welcome contributions. A good place to start is to following the instructions
-in the <a href="https://arrow.apache.org/ballista/">user guide</a> and try using Ballista with your own SQL queries and ETL pipelines, and file issues
-for any bugs or feature suggestions.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2023/01/19/datafusion-16.0.0/index.html b/output/2023/01/19/datafusion-16.0.0/index.html
deleted file mode 100644
index b462ae8..0000000
--- a/output/2023/01/19/datafusion-16.0.0/index.html
+++ /dev/null
@@ -1,277 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 16.0.0 Project Update - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow DataFusion 16.0.0 Project Update
- </h1>
- <p>Posted on: Thu 19 January 2023 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible
-query execution framework, written in <a href="https://www.rust-lang.org/">Rust</a>,
-that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its
-in-memory format. It is targeted primarily at developers creating data
-intensive analytics, and offers mature
-<a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>,
-a DataFrame API, and many extension points.</p>
-<p>Systems based on DataFusion perform very well in benchmarks,
-especially considering they operate directly on parquet files rather
-than first loading into a specialized format. Some recent highlights
-include <a href="https://benchmark.clickhouse.com/">clickbench</a> and the
-<a href="https://www.cloudfuse.io/dashboards/standalone-engines">Cloudfuse.io standalone query
-engines</a> page.</p>
-<p>DataFusion is also part of a longer term trend, articulated clearly by
-<a href="http://www.cs.cmu.edu/~pavlo/">Andy Pavlo</a> in his <a href="https://ottertune.com/blog/2022-databases-retrospective/">2022 Databases
-Retrospective</a>.
-Database frameworks are proliferating and it is likely that all OLAP
-DBMSs and other data heavy applications, such as machine learning,
-will <strong>require</strong> a vectorized, highly performant query engine in the next
-5 years to remain relevant. The only practical way to make such
-technology so widely available without many millions of dollars of
-investment is though open source engine such as DataFusion or
-<a href="https://github.com/facebookincubator/velox">Velox</a>.</p>
-<p>The rest of this post describes the improvements made to DataFusion
-over the last three months and some hints of where we are heading.</p>
-<h2>Community Growth</h2>
-<p>We again saw significant growth in the DataFusion community since <a href="https://arrow.apache.org/blog/2022/10/25/datafusion-13.0.0/">our last update</a>. There are some interesting metrics on <a href="https://ossrank.com/p/1573-apache-arrow-datafusion">OSSRank</a>.</p>
-<p>The DataFusion 16.0.0 release consists of 543 PRs from 73 distinct contributors, not including all the work that goes into dependencies such as <a href="https://crates.io/crates/arrow">arrow</a>, <a href="https://crates.io/crates/parquet">parquet</a>, and <a href="https://crates.io/crates/object_store">object_store</a>, that much of the same community helps support. Thank you all for your help</p>
-<!--
-$ git log --pretty=oneline 13.0.0..16.0.0 . | wc -l
- 543
-
-$ git shortlog -sn 13.0.0..16.0.0 . | wc -l
- 73
--->
-<p>Several <a href="https://github.com/apache/arrow-datafusion#known-uses">new systems based on DataFusion</a> were recently added:</p>
-<ul>
-<li><a href="https://github.com/GreptimeTeam/greptimedb">Greptime DB</a></li>
-<li><a href="https://synnada.ai/">Synnada</a></li>
-<li><a href="https://github.com/PRQL/prql-query">PRQL</a></li>
-<li><a href="https://github.com/parseablehq/parseable">Parseable</a></li>
-<li><a href="https://github.com/splitgraph/seafowl">SeaFowl</a></li>
-</ul>
-<h2>Performance 🚀</h2>
-<p>Performance and efficiency are core values for
-DataFusion. While there is still a gap between DataFusion and the best of
-breed, tightly integrated systems such as <a href="https://duckdb.org">DuckDB</a>
-and <a href="https://www.pola.rs/">Polars</a>, DataFusion is
-closing the gap quickly. Performance highlights from the last three
-months:</p>
-<ul>
-<li>Up to 30% Faster Sorting and Merging using the new <a href="https://arrow.apache.org/blog/2022/11/07/multi-column-sorts-in-arrow-rust-part-1/">Row Format</a></li>
-<li><a href="https://arrow.apache.org/blog/2022/12/26/querying-parquet-with-millisecond-latency/">Advanced predicate pushdown</a>, directly on parquet, directly from object storage, enabling sub millisecond filtering. <!-- Andrew nots: we should really get this turned on by default --></li>
-<li><code>70%</code> faster <code>IN</code> expressions evaluation (<a href="https://github.com/apache/arrow-datafusion/issues/4057">#4057</a>)</li>
-<li>Sort and partition aware optimizations (<a href="https://github.com/apache/arrow-datafusion/issues/3969">#3969</a> and <a href="https://github.com/apache/arrow-datafusion/issues/4691">#4691</a>)</li>
-<li>Filter selectivity analysis (<a href="https://github.com/apache/arrow-datafusion/issues/3868">#3868</a>)</li>
-</ul>
-<h2>Runtime Resource Limits</h2>
-<p>Previously, DataFusion could potentially use unbounded amounts of memory for certain queries that included Sorts, Grouping or Joins.</p>
-<p>In version 16.0.0, it is possible to limit DataFusion's memory usage for Sorting and Grouping. We are looking for help adding similar limiting for Joins as well as expanding our algorithms to optionally spill to secondary storage. See <a href="https://github.com/apache/arrow-datafusion/issues/3941">#3941</a> for more detail.</p>
-<h2>SQL Window Functions</h2>
-<p><a href="https://en.wikipedia.org/wiki/Window_function_(SQL)">SQL Window Functions</a> are useful for a variety of analysis and DataFusion's implementation support expanded significantly:</p>
-<ul>
-<li>Custom window frames such as <code>... OVER (ORDER BY ... RANGE BETWEEN 0.2 PRECEDING AND 0.2 FOLLOWING)</code></li>
-<li>Unbounded window frames such as <code>... OVER (ORDER BY ... RANGE UNBOUNDED ROWS PRECEDING)</code></li>
-<li>Support for the <code>NTILE</code> window function (<a href="https://github.com/apache/arrow-datafusion/issues/4676">#4676</a>)</li>
-<li>Support for <code>GROUPS</code> mode (<a href="https://github.com/apache/arrow-datafusion/issues/4155">#4155</a>)</li>
-</ul>
-<h1>Improved Joins</h1>
-<p>Joins are often the most complicated operations to handle well in
-analytics systems and DataFusion 16.0.0 offers significant improvements
-such as</p>
-<ul>
-<li>Cost based optimizer (CBO) automatically reorders join evaluations, selects algorithms (Merge / Hash), and pick build side based on available statistics and join type (<code>INNER</code>, <code>LEFT</code>, etc) (<a href="https://github.com/apache/arrow-datafusion/issues/4219">#4219</a>)</li>
-<li>Fast non <code>column=column</code> equijoins such as <code>JOIN ON a.x + 5 = b.y</code></li>
-<li>Better performance on non-equijoins (<a href="https://github.com/apache/arrow-datafusion/issues/4562">#4562</a>) <!-- TODO is this a good thing to mention as any time this is usd the query is going to go slow or the data size is small --></li>
-</ul>
-<h1>Streaming Execution</h1>
-<p>One emerging use case for Datafusion is as a foundation for
-streaming-first data platforms. An important prerequisite
-is support for incremental execution for queries that can be computed
-incrementally.</p>
-<p>With this release, DataFusion now supports the following streaming features:</p>
-<ul>
-<li>Data ingestion from infinite files such as FIFOs (<a href="https://github.com/apache/arrow-datafusion/issues/4694">#4694</a>),</li>
-<li>Detection of pipeline-breaking queries in streaming use cases (<a href="https://github.com/apache/arrow-datafusion/issues/4694">#4694</a>),</li>
-<li>Automatic input swapping for joins so probe side is a data stream (<a href="https://github.com/apache/arrow-datafusion/issues/4694">#4694</a>),</li>
-<li>Intelligent elision of pipeline-breaking sort operations whenever possible (<a href="https://github.com/apache/arrow-datafusion/issues/4691">#4691</a>),</li>
-<li>Incremental execution for more types of queries; e.g. queries involving finite window frames (<a href="https://github.com/apache/arrow-datafusion/issues/4777">#4777</a>).</li>
-</ul>
-<p>These are a major steps forward, and we plan even more improvements over the next few releases.</p>
-<h1>Better Support for Distributed Catalogs</h1>
-<p>16.0.0 has been enhanced support for asynchronous catalogs (<a href="https://github.com/apache/arrow-datafusion/issues/4607">#4607</a>)
-to better support distributed metadata stores such as
-<a href="https://delta.io/">Delta.io</a> and <a href="https://iceberg.apache.org/">Apache
-Iceberg</a> which require asynchronous I/O
-during planning to access remote catalogs. Previously, DataFusion
-required synchronous access to all relevant catalog information.</p>
-<h1>Additional SQL Support</h1>
-<p>SQL support continues to improve, including some of these highlights:</p>
-<ul>
-<li>Add TPC-DS query planning regression tests <a href="https://github.com/apache/arrow-datafusion/issues/4719">#4719</a></li>
-<li>Support for <code>PREPARE</code> statement <a href="https://github.com/apache/arrow-datafusion/issues/4490">#4490</a></li>
-<li>Automatic coercions ast between Date and Timestamp <a href="https://github.com/apache/arrow-datafusion/issues/4726">#4726</a></li>
-<li>Support type coercion for timestamp and utf8 <a href="https://github.com/apache/arrow-datafusion/issues/4312">#4312</a></li>
-<li>Full support for time32 and time64 literal values (<code>ScalarValue</code>) <a href="https://github.com/apache/arrow-datafusion/issues/4156">#4156</a></li>
-<li>New functions, incuding <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
-<li>Compressed CSV/JSON support <a href="https://github.com/apache/arrow-datafusion/issues/3642">#3642</a></li>
-</ul>
-<p>The community has also invested in new <a href="https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/tests/sqllogictests/README.md">sqllogic based</a> tests to keep improving DataFusion's quality with less effort.</p>
-<h1>Plan Serialization and Substrait</h1>
-<p>DataFusion now supports serialization of physical plans, with a custom protocol buffers format. In addition, we are adding initial support for <a href="https://substrait.io/">Substrait</a>, a Cross-Language Serialization for Relational Algebra</p>
-<h1>How to Get Involved</h1>
-<p>Kudos to everyone in the community who contributed ideas, discussions, bug reports, documentation and code. It is exciting to be building something so cool together!</p>
-<p>If you are interested in contributing to DataFusion, we would love to
-have you join us. You can try out DataFusion on some of your own
-data and projects and let us know how it goes or contribute a PR with
-documentation, tests or code. A list of open issues suitable for
-beginners is
-<a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
-<p>Check out our <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
-<h2>Appendix: Contributor Shoutout</h2>
-<p>Here is a list of people who have contributed PRs to this project over the last three releases, derived from <code>git shortlog -sn 13.0.0..16.0.0 .</code> Thank you all!</p>
-<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="mi">113</span><span class="w"> </span><span class="n">Andrew</span><span class="w"> </span><span class="n">Lamb</span><span class="w"></span>
-<span class="w"> </span><span class="mi">58</span><span class="w"> </span><span class="n">jakevin</span><span class="w"></span>
-<span class="w"> </span><span class="mi">46</span><span class="w"> </span><span class="n">Raphael</span><span class="w"> </span><span class="n">Taylor</span><span class="o">-</span><span class="n">Davies</span><span class="w"></span>
-<span class="w"> </span><span class="mi">30</span><span class="w"> </span><span class="n">Andy</span><span class="w"> </span><span class="n">Grove</span><span class="w"></span>
-<span class="w"> </span><span class="mi">19</span><span class="w"> </span><span class="n">Batuhan</span><span class="w"> </span><span class="n">Taskaya</span><span class="w"></span>
-<span class="w"> </span><span class="mi">19</span><span class="w"> </span><span class="n">Remzi</span><span class="w"> </span><span class="n">Yang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">17</span><span class="w"> </span><span class="n">ygf11</span><span class="w"></span>
-<span class="w"> </span><span class="mi">16</span><span class="w"> </span><span class="n">Burak</span><span class="w"></span>
-<span class="w"> </span><span class="mi">16</span><span class="w"> </span><span class="n">Jeffrey</span><span class="w"></span>
-<span class="w"> </span><span class="mi">16</span><span class="w"> </span><span class="n">Marco</span><span class="w"> </span><span class="n">Neumann</span><span class="w"></span>
-<span class="w"> </span><span class="mi">14</span><span class="w"> </span><span class="n">Kun</span><span class="w"> </span><span class="n">Liu</span><span class="w"></span>
-<span class="w"> </span><span class="mi">12</span><span class="w"> </span><span class="n">Yang</span><span class="w"> </span><span class="n">Jiang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="n">mingmwang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">Daniël</span><span class="w"> </span><span class="n">Heres</span><span class="w"></span>
-<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">Mustafa</span><span class="w"> </span><span class="n">akur</span><span class="w"></span>
-<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">comphead</span><span class="w"></span>
-<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">mvanschellebeeck</span><span class="w"></span>
-<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">xudong</span><span class="p">.</span><span class="n">w</span><span class="w"></span>
-<span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">dependabot</span><span class="o">[</span><span class="n">bot</span><span class="o">]</span><span class="w"></span>
-<span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">yahoNanJing</span><span class="w"></span>
-<span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">Brent</span><span class="w"> </span><span class="n">Gardner</span><span class="w"></span>
-<span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="n">AssHero</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">Jiayu</span><span class="w"> </span><span class="n">Liu</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">Wei</span><span class="o">-</span><span class="n">Ting</span><span class="w"> </span><span class="n">Kuo</span><span class="w"></span>
-<span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">askoa</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">André</span><span class="w"> </span><span class="n">Calado</span><span class="w"> </span><span class="n">Coroado</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Jie</span><span class="w"> </span><span class="n">Han</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Jon</span><span class="w"> </span><span class="n">Mease</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Metehan</span><span class="w"> </span><span class="n">Yıldırım</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Nga</span><span class="w"> </span><span class="k">Tran</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Ruihang</span><span class="w"> </span><span class="n">Xia</span><span class="w"></span>
-<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">baishen</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Berkay</span><span class="w"> </span><span class="n">Şahin</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Dan</span><span class="w"> </span><span class="n">Harris</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Dongyan</span><span class="w"> </span><span class="n">Zhou</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Eduard</span><span class="w"> </span><span class="n">Karacharov</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Kikkon</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Liang</span><span class="o">-</span><span class="n">Chi</span><span class="w"> </span><span class="n">Hsieh</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Marko</span><span class="w"> </span><span class="n">Milenković</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Martin</span><span class="w"> </span><span class="n">Grigorov</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Roman</span><span class="w"> </span><span class="n">Nozdrin</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Tim</span><span class="w"> </span><span class="n">Van</span><span class="w"> </span><span class="n">Wassenhove</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">r</span><span class="mf">.4</span><span class="n">ntix</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">unconsolable</span><span class="w"></span>
-<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">unvalley</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Ajaya</span><span class="w"> </span><span class="n">Agrawal</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Alexander</span><span class="w"> </span><span class="n">Spies</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">ArkashaJavelin</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Artjoms</span><span class="w"> </span><span class="n">Iskovs</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">BoredPerson</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Christian</span><span class="w"> </span><span class="n">Salvati</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Creampanda</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">Data</span><span class="w"> </span><span class="n">Psycho</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Francis</span><span class="w"> </span><span class="n">Du</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Francis</span><span class="w"> </span><span class="n">Le</span><span class="w"> </span><span class="n">Roy</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">LFC</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Marko</span><span class="w"> </span><span class="n">Grujic</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Matt</span><span class="w"> </span><span class="n">Willian</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Matthijs</span><span class="w"> </span><span class="n">Brobbel</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nf">Max</span><span class="w"> </span><span class="n">Burke</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Mehmet</span><span class="w"> </span><span class="n">Ozan</span><span class="w"> </span><span class="n">Kabak</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Rito</span><span class="w"> </span><span class="n">Takeuchi</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Roman</span><span class="w"> </span><span class="n">Zeyde</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Vrishabh</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Zhang</span><span class="w"> </span><span class="n">Li</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">ZuoTiJia</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">byteink</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">cfraz89</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">nbr</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">xxchan</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">yujie</span><span class="p">.</span><span class="n">zhang</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">zembunia</span><span class="w"></span>
-<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">哇呜哇呜呀咦耶</span><span class="w"></span>
-</code></pre></div>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2023/06/24/datafusion-25.0.0/index.html b/output/2023/06/24/datafusion-25.0.0/index.html
deleted file mode 100644
index 790b752..0000000
--- a/output/2023/06/24/datafusion-25.0.0/index.html
+++ /dev/null
@@ -1,255 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 26.0.0 - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow DataFusion 26.0.0
- </h1>
- <p>Posted on: Sat 24 June 2023 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>It has been a whirlwind 6 months of DataFusion development since <a href="https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0">our
-last update</a>: the community has grown, many features have been added,
-performance improved and we are <a href="https://github.com/apache/arrow-datafusion/discussions/6475">discussing</a> branching out to our own
-top level Apache Project.</p>
-<h2>Background</h2>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database
-toolkit, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory
-format.</p>
-<p>DataFusion, along with <a href="https://calcite.apache.org">Apache Calcite</a>, Facebook's <a href="https://github.com/facebookincubator/velox">Velox</a> and
-similar technology are part of the next generation "<a href="https://www.usenix.org/publications/login/winter2018/khurana">Deconstructed
-Database</a>" architectures, where new systems are built on a foundation
-of fast, modular components, rather as a single tightly integrated
-system.</p>
-<p>While single tightly integrated systems such as <a href="https://spark.apache.org/">Spark</a>, <a href="https://duckdb.org">DuckDB</a> and
-<a href="https://www.pola.rs/">Pola.rs</a> are great pieces of technology, our community believes that
-anyone developing new data heavy application, such as those common in
-machine learning in the next 5 years, will <strong>require</strong> a high
-performance, vectorized, query engine to remain relevant. The only
-practical way to gain access to such technology without investing many
-millions of dollars to build a new tightly integrated engine, is
-though open source projects like DataFusion and similar enabling
-technologies such as <a href="https://arrow.apache.org">Apache Arrow</a> and <a href="https://www.rust-lang.org/">Rust</a>.</p>
-<p>DataFusion is targeted primarily at developers creating other data
-intensive analytics, and offers:</p>
-<ul>
-<li>High performance, native, parallel streaming execution engine</li>
-<li>Mature <a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>, featuring subqueries, window functions, grouping sets, and more</li>
-<li>Built in support for Parquet, Avro, CSV, JSON and Arrow formats and easy extension for others</li>
-<li>Native DataFrame API and <a href="https://arrow.apache.org/datafusion-python/">python bindings</a></li>
-<li><a href="https://docs.rs/datafusion/latest/datafusion/index.html">Well documented</a> source code and architecture, designed to be customized to suit downstream project needs</li>
-<li>High quality, easy to use code <a href="https://crates.io/crates/datafusion/versions">released every 2 weeks to crates.io</a></li>
-<li>Welcoming, open community, governed by the highly regarded and well understood <a href="https://www.apache.org/">Apache Software Foundation</a></li>
-</ul>
-<p>The rest of this post highlights some of the improvements we have made
-to DataFusion over the last 6 months and a preview of where we are
-heading. You can see a list of all changes in the detailed
-<a href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/CHANGELOG.md">CHANGELOG</a>.</p>
-<h2>(Even) Better Performance</h2>
-<p><a href="https://voltrondata.com/resources/speeds-and-feeds-hardware-and-software-matter">Various</a> benchmarks show DataFusion to be quite close or <a href="https://github.com/tustvold/access-log-bench">even
-faster</a> to the state of the art in analytic performance (at the moment
-this seems to be DuckDB). We continually work on improving performance
-(see <a href="https://github.com/apache/arrow-datafusion/issues/5546">#5546</a> for a list) and would love additional help in this area.</p>
-<p>DataFusion now reads single large Parquet files significantly faster by
-<a href="https://github.com/apache/arrow-datafusion/pull/5057">parallelizing across multiple cores</a>. Native speeds for reading JSON
-and CSV files are also up to 2.5x faster thanks to improvements
-upstream in arrow-rs <a href="https://github.com/apache/arrow-rs/pull/3479#issuecomment-1384353159">JSON reader</a> and <a href="https://github.com/apache/arrow-rs/pull/3365">CSV reader</a>.</p>
-<p>Also, we have integrated the <a href="https://arrow.apache.org/blog/2022/11/07/multi-column-sorts-in-arrow-rust-part-1/">arrow-rs Row Format</a> into DataFusion resulting in up to <a href="https://github.com/apache/arrow-datafusion/pull/6163">2-3x faster sorting and merging</a>.</p>
-<h2>Improved Documentation and Website</h2>
-<p>Part of growing the DataFusion community is ensuring that DataFusion's
-features are understood and that it is easy to contribute and
-participate. To that end the <a href="https://arrow.apache.org/datafusion/">website</a> has been cleaned up, <a href="https://docs.rs/datafusion/latest/datafusion/index.html#architecture">the
-architecture guide</a> expanded, the <a href="https://arrow.apache.org/datafusion/contributor-guide/roadmap.html">roadmap</a> updated, and several
-overview talks created:</p>
-<ul>
-<li>Apr 2023 <em>Query Engine</em>: <a href="https://youtu.be/NVKujPxwSBA">recording</a> and <a href="https://docs.google.com/presentation/d/1D3GDVas-8y0sA4c8EOgdCvEjVND4s2E7I6zfs67Y4j8/edit#slide=id.p">slides</a></li>
-<li>April 2023 <em>Logical Plan and Expressions</em>: <a href="https://youtu.be/EzZTLiSJnhY">recording</a> and <a href="https://docs.google.com/presentation/d/1ypylM3-w60kVDW7Q6S99AHzvlBgciTdjsAfqNP85K30">slides</a></li>
-<li>April 2023 <em>Physical Plan and Execution</em>: <a href="https://youtu.be/2jkWU3_w6z0">recording</a> and <a href="https://docs.google.com/presentation/d/1cA2WQJ2qg6tx6y4Wf8FH2WVSm9JQ5UgmBWATHdik0hg">slides</a></li>
-</ul>
-<h2>New Features</h2>
-<h3>More Streaming, Less Memory</h3>
-<p>We have made significant progress on the <a href="https://github.com/apache/arrow-datafusion/issues/4285">streaming execution roadmap</a>
-such as <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.unbounded_output">unbounded datasources</a>, <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/aggregates/enum.GroupByOrderMode.html">streaming group by</a>, sophisticated
-<a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/global_sort_selection/index.html">sort</a> and <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/repartition/index.html">repartitioning</a> improvements in the optimizer, and support
-for <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/joins/struct.SymmetricHashJoinExec.html">symmetric hash join</a> (read more about that in the great <a href="https://www.synnada.ai/blog/general-purpose-stream-joins-via-pruning-symmetric-hash-joins">Synnada
-Blog Post</a> on the topic). Together, these features both 1) make it
-easier to build streaming systems using DataFusion that can
-incrementally generate output before (or ever) seeing the end of the
-input and 2) allow general queries to use less memory and generate their
-results faster.</p>
-<p>We have also improved the runtime <a href="https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/index.html">memory management</a> system so that
-DataFusion now stays within its declared memory budget <a href="https://github.com/apache/arrow-datafusion/issues/3941">generate
-runtime errors</a>.</p>
-<h3>DML Support (<code>INSERT</code>, <code>DELETE</code>, <code>UPDATE</code>, etc)</h3>
-<p>Part of building high performance data systems includes writing data,
-and DataFusion supports several features for creating new files:</p>
-<ul>
-<li><code>INSERT INTO</code> and <code>SELECT ... INTO</code> support for memory backed and CSV tables</li>
-<li>New <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/insert/trait.DataSink.html">API for writing data into TableProviders</a></li>
-</ul>
-<p>We are working on easier to use <a href="https://github.com/apache/arrow-datafusion/issues/5654">COPY INTO</a> syntax, better support
-for writing parquet, JSON, and AVRO, and more -- see our <a href="https://github.com/apache/arrow-datafusion/issues/6569">tracking epic</a>
-for more details.</p>
-<h3>Timestamp and Intervals</h3>
-<p>One mark of the maturity of a SQL engine is how it handles the tricky
-world of timestamp, date, times and interval arithmetic. DataFusion is
-feature complete in this area and behaves as you would expect,
-supporting queries such as</p>
-<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'1 month'</span> <span class="k">FROM</span> <span class="n">my_table</span><span class="p">;</span>
-</code></pre></div>
-<p>We still have a long tail of <a href="https://github.com/apache/arrow-datafusion/issues/3148">date and time improvements</a>, which we are working on as well.</p>
-<h3>Querying Structured Types (<code>List</code> and <code>Struct</code>s)</h3>
-<p>Arrow and Parquet <a href="https://arrow.apache.org/blog/2022/10/08/arrow-parquet-encoding-part-2/">support nested data</a> well and DataFusion lets you
-easily query such <code>Struct</code> and <code>List</code>. For example, you can use
-DataFusion to read and query the <a href="https://data.mendeley.com/datasets/ct8f9skv97">JSON Datasets for Exploratory OLAP -
-Mendeley Data</a> like this:</p>
-<div class="codehilite"><pre><span></span><code><span class="c1">----------</span>
-<span class="c1">-- Explore structured data using SQL</span>
-<span class="c1">----------</span>
-<span class="k">SELECT</span> <span class="k">delete</span> <span class="k">FROM</span> <span class="s1">'twitter-sample-head-100000.parquet'</span> <span class="k">WHERE</span> <span class="k">delete</span> <span class="k">IS</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">limit</span> <span class="mi">10</span><span class="p">;</span>
-<span class="o">+</span><span class="c1">---------------------------------------------------------------------------------------------------------------------------+</span>
-<span class="o">|</span> <span class="k">delete</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">---------------------------------------------------------------------------------------------------------------------------+</span>
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">135037425050320896</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">135037425050320896</span><span class="p">,</span> <span class= [...]
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">134703982051463168</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">134703982051463168</span><span class="p">,</span> <span class= [...]
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">134773741740765184</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">134773741740765184</span><span class="p">,</span> <span class= [...]
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">132543659655704576</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">132543659655704576</span><span class="p">,</span> <span class= [...]
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">133786431926697984</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">133786431926697984</span><span class="p">,</span> <span class= [...]
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">134619093570560002</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">134619093570560002</span><span class="p">,</span> <span class= [...]
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">134019857527214080</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">134019857527214080</span><span class="p">,</span> <span class= [...]
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">133931546469076993</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">133931546469076993</span><span class="p">,</span> <span class= [...]
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">134397743350296576</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">134397743350296576</span><span class="p">,</span> <span class= [...]
-<span class="o">|</span> <span class="err">{</span><span class="n">status</span><span class="p">:</span> <span class="err">{</span><span class="n">id</span><span class="p">:</span> <span class="err">{$</span><span class="n">numberLong</span><span class="p">:</span> <span class="mi">127833661767823360</span><span class="err">}</span><span class="p">,</span> <span class="n">id_str</span><span class="p">:</span> <span class="mi">127833661767823360</span><span class="p">,</span> <span class= [...]
-<span class="o">+</span><span class="c1">---------------------------------------------------------------------------------------------------------------------------+</span>
-
-<span class="c1">----------</span>
-<span class="c1">-- Select some deeply nested fields</span>
-<span class="c1">----------</span>
-<span class="k">SELECT</span>
- <span class="k">delete</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'id'</span><span class="p">][</span><span class="s1">'$numberLong'</span><span class="p">]</span> <span class="k">as</span> <span class="n">delete_id</span><span class="p">,</span>
- <span class="k">delete</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'user_id'</span><span class="p">]</span> <span class="k">as</span> <span class="n">delete_user_id</span>
-<span class="k">FROM</span> <span class="s1">'twitter-sample-head-100000.parquet'</span> <span class="k">WHERE</span> <span class="k">delete</span> <span class="k">IS</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">LIMIT</span> <span class="mi">10</span><span class="p">;</span>
-
-<span class="o">+</span><span class="c1">--------------------+----------------+</span>
-<span class="o">|</span> <span class="n">delete_id</span> <span class="o">|</span> <span class="n">delete_user_id</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">--------------------+----------------+</span>
-<span class="o">|</span> <span class="mi">135037425050320896</span> <span class="o">|</span> <span class="mi">334902461</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">134703982051463168</span> <span class="o">|</span> <span class="mi">405383453</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">134773741740765184</span> <span class="o">|</span> <span class="mi">64823441</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">132543659655704576</span> <span class="o">|</span> <span class="mi">45917834</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">133786431926697984</span> <span class="o">|</span> <span class="mi">67229952</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">134619093570560002</span> <span class="o">|</span> <span class="mi">182430773</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">134019857527214080</span> <span class="o">|</span> <span class="mi">257396311</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">133931546469076993</span> <span class="o">|</span> <span class="mi">124539548</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">134397743350296576</span> <span class="o">|</span> <span class="mi">139836391</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">127833661767823360</span> <span class="o">|</span> <span class="mi">244442687</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">--------------------+----------------+</span>
-</code></pre></div>
-<h3>Subqueries All the Way Down</h3>
-<p>DataFusion can run many different subqueries by rewriting them to
-joins. It has been able to run the full suite of TPC-H queries for at
-least the last year, but recently we have implemented significant
-improvements to this logic, sufficient to run almost all queries in
-the TPC-DS benchmark as well.</p>
-<h2>Community and Project Growth</h2>
-<p>The six months since <a href="https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0">our last update</a> saw significant growth in
-the DataFusion community. Between versions <code>17.0.0</code> and <code>26.0.0</code>,
-DataFusion merged 711 PRs from 107 distinct contributors, not
-including all the work that goes into our core dependencies such as
-<a href="https://crates.io/crates/arrow">arrow</a>,
-<a href="https://crates.io/crates/parquet">parquet</a>, and
-<a href="https://crates.io/crates/object_store">object_store</a>, that much of
-the same community helps support.</p>
-<p>In addition, we have added 7 new committers and 1 new PMC member to
-the Apache Arrow project, largely focused on DataFusion, and we
-learned about some of the cool <a href="https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users">new systems</a> which are using
-DataFusion. Given the growth of the community and interest in the
-project, we also clarified the <a href="https://github.com/apache/arrow-datafusion/discussions/6441">mission statement</a> and are
-<a href="https://github.com/apache/arrow-datafusion/discussions/6475">discussing</a> "graduate"ing DataFusion to a new top level
-Apache Software Foundation project.</p>
-<!--
-$ git log --pretty=oneline 17.0.0..26.0.0 . | wc -l
- 711
-
-$ git shortlog -sn 17.0.0..26.0.0 . | wc -l
- 107
--->
-<h1>How to Get Involved</h1>
-<p>Kudos to everyone in the community who has contributed ideas,
-discussions, bug reports, documentation and code. It is exciting to be
-innovating on the next generation of database architectures together!</p>
-<p>If you are interested in contributing to DataFusion, we would love to
-have you join us. You can try out DataFusion on some of your own
-data and projects and let us know how it goes or contribute a PR with
-documentation, tests or code. A list of open issues suitable for
-beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
-<p>Check out our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a> for more ways to engage with the
-community.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2023/08/05/datafusion_fast_grouping/index.html b/output/2023/08/05/datafusion_fast_grouping/index.html
deleted file mode 100644
index 1c39af7..0000000
--- a/output/2023/08/05/datafusion_fast_grouping/index.html
+++ /dev/null
@@ -1,357 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0 - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0
- </h1>
- <p>Posted on: Sat 05 August 2023 by alamb, Dandandan, tustvold</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- Converted from Google Docs using https://www.buymeacoffee.com/docstomarkdown -->
-<h2>Aggregating Millions of Groups Fast in Apache Arrow DataFusion</h2>
-<p>Andrew Lamb, Daniël Heres, Raphael Taylor-Davies,</p>
-<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
-<h2>TLDR</h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability is 2-3x faster in the <a href="https://crates.io/crates/datafusion/28.0.0">newly released version <code>28.0.0</code></a> for queries with a large number (10,000 or more) of groups.</p>
-<p>Improving aggregation performance matters to all users of DataFusion. For example, both InfluxDB, a <a href="https://github.com/influxdata/influxdb">time series data platform</a> and Coralogix, a <a href="https://coralogix.com/?utm_source=InfluxDB&utm_medium=Blog&utm_campaign=organic">full-stack observability</a> platform, aggregate vast amounts of raw data to monitor and create insights for our customers. Improving DataFusion’s performance lets us provide better user ex [...]
-<p>With the new optimizations, DataFusion’s grouping speed is now close to DuckDB, a system that regularly reports <a href="https://duckdblabs.github.io/db-benchmark/">great</a> <a href="https://duckdb.org/2022/03/07/aggregate-hashtable.html#experiments">grouping</a> benchmark performance numbers. Figure 1 contains a representative sample of <a href="https://github.com/ClickHouse/ClickBench/tree/main">ClickBench</a> on a single Parquet file, and the full results are at the end of t [...]
-<p><img src="../images/datafusion_fast_grouping/summary.png" width="700"/></p>
-<p><strong>Figure 1</strong>: Query performance for ClickBench queries on queries 16, 17, 18 and 19 on a single Parquet file for DataFusion <code>27.0.0</code>, DataFusion <code>28.0.0</code> and DuckDB <code>0.8.1</code>.</p>
-<h2>Introduction to high cardinality grouping</h2>
-<p>Aggregation is a fancy word for computing summary statistics across many rows that have the same value in one or more columns. We call the rows with the same values <em>groups</em> and “high cardinality” means there are a large number of distinct groups in the dataset. At the time of writing, a “large” number of groups in analytic engines is around 10,000.</p>
-<p>For example the <a href="https://github.com/ClickHouse/ClickBench">ClickBench</a> <em>hits</em> dataset contains 100 million anonymized user clicks across a set of websites. ClickBench Query 17 is:</p>
-<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="ss">"UserID"</span><span class="p">,</span> <span class="ss">"SearchPhrase"</span><span class="p">,</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span>
-<span class="k">FROM</span> <span class="n">hits</span>
-<span class="k">GROUP</span> <span class="k">BY</span> <span class="ss">"UserID"</span><span class="p">,</span> <span class="ss">"SearchPhrase"</span>
-<span class="k">ORDER</span> <span class="k">BY</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span>
-<span class="k">DESC</span> <span class="k">LIMIT</span> <span class="mi">10</span><span class="p">;</span>
-</code></pre></div>
-<p>In English, this query finds “the top ten (user, search phrase) combinations, across all clicks” and produces the following results (there are no search phrases for the top ten users):</p>
-<div class="codehilite"><pre><span></span><code>+---------------------+--------------+-----------------+
-| UserID | SearchPhrase | COUNT(UInt8(1)) |
-+---------------------+--------------+-----------------+
-| 1313338681122956954 | | 29097 |
-| 1907779576417363396 | | 25333 |
-| 2305303682471783379 | | 10597 |
-| 7982623143712728547 | | 6669 |
-| 7280399273658728997 | | 6408 |
-| 1090981537032625727 | | 6196 |
-| 5730251990344211405 | | 6019 |
-| 6018350421959114808 | | 5990 |
-| 835157184735512989 | | 5209 |
-| 770542365400669095 | | 4906 |
-+---------------------+--------------+-----------------+
-</code></pre></div>
-<p>The ClickBench dataset contains</p>
-<ul>
-<li>99,997,497 total rows[^1]</li>
-<li>17,630,976 different users (distinct UserIDs)[^2]</li>
-<li>6,019,103 different search phrases[^3]</li>
-<li>24,070,560 distinct combinations[^4] of (UserID, SearchPhrase)
-Thus, to answer the query, DataFusion must map each of the 100M different input rows into one of the <strong>24 million different groups</strong>, and keep count of how many such rows there are in each group.</li>
-</ul>
-<h2>The solution</h2>
-<p>Like most concepts in databases and other analytic systems, the basic ideas of this algorithm are straightforward and taught in introductory computer science courses. You could compute the query with a program such as this[^5]:</p>
-<div class="codehilite"><pre><span></span><code><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
-<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
-<span class="kn">from</span> <span class="nn">operator</span> <span class="kn">import</span> <span class="n">itemgetter</span>
-
-<span class="c1"># read file</span>
-<span class="n">hits</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_parquet</span><span class="p">(</span><span class="s1">'hits.parquet'</span><span class="p">,</span> <span class="n">engine</span><span class="o">=</span><span class="s1">'pyarrow'</span><span class="p">)</span>
-
-<span class="c1"># build groups</span>
-<span class="n">counts</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">int</span><span class="p">)</span>
-<span class="k">for</span> <span class="n">index</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">hits</span><span class="o">.</span><span class="n">iterrows</span><span class="p">():</span>
- <span class="n">group</span> <span class="o">=</span> <span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s1">'UserID'</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="s1">'SearchPhrase'</span><span class="p">]);</span>
- <span class="c1"># update the dict entry for the corresponding key</span>
- <span class="n">counts</span><span class="p">[</span><span class="n">group</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
-
-<span class="c1"># Print the top 10 values</span>
-<span class="nb">print</span> <span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">counts</span><span class="o">.</span><span class="n">items</span><span class="p">(),</span> <span class="n">key</span><span class="o">=</span><span class="n">itemgetter</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="n">reverse</span><span class="o">=</span><span class= [...]
-</code></pre></div>
-<p>This approach, while simple, is both slow and very memory inefficient. It requires over 40 seconds to compute the results for less than 1% of the dataset[^6]. Both DataFusion <code>28.0.0</code> and DuckDB <code>0.8.1</code> compute results in under 10 seconds for the <em>entire</em> dataset.</p>
-<p>To answer this query quickly and efficiently, you have to write your code such that it:</p>
-<ol>
-<li>Keeps all cores busy aggregating via parallelized computation</li>
-<li>Updates aggregate values quickly, using vectorizable loops that are easy for compilers to translate into the high performance <a href="https://en.wikipedia.org/wiki/Single_instruction,_multiple_data">SIMD</a> instructions available in modern CPUs.</li>
-</ol>
-<p>The rest of this article explains how grouping works in DataFusion and the improvements we made in <code>28.0.0</code>.</p>
-<h3>Two phase parallel partitioned grouping</h3>
-<p>Both DataFusion <code>27.0.</code> and <code>28.0.0</code> use state-of-the-art, two phase parallel hash partitioned grouping, similar to other high-performance vectorized engines like <a href="https://duckdb.org/2022/03/07/aggregate-hashtable.html">DuckDB’s Parallel Grouped Aggregates</a>. In pictures this looks like:</p>
-<div class="codehilite"><pre><span></span><code> ▲ ▲
- │ │
- │ │
- │ │
-┌───────────────────────┐ ┌───────────────────┐
-│ GroupBy │ │ GroupBy │ Step 4
-│ (Final) │ │ (Final) │
-└───────────────────────┘ └───────────────────┘
- ▲ ▲
- │ │
- └────────────┬───────────┘
- │
- │
- ┌─────────────────────────┐
- │ Repartition │ Step 3
- │ HASH(x) │
- └─────────────────────────┘
- ▲
- │
- ┌────────────┴──────────┐
- │ │
- │ │
- ┌────────────────────┐ ┌─────────────────────┐
- │ GroupyBy │ │ GroupBy │ Step 2
- │ (Partial) │ │ (Partial) │
- └────────────────────┘ └─────────────────────┘
- ▲ ▲
- ┌──┘ └─┐
- │ │
- .─────────. .─────────.
- ,─' '─. ,─' '─.
-; Input : ; Input : Step 1
-: Stream 1 ; : Stream 2 ;
- ╲ ╱ ╲ ╱
- '─. ,─' '─. ,─'
- `───────' `───────'
-</code></pre></div>
-<p><strong>Figure 2</strong>: Two phase repartitioned grouping: data flows from bottom (source) to top (results) in two phases. First (Steps 1 and 2), each core reads the data into a core-specific hash table, computing intermediate aggregates without any cross-core coordination. Then (Steps 3 and 4) DataFusion divides the data (“repartitions”) into distinct subsets by group value, and each subset is sent to a specific core which computes the final aggregate.</p>
-<p>The two phases are critical for keeping cores busy in a multi-core system. Both phases use the same hash table approach (explained in the next section), but differ in how the groups are distributed and the partial results emitted from the accumulators. The first phase aggregates data as soon as possible after it is produced. However, as shown in Figure 2, the groups can be anywhere in any input, so the same group is often found on many different cores. The second phase uses a hash fun [...]
-<div class="codehilite"><pre><span></span><code> <span class="err">┌─────┐</span> <span class="err">┌─────┐</span>
- <span class="err">│</span> <span class="mi">1</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">3</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">2</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">4</span> <span class="err">│</span> <span class="mi">2</span><span class="p">.</span> <span class="k">After</span> <span class="n">Repartitioning</span><span class="p">:</span> <span class="k">each</span>
- <span class="err">└─────┘</span> <span class="err">└─────┘</span> <span class="k">group</span> <span class="k">key</span> <span class="n">appears</span> <span class="k">in</span> <span class="n">exactly</span>
- <span class="err">┌─────┐</span> <span class="err">┌─────┐</span> <span class="n">one</span> <span class="n">partition</span>
- <span class="err">│</span> <span class="mi">1</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">3</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">2</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">4</span> <span class="err">│</span>
- <span class="err">└─────┘</span> <span class="err">└─────┘</span>
-
-<span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class= [...]
-
- <span class="err">┌─────┐</span> <span class="err">┌─────┐</span>
- <span class="err">│</span> <span class="mi">2</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">2</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">1</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">2</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">3</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">3</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">4</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">1</span> <span class="err">│</span>
- <span class="err">└─────┘</span> <span class="err">└─────┘</span> <span class="mi">1</span><span class="p">.</span> <span class="k">Input</span> <span class="n">Stream</span><span class="p">:</span> <span class="n">groups</span>
- <span class="p">...</span> <span class="p">...</span> <span class="k">values</span> <span class="k">are</span> <span class="n">spread</span>
- <span class="err">┌─────┐</span> <span class="err">┌─────┐</span> <span class="n">arbitrarily</span> <span class="n">over</span> <span class="k">each</span> <span class="k">input</span>
- <span class="err">│</span> <span class="mi">1</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">4</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">4</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">3</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">1</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">1</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">4</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">3</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">3</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">2</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">2</span> <span class="err">│</span> <span class="err">│</span> <span class="mi">2</span> <span class="err">│</span>
- <span class="err">│</span> <span class="mi">2</span> <span class="err">│</span> <span class="err">└─────┘</span>
- <span class="err">└─────┘</span>
-
- <span class="n">Core</span> <span class="n">A</span> <span class="n">Core</span> <span class="n">B</span>
-</code></pre></div>
-<p><strong>Figure 3</strong>: Group value distribution across 2 cores during aggregation phases. In the first phase, every group value <code>1</code>, <code>2</code>, <code>3</code>, <code>4</code>, is present in the input stream processed by each core. In the second phase, after repartitioning, the group values <code>1</code> and <code>2</code> are processed by core A, and values <code>3</code> and <code>4</code> are processed only by core B.</p>
-<p>There are some additional subtleties in the <a href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/physical_plan/aggregates/row_hash.rs">DataFusion implementation</a> not mentioned above due to space constraints, such as:</p>
-<ol>
-<li>The policy of when to emit data from the first phase’s hash table (e.g. because the data is partially sorted)</li>
-<li>Handling specific filters per aggregate (due to the <code>FILTER</code> SQL clause)</li>
-<li>Data types of intermediate values (which may not be the same as the final output for some aggregates such as <code>AVG</code>).</li>
-<li>Action taken when memory use exceeds its budget.</li>
-</ol>
-<h3>Hash grouping</h3>
-<p>DataFusion queries can compute many different aggregate functions for each group, both <a href="https://arrow.apache.org/datafusion/user-guide/sql/aggregate_functions.html">built in</a> and/or user defined <a href="https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.AggregateUDF.html"><code>AggregateUDFs</code></a>. The state for each aggregate function, called an <em>accumulator</em>, is tracked with a hash table (DataFusion uses the excellent <a href="https://docs.rs/ha [...]
-<h3>Hash grouping in <code>27.0.0</code></h3>
-<p>As shown in Figure 3, DataFusion <code>27.0.0</code> stores the data in a <a href="https://github.com/apache/arrow-datafusion/blob/4d93b6a3802151865b68967bdc4c7d7ef425b49a/datafusion/core/src/physical_plan/aggregates/utils.rs#L38-L50"><code>GroupState</code></a> structure which, unsurprisingly, tracks the state for each group. The state for each group consists of:</p>
-<ol>
-<li>The actual value of the group columns, in <a href="https://docs.rs/arrow-row/latest/arrow_row/index.html">Arrow Row</a> format.</li>
-<li>In-progress accumulations (e.g. the running counts for the <code>COUNT</code> aggregate) for each group, in one of two possible formats (<a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/expr/src/accumulator.rs#L24-L49"><code>Accumulator</code></a> or <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/aggregate/row_accumulator.rs#L26-L46"><code>RowAc [...]
-<li>Scratch space for tracking which rows match each aggregate in each batch.</li>
-</ol>
-<div class="codehilite"><pre><span></span><code> <span class="err">┌──────────────────────────────────────┐</span>
- <span class="err">│</span> <span class="err">│</span>
- <span class="err">│</span> <span class="p">...</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">┃</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">┌─────────┐</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">┌──────────────────────────────┐</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span><span class="k">group</span> <span class="k">values</span><span class="p">:</span> <span class="n">OwnedRow</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">┌─────┐</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">└──────────────────────────────┘</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="mi">5</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">┌──────────────────────────────┐</span> <span class="err">┃</span> <span class="er [...]
- <span class="err">│</span> <span class="err">├─────┤</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span><span class="k">Row</span> <span class="n">accumulator</span><span class="p">:</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="mi">9</span> <span class="err">│─┼────┐</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span><span class="n">Vec</span><span class="o"><</span><span class="n">u8</span><span class="o">></span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">├─────┤</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">└──────────────────────────────┘</span> <span class="err">┃</span> <span c [...]
- <span class="err">│</span> <span class="err">│</span> <span class="p">...</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">┌──────────────────────┐</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">├─────┤</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│┌──────────────┐</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="mi">1</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">││</span><span class="n">Accumulator</span> <span class="mi">1</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">├─────┤</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│└──────────────┘</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="p">...</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│┌──────────────┐</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">└─────┘</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">││</span><span class="n">Accumulator</span> <span class="mi">2</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│└──────────────┘</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">└─────────┘</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span> <span class="n">Box</span><span class="o"><</span><span class="n">dyn</span> <span class="n">Accumulator</span><span class="o">></span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="n">Hash</span> <span class="k">Table</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">└──────────────────────┘</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">┌─────────────────────────┐</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span><span class="n">scratch</span> <span class="n">indices</span><span class="p">:</span> <span class="n">Vec</span><span class="o"><</span><span class="n">u32</span><span class="o">></span><span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">└─────────────────────────┘</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="n">GroupState</span> <span class="err">┃</span> <span class="err">│</span>
- <span class="err">└─────▶</span> <span class="err">│</span> <span class="err">┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛</span> <span class="err">│</span>
- <span class="err">│</span> <span class="err">│</span>
- <span class="n">Hash</span> <span class="k">table</span> <span class="n">tracks</span> <span class="n">an</span> <span class="err">│</span> <span class="p">...</span> <span class="err">│</span>
- <span class="k">index</span> <span class="k">into</span> <span class="n">group_states</span> <span class="err">│</span> <span class="err">│</span>
- <span class="err">└──────────────────────────────────────┘</span>
- <span class="n">group_states</span><span class="p">:</span> <span class="n">Vec</span><span class="o"><</span><span class="n">GroupState</span><span class="o">></span>
-
- <span class="n">There</span> <span class="k">is</span> <span class="n">one</span> <span class="n">GroupState</span> <span class="n">PER</span> <span class="k">GROUP</span>
-</code></pre></div>
-<p><strong>Figure 4</strong>: Hash group operator structure in DataFusion <code>27.0.0</code>. A hash table maps each group to a GroupState which contains all the per-group states.</p>
-<p>To compute the aggregate, DataFusion performs the following steps for each input batch:</p>
-<ol>
-<li>Calculate hash using <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/hash_utils.rs#L264-L307">efficient vectorized code</a>, specialized for each data type.</li>
-<li>Determine group indexes for each input row using the hash table (creating new entries for newly seen groups).</li>
-<li><a href="https://github.com/apache/arrow-datafusion/blob/4ab8be57dee3bfa72dd105fbd7b8901b873a4878/datafusion/core/src/physical_plan/aggregates/row_hash.rs#L562-L602">Update Accumulators for each group that had input rows,</a> assembling the rows into a contiguous range for vectorized accumulator if there are a sufficient number of them.</li>
-</ol>
-<p>DataFusion also stores the hash values in the table to avoid potentially costly hash recomputation when resizing the hash table.</p>
-<p>This scheme works very well for a relatively small number of distinct groups: all accumulators are efficiently updated with large contiguous batches of rows.</p>
-<p>However, this scheme is not ideal for high cardinality grouping due to:</p>
-<ol>
-<li><strong>Multiple allocations per group</strong> for the group value row format, as well as for the <code>RowAccumulator</code>s and each <code>Accumulator</code>. The <code>Accumulator</code> may have additional allocations within it as well.</li>
-<li><strong>Non-vectorized updates:</strong> Accumulator updates often fall back to a slower non-vectorized form because the number of distinct groups is large (and thus number of values per group is small) in each input batch.</li>
-</ol>
-<h3>Hash grouping in <code>28.0.0</code></h3>
-<p>For <code>28.0.0</code>, we rewrote the core group by implementation following traditional system optimization principles: fewer allocations, type specialization, and aggressive vectorization.</p>
-<p>DataFusion <code>28.0.0</code> uses the same RawTable and still stores group indexes. The major differences, as shown in Figure 4, are:</p>
-<ol>
-<li>Group values are stored either<ol>
-<li>Inline in the <code>RawTable</code> (for single columns of primitive types), where the conversion to Row format costs more than its benefit</li>
-<li>In a separate <a href="https://docs.rs/arrow-row/latest/arrow_row/struct.Row.html">Rows</a> structure with a single contiguous allocation for all groups values, rather than an allocation per group. Accumulators manage the state for all the groups internally, so the code to update intermediate values is a tight type specialized loop. The new <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/aggregate/groups_a [...]
-</ol>
-</li>
-</ol>
-<div class="codehilite"><pre><span></span><code><span class="err">┌───────────────────────────────────┐</span> <span class="err">┌───────────────────────┐</span>
-<span class="err">│</span> <span class="err">┌</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">┐</span> <span class="err">┌─────────────────┐│</span> <span class="err">│</span> <span class="err">┏━━━━━━━━━━━━━━━━━━━┓</span> <s [...]
-<span class="err">│</span> <span class="err">│</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">┌──────────────┐</span> <span class="err">┃</span> <span class="err">│</span>
-<span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┌</span> <span class="err">─</span> <span class="err">─</span> <span class="err">┐┌─────┐</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│┌───────&bo [...]
-<span class="err">│</span> <span class="err">│</span> <span class="n">X</span> <span class="err">│</span> <span class="mi">5</span> <span class="err">│</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">││</span> <span class="n">value1</span> <span class="err">│</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
-<span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">├</span> <span class="err">─</span> <span class="err">─</span> <span class="err">┤├─────┤</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│└───────&bo [...]
-<span class="err">│</span> <span class="err">│</span> <span class="n">Q</span> <span class="err">│</span> <span class="mi">9</span> <span class="err">│──┼┼──┐</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span> <span class="p">...</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span>
-<span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">├</span> <span class="err">─</span> <span class="err">─</span> <span class="err">┤├─────┤</span> <span class="err">││</span> <span class="err">└──┼─╋─▶│</span> <span class="err">│</span> <span class="err">┃</s [...]
-<span class="err">│</span> <span class="err">│</span> <span class="p">...</span> <span class="err">│</span> <span class="p">...</span> <span class="err">│</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│┌───────────┐</span> <span class="err">│</span> <span class="err">┃</span> <span class="err"> [...]
-<span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">├</span> <span class="err">─</span> <span class="err">─</span> <span class="err">┤├─────┤</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">││</span> <span class="n">valueN</span> <span [...]
-<span class="err">│</span> <span class="err">│</span> <span class="n">H</span> <span class="err">│</span> <span class="mi">1</span> <span class="err">│</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│└───────────┘</span> <span class="err">│</span> <span class="err">┃</span> <span class="err" [...]
-<span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">├</span> <span class="err">─</span> <span class="err">─</span> <span class="err">┤├─────┤</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">│</span><span class="k">values</span><span class="p"> [...]
-<span class="err">│</span> <span class="k">Rows</span> <span class="err">│</span> <span class="p">...</span> <span class="err">│</span> <span class="p">...</span> <span class="err">│</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">└──────────────┘</span> <span class="err">┃</span> <span class="er [...]
-<span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="err">└</span> <span class="err">─</span> <span class="err">─</span> <span class="err">┘└─────┘</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="err">┃</span> <span class="err">│</span>
-<span class="err">│</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">─</span> <span class="err">│</span> <span class="err">││</span> <span class="err">│</span> <span class="err">┃</span> <span class="n">GroupsAccumulator</span> <span class="err">┃</span> <span class="err">│</span>
-<span class="err">│</span> <span class="err">└─────────────────┘│</span> <span class="err">│</span> <span class="err">┗━━━━━━━━━━━━━━━━━━━┛</span> <span class="err">│</span>
-<span class="err">│</span> <span class="n">Hash</span> <span class="k">Table</span> <span class="err">│</span> <span class="err">│</span> <span class="err">│</span>
-<span class="err">│</span> <span class="err">│</span> <span class="err">│</span> <span class="p">...</span> <span class="err">│</span>
-<span class="err">└───────────────────────────────────┘</span> <span class="err">└───────────────────────┘</span>
- <span class="n">GroupState</span> <span class="n">Accumulators</span>
-
-
-<span class="n">Hash</span> <span class="k">table</span> <span class="n">value</span> <span class="n">stores</span> <span class="n">group_indexes</span> <span class="n">One</span> <span class="n">GroupsAccumulator</span>
-<span class="k">and</span> <span class="k">group</span> <span class="k">values</span><span class="p">.</span> <span class="n">per</span> <span class="k">aggregate</span><span class="p">.</span> <span class="k">Each</span>
- <span class="n">stores</span> <span class="n">the</span> <span class="k">state</span> <span class="k">for</span>
-<span class="k">Group</span> <span class="k">values</span> <span class="k">are</span> <span class="n">stored</span> <span class="n">either</span> <span class="n">inline</span> <span class="o">*</span><span class="k">ALL</span><span class="o">*</span> <span class="n">groups</span><span class="p">,</span> <span class="n">typically</span>
-<span class="k">in</span> <span class="n">the</span> <span class="n">hash</span> <span class="k">table</span> <span class="k">or</span> <span class="k">in</span> <span class="n">a</span> <span class="n">single</span> <span class="k">using</span> <span class="n">a</span> <span class="n">native</span> <span class="n">Vec</span><span class="o"><</span><span class="n">T</span><span class="o">></span>
-<span class="n">allocation</span> <span class="k">using</span> <span class="n">the</span> <span class="n">arrow</span> <span class="k">Row</span> <span class="n">format</span>
-</code></pre></div>
-<p><strong>Figure 5</strong>: Hash group operator structure in DataFusion <code>28.0.0</code>. Group values are stored either directly in the hash table, or in a single allocation using the arrow Row format. The hash table contains group indexes. A single <code>GroupsAccumulator</code> stores the per-aggregate state for <em>all</em> groups.</p>
-<p>This new structure improves performance significantly for high cardinality groups due to:</p>
-<ol>
-<li><strong>Reduced allocations</strong>: There are no longer any individual allocations per group.</li>
-<li><strong>Contiguous native accumulator states</strong>: Type-specialized accumulators store the values for all groups in a single contiguous allocation using a <a href="https://doc.rust-lang.org/std/vec/struct.Vec.html">Rust Vec<T></a> of some native type.</li>
-<li><strong>Vectorized state update</strong>: The inner aggregate update loops, which are type-specialized and in terms of native <code>Vec</code>s, are well-vectorized by the Rust compiler (thanks <a href="https://llvm.org/">LLVM</a>!).</li>
-</ol>
-<h3>Notes</h3>
-<p>Some vectorized grouping implementations store the accumulator state row-wise directly in the hash table, which often uses modern CPU caches efficiently. Managing accumulator state in columnar fashion may sacrifice some cache locality, however it ensures the size of the hash table remains small, even when there are large numbers of groups and aggregates, making it easier for the compiler to vectorize the accumulator update.</p>
-<p>Depending on the cost of recomputing hash values, DataFusion <code>28.0.0</code> may or may not store the hash values in the table. This optimizes the tradeoff between the cost of computing the hash value (which is expensive for strings, for example) vs. the cost of storing it in the hash table.</p>
-<p>One subtlety that arises from pushing state updates into GroupsAccumulators is that each accumulator must handle similar variations with/without filtering and with/without nulls in the input. DataFusion <code>28.0.0</code> uses a templated <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/aggregate/groups_accumulator/accumulate.rs#L28-L54"><code>NullState</code></a> which encapsulates these common patterns ac [...]
-<p>The code structure is heavily influenced by the fact DataFusion is implemented using <a href="https://www.rust-lang.org/">Rust</a>, a new(ish) systems programming language focused on speed and safety. Rust heavily discourages many of the traditional pointer casting “tricks” used in C/C++ hash grouping implementations. The DataFusion aggregation code is almost entirely <a href="https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html#:~:text=Safe%20Rust%20is%20the%20true [...]
-<h2>ClickBench results</h2>
-<p>The full results of running the <a href="https://github.com/ClickHouse/ClickBench/tree/main">ClickBench</a> queries against the single Parquet file with DataFusion <code>27.0.0</code>, DataFusion <code>28.0.0</code>, and DuckDB <code>0.8.1</code> are below. These numbers were run on a GCP <code>e2-standard-8 machine</code> with 8 cores and 32 GB of RAM, using the scripts <a href="https://github.com/alamb/datafusion-duckdb-benchmark">here</a>.</p>
-<p>As the industry moves towards data systems assembled from components, it is increasingly important that they exchange data using open standards such as <a href="https://arrow.apache.org/">Apache Arrow</a> and <a href="https://parquet.apache.org/">Parquet</a> rather than custom storage and in-memory formats. Thus, this benchmark uses a single input Parquet file representative of many DataFusion users and aligned with the current trend in analytics of avoiding a costly load/transformati [...]
-<p>DataFusion now reaches near-DuckDB-speeds querying Parquet data. While we don’t plan to engage in a benchmarking shootout with a team that literally wrote <a href="https://dl.acm.org/doi/abs/10.1145/3209950.3209955">Fair Benchmarking Considered Difficult</a>, hopefully everyone can agree that DataFusion <code>28.0.0</code> is a significant improvement.</p>
-<p><img src="../images/datafusion_fast_grouping/full.png" width="700"/></p>
-<p><strong>Figure 6</strong>: Performance of DataFusion <code>27.0.0</code>, DataFusion <code>28.0.0</code>, and DuckDB <code>0.8.1</code> on all 43 ClickBench queries against a single <code>hits.parquet</code> file. Lower is better.</p>
-<h3>Notes</h3>
-<p>DataFusion <code>27.0.0</code> was not able to run several queries due to either planner bugs (Q9, Q11, Q12, 14) or running out of memory (Q33). DataFusion <code>28.0.0</code> solves those issues.</p>
-<p>DataFusion is faster than DuckDB for query 21 and 22, likely due to optimized implementations of string pattern matching.</p>
-<h2>Conclusion: performance matters</h2>
-<p>Improving aggregation performance by more than a factor of two allows developers building products and projects with DataFusion to spend more time on value-added domain specific features. We believe building systems with DataFusion is much faster than trying to build something similar from scratch. DataFusion increases productivity because it eliminates the need to rebuild well-understood, but costly to implement, analytic database technology. While we’re pleased with the improv [...]
-<h2>Acknowledgments</h2>
-<p>DataFusion is a <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">community effort</a> and this work was not possible without contributions from many in the community. A special shout out to <a href="https://github.com/sunchao">sunchao</a>, <a href="https://github.com/jyshen">yjshen</a>, <a href="https://github.com/yahoNanJing">yahoNanJing</a>, <a href="https://github.com/mingmwang">mingmwang</a>, <a href="https://github.com/ozankabak">ozankabak</a>, < [...]
-<h2>About DataFusion</h2>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database toolkit, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org/">Apache Arrow</a> as its in-memory format. DataFusion, along with <a href="https://calcite.apache.org/">Apache Calcite</a>, Facebook’s <a href="https://github.com/facebookincubator/velox">Velox</a>, and similar technology are part of the next generation [...]
-<!-- Footnotes themselves at the bottom. -->
-<h2>Notes</h2>
-<p>[^1]: <code>SELECT COUNT(*) FROM 'hits.parquet';</code></p>
-<p>[^2]: <code>SELECT COUNT(DISTINCT "UserID") as num_users FROM 'hits.parquet';</code></p>
-<p>[^3]: <code>SELECT COUNT(DISTINCT "SearchPhrase") as num_phrases FROM 'hits.parquet';</code></p>
-<p>[^4]: <code>SELECT COUNT(*) FROM (SELECT DISTINCT "UserID", "SearchPhrase" FROM 'hits.parquet')</code></p>
-<p>[^5]: Full script at <a href="https://github.com/alamb/datafusion-duckdb-benchmark/blob/main/hash.py">hash.py</a></p>
-<p>[^6]: <a href="https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_%7B%7D.parquet">hits_0.parquet</a>, one of the files from the partitioned ClickBench dataset, which has <code>100,000</code> rows and is 117 MB in size. The entire dataset has <code>100,000,000</code> rows in a single 14 GB Parquet file. The script did not complete on the entire dataset after 40 minutes, and used 212 GB RAM at peak.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/01/19/datafusion-34.0.0/index.html b/output/2024/01/19/datafusion-34.0.0/index.html
deleted file mode 100644
index 644019b..0000000
--- a/output/2024/01/19/datafusion-34.0.0/index.html
+++ /dev/null
@@ -1,304 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024 - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024
- </h1>
- <p>Posted on: Fri 19 January 2024 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>We recently <a href="https://crates.io/crates/datafusion/34.0.0">released DataFusion 34.0.0</a>. This blog highlights some of the major
-improvements since we <a href="https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/.">released DataFusion 26.0.0</a> (spoiler alert there are many)
-and a preview of where the community plans to focus in the next 6 months.</p>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that
-uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
-create new, fast data centric systems such as databases, dataframe libraries,
-machine learning and streaming applications. While <a href="https://arrow.apache.org/datafusion/user-guide/introduction.html#project-goals">DataFusion’s primary design
-goal</a> is to accelerate creating other data centric systems, it has a
-reasonable experience directly out of the box as a <a href="https://arrow.apache.org/datafusion-python/">dataframe library</a> and
-<a href="https://arrow.apache.org/datafusion/user-guide/cli.html">command line SQL tool</a>.</p>
-<p>This may also be our last update on the Apache Arrow Site. Future
-updates will likely be on the DataFusion website as we are working to <a href="https://github.com/apache/arrow-datafusion/discussions/6475">graduate
-to a top level project</a> (Apache Arrow DataFusion → Apache DataFusion!) which
-will help focus governance and project growth. Also exciting, our <a href="https://github.com/apache/arrow-datafusion/discussions/8522">first
-DataFusion in person meetup</a> is planned for March 2024.</p>
-<p>DataFusion is very much a community endeavor. Our core thesis is that as a
-community we can build much more advanced technology than any of us as
-individuals or companies could alone. In the last 6 months between <code>26.0.0</code> and
-<code>34.0.0</code>, community growth has been strong. We accepted and reviewed over a
-thousand PRs from 124 different committers, created over 650 issues and closed 517
-of them.
-You can find a list of all changes in the detailed <a href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/CHANGELOG.md">CHANGELOG</a>.</p>
-<!--
-$ git log --pretty=oneline 26.0.0..34.0.0 . | wc -l
- 1009
-
-$ git shortlog -sn 26.0.0..34.0.0 . | wc -l
- 124
-
-https://crates.io/crates/datafusion/26.0.0
-DataFusion 26 released June 7, 2023
-
-https://crates.io/crates/datafusion/34.0.0
-DataFusion 34 released Dec 17, 2023
-
-Issues created in this time: 214 open, 437 closed
-https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+created%3A2023-06-23..2023-12-17
-
-Issues closes: 517
-https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2023-06-23..2023-12-17+
-
-PRs merged in this time 908
-https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2023-06-23..2023-12-17
--->
-<h1>Improved Performance 🚀</h1>
-<p>Performance is a key feature of DataFusion, DataFusion is
-more than 2x faster on <a href="https://benchmark.clickhouse.com/">ClickBench</a> compared to version <code>25.0.0</code>, as shown below:</p>
-<!--
- Scripts: https://github.com/alamb/datafusion-duckdb-benchmark/tree/datafusion-25-34
- Spreadsheet: https://docs.google.com/spreadsheets/d/1FtI3652WIJMC5LmJbLfT3G06w0JQIxEPG4yfMafexh8/edit#gid=1879366976
- Average runtime on 25.0.0: 7.2s (for the queries that actually ran)
- Average runtime on 34.0.0: 3.6s (for the same queries that ran in 25.0.0)
--->
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-responsive" src="../images/datafusion-34.0.0/compare-new.png" width="100%"/>
-<figcaption>
-<b>Figure 1</b>: Performance improvement between <code>25.0.0</code> and <code>34.0.0</code> on ClickBench.
- Note that DataFusion <code>25.0.0</code>, could not run several queries due to
- unsupported SQL (Q9, Q11, Q12, Q14) or memory requirements (Q33).
- </figcaption>
-</figure>
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-responsive" src="../images/datafusion-34.0.0/compare.png" width="100%"/>
-<figcaption>
-<b>Figure 2</b>: Total query runtime for DataFusion <code>34.0.0</code> and DataFusion <code>25.0.0</code>.
- </figcaption>
-</figure>
-<p>Here are some specific enhancements we have made to improve performance:
-* <a href="https://arrow.apache.org/blog/2023/08/05/datafusion_fast_grouping/">2-3x better aggregation performance with many distinct groups</a>
-* Partially ordered grouping / streaming grouping
-* [Specialized operator for "TopK" <code>ORDER BY LIMIT XXX</code>]
-* [Specialized operator for <code>min(col) GROUP BY .. ORDER by min(col) LIMIT XXX</code>]
-* <a href="https://github.com/apache/arrow-datafusion/pull/8126">Improved join performance</a>
-* Eliminate redundant sorting with sort order aware optimizers</p>
-<h1>New Features ✨</h1>
-<h2>DML / Insert / Creating Files</h2>
-<p>DataFusion now supports writing data in parallel, to individual or multiple
-files, using <code>Parquet</code>, <code>CSV</code>, <code>JSON</code>, <code>ARROW</code> and user defined formats.
-<a href="https://github.com/apache/arrow-datafusion/pull/7655">Benchmark results</a> show improvements up to 5x in some cases.</p>
-<p>Similarly to reading, data can now be written to any [<code>ObjectStore</code>]
-implementation, including AWS S3, Azure Blob Storage, GCP Cloud Storage, local
-files, and user defined implementations. While reading from <a href="https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html#features">hive style
-partitioned tables</a> has long been supported, it is now possible to write to such
-tables as well.</p>
-<p>For example, to write to a local file:</p>
-<div class="codehilite"><pre><span></span><code><span class="err">❯</span> <span class="k">CREATE</span> <span class="k">EXTERNAL</span> <span class="k">TABLE</span> <span class="n">awesome_table</span><span class="p">(</span><span class="n">x</span> <span class="nb">INT</span><span class="p">)</span> <span class="n">STORED</span> <span class="k">AS</span> <span class="n">PARQUET</span> <span class="k">LOCATION</span> <span class="s1">'/tmp/my_awesome_table'</span><span class="p">;</span>
-<span class="mi">0</span> <span class="k">rows</span> <span class="k">in</span> <span class="k">set</span><span class="p">.</span> <span class="n">Query</span> <span class="n">took</span> <span class="mi">0</span><span class="p">.</span><span class="mi">003</span> <span class="n">seconds</span><span class="p">.</span>
-
-<span class="err">❯</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">awesome_table</span> <span class="k">SELECT</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">10</span> <span class="k">FROM</span> <span class="n">my_source_table</span><span class="p">;</span>
-<span class="o">+</span><span class="c1">-------+</span>
-<span class="o">|</span> <span class="k">count</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">-------+</span>
-<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">-------+</span>
-<span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span><span class="p">.</span> <span class="n">Query</span> <span class="n">took</span> <span class="mi">0</span><span class="p">.</span><span class="mi">024</span> <span class="n">seconds</span><span class="p">.</span>
-</code></pre></div>
-<p>You can also write to files with the [<code>COPY</code>], similarly to [DuckDB’s <code>COPY</code>]:</p>
-<div class="codehilite"><pre><span></span><code><span class="err">❯</span> <span class="k">COPY</span> <span class="p">(</span><span class="k">SELECT</span> <span class="n">x</span> <span class="o">+</span> <span class="mi">1</span> <span class="k">FROM</span> <span class="n">my_source_table</span><span class="p">)</span> <span class="k">TO</span> <span class="s1">'/tmp/output.json'</span><span class="p">;</span>
-<span class="o">+</span><span class="c1">-------+</span>
-<span class="o">|</span> <span class="k">count</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">-------+</span>
-<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">-------+</span>
-<span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span><span class="p">.</span> <span class="n">Query</span> <span class="n">took</span> <span class="mi">0</span><span class="p">.</span><span class="mi">014</span> <span class="n">seconds</span><span class="p">.</span>
-</code></pre></div>
-<div class="codehilite"><pre><span></span><code>$ cat /tmp/output.json
-<span class="o">{</span><span class="s2">"x"</span>:1<span class="o">}</span>
-<span class="o">{</span><span class="s2">"x"</span>:2<span class="o">}</span>
-<span class="o">{</span><span class="s2">"x"</span>:3<span class="o">}</span>
-</code></pre></div>
-<h2>Improved <code>STRUCT</code> and <code>ARRAY</code> support</h2>
-<p>DataFusion <code>34.0.0</code> has much improved <code>STRUCT</code> and <code>ARRAY</code>
-support, including a full range of <a href="https://arrow.apache.org/datafusion/user-guide/sql/scalar_functions.html#struct-functions">struct functions</a> and <a href="https://arrow.apache.org/datafusion/user-guide/sql/scalar_functions.html#array-functions">array functions</a>.</p>
-<!--
-❯ create table my_table as values ([1,2,3]), ([2]), ([4,5]);
--->
-<p>For example, you can now use <code>[]</code> syntax and <code>array_length</code> to access and inspect arrays:</p>
-<div class="codehilite"><pre><span></span><code><span class="err">❯</span> <span class="k">SELECT</span> <span class="n">column1</span><span class="p">,</span>
- <span class="n">column1</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">AS</span> <span class="n">first_element</span><span class="p">,</span>
- <span class="n">array_length</span><span class="p">(</span><span class="n">column1</span><span class="p">)</span> <span class="k">AS</span> <span class="n">len</span>
- <span class="k">FROM</span> <span class="n">my_table</span><span class="p">;</span>
-<span class="o">+</span><span class="c1">-----------+---------------+-----+</span>
-<span class="o">|</span> <span class="n">column1</span> <span class="o">|</span> <span class="n">first_element</span> <span class="o">|</span> <span class="n">len</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">-----------+---------------+-----+</span>
-<span class="o">|</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="mi">3</span> <span class="o">|</span>
-<span class="o">|</span> <span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
-<span class="o">|</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">]</span> <span class="o">|</span> <span class="mi">4</span> <span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">-----------+---------------+-----+</span>
-</code></pre></div>
-<div class="codehilite"><pre><span></span><code><span class="err">❯</span> <span class="k">SELECT</span> <span class="n">column1</span><span class="p">,</span> <span class="n">column1</span><span class="p">[</span><span class="s1">'c0'</span><span class="p">]</span> <span class="k">FROM</span> <span class="n">my_table</span><span class="p">;</span>
-<span class="o">+</span><span class="c1">------------------+----------------------+</span>
-<span class="o">|</span> <span class="n">column1</span> <span class="o">|</span> <span class="n">my_table</span><span class="p">.</span><span class="n">column1</span><span class="p">[</span><span class="n">c0</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">------------------+----------------------+</span>
-<span class="o">|</span> <span class="err">{</span><span class="n">c0</span><span class="p">:</span> <span class="n">foo</span><span class="p">,</span> <span class="n">c1</span><span class="p">:</span> <span class="mi">1</span><span class="err">}</span> <span class="o">|</span> <span class="n">foo</span> <span class="o">|</span>
-<span class="o">|</span> <span class="err">{</span><span class="n">c0</span><span class="p">:</span> <span class="n">bar</span><span class="p">,</span> <span class="n">c1</span><span class="p">:</span> <span class="mi">2</span><span class="err">}</span> <span class="o">|</span> <span class="n">bar</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">------------------+----------------------+</span>
-<span class="mi">2</span> <span class="k">rows</span> <span class="k">in</span> <span class="k">set</span><span class="p">.</span> <span class="n">Query</span> <span class="n">took</span> <span class="mi">0</span><span class="p">.</span><span class="mi">002</span> <span class="n">seconds</span><span class="p">.</span>
-</code></pre></div>
-<h2>Other Features</h2>
-<p>Other notable features include:
-* Support aggregating datasets that exceed memory size, with <a href="https://github.com/apache/arrow-datafusion/pull/7400">group by spill to disk</a>
-* All operators now track and limit their memory consumption, including Joins</p>
-<h1>Building Systems is Easier with DataFusion 🛠️</h1>
-<h2>Documentation</h2>
-<p>It is easier than ever to get started using DataFusion with the
-new <a href="https://arrow.apache.org/datafusion/library-user-guide/index.html">Library Users Guide</a> as well as significantly improved the <a href="https://docs.rs/datafusion/latest/datafusion/index.html">API documentation</a>. </p>
-<h2>User Defined Window and Table Functions</h2>
-<p>In addition to DataFusion's <a href="https://arrow.apache.org/datafusion/library-user-guide/adding-udfs.html#adding-a-scalar-udf">User Defined Scalar Functions</a>, and <a href="https://arrow.apache.org/datafusion/library-user-guide/adding-udfs.html#adding-an-aggregate-udf">User Defined Aggregate Functions</a>, DataFusion now supports <a href="https://arrow.apache.org/datafusion/library-user-guide/adding-udfs.html#adding-a-window-udf">User Defined Window Functions</a>
- and <a href="https://arrow.apache.org/datafusion/library-user-guide/adding-udfs.html#adding-a-user-defined-table-function">User Defined Table Functions</a>.</p>
-<p>For example, [the <code>datafusion-cli</code>] implements a DuckDB style [<code>parquet_metadata</code>]
-function as a user defined table function (<a href="https://github.com/apache/arrow-datafusion/blob/3f219bc929cfd418b0e3d3501f8eba1d5a2c87ae/datafusion-cli/src/functions.rs#L222-L248">source code here</a>): </p>
-<div class="codehilite"><pre><span></span><code><span class="err">❯</span> <span class="k">SELECT</span>
- <span class="n">path_in_schema</span><span class="p">,</span> <span class="n">row_group_id</span><span class="p">,</span> <span class="n">row_group_num_rows</span><span class="p">,</span> <span class="n">stats_min</span><span class="p">,</span> <span class="n">stats_max</span><span class="p">,</span> <span class="n">total_compressed_size</span>
-<span class="k">FROM</span>
- <span class="n">parquet_metadata</span><span class="p">(</span><span class="s1">'hits.parquet'</span><span class="p">)</span>
-<span class="k">WHERE</span> <span class="n">path_in_schema</span> <span class="o">=</span> <span class="s1">'"WatchID"'</span>
-<span class="k">LIMIT</span> <span class="mi">3</span><span class="p">;</span>
-
-<span class="o">+</span><span class="c1">----------------+--------------+--------------------+---------------------+---------------------+-----------------------+</span>
-<span class="o">|</span> <span class="n">path_in_schema</span> <span class="o">|</span> <span class="n">row_group_id</span> <span class="o">|</span> <span class="n">row_group_num_rows</span> <span class="o">|</span> <span class="n">stats_min</span> <span class="o">|</span> <span class="n">stats_max</span> <span class="o">|</span> <span class="n">total_compressed_size</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">----------------+--------------+--------------------+---------------------+---------------------+-----------------------+</span>
-<span class="o">|</span> <span class="ss">"WatchID"</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span class="mi">450560</span> <span class="o">|</span> <span class="mi">4611687214012840539</span> <span class="o">|</span> <span class="mi">9223369186199968220</span> <span class="o">|</span> <span class="mi">3883759</span> <span class="o">|</span>
-<span class="o">|</span> <span class="ss">"WatchID"</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="mi">612174</span> <span class="o">|</span> <span class="mi">4611689135232456464</span> <span class="o">|</span> <span class="mi">9223371478009085789</span> <span class="o">|</span> <span class="mi">5176803</span> <span class="o">|</span>
-<span class="o">|</span> <span class="ss">"WatchID"</span> <span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="mi">344064</span> <span class="o">|</span> <span class="mi">4611692774829951781</span> <span class="o">|</span> <span class="mi">9223363791697310021</span> <span class="o">|</span> <span class="mi">3031680</span> <span class="o">|</span>
-<span class="o">+</span><span class="c1">----------------+--------------+--------------------+---------------------+---------------------+-----------------------+</span>
-<span class="mi">3</span> <span class="k">rows</span> <span class="k">in</span> <span class="k">set</span><span class="p">.</span> <span class="n">Query</span> <span class="n">took</span> <span class="mi">0</span><span class="p">.</span><span class="mi">053</span> <span class="n">seconds</span><span class="p">.</span>
-</code></pre></div>
-<h3>Growth of DataFusion 📈</h3>
-<p>DataFusion has been appearing more publically in the wild. For example
-* New projects built using DataFusion such as <a href="https://lancedb.com/">lancedb</a>, <a href="https://glaredb.com/">GlareDB</a>, <a href="https://www.arroyo.dev/">Arroyo</a>, and <a href="https://github.com/cmu-db/optd">optd</a>.
-* Public talks such as <a href="https://www.youtube.com/watch?v=AJU9rdRNk9I">Apache Arrow Datafusion: Vectorized
- Execution Framework For Maximum Performance</a> in <a href="https://www.bagevent.com/event/8432178">CommunityOverCode Asia 2023</a>
-* Blogs posts such as <a href="https://www.synnada.ai/blog/apache-arrow-arrow-datafusion-ai-native-data-infra-an-interview-with-our-ceo-ozan">Apache Arrow, Arrow/DataFusion, AI-native Data Infra</a>,
- <a href="https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/">Flight, DataFusion, Arrow, and Parquet: Using the FDAP Architecture to build InfluxDB 3.0</a>, and
- <a href="https://www.linkedin.com/pulse/guide-user-defined-functions-apache-arrow-datafusion-dade-aderemi/">A Guide to User-Defined Functions in Apache Arrow DataFusion</a></p>
-<p>We have also <a href="https://github.com/apache/arrow-datafusion/issues/6782">submitted a paper</a> to <a href="https://2024.sigmod.org/">SIGMOD 2024</a>, one of the
-premiere database conferences, describing DataFusion in a technically formal
-style and making the case that it is possible to create a modular and extensive query engine
-without sacrificing performance. We hope this paper helps people
-evaluating DataFusion for their needs understand it better.</p>
-<h1>DataFusion in 2024 🥳</h1>
-<p>Some major initiatives from contributors we know of this year are:</p>
-<ol>
-<li>
-<p><em>Modularity</em>: Make DataFusion even more modular, such as <a href="https://github.com/apache/arrow-datafusion/issues/8045">unifying
- built in and user functions</a>, making it easier to customize
- DataFusion's behavior.</p>
-</li>
-<li>
-<p><em>Community Growth</em>: Graduate to our own top level Apache project, and
- subsequently add more committers and PMC members to keep pace with project
- growth.</p>
-</li>
-<li>
-<p><em>Use case white papers</em>: Write blog posts and videos explaining
- how to use DataFusion for real-world use cases.</p>
-</li>
-<li>
-<p><em>Testing</em>: Improve CI infrastructure and test coverage, more fuzz
- testing, and better functional and performance regression testing.</p>
-</li>
-<li>
-<p><em>Planning Time</em>: Reduce the time taken to plan queries, both <a href="https://github.com/apache/arrow-datafusion/issues/7698">wide
- tables of 1000s of columns</a>, and in <a href="https://github.com/apache/arrow-datafusion/issues/5637">general</a>.</p>
-</li>
-<li>
-<p><em>Aggregate Performance</em>: Improve the speed of <a href="https://github.com/apache/arrow-datafusion/issues/7000">aggregating "high cardinality"</a> data
- when there are many (e.g. millions) of distinct groups.</p>
-</li>
-<li>
-<p><em>Statistics</em>: <a href="https://github.com/apache/arrow-datafusion/issues/8227">Improved statistics handling</a> with an eye towards more
- sophisticated expression analysis and cost models.</p>
-</li>
-</ol>
-<h1>How to Get Involved</h1>
-<p>If you are interested in contributing to DataFusion we would love to have you
-join us. You can try out DataFusion on some of your own data and projects and
-let us know how it goes, contribute suggestions, documentation, bug reports, or
-a PR with documentation, tests or code. A list of open issues
-suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
-<p>As the community grows, we are also looking to restart biweekly calls /
-meetings. Timezones are always a challenge for such meetings, but we hope to
-have two calls that can work for most attendees. If you are interested
-in helping, or just want to say hi, please drop us a note via one of
-the methods listed in our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a>.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/03/06/comet-donation/index.html b/output/2024/03/06/comet-donation/index.html
deleted file mode 100644
index d9c8bb4..0000000
--- a/output/2024/03/06/comet-donation/index.html
+++ /dev/null
@@ -1,120 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Announcing Apache Arrow DataFusion Comet - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Announcing Apache Arrow DataFusion Comet
- </h1>
- <p>Posted on: Wed 06 March 2024 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p>The Apache Arrow PMC is pleased to announce the donation of the <a href="https://github.com/apache/arrow-datafusion-comet">Comet project</a>,
-a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/datafusion">Apache Arrow DataFusion</a>.</p>
-<p>Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to
-accelerate Spark workloads. It is designed as a drop-in
-replacement for Spark's JVM based SQL execution engine and offers significant
-performance improvements for some workloads as shown below.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-responsive" src="../images/datafusion-comet/comet-architecture.png" width="100%"/>
-<figcaption>
-<b>Figure 1</b>: With Comet, users interact with the same Spark ecosystem, tools
- and APIs such as Spark SQL. Queries still run through Spark's query optimizer and planner.
- However, the execution is delegated to Comet,
- which is significantly faster and more resource efficient than a JVM based
- implementation.
-</figcaption>
-</figure>
-<p>Comet is one of a growing class of projects that aim to accelerate Spark using
-native columnar engines such as the proprietary <a href="https://www.databricks.com/product/photon">Databricks Photon Engine</a> and
-open source projects <a href="https://incubator.apache.org/projects/gluten.html">Gluten</a>, <a href="https://github.com/NVIDIA/spark-rapids">Spark RAPIDS</a>, and <a href="https://github.com/kwai/blaze">Blaze</a> (also built using
-DataFusion).</p>
-<p>Comet was originally implemented at Apple and the engineers who worked on the
-project are also significant contributors to Arrow and DataFusion. Bringing
-Comet into the Apache Software Foundation will accelerate its development and
-grow its community of contributors and users.</p>
-<h1>Get Involved</h1>
-<p>Comet is still in the early stages of development and we would love to have you
-join us and help shape the project. We are working on an initial release, and
-expect to post another update with more details at that time.</p>
-<p>Before then, here are some ways to get involved:</p>
-<ul>
-<li>
-<p>Learn more by visiting the <a href="https://github.com/apache/arrow-datafusion-comet">Comet project</a> page and reading the <a href="https://lists.apache.org/thread/0q1rb11jtpopc7vt1ffdzro0omblsh0s">mailing list
- discussion</a> about the initial donation.</p>
-</li>
-<li>
-<p>Help us plan out the <a href="https://github.com/apache/arrow-datafusion-comet/issues/19">roadmap</a></p>
-</li>
-<li>
-<p>Try out the project and provide feedback, file issues, and contribute code.</p>
-</li>
-</ul>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/05/07/datafusion-tlp/index.html b/output/2024/05/07/datafusion-tlp/index.html
deleted file mode 100644
index 5946a13..0000000
--- a/output/2024/05/07/datafusion-tlp/index.html
+++ /dev/null
@@ -1,114 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Announcing Apache Arrow DataFusion is now Apache DataFusion - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Announcing Apache Arrow DataFusion is now Apache DataFusion
- </h1>
- <p>Posted on: Tue 07 May 2024 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>TLDR; <a href="https://arrow.apache.org/">Apache Arrow</a> DataFusion --> <a href="https://datafusion.apache.org/">Apache DataFusion</a></p>
-<p>The Arrow PMC and newly created DataFusion PMC are happy to announce that as of
-April 16, 2024 the Apache Arrow DataFusion subproject is now a top level
-<a href="https://www.apache.org/">Apache Software Foundation</a> project.</p>
-<h2>Background</h2>
-<p>Apache DataFusion is a fast, extensible query engine for building high-quality
-data-centric systems in Rust, using the Apache Arrow in-memory format.</p>
-<p>When DataFusion was <a href="https://arrow.apache.org/blog/2019/02/04/datafusion-donation/">donated to the Apache Software Foundation</a> in 2019, the
-DataFusion community was not large enough to stand on its own and the Arrow
-project agreed to help support it. The community has grown significantly since
-2019, benefiting immensely from being part of Arrow and following <a href="https://www.apache.org/theapacheway/">The Apache
-Way</a>.</p>
-<h2>Why now?</h2>
-<p>The community <a href="https://github.com/apache/datafusion/discussions/6475">discussed graduating to a top level project publicly</a> for almost
-a year, as the project seemed ready to stand on its own and would benefit from
-more focused governance. For example, earlier in DataFusion's life many
-contributed to both <a href="https://github.com/apache/arrow-rs">arrow-rs</a> and DataFusion, but as DataFusion has matured many
-contributors, committers and PMC members focused more and more exclusively on
-DataFusion.</p>
-<h2>Looking forward</h2>
-<p>The future looks bright. There are now <a href="https://datafusion.apache.org/user-guide/introduction.html#known-users">10s of known projects built with
-DataFusion</a>, and that number continues to grow. We recently held our <a href="https://github.com/apache/datafusion/discussions/8522">first in
-person meetup</a> passed <a href="https://github.com/apache/datafusion/stargazers">5000 stars</a> on GitHub, <a href="https://github.com/apache/datafusion/issues/8373#issuecomment-2025133714">wrote a paper that was accepted
-at SIGMOD 2024</a>, and began work on <a href="https://github.com/apache/datafusion-comet">Comet</a>, an <a href="https://spark.apache.org/">Apache Spark</a> accelerator
-<a href="https://arrow.apache.org/blog/2024/03/06/comet-donation/">initially donated by Apple</a>.</p>
-<p>Thank you to everyone in the Arrow community who helped DataFusion grow and
-mature over the years, and we look forward to continuing our collaboration as
-projects. All future blogs and announcements will be posted on the <a href="https://datafusion.apache.org/">Apache
-DataFusion</a> website.</p>
-<h2>Get Involved</h2>
-<p>If you are interested in joining the community, we would love to have you join
-us. Get in touch using <a href="https://datafusion.apache.org/contributor-guide/communication.html">Communication Doc</a> and learn how to get involved in the
-<a href="https://datafusion.apache.org/contributor-guide/index.html">Contributor Guide</a>. We welcome everyone to try DataFusion on their
-own data and projects and let us know how it goes, contribute suggestions,
-documentation, bug reports, or a PR with documentation, tests or code.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/07/20/datafusion-comet-0.1.0/index.html b/output/2024/07/20/datafusion-comet-0.1.0/index.html
deleted file mode 100644
index 09d7134..0000000
--- a/output/2024/07/20/datafusion-comet-0.1.0/index.html
+++ /dev/null
@@ -1,154 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.1.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache DataFusion Comet 0.1.0 Release
- </h1>
- <p>Posted on: Sat 20 July 2024 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce the first official source release of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers five months of development work since the project was <a href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/">donated</a> to the Apache DataFusion
-project and is the result of merging 343 PRs from 41 contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.1.0.md">change log</a> for more information.</p>
-<p>This first release supports 15 <a href="https://datafusion.apache.org/comet/user-guide/datatypes.html#">data types</a>, 13 <a href="https://datafusion.apache.org/comet/user-guide/operators.html#">operators</a>, and 106 <a href="https://datafusion.apache.org/comet/user-guide/expressions.html#">expressions</a>. Comet is compatible with Apache
-Spark versions 3.3, 3.4, and 3.5. There is also experimental support for preview versions of Spark 4.0.</p>
-<h2>Project Status</h2>
-<p>The project's recent focus has been on fixing correctness and stability issues and implementing additional
-native operators and expressions so that a broader range of queries can be executed natively.</p>
-<p>Here are some of the highlights since the project was donated:</p>
-<ul>
-<li>Implemented native support for:</li>
-<li>SortMergeJoin</li>
-<li>HashJoin</li>
-<li>BroadcastHashJoin</li>
-<li>Columnar Shuffle</li>
-<li>More aggregate expressions</li>
-<li>Window aggregates</li>
-<li>Many Spark-compatible CAST expressions</li>
-<li>Implemented a simple Spark Fuzz Testing utility to find correctness issues</li>
-<li>Published a <a href="https://datafusion.apache.org/comet/user-guide/overview.html">User Guide</a> and <a href="https://datafusion.apache.org/comet/contributor-guide/contributing.html">Contributors Guide</a></li>
-<li>Created a <a href="https://github.com/apache/datafusion-benchmarks">DataFusion Benchmarks</a> repository with scripts and documentation for running benchmarks derived<br/>
- from TPC-H and TPC-DS with DataFusion and Comet</li>
-</ul>
-<h2>Current Performance</h2>
-<p>Comet already delivers a modest performance speedup for many queries, enabling faster data processing and
-shorter time-to-insights.</p>
-<p>We use benchmarks derived from the industry standard TPC-H and TPC-DS benchmarks for tracking progress with
-performance. The following chart shows the time it takes to run the 22 TPC-H queries against 100 GB of data in
-Parquet format using a single executor with eight cores. See the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>
-for details of the environment used for these benchmarks.</p>
-<p><img alt="Chart showing TPC-H benchmark results for Comet 0.1.0" class="img-responsive" src="../images/comet-0.1.0/tpch_allqueries.png" width="100%"/></p>
-<p>Comet reduces the overall execution time from 626 seconds to 407 seconds, a 54% speedup (1.54x faster).</p>
-<p>Running the same queries with DataFusion standalone using the same number of cores results in a 3.9x speedup
-compared to Spark. Although this isn’t a fair comparison (DataFusion does not have shuffle or match Spark
-semantics in some cases, for example), it does give some idea about the potential future performance of
-Comet. Comet aims to provide a 2x-4x speedup for a wide range of queries once more operators and expressions
-can run natively.</p>
-<p>The following chart shows how much Comet currently accelerates each query from the benchmark.</p>
-<p><img alt="Chart showing TPC-H benchmark results for Comet 0.1.0" class="img-responsive" src="../images/comet-0.1.0/tpch_queries_speedup.png" width="100%"/></p>
-<p>These benchmarks can be reproduced in any environment using the documentation in the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>. We
-encourage you to run these benchmarks in your environment or, even better, try Comet out with your existing Spark jobs.</p>
-<h2>Roadmap</h2>
-<p>Comet is an open-source project, and contributors are welcome to work on any features they are interested in, but
-here are some current focus areas.</p>
-<ul>
-<li>Improve Performance & Reliability:</li>
-<li>Implement the remaining features needed so that all TPC-H queries can run entirely natively</li>
-<li>Implement spill support in SortMergeJoin</li>
-<li>Enable columnar shuffle by default</li>
-<li>Fully support Spark version 4.0.0</li>
-<li>Support more Spark operators and expressions</li>
-<li>We would like to support many more expressions natively in Comet, and this is a great place to start
- contributing. The contributors' guide has a section covering <a href="https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html">adding support for new expressions</a>.</li>
-<li>Move more Spark expressions into the <a href="https://crates.io/crates/datafusion-comet-spark-expr">datafusion-comet-spark-expr</a> crate. Although the main focus of the Comet
- project is to provide an accelerator for Apache Spark, we also publish a standalone crate containing
- Spark-compatible expressions that can be used by any project using DataFusion, without adding any dependencies
- on JVM or Apache Spark.</li>
-<li>Release Process & Documentation</li>
-<li>Implement a binary release process so that we can publish JAR files to Maven for all supported platforms</li>
-<li>Add documentation for running Spark and Comet in Kubernetes, and add example Dockerfiles.</li>
-</ul>
-<h2>Getting Involved</h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project, and there is a Comet community video call held every four weeks on Wednesdays at 11:30 a.m. Eastern Time,
-which is 16:30 UTC during Eastern Standard Time and 15:30 UTC during Eastern Daylight Time. See the
-<a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">Comet Community Meeting</a> Google Document for the next scheduled meeting date, the video call link, and
-recordings of previous calls.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/07/24/datafusion-40.0.0/index.html b/output/2024/07/24/datafusion-40.0.0/index.html
deleted file mode 100644
index e2102fc..0000000
--- a/output/2024/07/24/datafusion-40.0.0/index.html
+++ /dev/null
@@ -1,363 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion 40.0.0 Released - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache DataFusion 40.0.0 Released
- </h1>
- <p>Posted on: Wed 24 July 2024 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- see https://github.com/apache/datafusion/issues/9602 for details -->
-<h2>Introduction</h2>
-<p>We are proud to announce <a href="https://crates.io/crates/datafusion/40.0.0">DataFusion 40.0.0</a>. This blog highlights some of the
-many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0/">DataFusion 34.0.0</a> and a preview of
-what the community is thinking about in the next 6 months. We are hoping to make
-more regular blog posts -- if you are interested in helping write them, please
-reach out!</p>
-<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that
-uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
-create new, fast data centric systems such as databases, dataframe libraries,
-machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion’s primary design
-goal</a> is to accelerate the creation of other data centric systems, it has a
-reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/python/">dataframe library</a> and
-<a href="https://datafusion.apache.org/user-guide/cli/">command line SQL tool</a>.</p>
-<p>DataFusion's core thesis is that as a community, together we can build much more
-advanced technology than any of us as individuals or companies could do alone.
-Without DataFusion, highly performant vectorized query engines would remain
-the domain of a few large companies and world-class research institutions.
-With DataFusion, we can all build on top of a shared foundation, and focus on
-what makes our projects unique.</p>
-<h2>Community Growth 📈</h2>
-<p>In the last 6 months, between <code>34.0.0</code> and <code>40.0.0</code>, our community continues to
-grow in new and exciting ways.</p>
-<ol>
-<li>DataFusion became a top level Apache Software Foundation project (read the
- <a href="https://news.apache.org/foundation/entry/apache-software-foundation-announces-new-top-level-project-apache-datafusion">press release</a> and <a href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp/">blog post</a>).</li>
-<li>We added several PMC members and new
- committers: <a href="https://github.com/comphead">@comphead</a>, <a href="https://github.com/mustafasrepo">@mustafasrepo</a>, <a href="https://github.com/ozankabak">@ozankabak</a>, and <a href="https://github.com/waynexia">@waynexia</a> joined the PMC,
- <a href="https://github.com/jonahgao">@jonahgao</a> and <a href="https://github.com/lewiszlw">@lewiszlw</a> joined as committers. See the <a href="https://lists.apache.org/list.html?dev@datafusion.apache.org">mailing list</a> for
- more details.</li>
-<li><a href="https://datafusion.apache.org/comet/">DataFusion Comet</a> was <a href="https://arrow.apache.org/blog/2024/03/06/comet-donation/">donated</a> and is nearing its first release.</li>
-<li>In the <a href="https://github.com/apache/arrow-datafusion">core DataFusion repo</a> alone we reviewed and accepted almost 1500 PRs from 182 different
- committers, created over 1000 issues and closed 781 of them 🚀. This is up
- almost 50% from our last post (1000 PRs from 124 committers with 650 issues
- created in our last post) 🤯. All changes are listed in the detailed
- <a href="https://github.com/apache/datafusion/blob/main/datafusion/CHANGELOG.md">CHANGELOG</a>.</li>
-<li>DataFusion focused meetups happened or are happening in multiple cities
- around the world: <a href="https://github.com/apache/datafusion/discussions/8522">Austin</a>, <a href="https://github.com/apache/datafusion/discussions/10800">San Francisco</a>, <a href="https://www.huodongxing.com/event/5761971909400?td=1965290734055">Hangzhou</a>, <a href="https://github.com/apache/datafusion/discussions/11213">New York</a>, and
- <a href="https://github.com/apache/datafusion/discussions/11431">Belgrade</a>.</li>
-<li>Many new projects started in the <a href="https://github.com/datafusion-contrib">datafusion-contrib</a> organization, including
- <a href="https://github.com/datafusion-contrib/datafusion-table-providers">Table Providers</a>, <a href="https://github.com/datafusion-contrib/datafusion-sqlancer">SQLancer</a>, <a href="https://github.com/datafusion-contrib/datafusion-functions-variant">Open Variant</a>, <a href="https://github.com/datafusion-contrib/datafusion-functions-json">JSON</a>, and <a href="https://github.com/datafusion-contrib/datafusion-orc">ORC</a>. </li>
-</ol>
-<!--
-$ git log --pretty=oneline 34.0.0..40.0.0 . | wc -l
- 1453 (up from 1009)
-
-$ git shortlog -sn 34.0.0..40.0.0 . | wc -l
- 182 (up from 124)
-
-https://crates.io/crates/datafusion/34.0.0
-DataFusion 34 released Dec 17, 2023
-
-https://crates.io/crates/datafusion/40.0.0
-DataFusion 34 released July 12, 2024
-
-Issues created in this time: 321 open, 781 closed (up from 214 open, 437 closed)
-https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+created%3A2023-12-17..2024-07-12
-
-Issues closed: 911 (up from 517)
-https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2023-12-17..2024-07-12
-
-PRs merged in this time 1490 (up from 908)
-https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2023-12-17..2024-07-12
-
--->
-<p>In addition, DataFusion has been appearing publicly more and more, both online and offline. Here are some highlights:</p>
-<ol>
-<li><a href="https://dl.acm.org/doi/10.1145/3626246.3653368">Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine</a>, was presented in <a href="https://2024.sigmod.org/">SIGMOD '24</a>, one of the major database conferences</li>
-<li>As part of the trend to define "the POSIX of databases" in <a href="https://db.cs.cmu.edu/papers/2024/whatgoesaround-sigmodrec2024.pdf">"What Goes Around Comes Around... And Around..."</a> from Andy Pavlo and Mike Stonebraker</li>
-<li><a href="https://www.cpard.xyz/posts/datafusion/">"Why you should keep an eye on Apache DataFusion and its community"</a></li>
-<li><a href="https://www.tisonkun.org/2024/07/15/datafusion-meetup-san-francisco/">Apache DataFusion offline meetup in the Bay Area</a></li>
-</ol>
-<h2>Improved Performance 🚀</h2>
-<p>Performance is a key feature of DataFusion, and the community continues to work
-to keep DataFusion state of the art in this area. One major area DataFusion
-improved is the time it takes to convert a SQL query into a plan that can be
-executed. Planning is now almost 2x faster for TPC-DS and TPC-H queries, and
-over 10x faster for some queries with many columns.</p>
-<p>Here is a chart showing the improvement due to the concerted effort of many
-contributors including <a href="https://github.com/jackwener">@jackwener</a>, <a href="https://github.com/alamb">@alamb</a>, <a href="https://github.com/Lordworms">@Lordworms</a>, <a href="https://github.com/dmitrybugakov">@dmitrybugakov</a>,
-<a href="https://github.com/appletreeisyellow">@appletreeisyellow</a>, <a href="https://github.com/ClSlaid">@ClSlaid</a>, <a href="https://github.com/rohitrastogi">@rohitrastogi</a>, <a href="https://github.com/emgeee">@emgeee</a>, <a href="https://github.com/kevinmingtarja">@kevinmingtarja</a>,
-and <a href="https://github.com/peter-toth">@peter-toth</a> over several months (see <a href="https://github.com/apache/arrow-datafusion/issues/8045">ticket</a> for more details)</p>
-<p><img src="../images/datafusion-40.0.0/improved-planning-time.png" width="700"/></p>
-<p>DataFusion is now up to 40% faster for queries that <code>GROUP BY</code> a single string
-or binary column due to a <a href="https://github.com/apache/datafusion/pull/8827">specialization for single
-Uft8/LargeUtf8/Binary/LargeBinary</a>. We are working on improving performance when
-there are [multiple variable length columns in the <code>GROUP BY</code> clause].</p>
-<p>We are also in the final phases of <a href="https://github.com/apache/datafusion/issues/10918">integrating</a> the new <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html">Arrow StringView</a>
-which significantly improves performance for workloads that scan, filter and
-group by variable length string and binary data. We expect the improvement to be
-especially pronounced for Parquet files due to <a href="https://github.com/apache/arrow-rs/issues/5530">upstream work in the parquet
-reader</a>. Kudos to <a href="https://github.com/XiangpengHong">@XiangpengHong</a>, <a href="https://github.com/AriesDevil">@AriesDevil</a>, <a href="https://github.com/PsiACE">@PsiACE</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>,
-<a href="https://github.com/a10y">@a10y</a>, and <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a> for driving this project.</p>
-<h2>Improved Quality 📋</h2>
-<p>DataFusion continues to improve overall in quality. In addition to ongoing bug
-fixes, one of the most exciting improvements is the addition of a new <a href="https://github.com/datafusion-contrib/datafusion-sqlancer">SQLancer</a>
-based <a href="https://github.com/apache/datafusion/issues/11030">DataFusion Fuzzing</a> suite thanks to <a href="https://github.com/2010YOUY01">@2010YOUY01</a> that has already found
-several bugs and thanks to <a href="https://github.com/jonahgao">@jonahgao</a>, <a href="https://github.com/tshauck">@tshauck</a>, <a href="https://github.com/xinlifoobar">@xinlifoobar</a>,
-<a href="https://github.com/LorrensP-2158466">@LorrensP-2158466</a> for fixing them so fast.</p>
-<h2>Improved Documentation 📚</h2>
-<p>We continue to improve the documentation to make it easier to get started using DataFusion with
-the <a href="https://datafusion.apache.org/library-user-guide/index.html">Library Users Guide</a>, <a href="https://docs.rs/datafusion/latest/datafusion/index.html">API documentation</a>, and <a href="https://github.com/apache/datafusion/tree/main/datafusion-examples">Examples</a>.</p>
-<p>Some notable new examples include:
-* <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_analysis.rs">sql_analysis.rs</a> to analyse SQL queries with DataFusion structures (thanks <a href="https://github.com/LorrensP-2158466">@LorrensP-2158466</a>)
-* <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/function_factory.rs">function_factory.rs</a> to create custom functions via SQL (thanks <a href="https://github.com/milenkovicm">@milenkovicm</a>)
-* <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/plan_to_sql.rs">plan_to_sql.rs</a> to generate SQL from DataFusion Expr and LogicalPlan (thanks <a href="https://github.com/edmondop">@edmondop</a>)
-* <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/parquet_index.rs">parquet_index.rs</a> and <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_parquet_index.rs">advanced_parquet_index.rs</a> for parquet indexing, described more below (thanks <a href="https://github.com/alamb">@alamb</a>)</p>
-<h2>New Features ✨</h2>
-<p>There are too many new features in the last 6 months to list them all, but here
-are some highlights:</p>
-<h1>SQL</h1>
-<ul>
-<li>Support for <code>UNNEST</code> (thanks <a href="https://github.com/duongcongtoai">@duongcongtoai</a>, <a href="https://github.com/JasonLi-cn">@JasonLi-cn</a> and <a href="https://github.com/jayzhan211">@jayzhan211</a>) </li>
-<li>Support for <a href="https://github.com/apache/datafusion/issues/462">Recursive CTEs</a> (thanks <a href="https://github.com/jonahgao">@jonahgao</a> and <a href="https://github.com/matthewgapp">@matthewgapp</a>) </li>
-<li>Support for <code>CREATE FUNCTION</code> (see below) </li>
-<li>Many new SQL functions</li>
-</ul>
-<p>DataFusion now has much improved support for structured types such <code>STRUCT</code>,
-<code>LIST</code>/<code>ARRAY</code> and <code>MAP</code>. For example, you can now create <code>STRUCT</code> literals
-in SQL like this:</p>
-<div class="codehilite"><pre><span></span><code><span class="o">></span><span class="w"> </span><span class="n">select</span><span class="w"> </span><span class="p">{</span><span class="na">'foo</span><span class="err">'</span>: <span class="p">{</span><span class="na">'bar</span><span class="err">'</span>: <span class="mi">2</span><span class="p">}};</span><span class="w"></span>
-<span class="o">+--------------------------------------------------------------+</span><span class="w"></span>
-<span class="o">|</span><span class="w"> </span><span class="n">named_struct</span><span class="p">(</span><span class="n">Utf8</span><span class="p">(</span><span class="s">"foo"</span><span class="p">),</span><span class="n">named_struct</span><span class="p">(</span><span class="n">Utf8</span><span class="p">(</span><span class="s">"bar"</span><span class="p">),</span><span class="n">Int64</span><span class="p">(</span><span class="mi">2</span><span class="p">)))</span><span class="w" [...]
-<span class="o">+--------------------------------------------------------------+</span><span class="w"></span>
-<span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="n">foo</span>: <span class="p">{</span><span class="n">bar</span>: <span class="mi">2</span><span class="p">}}</span><span class="w"> </span><span class="o">|</span><span class="w"></span>
-<span class="o">+--------------------------------------------------------------+</span><span class="w"></span>
-<span class="mi">1</span><span class="w"> </span><span class="n">row</span><span class="p">(</span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="n">fetched</span><span class="p">.</span><span class="w"></span>
-<span class="n">Elapsed</span><span class="w"> </span><span class="mf">0.002</span><span class="w"> </span><span class="n">seconds</span><span class="p">.</span><span class="w"></span>
-</code></pre></div>
-<h1>SQL Unparser (SQL Formatter)</h1>
-<p>DataFusion now supports converting <code>Expr</code>s and <code>LogicalPlan</code>s BACK to SQL text.
-This can be useful in query federation to push predicates down into other
-systems that only accept SQL, and for building systems that generate SQL.</p>
-<p>For example, you can now convert a logical expression back to SQL text:</p>
-<div class="codehilite"><pre><span></span><code><span class="c1">// Form a logical expression that represents the SQL "a < 5 OR a = 8"</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">expr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">col</span><span class="p">(</span><span class="s">"a"</span><span class="p">).</span><span class="n">lt</span><span class="p">(</span><span class="n">lit</span><span class="p">(</span><span class="mi">5</span><span class="p">)).</span><span class="n">or</span><span class="p">(</span><span class="n">col</span><span class="p"> [...]
-<span class="c1">// convert the expression back to SQL text</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">sql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">expr_to_sql</span><span class="p">(</span><span class="o">&</span><span class="n">expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">to_string</span><span class="p">();</span><span class="w"></span>
-<span class="n">assert_eq</span><span class="o">!</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span><span class="w"> </span><span class="s">"a < 5 OR a = 8"</span><span class="p">);</span><span class="w"></span>
-</code></pre></div>
-<p>You can also do complex things like parsing SQL, modifying the plan, and convert
-it back to SQL:</p>
-<div class="codehilite"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">df</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ctx</span><span class="w"></span>
-<span class="w"> </span><span class="c1">// Use SQL to read some data from the parquet file</span>
-<span class="w"> </span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">"SELECT int_col, double_col, CAST(date_string_col as VARCHAR) FROM alltypes_plain"</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="k">await</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
-<span class="c1">// Programmatically add new filters `id > 1 and tinyint_col < double_col`</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">df</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">df</span><span class="p">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s">"id"</span><span class="p">).</span><span class="n">gt</span><span class="p">(</span><span class="n">lit</span><span class="p">(</span><span class="mi">1</span><span class="p"> [...]
-<span class="c1">// Convert the new logical plan back to SQL</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">sql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">plan_to_sql</span><span class="p">(</span><span class="n">df</span><span class="p">.</span><span class="n">logical_plan</span><span class="p">())</span><span class="o">?</span><span class="p">.</span><span class="n">to_string</span><span class="p">();</span><span class="w"></span>
-<span class="n">assert_eq</span><span class="o">!</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span><span class="w"> </span>
-<span class="w"> </span><span class="s">"SELECT alltypes_plain.int_col, alltypes_plain.double_col, CAST(alltypes_plain.date_string_col AS VARCHAR) \</span>
-<span class="s"> FROM alltypes_plain WHERE ((alltypes_plain.id > 1) AND (alltypes_plain.tinyint_col < alltypes_plain.double_col))"</span><span class="p">)</span><span class="w"></span>
-<span class="p">);</span><span class="w"></span>
-</code></pre></div>
-<p>See the <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/plan_to_sql.rs">Plan to SQL example</a> or the APIs <a href="https://docs.rs/datafusion/latest/datafusion/sql/unparser/fn.expr_to_sql.html">expr_to_sql</a> and <a href="https://docs.rs/datafusion/latest/datafusion/sql/unparser/fn.plan_to_sql.html">plan_to_sql</a> for more details.</p>
-<h1>Low Level APIs for Fast Parquet Access (indexing)</h1>
-<p>With their rising prevalence, supporting efficient access to Parquet files
-stored remotely on object storage is important. Part of doing this efficiently
-is minimizing the number of object store requests made by caching metadata and
-skipping over parts of the file that are not needed (e.g. via an index).</p>
-<p>DataFusion's Parquet reader has long internally supported <a href="https://arrow.apache.org/blog/2022/12/26/querying-parquet-with-millisecond-latency/">advanced predicate
-pushdown</a> by reading the parquet metadata from the file footer and pruning based
-on row group and data page statistics. DataFusion now also supports users
-supplying their own low level pruning information via the [<code>ParquetAccessPlan</code>]
-API.</p>
-<p>This API can be used along with index information to selectively skip decoding
-parts of the file. For example, Spice AI used this feature to add <a href="https://github.com/spiceai/spiceai/pull/1891">efficient
-support</a> for reading from DeltaLake tables and handling <a href="https://docs.delta.io/latest/delta-deletion-vectors.html">deletion vectors</a>.</p>
-<div class="codehilite"><pre><span></span><code> ┌───────────────────────┐ If the RowSelection does not include any
- │ ... │ rows from a particular Data Page, that
- │ │ Data Page is not fetched or decoded.
- │ ┌───────────────────┐ │ Note this requires a PageIndex
- │ │ ┌──────────┐ │ │
-Row │ │ │DataPage 0│ │ │ ┌────────────────────┐
-Groups │ │ └──────────┘ │ │ │ │
- │ │ ┌──────────┐ │ │ │ ParquetExec │
- │ │ ... │DataPage 1│ ◀┼ ┼ ─ ─ ─ │ (Parquet Reader) │
- │ │ └──────────┘ │ │ └ ─ ─ ─ ─ ─│ │
- │ │ ┌──────────┐ │ │ │ ╔═══════════════╗ │
- │ │ │DataPage 2│ │ │ If only rows │ ║ParquetMetadata║ │
- │ │ └──────────┘ │ │ from DataPage 1 │ ╚═══════════════╝ │
- │ └───────────────────┘ │ are selected, └────────────────────┘
- │ │ only DataPage 1
- │ ... │ is fetched and
- │ │ decoded
- │ ╔═══════════════════╗ │
- │ ║ Thrift metadata ║ │
- │ ╚═══════════════════╝ │
- └───────────────────────┘
- Parquet File
-</code></pre></div>
-<p>See the <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/parquet_index.rs">parquet_index.rs</a> and <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_parquet_index.rs">advanced_parquet_index.rs</a> examples for more details. </p>
-<p>Thanks to <a href="https://github.com/alamb">@alamb</a> and <a href="https://github.com/Ted-Jiang">@Ted-Jiang</a> for this feature. </p>
-<h2>Building Systems is Easier with DataFusion 🛠️</h2>
-<p>In addition to many incremental API improvements, there are several new APIs that make
-it easier to build systems on top of DataFusion:</p>
-<ul>
-<li>Faster and easier to use <a href="https://docs.rs/datafusion/latest/datafusion/common/tree_node/trait.TreeNode.html#overview">TreeNode API</a> for traversing and manipulating plans and expressions.</li>
-<li>All functions now use the same <a href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html">Scalar User Defined Function API</a>, making it easier to customize
- DataFusion's behavior without sacrificing performance. See <a href="https://github.com/apache/arrow-datafusion/issues/8045">ticket</a> for more details.</li>
-<li>DataFusion can now be compiled to <a href="https://github.com/apache/datafusion/discussions/9834">WASM</a>. </li>
-</ul>
-<h1>User Defined SQL Parsing Extensions</h1>
-<p>As of DataFusion 40.0.0, you can use the [<code>ExprPlanner</code>] trait to extend
-DataFusion's SQL planner to support custom operators or syntax.</p>
-<p>For example the <a href="https://github.com/datafusion-contrib/datafusion-functions-json">datafusion-functions-json</a> project uses this API to support
-JSON operators in SQL queries. It provides a custom implementation for
-planning JSON operators such as <code>-></code> and <code>->></code> with code like:</p>
-<div class="codehilite"><pre><span></span><code><span class="k">struct</span> <span class="nc">MyCustomPlanner</span><span class="p">;</span><span class="w"></span>
-
-<span class="k">impl</span><span class="w"> </span><span class="n">ExprPlanner</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">MyCustomPlanner</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="c1">// Provide custom implementation for planning a binary operators</span>
-<span class="w"> </span><span class="c1">// such as `->` and `->>`</span>
-<span class="w"> </span><span class="k">fn</span> <span class="nf">plan_binary_op</span><span class="p">(</span><span class="w"></span>
-<span class="w"> </span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">expr</span>: <span class="nc">RawBinaryExpr</span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">_schema</span>: <span class="kp">&</span><span class="nc">DFSchema</span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="n">PlannerResult</span><span class="o"><</span><span class="n">RawBinaryExpr</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="o">&</span><span class="n">expr</span><span class="p">.</span><span class="n">op</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="n">BinaryOperator</span>::<span class="n">Arrow</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* plan -> operator */</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="w"> </span><span class="n">BinaryOperator</span>::<span class="n">LongArrow</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* plan ->> operator */</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-</code></pre></div>
-<p>Thanks to <a href="https://github.com/samuelcolvin">@samuelcolvin</a>, <a href="https://github.com/jayzhan211">@jayzhan211</a> and <a href="https://github.com/dharanad">@dharanad</a> for helping make this
-feature happen.</p>
-<h1>Pluggable Support for <code>CREATE FUNCTION</code></h1>
-<p>DataFusion's new [<code>FunctionFactory</code>] API let's users provide a handler for
-<code>CREATE FUNCTION</code> SQL statements. This feature lets you build systems that
-support defining functions in SQL such as</p>
-<div class="codehilite"><pre><span></span><code><span class="c1">-- SQL based functions</span>
-<span class="k">CREATE</span> <span class="k">FUNCTION</span> <span class="n">my_func</span><span class="p">(</span><span class="n">DOUBLE</span><span class="p">,</span> <span class="n">DOUBLE</span><span class="p">)</span> <span class="k">RETURNS</span> <span class="n">DOUBLE</span>
- <span class="k">RETURN</span> <span class="err">$</span><span class="mi">1</span> <span class="o">+</span> <span class="err">$</span><span class="mi">3</span>
-<span class="p">;</span>
-
-<span class="c1">-- ML Models</span>
-<span class="k">CREATE</span> <span class="k">FUNCTION</span> <span class="n">iris</span><span class="p">(</span><span class="nb">FLOAT</span><span class="p">[])</span> <span class="k">RETURNS</span> <span class="nb">FLOAT</span><span class="p">[]</span>
-<span class="k">LANGUAGE</span> <span class="n">TORCH</span> <span class="k">AS</span> <span class="s1">'models:/iris@champion'</span><span class="p">;</span>
-
-<span class="c1">-- WebAssembly</span>
-<span class="k">CREATE</span> <span class="k">FUNCTION</span> <span class="n">func</span><span class="p">(</span><span class="nb">FLOAT</span><span class="p">[])</span> <span class="k">RETURNS</span> <span class="nb">FLOAT</span><span class="p">[]</span>
-<span class="k">LANGUAGE</span> <span class="n">WASM</span> <span class="k">AS</span> <span class="s1">'func.wasm'</span>
-</code></pre></div>
-<p>Huge thanks to <a href="https://github.com/milenkovicm">@milenkovicm</a> for this feature. There is an example of how to
-make macro like functions in <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/function_factory.rs">function_factory.rs</a>. It would be
-great if <a href="https://github.com/apache/datafusion/issues/9326">someone made a demo</a> showing how to create WASMs 🎣.</p>
-<h2>Looking Ahead: The Next Six Months 🔭</h2>
-<p>The community has been <a href="https://github.com/apache/datafusion/issues/11442">discussing what we will work on in the next six months</a>.
-Some major initiatives from that discussion are:</p>
-<ol>
-<li>
-<p><em>Performance</em>: Improve the speed of <a href="https://github.com/apache/arrow-datafusion/issues/7000">aggregating "high cardinality"</a>
- data when there are many (e.g. millions) of distinct groups as well as additional
- ideas to improve parquet performance. </p>
-</li>
-<li>
-<p><em>Modularity</em>: Make DataFusion even more modular, by completely unifying
- built in and user <a href="https://github.com/apache/datafusion/issues/8708">aggregate functions</a> and <a href="https://github.com/apache/datafusion/issues/8709">window functions</a>.</p>
-</li>
-<li>
-<p><em>LogicalTypes</em>: <a href="https://github.com/apache/datafusion/issues/11513">Introduce Logical Types</a> to make it easier to use
- different encodings like <code>StringView</code>, <code>RunEnd</code> and <code>Dictionary</code> arrays as well
- as user defined types. Thanks <a href="https://github.com/notfilippo">@notfilippo</a> for driving this. </p>
-</li>
-<li>
-<p><em>Improved Documentation</em>: Write blog posts and videos explaining
- how to use DataFusion for real-world use cases.</p>
-</li>
-<li>
-<p><em>Testing</em>: Improve CI infrastructure and test coverage, more fuzz
- testing, and better functional and performance regression testing.</p>
-</li>
-</ol>
-<h1>How to Get Involved</h1>
-<p>DataFusion is not a project built or driven by a single person, company, or
-foundation. Rather, our community of users and contributors work together to
-build a shared technology that none of us could have built alone.</p>
-<p>If you are interested in joining us we would love to have you. You can try out
-DataFusion on some of your own data and projects and let us know how it goes,
-contribute suggestions, documentation, bug reports, or a PR with documentation,
-tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
-can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/08/20/python-datafusion-40.0.0/index.html b/output/2024/08/20/python-datafusion-40.0.0/index.html
deleted file mode 100644
index cade72d..0000000
--- a/output/2024/08/20/python-datafusion-40.0.0/index.html
+++ /dev/null
@@ -1,210 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Python 40.1.0 Released, Significant usability updates - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache DataFusion Python 40.1.0 Released, Significant usability updates
- </h1>
- <p>Posted on: Tue 20 August 2024 by timsaucer</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/40.1.0/">DataFusion in Python 40.1.0</a> has been released. In addition to
-bringing in all of the new features of the core <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> package, this release
-contains <em>significant</em> updates to the user interface and documentation. We listened to the python
-user community to create a more <em>pythonic</em> experience. If you have not used the python interface to
-DataFusion before, this is an excellent time to give it a try!</p>
-<h2>Background</h2>
-<p>Until now, the python bindings for DataFusion have primarily been a thin layer to expose the
-underlying Rust functionality. This has been worked well for early adopters to use DataFusion
-within their Python projects, but some users have found it difficult to work with. As compared to
-other DataFrame libraries, these issues were raised:</p>
-<ol>
-<li>Most of the functions had little or no documentation. Users often had to refer to the Rust
-documentation or code to learn how to use DataFusion. This alienated some python users.</li>
-<li>Users could not take advantage of modern IDE features such as type hinting. These are valuable
-tools for rapid testing and development.</li>
-<li>Some of the interfaces felt “clunky” to users since some Python concepts do not always map well
-to their Rust counterparts.</li>
-</ol>
-<p>This release aims to bring a better user experience to the DataFusion Python community.</p>
-<h2>What's Changed</h2>
-<p>The most significant difference is that we have added wrapper functions and classes for most of the
-user facing interface. These wrappers, written in Python, contain both documentation and type
-annotations.</p>
-<p>This documenation is now available on the <a href="https://datafusion.apache.org/python/autoapi/datafusion/index.html">DataFusion in Python API</a> website. There you can browse
-the available functions and classes to see the breadth of available functionality.</p>
-<p>Modern IDEs use language servers such as
-<a href="https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance">Pylance</a> or
-<a href="https://jedi.readthedocs.io/en/latest/">Jedi</a> to perform analysis of python code, provide useful
-hints, and identify usage errors. These are major tools in the python user community. With this
-release, users can fully use these tools in their workflow.</p>
-<figure style="text-align: center;">
-<img alt="Fig 1: Enhanced tooltips in an IDE." class="img-responsive" src="../images/python-datafusion-40.0.0/vscode_hover_tooltip.png" width="100%"/>
-<figcaption>
-<b>Figure 1</b>: With the enhanced python wrappers, users can see helpful tool tips with
- type annotations directly in modern IDEs.
-</figcaption>
-</figure>
-<p>By having the type annotations, these IDEs can also identify quickly when a user has incorrectly
-used a function's arguments as shown in Figure 2.</p>
-<figure style="text-align: center;">
-<img alt="Fig 2: Error checking in static analysis" class="img-responsive" src="../images/python-datafusion-40.0.0/pylance_error_checking.png" width="100%"/>
-<figcaption>
-<b>Figure 2</b>: Modern Python language servers can perform static analysis and quickly find
- errors in the arguments to functions.
-</figcaption>
-</figure>
-<p>In addition to these wrapper libraries, we have enhancements to some of the functions to feel more
-easy to use.</p>
-<h3>Improved DataFrame filter arguments</h3>
-<p>You can now apply multiple <code>filter</code> statements in a single step. When using <code>DataFrame.filter</code> you
-can pass in multiple arguments, separated by a comma. These will act as a logical <code>AND</code> of all of
-the filter arguments. The following two statements are equivalent:</p>
-<div class="codehilite"><pre><span></span><code><span class="n">df</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"size"</span><span class="p">)</span> <span class="o"><</span> <span class="n">col</span><span class="p">(</span><span class="s2">"max_size"</span><span class="p">))</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</sp [...]
-<span class="n">df</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"size"</span><span class="p">)</span> <span class="o"><</span> <span class="n">col</span><span class="p">(</span><span class="s2">"max_size"</span><span class="p">),</span> <span class="n">col</span><span class="p">(</span><span class="s2">"color"</span><span class="p">)</span> <span class="o">==</span> <span class="n" [...]
-</code></pre></div>
-<h3>Comparison against literal values</h3>
-<p>It is very common to write DataFrame operations that compare an expression to some fixed value.
-For example, filtering a DataFrame might have an operation such as <code>df.filter(col("size") < lit(16))</code>.
-To make these common operations more ergonomic, you can now simply use <code>df.filter(col("size") < 16)</code>.</p>
-<p>For the right hand side of the comparison operator, you can now use any Python value that can be
-coerced into a <code>Literal</code>. This gives an easy to ready expression. For example, consider these few
-lines from one of the
-<a href="https://github.com/apache/datafusion-python/tree/main/examples/tpch">TPC-H examples</a> provided in
-the DataFusion Python repository.</p>
-<div class="codehilite"><pre><span></span><code><span class="n">df</span> <span class="o">=</span> <span class="p">(</span>
- <span class="n">df_lineitem</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"l_shipdate"</span><span class="p">)</span> <span class="o">>=</span> <span class="n">lit</span><span class="p">(</span><span class="n">date</span><span class="p">))</span>
- <span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"l_discount"</span><span class="p">)</span> <span class="o">>=</span> <span class="n">lit</span><span class="p">(</span><span class="n">DISCOUNT</span><span class="p">)</span> <span class="o">-</span> <span class="n">lit</span><span class="p">(</span><span class="n">DELTA</span><span class="p">))</span>
- <span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"l_discount"</span><span class="p">)</span> <span class="o"><=</span> <span class="n">lit</span><span class="p">(</span><span class="n">DISCOUNT</span><span class="p">)</span> <span class="o">+</span> <span class="n">lit</span><span class="p">(</span><span class="n">DELTA</span><span class="p">))</span>
- <span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"l_quantity"</span><span class="p">)</span> <span class="o"><</span> <span class="n">lit</span><span class="p">(</span><span class="n">QUANTITY</span><span class="p">))</span>
-<span class="p">)</span>
-</code></pre></div>
-<p>The above code mirrors closely how these filters would need to be applied in rust. With this new
-release, the user can simplify these lines. Also shown in the example below is that <code>filter()</code>
-now accepts a variable number of arguments and filters on all such arguments (boolean AND).</p>
-<div class="codehilite"><pre><span></span><code><span class="n">df</span> <span class="o">=</span> <span class="n">df_lineitem</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
- <span class="n">col</span><span class="p">(</span><span class="s2">"l_shipdate"</span><span class="p">)</span> <span class="o">>=</span> <span class="n">date</span><span class="p">,</span>
- <span class="n">col</span><span class="p">(</span><span class="s2">"l_discount"</span><span class="p">)</span> <span class="o">>=</span> <span class="n">DISCOUNT</span> <span class="o">-</span> <span class="n">DELTA</span><span class="p">,</span>
- <span class="n">col</span><span class="p">(</span><span class="s2">"l_discount"</span><span class="p">)</span> <span class="o"><=</span> <span class="n">DISCOUNT</span> <span class="o">+</span> <span class="n">DELTA</span><span class="p">,</span>
- <span class="n">col</span><span class="p">(</span><span class="s2">"l_quantity"</span><span class="p">)</span> <span class="o"><</span> <span class="n">QUANTITY</span><span class="p">,</span>
-<span class="p">)</span>
-</code></pre></div>
-<h3>Select columns by name</h3>
-<p>It is very common for users to perform <code>DataFrame</code> selection where they simply want a column. For
-this we have had the function <code>select_columns("a", "b")</code> or the user could perform
-<code>select(col("a"), col("b"))</code>. In the new release, we accept either full expressions in <code>select()</code>
-or strings of the column names. You can mix these as well.</p>
-<p>Where before you may have to do an operation like</p>
-<div class="codehilite"><pre><span></span><code><span class="n">df_subset</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span> <span class="n">f</span><span class="o">.</span><span class="n">abs</span><spa [...]
-</code></pre></div>
-<p>You can now simplify this to</p>
-<div class="codehilite"><pre><span></span><code><span class="n">df_subset</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"c"</span><span [...]
-</code></pre></div>
-<h3>Creating named structs</h3>
-<p>Creating a <code>struct</code> with named fields was previously difficult to use and allowed for potential
-user errors when specifying the name of each field. Now we have a cleaner interface where the
-user passes a list of tuples containing the name of the field and the expression to create.</p>
-<div class="codehilite"><pre><span></span><code><span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">named_struct</span><span class="p">([</span>
- <span class="p">(</span><span class="s2">"a"</span><span class="p">,</span> <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)),</span>
- <span class="p">(</span><span class="s2">"b"</span><span class="p">,</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">))</span>
-<span class="p">]))</span>
-</code></pre></div>
-<h2>Next Steps</h2>
-<p>While most of the user facing classes and functions have been exposed, there are a few that require
-exposure. Namely the classes in <code>datafusion.object_store</code> and the logical plans used by
-<code>datafusion.substrait</code>. The team is working on
-<a href="https://github.com/apache/datafusion-python/issues/767">these issues</a>.</p>
-<p>Additionally, in the next release of DataFusion there have been improvements made to the user-defined
-aggregate and window functions to make them easier to use. We plan on
-<a href="https://github.com/apache/datafusion-python/issues/780">bringing these enhancements</a> to this project.</p>
-<h2>Thank You</h2>
-<p>We would like to thank the following members for their very helpful discussions regarding these
-updates: <a href="https://github.com/andygrove">@andygrove</a>, <a href="https://github.com/max-muoto">@max-muoto</a>, <a href="https://github.com/slyons">@slyons</a>, <a href="https://github.com/Throne3d">@Throne3d</a>, <a href="https://github.com/Michael-J-Ward">@Michael-J-Ward</a>, <a href="https://github.com/datapythonista">@datapythonista</a>,
-<a href="https://github.com/austin362667">@austin362667</a>, <a href="https://github.com/kylebarron">@kylebarron</a>, <a href="https://github.com/simicd">@simicd</a>. The <a href="https://github.com/apache/datafusion-python/pull/750">primary PR (#750)</a> that includes these updates
-had an extensive conversation, leading to a significantly improved end product. Again, thank you
-to all who provided input!</p>
-<p>We would like to give an special thank you to <a href="https://github.com/3ok">@3ok</a> who created the initial version of the wrapper
-definitions. The work they did was time consuming and required exceptional attention to detail. It
-provided enormous value to starting this project. Thank you!</p>
-<h2>Get Involved</h2>
-<p>The DataFusion Python team is an active and engaging community and we would love
-to have you join us and help the project.</p>
-<p>Here are some ways to get involved:</p>
-<ul>
-<li>
-<p>Learn more by visiting the <a href="https://datafusion.apache.org/python/index.html">DataFusion Python project</a>
-page.</p>
-</li>
-<li>
-<p>Try out the project and provide feedback, file issues, and contribute code.</p>
-</li>
-</ul>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/08/28/datafusion-comet-0.2.0/index.html b/output/2024/08/28/datafusion-comet-0.2.0/index.html
deleted file mode 100644
index 816dabd..0000000
--- a/output/2024/08/28/datafusion-comet-0.2.0/index.html
+++ /dev/null
@@ -1,127 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.2.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache DataFusion Comet 0.2.0 Release
- </h1>
- <p>Posted on: Wed 28 August 2024 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.2.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately four weeks of development work and is the result of merging 87 PRs from 14
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.2.0.md">change log</a> for more information.</p>
-<h2>Release Highlights</h2>
-<h3>Docker Images</h3>
-<p>Docker images are now available from the <a href="https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet/265110454?tag=spark-3.4-scala-2.12-0.2.0">GitHub Container Registry</a>.</p>
-<h3>Performance improvements</h3>
-<ul>
-<li>Native shuffle is now enabled by default</li>
-<li>Improved handling of decimal types</li>
-<li>Reduced some redundant copying of batches in Filter/Scan operations</li>
-<li>Optimized performance of count aggregates</li>
-<li>Optimized performance of CASE expressions for specific uses:</li>
-<li>CASE WHEN expr THEN column ELSE null END</li>
-<li>CASE WHEN expr THEN literal ELSE literal END</li>
-<li>Optimized performance of IS NOT NULL</li>
-</ul>
-<h3>New Features</h3>
-<ul>
-<li>Window operations now support count and sum aggregates</li>
-<li>CreateArray</li>
-<li>GetStructField</li>
-<li>Support nested types in hash join</li>
-<li>Basic implementation of RLIKE expression</li>
-</ul>
-<h2>Current Performance</h2>
-<p>We use benchmarks derived from the industry standard TPC-H and TPC-DS benchmarks for tracking progress with
-performance. The following charts shows the time it takes to run the queries against 100 GB of data in
-Parquet format using a single executor with eight cores. See the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>
-for details of the environment used for these benchmarks.</p>
-<h3>Benchmark derived from TPC-H</h3>
-<p>Comet 0.2.0 provides a 62% speedup compared to Spark. This is slightly better than the Comet 0.1.0 release.</p>
-<p><img alt="Chart showing TPC-H benchmark results for Comet 0.2.0" class="img-responsive" src="../images/comet-0.2.0/tpch_allqueries.png" width="100%"/></p>
-<h3>Benchmark derived from TPC-DS</h3>
-<p>Comet 0.2.0 provides a 21% speedup compared to Spark, which is a significant improvement compared to
-Comet 0.1.0, which did not provide any speedup for this benchmark.</p>
-<p><img alt="Chart showing TPC-DS benchmark results for Comet 0.2.0" class="img-responsive" src="../images/comet-0.2.0/tpcds_allqueries.png" width="100%"/></p>
-<h2>Getting Involved</h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/09/13/string-view-german-style-strings-part-1/index.html b/output/2024/09/13/string-view-german-style-strings-part-1/index.html
deleted file mode 100644
index bae82ff..0000000
--- a/output/2024/09/13/string-view-german-style-strings-part-1/index.html
+++ /dev/null
@@ -1,171 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet
- </h1>
- <p>Posted on: Fri 13 September 2024 by Xiangpeng Hao, Andrew Lamb</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p><em>Editor's Note: This is the first of a <a href="../2024/09/13/string-view-german-style-strings-part-2/">two part</a> blog series that was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating it into <a href="https://datafusion.apache.org/">Apache DataFusion</a>, significantly accelerating string-intensive queries in the <a href="https://benchmark.clickhouse.com/">ClickBen [...]
-<p>Getting significant end-to-end performance improvements was non-trivial. Implementing StringView itself was only a fraction of the effort required. Among other things, we had to optimize UTF-8 validation, implement unintuitive compiler optimizations, tune block sizes, and time GC to realize the <a href="https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/">FDAP ecosystem</a>’s benefit. With other members of the open source community, we wer [...]
-<p>StringView is based on a simple idea: avoid some string copies and accelerate comparisons with inlined prefixes. Like most great ideas, it is “obvious” only after <a href="https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf">someone describes it clearly</a>. Although simple, straightforward implementation actually <em>slows down performance for almost every query</em>. We must, therefore, apply astute observations and diligent engineering to realize the actual bene [...]
-<p>Although this journey was successful, not all research ideas are as lucky. To accelerate the adoption of research into industry, it is valuable to integrate research prototypes with practical systems. Understanding the nuances of real-world systems makes it more likely that research designs[^2] will lead to practical system improvements.</p>
-<p>StringView support was released as part of <a href="https://crates.io/crates/arrow/52.2.0">arrow-rs v52.2.0</a> and <a href="https://crates.io/crates/datafusion/41.0.0">DataFusion v41.0.0</a>. You can try it by setting the <code>schema_force_view_types</code> <a href="https://datafusion.apache.org/user-guide/configs.html">DataFusion configuration option</a>, and we are<a href="https://github.com/apache/datafusion/issues/11682"> hard at work with the community to </a>make it the defaul [...]
-<p><img alt="End to end performance improvements for ClickBench queries" class="img-responsive" src="../images/string-view-1/figure1-performance.png" width="100%"/></p>
-<p>Figure 1: StringView improves string-intensive ClickBench query performance by 20% - 200%</p>
-<h2>What is StringView?</h2>
-<p><img alt="Diagram of using StringArray and StringViewArray to represent the same string content" class="img-responsive" src="../images/string-view-1/figure2-string-view.png" width="100%"/></p>
-<p>Figure 2: Use StringArray and StringViewArray to represent the same string content.</p>
-<p>The concept of inlined strings with prefixes (called “German Strings” <a href="https://x.com/andy_pavlo/status/1813258735965643203">by Andy Pavlo</a>, in homage to <a href="https://www.tum.de/">TUM</a>, where the <a href="https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf">Umbra paper that describes</a> them originated)
-has been used in many recent database systems (<a href="https://engineering.fb.com/2024/02/20/developer-tools/velox-apache-arrow-15-composable-data-management/">Velox</a>, <a href="https://pola.rs/posts/polars-string-type/">Polars</a>, <a href="https://duckdb.org/2021/12/03/duck-arrow.html">DuckDB</a>, <a href="https://cedardb.com/blog/german_strings/">CedarDB</a>, etc.)
-and was introduced to Arrow as a new <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringViewArray</a>[^3] type. Arrow’s original <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout">StringArray</a> is very memory efficient but less effective for certain operations.
-StringViewArray accelerates string-intensive operations via prefix inlining and a more flexible and compact string representation.</p>
-<p>A StringViewArray consists of three components:</p>
-<ol>
-<li>The <code><em>view</em></code> array</li>
-<li>The buffers</li>
-<li>The buffer pointers (IDs) that map buffer offsets to their physical locations</li>
-</ol>
-<p>Each <code>view</code> is 16 bytes long, and its contents differ based on the string’s length:</p>
-<ul>
-<li>string length < 12 bytes: the first four bytes store the string length, and the remaining 12 bytes store the inlined string.</li>
-<li>string length > 12 bytes: the string is stored in a separate buffer. The length is again stored in the first 4 bytes, followed by the buffer id (4 bytes), the buffer offset (4 bytes), and the prefix (first 4 bytes) of the string.</li>
-</ul>
-<p>Figure 2 shows an example of the same logical content (left) using StringArray (middle) and StringViewArray (right):</p>
-<ul>
-<li>The first string – <code>"Apache DataFusion"</code> – is 17 bytes long, and both StringArray and StringViewArray store the string’s bytes at the beginning of the buffer. The StringViewArray also inlines the first 4 bytes – <code>"Apac"</code> – in the view.</li>
-<li>The second string, <code>"InfluxDB"</code> is only 8 bytes long, so StringViewArray completely inlines the string content in the <code>view</code> struct while StringArray stores the string in the buffer as well.</li>
-<li>The third string <code>"Arrow Rust Impl"</code> is 15 bytes long and cannot be fully inlined. StringViewArray stores this in the same form as the first string.</li>
-<li>The last string <code>"Apache DataFusion"</code> has the same content as the first string. It’s possible to use StringViewArray to avoid this duplication and reuse the bytes by pointing the view to the previous location.</li>
-</ul>
-<p>StringViewArray provides three opportunities for outperforming StringArray:</p>
-<ol>
-<li>Less copying via the offset + buffer format</li>
-<li>Faster comparisons using the inlined string prefix</li>
-<li>Reusing repeated string values with the flexible <code>view</code> layout</li>
-</ol>
-<p>The rest of this blog post discusses how to apply these opportunities in real query scenarios to improve performance, what challenges we encountered along the way, and how we solved them.</p>
-<h2>Faster Parquet Loading</h2>
-<p><a href="https://parquet.apache.org/">Apache Parquet</a> is the de facto format for storing large-scale analytical data commonly stored LakeHouse-style, such as <a href="https://iceberg.apache.org">Apache Iceberg</a> and <a href="https://delta.io">Delta Lake</a>. Efficiently loading data from Parquet is thus critical to query performance in many important real-world workloads.</p>
-<p>Parquet encodes strings (i.e., <a href="https://docs.rs/parquet/latest/parquet/data_type/struct.ByteArray.html">byte array</a>) in a slightly different format than required for the original Arrow StringArray. The string length is encoded inline with the actual string data (as shown in Figure 4 left). As mentioned previously, StringArray requires the data buffer to be continuous and compact—the strings have to follow one after another. This requirement means that reading Parquet [...]
-<p>On the other hand, reading Parquet data as a StringViewArray can re-use the same data buffer as storing the Parquet pages because StringViewArray does not require strings to be contiguous. For example, in Figure 4, the StringViewArray directly references the buffer with the decoded Parquet page. The string <code>"Arrow Rust Impl"</code> is represented by a <code>view</code> with offset 37 and length 15 into that buffer.</p>
-<p><img alt="Diagram showing how StringViewArray can avoid copying by reusing decoded Parquet pages." class="img-responsive" src="../images/string-view-1/figure4-copying.png" width="100%"/></p>
-<p>Figure 4: StringViewArray avoids copying by reusing decoded Parquet pages.</p>
-<p><strong>Mini benchmark</strong></p>
-<p>Reusing Parquet buffers is great in theory, but how much does saving a copy actually matter? We can run the following benchmark in arrow-rs to find out:</p>
-<p>Our benchmarking machine shows that loading <em>BinaryViewArray</em> is almost 2x faster than loading BinaryArray (see next section about why this isn’t <em>String</em> ViewArray).</p>
-<p>You can read more on this arrow-rs issue: <a href="https://github.com/apache/arrow-rs/issues/5904">https://github.com/apache/arrow-rs/issues/5904</a></p>
-<h1>From Binary to Strings</h1>
-<p>You may wonder why we reported performance for BinaryViewArray when this post is about StringViewArray. Surprisingly, initially, our implementation to read StringViewArray from Parquet was much <em>slower</em> than StringArray. Why? TLDR: Although reading StringViewArray copied less data, the initial implementation also spent much more time validating <a href="https://en.wikipedia.org/wiki/UTF-8#:~:text=UTF%2D8%20is%20a%20variable,Unicode%20Standard">UTF-8</a> (as shown in Figure 5).</p>
-<p>Strings are stored as byte sequences. When reading data from (potentially untrusted) Parquet files, a Parquet decoder must ensure those byte sequences are valid UTF-8 strings, and most programming languages, including Rust, include highly<a href="https://doc.rust-lang.org/std/str/fn.from_utf8.html"> optimized routines</a> for doing so.</p>
-<p><img alt="Figure showing time to load strings from Parquet and the effect of optimized UTF-8 validation." class="img-responsive" src="../images/string-view-1/figure5-loading-strings.png" width="100%"/></p>
-<p>Figure 5: Time to load strings from Parquet. The UTF-8 validation advantage initially eliminates the advantage of reduced copying for StringViewArray.</p>
-<p>A StringArray can be validated in a single call to the UTF-8 validation function as it has a continuous string buffer. As long as the underlying buffer is UTF-8[^4], all strings in the array must be UTF-8. The Rust parquet reader makes a single function call to validate the entire buffer.</p>
-<p>However, validating an arbitrary StringViewArray requires validating each string with a separate call to the validation function, as the underlying buffer may also contain non-string data (for example, the lengths in Parquet pages).</p>
-<p>UTF-8 validation in Rust is highly optimized and favors longer strings (as shown in Figure 6), likely because it leverages SIMD instructions to perform parallel validation. The benefit of a single function call to validate UTF-8 over a function call for each string more than eliminates the advantage of avoiding the copy for StringViewArray.</p>
-<p><img alt="Figure showing UTF-8 validation throughput vs string length." class="img-responsive" src="../images/string-view-1/figure6-utf8-validation.png" width="100%"/></p>
-<p>Figure 6: UTF-8 validation throughput vs string length—StringArray’s contiguous buffer can be validated much faster than StringViewArray’s buffer.</p>
-<p>Does this mean we should only use StringArray? No! Thankfully, there’s a clever way out. The key observation is that in many real-world datasets,<a href="https://www.vldb.org/pvldb/vol17/p148-zeng.pdf"> 99% of strings are shorter than 128 bytes</a>, meaning the encoded length values are smaller than 128, <strong>in which case the length itself is also valid UTF-8</strong> (in fact, it is <a href="https://en.wikipedia.org/wiki/ASCII">ASCII</a>).</p>
-<p>This observation means we can optimize validating UTF-8 strings in Parquet pages by treating the length bytes as part of a single large string as long as the length <em>value</em> is less than 128. Put another way, prior to this optimization, the length bytes act as string boundaries, which require a UTF-8 validation on each string. After this optimization, only those strings with lengths larger than 128 bytes (less than 1% of the strings in the ClickBench dataset) are string boundari [...]
-<p>The <a href="https://github.com/apache/arrow-rs/pull/6009/files">actual implementation</a> is only nine lines of Rust (with 30 lines of comments). You can find more details in the related arrow-rs issue:<a href="https://github.com/apache/arrow-rs/issues/5995"> https://github.com/apache/arrow-rs/issues/5995</a>. As expected, with this optimization, loading StringViewArray is almost 2x faster than loading StringArray.</p>
-<h1>Be Careful About Implicit Copies</h1>
-<p>After all the work to avoid copying strings when loading from Parquet, performance was still not as good as expected. We tracked the problem to a few implicit data copies that we weren't aware of, as described in<a href="https://github.com/apache/arrow-rs/issues/6033"> this issue</a>.</p>
-<p>The copies we eventually identified come from the following innocent-looking line of Rust code, where <code>self.buf</code> is a <a href="https://en.wikipedia.org/wiki/Reference_counting">reference counted</a> pointer that should transform without copying into a buffer for use in StringViewArray.</p>
-<p>However, Rust-type coercion rules favored a blanket implementation that <em>did</em> copy data. This implementation is shown in the following code block where the <code>impl<T: AsRef<[u8]>></code> will accept any type that implements <code>AsRef<[u8]></code> and copies the data to create a new buffer. To avoid copying, users need to explicitly call <code>from_vec</code>, which consumes the <code>Vec</code> and transforms it into a buffer.</p>
-<p>Diagnosing this implicit copy was time-consuming as it relied on subtle Rust language semantics. We needed to track every step of the data flow to ensure every copy was necessary. To help other users and prevent future mistakes, we also <a href="https://github.com/apache/arrow-rs/pull/6043">removed</a> the implicit API from arrow-rs in favor of an explicit API. Using this approach, we found and fixed several <a href="https://github.com/apache/arrow-rs/pull/6039">other unintentional co [...]
-<h1>Help the Compiler by Giving it More Information</h1>
-<p>The Rust compiler’s automatic optimizations mostly work very well for a wide variety of use cases, but sometimes, it needs additional hints to generate the most efficient code. When profiling the performance of <code>view</code> construction, we found, counterintuitively, that constructing <strong>long</strong> strings was 10x faster than constructing <strong>short</strong> strings, which made short strings slower on StringViewArray than on StringArray!</p>
-<p>As described in the first section, StringViewArray treats long and short strings differently. Short strings (<12 bytes) directly inline to the <code>view</code> struct, while long strings only inline the first 4 bytes. The code to construct a <code>view</code> looks something like this:</p>
-<p>It appears that both branches of the code should be fast: they both involve copying at most 16 bytes of data and some memory shift/store operations. How could the branch for short strings be 10x slower?</p>
-<p>Looking at the assembly code using <a href="https://godbolt.org/">Compiler Explorer</a>, we (with help from <a href="https://github.com/aoli-al">Ao Li</a>) found the compiler used CPU <strong>load instructions</strong> to copy the fixed-sized 4 bytes to the <code>view</code> for long strings, but it calls a function, <a href="https://doc.rust-lang.org/std/ptr/fn.copy_nonoverlapping.html"><code>ptr::copy_non_overlapping</code></a>, to copy the inlined bytes to the <code>view</code> for [...]
-<p>However, we know something the compiler doesn’t know: the short string size is not arbitrary—it must be between 0 and 12 bytes, and we can leverage this information to avoid the function call. Our solution generates 13 copies of the function using generics, one for each of the possible prefix lengths. The code looks as follows, and <a href="https://godbolt.org/z/685YPsd5G">checking the assembly code</a>, we confirmed there are no calls to <code>ptr::copy_non_overlapping</c [...]
-<h1>End-to-End Query Performance</h1>
-<p>In the previous sections, we went out of our way to make sure loading StringViewArray is faster than StringArray. Before going further, we wanted to verify if obsessing about reducing copies and function calls has actually improved end-to-end performance in real-life queries. To do this, we evaluated a ClickBench query (Q20) in DataFusion that counts how many URLs contain the word <code>"google"</code>:</p>
-<p>This is a relatively simple query; most of the time is spent on loading the “URL” column to find matching rows. The query plan looks like this:</p>
-<p>We ran the benchmark in the DataFusion repo like this:</p>
-<p>With StringViewArray we saw a 24% end-to-end performance improvement, as shown in Figure 7. With the <code>--string-view</code> argument, the end-to-end query time is <code>944.3 ms, 869.6 ms, 861.9 ms</code> (three iterations). Without <code>--string-view</code>, the end-to-end query time is <code>1186.1 ms, 1126.1 ms, 1138.3 ms</code>.</p>
-<p><img alt="Figure showing StringView improves end to end performance by 24 percent." class="img-responsive" src="../images/string-view-1/figure7-end-to-end.png" width="100%"/></p>
-<p>Figure 7: StringView reduces end-to-end query time by 24% on ClickBench Q20.</p>
-<p>We also double-checked with detailed profiling and verified that the time reduction is indeed due to faster Parquet loading.</p>
-<h2>Conclusion</h2>
-<p>In this first blog post, we have described what it took to improve the
-performance of simply reading strings from Parquet files using StringView. While
-this resulted in real end-to-end query performance improvements, in our <a href="https://datafusion.apache.org/blog/2024/09/13/using-stringview-to-make-queries-faster-part-2.html">next
-post</a>, we explore additional optimizations enabled by StringView in DataFusion,
-along with some of the pitfalls we encountered while implementing them.</p>
-<h1>Footnotes</h1>
-<p>[^1]: Benchmarked with AMD Ryzen 7600x (12 core, 24 threads, 32 MiB L3), WD Black SN770 NVMe SSD (5150MB/4950MB seq RW bandwidth)</p>
-<p>[^2]: Xiangpeng is a PhD student at the University of Wisconsin-Madison</p>
-<p>[^3]: There is also a corresponding <em>BinaryViewArray</em> which is similar except that the data is not constrained to be UTF-8 encoded strings.</p>
-<p>[^4]: We also make sure that offsets do not break a UTF-8 code point, which is <a href="https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/buffer/offset_buffer.rs#L62-L71">cheaply validated</a>.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/09/13/string-view-german-style-strings-part-2/index.html b/output/2024/09/13/string-view-german-style-strings-part-2/index.html
deleted file mode 100644
index 9b049df..0000000
--- a/output/2024/09/13/string-view-german-style-strings-part-2/index.html
+++ /dev/null
@@ -1,148 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations
- </h1>
- <p>Posted on: Fri 13 September 2024 by Xiangpeng Hao, Andrew Lamb</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p><em>Editor's Note: This blog series was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>In the <a href="https://datafusion.staged.apache.org/2024/09/13/string-view-german-style-strings-part-1">first post</a>, we discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies.
-In this second part of the post, we describe the rest of the journey: implementing additional efficient operations for real query processing.</p>
-<h2>Faster String Operations</h2>
-<h1>Faster comparison</h1>
-<p>String comparison is ubiquitous; it is the core of
-<a href="https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/index.html"><code>cmp</code></a>,
-<a href="https://docs.rs/arrow/latest/arrow/compute/fn.min.html"><code>min</code></a>/<a href="https://docs.rs/arrow/latest/arrow/compute/fn.max.html"><code>max</code></a>,
-and <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/comparison/fn.like.html"><code>like</code></a>/<a href="https://docs.rs/arrow/latest/arrow/compute/kernels/comparison/fn.ilike.html"><code>ilike</code></a> kernels. StringViewArray is designed to accelerate such comparisons using the inlined prefix—the key observation is that, in many cases, only the first few bytes of the string determine the string comparison results.</p>
-<p>For example, to compare the strings <code>InfluxDB</code> with <code>Apache DataFusion</code>, we only need to look at the first byte to determine the string ordering or equality. In this case, since <code>A</code> is earlier in the alphabet than <code>I,</code> <code>Apache DataFusion</code> sorts first, and we know the strings are not equal. Despite only needing the first byte, comparing these strings when stored as a StringArray requires two memory accesses: 1) load the string offs [...]
-<p>For StringViewArray, typically, only one memory access is needed to load the view struct. Only if the result can not be determined from the prefix is the second memory access required. For the example above, there is no need for the second access. This technique is very effective in practice: the second access is never necessary for the more than <a href="https://www.vldb.org/pvldb/vol17/p148-zeng.pdf">60% of real-world strings which are shorter than 12 bytes</a>, as they are stored c [...]
-<p>However, functions that operate on strings must be specialized to take advantage of the inlined prefix. In addition to low-level comparison kernels, we implemented <a href="https://github.com/apache/arrow-rs/issues/5374">a wide range</a> of other StringViewArray operations that cover the functions and operations seen in ClickBench queries. Supporting StringViewArray in all string operations takes quite a bit of effort, and thankfully the Arrow and DataFusion communities are already ha [...]
-<h1>Faster <code>take</code>and<code>filter</code></h1>
-<p>After a filter operation such as <code>WHERE url <> ''</code> to avoid processing empty urls, DataFusion will often <em>coalesce</em> results to form a new array with only the passing elements.
-This coalescing ensures the batches are sufficiently sized to benefit from <a href="https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf">vectorized processing</a> in subsequent steps.</p>
-<p>The coalescing operation is implemented using the <a href="https://docs.rs/arrow/latest/arrow/compute/fn.take.html">take</a> and <a href="https://arrow.apache.org/rust/arrow/compute/kernels/filter/fn.filter.html">filter</a> kernels in arrow-rs. For StringArray, these kernels require copying the string contents to a new buffer without “holes” in between. This copy can be expensive especially when the new array is large.</p>
-<p>However, <code>take</code> and <code>filter</code> for StringViewArray can avoid the copy by reusing buffers from the old array. The kernels only need to create a new list of <code>view</code>s that point at the same strings within the old buffers.
-Figure 1 illustrates the difference between the output of both string representations. StringArray creates two new strings at offsets 0-17 and 17-32, while StringViewArray simply points to the original buffer at offsets 0 and 25.</p>
-<p><img alt="Diagram showing Zero-copy <code>take</code>/<code>filter</code> for StringViewArray" class="img-responsive" src="../images/string-view-2/figure1-zero-copy-take.png" width="100%"/></p>
-<p>Figure 1: Zero-copy <code>take</code>/<code>filter</code> for StringViewArray</p>
-<h1>When to GC?</h1>
-<p>Zero-copy <code>take/filter</code> is great for generating large arrays quickly, but it is suboptimal for highly selective filters, where most of the strings are filtered out. When the cardinality drops, StringViewArray buffers become sparse—only a small subset of the bytes in the buffer’s memory are referred to by any <code>view</code>. This leads to excessive memory usage, especially in a <a href="https://github.com/apache/datafusion/issues/11628">filter-then-coalesce sc [...]
-<p>To release unused memory, we implemented a <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html#method.gc">garbage collection (GC)</a> routine to consolidate the data into a new buffer to release the old sparse buffer(s). As the GC operation copies strings, similarly to StringArray, we must be careful about when to call it. If we call GC too early, we cause unnecessary copying, losing much of the benefit of StringViewArray. If we call GC too late, we hold [...]
-<p><code>arrow-rs</code> implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/coalesce_batches/struct.CoalesceBatchesExec.html"><code>CoalseceBatchesExec</code></a> operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, which aligns perfectly with the scenario of G [...]
-We, therefore,<a href="https://github.com/apache/datafusion/pull/11587"> implemented the GC procedure</a> inside <code>CoalseceBatchesExec</code>[^5] with a heuristic that estimates when the buffers are too sparse.</p>
-<h2>The art of function inlining: not too much, not too little</h2>
-<p>Like string inlining, <em>function</em> inlining is the process of embedding a short function into the caller to avoid the overhead of function calls (caller/callee save).
-Usually, the Rust compiler does a good job of deciding when to inline. However, it is possible to override its default using the <a href="https://doc.rust-lang.org/reference/attributes/codegen.html#the-inline-attribute"><code>#[inline(always)]</code> directive</a>.
-In performance-critical code, inlined code allows us to organize large functions into smaller ones without paying the runtime cost of function invocation.</p>
-<p>However, function inlining is <strong><em>not</em></strong> always better, as it leads to larger function bodies that are harder for LLVM to optimize (for example, suboptimal <a href="https://en.wikipedia.org/wiki/Register_allocation">register spilling</a>) and risk overflowing the CPU’s instruction cache. We observed several performance regressions where function inlining caused <em>slower</em> performance when implementing the StringViewArray comparison kernels. Careful inspec [...]
-<h2>Buffer size tuning</h2>
-<p>StringViewArray permits multiple buffers, which enables a flexible buffer layout and potentially reduces the need to copy data. However, a large number of buffers slows down the performance of other operations.
-For example, <a href="https://docs.rs/arrow/latest/arrow/array/trait.Array.html#tymethod.get_array_memory_size"><code>get_array_memory_size</code></a> needs to sum the memory size of each buffer, which takes a long time with thousands of small buffers.
-In certain cases, we found that multiple calls to <a href="https://docs.rs/arrow/latest/arrow/compute/fn.concat_batches.html"><code>concat_batches</code></a> lead to arrays with millions of buffers, which was prohibitively expensive.</p>
-<p>For example, consider a StringViewArray with the previous default buffer size of 8 KB. With this configuration, holding 4GB of string data requires almost half a million buffers! Larger buffer sizes are needed for larger arrays, but we cannot arbitrarily increase the default buffer size, as small arrays would consume too much memory (most arrays require at least one buffer). Buffer sizing is especially problematic in query processing, as we often need to construct small batches of str [...]
-<p>To balance the buffer size trade-off, we again leverage the query processing (DataFusion) semantics to decide when to use larger buffers. While coalescing batches, we combine multiple small string arrays and set a smaller buffer size to keep the total memory consumption low. In string aggregation, we aggregate over an entire Datafusion partition, which can generate a large number of strings, so we set a larger buffer size (2MB).</p>
-<p>To assist situations where the semantics are unknown, we also <a href="https://github.com/apache/arrow-rs/pull/6136">implemented</a> a classic dynamic exponential buffer size growth strategy, which starts with a small buffer size (8KB) and doubles the size of each new buffer up to 2MB. We implemented this strategy in arrow-rs and enabled it by default so that other users of StringViewArray can also benefit from this optimization. See this issue for more details: <a href="https://githu [...]
-<h2>End-to-end query performance</h2>
-<p>We have made significant progress in optimizing StringViewArray filtering operations. Now, let’s test it in the real world to see how it works!</p>
-<p>Let’s consider ClickBench query 22, which selects multiple string fields (<code>URL</code>, <code>Title</code>, and <code>SearchPhase</code>) and applies several filters.</p>
-<p>We ran the benchmark using the following command in the DataFusion repo. Again, the <code>--string-view</code> option means we use StringViewArray instead of StringArray.</p>
-<p>To eliminate the impact of the faster Parquet reading using StringViewArray (see the first part of this blog), Figure 2 plots only the time spent in <code>FilterExec</code>. Without StringViewArray, the filter takes 7.17s; with StringViewArray, the filter only takes 4.86s, a 32% reduction in time. Moreover, we see a 17% improvement in end-to-end query performance.</p>
-<p><img alt="Figure showing StringViewArray reduces the filter time by 32% on ClickBench query 22." class="img-responsive" src="../images/string-view-2/figure2-filter-time.png" width="100%"/></p>
-<p>Figure 2: StringViewArray reduces the filter time by 32% on ClickBench query 22.</p>
-<h1>Faster String Aggregation</h1>
-<p>So far, we have discussed how to exploit two StringViewArray features: reduced copy and faster filtering. This section focuses on reusing string bytes to repeat string values.</p>
-<p>As described in part one of this blog, if two strings have identical values, StringViewArray can use two different <code>view</code>s pointing at the same buffer range, thus avoiding repeating the string bytes in the buffer. This makes StringViewArray similar to an Arrow <a href="https://docs.rs/arrow/latest/arrow/array/struct.DictionaryArray.html">DictionaryArray</a> that stores Strings—both array types work well for strings with only a few distinct values.</p>
-<p>Deduplicating string values can significantly reduce memory consumption in StringViewArray. However, this process is expensive and involves hashing every string and maintaining a hash table, and so it cannot be done by default when creating a StringViewArray. We introduced an<a href="https://docs.rs/arrow/latest/arrow/array/builder/struct.GenericByteViewBuilder.html#method.with_deduplicate_strings"> opt-in string deduplication mode</a> in arrow-rs for advanced users who know their dat [...]
-<p>Once again, we leverage DataFusion query semantics to identify StringViewArray with duplicate values, such as aggregation queries with multiple group keys. For example, some <a href="https://github.com/apache/datafusion/blob/main/benchmarks/queries/clickbench/queries.sql">ClickBench queries</a> group by two columns:</p>
-<ul>
-<li><code>UserID</code> (an integer with close to 1 M distinct values)</li>
-<li><code>MobilePhoneModel</code> (a string with less than a hundred distinct values)</li>
-</ul>
-<p>In this case, the output row count is<code>count(distinct UserID) * count(distinct MobilePhoneModel)</code>, which is 100M. Each string value of <code>MobilePhoneModel</code> is repeated 1M times. With StringViewArray, we can save space by pointing the repeating values to the same underlying buffer.</p>
-<p>Faster string aggregation with StringView is part of a larger project to <a href="https://github.com/apache/datafusion/issues/7000">improve DataFusion aggregation performance</a>. We have a <a href="https://github.com/apache/datafusion/pull/11794">proof of concept implementation</a> with StringView that can improve the multi-column string aggregation by 20%. We would love your help to get it production ready!</p>
-<h1>StringView Pitfalls</h1>
-<p>Most existing blog posts (including this one) focus on the benefits of using StringViewArray over other string representations such as StringArray. As we have discussed, even though it requires a significant engineering investment to realize, StringViewArray is a major improvement over StringArray in many cases.</p>
-<p>However, there are several cases where StringViewArray is slower than StringArray. For completeness, we have listed those instances here:</p>
-<ol>
-<li><strong>Tiny strings (when strings are shorter than 8 bytes)</strong>: every element of the StringViewArray consumes at least 16 bytes of memory—the size of the <code>view</code> struct. For an array of tiny strings, StringViewArray consumes more memory than StringArray and thus can cause slower performance due to additional memory pressure on the CPU cache.</li>
-<li><strong>Many repeated short strings</strong>: Similar to the first point, StringViewArray can be slower and require more memory than a DictionaryArray because 1) it can only reuse the bytes in the buffer when the strings are longer than 12 bytes and 2) 32-bit offsets are always used, even when a smaller size (8 bit or 16 bit) could represent all the distinct values.</li>
-<li><strong>Filtering:</strong> As we mentioned above, StringViewArrays often consume more memory than the corresponding StringArray, and memory bloat quickly dominates the performance without GC. However, invoking GC also reduces the benefits of less copying so must be carefully tuned.</li>
-</ol>
-<h1>Conclusion and Takeaways</h1>
-<p>In these two blog posts, we discussed what it takes to implement StringViewArray in arrow-rs and then integrate it into DataFusion. Our evaluations on ClickBench queries show that StringView can improve the performance of string-intensive workloads by up to 2x.</p>
-<p>Given that DataFusion already <a href="https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQiI6ZmFsc2UsIkF0aGVuYSAocGFydGl0aW9uZWQpIjpmYWxzZSwiQXRoZW5hIChzaW5nbGUpIjpmYWxzZSwiQXVyb3JhIGZvciBNeVNRTCI6ZmFsc2UsIkF1cm9yYSBmb3IgUG9zdGdyZVNRTCI6ZmFsc2UsIkJ5Q29uaXR5IjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiY2hEQiAoUGFycXVldCwgcGFydGl0aW9uZWQpIjpmYWxzZSwiY2hEQiI6ZmFsc2UsIkNpdHVzIjpmYWxzZSwiQ2xpY2tIb3VzZSBDbG91ZCAoYXdzKSI6ZmFsc2UsIkNsaWNrSG91c2UgQ2xvdWQgKGF3cykgUGFyYWxsZWwgUmVwbGljYXMgT04iOmZh [...]
-<p>StringView is a big project that has received tremendous community support. Specifically, we would like to thank <a href="https://github.com/tustvold">@tustvold</a>, <a href="https://github.com/ariesdevil">@ariesdevil</a>, <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a>, <a href="https://github.com/ClSlaid">@ClSlaid</a>, <a href="https://github.com/2010YOUY01">@2010YOUY01</a>, <a href="https://github.com/chloro-pn">@chloro-pn</a>, <a href="https://github.com/a10y">@a10y</a [...]
-<p>As the introduction states, “German Style Strings” is a relatively straightforward research idea that avoid some string copies and accelerates comparisons. However, applying this (great) idea in practice requires a significant investment in careful software engineering. Again, we encourage the research community to continue to help apply research ideas to industrial systems, such as DataFusion, as doing so provides valuable perspectives when evaluating future research ques [...]
-<h3>Footnotes</h3>
-<p>[^5]: There are additional optimizations possible in this operation that the community is working on, such as <a href="https://github.com/apache/datafusion/issues/7957">https://github.com/apache/datafusion/issues/7957</a>.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/09/27/datafusion-comet-0.3.0/index.html b/output/2024/09/27/datafusion-comet-0.3.0/index.html
deleted file mode 100644
index 5251cee..0000000
--- a/output/2024/09/27/datafusion-comet-0.3.0/index.html
+++ /dev/null
@@ -1,124 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.3.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache DataFusion Comet 0.3.0 Release
- </h1>
- <p>Posted on: Fri 27 September 2024 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.3.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately four weeks of development work and is the result of merging 57 PRs from 12
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.3.0.md">change log</a> for more information.</p>
-<h2>Release Highlights</h2>
-<h3>Binary Releases</h3>
-<p>Comet jar files are now published to Maven central for amd64 and arm64 architectures (Linux only).</p>
-<p>Files can be found at https://central.sonatype.com/search?q=org.apache.datafusion</p>
-<ul>
-<li>Spark versions 3.3, 3.4, and 3.5 are supported.</li>
-<li>Scala versions 2.12 and 2.13 are supported.</li>
-</ul>
-<h3>New Features</h3>
-<p>The following expressions are now supported natively:</p>
-<ul>
-<li><code>DateAdd</code></li>
-<li><code>DateSub</code></li>
-<li><code>ElementAt</code></li>
-<li><code>GetArrayElement</code></li>
-<li><code>ToJson</code></li>
-</ul>
-<h3>Performance & Stability</h3>
-<ul>
-<li>Upgraded to DataFusion 42.0.0</li>
-<li>Reduced memory overhead due to some memory leaks being fixed</li>
-<li>Comet will now fall back to Spark for queries that use DPP, to avoid performance regressions because Comet does
- not have native support for DPP yet</li>
-<li>Improved performance when converting Spark columnar data to Arrow format</li>
-<li>Faster decimal sum and avg functions </li>
-</ul>
-<h3>Documentation Updates</h3>
-<ul>
-<li>Improved documentation for deploying Comet with Kubernetes and Helm in the <a href="https://datafusion.apache.org/comet/user-guide/kubernetes.html">Comet Kubernetes Guide</a></li>
-<li>More detailed architectural overview of Comet scan and execution in the <a href="https://datafusion.apache.org/comet/contributor-guide/plugin_overview.html">Comet Plugin Overview</a> in the contributor guide</li>
-</ul>
-<h2>Getting Involved</h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html b/output/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html
deleted file mode 100644
index f9bace5..0000000
--- a/output/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html
+++ /dev/null
@@ -1,263 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet files - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache DataFusion is now the fastest single node engine for querying Apache Parquet files
- </h1>
- <p>Posted on: Mon 18 November 2024 by Andrew Lamb, Staff Engineer at InfluxData</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>I am extremely excited to announce that <a href="https://crates.io/crates/datafusion">Apache DataFusion</a> is the
-fastest engine for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">ClickBench</a>. It is faster
-than <a href="https://duckdb.org/">DuckDB</a>, <a href="https://clickhouse.com/chdb">chDB</a> and <a href="https://clickhouse.com/">Clickhouse</a> using the same hardware. It also marks
-the first time a <a href="https://www.rust-lang.org/">Rust</a>-based engine holds the top spot, which has previously
-been held by traditional C/C++-based engines.</p>
-<p><img alt="Apache DataFusion Logo" class="img-responsive" src="../images/2x_bgwhite_original.png" width="80%"/></p>
-<p><img alt="ClickBench performance for DataFusion 43.0.0" class="img-responsive" src="../images/clickbench-datafusion-43/perf.png" width="100%"/></p>
-<p><strong>Figure 1</strong>: 2024-11-16 <a href="https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQiI6ZmFsc2UsIkFsbG95REIgKHR1bmVkKSI6ZmFsc2UsIkF0aGVuYSAocGFydGl0aW9uZWQpIjpmYWxzZSwiQXRoZW5hIChzaW5nbGUpIjpmYWxzZSwiQXVyb3JhIGZvciBNeVNRTCI6ZmFsc2UsIkF1cm9yYSBmb3IgUG9zdGdyZVNRTCI6ZmFsc2UsIkJ5Q29uaXR5IjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiY2hEQiAoRGF0YUZyYW1lKSI6ZmFsc2UsImNoREIgKFBhcnF1ZXQsIHBhcnRpdGlvbmVkKSI6dHJ1ZSwiY2hEQiI6ZmFsc2UsIkNpdHVzIjpmYWxzZSwiQ2xpY2tIb3VzZSBDbG91ZCAoYXdzKSI6 [...]
-partitioned 14 GB Parquet dataset (100 files, each ~140MB) on a <code>c6a.4xlarge</code> (16
-CPU / 32 GB RAM) VM. Measurements are relative (<code>1.x</code>) to results using
-different hardware.</p>
-<p>Best in class performance on Parquet is now available to anyone. DataFusion’s
-open design lets you start quickly with a full featured Query Engine, including
-SQL, data formats, catalogs, and more, and then customize any behavior you need.
-I predict the continued emergence of new classes of data systems now that
-creators can focus the bulk of their innovation on areas such as query
-languages, system integrations, and data formats rather than trying to play
-catchup with core engine performance.</p>
-<p>ClickBench also includes results for proprietary storage formats, which require
-costly load / export steps, making them useful in fewer use cases and thus much
-less important than open formats (though the idea of use case specific formats
-is interesting[^2]).</p>
-<p>This blog post highlights some of the techniques we used to achieve this
-performance, and celebrates the teamwork involved.</p>
-<h1>A Strong History of Performance Improvements</h1>
-<p>Performance has long been a core focus for DataFusion's community, and
-speed attracts users and contributors. Recently, we seem to have been
-even more focused on performance, including in July, 2024 when <a href="https://www.linkedin.com/in/mehmet-ozan-kabak/">Mehmet Ozan
-Kabak</a>, CEO of <a href="https://www.synnada.ai/">Synnada</a>, again <a href="https://github.com/apache/datafusion/issues/11442#issuecomment-2226834443">suggested focusing on performance</a>. This
-got many of us excited (who doesn’t love a challenge!), and we have subsequently
-rallied to steadily improve the performance release on release as shown in
-Figure 2.</p>
-<p><img alt="ClickBench performance results over time for DataFusion" class="img-responsive" src="../images/clickbench-datafusion-43/perf-over-time.png" width="100%"/></p>
-<p><strong>Figure 2</strong>: ClickBench performance improved over 30% between DataFusion 34
-(released Dec. 2023) and DataFusion 43 (released Nov. 2024).</p>
-<p>Like all good optimization efforts, ours took sustained effort as DataFusion ran
-out of <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">single 2x performance improvements</a> several years ago. Working together our
-community of engineers from around the world[^3] and all experience levels[^4]
-pulled it off (check out <a href="https://github.com/apache/datafusion/issues/12821">this discussion</a> to get a sense). It may be a "<a href="https://db.cs.cmu.edu/seminar2024/">hobo
-sandwich</a>" [^5], but it is a tasty one!</p>
-<p>Of course, most of these techniques have been implemented and described before,
-but until now they were only available in proprietary systems such as
-<a href="https://www.vertica.com/">Vertica</a>, <a href="https://www.databricks.com/product/photon">DataBricks
-Photon</a>, or
-<a href="https://www.snowflake.com/en/">Snowflake</a> or in tightly integrated open source
-systems such as <a href="https://duckdb.org/">DuckDB</a> or
-<a href="https://clickhouse.com/">ClickHouse</a> which were not designed to be extended.</p>
-<h2>StringView</h2>
-<p>Performance improved for all queries when DataFusion switched to using Arrow
-<code>StringView</code>. Using <code>StringView</code> “just” saves some copies and avoids one memory
-access for certain comparisons. However, these copies and comparisons happen to
-occur in many of the hottest loops during query processing, so optimizing them
-resulted in measurable performance improvements.</p>
-<p><img alt="Illustration of how take works with StringView" class="img-responsive" src="../images/clickbench-datafusion-43/string-view-take.png" width="80%"/></p>
-<p><strong>Figure 3:</strong> Figure from <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">Using StringView / German Style Strings to Make
-Queries Faster: Part 1</a> showing how <code>StringView</code> saves copying data in many cases.</p>
-<p>Using StringView to make DataFusion faster for ClickBench required substantial
-careful, low level optimization work described in <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">Using StringView / German
-Style Strings to Make Queries Faster: Part 1</a> and <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">Part 2</a>. However, it <em>also</em>
-required extending the rest of DataFusion’s operations to support the new type.
-You can get a sense of the magnitude of the work required by looking at the 100+
-pull requests linked to the epic in arrow-rs
-(<a href="https://github.com/apache/arrow-rs/issues/5374">here</a>) and three major epics
-(<a href="https://github.com/apache/datafusion/issues/10918">here</a>,
-<a href="https://github.com/apache/datafusion/issues/11790">here</a> and
-<a href="https://github.com/apache/datafusion/issues/11752">here</a>) in DataFusion.</p>
-<p>Here is a partial list of people involved in the project (I am sorry to those whom I forgot)</p>
-<ul>
-<li><strong>Arrow</strong>: <a href="https://github.com/XiangpengHao">Xiangpeng Hao</a> (InfluxData’s amazing 2024 summer intern and UW Madison PhD), <a href="https://github.com/ariesdevil">Yijun Zhao</a> from DataBend Labs, and <a href="https://github.com/tustvold">Raphael Taylor-Davies</a> laid the foundation. <a href="https://github.com/RinChanNOWWW">RinChanNOW</a> from Tencent and <a href="https://github.com/a10y">Andrew Duffy</a> from SpiralDB helped push it along in the ear [...]
-<li><strong>DataFusion</strong>: <a href="https://github.com/XiangpengHao">Xiangpeng Hao</a>, again charted the initial path and <a href="https://github.com/Weijun-H">Weijun Huang</a>, <a href="https://github.com/dharanad">Dharan Aditya</a> <a href="https://github.com/Lordworms">Lordworms</a>, <a href="https://github.com/goldmedal">Jax Liu</a>, <a href="https://github.com/wiedld">wiedld</a>, <a href="https://github.com/tlm365">Tai Le Manh</a>, <a href="https://github.com/my-vegetable-h [...]
-<li><strong>DataFusion String Function Migration</strong>: <a href="https://github.com/tshauck">Trent Hauck</a> organized the effort and set the patterns, <a href="https://github.com/goldmedal">Jax Liu</a> made a clever testing framework, and <a href="https://github.com/austin362667">Austin Liu</a>, <a href="https://github.com/demetribu">Dmitrii Bu</a>, <a href="https://github.com/tlm365">Tai Le Manh</a>, <a href="https://github.com/PsiACE">Chojan Shang</a>, <a href="https://github.com/ [...]
-</ul>
-<h2>Parquet</h2>
-<p>Part of the reason for DataFusion's speed in ClickBench is reading Parquet files (really) quickly,
-which reflects invested effort in the Parquet reading system (see <a href="https://www.influxdata.com/blog/querying-parquet-millisecond-latency/">Querying
-Parquet with Millisecond Latency</a> )</p>
-<p>The <a href="https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/parquet/struct.ParquetExec.html">DataFusion ParquetExec</a> (built on the <a href="https://crates.io/crates/parquet">Rust Parquet Implementation</a>) is now the most
-sophisticated open source Parquet reader I know of. It has every optimization we
-can think of for reading Parquet, including projection pushdown, predicate
-pushdown (row group metadata, page index, and bloom filters), limit pushdown,
-parallel reading, interleaved I/O, and late materialized filtering (coming soon ™️
-by default). Some recent work from <a href="https://github.com/itsjunetime">June</a>
-<a href="https://github.com/apache/datafusion/pull/12135">recently unblocked a remaining hurdle</a> for enabling late materialized
-filtering, and conveniently <a href="https://github.com/XiangpengHao">Xiangpeng Hao</a> is
-working on the <a href="https://github.com/apache/arrow-datafusion/issues/3463">final piece</a> (no pressure😅)</p>
-<h2>Skipping Partial Aggregation When It Doesn't Help</h2>
-<p>Many ClickBench queries are aggregations that summarize millions of rows, a
-common task for reporting and dashboarding. DataFusion uses state of the art
-<a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.Accumulator.html#tymethod.state">two phase aggregation</a> plans. Normally, two phase aggregation works well as the
-first phase consolidates many rows immediately after reading, while the data is
-still in cache. However, for certain “high cardinality” aggregate queries (that
-have large numbers of groups), <a href="https://github.com/apache/datafusion/issues/6937">the two phase aggregation strategy used in
-DataFusion was inefficient</a>,
-manifesting in relatively slower performance compared to other engines for
-ClickBench queries such as</p>
-<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="ss">"WatchID"</span><span class="p">,</span> <span class="ss">"ClientIP"</span><span class="p">,</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">AS</span> <span class="k">c</span><span class="p">,</span> <span class="p">...</span>
-<span class="k">FROM</span> <span class="n">hits</span>
-<span class="k">GROUP</span> <span class="k">BY</span> <span class="ss">"WatchID"</span><span class="p">,</span> <span class="ss">"ClientIP"</span> <span class="cm">/* <----- 13M Distinct Groups!!! */</span>
-<span class="k">ORDER</span> <span class="k">BY</span> <span class="k">c</span> <span class="k">DESC</span>
-<span class="k">LIMIT</span> <span class="mi">10</span><span class="p">;</span>
-</code></pre></div>
-<p>For such queries, the first aggregation phase does not significantly
-reduce the number of rows, which wastes significant effort. <a href="https://github.com/korowa">Eduard
-Karacharov</a> contributed a <a href="https://github.com/apache/datafusion/pull/11627">dynamic strategy</a> to
-bypass the first phase when it is not working efficiently, shown in Figure 4.</p>
-<p><img alt="Two phase aggregation diagram from DataFusion API docs annotated to show first phase not helping" class="img-responsive" src="../images/clickbench-datafusion-43/skipping-partial-aggregation.png" width="100%"/></p>
-<p><strong>Figure 4</strong>: Diagram from <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.Accumulator.html#tymethod.state">DataFusion API docs</a> showing when the multi-phase
-grouping is not effective</p>
-<h2>Optimized Multi-Column Grouping</h2>
-<p>Another method for improving analytic database performance is specialized (aka
-highly optimized) versions of operations for different data types, which the
-system picks at runtime based on the query. Like other systems, DataFusion has
-specialized code for handling different types of group columns. For example,
-there is <a href="https://github.com/apache/datafusion/blob/73507c307487708deb321e1ba4e0d302084ca27e/datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs">special code</a> that handles <code>GROUP BY int_id</code> and <a href="https://github.com/apache/datafusion/blob/73507c307487708deb321e1ba4e0d302084ca27e/datafusion/physical-plan/src/aggregates/group_values/single_group_by/bytes.rs">different special
-code</a> that handles <code>GROUP BY string_id</code> .</p>
-<p>When a query groups by multiple columns, it is tricker to apply this technique.
-For example <code>GROUP BY string_id, int_id</code> and <code>GROUP BY int_id, string_id</code> have
-different optimal structures, but it is not possible to include specialized
-versions for all possible combinations of group column types.</p>
-<p>DataFusion includes <a href="https://github.com/apache/datafusion/blob/73507c307487708deb321e1ba4e0d302084ca27e/datafusion/physical-plan/src/aggregates/group_values/row.rs#L33-L39">a general Row based mechanism</a> that works for any
-combination of column types, but this general mechanism copies each value twice
-as shown in Figure 5. The cost of this copy <a href="https://github.com/apache/datafusion/issues/9403">is especially high for variable
-length strings and binary data</a>.</p>
-<p><img alt="Row based storage for multiple group columns" class="img-responsive" src="../images/clickbench-datafusion-43/row-based-storage.png" width="100%"/></p>
-<p><strong>Figure 5</strong>: Prior to DataFusion 43.0.0, queries with multiple group columns
-used Row based group storage and copied each group value twice. This copy
-consumes a substantial amount of the query time for queries with many distinct
-groups, such as several of the queries in ClickBench.</p>
-<p>Many optimizations in Databases boil down to simply avoiding copies, and this
-was no exception. The trick was to figure out how to avoid copies without
-causing per-column comparison overhead to dominate or complexity to get out of
-hand. In a great example of diligent and disciplined engineering, <a href="https://github.com/jayzhan211">Jay
-Zhan</a> tried <a href="https://github.com/apache/datafusion/pull/10937">several</a>, <a href="https://github.com/apache/datafusion/pull/10976">different</a> approaches until arriving
-at the [one shipped in DataFusion <code>43.0.0</code>], shown in Figure 6.</p>
-<p><img alt="Column based storage for multiple group columns" class="img-responsive" src="../images/clickbench-datafusion-43/column-based-storage.png" width="100%"/></p>
-<p><strong>Figure 6</strong>: DataFusion 43.0.0’s new columnar group storage copies each group
-value exactly once, which is significantly faster when grouping by multiple
-columns.</p>
-<p>Huge thanks as well to <a href="https://github.com/eejbyfeldt">Emil Ejbyfeldt</a> and
-<a href="https://github.com/Dandandan">Daniël Heres</a> for their help reviewing and to
-<a href="https://github.com/Rachelint">Rachelint (kamille</a>) for reviewing and
-contributing a faster <a href="https://github.com/apache/datafusion/pull/12996">vectorized append and compare for multiple groups</a> which
-will be released in DataFusion 44. The discussion on <a href="https://github.com/apache/datafusion/issues/9403">the ticket</a> is another
-great example of the power of the DataFusion community working together to build
-great software.</p>
-<h1>What’s Next 🚀</h1>
-<p>Just as I expect the performance of other engines to improve, DataFusion has
-several more performance improvements lined up itself:</p>
-<ol>
-<li><a href="https://github.com/apache/datafusion/pull/11943#top">Intermediate results blocked management</a> (thanks again <a href="https://github.com/Rachelint">Rachelint (kamille</a>)</li>
-<li><a href="https://github.com/apache/datafusion/issues/3463">Enable parquet filter pushdown by default</a></li>
-</ol>
-<p>We are also talking about what to focus on over the <a href="https://github.com/apache/datafusion/issues/13274">next three
-months</a> and are always
-looking for people to help! If you want to geek out (obsess??) about performance
-and other features with engineers from around the world, <a href="https://datafusion.apache.org/contributor-guide/communication.html">we would love you to
-join us</a>.</p>
-<h1>Additional Thanks</h1>
-<p>In addition to the people called out above, thanks:</p>
-<ol>
-<li><a href="https://github.com/pmcgleenon">Patrick McGleenon</a> for running ClickBench and gathering this data (<a href="https://github.com/apache/datafusion/issues/13099#issuecomment-2478314793">source</a>).</li>
-<li>Everyone I missed in the shoutouts – there are so many of you. We appreciate everyone.</li>
-</ol>
-<h1>Conclusion</h1>
-<p>I have dreamed about DataFusion being on top of the ClickBench leaderboard for
-several years. I often watched with envy improvements in systems backed by large
-VC investments, internet companies, or world class research institutions, and
-doubted that we could pull off something similar in an open source project with
-always limited time.</p>
-<p>The fact that we have now surpassed those other systems in query performance I
-think speaks to the power and possibility of focusing on community and aligning
-our collective enthusiasm and skills towards a common goal. Of course, being on
-the top in any particular benchmark is likely fleeting as other engines will
-improve, but so will DataFusion!</p>
-<p>I love working on DataFusion – the people, the quality of the code, my
-interactions and the results we have achieved together far surpass my
-expectations as well as most of my other software development experiences. I
-can’t wait to see what people will build next, and hope to <a href="https://github.com/apache/datafusion">see you
-online</a>.</p>
-<h2>Notes</h2>
-<p>[^1]: Note that DuckDB is slightly faster on the ‘cold’ run.</p>
-<p>[^2]: Want to try your hand at a custom format for ClickBench fame / glory?: <a href="https://github.com/apache/datafusion/issues/13448">Make DataFusion the fastest engine in ClickBench with custom file format</a></p>
-<p>[^3]: We have contributors from North America, South American, Europe, Asia, Africa and Australia</p>
-<p>[^4]: Undergraduates, PhD, Junior engineers, and getting-kind-of-crotchety experienced engineers</p>
-<p>[^5]: Thanks to Andy Pavlo, I love that nomenclature</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/11/19/datafusion-python-udf-comparisons/index.html b/output/2024/11/19/datafusion-python-udf-comparisons/index.html
deleted file mode 100644
index 82a3f1c..0000000
--- a/output/2024/11/19/datafusion-python-udf-comparisons/index.html
+++ /dev/null
@@ -1,601 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Comparing approaches to User Defined Functions in Apache DataFusion using Python - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Comparing approaches to User Defined Functions in Apache DataFusion using Python
- </h1>
- <p>Posted on: Tue 19 November 2024 by timsaucer</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Personal Context</h2>
-<p>For a few months now I’ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
-fast query engine written in Rust. From my experience the language that nearly all data scientists
-are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger tasks that require
-distributed processing.</p>
-<p>In addition to DataFusion, there is another Rust based newcomer to the DataFrame world,
-<a href="https://pola.rs/">Polars</a>. The latter is growing extremely fast, and it serves many of the same
-use cases as DataFusion. For my use cases, I'm interested in DataFusion because I want to be able
-to build small scale tests rapidly and then scale them up to larger distributed systems with ease.
-I do recommend evaluating Polars for in-memory work.</p>
-<p>Personally, I would love a single query approach that is fast for both in-memory usage and can
-extend to large batch processing to exploit parallelization. I think DataFusion, coupled with
-<a href="https://datafusion.apache.org/ballista/">Ballista</a> or
-<a href="https://github.com/apache/datafusion-ray">DataFusion-Ray</a>, may provide this solution.</p>
-<p>As I’m testing, I’m primarily limiting my work to the
-<a href="https://datafusion.apache.org/python/">datafusion-python</a> project, a wrapper around the Rust
-DataFusion library. This wrapper gives you the speed advantages of keeping all of the data in the
-Rust implementation and the ergonomics of working in Python. Personally, I would prefer to work
-purely in Rust, but I also recognize that since the industry works in Python we should meet the
-people where they are.</p>
-<h2>User-Defined Functions</h2>
-<p>The focus of this post is User-Defined Functions (UDFs). The DataFusion library gives a lot of
-useful functions already for doing DataFrame manipulation. These are going to be similar to those
-you find in other DataFrame libraries. You’ll be able to do simple arithmetic, create substrings of
-columns, or find the average value across a group of rows. These cover most of the use cases
-you’ll need in a DataFrame.</p>
-<p>However, there will always arise times when you want a custom function. With UDFs you open a
-world of possibilities in your code. Sometimes there simply isn’t an easy way to use built-in
-functions to achieve your goals.</p>
-<p>In the following, I’m going to demonstrate two example use cases. These are based on real world
-problems I’ve encountered. Also I want to demonstrate the approach of “make it work, make it work
-well, make it work fast” that is a motto I’ve seen thrown around in data science.</p>
-<p>I will demonstrate three approaches to writing UDFs. In order of increasing performance they are</p>
-<ul>
-<li>Writing a pure Python function to do your computation</li>
-<li>Using the PyArrow libraries in Python to accelerate your processing</li>
-<li>Writing a UDF in Rust and exposing it to Python</li>
-</ul>
-<p>Additionally I will demonstrate two variants of this. The first will be nearly identical to the
-PyArrow library approach to simplify understanding how to connect the Rust code to Python. In the
-second version we will do the iteration through the input arrays ourselves to give even greater
-flexibility to the user.</p>
-<p>Here are the two example use cases, taken from my own work but generalized.</p>
-<h3>Use Case 1: Scalar Function</h3>
-<p>I have a DataFrame and a list of tuples that I’m interested in. I want to filter out the DataFrame
-to only have values that match those tuples from certain columns in the DataFrame.</p>
-<p>To give a concrete example, we will use data generated for the <a href="https://www.tpc.org/tpch/">TPC-H benchmarks</a>.
-Suppose I have a table of sales line items. There are many columns, but I am interested in three: a
-part key (<code>p_partkey</code>), supplier key (<code>p_suppkey</code>), and return status (<code>p_returnflag</code>). I want
-only to return a DataFrame with a specific combination of these three values. That is, I want
-to know if part number 1530 from supplier 4031 was sold (not returned), so I want a specific
-combination of <code>p_partkey = 1530</code>, <code>p_suppkey = 4031</code>, and <code>p_returnflag = 'N'</code>. I have a small
-handful of these combinations I want to return.</p>
-<p>Probably the most ergonomic way to do this without UDF is to turn that list of tuples into a
-DataFrame itself, perform a join, and select the columns from the original DataFrame. If we were
-working in PySpark we would probably broadcast join the DataFrame created from the tuple list since
-it is tiny. In practice, I have found that with some DataFrame libraries performing a filter rather
-than a join can be significantly faster. This is worth profiling for your specific use case.</p>
-<h3>Use Case 2: Aggregate Function</h3>
-<p>I have a DataFrame with many values that I want to aggregate. I have already analyzed it and
-determined there is a noise level below which I do not want to include in my analysis. I want to
-compute a sum of only values that are above my noise threshold.</p>
-<p>This can be done fairly easy without leaning on a User Defined Aggegate Function (UDAF). You can
-simply filter the DataFrame and then aggregate using the built-in <code>sum</code> function. Here, we
-demonstrate doing this as a UDF primarily as an example of how to write UDAFs. We will use the
-PyArrow compute approach.</p>
-<h2>Pure Python approach</h2>
-<p>The fastest way (developer time, not code time) for me to implement the scalar problem solution
-was to do something along the lines of “for each row, check the values of interest contains that
-tuple”. I’ve published this as
-<a href="https://github.com/apache/datafusion-python/blob/main/examples/python-udf-comparisons.py">an example</a>
-in the <a href="https://github.com/apache/datafusion-python">datafusion-python repository</a>. Here is an
-example of how this can be done:</p>
-<div class="codehilite"><pre><span></span><code><span class="n">values_of_interest</span> <span class="o">=</span> <span class="p">[</span>
- <span class="p">(</span><span class="mi">1530</span><span class="p">,</span> <span class="mi">4031</span><span class="p">,</span> <span class="s2">"N"</span><span class="p">),</span>
- <span class="p">(</span><span class="mi">6530</span><span class="p">,</span> <span class="mi">1531</span><span class="p">,</span> <span class="s2">"N"</span><span class="p">),</span>
- <span class="p">(</span><span class="mi">5618</span><span class="p">,</span> <span class="mi">619</span><span class="p">,</span> <span class="s2">"N"</span><span class="p">),</span>
- <span class="p">(</span><span class="mi">8118</span><span class="p">,</span> <span class="mi">8119</span><span class="p">,</span> <span class="s2">"N"</span><span class="p">),</span>
-<span class="p">]</span>
-
-<span class="k">def</span> <span class="nf">is_of_interest_impl</span><span class="p">(</span>
- <span class="n">partkey_arr</span><span class="p">:</span> <span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">,</span>
- <span class="n">suppkey_arr</span><span class="p">:</span> <span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">,</span>
- <span class="n">returnflag_arr</span><span class="p">:</span> <span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">,</span>
-<span class="p">)</span> <span class="o">-></span> <span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">:</span>
- <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="k">for</span> <span class="n">idx</span><span class="p">,</span> <span class="n">partkey</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">partkey_arr</span><span class="p">):</span>
- <span class="n">partkey</span> <span class="o">=</span> <span class="n">partkey</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
- <span class="n">suppkey</span> <span class="o">=</span> <span class="n">suppkey_arr</span><span class="p">[</span><span class="n">idx</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
- <span class="n">returnflag</span> <span class="o">=</span> <span class="n">returnflag_arr</span><span class="p">[</span><span class="n">idx</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
- <span class="n">value</span> <span class="o">=</span> <span class="p">(</span><span class="n">partkey</span><span class="p">,</span> <span class="n">suppkey</span><span class="p">,</span> <span class="n">returnflag</span><span class="p">)</span>
- <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">value</span> <span class="ow">in</span> <span class="n">values_of_interest</span><span class="p">)</span>
-
- <span class="k">return</span> <span class="n">pa</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
-
-<span class="c1"># Wrap our custom function with `datafusion.udf`, annotating expected </span>
-<span class="c1"># parameter and return types</span>
-<span class="n">is_of_interest</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span>
- <span class="n">is_of_interest_impl</span><span class="p">,</span>
- <span class="p">[</span><span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p">(),</span> <span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p">(),</span> <span class="n">pa</span><span class="o">.</span><span class="n">utf8</span><span class="p">()],</span>
- <span class="n">pa</span><span class="o">.</span><span class="n">bool_</span><span class="p">(),</span>
- <span class="s2">"stable"</span><span class="p">,</span>
-<span class="p">)</span>
-
-<span class="n">df_udf_filter</span> <span class="o">=</span> <span class="n">df_lineitem</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
- <span class="n">is_of_interest</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"l_partkey"</span><span class="p">),</span> <span class="n">col</span><span class="p">(</span><span class="s2">"l_suppkey"</span><span class="p">),</span> <span class="n">col</span><span class="p">(</span><span class="s2">"l_returnflag"</span><span class="p">))</span>
-<span class="p">)</span>
-</code></pre></div>
-<p>When working with a DataFusion UDF in Python, you define your function to take in some number of
-expressions. During the evaluation, these will get computed into their corresponding values and
-passed to your UDF as a PyArrow Array. We must return an Array also with the same number of
-elements (rows). So the UDF example just iterates through all of the arrays and checks to see if
-the tuple created from these columns matches any of those that we’re looking for.</p>
-<p>I’ll repeat because this is something that tripped me up the first time I wrote a UDF for
-datafusion: <strong>DataFusion UDFs, even scalar UDFs, process an array of values at a time not a single
-row.</strong> This is different from some other DataFrame libraries and you may need to recognize a slight
-change in mentality.</p>
-<p>Some important lines here are the lines like <code>partkey = partkey.as_py()</code>. When we do this, we pay a
-heavy cost. Now instead of keeping the analysis in the Rust code, we have to take the values in the
-array and convert them over to Python objects. In this case we end up getting two numbers and a
-string as real Python objects, complete with reference counting and all. Also we are iterating
-through the array in Python rather than Rust native. These will <strong>significantly</strong> slow down your
-code. Any time you have to cross the barrier where you change values inside the Rust arrays into
-Python objects or vice versa you will pay <strong>heavy</strong> cost in that transformation. You will want to
-design your UDFs to avoid this as much as possible.</p>
-<h2>Python approach using PyArrow compute</h2>
-<p>DataFusion uses <a href="https://arrow.apache.org/">Apache Arrow</a> as its in-memory data format. This can
-be seen in the way that Arrow Arrays are passed into the UDFs. We can take advantage of the fact
-that <a href="https://arrow.apache.org/docs/python/">PyArrow</a>, the canonical Python Arrow implementation,
-provides a variety of
-useful functions. In the example below, we are only using a few of the boolean functions and the
-equality function. Each of these functions takes two arrays and analyzes them row by row. In the
-below example, we shift the logic around a little since we are now operating on an entire array of
-values instead of checking a single row ourselves.</p>
-<div class="codehilite"><pre><span></span><code><span class="kn">import</span> <span class="nn">pyarrow.compute</span> <span class="k">as</span> <span class="nn">pc</span>
-
-<span class="k">def</span> <span class="nf">udf_using_pyarrow_compute_impl</span><span class="p">(</span>
- <span class="n">partkey_arr</span><span class="p">:</span> <span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">,</span>
- <span class="n">suppkey_arr</span><span class="p">:</span> <span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">,</span>
- <span class="n">returnflag_arr</span><span class="p">:</span> <span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">,</span>
-<span class="p">)</span> <span class="o">-></span> <span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">:</span>
- <span class="n">results</span> <span class="o">=</span> <span class="kc">None</span>
- <span class="k">for</span> <span class="n">partkey</span><span class="p">,</span> <span class="n">suppkey</span><span class="p">,</span> <span class="n">returnflag</span> <span class="ow">in</span> <span class="n">values_of_interest</span><span class="p">:</span>
- <span class="n">filtered_partkey_arr</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="n">equal</span><span class="p">(</span><span class="n">partkey_arr</span><span class="p">,</span> <span class="n">partkey</span><span class="p">)</span>
- <span class="n">filtered_suppkey_arr</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="n">equal</span><span class="p">(</span><span class="n">suppkey_arr</span><span class="p">,</span> <span class="n">suppkey</span><span class="p">)</span>
- <span class="n">filtered_returnflag_arr</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="n">equal</span><span class="p">(</span><span class="n">returnflag_arr</span><span class="p">,</span> <span class="n">returnflag</span><span class="p">)</span>
-
- <span class="n">resultant_arr</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="n">and_</span><span class="p">(</span><span class="n">filtered_partkey_arr</span><span class="p">,</span> <span class="n">filtered_suppkey_arr</span><span class="p">)</span>
- <span class="n">resultant_arr</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="n">and_</span><span class="p">(</span><span class="n">resultant_arr</span><span class="p">,</span> <span class="n">filtered_returnflag_arr</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="n">results</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">results</span> <span class="o">=</span> <span class="n">resultant_arr</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">results</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="n">or_</span><span class="p">(</span><span class="n">results</span><span class="p">,</span> <span class="n">resultant_arr</span><span class="p">)</span>
-
- <span class="k">return</span> <span class="n">results</span>
-
-
-<span class="n">udf_using_pyarrow_compute</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span>
- <span class="n">udf_using_pyarrow_compute_impl</span><span class="p">,</span>
- <span class="p">[</span><span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p">(),</span> <span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p">(),</span> <span class="n">pa</span><span class="o">.</span><span class="n">utf8</span><span class="p">()],</span>
- <span class="n">pa</span><span class="o">.</span><span class="n">bool_</span><span class="p">(),</span>
- <span class="s2">"stable"</span><span class="p">,</span>
-<span class="p">)</span>
-
-<span class="n">df_udf_pyarrow_compute</span> <span class="o">=</span> <span class="n">df_lineitem</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
- <span class="n">udf_using_pyarrow_compute</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"l_partkey"</span><span class="p">),</span> <span class="n">col</span><span class="p">(</span><span class="s2">"l_suppkey"</span><span class="p">),</span> <span class="n">col</span><span class="p">(</span><span class="s2">"l_returnflag"</span><span class="p">))</span>
-<span class="p">)</span>
-</code></pre></div>
-<p>The idea in the code above is that we will iterate through each of the values of interest, which we
-expect to be small. For each of the columns, we compare the value of interest to it’s corresponding
-array using <code>pyarrow.compute.equal</code>. This will give use three boolean arrays. We have a match to
-the tuple if we have a row in all three arrays that is true, so we use <code>pyarrow.compute.and_</code>. Now
-our return value from the UDF needs to include arrays for which any of the values of interest list
-of tuples exists, so we take the result from the current loop and perform a <code>pyarrow.compute.or_</code>
-on it.</p>
-<p>From my benchmarking, switching from approach of converting values into Python objects to this
-approach of using the PyArrow built-in functions leads to about a 10x speed improvement in this
-simple problem.</p>
-<p>It’s worth noting that almost all of the PyArrow compute functions expect to take one or two arrays
-as their arguments. If you need to write a UDF that is evaluating three or more columns, you’ll
-need to do something akin to what we’ve shown here.</p>
-<h2>Rust UDF with Python wrapper</h2>
-<p>This is the most complicated approach, but has the potential to be the most performant. What we
-will do here is write a Rust function to perform our computation and then expose that function to
-Python. I know of two use cases where I would recommend this approach. The first is the case when
-the PyArrow compute functions are insufficient for your needs. Perhaps your code is too complex or
-could be greatly simplified if you pulled in some outside dependency. The second use case is when
-you have written a UDF that you’re sharing across multiple projects and have hardened the approach.
-It is possible that you can implement your function in Rust to give a speed improvement and then
-every project that is using this shared UDF will benefit from those updates.</p>
-<p>When deciding to use this approach, it’s worth considering how much you think you’ll actually
-benefit from the Rust implementation to decide if it’s worth the additional effort to maintain and
-deploy the Python wheels you generate. It is certainly not necessary for every use case.</p>
-<p>Due to the excellent work by the Python arrow team, we can simplify our work to needing only two
-dependencies on the Rust side, <a href="https://github.com/apache/arrow-rs">arrow-rs</a> and
-<a href="https://pyo3.rs/">pyo3</a>. I have posted a <a href="https://github.com/timsaucer/tuple_filter_example">minimal example</a>.
-You’ll need <a href="https://github.com/PyO3/maturin">maturin</a> to build the project, and you must use
-release mode when building to get the expected performance.</p>
-<div class="codehilite"><pre><span></span><code>maturin develop --release
-</code></pre></div>
-<p>When you write your UDF in Rust you generally will need to take these steps</p>
-<ol>
-<li>Write a function description that takes in some number of Python generic objects.</li>
-<li>Convert these objects to Arrow Arrays of the appropriate type(s).</li>
-<li>Perform your computation and create a resultant Array.</li>
-<li>Convert the array into a Python generic object.</li>
-</ol>
-<p>For the conversion to and from Python objects, we can take advantage of the
-<code>ArrayData::from_pyarrow_bound</code> and <code>ArrayData::to_pyarrow</code> functions. All that remains is to
-perform your computation.</p>
-<p>We are going to demonstrate doing this computation in two ways. The first is to mimic what we’ve
-done in the above approach using PyArrow. In the second we demonstrate iterating through the three
-arrays ourselves.</p>
-<p>In our first approach, we can expect the performance to be nearly identical to when we used the
-PyArrow compute functions. On the Rust side we will have slightly less overhead but the heavy
-lifting portions of the code are essentially the same between this Rust implementation and the
-PyArrow approach above.</p>
-<p>The reason for demonstrating this, even though it doesn’t provide a significant speedup over
-Python, is to primarily demonstrate how to make the Python to Rust with Python wrapper
-transition. In the second implementation you can see how we can iterate through all of the arrays
-ourselves.</p>
-<p>In this first example, we are hard coding the values of interest, but in the following section
-we demonstrate passing these in during initalization.</p>
-<div class="codehilite"><pre><span></span><code><span class="cp">#[pyfunction]</span><span class="w"></span>
-<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">tuple_filter_fn</span><span class="p">(</span><span class="w"></span>
-<span class="w"> </span><span class="n">py</span>: <span class="nc">Python</span><span class="o"><</span><span class="nb">'_</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">partkey_expr</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyAny</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">suppkey_expr</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyAny</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">returnflag_expr</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyAny</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="p">)</span><span class="w"> </span>-> <span class="nc">PyResult</span><span class="o"><</span><span class="n">Py</span><span class="o"><</span><span class="n">PyAny</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">partkey_arr</span>: <span class="nc">PrimitiveArray</span><span class="o"><</span><span class="n">Int64Type</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">ArrayData</span>::<span class="n">from_pyarrow_bound</span><span class="p">(</span><span class="n">partkey_expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">into</span><span class="p">();</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">suppkey_arr</span>: <span class="nc">PrimitiveArray</span><span class="o"><</span><span class="n">Int64Type</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">ArrayData</span>::<span class="n">from_pyarrow_bound</span><span class="p">(</span><span class="n">suppkey_expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">into</span><span class="p">();</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">returnflag_arr</span>: <span class="nc">StringArray</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ArrayData</span>::<span class="n">from_pyarrow_bound</span><span class="p">(</span><span class="n">returnflag_expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">into</span><span class="p">();</span><span [...]
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">values_of_interest</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">vec</span><span class="o">!</span><span class="p">[</span><span class="w"></span>
-<span class="w"> </span><span class="p">(</span><span class="mi">1530</span><span class="p">,</span><span class="w"> </span><span class="mi">4031</span><span class="p">,</span><span class="w"> </span><span class="s">"N"</span><span class="p">.</span><span class="n">to_string</span><span class="p">()),</span><span class="w"></span>
-<span class="w"> </span><span class="p">(</span><span class="mi">6530</span><span class="p">,</span><span class="w"> </span><span class="mi">1531</span><span class="p">,</span><span class="w"> </span><span class="s">"N"</span><span class="p">.</span><span class="n">to_string</span><span class="p">()),</span><span class="w"></span>
-<span class="w"> </span><span class="p">(</span><span class="mi">5618</span><span class="p">,</span><span class="w"> </span><span class="mi">619</span><span class="p">,</span><span class="w"> </span><span class="s">"N"</span><span class="p">.</span><span class="n">to_string</span><span class="p">()),</span><span class="w"></span>
-<span class="w"> </span><span class="p">(</span><span class="mi">8118</span><span class="p">,</span><span class="w"> </span><span class="mi">8119</span><span class="p">,</span><span class="w"> </span><span class="s">"N"</span><span class="p">.</span><span class="n">to_string</span><span class="p">()),</span><span class="w"></span>
-<span class="w"> </span><span class="p">];</span><span class="w"></span>
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">res</span>: <span class="nb">Option</span><span class="o"><</span><span class="n">BooleanArray</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">None</span><span class="p">;</span><span class="w"></span>
-
-<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">partkey</span><span class="p">,</span><span class="w"> </span><span class="n">suppkey</span><span class="p">,</span><span class="w"> </span><span class="n">returnflag</span><span class="p">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">&</span><span class="n">values_of_interest</span><span class="w"> </span><span class= [...]
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">filtered_partkey_arr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BooleanArray</span>::<span class="n">from_unary</span><span class="p">(</span><span class="o">&</span><span class="n">partkey_arr</span><span class="p">,</span><span class="w"> </span><span class="o">|</span><span class="n">p</span><span class="o">|</span><span class="w"> </ [...]
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">filtered_suppkey_arr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BooleanArray</span>::<span class="n">from_unary</span><span class="p">(</span><span class="o">&</span><span class="n">suppkey_arr</span><span class="p">,</span><span class="w"> </span><span class="o">|</span><span class="n">s</span><span class="o">|</span><span class="w"> </ [...]
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">filtered_returnflag_arr</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">BooleanArray</span>::<span class="n">from_unary</span><span class="p">(</span><span class="o">&</span><span class="n">returnflag_arr</span><span class="p">,</span><span class="w"> </span><span class="o">|</span><span class="n">s</span><span class="o">|</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">returnflag</span><span class="p">);</span [...]
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">part_and_supp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">compute</span>::<span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">filtered_partkey_arr</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">filtered_suppkey_arr</span><span class="p">)</span><span [...]
-<span class="w"> </span><span class="p">.</span><span class="n">map_err</span><span class="p">(</span><span class="o">|</span><span class="n">e</span><span class="o">|</span><span class="w"> </span><span class="n">PyValueError</span>::<span class="n">new_err</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">to_string</span><span class="p">()))</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">resultant_arr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">compute</span>::<span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">part_and_supp</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">filtered_returnflag_arr</span><span class="p">)</span><span clas [...]
-<span class="w"> </span><span class="p">.</span><span class="n">map_err</span><span class="p">(</span><span class="o">|</span><span class="n">e</span><span class="o">|</span><span class="w"> </span><span class="n">PyValueError</span>::<span class="n">new_err</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">to_string</span><span class="p">()))</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
-
-<span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">r</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">compute</span>::<span class="n">or</span><span class="p">(</span><span class="o">&</span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">resultant_arr</span><span class="p">).</span><span class [...]
-<span class="w"> </span><span class="nb">None</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">resultant_arr</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="p">};</span><span class="w"></span>
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-
-<span class="w"> </span><span class="n">res</span><span class="p">.</span><span class="n">unwrap</span><span class="p">().</span><span class="n">into_data</span><span class="p">().</span><span class="n">to_pyarrow</span><span class="p">(</span><span class="n">py</span><span class="p">)</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-
-
-<span class="cp">#[pymodule]</span><span class="w"></span>
-<span class="k">fn</span> <span class="nf">tuple_filter_example</span><span class="p">(</span><span class="n">module</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyModule</span><span class="o">></span><span class="p">)</span><span class="w"> </span>-> <span class="nc">PyResult</span><span class="o"><</span><span class="p">()</span><span class= [...]
-<span class="w"> </span><span class="n">module</span><span class="p">.</span><span class="n">add_function</span><span class="p">(</span><span class="n">wrap_pyfunction</span><span class="o">!</span><span class="p">(</span><span class="n">tuple_filter_fn</span><span class="p">,</span><span class="w"> </span><span class="n">module</span><span class="p">)</span><span class="o">?</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
-<span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-</code></pre></div>
-<p>To use this we use the <code>udf</code> function in <code>datafusion-python</code> just as before.</p>
-<div class="codehilite"><pre><span></span><code><span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">udf</span>
-<span class="kn">import</span> <span class="nn">pyarrow</span> <span class="k">as</span> <span class="nn">pa</span>
-<span class="kn">from</span> <span class="nn">tuple_filter_example</span> <span class="kn">import</span> <span class="n">tuple_filter_fn</span>
-
-<span class="n">udf_using_custom_rust_fn</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span>
- <span class="n">tuple_filter_fn</span><span class="p">,</span>
- <span class="p">[</span><span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p">(),</span> <span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p">(),</span> <span class="n">pa</span><span class="o">.</span><span class="n">utf8</span><span class="p">()],</span>
- <span class="n">pa</span><span class="o">.</span><span class="n">bool_</span><span class="p">(),</span>
- <span class="s2">"stable"</span><span class="p">,</span>
-<span class="p">)</span>
-</code></pre></div>
-<p>That's it! We've now got a third party Rust UDF with Python wrappers working with DataFusion's
-Python bindings!</p>
-<h3>Rust UDF with initialization</h3>
-<p>Looking at the code above, you can see that it is hard coding the values we're interested in. There
-are many types of UDFs that don't require any additional data provided to them before they start
-the computation. The code above is sloppy, so let's clean it up.</p>
-<p>We want to write the function to take some additional data. A limitation of the UDFs we create is
-that they expect to operate on entire arrays of data at a time. We can get around this problem by
-creating an initializer for our UDF. We do this by defining a Rust struct that contains the data we
-need and implement two methods on this struct, <code>new</code> and <code>__call__</code>. By doing this we will create a
-Python object that is callable, so it can be the function we provide to <code>udf</code>.</p>
-<div class="codehilite"><pre><span></span><code><span class="cp">#[pyclass]</span><span class="w"></span>
-<span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">TupleFilterClass</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="n">values_of_interest</span>: <span class="nb">Vec</span><span class="o"><</span><span class="p">(</span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="nb">String</span><span class="p">)</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-
-<span class="cp">#[pymethods]</span><span class="w"></span>
-<span class="k">impl</span><span class="w"> </span><span class="n">TupleFilterClass</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="cp">#[new]</span><span class="w"></span>
-<span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">values_of_interest</span>: <span class="nb">Vec</span><span class="o"><</span><span class="p">(</span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="nb">String</span><span class="p">)</span><span class="o">></span><span class="p">)</span><span class= [...]
-<span class="w"> </span><span class="bp">Self</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="n">values_of_interest</span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-
-<span class="w"> </span><span class="k">fn</span> <span class="nf">__call__</span><span class="p">(</span><span class="w"></span>
-<span class="w"> </span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">py</span>: <span class="nc">Python</span><span class="o"><</span><span class="nb">'_</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">partkey_expr</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyAny</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">suppkey_expr</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyAny</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">returnflag_expr</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyAny</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="p">)</span><span class="w"> </span>-> <span class="nc">PyResult</span><span class="o"><</span><span class="n">Py</span><span class="o"><</span><span class="n">PyAny</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">partkey_arr</span>: <span class="nc">PrimitiveArray</span><span class="o"><</span><span class="n">Int64Type</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">ArrayData</span>::<span class="n">from_pyarrow_bound</span><span class="p">(</span><span class="n">partkey_expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">into</span><span class="p">();</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">suppkey_arr</span>: <span class="nc">PrimitiveArray</span><span class="o"><</span><span class="n">Int64Type</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">ArrayData</span>::<span class="n">from_pyarrow_bound</span><span class="p">(</span><span class="n">suppkey_expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">into</span><span class="p">();</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">returnflag_arr</span>: <span class="nc">StringArray</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ArrayData</span>::<span class="n">from_pyarrow_bound</span><span class="p">(</span><span class="n">returnflag_expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">into</span><span class="p">();</span><s [...]
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">res</span>: <span class="nb">Option</span><span class="o"><</span><span class="n">BooleanArray</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">None</span><span class="p">;</span><span class="w"></span>
-
-<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">partkey</span><span class="p">,</span><span class="w"> </span><span class="n">suppkey</span><span class="p">,</span><span class="w"> </span><span class="n">returnflag</span><span class="p">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">&</span><span class="bp">self</span><span class="p">.</span><span class="n">value [...]
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">filtered_partkey_arr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BooleanArray</span>::<span class="n">from_unary</span><span class="p">(</span><span class="o">&</span><span class="n">partkey_arr</span><span class="p">,</span><span class="w"> </span><span class="o">|</span><span class="n">p</span><span class="o">|</span><span class="w" [...]
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">filtered_suppkey_arr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BooleanArray</span>::<span class="n">from_unary</span><span class="p">(</span><span class="o">&</span><span class="n">suppkey_arr</span><span class="p">,</span><span class="w"> </span><span class="o">|</span><span class="n">s</span><span class="o">|</span><span class="w" [...]
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">filtered_returnflag_arr</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">BooleanArray</span>::<span class="n">from_unary</span><span class="p">(</span><span class="o">&</span><span class="n">returnflag_arr</span><span class="p">,</span><span class="w"> </span><span class="o">|</span><span class="n">s</span><span class="o">|</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">returnflag</span><span class="p">);</ [...]
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">part_and_supp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">compute</span>::<span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">filtered_partkey_arr</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">filtered_suppkey_arr</span><span class="p">)</span><s [...]
-<span class="w"> </span><span class="p">.</span><span class="n">map_err</span><span class="p">(</span><span class="o">|</span><span class="n">e</span><span class="o">|</span><span class="w"> </span><span class="n">PyValueError</span>::<span class="n">new_err</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">to_string</span><span class="p">()))</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">resultant_arr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">compute</span>::<span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">part_and_supp</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">filtered_returnflag_arr</span><span class="p">)</span><span [...]
-<span class="w"> </span><span class="p">.</span><span class="n">map_err</span><span class="p">(</span><span class="o">|</span><span class="n">e</span><span class="o">|</span><span class="w"> </span><span class="n">PyValueError</span>::<span class="n">new_err</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">to_string</span><span class="p">()))</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
-
-<span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">r</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">compute</span>::<span class="n">or</span><span class="p">(</span><span class="o">&</span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">resultant_arr</span><span class="p">).</span><span c [...]
-<span class="w"> </span><span class="nb">None</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">resultant_arr</span><span class="p">),</span><span class="w"></span>
-<span class="w"> </span><span class="p">};</span><span class="w"></span>
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-
-<span class="w"> </span><span class="n">res</span><span class="p">.</span><span class="n">unwrap</span><span class="p">().</span><span class="n">into_data</span><span class="p">().</span><span class="n">to_pyarrow</span><span class="p">(</span><span class="n">py</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-
-<span class="cp">#[pymodule]</span><span class="w"></span>
-<span class="k">fn</span> <span class="nf">tuple_filter_example</span><span class="p">(</span><span class="n">module</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyModule</span><span class="o">></span><span class="p">)</span><span class="w"> </span>-> <span class="nc">PyResult</span><span class="o"><</span><span class="p">()</span><span class= [...]
-<span class="w"> </span><span class="n">module</span><span class="p">.</span><span class="n">add_class</span>::<span class="o"><</span><span class="n">TupleFilterClass</span><span class="o">></span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
-<span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-</code></pre></div>
-<p>When you write this, you don't have to call your constructor <code>new</code>. The more important part is that
-you have <code>#[new]</code> designated on the function. With this you can provide any kinds of data you need
-during processing. Using this initializer in Python is fairly straightforward.</p>
-<div class="codehilite"><pre><span></span><code><span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">udf</span>
-<span class="kn">import</span> <span class="nn">pyarrow</span> <span class="k">as</span> <span class="nn">pa</span>
-<span class="kn">from</span> <span class="nn">tuple_filter_example</span> <span class="kn">import</span> <span class="n">TupleFilterClass</span>
-
-<span class="n">tuple_filter_class</span> <span class="o">=</span> <span class="n">TupleFilterClass</span><span class="p">(</span><span class="n">values_of_interest</span><span class="p">)</span>
-
-<span class="n">udf_using_custom_rust_fn_with_data</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span>
- <span class="n">tuple_filter_class</span><span class="p">,</span>
- <span class="p">[</span><span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p">(),</span> <span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p">(),</span> <span class="n">pa</span><span class="o">.</span><span class="n">utf8</span><span class="p">()],</span>
- <span class="n">pa</span><span class="o">.</span><span class="n">bool_</span><span class="p">(),</span>
- <span class="s2">"stable"</span><span class="p">,</span>
- <span class="n">name</span><span class="o">=</span><span class="s2">"tuple_filter_with_data"</span>
-<span class="p">)</span>
-</code></pre></div>
-<p>When you use this approach you will need to provide a <code>name</code> argument to <code>udf</code>. This is because our
-class/struct does not get the <code>__qualname__</code> attribute that the <code>udf</code> function is looking for. You
-can give this udf any name you choose.</p>
-<h3>Rust UDF with direct iteration</h3>
-<p>The final version of our scalar UDF is one where we implement it in Rust and iterate through all of
-the arrays ourselves. If you are iterating through more than 3 arrays at a time I recommend looking
-at <a href="https://docs.rs/itertools/latest/itertools/macro.izip.html">izip</a> in the
-<a href="https://crates.io/crates/itertools">itertools crate</a>. For ease of understanding and since we only
-have 3 arrays here I will just explicitly create my own tuple here.</p>
-<div class="codehilite"><pre><span></span><code><span class="cp">#[pyclass]</span><span class="w"></span>
-<span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">TupleFilterDirectIterationClass</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="n">values_of_interest</span>: <span class="nb">Vec</span><span class="o"><</span><span class="p">(</span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="nb">String</span><span class="p">)</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-
-<span class="cp">#[pymethods]</span><span class="w"></span>
-<span class="k">impl</span><span class="w"> </span><span class="n">TupleFilterDirectIterationClass</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="cp">#[new]</span><span class="w"></span>
-<span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">values_of_interest</span>: <span class="nb">Vec</span><span class="o"><</span><span class="p">(</span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="nb">String</span><span class="p">)</span><span class="o">></span><span class="p">)</span><span class= [...]
-<span class="w"> </span><span class="bp">Self</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">values_of_interest</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-
-<span class="w"> </span><span class="k">fn</span> <span class="nf">__call__</span><span class="p">(</span><span class="w"></span>
-<span class="w"> </span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">py</span>: <span class="nc">Python</span><span class="o"><</span><span class="nb">'_</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">partkey_expr</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyAny</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">suppkey_expr</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyAny</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">returnflag_expr</span>: <span class="kp">&</span><span class="nc">Bound</span><span class="o"><</span><span class="nb">'_</span><span class="p">,</span><span class="w"> </span><span class="n">PyAny</span><span class="o">></span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="p">)</span><span class="w"> </span>-> <span class="nc">PyResult</span><span class="o"><</span><span class="n">Py</span><span class="o"><</span><span class="n">PyAny</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">partkey_arr</span>: <span class="nc">PrimitiveArray</span><span class="o"><</span><span class="n">Int64Type</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">ArrayData</span>::<span class="n">from_pyarrow_bound</span><span class="p">(</span><span class="n">partkey_expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">into</span><span class="p">();</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">suppkey_arr</span>: <span class="nc">PrimitiveArray</span><span class="o"><</span><span class="n">Int64Type</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">ArrayData</span>::<span class="n">from_pyarrow_bound</span><span class="p">(</span><span class="n">suppkey_expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">into</span><span class="p">();</span><span class="w"></span>
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">returnflag_arr</span>: <span class="nc">StringArray</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ArrayData</span>::<span class="n">from_pyarrow_bound</span><span class="p">(</span><span class="n">returnflag_expr</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="n">into</span><span class="p">();</span><s [...]
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">values_to_search</span>: <span class="nb">Vec</span><span class="o"><</span><span class="p">(</span><span class="o">&</span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="kt">i64</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="kt">str</span><span class="p">)</span><span [...]
-<span class="w"> </span><span class="p">.</span><span class="n">iter</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="w"> </span [...]
-<span class="w"> </span><span class="p">.</span><span class="n">collect</span><span class="p">();</span><span class="w"></span>
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">values</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">partkey_arr</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">values</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">iter</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">zip</span><span class="p">(</span><span class="n">suppkey_arr</span><span class="p">.</span><span class="n">values</span><span class="p">().</span><span class="n">iter</span><span class="p">())</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">zip</span><span class="p">(</span><span class="n">returnflag_arr</span><span class="p">.</span><span class="n">iter</span><span class="p">())</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="p">((</span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="p">),</span><span class="w"> </span><span class="n">c</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="w"> </sp [...]
-<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="n">v</span><span class="o">|</span><span class="w"> </span><span class="n">values_to_search</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="o">&</span><span class="n">v</span><span class="p">));</span><span class="w"></span>
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">res</span>: <span class="nc">BooleanArray</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BooleanBuffer</span>::<span class="n">from_iter</span><span class="p">(</span><span class="n">values</span><span class="p">).</span><span class="n">into</span><span class="p">();</span><span class="w"></span>
-
-<span class="w"> </span><span class="n">res</span><span class="p">.</span><span class="n">into_data</span><span class="p">().</span><span class="n">to_pyarrow</span><span class="p">(</span><span class="n">py</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-</code></pre></div>
-<p>We convert the <code>values_of_interest</code> into a vector of borrowed types so that we can do a fast search
-without creating additional memory. The other option is to turn the <code>returnflag</code> into a <code>String</code>
-but that memory allocation is unnecessary. After that we use two <code>zip</code> operations so that we can
-iterate over all three columns in a single pass. Since each <code>zip</code> will return a tuple of two
-elements, a quick <code>map</code> turns them into the tuple format we need. Also, <code>StringArray</code> is a little
-different in the buffer it uses, so it is treated slightly differently from the others.</p>
-<h2>User Defined Aggregate Function</h2>
-<p>Writing a user defined aggregate function or user defined window function is slightly more complex
-than scalar functions. This is because we must accumulate values and there is no guarantee that one
-batch will contain all the values we are aggregating over. For this we need to define an
-<code>Accumulator</code> which will do a few things.</p>
-<ul>
-<li>Process a batch and compute an internal state</li>
-<li>Share the state so that we can combine multiple batches</li>
-<li>Merge the results across multiple batches</li>
-<li>Return the final result</li>
-</ul>
-<p>In the example below, we're going to look at customer orders and we want to know per customer ID,
-how much they have ordered total. We want to ignore small orders, which we define as anything under
-5000.</p>
-<div class="codehilite"><pre><span></span><code><span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">Accumulator</span><span class="p">,</span> <span class="n">udaf</span>
-<span class="kn">import</span> <span class="nn">pyarrow</span> <span class="k">as</span> <span class="nn">pa</span>
-<span class="kn">import</span> <span class="nn">pyarrow.compute</span> <span class="k">as</span> <span class="nn">pc</span>
-
-<span class="n">IGNORE_THESHOLD</span> <span class="o">=</span> <span class="mf">5000.0</span>
-<span class="k">class</span> <span class="nc">AboveThresholdAccum</span><span class="p">(</span><span class="n">Accumulator</span><span class="p">):</span>
- <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span class="o">=</span> <span class="mf">0.0</span>
-
- <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">values</span><span class="p">:</span> <span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">over_threshold</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="n">greater</span><span class="p">(</span><span class="n">values</span><span class="p">,</span> <span class="n">pa</span><span class="o">.</span><span class="n">scalar</span><span class="p">(</span><span class="n">IGNORE_THESHOLD</span><span class="p">))</span>
- <span class="n">sum_above</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">values</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">over_threshold</span><span class="p">))</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
- <span class="k">if</span> <span class="n">sum_above</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">sum_above</span> <span class="o">=</span> <span class="mf">0.0</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span class="o">+</span> <span class="n">sum_above</span>
-
- <span class="k">def</span> <span class="nf">merge</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">states</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="n">pa</span><span class="o">.</span><span class="n">Array</span><span class="p">])</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span class="o">+</span> <span class="n">pc</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">states</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
-
- <span class="k">def</span> <span class="nf">state</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="n">pa</span><span class="o">.</span><span class="n">Scalar</span><span class="p">]:</span>
- <span class="k">return</span> <span class="p">[</span><span class="n">pa</span><span class="o">.</span><span class="n">scalar</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_sum</span><span class="p">)]</span>
-
- <span class="k">def</span> <span class="nf">evaluate</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">pa</span><span class="o">.</span><span class="n">Scalar</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">pa</span><span class="o">.</span><span class="n">scalar</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_sum</span><span class="p">)</span>
-
-<span class="n">sum_above_threshold</span> <span class="o">=</span> <span class="n">udaf</span><span class="p">(</span><span class="n">AboveThresholdAccum</span><span class="p">,</span> <span class="p">[</span><span class="n">pa</span><span class="o">.</span><span class="n">float64</span><span class="p">()],</span> <span class="n">pa</span><span class="o">.</span><span class="n">float64</span><span class="p">(),</span> <span class="p">[</span><span class="n">pa</span><span class="o">.</s [...]
-
-<span class="n">df_orders</span><span class="o">.</span><span class="n">aggregate</span><span class="p">([</span><span class="n">col</span><span class="p">(</span><span class="s2">"o_custkey"</span><span class="p">)],[</span><span class="n">sum_above_threshold</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"o_totalprice"</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class= [...]
-</code></pre></div>
-<p>Since we are doing a <code>sum</code> we can keep a single value as our internal state. When we call <code>update()</code>
-we will process a single array and update the internal state, which we share with the <code>state()</code>
-function. For larger batches we may <code>merge()</code> these states. It is important to note that the
-<code>states</code> in the <code>merge()</code> function are an array of the values returned from <code>state()</code>. It is
-entirely possible that the <code>merge</code> function is significantly different than the <code>update</code>, though in
-our example they are very similar.</p>
-<p>One example of implementing a user defined aggregate function where the <code>update()</code> and <code>merge()</code>
-operations are different is computing an average. In <code>update()</code> we would create a state that is both
-a sum and a count. <code>state()</code> would return a list of these two values, and <code>merge()</code> would compute
-the final result.</p>
-<h2>User Defined Window Functions</h2>
-<p>Writing a user defined window function is slightly more complex than an aggregate function due
-to the variety of ways that window functions are called. I recommend reviewing the
-<a href="https://datafusion.apache.org/python/user-guide/common-operations/udf-and-udfa.html">online documentation</a>
-for a description of which functions need to be implemented. The details of how to implement
-these generally follow the same patterns as described above for aggregate functions.</p>
-<h2>Performance Comparison</h2>
-<p>For the scalar functions above, we performed a timing evaluation, repeating the operation 100
-times. For this simple example these are our results.</p>
-<div class="codehilite"><pre><span></span><code><span class="err">+-----------------------------+--------------+---------+</span>
-<span class="err">| approach | Average Time | Std Dev |</span>
-<span class="err">+-----------------------------+--------------+---------+</span>
-<span class="err">| python udf | 4.969 | 0.062 |</span>
-<span class="err">| simple filter | 1.075 | 0.022 |</span>
-<span class="err">| explicit filter | 0.685 | 0.063 |</span>
-<span class="err">| pyarrow compute | 0.529 | 0.017 |</span>
-<span class="err">| arrow rust compute | 0.511 | 0.034 |</span>
-<span class="err">| arrow rust compute as class | 0.502 | 0.011 |</span>
-<span class="err">| rust custom iterator | 0.478 | 0.009 |</span>
-<span class="err">+-----------------------------+--------------+---------+</span>
-</code></pre></div>
-<p>As expected, the conversion to Python objects is by far the worst performance. As soon as we drop
-into using any functions that keep the data entirely on the Native (Rust or C/C++) side we see a
-near 10x speed improvement. Then as we increase our complexity from using PyArrow compute functions
-to implementing the UDF in Rust we see incremental improvements. Our fastest approach - iterating
-through the arrays ourselves does operate nearly 10% faster than the PyArrow compute approach.</p>
-<h2>Final Thoughts and Recommendations</h2>
-<p>For anyone who is curious about <a href="https://datafusion.apache.org/">DataFusion</a> I highly recommend
-giving it a try. This post was designed to make it easier for new users to the Python implementation
-to work with User Defined Functions by giving a few examples of how one might implement these.</p>
-<p>When it comes to designing UDFs, I strongly recommend seeing if you can write your UDF using
-<a href="https://arrow.apache.org/docs/python/api/compute.html">PyArrow functions</a> rather than pure Python
-objects. As shown in the scalar example above, you can achieve a 10x speedup by using PyArrow
-functions. If you must do something that isn't well represented by the PyArrow compute functions,
-then I would consider using a Rust based UDF in the manner shown above.</p>
-<p>I would like to thank <a href="https://github.com/alamb">@alamb</a>, <a href="https://github.com/andygrove">@andygrove</a>, <a href="https://github.com/comphead">@comphead</a>, <a href="https://github.com/emgeee">@emgeee</a>, <a href="https://github.com/kylebarron">@kylebarron</a>, and <a href="https://github.com/Omega359">@Omega359</a>
-for their helpful reviews and feedback.</p>
-<p>Lastly, the Apache Arrow and DataFusion community is an active group of very helpful people working
-to make a great tool. If you want to get involved, please take a look at the
-<a href="https://datafusion.apache.org/python/">online documentation</a> and jump in to help with one of the
-<a href="https://github.com/apache/datafusion-python/issues">open issues</a>.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/2024/11/20/datafusion-comet-0.4.0/index.html b/output/2024/11/20/datafusion-comet-0.4.0/index.html
deleted file mode 100644
index 41ffebe..0000000
--- a/output/2024/11/20/datafusion-comet-0.4.0/index.html
+++ /dev/null
@@ -1,138 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.4.0 Release - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- Apache DataFusion Comet 0.4.0 Release
- </h1>
- <p>Posted on: Wed 20 November 2024 by pmc</p>
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately six weeks of development work and is the result of merging 51 PRs from 10
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.4.0.md">change log</a> for more information.</p>
-<h2>Release Highlights</h2>
-<h3>Performance & Stability</h3>
-<p>There are a number of performance and stability improvements in this release. Here is a summary of some of the
-larger changes. Current benchmarking results can be found in the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>.</p>
-<h4>Unified Memory Management</h4>
-<p>Comet now uses a unified memory management approach that shares an off-heap memory pool with Apache Spark, resulting
-in a much simpler configuration. Comet now requires <code>spark.memory.offHeap.enabled=true</code>. This approach provides a
-holistic view of memory usage in Spark and Comet and makes it easier to optimize system performance.</p>
-<h4>Faster Joins</h4>
-<p>Apache Spark supports sort-merge and hash joins, which have similar performance characteristics. Spark defaults to
-using sort-merge joins because they are less likely to result in OutOfMemory exceptions. In vectorized query
-engines such as DataFusion, hash joins outperform sort-merge joins. Comet now has an experimental feature to
-replace Spark sort-merge joins with hash joins for improved performance. This feature is experimental because
-there is currently no spill-to-disk support in the hash join implementation. This feature can be enabled by
-setting <code>spark.comet.exec.replaceSortMergeJoin=true</code>.</p>
-<h4>Bloom Filter Aggregates</h4>
-<p>Spark’s optimizer can insert Bloom filter aggregations and filters to prune large result sets before a shuffle. However,
-Comet would fall back to Spark for the aggregation. Comet now has native support for Bloom filter aggregations
-after previously supporting Bloom filter testing. Users no longer need to set
-<code>spark.sql.optimizer.runtime.bloomFilter.enabled=false</code> when using Comet.</p>
-<h4>Complex Type support</h4>
-<p>This release has the following improvements to complex type support:</p>
-<ul>
-<li>Implemented <code>ArrayAppend</code> and <code>GetArrayStructFields</code>.</li>
-<li>Implemented native cast between structs</li>
-<li>Implemented native cast from structs to string</li>
-</ul>
-<h2>Roadmap</h2>
-<p>One of the highest priority items on the roadmap is to add support for reading complex types (maps, structs, and arrays)
-from Parquet sources, both when reading Parquet directly and from Iceberg.</p>
-<p>Comet currently has proprietary native code for decoding Parquet pages, native column readers for all of Spark’s
-primitive types, and special handling for Spark-specific use cases such as timestamp rebasing and decimal type
-promotion. This implementation does not yet support complex types. File IO, decryption, and decompression are handled
-in JVM code, and Parquet pages are passed on to native code for decoding.</p>
-<p>Rather than add complex type support to this existing code, we are exploring two main options to allow us to
-leverage more of the upstream Arrow and DataFusion code.</p>
-<h3>Use DataFusion’s ParquetExec</h3>
-<p>For use cases where DataFusion can support reading a Parquet source, Comet could create a native plan that uses
-DataFusion’s ParquetExec. We are investigating using DataFusion’s SchemaAdapter to handle some Spark-specific
-handling of timestamps and decimals.</p>
-<h3>Use Arrow’s Parquet Batch Reader</h3>
-<p>For use cases not supported by DataFusion’s ParquetExec, such as integrating with Iceberg, we are exploring
-replacing our current native Parquet decoding logic with the Arrow readers provided by the Parquet crate.</p>
-<p>Iceberg already provides a vectorized Spark reader for Parquet. A <a href="https://github.com/apache/iceberg/pull/9841">PR</a> is open against Iceberg for adding a native
-version based on Comet, and we hope to update this to leverage the improvements outlined above.</p>
-<h2>Getting Involved</h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/output/about.html b/output/about.html
deleted file mode 100644
index 47c3af4..0000000
--- a/output/about.html
+++ /dev/null
@@ -1,64 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>About - Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
- <div>
-
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
-
-<!-- page contents -->
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
- <h1>
- About
- </h1>
- <p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is a very fast, extensible query engine for building high-quality data-centric systems in Rust, using the Apache Arrow in-memory format.</p>
-<p>To update this site, please propose a change to the <a href="https://github.com/apache/datafusion-site">source code</a>.</p>
- </div>
- </div>
- </div>
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </div>
- </main>
- </body>
-</html>
diff --git a/output/author/agrove.html b/output/author/agrove.html
deleted file mode 100644
index 7c27cea..0000000
--- a/output/author/agrove.html
+++ /dev/null
@@ -1,147 +0,0 @@
- <!doctype html>
- <html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> <link href="/css/blog_index.css" rel="stylesheet">
- </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
- <div>
-
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
- <div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-
- <h3>Welcome to the Apache DataFusion Blog!</h3>
- <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
-
-
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2021/04/12/ballista-donation">Ballista: A Distributed Scheduler for Apache Arrow</a></h1>
- <p>Posted on: Mon 12 April 2021 by agrove</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
-to the Apache Arrow project. </p>
-<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2021/04/12/ballista-donation" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2019/02/04/datafusion-donation">DataFusion: A Rust-native Query Engine for Apache Arrow</a></h1>
- <p>Posted on: Mon 04 February 2019 by agrove</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2019/02/04/datafusion-donation" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
-
- </div>
- </div>
-</div> </div>
- </div>
- </div>
-
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </div>
- </main>
- </body>
- </html>
diff --git a/output/author/alamb-dandandan-tustvold.html b/output/author/alamb-dandandan-tustvold.html
deleted file mode 100644
index c79588a..0000000
--- a/output/author/alamb-dandandan-tustvold.html
+++ /dev/null
@@ -1,111 +0,0 @@
- <!doctype html>
- <html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> <link href="/css/blog_index.css" rel="stylesheet">
- </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
- <div>
-
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
- <div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-
- <h3>Welcome to the Apache DataFusion Blog!</h3>
- <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
-
-
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2023/08/05/datafusion_fast_grouping">Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</a></h1>
- <p>Posted on: Sat 05 August 2023 by alamb, Dandandan, tustvold</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- Converted from Google Docs using https://www.buymeacoffee.com/docstomarkdown -->
-<h2>Aggregating Millions of Groups Fast in Apache Arrow DataFusion</h2>
-<p>Andrew Lamb, Daniël Heres, Raphael Taylor-Davies,</p>
-<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
-<h2>TLDR</h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2023/08/05/datafusion_fast_grouping" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
-
- </div>
- </div>
-</div> </div>
- </div>
- </div>
-
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </div>
- </main>
- </body>
- </html>
diff --git a/output/author/andrew-lamb-staff-engineer-at-influxdata.html b/output/author/andrew-lamb-staff-engineer-at-influxdata.html
deleted file mode 100644
index 6d9c003..0000000
--- a/output/author/andrew-lamb-staff-engineer-at-influxdata.html
+++ /dev/null
@@ -1,110 +0,0 @@
- <!doctype html>
- <html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> <link href="/css/blog_index.css" rel="stylesheet">
- </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
- <div>
-
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
- <div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-
- <h3>Welcome to the Apache DataFusion Blog!</h3>
- <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
-
-
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench">Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</a></h1>
- <p>Posted on: Mon 18 November 2024 by Andrew Lamb, Staff Engineer at InfluxData</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>I am extremely excited to announce that <a href="https://crates.io/crates/datafusion">Apache DataFusion</a> is the
-fastest engine for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">ClickBench</a>. It is faster
-than <a href="https://duckdb.org/">DuckDB</a>, <a href="https://clickhouse.com/chdb">chDB</a> and <a href="https://clickhouse.com/">Clickhouse</a> using the same hardware. It also marks
-the first time a <a href="https://www.rust-lang.org/">Rust</a>-based engine holds the top spot, which has previously
-been …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
-
- </div>
- </div>
-</div> </div>
- </div>
- </div>
-
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </div>
- </main>
- </body>
- </html>
diff --git a/output/author/pmc.html b/output/author/pmc.html
deleted file mode 100644
index c5a8d65..0000000
--- a/output/author/pmc.html
+++ /dev/null
@@ -1,810 +0,0 @@
- <!doctype html>
- <html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> <link href="/css/blog_index.css" rel="stylesheet">
- </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
- <div>
-
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
- <div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-
- <h3>Welcome to the Apache DataFusion Blog!</h3>
- <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
-
-
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/11/20/datafusion-comet-0.4.0">Apache DataFusion Comet 0.4.0 Release</a></h1>
- <p>Posted on: Wed 20 November 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/11/20/datafusion-comet-0.4.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/09/27/datafusion-comet-0.3.0">Apache DataFusion Comet 0.3.0 Release</a></h1>
- <p>Posted on: Fri 27 September 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.3.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/09/27/datafusion-comet-0.3.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/08/28/datafusion-comet-0.2.0">Apache DataFusion Comet 0.2.0 Release</a></h1>
- <p>Posted on: Wed 28 August 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.2.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/08/28/datafusion-comet-0.2.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/07/24/datafusion-40.0.0">Apache DataFusion 40.0.0 Released</a></h1>
- <p>Posted on: Wed 24 July 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- see https://github.com/apache/datafusion/issues/9602 for details -->
-<h2>Introduction</h2>
-<p>We are proud to announce <a href="https://crates.io/crates/datafusion/40.0.0">DataFusion 40.0.0</a>. This blog highlights some of the
-many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0/">DataFusion 34.0.0</a> and a preview of
-what the community is thinking about in the next 6 months. We are hoping to make
-more regular blog posts …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/07/24/datafusion-40.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/07/20/datafusion-comet-0.1.0">Apache DataFusion Comet 0.1.0 Release</a></h1>
- <p>Posted on: Sat 20 July 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce the first official source release of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/07/20/datafusion-comet-0.1.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/05/07/datafusion-tlp">Announcing Apache Arrow DataFusion is now Apache DataFusion</a></h1>
- <p>Posted on: Tue 07 May 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>TLDR; <a href="https://arrow.apache.org/">Apache Arrow</a> DataFusion --> <a href="https://datafusion.apache.org/">Apache DataFusion</a></p>
-<p>The Arrow PMC and newly created DataFusion PMC are happy to announce that as of
-April 16, 2024 the Apache Arrow DataFusion subproject is now a top level
-<a href="https://www.apache.org/">Apache Software Foundation</a> project.</p>
-<h2>Background</h2>
-<p>Apache DataFusion is a fast, extensible query engine for building …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/05/07/datafusion-tlp" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/03/06/comet-donation">Announcing Apache Arrow DataFusion Comet</a></h1>
- <p>Posted on: Wed 06 March 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p>The Apache Arrow PMC is pleased to announce the donation of the <a href="https://github.com/apache/arrow-datafusion-comet">Comet project</a>,
-a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/datafusion">Apache Arrow DataFusion</a>.</p>
-<p>Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to
-accelerate Spark workloads. It is designed as a drop-in
-replacement for Spark's JVM …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/03/06/comet-donation" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/01/19/datafusion-34.0.0">Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</a></h1>
- <p>Posted on: Fri 19 January 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>We recently <a href="https://crates.io/crates/datafusion/34.0.0">released DataFusion 34.0.0</a>. This blog highlights some of the major
-improvements since we <a href="https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/.">released DataFusion 26.0.0</a> (spoiler alert there are many)
-and a preview of where the community plans to focus in the next 6 months.</p>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/01/19/datafusion-34.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2023/06/24/datafusion-25.0.0">Apache Arrow DataFusion 26.0.0</a></h1>
- <p>Posted on: Sat 24 June 2023 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>It has been a whirlwind 6 months of DataFusion development since <a href="https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0">our
-last update</a>: the community has grown, many features have been added,
-performance improved and we are <a href="https://github.com/apache/arrow-datafusion/discussions/6475">discussing</a> branching out to our own
-top level Apache Project.</p>
-<h2>Background</h2>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database
-toolkit …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2023/06/24/datafusion-25.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2023/01/19/datafusion-16.0.0">Apache Arrow DataFusion 16.0.0 Project Update</a></h1>
- <p>Posted on: Thu 19 January 2023 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible
-query execution framework, written in <a href="https://www.rust-lang.org/">Rust</a>,
-that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its
-in-memory format. It is targeted primarily at developers creating data
-intensive analytics, and offers mature
-<a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>,
-a DataFrame API, and many extension points.</p>
-<p>Systems based on DataFusion perform very well in benchmarks …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2023/01/19/datafusion-16.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/10/28/ballista-0.9.0">Apache Arrow Ballista 0.9.0 Release</a></h1>
- <p>Posted on: Fri 28 October 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://github.com/apache/arrow-ballista">Ballista</a> is an Arrow-native distributed SQL query engine implemented in Rust.</p>
-<p>Ballista 0.9.0 is now available and is the most significant release since the project was <a href="http://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to Apache
-Arrow in 2021.</p>
-<p>This release represents 4 weeks of work, with 66 commits from 14 contributors:</p>
-<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="mi">22</span><span class="w"> </span><span class="n">Andy …</span></code></pre></div></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/10/28/ballista-0.9.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/10/25/datafusion-13.0.0">Apache Arrow DataFusion 13.0.0 Project Update</a></h1>
- <p>Posted on: Tue 25 October 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> <a href="https://crates.io/crates/datafusion"><code>13.0.0</code></a> is released, and this blog contains an update on the project for the 5 months since our <a href="https://arrow.apache.org/blog/2022/05/16/datafusion-8.0.0/">last update in May 2022</a>.</p>
-<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/10/25/datafusion-13.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/05/16/datafusion-8.0.0">Apache Arrow DataFusion 8.0.0 Release</a></h1>
- <p>Posted on: Mon 16 May 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that
-uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>,
-a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/05/16/datafusion-8.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/03/21/datafusion-contrib">Introducing Apache Arrow DataFusion Contrib</a></h1>
- <p>Posted on: Mon 21 March 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p>Apache Arrow <a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/03/21/datafusion-contrib" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/02/28/datafusion-7.0.0">Apache Arrow DataFusion 7.0.0 Release</a></h1>
- <p>Posted on: Mon 28 February 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/02/28/datafusion-7.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md">Apache Arrow DataFusion 6.0.0 Release</a></h1>
- <p>Posted on: Fri 19 November 2021 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an embedded
-query engine which leverages the unique features of
-<a href="https://www.rust-lang.org/">Rust</a> and <a href="https://arrow.apache.org/">Apache
-Arrow</a> to provide a system that is high
-performance, easy to connect, easy to embed, and high quality.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2021/08/18/ballista-0.5.0">Apache Arrow Ballista 0.5.0 Release</a></h1>
- <p>Posted on: Wed 18 August 2021 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>Ballista extends DataFusion to provide support for distributed queries. This is the first release of Ballista since
-the project was <a href="https://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to the Apache Arrow project
-and includes 80 commits from 11 contributors.</p>
-<div class="codehilite"><pre><span></span><code><span class="err">git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust …</span></code></pre></div></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2021/08/18/ballista-0.5.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2021/08/18/datafusion-5.0.0">Apache Arrow DataFusion 5.0.0 Release</a></h1>
- <p>Posted on: Wed 18 August 2021 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache Arrow team is pleased to announce the DataFusion 5.0.0 release. This covers 4 months of development work
-and includes 211 commits from the following 31 distinct contributors.</p>
-<div class="codehilite"><pre><span></span><code>$ git shortlog -sn <span class="m">4</span>.0.0..5.0.0 datafusion datafusion-cli datafusion-examples
- <span class="m">61</span> Jiayu Liu
- <span class="m">47</span> Andrew Lamb
- <span class="m">27 …</span></code></pre></div></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2021/08/18/datafusion-5.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
-
- </div>
- </div>
-</div> </div>
- </div>
- </div>
-
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </div>
- </main>
- </body>
- </html>
diff --git a/output/author/timsaucer.html b/output/author/timsaucer.html
deleted file mode 100644
index 4564850..0000000
--- a/output/author/timsaucer.html
+++ /dev/null
@@ -1,150 +0,0 @@
- <!doctype html>
- <html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> <link href="/css/blog_index.css" rel="stylesheet">
- </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
- <div>
-
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
- <div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-
- <h3>Welcome to the Apache DataFusion Blog!</h3>
- <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
-
-
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/11/19/datafusion-python-udf-comparisons">Comparing approaches to User Defined Functions in Apache DataFusion using Python</a></h1>
- <p>Posted on: Tue 19 November 2024 by timsaucer</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Personal Context</h2>
-<p>For a few months now I’ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
-fast query engine written in Rust. From my experience the language that nearly all data scientists
-are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/11/19/datafusion-python-udf-comparisons" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/08/20/python-datafusion-40.0.0">Apache DataFusion Python 40.1.0 Released, Significant usability updates</a></h1>
- <p>Posted on: Tue 20 August 2024 by timsaucer</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/40.1.0/">DataFusion in Python 40.1.0</a> has been released. In addition to
-bringing in all of the new features of the core <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> package, this release
-contains <em>significant</em> updates to the user interface and documentation. We listened to the python …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/08/20/python-datafusion-40.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
-
- </div>
- </div>
-</div> </div>
- </div>
- </div>
-
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </div>
- </main>
- </body>
- </html>
diff --git a/output/author/xiangpeng-hao-andrew-lamb.html b/output/author/xiangpeng-hao-andrew-lamb.html
deleted file mode 100644
index b85cef3..0000000
--- a/output/author/xiangpeng-hao-andrew-lamb.html
+++ /dev/null
@@ -1,146 +0,0 @@
- <!doctype html>
- <html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> <link href="/css/blog_index.css" rel="stylesheet">
- </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
- <div>
-
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
- <div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-
- <h3>Welcome to the Apache DataFusion Blog!</h3>
- <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
-
-
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/09/13/string-view-german-style-strings-part-1">Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</a></h1>
- <p>Posted on: Fri 13 September 2024 by Xiangpeng Hao, Andrew Lamb</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p><em>Editor's Note: This is the first of a <a href="../2024/09/13/string-view-german-style-strings-part-2/">two part</a> blog series that was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/09/13/string-view-german-style-strings-part-1" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/09/13/string-view-german-style-strings-part-2">Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</a></h1>
- <p>Posted on: Fri 13 September 2024 by Xiangpeng Hao, Andrew Lamb</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p><em>Editor's Note: This blog series was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>In the <a href="https://datafusion.staged.apache.org/2024/09/13/string-view-german-style-strings-part-1">first post</a>, we discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies.
-In this second …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/09/13/string-view-german-style-strings-part-2" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
-
- </div>
- </div>
-</div> </div>
- </div>
- </div>
-
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </div>
- </main>
- </body>
- </html>
diff --git a/output/category/blog.html b/output/category/blog.html
deleted file mode 100644
index 1aa9d81..0000000
--- a/output/category/blog.html
+++ /dev/null
@@ -1,1129 +0,0 @@
- <!doctype html>
- <html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Blog</title>
-<link href="/css/bootstrap.min.css" rel="stylesheet">
-<link href="/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/css/headerlink.css" rel="stylesheet">
-<link href="/highlight/default.min.css" rel="stylesheet">
-<script src="/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> <link href="/css/blog_index.css" rel="stylesheet">
- </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
- <div>
-
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/"><img src="/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- </div>
- </div>
-</nav>
- <div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-<div id="contents">
- <div class="bg-white p-5 rounded">
- <div class="col-sm-8 mx-auto">
-
- <h3>Welcome to the Apache DataFusion Blog!</h3>
- <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
-
-
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/11/20/datafusion-comet-0.4.0">Apache DataFusion Comet 0.4.0 Release</a></h1>
- <p>Posted on: Wed 20 November 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/11/20/datafusion-comet-0.4.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/11/19/datafusion-python-udf-comparisons">Comparing approaches to User Defined Functions in Apache DataFusion using Python</a></h1>
- <p>Posted on: Tue 19 November 2024 by timsaucer</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Personal Context</h2>
-<p>For a few months now I’ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
-fast query engine written in Rust. From my experience the language that nearly all data scientists
-are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/11/19/datafusion-python-udf-comparisons" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench">Apache DataFusion is now the fastest single node engine for querying Apache Parquet files</a></h1>
- <p>Posted on: Mon 18 November 2024 by Andrew Lamb, Staff Engineer at InfluxData</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>I am extremely excited to announce that <a href="https://crates.io/crates/datafusion">Apache DataFusion</a> is the
-fastest engine for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">ClickBench</a>. It is faster
-than <a href="https://duckdb.org/">DuckDB</a>, <a href="https://clickhouse.com/chdb">chDB</a> and <a href="https://clickhouse.com/">Clickhouse</a> using the same hardware. It also marks
-the first time a <a href="https://www.rust-lang.org/">Rust</a>-based engine holds the top spot, which has previously
-been …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/09/27/datafusion-comet-0.3.0">Apache DataFusion Comet 0.3.0 Release</a></h1>
- <p>Posted on: Fri 27 September 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.3.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/09/27/datafusion-comet-0.3.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/09/13/string-view-german-style-strings-part-1">Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</a></h1>
- <p>Posted on: Fri 13 September 2024 by Xiangpeng Hao, Andrew Lamb</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p><em>Editor's Note: This is the first of a <a href="../2024/09/13/string-view-german-style-strings-part-2/">two part</a> blog series that was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/09/13/string-view-german-style-strings-part-1" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/09/13/string-view-german-style-strings-part-2">Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations</a></h1>
- <p>Posted on: Fri 13 September 2024 by Xiangpeng Hao, Andrew Lamb</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p><em>Editor's Note: This blog series was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>In the <a href="https://datafusion.staged.apache.org/2024/09/13/string-view-german-style-strings-part-1">first post</a>, we discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies.
-In this second …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/09/13/string-view-german-style-strings-part-2" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/08/28/datafusion-comet-0.2.0">Apache DataFusion Comet 0.2.0 Release</a></h1>
- <p>Posted on: Wed 28 August 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.2.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/08/28/datafusion-comet-0.2.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/08/20/python-datafusion-40.0.0">Apache DataFusion Python 40.1.0 Released, Significant usability updates</a></h1>
- <p>Posted on: Tue 20 August 2024 by timsaucer</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/40.1.0/">DataFusion in Python 40.1.0</a> has been released. In addition to
-bringing in all of the new features of the core <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> package, this release
-contains <em>significant</em> updates to the user interface and documentation. We listened to the python …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/08/20/python-datafusion-40.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/07/24/datafusion-40.0.0">Apache DataFusion 40.0.0 Released</a></h1>
- <p>Posted on: Wed 24 July 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- see https://github.com/apache/datafusion/issues/9602 for details -->
-<h2>Introduction</h2>
-<p>We are proud to announce <a href="https://crates.io/crates/datafusion/40.0.0">DataFusion 40.0.0</a>. This blog highlights some of the
-many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0/">DataFusion 34.0.0</a> and a preview of
-what the community is thinking about in the next 6 months. We are hoping to make
-more regular blog posts …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/07/24/datafusion-40.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/07/20/datafusion-comet-0.1.0">Apache DataFusion Comet 0.1.0 Release</a></h1>
- <p>Posted on: Sat 20 July 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce the first official source release of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/07/20/datafusion-comet-0.1.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/05/07/datafusion-tlp">Announcing Apache Arrow DataFusion is now Apache DataFusion</a></h1>
- <p>Posted on: Tue 07 May 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>TLDR; <a href="https://arrow.apache.org/">Apache Arrow</a> DataFusion --> <a href="https://datafusion.apache.org/">Apache DataFusion</a></p>
-<p>The Arrow PMC and newly created DataFusion PMC are happy to announce that as of
-April 16, 2024 the Apache Arrow DataFusion subproject is now a top level
-<a href="https://www.apache.org/">Apache Software Foundation</a> project.</p>
-<h2>Background</h2>
-<p>Apache DataFusion is a fast, extensible query engine for building …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/05/07/datafusion-tlp" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/03/06/comet-donation">Announcing Apache Arrow DataFusion Comet</a></h1>
- <p>Posted on: Wed 06 March 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p>The Apache Arrow PMC is pleased to announce the donation of the <a href="https://github.com/apache/arrow-datafusion-comet">Comet project</a>,
-a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/datafusion">Apache Arrow DataFusion</a>.</p>
-<p>Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to
-accelerate Spark workloads. It is designed as a drop-in
-replacement for Spark's JVM …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/03/06/comet-donation" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2024/01/19/datafusion-34.0.0">Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</a></h1>
- <p>Posted on: Fri 19 January 2024 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>We recently <a href="https://crates.io/crates/datafusion/34.0.0">released DataFusion 34.0.0</a>. This blog highlights some of the major
-improvements since we <a href="https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/.">released DataFusion 26.0.0</a> (spoiler alert there are many)
-and a preview of where the community plans to focus in the next 6 months.</p>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2024/01/19/datafusion-34.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2023/08/05/datafusion_fast_grouping">Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</a></h1>
- <p>Posted on: Sat 05 August 2023 by alamb, Dandandan, tustvold</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- Converted from Google Docs using https://www.buymeacoffee.com/docstomarkdown -->
-<h2>Aggregating Millions of Groups Fast in Apache Arrow DataFusion</h2>
-<p>Andrew Lamb, Daniël Heres, Raphael Taylor-Davies,</p>
-<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
-<h2>TLDR</h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2023/08/05/datafusion_fast_grouping" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2023/06/24/datafusion-25.0.0">Apache Arrow DataFusion 26.0.0</a></h1>
- <p>Posted on: Sat 24 June 2023 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>It has been a whirlwind 6 months of DataFusion development since <a href="https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0">our
-last update</a>: the community has grown, many features have been added,
-performance improved and we are <a href="https://github.com/apache/arrow-datafusion/discussions/6475">discussing</a> branching out to our own
-top level Apache Project.</p>
-<h2>Background</h2>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database
-toolkit …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2023/06/24/datafusion-25.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2023/01/19/datafusion-16.0.0">Apache Arrow DataFusion 16.0.0 Project Update</a></h1>
- <p>Posted on: Thu 19 January 2023 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible
-query execution framework, written in <a href="https://www.rust-lang.org/">Rust</a>,
-that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its
-in-memory format. It is targeted primarily at developers creating data
-intensive analytics, and offers mature
-<a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>,
-a DataFrame API, and many extension points.</p>
-<p>Systems based on DataFusion perform very well in benchmarks …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2023/01/19/datafusion-16.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/10/28/ballista-0.9.0">Apache Arrow Ballista 0.9.0 Release</a></h1>
- <p>Posted on: Fri 28 October 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://github.com/apache/arrow-ballista">Ballista</a> is an Arrow-native distributed SQL query engine implemented in Rust.</p>
-<p>Ballista 0.9.0 is now available and is the most significant release since the project was <a href="http://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to Apache
-Arrow in 2021.</p>
-<p>This release represents 4 weeks of work, with 66 commits from 14 contributors:</p>
-<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="mi">22</span><span class="w"> </span><span class="n">Andy …</span></code></pre></div></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/10/28/ballista-0.9.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/10/25/datafusion-13.0.0">Apache Arrow DataFusion 13.0.0 Project Update</a></h1>
- <p>Posted on: Tue 25 October 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> <a href="https://crates.io/crates/datafusion"><code>13.0.0</code></a> is released, and this blog contains an update on the project for the 5 months since our <a href="https://arrow.apache.org/blog/2022/05/16/datafusion-8.0.0/">last update in May 2022</a>.</p>
-<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/10/25/datafusion-13.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/05/16/datafusion-8.0.0">Apache Arrow DataFusion 8.0.0 Release</a></h1>
- <p>Posted on: Mon 16 May 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that
-uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>,
-a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/05/16/datafusion-8.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/03/21/datafusion-contrib">Introducing Apache Arrow DataFusion Contrib</a></h1>
- <p>Posted on: Mon 21 March 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p>Apache Arrow <a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/03/21/datafusion-contrib" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2022/02/28/datafusion-7.0.0">Apache Arrow DataFusion 7.0.0 Release</a></h1>
- <p>Posted on: Mon 28 February 2022 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2022/02/28/datafusion-7.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md">Apache Arrow DataFusion 6.0.0 Release</a></h1>
- <p>Posted on: Fri 19 November 2021 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an embedded
-query engine which leverages the unique features of
-<a href="https://www.rust-lang.org/">Rust</a> and <a href="https://arrow.apache.org/">Apache
-Arrow</a> to provide a system that is high
-performance, easy to connect, easy to embed, and high quality.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2021/08/18/ballista-0.5.0">Apache Arrow Ballista 0.5.0 Release</a></h1>
- <p>Posted on: Wed 18 August 2021 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>Ballista extends DataFusion to provide support for distributed queries. This is the first release of Ballista since
-the project was <a href="https://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to the Apache Arrow project
-and includes 80 commits from 11 contributors.</p>
-<div class="codehilite"><pre><span></span><code><span class="err">git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust …</span></code></pre></div></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2021/08/18/ballista-0.5.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2021/08/18/datafusion-5.0.0">Apache Arrow DataFusion 5.0.0 Release</a></h1>
- <p>Posted on: Wed 18 August 2021 by pmc</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache Arrow team is pleased to announce the DataFusion 5.0.0 release. This covers 4 months of development work
-and includes 211 commits from the following 31 distinct contributors.</p>
-<div class="codehilite"><pre><span></span><code>$ git shortlog -sn <span class="m">4</span>.0.0..5.0.0 datafusion datafusion-cli datafusion-examples
- <span class="m">61</span> Jiayu Liu
- <span class="m">47</span> Andrew Lamb
- <span class="m">27 …</span></code></pre></div></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2021/08/18/datafusion-5.0.0" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2021/04/12/ballista-donation">Ballista: A Distributed Scheduler for Apache Arrow</a></h1>
- <p>Posted on: Mon 12 April 2021 by agrove</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
-to the Apache Arrow project. </p>
-<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2021/04/12/ballista-donation" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
- <!-- Post -->
- <div class="row">
- <div class="callout">
- <article class="post">
- <header>
- <div class="title">
- <h1><a href="/blog/2019/02/04/datafusion-donation">DataFusion: A Rust-native Query Engine for Apache Arrow</a></h1>
- <p>Posted on: Mon 04 February 2019 by agrove</p>
- <p><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></p>
- <footer>
- <ul class="actions">
- <div style="text-align: right"><a href="/blog/2019/02/04/datafusion-donation" class="button medium">Continue Reading</a></div>
- </ul>
- <ul class="stats">
- </ul>
- </footer>
- </article>
- </div>
- </div>
-
- </div>
- </div>
-</div> </div>
- </div>
- </div>
-
- <!-- footer -->
- <div class="row">
- <div class="large-12 medium-12 columns">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/js/bootstrap.bundle.min.js"></script> </div>
- </main>
- </body>
- </html>
diff --git a/output/css/app.css b/output/css/app.css
deleted file mode 100644
index e69de29..0000000
diff --git a/output/css/blog_index.css b/output/css/blog_index.css
deleted file mode 100644
index cc36581..0000000
--- a/output/css/blog_index.css
+++ /dev/null
@@ -1,16 +0,0 @@
-/*
-These adjustments override bootstrap.min.css to give smaller headers in the blog index page.
-Otherwise the headers appear too large.
-*/
-
-h1 {
- font-size: calc(1.325rem + .9vw); /* Article title */
-}
-
-h2 {
- font-size: calc(1.3rem + .6vw); /* Main headers within article */
-}
-
-h3 {
- font-size: calc(1.275rem + .3vw);
-}
diff --git a/output/css/bootstrap.min.css b/output/css/bootstrap.min.css
deleted file mode 100644
index 3993414..0000000
--- a/output/css/bootstrap.min.css
+++ /dev/null
@@ -1,6 +0,0 @@
-@charset "UTF-8";/*!
- * Bootstrap v5.3.3 (https://getbootstrap.com/)
- * Copyright 2011-2024 The Bootstrap Authors
- * Licensed under MIT (https://github.com/twbs/bootstrap/blob/main/LICENSE)
- */:root,[data-bs-theme=light]{--bs-blue:#0d6efd;--bs-indigo:#6610f2;--bs-purple:#6f42c1;--bs-pink:#d63384;--bs-red:#dc3545;--bs-orange:#fd7e14;--bs-yellow:#ffc107;--bs-green:#198754;--bs-teal:#20c997;--bs-cyan:#0dcaf0;--bs-black:#000;--bs-white:#fff;--bs-gray:#6c757d;--bs-gray-dark:#343a40;--bs-gray-100:#f8f9fa;--bs-gray-200:#e9ecef;--bs-gray-300:#dee2e6;--bs-gray-400:#ced4da;--bs-gray-500:#adb5bd;--bs-gray-600:#6c757d;--bs-gray-700:#495057;--bs-gray-800:#343a40;--bs-gray-900:#212529;-- [...]
-/*# sourceMappingURL=bootstrap.min.css.map */
\ No newline at end of file
diff --git a/output/css/font-awesome.min.css b/output/css/font-awesome.min.css
deleted file mode 100644
index 540440c..0000000
--- a/output/css/font-awesome.min.css
+++ /dev/null
@@ -1,4 +0,0 @@
-/*!
- * Font Awesome 4.7.0 by @davegandy - http://fontawesome.io - @fontawesome
- * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License)
- */@font-face{font-family:'FontAwesome';src:url('../fonts/fontawesome-webfont.eot?v=4.7.0');src:url('../fonts/fontawesome-webfont.eot?#iefix&v=4.7.0') format('embedded-opentype'),url('../fonts/fontawesome-webfont.woff2?v=4.7.0') format('woff2'),url('../fonts/fontawesome-webfont.woff?v=4.7.0') format('woff'),url('../fonts/fontawesome-webfont.ttf?v=4.7.0') format('truetype'),url('../fonts/fontawesome-webfont.svg?v=4.7.0#fontawesomeregular') format('svg');font-weight:normal;font-style:norma [...]
diff --git a/output/css/fontawesome.all.min.css b/output/css/fontawesome.all.min.css
deleted file mode 100644
index d278768..0000000
--- a/output/css/fontawesome.all.min.css
+++ /dev/null
@@ -1,6 +0,0 @@
-/*!
- * Font Awesome Free 6.2.0 by @fontawesome - https://fontawesome.com
- * License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License)
- * Copyright 2022 Fonticons, Inc.
- */
-.fa{font-family:var(--fa-style-family,"Font Awesome 6 Free");font-weight:var(--fa-style,900)}.fa,.fa-brands,.fa-classic,.fa-regular,.fa-sharp,.fa-solid,.fab,.far,.fas{-moz-osx-font-smoothing:grayscale;-webkit-font-smoothing:antialiased;display:var(--fa-display,inline-block);font-style:normal;font-variant:normal;line-height:1;text-rendering:auto}.fa-classic,.fa-regular,.fa-solid,.far,.fas{font-family:"Font Awesome 6 Free"}.fa-brands,.fab{font-family:"Font Awesome 6 Brands"}.fa-1x{font-siz [...]
diff --git a/output/css/headerlink.css b/output/css/headerlink.css
deleted file mode 100644
index ed25829..0000000
--- a/output/css/headerlink.css
+++ /dev/null
@@ -1,16 +0,0 @@
-/*
-* Hide class="elementid-permalink", except when an enclosing heading
-* has the :hover property.
-*/
-.headerlink, .elementid-permalink {
- visibility: hidden;
-}
-h1:hover > .headerlink,
-h2:hover > .headerlink,
-h3:hover > .headerlink,
-h4:hover > .headerlink,
-h5:hover > .headerlink,
-h6:hover > .headerlink,
-dt:hover > .elementid-permalink {
- visibility: visible
-}
diff --git a/output/css/webfonts/fa-brands-400.ttf b/output/css/webfonts/fa-brands-400.ttf
deleted file mode 100644
index 502f362..0000000
Binary files a/output/css/webfonts/fa-brands-400.ttf and /dev/null differ
diff --git a/output/css/webfonts/fa-brands-400.woff2 b/output/css/webfonts/fa-brands-400.woff2
deleted file mode 100644
index d801b51..0000000
Binary files a/output/css/webfonts/fa-brands-400.woff2 and /dev/null differ
diff --git a/output/css/webfonts/fa-regular-400.ttf b/output/css/webfonts/fa-regular-400.ttf
deleted file mode 100644
index e0abe27..0000000
Binary files a/output/css/webfonts/fa-regular-400.ttf and /dev/null differ
diff --git a/output/css/webfonts/fa-regular-400.woff2 b/output/css/webfonts/fa-regular-400.woff2
deleted file mode 100644
index d736e4b..0000000
Binary files a/output/css/webfonts/fa-regular-400.woff2 and /dev/null differ
diff --git a/output/css/webfonts/fa-solid-900.ttf b/output/css/webfonts/fa-solid-900.ttf
deleted file mode 100644
index 13c9489..0000000
Binary files a/output/css/webfonts/fa-solid-900.ttf and /dev/null differ
diff --git a/output/css/webfonts/fa-solid-900.woff2 b/output/css/webfonts/fa-solid-900.woff2
deleted file mode 100644
index 3516fdb..0000000
Binary files a/output/css/webfonts/fa-solid-900.woff2 and /dev/null differ
diff --git a/output/css/webfonts/fa-v4compatibility.ttf b/output/css/webfonts/fa-v4compatibility.ttf
deleted file mode 100644
index dc29819..0000000
Binary files a/output/css/webfonts/fa-v4compatibility.ttf and /dev/null differ
diff --git a/output/css/webfonts/fa-v4compatibility.woff2 b/output/css/webfonts/fa-v4compatibility.woff2
deleted file mode 100644
index 28d46b1..0000000
Binary files a/output/css/webfonts/fa-v4compatibility.woff2 and /dev/null differ
diff --git a/output/favicon.ico b/output/favicon.ico
deleted file mode 100644
index fa733f2..0000000
Binary files a/output/favicon.ico and /dev/null differ
diff --git a/output/feed.xml b/output/feed.xml
deleted file mode 100644
index deb5cc0..0000000
--- a/output/feed.xml
+++ /dev/null
@@ -1,568 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog</title><link>https://datafusion.staged.apache.org/</link><description></description><lastBuildDate>Wed, 20 Nov 2024 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet 0.4.0 Release</title><link>https://datafusion.staged.apache.org/2024/11/20/datafusion-comet-0.4.0</link><description><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 20 Nov 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-11-20:/2024/11/20/datafusion-comet-0.4.0</guid><category>blog</category></item><item><title>Comparing approaches to User Defined Functions in Apache DataFusion using Python</title><link>https://datafusion.staged.apache.org/2024 [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Personal Context</h2>
-<p>For a few months now I&rsquo;ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
-fast query engine written in Rust. From my experience the language that nearly all data scientists
-are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Tue, 19 Nov 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-11-19:/2024/11/19/datafusion-python-udf-comparisons</guid><category>blog</category></item><item><title>Apache DataFusion is now the fastest single node engine for querying Apache P [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>I am extremely excited to announce that <a href="https://crates.io/crates/datafusion">Apache DataFusion</a> is the
-fastest engine for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">ClickBench</a>. It is faster
-than <a href="https://duckdb.org/">DuckDB</a>, <a href="https://clickhouse.com/chdb">chDB</a> and <a href="https://clickhouse.com/">Clickhouse</a> using the same hardware. It also marks
-the first time a <a href="https://www.rust-lang.org/">Rust</a>-based engine holds the top spot, which has previously
-been …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Lamb, Staff Engineer at InfluxData</dc:creator><pubDate>Mon, 18 Nov 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-11-18:/2024/11/18/datafusion-fastest-single-node-parquet-clickbench</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.3.0 Release</title><link>https://datafusion.staged.apache.org/2024/09/27/datafusion-comet-0.3 [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.3.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 27 Sep 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-09-27:/2024/09/27/datafusion-comet-0.3.0</guid><category>blog</category></item><item><title>Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet</title><link>https://datafusion.staged.apache.o [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p><em>Editor's Note: This is the first of a <a href="../2024/09/13/string-view-german-style-strings-part-2/">two part</a> blog series that was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao, Andrew Lamb</dc:creator><pubDate>Fri, 13 Sep [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p><em>Editor's Note: This blog series was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>In the <a href="https://datafusion.staged.apache.org/2024/09/13/string-view-german-style-strings-part-1">first post</a>, we discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies.
-In this second …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Xiangpeng Hao, Andrew Lamb</dc:creator><pubDate>Fri, 13 Sep 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-09-13:/2024/09/13/string-view-german-style-strings-part-2</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.2.0 Release</title><link>https://datafusion.staged.apache.org/2024/08/28/datafusion-comet-0.2.0</link><descr [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.2.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 28 Aug 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-08-28:/2024/08/28/datafusion-comet-0.2.0</guid><category>blog</category></item><item><title>Apache DataFusion Python 40.1.0 Released, Significant usability updates</title><link>https://datafusion.staged.apache.org/2024/08/20/py [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/40.1.0/">DataFusion in Python 40.1.0</a> has been released. In addition to
-bringing in all of the new features of the core <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> package, this release
-contains <em>significant</em> updates to the user interface and documentation. We listened to the python …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">timsaucer</dc:creator><pubDate>Tue, 20 Aug 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-08-20:/2024/08/20/python-datafusion-40.0.0</guid><category>blog</category></item><item><title>Apache DataFusion 40.0.0 Released</title><link>https://datafusio [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- see https://github.com/apache/datafusion/issues/9602 for details -->
-<h2>Introduction</h2>
-<p>We are proud to announce <a href="https://crates.io/crates/datafusion/40.0.0">DataFusion 40.0.0</a>. This blog highlights some of the
-many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0/">DataFusion 34.0.0</a> and a preview of
-what the community is thinking about in the next 6 months. We are hoping to make
-more regular blog posts …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 24 Jul 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-07-24:/2024/07/24/datafusion-40.0.0</guid><category>blog</category></item><item><title>Apache DataFusion Comet 0.1.0 Release</title><link>https://datafusion.staged.apache.org/2024/07/20/datafusion-comet-0.1.0</link><description><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce the first official source release of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Sat, 20 Jul 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-07-20:/2024/07/20/datafusion-comet-0.1.0</guid><category>blog</category></item><item><title>Announcing Apache Arrow DataFusion is now Apache DataFusion</title><link>https://datafusion.staged.apache.org/2024/05/07/datafusion-tlp</l [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>TLDR; <a href="https://arrow.apache.org/">Apache Arrow</a> DataFusion --&gt; <a href="https://datafusion.apache.org/">Apache DataFusion</a></p>
-<p>The Arrow PMC and newly created DataFusion PMC are happy to announce that as of
-April 16, 2024 the Apache Arrow DataFusion subproject is now a top level
-<a href="https://www.apache.org/">Apache Software Foundation</a> project.</p>
-<h2>Background</h2>
-<p>Apache DataFusion is a fast, extensible query engine for building …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 07 May 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-05-07:/2024/05/07/datafusion-tlp</guid><category>blog</category></item><item><title>Announcing Apache Arrow DataFusion Comet</title><link>https://datafusion.staged.apache.org/2024/03/06/comet-donation</link [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p>The Apache Arrow PMC is pleased to announce the donation of the <a href="https://github.com/apache/arrow-datafusion-comet">Comet project</a>,
-a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/datafusion">Apache Arrow DataFusion</a>.</p>
-<p>Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to
-accelerate Spark workloads. It is designed as a drop-in
-replacement for Spark's JVM …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 06 Mar 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-03-06:/2024/03/06/comet-donation</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024</title><link>https://datafusion.staged.apache.org/2024/01/19/datafusion-34.0.0</link><description><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Introduction</h2>
-<p>We recently <a href="https://crates.io/crates/datafusion/34.0.0">released DataFusion 34.0.0</a>. This blog highlights some of the major
-improvements since we <a href="https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/.">released DataFusion 26.0.0</a> (spoiler alert there are many)
-and a preview of where the community plans to focus in the next 6 months.</p>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 19 Jan 2024 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2024-01-19:/2024/01/19/datafusion-34.0.0</guid><category>blog</category></item><item><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</titl [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- Converted from Google Docs using https://www.buymeacoffee.com/docstomarkdown -->
-<h2>Aggregating Millions of Groups Fast in Apache Arrow DataFusion</h2>
-<p>Andrew Lamb, Dani&euml;l Heres, Raphael Taylor-Davies,</p>
-<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
-<h2>TLDR</h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>&rsquo;s parallel aggregation capability …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, Dandandan, tustvold</dc:creator><pubDate>Sat, 05 Aug 2023 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2023-08- [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>It has been a whirlwind 6 months of DataFusion development since <a href="https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0">our
-last update</a>: the community has grown, many features have been added,
-performance improved and we are <a href="https://github.com/apache/arrow-datafusion/discussions/6475">discussing</a> branching out to our own
-top level Apache Project.</p>
-<h2>Background</h2>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database
-toolkit …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Sat, 24 Jun 2023 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2023-06-24:/2023/06/24/datafusion-25.0.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 16.0.0 Project Update</title><link>https://datafusion.staged.apache.org/2023/01/19/datafusion-16.0.0</link><description><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible
-query execution framework, written in <a href="https://www.rust-lang.org/">Rust</a>,
-that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its
-in-memory format. It is targeted primarily at developers creating data
-intensive analytics, and offers mature
-<a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>,
-a DataFrame API, and many extension points.</p>
-<p>Systems based on DataFusion perform very well in benchmarks …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 19 Jan 2023 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2023-01-19:/2023/01/19/datafusion-16.0.0</guid><category>blog</category></item><item><title>Apache Arrow Ballista 0.9.0 Release</title><link>https://datafusion.staged.apache.org/2022/10/28/ballista-0.9.0</link><descri [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://github.com/apache/arrow-ballista">Ballista</a> is an Arrow-native distributed SQL query engine implemented in Rust.</p>
-<p>Ballista 0.9.0 is now available and is the most significant release since the project was <a href="http://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to Apache
-Arrow in 2021.</p>
-<p>This release represents 4 weeks of work, with 66 commits from 14 contributors:</p>
-<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="mi">22</span><span class="w"> </span><span class="n">Andy …</span></code></pre></div></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 28 Oct 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2022-10-28:/2022/10/28/ball [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> <a href="https://crates.io/crates/datafusion"><code>13.0.0</code></a> is released, and this blog contains an update on the project for the 5 months since our <a href="https://arrow.apache.org/blog/2022/05/16/datafusion-8.0.0/">last update in May 2022</a>.</p>
-<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Tue, 25 Oct 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2022-10-25:/2022/10/25/datafusion-13.0.0</guid><category>blog</category></item><item><title>Apache Arrow DataFusion 8.0.0 Release</title><link [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that
-uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>,
-a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 16 May 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2022-05-16:/2022/05/16/datafusion-8.0.0</guid><category>blog</category></item><item><title>Introducing Apache Arrow DataFusion Contrib</title><link>https://datafusion.sta [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p>Apache Arrow <a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 21 Mar 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2022-03-21:/2022/03/21/da [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Mon, 28 Feb 2022 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2022-02- [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1>Introduction</h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an embedded
-query engine which leverages the unique features of
-<a href="https://www.rust-lang.org/">Rust</a> and <a href="https://arrow.apache.org/">Apache
-Arrow</a> to provide a system that is high
-performance, easy to connect, easy to embed, and high quality.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 19 Nov 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2021-11-19:/2021/11/19/2021-11-8-datafusion-6.0.0.md</guid><category>blog</category></item><item><title>Apache Arrow Ballista 0.5.0 Release</title><link>https://datafusion.staged.apache.org/ [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>Ballista extends DataFusion to provide support for distributed queries. This is the first release of Ballista since
-the project was <a href="https://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to the Apache Arrow project
-and includes 80 commits from 11 contributors.</p>
-<div class="codehilite"><pre><span></span><code><span class="err">git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust …</span></code></pre></div></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 18 Aug 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2021-08-18:/2021/08/18/ballista-0.5.0</guid><category>blog</ [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache Arrow team is pleased to announce the DataFusion 5.0.0 release. This covers 4 months of development work
-and includes 211 commits from the following 31 distinct contributors.</p>
-<div class="codehilite"><pre><span></span><code>$ git shortlog -sn <span class="m">4</span>.0.0..5.0.0 datafusion datafusion-cli datafusion-examples
- <span class="m">61</span> Jiayu Liu
- <span class="m">47</span> Andrew Lamb
- <span class="m">27 …</span></code></pre></div></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Wed, 18 Aug 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2021-08-18:/2021/08/18/datafusion-5.0.0</guid><category>blog</category></item><item><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link>https://datafusion.staged.apache.org/2021/04/12/ballista-donat [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
-to the Apache Arrow project. </p>
-<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 12 Apr 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2021-04-12:/2021/04/12/ballista-donation</guid><category>blog</category></item><item><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link>https://datafus [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 04 Feb 2019 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2019-02-04:/2019/02/04/datafusion-donation</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/output/feeds/agrove.atom.xml b/output/feeds/agrove.atom.xml
deleted file mode 100644
index bec5e24..0000000
--- a/output/feeds/agrove.atom.xml
+++ /dev/null
@@ -1,189 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - agrove</title><link href="https://datafusion.staged.apache.org/" rel="alternate"></link><link href="https://datafusion.staged.apache.org/feeds/agrove.atom.xml" rel="self"></link><id>https://datafusion.staged.apache.org/</id><updated>2021-04-12T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link href="https://datafusion.staged.apache.org/2021/04/ [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
-to the Apache Arrow project. </p>
-<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></summary><content type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
-to the Apache Arrow project. </p>
-<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported as
-first-class citizens without paying a penalty for serialization costs.</p>
-<p>The foundational technologies in Ballista are:</p>
-<ul>
-<li><a href="https://arrow.apache.org/">Apache Arrow</a> memory model and compute kernels for efficient processing of data.</li>
-<li><a href="https://github.com/apache/arrow-datafusion">Apache Arrow DataFusion</a> query planning and
- execution framework, extended by Ballista to provide distributed planning and execution.</li>
-<li><a href="https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/">Apache Arrow Flight Protocol</a> for efficient
- data transfer between processes.</li>
-<li><a href="https://developers.google.com/protocol-buffers">Google Protocol Buffers</a> for serializing query plans.</li>
-<li><a href="https://www.docker.com/">Docker</a> for packaging up executors along with user-defined code.</li>
-</ul>
-<p>Ballista can be deployed as a standalone cluster and also supports <a href="https://kubernetes.io/">Kubernetes</a>. In either
-case, the scheduler can be configured to use <a href="https://etcd.io/">etcd</a> as a backing store to (eventually) provide
-redundancy in the case of a scheduler failing.</p>
-<h2>Status</h2>
-<p>The Ballista project is at an early stage of development. However, it is capable of running complex analytics queries
-in a distributed cluster with reasonable performance (comparable to more established distributed query frameworks).</p>
-<p>One of the benefits of Ballista being part of the Arrow codebase is that there is now an opportunity to push parts of
-the scheduler down to DataFusion so that is possible to seamlessly scale across cores in DataFusion, and across nodes
-in Ballista, using the same unified query scheduler.</p>
-<h2>Contributors Welcome!</h2>
-<p>If you are excited about being able to use Rust for distributed compute and ETL and would like to contribute to this
-work then there are many ways to get involved. The simplest way to get started is to try out Ballista against your own
-datasets and file bug reports for any issues that you find. You could also check out the current
-<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20component%20%3D%20%22Rust%20-%20Ballista%22">list of issues</a> and have a go at fixing one.</p>
-<p>The <a href="https://github.com/apache/arrow/blob/master/rust/README.md#arrow-rust-community">Arrow Rust Community</a>
-section of the Rust README provides information on other ways to interact with the Ballista contributors and
-maintainers.</p></content><category term="blog"></category></entry><entry><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link href="https://datafusion.staged.apache.org/2019/02/04/datafusion-donation" rel="alternate"></link><published>2019-02-04T00:00:00+00:00</published><updated>2019-02-04T00:00:00+00:00</updated><author><name>agrove</name></author><id>tag:datafusion.staged.apache.org,2019-02-04:/2019/02/04/datafusion-donation</id><summary type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></summary><content type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support SQL queries against iterators of RecordBatch and has support for CSV files. There are plans to <a href="https://issues.apache.org/jira/browse/ARROW-4466">add support for Parquet files</a>.</p>
-<p>SQL support is limited to projection (<code>SELECT</code>), selection (<code>WHERE</code>), and simple aggregates (<code>MIN</code>, <code>MAX</code>, <code>SUM</code>) with an optional <code>GROUP BY</code> clause.</p>
-<p>Supported expressions are identifiers, literals, simple math operations (<code>+</code>, <code>-</code>, <code>*</code>, <code>/</code>), binary expressions (<code>AND</code>, <code>OR</code>), equality and comparison operators (<code>=</code>, <code>!=</code>, <code>&lt;</code>, <code>&lt;=</code>, <code>&gt;=</code>, <code>&g [...]
-<h2>Example</h2>
-<p>The following example demonstrates running a simple aggregate SQL query against a CSV file.</p>
-<div class="codehilite"><pre><span></span><code><span class="c1">// create execution context</span>
-<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">ctx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ExecutionContext</span>::<span class="n">new</span><span class="p">();</span><span class="w"></span>
-
-<span class="c1">// define schema for data source (csv file)</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Arc</span>::<span class="n">new</span><span class="p">(</span><span class="n">Schema</span>::<span class="n">new</span><span class="p">(</span><span class="n">vec</span> [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c1"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</ [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c2"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt32</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),< [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c3"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int8</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</ [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c4"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int16</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),< [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c5"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int32</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),< [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c6"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int64</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),< [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c7"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt8</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),< [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c8"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt16</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),< [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c9"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt32</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),< [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c10"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">UInt64</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),&l [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c11"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float32</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),& [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c12"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float64</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),& [...]
-<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"c13"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),< [...]
-<span class="p">]));</span><span class="w"></span>
-
-<span class="c1">// register csv file with the execution context</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">csv_datasource</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
-<span class="w"> </span><span class="n">CsvDataSource</span>::<span class="n">new</span><span class="p">(</span><span class="s">"test/data/aggregate_test_100.csv"</span><span class="p">,</span><span class="w"> </span><span class="n">schema</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span><span class="w"> </span& [...]
-<span class="n">ctx</span><span class="p">.</span><span class="n">register_datasource</span><span class="p">(</span><span class="s">"aggregate_test_100"</span><span class="p">,</span><span class="w"> </span><span class="n">Rc</span>::<span class="n">new</span><span class="p">(</span><span class="n">RefCell</span>::<span class="n">new</span><sp [...]
-
-<span class="kd">let</span><span class="w"> </span><span class="n">sql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"SELECT c1, MIN(c12), MAX(c12) FROM aggregate_test_100 WHERE c11 &gt; 0.1 AND c11 &lt; 0.9 GROUP BY c1"</span><span class="p">;</span><span class="w"></span>
-
-<span class="c1">// execute the query</span>
-<span class="kd">let</span><span class="w"> </span><span class="n">relation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ctx</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="o">&amp;</span><span class="n">sql</span><span class="p">).</span> [...]
-<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">results</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">relation</span><span class="p">.</span><span class="n">borrow_mut</span><span class="p">();</span><span class="w"></span>
-
-<span class="c1">// iterate over the results</span>
-<span class="k">while</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">batch</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">results</span><span class="p">.</span>< [...]
-<span class="w"> </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="w"></span>
-<span class="w"> </span><span class="s">"RecordBatch has {} rows and {} columns"</span><span class="p">,</span><span class="w"></span>
-<span class="w"> </span><span class="n">batch</span><span class="p">.</span><span class="n">num_rows</span><span class="p">(),</span><span class="w"></span>
-<span class="w"> </span><span class="n">batch</span><span class="p">.</span><span class="n">num_columns</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">);</span><span class="w"></span>
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">c1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">batch</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">as_any</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">downcast_ref</span>::<span class="o">&lt;</span><span class="n">BinaryArray</span><span class="o">&gt;</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">min</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">batch</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">column</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">as_any</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">downcast_ref</span>::<span class="o">&lt;</span><span class="n">Float64Array</span><span class="o">&gt;</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
-
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">max</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">batch</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">column</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">as_any</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">downcast_ref</span>::<span class="o">&lt;</span><span class="n">Float64Array</span><span class="o">&gt;</span><span class="p">()</span><span class="w"></span>
-<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
-
-<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">0</span><span class="o">..</span><span class="n">batch</span><span class="p">.</span><span class="n">num_rows</span><span class="p">()</span>< [...]
-<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">c1_value</span>: <span class="nb">String</span> <span class="o">=</span><span class="w"> </span><span class="nb">String</span>::<span class="n">from_utf8</span><span class="p">(</span><span class="n">c1</span><span class="p">.</span><span class="n [...]
-<span class="w"> </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{}, Min: {}, Max: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">c1_value</span><span class="p">,</span><span class="w"> </span><span class="n">min</span><span class="p">.</span><span class="n [...]
-<span class="w"> </span><span class="p">}</span><span class="w"></span>
-<span class="p">}</span><span class="w"></span>
-</code></pre></div>
-<h2>Roadmap</h2>
-<p>The roadmap for DataFusion will depend on interest from the Rust community, but here are some of the short term items that are planned:</p>
-<ul>
-<li>Extending test coverage of the existing functionality</li>
-<li>Adding support for Parquet data sources</li>
-<li>Implementing more SQL features such as <code>JOIN</code>, <code>ORDER BY</code> and <code>LIMIT</code></li>
-<li>Implement a DataFrame API as an alternative to SQL</li>
-<li>Adding support for partitioning and parallel query execution using Rust's async and await functionality</li>
-<li>Creating a Docker image to make it easy to use DataFusion as a standalone query tool for interactive and batch queries</li>
-</ul>
-<h2>Contributors Welcome!</h2>
-<p>If you are excited about being able to use Rust for data science and would like to contribute to this work then there are many ways to get involved. The simplest way to get started is to try out DataFusion against your own data sources and file bug reports for any issues that you find. You could also check out the current <a href="https://cwiki.apache.org/confluence/display/ARROW/Rust+JIRA+Dashboard">list of issues</a> and have a go at fixing one. You can also join t [...]
\ No newline at end of file
diff --git a/output/feeds/agrove.rss.xml b/output/feeds/agrove.rss.xml
deleted file mode 100644
index 8afe237..0000000
--- a/output/feeds/agrove.rss.xml
+++ /dev/null
@@ -1,42 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - agrove</title><link>https://datafusion.staged.apache.org/</link><description></description><lastBuildDate>Mon, 12 Apr 2021 00:00:00 +0000</lastBuildDate><item><title>Ballista: A Distributed Scheduler for Apache Arrow</title><link>https://datafusion.staged.apache.org/2021/04/12/ballista-donation</link><description><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
-to the Apache Arrow project. </p>
-<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 12 Apr 2021 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2021-04-12:/2021/04/12/ballista-donation</guid><category>blog</category></item><item><title>DataFusion: A Rust-native Query Engine for Apache Arrow</title><link>https://datafus [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">agrove</dc:creator><pubDate>Mon, 04 Feb 2019 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2019-02-04:/2019/02/04/datafusion-donation</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/output/feeds/alamb-dandandan-tustvold.atom.xml b/output/feeds/alamb-dandandan-tustvold.atom.xml
deleted file mode 100644
index df0ec27..0000000
--- a/output/feeds/alamb-dandandan-tustvold.atom.xml
+++ /dev/null
@@ -1,321 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - alamb, Dandandan, tustvold</title><link href="https://datafusion.staged.apache.org/" rel="alternate"></link><link href="https://datafusion.staged.apache.org/feeds/alamb-dandandan-tustvold.atom.xml" rel="self"></link><id>https://datafusion.staged.apache.org/</id><updated>2023-08-05T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title> [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- Converted from Google Docs using https://www.buymeacoffee.com/docstomarkdown -->
-<h2>Aggregating Millions of Groups Fast in Apache Arrow DataFusion</h2>
-<p>Andrew Lamb, Dani&euml;l Heres, Raphael Taylor-Davies,</p>
-<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
-<h2>TLDR</h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>&rsquo;s parallel aggregation capability …</p></summary><content type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- Converted from Google Docs using https://www.buymeacoffee.com/docstomarkdown -->
-<h2>Aggregating Millions of Groups Fast in Apache Arrow DataFusion</h2>
-<p>Andrew Lamb, Dani&euml;l Heres, Raphael Taylor-Davies,</p>
-<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
-<h2>TLDR</h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>&rsquo;s parallel aggregation capability is 2-3x faster in the <a href="https://crates.io/crates/datafusion/28.0.0">newly released version <code>28.0.0</code></a> for queries with a large number (10,000 or more) of groups.</p>
-<p>Improving aggregation performance matters to all users of DataFusion. For example, both InfluxDB, a <a href="https://github.com/influxdata/influxdb">time series data platform</a> and Coralogix, a <a href="https://coralogix.com/?utm_source=InfluxDB&amp;utm_medium=Blog&amp;utm_campaign=organic">full-stack observability</a> platform, aggregate vast amounts of raw data to monitor and create insights for our customers. Improving DataFusion&rsquo;s [...]
-<p>With the new optimizations, DataFusion&rsquo;s grouping speed is now close to DuckDB, a system that regularly reports <a href="https://duckdblabs.github.io/db-benchmark/">great</a> <a href="https://duckdb.org/2022/03/07/aggregate-hashtable.html#experiments">grouping</a> benchmark performance numbers. Figure 1 contains a representative sample of <a href="https://github.com/ClickHouse/ClickBench/tree/main">ClickBench</a> on a single Parquet [...]
-<p><img src="../images/datafusion_fast_grouping/summary.png" width="700"/></p>
-<p><strong>Figure 1</strong>: Query performance for ClickBench queries on queries 16, 17, 18 and 19 on a single Parquet file for DataFusion <code>27.0.0</code>, DataFusion <code>28.0.0</code> and DuckDB <code>0.8.1</code>.</p>
-<h2>Introduction to high cardinality grouping</h2>
-<p>Aggregation is a fancy word for computing summary statistics across many rows that have the same value in one or more columns. We call the rows with the same values <em>groups</em> and &ldquo;high cardinality&rdquo; means there are a large number of distinct groups in the dataset. At the time of writing, a &ldquo;large&rdquo; number of groups in analytic engines is around 10,000.</p>
-<p>For example the <a href="https://github.com/ClickHouse/ClickBench">ClickBench</a> <em>hits</em> dataset contains 100 million anonymized user clicks across a set of websites. ClickBench Query 17 is:</p>
-<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span> <span class="ss">"UserID"</span><span class="p">,</span> <span class="ss">"SearchPhrase"</span><span class="p">,</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span>
-<span class="k">FROM</span> <span class="n">hits</span>
-<span class="k">GROUP</span> <span class="k">BY</span> <span class="ss">"UserID"</span><span class="p">,</span> <span class="ss">"SearchPhrase"</span>
-<span class="k">ORDER</span> <span class="k">BY</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span>
-<span class="k">DESC</span> <span class="k">LIMIT</span> <span class="mi">10</span><span class="p">;</span>
-</code></pre></div>
-<p>In English, this query finds &ldquo;the top ten (user, search phrase) combinations, across all clicks&rdquo; and produces the following results (there are no search phrases for the top ten users):</p>
-<div class="codehilite"><pre><span></span><code>+---------------------+--------------+-----------------+
-| UserID | SearchPhrase | COUNT(UInt8(1)) |
-+---------------------+--------------+-----------------+
-| 1313338681122956954 | | 29097 |
-| 1907779576417363396 | | 25333 |
-| 2305303682471783379 | | 10597 |
-| 7982623143712728547 | | 6669 |
-| 7280399273658728997 | | 6408 |
-| 1090981537032625727 | | 6196 |
-| 5730251990344211405 | | 6019 |
-| 6018350421959114808 | | 5990 |
-| 835157184735512989 | | 5209 |
-| 770542365400669095 | | 4906 |
-+---------------------+--------------+-----------------+
-</code></pre></div>
-<p>The ClickBench dataset contains</p>
-<ul>
-<li>99,997,497 total rows[^1]</li>
-<li>17,630,976 different users (distinct UserIDs)[^2]</li>
-<li>6,019,103 different search phrases[^3]</li>
-<li>24,070,560 distinct combinations[^4] of (UserID, SearchPhrase)
-Thus, to answer the query, DataFusion must map each of the 100M different input rows into one of the <strong>24 million different groups</strong>, and keep count of how many such rows there are in each group.</li>
-</ul>
-<h2>The solution</h2>
-<p>Like most concepts in databases and other analytic systems, the basic ideas of this algorithm are straightforward and taught in introductory computer science courses. You could compute the query with a program such as this[^5]:</p>
-<div class="codehilite"><pre><span></span><code><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
-<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
-<span class="kn">from</span> <span class="nn">operator</span> <span class="kn">import</span> <span class="n">itemgetter</span>
-
-<span class="c1"># read file</span>
-<span class="n">hits</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_parquet</span><span class="p">(</span><span class="s1">'hits.parquet'</span><span class="p">,</span> <span class="n">engine</span><span class="o">=</span><span class="s1">'pyarrow'</span><span class="p">)</span>
-
-<span class="c1"># build groups</span>
-<span class="n">counts</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">int</span><span class="p">)</span>
-<span class="k">for</span> <span class="n">index</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">hits</span><span class="o">.</span><span class="n">iterrows</span><span class="p">():</span>
- <span class="n">group</span> <span class="o">=</span> <span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s1">'UserID'</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="s1">'SearchPhrase'</span><span class="p">]);</span>
- <span class="c1"># update the dict entry for the corresponding key</span>
- <span class="n">counts</span><span class="p">[</span><span class="n">group</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
-
-<span class="c1"># Print the top 10 values</span>
-<span class="nb">print</span> <span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">counts</span><span class="o">.</span><span class="n">items</span><span class="p">(),</span> <span class="n">key</span><span class="o">=</span><span class="n">itemgett [...]
-</code></pre></div>
-<p>This approach, while simple, is both slow and very memory inefficient. It requires over 40 seconds to compute the results for less than 1% of the dataset[^6]. Both DataFusion <code>28.0.0</code> and DuckDB <code>0.8.1</code> compute results in under 10 seconds for the <em>entire</em> dataset.</p>
-<p>To answer this query quickly and efficiently, you have to write your code such that it:</p>
-<ol>
-<li>Keeps all cores busy aggregating via parallelized computation</li>
-<li>Updates aggregate values quickly, using vectorizable loops that are easy for compilers to translate into the high performance <a href="https://en.wikipedia.org/wiki/Single_instruction,_multiple_data">SIMD</a> instructions available in modern CPUs.</li>
-</ol>
-<p>The rest of this article explains how grouping works in DataFusion and the improvements we made in <code>28.0.0</code>.</p>
-<h3>Two phase parallel partitioned grouping</h3>
-<p>Both DataFusion <code>27.0.</code> and <code>28.0.0</code> use state-of-the-art, two phase parallel hash partitioned grouping, similar to other high-performance vectorized engines like <a href="https://duckdb.org/2022/03/07/aggregate-hashtable.html">DuckDB&rsquo;s Parallel Grouped Aggregates</a>. In pictures this looks like:</p>
-<div class="codehilite"><pre><span></span><code> ▲ ▲
- &boxv; &boxv;
- &boxv; &boxv;
- &boxv; &boxv;
-&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl; &boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;
-&boxv; GroupBy &boxv; &boxv; GroupBy &boxv; Step 4
-&boxv; (Final) &boxv; &boxv; (Final) &boxv;
-&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul; &boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;
- ▲ ▲
- &boxv; &boxv;
- &boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxhd;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;
- &boxv;
- &boxv;
- &boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;
- &boxv; Repartition &boxv; Step 3
- &boxv; HASH(x) &boxv;
- &boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;
- ▲
- &boxv;
- &boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxhu;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;
- &boxv; &boxv;
- &boxv; &boxv;
- &boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl; &boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;
- &boxv; GroupyBy &boxv; &boxv; GroupBy &boxv; Step 2
- &boxv; (Partial) &boxv; &boxv; (Partial) &boxv;
- &boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul; &boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;
- ▲ ▲
- &boxdr;&boxh;&boxh;&boxul; &boxur;&boxh;&boxdl;
- &boxv; &boxv;
- .&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;. .&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;.
- ,&boxh;' '&boxh;. ,&boxh;' '&boxh;.
-; Input : ; Input : Step 1
-: Stream 1 ; : Stream 2 ;
- ╲ ╱ ╲ ╱
- '&boxh;. ,&boxh;' '&boxh;. ,&boxh;'
- `&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;' `&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;'
-</code></pre></div>
-<p><strong>Figure 2</strong>: Two phase repartitioned grouping: data flows from bottom (source) to top (results) in two phases. First (Steps 1 and 2), each core reads the data into a core-specific hash table, computing intermediate aggregates without any cross-core coordination. Then (Steps 3 and 4) DataFusion divides the data (&ldquo;repartitions&rdquo;) into distinct subsets by group value, and each subset is sent to a specific core which computes the final ag [...]
-<p>The two phases are critical for keeping cores busy in a multi-core system. Both phases use the same hash table approach (explained in the next section), but differ in how the groups are distributed and the partial results emitted from the accumulators. The first phase aggregates data as soon as possible after it is produced. However, as shown in Figure 2, the groups can be anywhere in any input, so the same group is often found on many different cores. The second phase uses a ha [...]
-<div class="codehilite"><pre><span></span><code> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span>
- <span class="err">&boxv;</span> <span class="mi">1</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">3</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">2</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">4</span> <span class="err">&boxv;</span> <span class="mi">2</span><span class="p">.</span> <span class="k">After</span> <span class="n">Repartitioning</span><span class="p">:</span> <span c [...]
- <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="k">group</span> <span class="k">key</span> <span class="n">appears</span> <span class="k">in</span> <span class="n">exactly</span>
- <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="n">one</span> <span class="n">partition</span>
- <span class="err">&boxv;</span> <span class="mi">1</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">3</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">2</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">4</span> <span class="err">&boxv;</span>
- <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span>
-
-<span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span cla [...]
-
- <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span>
- <span class="err">&boxv;</span> <span class="mi">2</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">2</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">1</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">2</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">3</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">3</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">4</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">1</span> <span class="err">&boxv;</span>
- <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="mi">1</span><span class="p">.</span> <span class="k">Input</span> <span class="n">Stream</span><span class="p">:</span> <span class="n">groups</span>
- <span class="p">...</span> <span class="p">...</span> <span class="k">values</span> <span class="k">are</span> <span class="n">spread</span>
- <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="n">arbitrarily</span> <span class="n">over</span> <span class="k">each</span> <span class="k">input</span>
- <span class="err">&boxv;</span> <span class="mi">1</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">4</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">4</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">3</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">1</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">1</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">4</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">3</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">3</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">2</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">2</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">2</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="mi">2</span> <span class="err">&boxv;</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span>
- <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span>
-
- <span class="n">Core</span> <span class="n">A</span> <span class="n">Core</span> <span class="n">B</span>
-</code></pre></div>
-<p><strong>Figure 3</strong>: Group value distribution across 2 cores during aggregation phases. In the first phase, every group value <code>1</code>, <code>2</code>, <code>3</code>, <code>4</code>, is present in the input stream processed by each core. In the second phase, after repartitioning, the group values <code>1</code> and <code>2</code> are processed by core A, and values <code>3</ [...]
-<p>There are some additional subtleties in the <a href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/physical_plan/aggregates/row_hash.rs">DataFusion implementation</a> not mentioned above due to space constraints, such as:</p>
-<ol>
-<li>The policy of when to emit data from the first phase&rsquo;s hash table (e.g. because the data is partially sorted)</li>
-<li>Handling specific filters per aggregate (due to the <code>FILTER</code> SQL clause)</li>
-<li>Data types of intermediate values (which may not be the same as the final output for some aggregates such as <code>AVG</code>).</li>
-<li>Action taken when memory use exceeds its budget.</li>
-</ol>
-<h3>Hash grouping</h3>
-<p>DataFusion queries can compute many different aggregate functions for each group, both <a href="https://arrow.apache.org/datafusion/user-guide/sql/aggregate_functions.html">built in</a> and/or user defined <a href="https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.AggregateUDF.html"><code>AggregateUDFs</code></a>. The state for each aggregate function, called an <em>accumulator</em>, is tracked with a hash table (Dat [...]
-<h3>Hash grouping in <code>27.0.0</code></h3>
-<p>As shown in Figure 3, DataFusion <code>27.0.0</code> stores the data in a <a href="https://github.com/apache/arrow-datafusion/blob/4d93b6a3802151865b68967bdc4c7d7ef425b49a/datafusion/core/src/physical_plan/aggregates/utils.rs#L38-L50"><code>GroupState</code></a> structure which, unsurprisingly, tracks the state for each group. The state for each group consists of:</p>
-<ol>
-<li>The actual value of the group columns, in <a href="https://docs.rs/arrow-row/latest/arrow_row/index.html">Arrow Row</a> format.</li>
-<li>In-progress accumulations (e.g. the running counts for the <code>COUNT</code> aggregate) for each group, in one of two possible formats (<a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/expr/src/accumulator.rs#L24-L49"><code>Accumulator</code></a> or <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/aggr [...]
-<li>Scratch space for tracking which rows match each aggregate in each batch.</li>
-</ol>
-<div class="codehilite"><pre><span></span><code> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&a [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="p">...</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="err">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">┃</span> <span class="err">&boxv;</span>
- <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;& [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;</span><span class="k">group</span> <span class="k">values</span><span class="p">:</span> <span class="n">OwnedRow</span> <span class="err">&boxv;</span> <span class="err"& [...]
- <span class="err">&boxv;</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">5</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&am [...]
- <span class="err">&boxv;</span> <span class="err">&boxvr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxvl;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;</span><span class="k">Row</span> <span class="n">accumulator</span><span class="p">:</span> &l [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">9</span> <span class="err">&boxv;&boxh;&boxvh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;</span><span class="n">Vec</span><span class="o">&lt;</span><span cl [...]
- <span class="err">&boxv;</span> <span class="err">&boxvr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxvl;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&am [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="p">...</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&box [...]
- <span class="err">&boxv;</span> <span class="err">&boxvr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxvl;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&am [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="mi">1</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;&boxv;</span><span class="n">Accumulator</span> <span [...]
- <span class="err">&boxv;</span> <span class="err">&boxvr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxvl;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&am [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="p">...</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&box [...]
- <span class="err">&boxv;</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;&boxv;</span><span class="n">Accumulator</span> <span class="mi">2</span& [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="err">& [...]
- <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;</span> <span class="n">Box</span><span class="o">&lt;</span><span class="n">dyn</span> <span class="n">Accu [...]
- <span class="n">Hash</span> <span class="k">Table</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&bo [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="e [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;</span><span class="n">scratch</span> <span class="n">indices</span><span class="p">:</span> <span class="n">Vec</span><span class="o">&lt;</span><span class="n">u32</span><span class="o">&gt;</span>& [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="e [...]
- <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="n">GroupState</span> <span class="err">┃</span> <span class="err">&boxv;</span>
- <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;▶</span> <span class="err">&boxv;</span> <span class="err">┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛</span> <span class="err">&boxv;</span>
- <span class="err">&boxv;</span> <span class="err">&boxv;</span>
- <span class="n">Hash</span> <span class="k">table</span> <span class="n">tracks</span> <span class="n">an</span> <span class="err">&boxv;</span> <span class="p">...</span> <span class="err">&boxv;</span>
- <span class="k">index</span> <span class="k">into</span> <span class="n">group_states</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span>
- <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span>
- <span class="n">group_states</span><span class="p">:</span> <span class="n">Vec</span><span class="o">&lt;</span><span class="n">GroupState</span><span class="o">&gt;</span>
-
- <span class="n">There</span> <span class="k">is</span> <span class="n">one</span> <span class="n">GroupState</span> <span class="n">PER</span> <span class="k">GROUP</span>
-</code></pre></div>
-<p><strong>Figure 4</strong>: Hash group operator structure in DataFusion <code>27.0.0</code>. A hash table maps each group to a GroupState which contains all the per-group states.</p>
-<p>To compute the aggregate, DataFusion performs the following steps for each input batch:</p>
-<ol>
-<li>Calculate hash using <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/hash_utils.rs#L264-L307">efficient vectorized code</a>, specialized for each data type.</li>
-<li>Determine group indexes for each input row using the hash table (creating new entries for newly seen groups).</li>
-<li><a href="https://github.com/apache/arrow-datafusion/blob/4ab8be57dee3bfa72dd105fbd7b8901b873a4878/datafusion/core/src/physical_plan/aggregates/row_hash.rs#L562-L602">Update Accumulators for each group that had input rows,</a> assembling the rows into a contiguous range for vectorized accumulator if there are a sufficient number of them.</li>
-</ol>
-<p>DataFusion also stores the hash values in the table to avoid potentially costly hash recomputation when resizing the hash table.</p>
-<p>This scheme works very well for a relatively small number of distinct groups: all accumulators are efficiently updated with large contiguous batches of rows.</p>
-<p>However, this scheme is not ideal for high cardinality grouping due to:</p>
-<ol>
-<li><strong>Multiple allocations per group</strong> for the group value row format, as well as for the <code>RowAccumulator</code>s and each <code>Accumulator</code>. The <code>Accumulator</code> may have additional allocations within it as well.</li>
-<li><strong>Non-vectorized updates:</strong> Accumulator updates often fall back to a slower non-vectorized form because the number of distinct groups is large (and thus number of values per group is small) in each input batch.</li>
-</ol>
-<h3>Hash grouping in <code>28.0.0</code></h3>
-<p>For <code>28.0.0</code>, we rewrote the core group by implementation following traditional system optimization principles: fewer allocations, type specialization, and aggressive vectorization.</p>
-<p>DataFusion <code>28.0.0</code> uses the same RawTable and still stores group indexes. The major differences, as shown in Figure 4, are:</p>
-<ol>
-<li>Group values are stored either<ol>
-<li>Inline in the <code>RawTable</code> (for single columns of primitive types), where the conversion to Row format costs more than its benefit</li>
-<li>In a separate <a href="https://docs.rs/arrow-row/latest/arrow_row/struct.Row.html">Rows</a> structure with a single contiguous allocation for all groups values, rather than an allocation per group. Accumulators manage the state for all the groups internally, so the code to update intermediate values is a tight type specialized loop. The new <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/s [...]
-</ol>
-</li>
-</ol>
-<div class="codehilite"><pre><span></span><code><span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> [...]
-<span class="err">&boxv;</span> <span class="err">&boxdr;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxdl;</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh; [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class= [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxdr;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxdl;&boxdr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxdl;</span> <span class="err"&g [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="n">X</span> <span class="err">&boxv;</span> <span class="mi">5</span> <span class="err">&boxv;</span> <span class="err">&boxv;&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;&boxv;</spa [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxvr;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxvl;&boxvr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxvl;</span> <span class="err"&g [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="n">Q</span> <span class="err">&boxv;</span> <span class="mi">9</span> <span class="err">&boxv;&boxh;&boxh;&boxvh;&boxvh;&boxh;&boxh;&boxdl;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;</sp [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxvr;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxvl;&boxvr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxvl;</span> <span class="err"&g [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="p">...</span> <span class="err">&boxv;</span> <span class="p">...</span> <span class="err">&boxv;</span> <span class="err">&boxv;&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;&boxdr;&box [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxvr;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxvl;&boxvr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxvl;</span> <span class="err"&g [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="n">H</span> <span class="err">&boxv;</span> <span class="mi">1</span> <span class="err">&boxv;</span> <span class="err">&boxv;&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="err">&boxv;&boxur;&bo [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxvr;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxvl;&boxvr;&boxh;&boxh;&boxh;&boxh;&boxh;&boxvl;</span> <span class="err"&g [...]
-<span class="err">&boxv;</span> <span class="k">Rows</span> <span class="err">&boxv;</span> <span class="p">...</span> <span class="err">&boxv;</span> <span class="p">...</span> <span class="err">&boxv;</span> <span class="err">&boxv;&boxv;</span> <span class="err">&boxv;</span> <span class="err">┃</span> <span class="er [...]
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxur;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxul;&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="err"&g [...]
-<span class="err">&boxv;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxh;</span> <span class="err">&boxv;</span> <span class="err">&boxv;&boxv;</span> <span class="err">&am [...]
-<span class="err">&boxv;</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;&boxv;</span> <span class="err">&boxv;</span> <span class="err">┗━━━━━━━━━━━━━━━━━━━┛</span> <span class="err">&boxv;</span>
-<span class="err">&boxv;</span> <span class="n">Hash</span> <span class="k">Table</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span>
-<span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="err">&boxv;</span> <span class="p">...</span> <span class="err">&boxv;</span>
-<span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxh;&boxul;</span> <span class="err">&boxur;&boxh;&boxh;&boxh;&boxh;&am [...]
- <span class="n">GroupState</span> <span class="n">Accumulators</span>
-
-
-<span class="n">Hash</span> <span class="k">table</span> <span class="n">value</span> <span class="n">stores</span> <span class="n">group_indexes</span> <span class="n">One</span> <span class="n">GroupsAccumulator</span>
-<span class="k">and</span> <span class="k">group</span> <span class="k">values</span><span class="p">.</span> <span class="n">per</span> <span class="k">aggregate</span><span class="p">.</span> <span class="k">Each</span>
- <span class="n">stores</span> <span class="n">the</span> <span class="k">state</span> <span class="k">for</span>
-<span class="k">Group</span> <span class="k">values</span> <span class="k">are</span> <span class="n">stored</span> <span class="n">either</span> <span class="n">inline</span> <span class="o">*</span><span class="k">ALL</span><span class="o">*</span> <span class="n">groups</span><span class="p">,</span> <span class="n">typically</span>
-<span class="k">in</span> <span class="n">the</span> <span class="n">hash</span> <span class="k">table</span> <span class="k">or</span> <span class="k">in</span> <span class="n">a</span> <span class="n">single</span> <span class="k">using</span> <span class="n">a</span> <span class="n">native</span> <span class="n">Vec</span><span cla [...]
-<span class="n">allocation</span> <span class="k">using</span> <span class="n">the</span> <span class="n">arrow</span> <span class="k">Row</span> <span class="n">format</span>
-</code></pre></div>
-<p><strong>Figure 5</strong>: Hash group operator structure in DataFusion <code>28.0.0</code>. Group values are stored either directly in the hash table, or in a single allocation using the arrow Row format. The hash table contains group indexes. A single <code>GroupsAccumulator</code> stores the per-aggregate state for <em>all</em> groups.</p>
-<p>This new structure improves performance significantly for high cardinality groups due to:</p>
-<ol>
-<li><strong>Reduced allocations</strong>: There are no longer any individual allocations per group.</li>
-<li><strong>Contiguous native accumulator states</strong>: Type-specialized accumulators store the values for all groups in a single contiguous allocation using a <a href="https://doc.rust-lang.org/std/vec/struct.Vec.html">Rust Vec&lt;T&gt;</a> of some native type.</li>
-<li><strong>Vectorized state update</strong>: The inner aggregate update loops, which are type-specialized and in terms of native <code>Vec</code>s, are well-vectorized by the Rust compiler (thanks <a href="https://llvm.org/">LLVM</a>!).</li>
-</ol>
-<h3>Notes</h3>
-<p>Some vectorized grouping implementations store the accumulator state row-wise directly in the hash table, which often uses modern CPU caches efficiently. Managing accumulator state in columnar fashion may sacrifice some cache locality, however it ensures the size of the hash table remains small, even when there are large numbers of groups and aggregates, making it easier for the compiler to vectorize the accumulator update.</p>
-<p>Depending on the cost of recomputing hash values, DataFusion <code>28.0.0</code> may or may not store the hash values in the table. This optimizes the tradeoff between the cost of computing the hash value (which is expensive for strings, for example) vs. the cost of storing it in the hash table.</p>
-<p>One subtlety that arises from pushing state updates into GroupsAccumulators is that each accumulator must handle similar variations with/without filtering and with/without nulls in the input. DataFusion <code>28.0.0</code> uses a templated <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/aggregate/groups_accumulator/accumulate.rs#L28-L54"><code>NullState</code></a> w [...]
-<p>The code structure is heavily influenced by the fact DataFusion is implemented using <a href="https://www.rust-lang.org/">Rust</a>, a new(ish) systems programming language focused on speed and safety. Rust heavily discourages many of the traditional pointer casting &ldquo;tricks&rdquo; used in C/C++ hash grouping implementations. The DataFusion aggregation code is almost entirely <a href="https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html#:~:text= [...]
-<h2>ClickBench results</h2>
-<p>The full results of running the <a href="https://github.com/ClickHouse/ClickBench/tree/main">ClickBench</a> queries against the single Parquet file with DataFusion <code>27.0.0</code>, DataFusion <code>28.0.0</code>, and DuckDB <code>0.8.1</code> are below. These numbers were run on a GCP <code>e2-standard-8 machine</code> with 8 cores and 32 GB of RAM, using the scripts <a href="https://github.com/alamb/datafusion-d [...]
-<p>As the industry moves towards data systems assembled from components, it is increasingly important that they exchange data using open standards such as <a href="https://arrow.apache.org/">Apache Arrow</a> and <a href="https://parquet.apache.org/">Parquet</a> rather than custom storage and in-memory formats. Thus, this benchmark uses a single input Parquet file representative of many DataFusion users and aligned with the current trend in analytics of avoid [...]
-<p>DataFusion now reaches near-DuckDB-speeds querying Parquet data. While we don&rsquo;t plan to engage in a benchmarking shootout with a team that literally wrote <a href="https://dl.acm.org/doi/abs/10.1145/3209950.3209955">Fair Benchmarking Considered Difficult</a>, hopefully everyone can agree that DataFusion <code>28.0.0</code> is a significant improvement.</p>
-<p><img src="../images/datafusion_fast_grouping/full.png" width="700"/></p>
-<p><strong>Figure 6</strong>: Performance of DataFusion <code>27.0.0</code>, DataFusion <code>28.0.0</code>, and DuckDB <code>0.8.1</code> on all 43 ClickBench queries against a single <code>hits.parquet</code> file. Lower is better.</p>
-<h3>Notes</h3>
-<p>DataFusion <code>27.0.0</code> was not able to run several queries due to either planner bugs (Q9, Q11, Q12, 14) or running out of memory (Q33). DataFusion <code>28.0.0</code> solves those issues.</p>
-<p>DataFusion is faster than DuckDB for query 21 and 22, likely due to optimized implementations of string pattern matching.</p>
-<h2>Conclusion: performance matters</h2>
-<p>Improving aggregation performance by more than a factor of two allows developers building products and projects with DataFusion to spend more time on value-added domain specific features. We believe building systems with DataFusion is much faster than trying to build something similar from scratch. DataFusion increases productivity because it eliminates the need to rebuild well-understood, but costly to implement, analytic database technology. While we&rsquo;re pleased with [...]
-<h2>Acknowledgments</h2>
-<p>DataFusion is a <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">community effort</a> and this work was not possible without contributions from many in the community. A special shout out to <a href="https://github.com/sunchao">sunchao</a>, <a href="https://github.com/jyshen">yjshen</a>, <a href="https://github.com/yahoNanJing">yahoNanJing</a>, <a href="https://github.com/mingmwang">mingmwang& [...]
-<h2>About DataFusion</h2>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database toolkit, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org/">Apache Arrow</a> as its in-memory format. DataFusion, along with <a href="https://calcite.apache.org/">Apache Calcite</a>, Facebook&rsquo;s <a href="https://github.com/facebookincubator/velox">V [...]
-<!-- Footnotes themselves at the bottom. -->
-<h2>Notes</h2>
-<p>[^1]: <code>SELECT COUNT(*) FROM 'hits.parquet';</code></p>
-<p>[^2]: <code>SELECT COUNT(DISTINCT "UserID") as num_users FROM 'hits.parquet';</code></p>
-<p>[^3]: <code>SELECT COUNT(DISTINCT "SearchPhrase") as num_phrases FROM 'hits.parquet';</code></p>
-<p>[^4]: <code>SELECT COUNT(*) FROM (SELECT DISTINCT "UserID", "SearchPhrase" FROM 'hits.parquet')</code></p>
-<p>[^5]: Full script at <a href="https://github.com/alamb/datafusion-duckdb-benchmark/blob/main/hash.py">hash.py</a></p>
-<p>[^6]: <a href="https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_%7B%7D.parquet">hits_0.parquet</a>, one of the files from the partitioned ClickBench dataset, which has <code>100,000</code> rows and is 117 MB in size. The entire dataset has <code>100,000,000</code> rows in a single 14 GB Parquet file. The script did not complete on the entire dataset after 40 minutes, and used 212 GB RAM at peak.</p></content><categ [...]
\ No newline at end of file
diff --git a/output/feeds/alamb-dandandan-tustvold.rss.xml b/output/feeds/alamb-dandandan-tustvold.rss.xml
deleted file mode 100644
index 9f51683..0000000
--- a/output/feeds/alamb-dandandan-tustvold.rss.xml
+++ /dev/null
@@ -1,25 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog - alamb, Dandandan, tustvold</title><link>https://datafusion.staged.apache.org/</link><description></description><lastBuildDate>Sat, 05 Aug 2023 00:00:00 +0000</lastBuildDate><item><title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0</title><link>https://datafusion.staged.apache.org/2023/08/05/datafusion_fast_grouping</link><description><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<!-- Converted from Google Docs using https://www.buymeacoffee.com/docstomarkdown -->
-<h2>Aggregating Millions of Groups Fast in Apache Arrow DataFusion</h2>
-<p>Andrew Lamb, Dani&euml;l Heres, Raphael Taylor-Davies,</p>
-<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
-<h2>TLDR</h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>&rsquo;s parallel aggregation capability …</p></description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">alamb, Dandandan, tustvold</dc:creator><pubDate>Sat, 05 Aug 2023 00:00:00 +0000</pubDate><guid isPermaLink="false">tag:datafusion.staged.apache.org,2023-08- [...]
\ No newline at end of file
diff --git a/output/feeds/all-en.atom.xml b/output/feeds/all-en.atom.xml
deleted file mode 100644
index f80fbff..0000000
--- a/output/feeds/all-en.atom.xml
+++ /dev/null
@@ -1,4462 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog</title><link href="https://datafusion.staged.apache.org/" rel="alternate"></link><link href="https://datafusion.staged.apache.org/feeds/all-en.atom.xml" rel="self"></link><id>https://datafusion.staged.apache.org/</id><updated>2024-11-20T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache DataFusion Comet 0.4.0 Release</title><link href="https://datafusion.staged.apache.org/2024/11/20/datafusion-comet-0. [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to …</p></summary><content type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately six weeks of development work and is the result of merging 51 PRs from 10
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.4.0.md">change log</a> for more information.</p>
-<h2>Release Highlights</h2>
-<h3>Performance &amp; Stability</h3>
-<p>There are a number of performance and stability improvements in this release. Here is a summary of some of the
-larger changes. Current benchmarking results can be found in the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>.</p>
-<h4>Unified Memory Management</h4>
-<p>Comet now uses a unified memory management approach that shares an off-heap memory pool with Apache Spark, resulting
-in a much simpler configuration. Comet now requires <code>spark.memory.offHeap.enabled=true</code>. This approach provides a
-holistic view of memory usage in Spark and Comet and makes it easier to optimize system performance.</p>
-<h4>Faster Joins</h4>
-<p>Apache Spark supports sort-merge and hash joins, which have similar performance characteristics. Spark defaults to
-using sort-merge joins because they are less likely to result in OutOfMemory exceptions. In vectorized query
-engines such as DataFusion, hash joins outperform sort-merge joins. Comet now has an experimental feature to
-replace Spark sort-merge joins with hash joins for improved performance. This feature is experimental because
-there is currently no spill-to-disk support in the hash join implementation. This feature can be enabled by
-setting <code>spark.comet.exec.replaceSortMergeJoin=true</code>.</p>
-<h4>Bloom Filter Aggregates</h4>
-<p>Spark&rsquo;s optimizer can insert Bloom filter aggregations and filters to prune large result sets before a shuffle. However,
-Comet would fall back to Spark for the aggregation. Comet now has native support for Bloom filter aggregations
-after previously supporting Bloom filter testing. Users no longer need to set
-<code>spark.sql.optimizer.runtime.bloomFilter.enabled=false</code> when using Comet.</p>
-<h4>Complex Type support</h4>
-<p>This release has the following improvements to complex type support:</p>
-<ul>
-<li>Implemented <code>ArrayAppend</code> and <code>GetArrayStructFields</code>.</li>
-<li>Implemented native cast between structs</li>
-<li>Implemented native cast from structs to string</li>
-</ul>
-<h2>Roadmap</h2>
-<p>One of the highest priority items on the roadmap is to add support for reading complex types (maps, structs, and arrays)
-from Parquet sources, both when reading Parquet directly and from Iceberg.</p>
-<p>Comet currently has proprietary native code for decoding Parquet pages, native column readers for all of Spark&rsquo;s
-primitive types, and special handling for Spark-specific use cases such as timestamp rebasing and decimal type
-promotion. This implementation does not yet support complex types. File IO, decryption, and decompression are handled
-in JVM code, and Parquet pages are passed on to native code for decoding.</p>
-<p>Rather than add complex type support to this existing code, we are exploring two main options to allow us to
-leverage more of the upstream Arrow and DataFusion code.</p>
-<h3>Use DataFusion&rsquo;s ParquetExec</h3>
-<p>For use cases where DataFusion can support reading a Parquet source, Comet could create a native plan that uses
-DataFusion&rsquo;s ParquetExec. We are investigating using DataFusion&rsquo;s SchemaAdapter to handle some Spark-specific
-handling of timestamps and decimals.</p>
-<h3>Use Arrow&rsquo;s Parquet Batch Reader</h3>
-<p>For use cases not supported by DataFusion&rsquo;s ParquetExec, such as integrating with Iceberg, we are exploring
-replacing our current native Parquet decoding logic with the Arrow readers provided by the Parquet crate.</p>
-<p>Iceberg already provides a vectorized Spark reader for Parquet. A <a href="https://github.com/apache/iceberg/pull/9841">PR</a> is open against Iceberg for adding a native
-version based on Comet, and we hope to update this to leverage the improvements outlined above.</p>
-<h2>Getting Involved</h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p></content><category term="blog"></category></entry><entry><title>Comparing approaches to User Defined Functions in Apache DataFusion using Python</title><link href="https://datafusion.staged.apache.org/2024/11/19/datafusion-python-udf-comparisons" rel="alternate"></link><published>2024-11-19T00:00:00+00:00</published><updated>2 [...]
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Personal Context</h2>
-<p>For a few months now I&rsquo;ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
-fast query engine written in Rust. From my experience the language that nearly all data scientists
-are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger …</p></summary><content type="html"><!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h2>Personal Context</h2>
-<p>For a few months now I&rsquo;ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
-fast query engine written in Rust. From my experience the language that nearly all data scientists
-are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger tasks that require
-distributed processing.</p>
-<p>In addition to DataFusion, there is another Rust based newcomer to the DataFrame world,
-<a href="https://pola.rs/">Polars</a>. The latter is growing extremely fast, and it serves many of the same
-use cases as DataFusion. For my use cases, I'm interested in DataFusion because I want to be able
-to build small scale tests rapidly and then scale them up to larger distributed systems with ease.
-I do recommend evaluating Polars for in-memory work.</p>
-<p>Personally, I would love a single query approach that is fast for both in-memory usage and can
-extend to large batch processing to exploit parallelization. I think DataFusion, coupled with
-<a href="https://datafusion.apache.org/ballista/">Ballista</a> or
-<a href="https://github.com/apache/datafusion-ray">DataFusion-Ray</a>, may provide this solution.</p>
-<p>As I&rsquo;m testing, I&rsquo;m primarily limiting my work to the
-<a href="https://datafusion.apache.org/python/">datafusion-python</a> project, a wrapper around the Rust
-DataFusion library. This wrapper gives you the speed advantages of keeping all of the data in the
-Rust implementation and the ergonomics of working in Python. Personally, I would prefer to work
-purely in Rust, but I also recognize that since the industry works in Python we should meet the
-people where they are.</p>
-<h2>User-Defined Functions</h2>
-<p>The focus of this post is User-Defined Functions (UDFs). The DataFusion library gives a lot of
-useful functions already for doing DataFrame manipulation. These are going to be similar to those
-you find in other DataFrame libraries. You&rsquo;ll be able to do simple arithmetic, create substrings of
-columns, or find the average value across a group of rows. These cover most of the use cases
-you&rsquo;ll need in a DataFrame.</p>
-<p>However, there will always arise times when you want a custom function. With UDFs you open a
-world of possibilities in your code. Sometimes there simply isn&rsquo;t an easy way to use built-in
-functions to achieve your goals.</p>
-<p>In the following, I&rsquo;m going to demonstrate two example use cases. These are based on real world
-problems I&rsquo;ve encountered. Also I want to demonstrate the approach of &ldquo;make it work, make it work
-well, make it work fast&rdquo; that is a motto I&rsquo;ve seen thrown around in data science.</p>
-<p>I will demonstrate three approaches to writing UDFs. In order of increasing performance they are</p>
-<ul>
-<li>Writing a pure Python function to do your computation</li>
-<li>Using the PyArrow libraries in Python to accelerate your processing</li>
-<li>Writing a UDF in Rust and exposing it to Python</li>
-</ul>
-<p>Additionally I will demonstrate two variants of this. The first will be nearly identical to the
-PyArrow library approach to simplify understanding how to connect the Rust code to Python. In the
-second version we will do the iteration through the input arrays ourselves to give even greater
-flexibility to the user.</p>
-<p>Here are the two example use cases, taken from my own work but generalized.</p>
-<h3>Use Case 1: Scalar Function</h3>
-<p>I have a DataFrame and a list of tuples that I&rsquo;m interested in. I want to filter out the DataFrame
-to only have values that match those tuples from certain columns in the DataFrame.</p>
-<p>To give a concrete example, we will use data generated for the <a href="https://www.tpc.org/tpch/">TPC-H benchmarks</a>.
-Suppose I have a table of sales line items. There are many columns, but I am interested in three: a
-part key (<code>p_partkey</code>), supplier key (<code>p_suppkey</code>), and return status (<code>p_returnflag</code>). I want
-only to return a DataFrame with a specific combination of these three values. That is, I want
-to know if part number 1530 from supplier 4031 was sold (not returned), so I want a specific
-combination of <code>p_partkey = 1530</code>, <code>p_suppkey = 4031</code>, and <code>p_returnflag = 'N'</code>. I have a small
-handful of these combinations I want to return.</p>
-<p>Probably the most ergonomic way to do this without UDF is to turn that list of tuples into a
-DataFrame itself, perform a join, and select the columns from the original DataFrame. If we were
-working in PySpark we would probably broadcast join the DataFrame created from the tuple list since
-it is tiny. In practice, I have found that with some DataFrame libraries performing a filter rather
-than a join can be significantly faster. This is worth profiling for your specific use case.</p>
-<h3>Use Case 2: Aggregate Function</h3>
-<p>I have a DataFrame with many values that I want to aggregate. I have already analyzed it and
-determined there is a noise level below which I do not want to include in my analysis. I want to
-compute a sum of only values that are above my noise threshold.</p>
-<p>This can be done fairly easy without leaning on a User Defined Aggegate Function (UDAF). You can
-simply filter the DataFrame and then aggregate using the built-in <code>sum</code> function. Here, we
-demonstrate doing this as a UDF primarily as an example of how to write UDAFs. We will use the
-PyArrow compute approach.</p>
-<h2>Pure Python approach</h2>
-<p>The fastest way (developer time, not code time) for me to implement the scalar problem solution
-was to do something along the lines of &ldquo;for each row, check the values of interest contains that
-tuple&rdquo;. I&rsquo;ve published this as
-<a href="https://github.com/apache/datafusion-python/blob/main/examples/python-udf-comparisons.py">an example</a>
-in the <a href="https://github.com/apache/datafusion-python">datafusion-python repository</a>. Here is an
-example of how this can be done:</p>
-<div class="codehilite"><pre><span></span><code><span class="n">values_of_interest</span> <span class="o">=</span> <span class="p">[</span>
- <span class="p">(</span><span class="mi">1530</span><span class="p">,</span> <span class="mi">4031</span><span class="p">,</span> <span class="s2">"N"</span><span class="p">),</span>
- <span class="p">(</span><span class="mi">6530</span><span class="p">,</span> <span class="mi">1531</span><span class="p">,</span> <span class="s2">"N"</span><span class="p">),</span>
... 31733 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org