You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by al...@apache.org on 2026/05/12 15:08:26 UTC
(datafusion-site) branch asf-site updated: Remove extra copy of content
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e852342 Remove extra copy of content
e852342 is described below
commit e8523421429560a59d05c94a7cce91679e8f3a31
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Tue May 12 11:08:19 2026 -0400
Remove extra copy of content
---
blog/2019/02/04/datafusion-donation/index.html | 217 -
blog/2021/04/12/ballista-donation/index.html | 166 -
blog/2021/08/18/ballista-0.5.0/index.html | 182 -
blog/2021/08/18/datafusion-5.0.0/index.html | 196 -
.../11/19/2021-11-8-datafusion-6.0.0.md/index.html | 234 -
blog/2022/02/28/datafusion-7.0.0/index.html | 240 -
blog/2022/03/21/datafusion-contrib/index.html | 236 -
blog/2022/05/16/datafusion-8.0.0/index.html | 301 -
blog/2022/10/25/datafusion-13.0.0/index.html | 348 -
blog/2022/10/28/ballista-0.9.0/index.html | 216 -
blog/2023/01/19/datafusion-16.0.0/index.html | 361 -
blog/2023/06/24/datafusion-25.0.0/index.html | 333 -
.../2023/08/05/datafusion_fast_grouping/index.html | 449 -
blog/2024/01/19/datafusion-34.0.0/index.html | 392 -
blog/2024/03/06/comet-donation/index.html | 176 -
blog/2024/05/07/datafusion-tlp/index.html | 176 -
blog/2024/07/20/datafusion-comet-0.1.0/index.html | 214 -
blog/2024/07/24/datafusion-40.0.0/index.html | 451 -
.../2024/08/20/python-datafusion-40.0.0/index.html | 286 -
blog/2024/08/28/datafusion-comet-0.2.0/index.html | 203 -
.../index.html | 243 -
.../index.html | 230 -
blog/2024/09/27/datafusion-comet-0.3.0/index.html | 192 -
.../index.html | 341 -
.../datafusion-python-udf-comparisons/index.html | 687 -
blog/2024/11/20/datafusion-comet-0.4.0/index.html | 222 -
.../2024/12/14/datafusion-python-43.1.0/index.html | 255 -
blog/2025/01/17/datafusion-comet-0.5.0/index.html | 205 -
.../02/02/datafusion-ballista-43.0.0/index.html | 198 -
blog/2025/02/17/datafusion-comet-0.6.0/index.html | 186 -
blog/2025/02/20/datafusion-45.0.0/index.html | 306 -
blog/2025/03/11/ordering-analysis/index.html | 526 -
blog/2025/03/20/datafusion-comet-0.7.0/index.html | 214 -
blog/2025/03/20/parquet-pruning/index.html | 200 -
blog/2025/03/21/parquet-pushdown/index.html | 383 -
blog/2025/03/24/datafusion-46.0.0/index.html | 229 -
.../2025/03/30/datafusion-python-46.0.0/index.html | 331 -
blog/2025/04/10/fastest-tpch-generator/index.html | 573 -
.../04/19/user-defined-window-functions/index.html | 465 -
blog/2025/05/06/datafusion-comet-0.8.0/index.html | 193 -
.../optimizing-sql-dataframes-part-one/index.html | 321 -
.../optimizing-sql-dataframes-part-two/index.html | 522 -
blog/2025/06/30/cancellation/index.html | 519 -
blog/2025/07/01/datafusion-comet-0.9.0/index.html | 259 -
blog/2025/07/11/datafusion-47.0.0/index.html | 353 -
.../07/14/user-defined-parquet-indexes/index.html | 621 -
blog/2025/07/16/datafusion-48.0.0/index.html | 308 -
blog/2025/07/28/datafusion-49.0.0/index.html | 428 -
.../2025/08/15/external-parquet-indexes/index.html | 721 -
blog/2025/09/10/dynamic-filters/index.html | 697 -
blog/2025/09/16/datafusion-comet-0.10.0/index.html | 227 -
.../09/21/custom-types-using-metadata/index.html | 378 -
blog/2025/09/29/datafusion-50.0.0/index.html | 461 -
blog/2025/10/21/datafusion-comet-0.11.0/index.html | 271 -
blog/2025/11/25/datafusion-51.0.0/index.html | 351 -
blog/2025/12/04/datafusion-comet-0.12.0/index.html | 238 -
.../15/avoid-consecutive-repartitions/index.html | 347 -
blog/2026/01/12/datafusion-52.0.0/index.html | 367 -
blog/2026/01/12/extending-sql/index.html | 424 -
blog/2026/01/30/datafusion-comet-0.13.0/index.html | 230 -
blog/2026/02/02/datafusion_case/index.html | 505 -
blog/2026/03/18/datafusion-comet-0.14.0/index.html | 213 -
blog/2026/03/20/limit-pruning/index.html | 415 -
blog/2026/03/31/writing-table-providers/index.html | 997 --
blog/2026/04/02/datafusion-53.0.0/index.html | 291 -
blog/2026/04/18/datafusion-comet-0.15.0/index.html | 315 -
blog/about.html | 68 -
blog/author/aditya-singh-rathore-andrew-lamb.html | 64 -
...-badaracco-pydantic-andrew-lamb-influxdata.html | 74 -
blog/author/agrove.html | 96 -
blog/author/alamb-akurmustafa.html | 104 -
blog/author/alamb-dandandan-tustvold.html | 68 -
.../andrew-lamb-achraf-b-and-sean-smith.html | 65 -
blog/author/andrew-lamb-influxdata.html | 69 -
.../andrew-lamb-staff-engineer-at-influxdata.html | 67 -
blog/author/gene-bordegaray.html | 65 -
blog/author/geoffrey-claude-datadog.html | 64 -
blog/author/milenkovicm.html | 66 -
blog/author/mustafa-akur-andrew-lamb.html | 67 -
...anci-and-berkay-sahin-on-behalf-of-the-pmc.html | 61 -
blog/author/pepijn-van-eeckhoudt.html | 134 -
blog/author/pmc.html | 1249 --
...at-tu-darmstadt-and-andrew-lamb-influxdata.html | 63 -
blog/author/tim-saucer-rerunio.html | 66 -
...-dunningtonwherobots-andrew-lambinfluxdata.html | 66 -
blog/author/timsaucer.html | 163 -
blog/author/xiangpeng-hao-andrew-lamb.html | 95 -
blog/author/xiangpeng-hao.html | 115 -
blog/author/xudong.html | 84 -
blog/category/blog.html | 2252 ---
blog/css/app.css | 27 -
blog/css/blog_index.css | 20 -
blog/css/bootstrap.min.css | 6 -
blog/css/dark-mode.css | 100 -
blog/css/font-awesome.min.css | 4 -
blog/css/fontawesome.all.min.css | 6 -
blog/css/headerlink.css | 16 -
blog/css/webfonts/fa-brands-400.ttf | Bin 186112 -> 0 bytes
blog/css/webfonts/fa-brands-400.woff2 | Bin 107460 -> 0 bytes
blog/css/webfonts/fa-regular-400.ttf | Bin 62048 -> 0 bytes
blog/css/webfonts/fa-regular-400.woff2 | Bin 25096 -> 0 bytes
blog/css/webfonts/fa-solid-900.ttf | Bin 397728 -> 0 bytes
blog/css/webfonts/fa-solid-900.woff2 | Bin 150472 -> 0 bytes
blog/css/webfonts/fa-v4compatibility.ttf | Bin 10136 -> 0 bytes
blog/css/webfonts/fa-v4compatibility.woff2 | Bin 4584 -> 0 bytes
blog/favicon.ico | Bin 8894 -> 0 bytes
blog/feed.xml | 1559 --
.../aditya-singh-rathore-andrew-lamb.atom.xml | 355 -
.../feeds/aditya-singh-rathore-andrew-lamb.rss.xml | 22 -
...aracco-pydantic-andrew-lamb-influxdata.atom.xml | 575 -
...daracco-pydantic-andrew-lamb-influxdata.rss.xml | 32 -
blog/feeds/agrove.atom.xml | 193 -
blog/feeds/agrove.rss.xml | 44 -
blog/feeds/alamb-akurmustafa.atom.xml | 603 -
blog/feeds/alamb-akurmustafa.rss.xml | 52 -
blog/feeds/alamb-dandandan-tustvold.atom.xml | 325 -
blog/feeds/alamb-dandandan-tustvold.rss.xml | 26 -
blog/feeds/all-en.atom.xml | 14874 ----------------
.../andrew-lamb-achraf-b-and-sean-smith.atom.xml | 454 -
.../andrew-lamb-achraf-b-and-sean-smith.rss.xml | 23 -
blog/feeds/andrew-lamb-influxdata.atom.xml | 592 -
blog/feeds/andrew-lamb-influxdata.rss.xml | 27 -
...drew-lamb-staff-engineer-at-influxdata.atom.xml | 228 -
...ndrew-lamb-staff-engineer-at-influxdata.rss.xml | 25 -
blog/feeds/blog.atom.xml | 14874 ----------------
blog/feeds/gene-bordegaray.atom.xml | 268 -
blog/feeds/gene-bordegaray.rss.xml | 23 -
blog/feeds/geoffrey-claude-datadog.atom.xml | 286 -
blog/feeds/geoffrey-claude-datadog.rss.xml | 22 -
blog/feeds/milenkovicm.atom.xml | 94 -
blog/feeds/milenkovicm.rss.xml | 24 -
blog/feeds/mustafa-akur-andrew-lamb.atom.xml | 395 -
blog/feeds/mustafa-akur-andrew-lamb.rss.xml | 25 -
...-and-berkay-sahin-on-behalf-of-the-pmc.atom.xml | 106 -
...i-and-berkay-sahin-on-behalf-of-the-pmc.rss.xml | 19 -
blog/feeds/pepijn-van-eeckhoudt.atom.xml | 804 -
blog/feeds/pepijn-van-eeckhoudt.rss.xml | 82 -
blog/feeds/pmc.atom.xml | 6004 -------
blog/feeds/pmc.rss.xml | 847 -
...u-darmstadt-and-andrew-lamb-influxdata.atom.xml | 492 -
...tu-darmstadt-and-andrew-lamb-influxdata.rss.xml | 21 -
blog/feeds/tim-saucer-rerunio.atom.xml | 829 -
blog/feeds/tim-saucer-rerunio.rss.xml | 24 -
...ningtonwherobots-andrew-lambinfluxdata.atom.xml | 272 -
...nningtonwherobots-andrew-lambinfluxdata.rss.xml | 24 -
blog/feeds/timsaucer.atom.xml | 1106 --
blog/feeds/timsaucer.rss.xml | 91 -
blog/feeds/xiangpeng-hao-andrew-lamb.atom.xml | 242 -
blog/feeds/xiangpeng-hao-andrew-lamb.rss.xml | 43 -
blog/feeds/xiangpeng-hao.atom.xml | 380 -
blog/feeds/xiangpeng-hao.rss.xml | 63 -
blog/feeds/xudong.atom.xml | 313 -
blog/feeds/xudong.rss.xml | 42 -
blog/highlight/default.min.css | 9 -
blog/highlight/highlight.js | 17194 -------------------
blog/images/2021-08-18-datafusion500perf.png | Bin 21245 -> 0 bytes
blog/images/2x_bgwhite_original.png | Bin 60451 -> 0 bytes
.../basic_before_query_plan.png | Bin 389431 -> 0 bytes
.../database_system_diagram.png | Bin 60386 -> 0 bytes
.../hash_repartitioning.png | Bin 34315 -> 0 bytes
.../hash_repartitioning_example.png | Bin 285156 -> 0 bytes
.../in_depth_before_query_plan.png | Bin 213614 -> 0 bytes
.../logic_tree_after.png | Bin 267069 -> 0 bytes
.../logic_tree_before.png | Bin 266326 -> 0 bytes
.../noot_noot_database_meme.png | Bin 810185 -> 0 bytes
.../optimal_query_plans.png | Bin 170114 -> 0 bytes
.../round_robin_repartitioning.png | Bin 31484 -> 0 bytes
.../tpch10_benchmark.png | Bin 144141 -> 0 bytes
.../tpch_benchmark.png | Bin 139612 -> 0 bytes
.../volcano_model_diagram.png | Bin 232350 -> 0 bytes
blog/images/case/evaluate_selection.svg | 4 -
blog/images/case/merge.svg | 4 -
blog/images/case/merge_n.svg | 4 -
blog/images/case/merging.svg | 4 -
blog/images/case/no_projection.svg | 4 -
blog/images/case/original_loop.svg | 4 -
blog/images/case/projection.svg | 4 -
blog/images/case/results.png | Bin 111593 -> 0 bytes
.../column-based-storage.png | Bin 47390 -> 0 bytes
.../clickbench-datafusion-43/perf-over-time.png | Bin 62185 -> 0 bytes
blog/images/clickbench-datafusion-43/perf.png | Bin 57326 -> 0 bytes
.../clickbench-datafusion-43/row-based-storage.png | Bin 45024 -> 0 bytes
.../skipping-partial-aggregation.png | Bin 133362 -> 0 bytes
.../clickbench-datafusion-43/string-view-take.png | Bin 66566 -> 0 bytes
blog/images/comet-0.1.0/tpch_allqueries.png | Bin 26393 -> 0 bytes
blog/images/comet-0.1.0/tpch_queries_compare.png | Bin 29269 -> 0 bytes
blog/images/comet-0.1.0/tpch_queries_speedup.png | Bin 42444 -> 0 bytes
blog/images/comet-0.11.0/tpch_allqueries.png | Bin 32703 -> 0 bytes
blog/images/comet-0.11.0/tpch_queries_compare.png | Bin 34547 -> 0 bytes
blog/images/comet-0.15.0/tpch_allqueries.png | Bin 23347 -> 0 bytes
blog/images/comet-0.15.0/tpch_queries_compare.png | Bin 26102 -> 0 bytes
blog/images/comet-0.2.0/tpcds_allqueries.png | Bin 24925 -> 0 bytes
blog/images/comet-0.2.0/tpch_allqueries.png | Bin 25972 -> 0 bytes
blog/images/comet-0.5.0/tpch_allqueries.png | Bin 28138 -> 0 bytes
blog/images/comet-0.5.0/tpch_queries_compare.png | Bin 33059 -> 0 bytes
blog/images/comet-0.7.0/performance.png | Bin 34131 -> 0 bytes
blog/images/comet-0.9.0/tracing.png | Bin 153447 -> 0 bytes
blog/images/datafusion-34.0.0/compare-new.png | Bin 176796 -> 0 bytes
blog/images/datafusion-34.0.0/compare.png | Bin 149835 -> 0 bytes
.../datafusion-40.0.0/improved-planning-time.png | Bin 47905 -> 0 bytes
.../datafusion-45.0.0/performance_over_time.png | Bin 52136 -> 0 bytes
.../datafusion-46.0.0/diagnostic-example.png | Bin 140405 -> 0 bytes
.../datafusion-47.0.0/datafusion-telemetry.png | Bin 142838 -> 0 bytes
.../performance_over_time_clickbench.png | Bin 64862 -> 0 bytes
.../performance_over_time_planning.png | Bin 11393 -> 0 bytes
.../performance_over_time_clickbench.png | Bin 63544 -> 0 bytes
.../arrow-57-metadata-parsing.png | Bin 78434 -> 0 bytes
.../performance_over_time_clickbench.png | Bin 61910 -> 0 bytes
.../datafusion-53.0.0/field-access-pushdown.svg | 58 -
.../datafusion-53.0.0/join-filter-pushdown.svg | 99 -
.../performance_over_time_clickbench.png | Bin 57883 -> 0 bytes
.../datafusion-ballista-43.0.0/ballista-logo.png | Bin 65501 -> 0 bytes
.../datafusion-ballista-43.0.0/tpch_allqueries.png | Bin 27455 -> 0 bytes
.../tpch_queries_compare.png | Bin 32843 -> 0 bytes
.../tpch_queries_speedup_rel.png | Bin 47169 -> 0 bytes
.../images/datafusion-comet/comet-architecture.png | Bin 30027 -> 0 bytes
blog/images/datafusion_fast_grouping/full.png | Bin 50146 -> 0 bytes
blog/images/datafusion_fast_grouping/summary.png | Bin 52822 -> 0 bytes
blog/images/dynamic-filters/execution-time.svg | 1 -
blog/images/dynamic-filters/join-performance.svg | 1 -
blog/images/dynamic-filters/query-plan-naive.png | Bin 111826 -> 0 bytes
.../query-plan-topk-dynamic-filters.png | Bin 160390 -> 0 bytes
blog/images/dynamic-filters/query-plan-topk.png | Bin 112692 -> 0 bytes
blog/images/extending-sql/architecture.svg | 151 -
.../external-index-overview.png | Bin 244699 -> 0 bytes
.../parquet-filter-pushdown.png | Bin 236537 -> 0 bytes
.../external-parquet-indexes/parquet-layout.png | Bin 244089 -> 0 bytes
.../external-parquet-indexes/parquet-metadata.png | Bin 302935 -> 0 bytes
.../processing-pipeline.png | Bin 95998 -> 0 bytes
.../external-parquet-indexes/prune-files.png | Bin 344697 -> 0 bytes
.../external-parquet-indexes/prune-row-groups.png | Bin 221498 -> 0 bytes
blog/images/fastest-tpch-generator/lamb-theory.png | Bin 300479 -> 0 bytes
.../fastest-tpch-generator/parquet-performance.png | Bin 61946 -> 0 bytes
.../fastest-tpch-generator/tbl-performance.png | Bin 49477 -> 0 bytes
blog/images/limit-pruning/before-after.svg | 74 -
.../limit-pruning/fully-matched-detection.svg | 54 -
.../limit-pruning/limit-rewrite-algorithm.svg | 71 -
blog/images/limit-pruning/pruning-phases.svg | 77 -
blog/images/limit-pruning/pruning-pipeline.svg | 52 -
blog/images/limit-pruning/row-group-states.svg | 75 -
blog/images/limit-pruning/wasted-io.svg | 101 -
blog/images/logo_original4x.png | Bin 119116 -> 0 bytes
.../metadata-handling/arrow_record_batch.png | Bin 224968 -> 0 bytes
.../access-path-and-join-order.png | Bin 39660 -> 0 bytes
.../common-subexpression-elimination.png | Bin 131011 -> 0 bytes
.../optimizing-sql-dataframes/filter-pushdown.png | Bin 60561 -> 0 bytes
.../initial-logical-plan.png | Bin 81521 -> 0 bytes
.../optimizing-sql-dataframes/join-rewrite.png | Bin 81151 -> 0 bytes
.../optimizing-sql-dataframes/limit-pushdown.png | Bin 105326 -> 0 bytes
.../optimized-logical-plan.png | Bin 104882 -> 0 bytes
.../optimizing-sql-dataframes/optimizer-passes.png | Bin 39157 -> 0 bytes
.../projection-pushdown.png | Bin 36469 -> 0 bytes
.../optimizing-sql-dataframes/query-execution.png | Bin 70217 -> 0 bytes
.../specialized-grouping.png | Bin 119133 -> 0 bytes
blog/images/optimizing-sql-dataframes/table1.png | Bin 61798 -> 0 bytes
.../optimizing-sql-dataframes/using-statistics.png | Bin 107342 -> 0 bytes
.../images/ordering_analysis/query_window_plan.png | Bin 189377 -> 0 bytes
blog/images/parquet-pruning/filter-pushdown.jpg | Bin 376764 -> 0 bytes
blog/images/parquet-pruning/read-parquet.jpg | Bin 284097 -> 0 bytes
blog/images/parquet-pushdown/baseline-impl.jpg | Bin 166560 -> 0 bytes
blog/images/parquet-pushdown/cached-pages.jpg | Bin 37963 -> 0 bytes
blog/images/parquet-pushdown/new-pipeline.jpg | Bin 106800 -> 0 bytes
blog/images/parquet-pushdown/parquet-viewer.jpg | Bin 344238 -> 0 bytes
.../parquet-pushdown/pushdown-vs-no-pushdown.jpg | Bin 194575 -> 0 bytes
.../pylance_error_checking.png | Bin 39119 -> 0 bytes
.../vscode_hover_tooltip.png | Bin 87320 -> 0 bytes
.../python-datafusion-46.0.0/html_rendering.png | Bin 189126 -> 0 bytes
blog/images/string-view-1/figure1-performance.png | Bin 153348 -> 0 bytes
blog/images/string-view-1/figure2-string-view.png | Bin 197178 -> 0 bytes
blog/images/string-view-1/figure4-copying.png | Bin 115588 -> 0 bytes
.../string-view-1/figure5-loading-strings.png | Bin 68999 -> 0 bytes
.../string-view-1/figure6-utf8-validation.png | Bin 187490 -> 0 bytes
blog/images/string-view-1/figure7-end-to-end.png | Bin 64736 -> 0 bytes
.../string-view-2/figure1-zero-copy-take.png | Bin 95659 -> 0 bytes
blog/images/string-view-2/figure2-filter-time.png | Bin 52437 -> 0 bytes
blog/images/task-cancellation/merge_plan.png | Bin 21711 -> 0 bytes
blog/images/task-cancellation/tokio_budget.png | Bin 221115 -> 0 bytes
.../images/task-cancellation/tokio_budget_plan.png | Bin 27840 -> 0 bytes
.../custom_index_structures.png | Bin 235929 -> 0 bytes
.../standard_index_structures.png | Bin 123351 -> 0 bytes
blog/index.html | 2883 ----
blog/js/bootstrap.bundle.min.js | 7 -
blog/js/bootstrap.bundle.min.js.map | 1 -
blog/js/dark-mode.js | 46 -
blog/js/giscus-consent.js | 115 -
285 files changed, 96720 deletions(-)
diff --git a/blog/2019/02/04/datafusion-donation/index.html b/blog/2019/02/04/datafusion-donation/index.html
deleted file mode 100644
index 8a31bb0..0000000
--- a/blog/2019/02/04/datafusion-donation/index.html
+++ /dev/null
@@ -1,217 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>DataFusion: A Rust-native Query Engine for Apache Arrow - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- DataFusion: A Rust-native Query Engine for Apache Arrow
- </h1>
- <p>Posted on: Mon 04 February 2019 by agrove</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#example">Example</a></li>
-<li><a href="#roadmap">Roadmap</a></li>
-<li><a href="#contributors-welcome">Contributors Welcome!</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion">DataFusion</a> has been donated to the Apache Arrow project. DataFusion is an in-memory query engine for the Rust implementation of Apache Arrow.</p>
-<p>Although DataFusion was started two years ago, it was recently re-implemented to be Arrow-native and currently has limited capabilities but does support SQL queries against iterators of RecordBatch and has support for CSV files. There are plans to <a href="https://issues.apache.org/jira/browse/ARROW-4466">add support for Parquet files</a>.</p>
-<p>SQL support is limited to projection (<code>SELECT</code>), selection (<code>WHERE</code>), and simple aggregates (<code>MIN</code>, <code>MAX</code>, <code>SUM</code>) with an optional <code>GROUP BY</code> clause.</p>
-<p>Supported expressions are identifiers, literals, simple math operations (<code>+</code>, <code>-</code>, <code>*</code>, <code>/</code>), binary expressions (<code>AND</code>, <code>OR</code>), equality and comparison operators (<code>=</code>, <code>!=</code>, <code><</code>, <code><=</code>, <code>>=</code>, <code>></code>), and <code>CAST(expr AS type)</code>.</p>
-<h2 id="example">Example<a class="headerlink" href="#example" title="Permanent link">¶</a></h2>
-<p>The following example demonstrates running a simple aggregate SQL query against a CSV file.</p>
-<pre><code class="language-rust">// create execution context
-let mut ctx = ExecutionContext::new();
-
-// define schema for data source (csv file)
-let schema = Arc::new(Schema::new(vec![
- Field::new("c1", DataType::Utf8, false),
- Field::new("c2", DataType::UInt32, false),
- Field::new("c3", DataType::Int8, false),
- Field::new("c4", DataType::Int16, false),
- Field::new("c5", DataType::Int32, false),
- Field::new("c6", DataType::Int64, false),
- Field::new("c7", DataType::UInt8, false),
- Field::new("c8", DataType::UInt16, false),
- Field::new("c9", DataType::UInt32, false),
- Field::new("c10", DataType::UInt64, false),
- Field::new("c11", DataType::Float32, false),
- Field::new("c12", DataType::Float64, false),
- Field::new("c13", DataType::Utf8, false),
-]));
-
-// register csv file with the execution context
-let csv_datasource =
- CsvDataSource::new("test/data/aggregate_test_100.csv", schema.clone(), 1024);
-ctx.register_datasource("aggregate_test_100", Rc::new(RefCell::new(csv_datasource)));
-
-let sql = "SELECT c1, MIN(c12), MAX(c12) FROM aggregate_test_100 WHERE c11 > 0.1 AND c11 < 0.9 GROUP BY c1";
-
-// execute the query
-let relation = ctx.sql(&sql).unwrap();
-let mut results = relation.borrow_mut();
-
-// iterate over the results
-while let Some(batch) = results.next().unwrap() {
- println!(
- "RecordBatch has {} rows and {} columns",
- batch.num_rows(),
- batch.num_columns()
- );
-
- let c1 = batch
- .column(0)
- .as_any()
- .downcast_ref::<BinaryArray>()
- .unwrap();
-
- let min = batch
- .column(1)
- .as_any()
- .downcast_ref::<Float64Array>()
- .unwrap();
-
- let max = batch
- .column(2)
- .as_any()
- .downcast_ref::<Float64Array>()
- .unwrap();
-
- for i in 0..batch.num_rows() {
- let c1_value: String = String::from_utf8(c1.value(i).to_vec()).unwrap();
- println!("{}, Min: {}, Max: {}", c1_value, min.value(i), max.value(i),);
- }
-}
-</code></pre>
-<h2 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">¶</a></h2>
-<p>The roadmap for DataFusion will depend on interest from the Rust community, but here are some of the short term items that are planned:</p>
-<ul>
-<li>Extending test coverage of the existing functionality</li>
-<li>Adding support for Parquet data sources</li>
-<li>Implementing more SQL features such as <code>JOIN</code>, <code>ORDER BY</code> and <code>LIMIT</code></li>
-<li>Implement a DataFrame API as an alternative to SQL</li>
-<li>Adding support for partitioning and parallel query execution using Rust's async and await functionality</li>
-<li>Creating a Docker image to make it easy to use DataFusion as a standalone query tool for interactive and batch queries</li>
-</ul>
-<h2 id="contributors-welcome">Contributors Welcome!<a class="headerlink" href="#contributors-welcome" title="Permanent link">¶</a></h2>
-<p>If you are excited about being able to use Rust for data science and would like to contribute to this work then there are many ways to get involved. The simplest way to get started is to try out DataFusion against your own data sources and file bug reports for any issues that you find. You could also check out the current <a href="https://cwiki.apache.org/confluence/display/ARROW/Rust+JIRA+Dashboard">list of issues</a> and have a go at fixing one. You can also join the <a href="http:/ [...]
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#example">Example</a></li>
-<li><a href="#roadmap">Roadmap</a></li>
-<li><a href="#contributors-welcome">Contributors Welcome!</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2021/04/12/ballista-donation/index.html b/blog/2021/04/12/ballista-donation/index.html
deleted file mode 100644
index 7ca22ac..0000000
--- a/blog/2021/04/12/ballista-donation/index.html
+++ /dev/null
@@ -1,166 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Ballista: A Distributed Scheduler for Apache Arrow - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Ballista: A Distributed Scheduler for Apache Arrow
- </h1>
- <p>Posted on: Mon 12 April 2021 by agrove</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#status">Status</a></li>
-<li><a href="#contributors-welcome">Contributors Welcome!</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>We are excited to announce that <a href="https://github.com/apache/arrow-datafusion/tree/master/ballista">Ballista</a> has been donated
-to the Apache Arrow project. </p>
-<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is built
-on an architecture that allows other programming languages (such as Python, C++, and Java) to be supported as
-first-class citizens without paying a penalty for serialization costs.</p>
-<p>The foundational technologies in Ballista are:</p>
-<ul>
-<li><a href="https://arrow.apache.org/">Apache Arrow</a> memory model and compute kernels for efficient processing of data.</li>
-<li><a href="https://github.com/apache/arrow-datafusion">Apache Arrow DataFusion</a> query planning and
- execution framework, extended by Ballista to provide distributed planning and execution.</li>
-<li><a href="https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/">Apache Arrow Flight Protocol</a> for efficient
- data transfer between processes.</li>
-<li><a href="https://developers.google.com/protocol-buffers">Google Protocol Buffers</a> for serializing query plans.</li>
-<li><a href="https://www.docker.com/">Docker</a> for packaging up executors along with user-defined code.</li>
-</ul>
-<p>Ballista can be deployed as a standalone cluster and also supports <a href="https://kubernetes.io/">Kubernetes</a>. In either
-case, the scheduler can be configured to use <a href="https://etcd.io/">etcd</a> as a backing store to (eventually) provide
-redundancy in the case of a scheduler failing.</p>
-<h2 id="status">Status<a class="headerlink" href="#status" title="Permanent link">¶</a></h2>
-<p>The Ballista project is at an early stage of development. However, it is capable of running complex analytics queries
-in a distributed cluster with reasonable performance (comparable to more established distributed query frameworks).</p>
-<p>One of the benefits of Ballista being part of the Arrow codebase is that there is now an opportunity to push parts of
-the scheduler down to DataFusion so that is possible to seamlessly scale across cores in DataFusion, and across nodes
-in Ballista, using the same unified query scheduler.</p>
-<h2 id="contributors-welcome">Contributors Welcome!<a class="headerlink" href="#contributors-welcome" title="Permanent link">¶</a></h2>
-<p>If you are excited about being able to use Rust for distributed compute and ETL and would like to contribute to this
-work then there are many ways to get involved. The simplest way to get started is to try out Ballista against your own
-datasets and file bug reports for any issues that you find. You could also check out the current
-<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20component%20%3D%20%22Rust%20-%20Ballista%22">list of issues</a> and have a go at fixing one.</p>
-<p>The <a href="https://github.com/apache/arrow/blob/master/rust/README.md#arrow-rust-community">Arrow Rust Community</a>
-section of the Rust README provides information on other ways to interact with the Ballista contributors and
-maintainers.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#status">Status</a></li>
-<li><a href="#contributors-welcome">Contributors Welcome!</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2021/08/18/ballista-0.5.0/index.html b/blog/2021/08/18/ballista-0.5.0/index.html
deleted file mode 100644
index 9c79d12..0000000
--- a/blog/2021/08/18/ballista-0.5.0/index.html
+++ /dev/null
@@ -1,182 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow Ballista 0.5.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow Ballista 0.5.0 Release
- </h1>
- <p>Posted on: Wed 18 August 2021 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#performance-and-scalability">Performance and Scalability</a></li>
-<li><a href="#new-features">New Features</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>Ballista extends DataFusion to provide support for distributed queries. This is the first release of Ballista since
-the project was <a href="https://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to the Apache Arrow project
-and includes 80 commits from 11 contributors.</p>
-<pre><code>git shortlog -sn 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust/executor ballista/rust/scheduler
- 27 Andy Grove
- 15 Jiayu Liu
- 12 Andrew Lamb
- 8 Ximo Guanter
- 6 Daniël Heres
- 5 QP Hou
- 2 Jorge Leitao
- 1 Javier Goday
- 1 K.I. (Dennis) Jung
- 1 Mike Seddon
- 1 sathis
-</code></pre>
-<!--
-$ git log --pretty=oneline 4.0.0..5.0.0 ballista/rust/client ballista/rust/core ballista/rust/executor ballista/rust/scheduler ballista-examples/ | wc -l
-80
--->
-<p>The release notes below are not exhaustive and only expose selected highlights of the release. Many other bug fixes
-and improvements have been made: we refer you to the <a href="https://github.com/apache/arrow-datafusion/blob/5.0.0/ballista/CHANGELOG.md">complete changelog</a>.</p>
-<h1 id="performance-and-scalability">Performance and Scalability<a class="headerlink" href="#performance-and-scalability" title="Permanent link">¶</a></h1>
-<p>Ballista is now capable of running complex SQL queries at scale and supports scalable distributed joins. We have been
-benchmarking using individual queries from the TPC-H benchmark at scale factors up to 1000 (1 TB). When running against
-CSV files, performance is generally very close to DataFusion, and significantly faster in some cases due to the fact
-that the scheduler limits the number of concurrent tasks that run at any given time. Performance against large Parquet
-datasets is currently non ideal due to some issues (<a href="https://github.com/apache/arrow-datafusion/issues/867">#867</a>,
-<a href="https://github.com/apache/arrow-datafusion/issues/868">#868</a>) that we hope to resolve for the next release. </p>
-<h1 id="new-features">New Features<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h1>
-<p>The main new features in this release are:</p>
-<ul>
-<li>Ballista queries can now be executed by calling DataFrame.collect()</li>
-<li>The shuffle mechanism has been re-implemented</li>
-<li>Distributed hash-partitioned joins are now supported</li>
-<li>Keda autoscaling is supported</li>
-</ul>
-<p>To get started with Ballista, refer to the <a href="https://docs.rs/ballista/0.5.0/ballista/">crate documentation</a>.</p>
-<p>Now that the basic functionality is in place, the focus for the next release will be to improve the performance and
-scalability as well as improving the documentation.</p>
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>If you are interested in contributing to Ballista, we would love to have you! You
-can help by trying out Ballista on some of your own data and projects and filing bug reports and helping to
-improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
-beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#performance-and-scalability">Performance and Scalability</a></li>
-<li><a href="#new-features">New Features</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2021/08/18/datafusion-5.0.0/index.html b/blog/2021/08/18/datafusion-5.0.0/index.html
deleted file mode 100644
index 0db760d..0000000
--- a/blog/2021/08/18/datafusion-5.0.0/index.html
+++ /dev/null
@@ -1,196 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 5.0.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow DataFusion 5.0.0 Release
- </h1>
- <p>Posted on: Wed 18 August 2021 by pmc</p>
-
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<p>The Apache Arrow team is pleased to announce the DataFusion 5.0.0 release. This covers 4 months of development work
-and includes 211 commits from the following 31 distinct contributors.</p>
-<pre><code>$ git shortlog -sn 4.0.0..5.0.0 datafusion datafusion-cli datafusion-examples
- 61 Jiayu Liu
- 47 Andrew Lamb
- 27 Daniël Heres
- 13 QP Hou
- 13 Andy Grove
- 4 Javier Goday
- 4 sathis
- 3 Ruan Pearce-Authers
- 3 Raphael Taylor-Davies
- 3 Jorge Leitao
- 3 Cui Wenzheng
- 3 Mike Seddon
- 3 Edd Robinson
- 2 思维
- 2 Liang-Chi Hsieh
- 2 Michael Lu
- 2 Parth Sarthy
- 2 Patrick More
- 2 Rich
- 1 Charlie Evans
- 1 Gang Liao
- 1 Agata Naomichi
- 1 Ritchie Vink
- 1 Evan Chan
- 1 Ruihang Xia
- 1 Todd Treece
- 1 Yichen Wang
- 1 baishen
- 1 Nga Tran
- 1 rdettai
- 1 Marco Neumann
-</code></pre>
-<!--
-$ git log --pretty=oneline 4.0.0..5.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 211
--->
-<p>The release notes below are not exhaustive and only expose selected highlights of the release. Many other bug fixes
-and improvements have been made: we refer you to the complete
-<a href="https://github.com/apache/arrow-datafusion/blob/5.0.0/datafusion/CHANGELOG.md">changelog</a>.</p>
-<h1 id="performance">Performance<a class="headerlink" href="#performance" title="Permanent link">¶</a></h1>
-<p>There have been numerous performance improvements in this release. The following chart shows the relative
-performance of individual TPC-H queries compared to the previous release.</p>
-<p><i>TPC-H @ scale factor 100, in parquet format. Concurrency 24.</i></p>
-<p><img src="/blog/images/2021-08-18-datafusion500perf.png"/></p>
-<p>We also extended support for more TPC-H queries: q7, q8, q9 and q13 are running successfully in DataFusion 5.0.</p>
-<h1 id="new-features">New Features<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h1>
-<ul>
-<li>Initial support for SQL-99 Analytics (WINDOW functions)</li>
-<li>Improved JOIN support: cross join, semi-join, anti join, and fixes to null handling</li>
-<li>Improved EXPLAIN support</li>
-<li>Initial implementation of metrics in the physical plan</li>
-<li>Support for SELECT DISTINCT</li>
-<li>Support for Json and NDJson formatted inputs</li>
-<li>Query column with relations</li>
-<li>Added more datetime related functions: <code>now</code>, <code>date_trunc</code>, <code>to_timestamp_millis</code>, <code>to_timestamp_micros</code>, <code>to_timestamp_seconds</code></li>
-<li>Streaming Dataframe.collect</li>
-<li>Support table column aliases</li>
-<li>Answer count(*), min() and max() queries using only statistics</li>
-<li>Non-equi-join filters in JOIN conditions</li>
-<li>Modulus operation</li>
-<li>Support group by column positions</li>
-<li>Added constant folding query optimizer</li>
-<li>Hash partitioned aggregation</li>
-<li>Added <code>random</code> SQL function</li>
-<li>Implemented count distinct for floats and dictionary types</li>
-<li>Re-exported arrow and parquet crates in Datafusion</li>
-<li>General row group pruning logic that’s agnostic to storage format</li>
-</ul>
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>If you are interested in contributing to DataFusion, we would love to have you! You
-can help by trying out DataFusion on some of your own data and projects and filing bug reports and helping to
-improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
-beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html b/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html
deleted file mode 100644
index 5b274b3..0000000
--- a/blog/2021/11/19/2021-11-8-datafusion-6.0.0.md/index.html
+++ /dev/null
@@ -1,234 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 6.0.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow DataFusion 6.0.0 Release
- </h1>
- <p>Posted on: Fri 19 November 2021 by pmc</p>
-
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an embedded
-query engine which leverages the unique features of
-<a href="https://www.rust-lang.org/">Rust</a> and <a href="https://arrow.apache.org/">Apache
-Arrow</a> to provide a system that is high
-performance, easy to connect, easy to embed, and high quality.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers 4 months of development work
-and includes 134 commits from the following 28 distinct contributors.</p>
-<!--
-git log --pretty=oneline 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 134
-
-git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 29
-
- Carlos and xudong963 are same individual
--->
-<pre><code> 28 Andrew Lamb
- 26 Jiayu Liu
- 13 xudong963
- 9 rdettai
- 9 QP Hou
- 6 Matthew Turner
- 5 Daniël Heres
- 4 Guillaume Balaine
- 3 Francis Du
- 3 Marco Neumann
- 3 Jon Mease
- 3 Nga Tran
- 2 Yijie Shen
- 2 Ruihang Xia
- 2 Liang-Chi Hsieh
- 2 baishen
- 2 Andy Grove
- 2 Jason Tianyi Wang
- 1 Nan Zhu
- 1 Antoine Wendlinger
- 1 Krisztián Szűcs
- 1 Mike Seddon
- 1 Conner Murphy
- 1 Patrick More
- 1 Taehoon Moon
- 1 Tiphaine Ruy
- 1 adsharma
- 1 lichuan6
-</code></pre>
-<p>The release notes below are not exhaustive and only expose selected highlights of the release. Many other bug fixes
-and improvements have been made: we refer you to the complete
-<a href="https://github.com/apache/arrow-datafusion/blob/6.0.0/datafusion/CHANGELOG.md">changelog</a>.</p>
-<h1 id="new-website">New Website<a class="headerlink" href="#new-website" title="Permanent link">¶</a></h1>
-<p>Befitting a growing project, DataFusion now has its
-<a href="https://arrow.apache.org/datafusion/">own website</a> hosted as part of the
-main <a href="https://arrow.apache.org">Apache Arrow Website</a></p>
-<h1 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">¶</a></h1>
-<p>The community worked to gather their thoughts about where we are
-taking DataFusion into a public
-<a href="https://arrow.apache.org/datafusion/specification/roadmap.html">Roadmap</a>
-for the first time</p>
-<h1 id="new-features">New Features<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h1>
-<ul>
-<li>Runtime operator metrics collection framework</li>
-<li>Object store abstraction for unified access to local or remote storage</li>
-<li>Hive style table partitioning support, for Parquet, CSV, Avro and Json files</li>
-<li>DataFrame API support for: <code>except</code>, <code>intersect</code>, <code>show</code>, <code>limit</code> and window functions</li>
-<li>SQL</li>
-<li><code>EXPLAIN ANALYZE</code> with runtime metrics</li>
-<li><code>trim ( [ LEADING | TRAILING | BOTH ] [ FROM ] string text [, characters text ] )</code> syntax</li>
-<li>Postgres style regular expression matching operators <code>~</code>, <code>~*</code>, <code>!~</code>, and <code>!~*</code></li>
-<li>SQL set operators <code>UNION</code>, <code>INTERSECT</code>, and <code>EXCEPT</code></li>
-<li><code>cume_dist</code>, <code>percent_rank</code> window functions</li>
-<li><code>digest</code>, <code>blake2s</code>, <code>blake2b</code>, <code>blake3</code> crypto functions</li>
-<li>HyperLogLog based <code>approx_distinct</code></li>
-<li><code>is distinct from</code> and <code>is not distinct from</code></li>
-<li><code>CREATE TABLE AS SELECT</code></li>
-<li>Accessing elements of nested <code>Struct</code> and <code>List</code> columns (e.g. <code>SELECT struct_column['field_name'], array_column[0] FROM ...</code>)</li>
-<li>Boolean expressions in <code>CASE</code> statement</li>
-<li><code>DROP TABLE</code></li>
-<li><code>VALUES</code> List</li>
-<li>Postgres regex match operators</li>
-<li>Support for Avro format</li>
-<li>Support for <code>ScalarValue::Struct</code></li>
-<li>Automatic schema inference for CSV files</li>
-<li>Better interactive editing support in <code>datafusion-cli</code> as well as <code>psql</code> style commands such as <code>\d</code>, <code>\?</code>, and <code>\q</code></li>
-<li>Generic constant evaluation and simplification framework</li>
-<li>Added common subexpression eliminate query plan optimization rule</li>
-<li>Python binding 0.4.0 with all Datafusion 6.0.0 features</li>
-</ul>
-<p>With these new features, we are also now passing TPC-H queries 8, 13 and 21.</p>
-<p>For the full list of new features with their relevant PRs, see the
-<a href="https://github.com/apache/arrow-datafusion/blob/6.0.0/datafusion/CHANGELOG.md">enhancements section</a>
-in the changelog.</p>
-<h1 id="async-planning-and-decoupling-file-format-from-table-layout"><code>async</code> planning and decoupling file format from table layout<a class="headerlink" href="#async-planning-and-decoupling-file-format-from-table-layout" title="Permanent link">¶</a></h1>
-<p>Driven by the need to support Hive style table partitioning, @rdettai
-introduced the following design change to the Datafusion core.</p>
-<ul>
-<li>The code for reading specific file formats (<code>Parquet</code>, <code>Avro</code>, <code>CSV</code>, and
-<code>JSON</code>) was separated from the logic that handles grouping sets of
-files into execution partitions.</li>
-<li>The query planning process was made <code>async</code>.</li>
-</ul>
-<p>As a result, we are able to replace the old <code>Parquet</code>, <code>CSV</code> and <code>JSON</code> table
-providers with a single <code>ListingTable</code> table provider.</p>
-<p>This also sets up DataFusion and its plug-in ecosystem to
-supporting a wide range of catalogs and various object store implementations.
-You can read more about this change in the
-<a href="https://docs.google.com/document/d/1Bd4-PLLH-pHj0BquMDsJ6cVr_awnxTuvwNJuWsTHxAQ">design document</a>
-and on the <a href="https://github.com/apache/arrow-datafusion/pull/1010">arrow-datafusion#1010 PR</a>.</p>
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>If you are interested in contributing to DataFusion, we would love to have you! You
-can help by trying out DataFusion on some of your own data and projects and filing bug reports and helping to
-improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for
-beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>
-and the full list is <a href="https://github.com/apache/arrow-datafusion/issues">here</a>.</p>
-<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2022/02/28/datafusion-7.0.0/index.html b/blog/2022/02/28/datafusion-7.0.0/index.html
deleted file mode 100644
index 50a1274..0000000
--- a/blog/2022/02/28/datafusion-7.0.0/index.html
+++ /dev/null
@@ -1,240 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 7.0.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow DataFusion 7.0.0 Release
- </h1>
- <p>Posted on: Mon 28 February 2022 by pmc</p>
-
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth checking out.</p>
-<p>DataFusion's SQL, <code>DataFrame</code>, and manual <code>PlanBuilder</code> API let users access a sophisticated query optimizer and execution engine capable of fast, resource efficient, and parallel execution that takes optimal advantage of todays multicore hardware. Being written in Rust means DataFusion can offer <em>both</em> the safety of dynamic languages as well as the resource efficiency of a compiled language.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 7.0.0 release. This covers 4 months of development work
-and includes 195 commits from the following 37 distinct contributors.</p>
-<!--
-git log --pretty=oneline 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 134
-
-git shortlog -sn 5.0.0..6.0.0 datafusion datafusion-cli datafusion-examples | wc -l
- 29
-
- Carlos and xudong963 are same individual
--->
-<pre><code> 44 Andrew Lamb
- 24 Kun Liu
- 23 Jiayu Liu
- 17 xudong.w
- 11 Yijie Shen
- 9 Matthew Turner
- 7 Liang-Chi Hsieh
- 5 Lin Ma
- 4 Stephen Carman
- 4 James Katz
- 4 Dmitry Patsura
- 4 QP Hou
- 3 dependabot[bot]
- 3 Remzi Yang
- 3 Yang
- 3 ic4y
- 3 Daniël Heres
- 2 Andy Grove
- 2 Raphael Taylor-Davies
- 2 Jason Tianyi Wang
- 2 Dan Harris
- 2 Sergey Melnychuk
- 1 Nitish Tiwari
- 1 Dom
- 1 Eduard Karacharov
- 1 Javier Goday
- 1 Boaz
- 1 Marko Mikulicic
- 1 Max Burke
- 1 Carol (Nichols || Goulding)
- 1 Phillip Cloud
- 1 Rich
- 1 Toby Hede
- 1 Will Jones
- 1 r.4ntix
- 1 rdettai
-</code></pre>
-<p>The following section highlights some of the improvements in this release. Of course, many other bug fixes and improvements have also been made and we refer you to the complete <a href="https://github.com/apache/arrow-datafusion/blob/7.0.0/datafusion/CHANGELOG.md">changelog</a> for the full detail.</p>
-<h1 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h1>
-<ul>
-<li>DataFusion Crate</li>
-<li>The DataFusion crate is being split into multiple crates to decrease compilation times and improve the development experience. Initially, <code>datafusion-common</code> (the core DataFusion components) and <code>datafusion-expr</code> (DataFusion expressions, functions, and operators) have been split out. There will be additional splits after the 7.0 release.</li>
-<li>Performance Improvements and Optimizations</li>
-<li>Arrow’s dyn scalar kernels are now used to enable efficient operations on <code>DictionaryArray</code>s <a href="https://github.com/apache/arrow-datafusion/pull/1685">#1685</a></li>
-<li>Switch from <code>std::sync::Mutex</code> to <code>parking_lot::Mutex</code> <a href="https://github.com/apache/arrow-datafusion/pull/1720">#1720</a></li>
-<li>New Features</li>
-<li>Support for memory tracking and spilling to disk<ul>
-<li>MemoryManager and DiskManager <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
-<li>Out of core sort <a href="https://github.com/apache/arrow-datafusion/pull/1526">#1526</a></li>
-<li>New metrics</li>
-<li><code>Gauge</code> and <code>CurrentMemoryUsage</code> <a href="https://github.com/apache/arrow-datafusion/pull/1682">#1682</a></li>
-<li><code>Spill_count</code> and <code>spilled_bytes</code> <a href="https://github.com/apache/arrow-datafusion/pull/1641">#1641</a></li>
-</ul>
-</li>
-<li>New math functions<ul>
-<li><code>Approx_quantile</code> <a href="https://github.com/apache/arrow-datafusion/pull/1539">#1529</a></li>
-<li><code>stddev</code> and <code>variance</code> (sample and population) <a href="https://github.com/apache/arrow-datafusion/pull/1525">#1525</a></li>
-<li><code>corr</code> <a href="https://github.com/apache/arrow-datafusion/pull/1561">#1561</a></li>
-</ul>
-</li>
-<li>Support decimal type <a href="https://github.com/apache/arrow-datafusion/pull/1394">#1394</a><a href="https://github.com/apache/arrow-datafusion/pull/1407">#1407</a><a href="https://github.com/apache/arrow-datafusion/pull/1408">#1408</a><a href="https://github.com/apache/arrow-datafusion/pull/1431">#1431</a><a href="https://github.com/apache/arrow-datafusion/pull/1483">#1483</a><a href="https://github.com/apache/arrow-datafusion/pull/1554">#1554</a><a href="https://github.com/apache/ [...]
-<li>Support for reading Parquet files with evolved schemas <a href="https://github.com/apache/arrow-datafusion/pull/1622">#1622</a><a href="https://github.com/apache/arrow-datafusion/pull/1709">#1709</a></li>
-<li>Support for registering <code>DataFrame</code> as table <a href="https://github.com/apache/arrow-datafusion/pull/1699">#1699</a></li>
-<li>Support for the <code>substring</code> function <a href="https://github.com/apache/arrow-datafusion/pull/1621">#1621</a></li>
-<li>Support <code>array_agg(distinct ...)</code> <a href="https://github.com/apache/arrow-datafusion/pull/1579">#1579</a></li>
-<li>Support <code>sort</code> on unprojected columns <a href="https://github.com/apache/arrow-datafusion/pull/1415">#1415</a></li>
-<li>Additional Integration Points</li>
-<li>A new public Expression simplification API <a href="https://github.com/apache/arrow-datafusion/pull/1717">#1717</a></li>
-<li><a href="https://github.com/datafusion-contrib">DataFusion-Contrib</a></li>
-<li>A new GitHub organization created as a home for both <code>DataFusion</code> extensions and as a testing ground for new features.<ul>
-<li>Extensions</li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-python">DataFusion-Python</a></li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-java">DataFusion-Java</a></li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-hdfs-native">DataFusion-hdsfs-native</a></li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-objectstore-s3">DataFusion-ObjectStore-s3</a></li>
-<li>New Features</li>
-<li><a href="https://github.com/datafusion-contrib/datafusion-streams">DataFusion-Streams</a></li>
-</ul>
-</li>
-<li><a href="https://github.com/jorgecarleitao/arrow2">Arrow2</a></li>
-<li>An <a href="https://github.com/apache/arrow-datafusion/tree/arrow2">Arrow2 Branch</a> has been created. There are ongoing discussions in <a href="https://github.com/apache/arrow-datafusion/issues/1532">DataFusion</a> and <a href="https://github.com/apache/arrow-rs/issues/1176">arrow-rs</a> about migrating <code>DataFusion</code> to <code>Arrow2</code></li>
-</ul>
-<h1 id="documentation-and-roadmap">Documentation and Roadmap<a class="headerlink" href="#documentation-and-roadmap" title="Permanent link">¶</a></h1>
-<p>We are working to consolidate the documentation into the <a href="https://arrow.apache.org/datafusion">official site</a>. You can find more details there on topics such as the <a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL status</a> and a <a href="https://arrow.apache.org/datafusion/user-guide/introduction.html#introduction">user guide</a>. This is also an area we would love to get help from the broader community <a href="https://github.com/apache/arrow [...]
-<p>To provide transparency on DataFusion’s priorities to users and developers a three month roadmap will be published at the beginning of each quarter. This can be found here <a href="https://arrow.apache.org/datafusion/specification/roadmap.html">here</a>.</p>
-<h1 id="upcoming-attractions">Upcoming Attractions<a class="headerlink" href="#upcoming-attractions" title="Permanent link">¶</a></h1>
-<ul>
-<li>Ballista is gaining momentum, and several groups are now evaluating and contributing to the project.</li>
-<li>Some of the proposed improvements<ul>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/1701">Improvements Overview</a></li>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/1675">Extensibility</a></li>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/1702">File system access</a></li>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/1704">Cluster state</a></li>
-</ul>
-</li>
-<li>Continued improvements for working with limited resources and large datasets</li>
-<li>Memory limited joins<a href="https://github.com/apache/arrow-datafusion/issues/1599">#1599</a></li>
-<li>Sort-merge join<a href="https://github.com/apache/arrow-datafusion/issues/141">#141</a><a href="https://github.com/apache/arrow-datafusion/pull/1776">#1776</a></li>
-<li>Introduce row based bytes representation <a href="https://github.com/apache/arrow-datafusion/pull/1708">#1708</a></li>
-</ul>
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>If you are interested in contributing to DataFusion, and learning about state of
-the art query processing, we would love to have you join us on the journey! You
-can help by trying out DataFusion on some of your own data and projects and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
-<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2022/03/21/datafusion-contrib/index.html b/blog/2022/03/21/datafusion-contrib/index.html
deleted file mode 100644
index e68c342..0000000
--- a/blog/2022/03/21/datafusion-contrib/index.html
+++ /dev/null
@@ -1,236 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Introducing Apache Arrow DataFusion Contrib - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Introducing Apache Arrow DataFusion Contrib
- </h1>
- <p>Posted on: Mon 21 March 2022 by pmc</p>
-
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
-<p>Apache Arrow <a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>, a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth checking out. DataFusion's pluggable design makes creating extensions at various points particular easy to build.</p>
-<p>DataFusion's SQL, <code>DataFrame</code>, and manual <code>PlanBuilder</code> API let users access a sophisticated query optimizer and execution engine capable of fast, resource efficient, and parallel execution that takes optimal advantage of todays multicore hardware. Being written in Rust means DataFusion can offer <em>both</em> the safety of dynamic languages as well as the resource efficiency of a compiled language.</p>
-<p>The DataFusion team is pleased to announce the creation of the <a href="https://github.com/datafusion-contrib">DataFusion-Contrib</a> GitHub organization to support and accelerate other projects. While the core DataFusion library remains under Apache governance, the contrib organization provides a more flexible testing ground for new DataFusion features and a home for DataFusion extensions. With this announcement, we are pleased to introduce the following inaugural DataFusion-Contri [...]
-<h2 id="datafusion-python">DataFusion-Python<a class="headerlink" href="#datafusion-python" title="Permanent link">¶</a></h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-python">project</a> provides Python bindings to the core Rust implementation of DataFusion, which allows users to:</p>
-<ul>
-<li>Work with familiar SQL or DataFrame APIs to run queries in a safe, multi-threaded environment, returning results in Python</li>
-<li>Create User Defined Functions and User Defined Aggregate Functions for complex operations</li>
-<li>Pay no overhead to copy between Python and underlying Rust execution engine (by way of Apache Arrow arrays)</li>
-</ul>
-<h3 id="upcoming-enhancements">Upcoming enhancements<a class="headerlink" href="#upcoming-enhancements" title="Permanent link">¶</a></h3>
-<p>The team is focusing on exposing more features from the underlying Rust implementation of DataFusion and improving documentation.</p>
-<h3 id="how-to-install">How to install<a class="headerlink" href="#how-to-install" title="Permanent link">¶</a></h3>
-<p>From <code>pip</code></p>
-<pre><code class="language-bash">pip install datafusion
-</code></pre>
-<p>Or</p>
-<pre><code class="language-bash">python -m pip install datafusion
-</code></pre>
-<h2 id="datafusion-objectstore-s3">DataFusion-ObjectStore-S3<a class="headerlink" href="#datafusion-objectstore-s3" title="Permanent link">¶</a></h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-objectstore-s3">crate</a> provides an <code>ObjectStore</code> implementation for querying data stored in S3 or S3 compatible storage. This makes it almost as easy to query data that lives on S3 as lives in local files</p>
-<ul>
-<li>Ability to create <code>S3FileSystem</code> to register as part of DataFusion <code>ExecutionContext</code></li>
-<li>Register files or directories stored on S3 with <code>ctx.register_listing_table</code></li>
-</ul>
-<h3 id="upcoming-enhancements_1">Upcoming enhancements<a class="headerlink" href="#upcoming-enhancements_1" title="Permanent link">¶</a></h3>
-<p>The current priority is adding python bindings for <code>S3FileSystem</code>. After that there will be async improvements as DataFusion adopts more of that functionality and we are looking into S3 Select functionality.</p>
-<h3 id="how-to-install_1">How to Install<a class="headerlink" href="#how-to-install_1" title="Permanent link">¶</a></h3>
-<p>Add the below to your <code>Cargo.toml</code> in your Rust Project with DataFusion.</p>
-<pre><code class="language-toml">datafusion-objectstore-s3 = "0.1.0"
-</code></pre>
-<h2 id="datafusion-substrait">DataFusion-Substrait<a class="headerlink" href="#datafusion-substrait" title="Permanent link">¶</a></h2>
-<p><a href="https://substrait.io/">Substrait</a> is an emerging standard that provides a cross-language serialization format for relational algebra (e.g. expressions and query plans).</p>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-substrait">crate</a> provides a Substrait producer and consumer for DataFusion. A producer converts a DataFusion logical plan into a Substrait protobuf and a consumer does the reverse.</p>
-<p>Examples of how to use this crate can be found <a href="https://github.com/datafusion-contrib/datafusion-substrait/blob/main/src/lib.rs">here</a>.</p>
-<h3 id="potential-use-cases">Potential Use Cases<a class="headerlink" href="#potential-use-cases" title="Permanent link">¶</a></h3>
-<ul>
-<li>Replace custom DataFusion protobuf serialization.</li>
-<li>Make it easier to pass query plans over FFI boundaries, such as from Python to Rust</li>
-<li>Allow Apache Calcite query plans to be executed in DataFusion</li>
-</ul>
-<h2 id="datafusion-bigtable">DataFusion-BigTable<a class="headerlink" href="#datafusion-bigtable" title="Permanent link">¶</a></h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-bigtable">crate</a> implements <a href="https://cloud.google.com/bigtable">Bigtable</a> as a data source and physical executor for DataFusion queries. It currently supports both UTF-8 string and 64-bit big-endian signed integers in Bigtable. From a SQL perspective it supports both simple and composite row keys with <code>=</code>, <code>IN</code>, and <code>BETWEEN</code> operators as well as projection pushdown. The ph [...]
-<h3 id="upcoming-enhancements_2">Upcoming Enhancements<a class="headerlink" href="#upcoming-enhancements_2" title="Permanent link">¶</a></h3>
-<ul>
-<li>Predicate pushdown</li>
-<li>Value range</li>
-<li>Value Regex</li>
-<li>Timestamp range</li>
-<li>Multithreaded</li>
-<li>Partition aware execution</li>
-<li>Production ready</li>
-</ul>
-<h3 id="how-to-install_2">How to Install<a class="headerlink" href="#how-to-install_2" title="Permanent link">¶</a></h3>
-<p>Add the below to your <code>Cargo.toml</code> in your Rust Project with DataFusion.</p>
-<pre><code class="language-toml">datafusion-bigtable = "0.1.0"
-</code></pre>
-<h2 id="datafusion-hdfs">DataFusion-HDFS<a class="headerlink" href="#datafusion-hdfs" title="Permanent link">¶</a></h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-objectstore-hdfs">crate</a> introduces <code>HadoopFileSystem</code> as a remote <code>ObjectStore</code> which provides the ability to query HDFS files. For HDFS access the <a href="https://github.com/yahoNanJing/fs-hdfs">fs-hdfs</a> library is used.</p>
-<h2 id="datafusion-tokomak">DataFusion-Tokomak<a class="headerlink" href="#datafusion-tokomak" title="Permanent link">¶</a></h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-tokomak">crate</a> provides an e-graph based DataFusion optimization framework based on the Rust <a href="https://egraphs-good.github.io">egg</a> library. An e-graph is a data structure that powers the equality saturation optimization technique.</p>
-<p>As context, the optimizer framework within DataFusion is currently <a href="https://github.com/apache/arrow-datafusion/issues/1972">under review</a> with the objective of implementing a more strategic long term solution that is more efficient and simpler to develop.</p>
-<p>Some of the benefits of using <code>egg</code> within DataFusion are:</p>
-<ul>
-<li>Implements optimized algorithms that are hard to match with manually written optimization passes</li>
-<li>Makes it easy and less verbose to add optimization rules</li>
-<li>Plugin framework to add more complex optimizations</li>
-<li>Egg does not depend on rule order and can lead to a higher level of optimization by being able to apply multiple rules at the same time until it converges</li>
-<li>Allows for cost-based optimizations</li>
-</ul>
-<p>This is an exciting new area for DataFusion with lots of opportunity for community involvement!</p>
-<h2 id="datafusion-tui">DataFusion-Tui<a class="headerlink" href="#datafusion-tui" title="Permanent link">¶</a></h2>
-<p><a href="https://github.com/datafusion-contrib/datafusion-tui">DataFusion-tui</a> aka <code>dft</code> provides a feature rich terminal application for using DataFusion. It has drawn inspiration and several features from <code>datafusion-cli</code>. In contrast to <code>datafusion-cli</code> the objective of this tool is to provide a light SQL IDE experience for querying data with DataFusion. This includes features such as the following which are currently implemented:</p>
-<ul>
-<li>Tab Management to provide clean and structured organization of DataFusion queries, results, <code>ExecutionContext</code> information, and logs</li>
-<li>SQL Editor<ul>
-<li>Text editor for writing SQL queries</li>
-</ul>
-</li>
-<li>Query History<ul>
-<li>History of executed queries, their execution time, and the number of returned rows</li>
-</ul>
-</li>
-<li><code>ExecutionContext</code> information<ul>
-<li>Expose information on which physical optimizers are used and which <code>ExecutionConfig</code> settings are set</li>
-</ul>
-</li>
-<li>Logs<ul>
-<li>Logs from <code>dft</code>, DataFusion, and any dependent libraries</li>
-</ul>
-</li>
-<li>Support for custom <code>ObjectStore</code>s</li>
-<li>S3</li>
-<li>Preload DDL from <code>~/.datafusionrc</code> to enable having local "database" available at startup</li>
-</ul>
-<h3 id="upcoming-enhancements_3">Upcoming Enhancements<a class="headerlink" href="#upcoming-enhancements_3" title="Permanent link">¶</a></h3>
-<ul>
-<li>SQL Editor</li>
-<li>Command to write query results to file</li>
-<li>Multiple SQL editor tabs</li>
-<li>Expose more information from <code>ExecutionContext</code></li>
-<li>A help tab that provides information on functions</li>
-<li>Query custom <code>TableProvider</code>s such as <a href="https://github.com/delta-io/delta-rs">DeltaTable</a> or <a href="https://github.com/datafusion-contrib/datafusion-bigtable">BigTable</a></li>
-</ul>
-<h2 id="datafusion-streams">DataFusion-Streams<a class="headerlink" href="#datafusion-streams" title="Permanent link">¶</a></h2>
-<p><a href="https://github.com/datafusion-contrib/datafusion-streams">DataFusion-Stream</a> is a new testing ground for creating a <code>StreamProvider</code> in DataFusion that will enable querying streaming data sources such as Apache Kafka. The implementation for this feature is currently being designed and is under active review. Once the design is finalized the trait and attendant data structures will be added back to the core DataFusion crate.</p>
-<h2 id="datafusion-java">DataFusion-Java<a class="headerlink" href="#datafusion-java" title="Permanent link">¶</a></h2>
-<p>This <a href="https://github.com/datafusion-contrib/datafusion-java">project</a> created an initial set of Java bindings to DataFusion. The project is currently in maintenance mode and is looking for maintainers to drive future development.</p>
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>If you are interested in contributing to DataFusion, and learning about state of
-the art query processing, we would love to have you join us on the journey! You
-can help by trying out DataFusion on some of your own data and projects and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
-<p>The best way to find out about creating new extensions within DataFusion-Contrib is reaching out on the <code>#arrow-rust</code> channel of the Apache Software Foundation <a href="https://join.slack.com/t/the-asf/shared_invite/zt-vlfbf7ch-HkbNHiU_uDlcH_RvaHv9gQ">Slack</a> workspace.</p>
-<p>You can also check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more ways to engage with the community.</p>
-<p>Links for each DataFusion-Contrib repository are provided above if you would like to contribute to those.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2022/05/16/datafusion-8.0.0/index.html b/blog/2022/05/16/datafusion-8.0.0/index.html
deleted file mode 100644
index 681fa9f..0000000
--- a/blog/2022/05/16/datafusion-8.0.0/index.html
+++ /dev/null
@@ -1,301 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 8.0.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow DataFusion 8.0.0 Release
- </h1>
- <p>Posted on: Mon 16 May 2022 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#summary">Summary</a><ul>
-<li><a href="#ddl-support">DDL Support</a></li>
-<li><a href="#sql-support">SQL Support</a></li>
-<li><a href="#query-execution-internals">Query Execution & Internals</a></li>
-<li><a href="#improved-file-support">Improved file support</a></li>
-<li><a href="#ballista">Ballista</a></li>
-<li><a href="#upcoming-work">Upcoming Work</a></li>
-</ul>
-</li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible query execution framework, written in Rust, that
-uses Apache Arrow as its in-memory format.</p>
-<p>When you want to extend your Rust project with <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a>,
-a DataFrame API, or the ability to read and process Parquet, JSON, Avro or CSV data, DataFusion is definitely worth
-checking out.</p>
-<p>DataFusion's SQL, <code>DataFrame</code>, and manual <code>PlanBuilder</code> API let users access a sophisticated query optimizer and
-execution engine capable of fast, resource efficient, and parallel execution that takes optimal advantage of
-today's multicore hardware. Being written in Rust means DataFusion can offer <em>both</em> the safety of a dynamic language and
-the resource efficiency of a compiled language.</p>
-<p>The Apache Arrow team is pleased to announce the DataFusion 8.0.0 release (and also the release of version 0.7.0 of
-the Ballista subproject). This covers 3 months of development work and includes 279 commits from the following 49
-distinct contributors.</p>
-<!--
-$ git log --pretty=oneline 7.0.0..8.0.0 datafusion datafusion-cli datafusion-examples ballista ballista-cli ballista-examples | wc -l
-279
-
-$ git shortlog -sn 7.0.0..8.0.0 datafusion datafusion-cli datafusion-examples ballista ballista-cli ballista-examples | wc -l
-49
-
-(feynman han, feynman.h, Feynman Han were assumed to be the same person)
--->
-<pre><code> 39 Andy Grove
- 33 Andrew Lamb
- 21 DuRipeng
- 20 Yijie Shen
- 19 Yang Jiang
- 17 Raphael Taylor-Davies
- 11 Dan Harris
- 11 Matthew Turner
- 11 yahoNanJing
- 9 dependabot[bot]
- 8 jakevin
- 6 Kun Liu
- 5 Jiayu Liu
- 4 Daniël Heres
- 4 mingmwang
- 4 xudong.w
- 3 Carol (Nichols || Goulding)
- 3 Dmitry Patsura
- 3 Eduard Karacharov
- 3 Jeremy Dyer
- 3 Kaushik
- 3 Rich
- 3 comphead
- 3 gaojun2048
- 3 Feynman Han
- 2 Jie Han
- 2 Jon Mease
- 2 Tim Van Wassenhove
- 2 Yt
- 2 Zhang Li
- 2 silence-coding
- 1 Alexander Spies
- 1 George Andronchik
- 1 Guillaume Balaine
- 1 Hao Xin
- 1 Jiacai Liu
- 1 Jörn Horstmann
- 1 Liang-Chi Hsieh
- 1 Max Burke
- 1 NaincyKumariKnoldus
- 1 Nga Tran
- 1 Patrick More
- 1 Pierre Zemb
- 1 Remzi Yang
- 1 Sergey Melnychuk
- 1 Stephen Carman
- 1 doki
-</code></pre>
-<p>The following sections highlight some of the changes in this release. Of course, many other bug fixes and
-improvements have been made and we encourage you to check out the
-<a href="https://github.com/apache/arrow-datafusion/blob/8.0.0/datafusion/CHANGELOG.md">changelog</a> for full details.</p>
-<h1 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h1>
-<h2 id="ddl-support">DDL Support<a class="headerlink" href="#ddl-support" title="Permanent link">¶</a></h2>
-<p>DDL support has been expanded to include the following commands for creating databases, schemas, and views. This
-allows DataFusion to be used more effectively from the CLI.</p>
-<ul>
-<li><code>CREATE DATABASE</code></li>
-<li><code>CREATE VIEW</code></li>
-<li><code>CREATE SCHEMA</code></li>
-<li><code>CREATE EXTERNAL TABLE</code> now supports JSON files, <code>IF NOT EXISTS</code>, and partition columns</li>
-</ul>
-<h2 id="sql-support">SQL Support<a class="headerlink" href="#sql-support" title="Permanent link">¶</a></h2>
-<p>The SQL query planner now supports a number of new SQL features, including:</p>
-<ul>
-<li><em>Subqueries</em>: when used via <code>IN</code>, <code>EXISTS</code>, and as scalars</li>
-<li><em>Grouping Sets</em>: <code>CUBE</code> and <code>ROLLUP</code> grouping sets.</li>
-<li><em>Aggregate functions</em>: <code>approx_percentile</code>, <code>approx_percentile_cont</code>, <code>approx_percentile_cont_with_weight</code>, <code>approx_distinct</code>, <code>approx_median</code> and <code>array</code></li>
-<li><em><code>null</code> literals</em></li>
-<li><em>bitwise operations</em>: for example '<code>|</code>'</li>
-</ul>
-<p>There are also many bug fixes and improvements around normalizing identifiers consistently.</p>
-<p>We continue our tradition of incrementally releasing support for new
-features as they are developed. Thus, while the physical plan may not yet
-support all new features, it gets more complete each release. These
-changes also make DataFusion an increasingly compelling choice for
-projects looking for a SQL parser and query planner that can produce
-optimized logical plans that can be translated to
-their own execution engine.</p>
-<h2 id="query-execution-internals">Query Execution & Internals<a class="headerlink" href="#query-execution-internals" title="Permanent link">¶</a></h2>
-<p>There are several notable improvements and new features in the query execution engine:</p>
-<ul>
-<li>The <code>ExecutionContext</code> has been renamed to <code>SessionContext</code> and now supports multi-tenancy</li>
-<li>The <code>ExecutionPlan</code> trait is no longer <code>async</code></li>
-<li>A new serialization API for serializing plans to bytes (based on protobuf)</li>
-</ul>
-<p>In addition, we have added several foundational features to drive even
-more advanced query processing into DataFusion, focusing on running
-arbitrary queries larger than available memory, and pushing the
-envelope for performance of sorting, grouping, and joining even
-further:</p>
-<ul>
-<li>Morsel-Driven Scheduler based on <a href="https://15721.courses.cs.cmu.edu/spring2016/papers/p743-leis.pdf">"Morsel-Driven Parallelism: A NUMA-Aware Query
- Evaluation Framework for the Many-Core Age"</a></li>
-<li>Consolidated object store implementation and integration with parquet decoding</li>
-<li>Memory Limited Spilling sort operator</li>
-<li>Memory Limited Sort-Merge join operator</li>
-<li>High performance JIT code generation for tuple comparisons</li>
-<li>Memory efficient Row Format</li>
-</ul>
-<h2 id="improved-file-support">Improved file support<a class="headerlink" href="#improved-file-support" title="Permanent link">¶</a></h2>
-<p>DataFusion now supports JSON, both for reading and writing. There are also new DataFrame methods for writing query
-results to files in CSV, Parquet, and JSON format.</p>
-<h2 id="ballista">Ballista<a class="headerlink" href="#ballista" title="Permanent link">¶</a></h2>
-<p>Ballista continues to mature and now supports a wider range of operators and expressions. There are also improvements
-to the scheduler to support UDFs, and there are some robustness improvements, such as cleaning up work directories
-and persisting session configs to allow schedulers to restart and continue processing in-flight jobs.</p>
-<h2 id="upcoming-work">Upcoming Work<a class="headerlink" href="#upcoming-work" title="Permanent link">¶</a></h2>
-<p>Here are some of the initiatives that the community plans on working on prior to the next release.</p>
-<ul>
-<li>There is a <a href="https://docs.google.com/document/d/1jNRbadyStSrV5kifwn0khufAwq6OnzGczG4z8oTQJP4/edit?usp=sharing">proposal to move Ballista to its own top-level arrow-ballista repository</a>
- to decouple DataFusion and Ballista releases and to allow each project to have documentation better targeted at
- its particular audience.</li>
-<li>We plan on increasing the frequency of DataFusion releases, with monthly releases now instead of quarterly. This
- is driven by requests from the increasing number of projects that now depend on DataFusion.</li>
-<li>There is ongoing work to implement new optimizer rules to rewrite queries containing subquery expressions as
- joins, to support a wider range of queries.</li>
-<li>The new scheduler based on morsel-driven execution will continue to evolve in this next release, with work to
- refine IO abstractions to improve performance and integration with the new scheduler.</li>
-<li>Improved performance for Sort, Grouping and Joins</li>
-</ul>
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>If you are interested in contributing to DataFusion, and learning about state-of-the-art query processing, we would
-love to have you join us on the journey! You can help by trying out DataFusion on some of your own data and projects
-and let us know how it goes or contribute a PR with documentation, tests or code. A list of open issues suitable
-for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a></p>
-<p>Check out our new <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#summary">Summary</a><ul>
-<li><a href="#ddl-support">DDL Support</a></li>
-<li><a href="#sql-support">SQL Support</a></li>
-<li><a href="#query-execution-internals">Query Execution & Internals</a></li>
-<li><a href="#improved-file-support">Improved file support</a></li>
-<li><a href="#ballista">Ballista</a></li>
-<li><a href="#upcoming-work">Upcoming Work</a></li>
-</ul>
-</li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2022/10/25/datafusion-13.0.0/index.html b/blog/2022/10/25/datafusion-13.0.0/index.html
deleted file mode 100644
index a8478c8..0000000
--- a/blog/2022/10/25/datafusion-13.0.0/index.html
+++ /dev/null
@@ -1,348 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 13.0.0 Project Update - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow DataFusion 13.0.0 Project Update
- </h1>
- <p>Posted on: Tue 25 October 2022 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#background">Background</a></li>
-<li><a href="#summary">Summary</a></li>
-<li><a href="#improved-support-for-cloud-object-stores">Improved Support for Cloud Object Stores</a><ul>
-<li><a href="#advanced-sql">Advanced SQL</a></li>
-</ul>
-</li>
-<li><a href="#more-ddl-support">More DDL Support</a></li>
-<li><a href="#faster-execution">Faster Execution</a></li>
-<li><a href="#optimizer-enhancements">Optimizer Enhancements</a></li>
-<li><a href="#parquet">Parquet</a></li>
-<li><a href="#datatype-support">DataType Support</a><ul>
-<li><a href="#upcoming-work">Upcoming Work</a></li>
-</ul>
-</li>
-<li><a href="#community-growth">Community Growth</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a><ul>
-<li><a href="#appendix-contributor-shoutout">Appendix: Contributor Shoutout</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> <a href="https://crates.io/crates/datafusion"><code>13.0.0</code></a> is released, and this blog contains an update on the project for the 5 months since our <a href="https://arrow.apache.org/blog/2022/05/16/datafusion-8.0.0/">last update in May 2022</a>.</p>
-<p>DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL processes, and database systems. You may want to check out DataFusion to extend your Rust project to:</p>
-<ul>
-<li>Support <a href="https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html">SQL support</a></li>
-<li>Support <a href="https://docs.rs/datafusion/13.0.0/datafusion/dataframe/struct.DataFrame.html">DataFrame API</a></li>
-<li>Support a Domain Specific Query Language</li>
-<li>Easily and quickly read and process Parquet, JSON, Avro or CSV data.</li>
-<li>Read from remote object stores such as AWS S3, Azure Blob Storage, GCP.</li>
-</ul>
-<p>Even though DataFusion is 4 years "young," it has seen significant community growth in the last few months and the momentum continues to accelerate.</p>
-<h1 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h1>
-<p>DataFusion is used as the engine in <a href="https://github.com/apache/arrow-datafusion#known-uses">many open source and commercial projects</a> and was one of the early open source projects to provide this capability. 2022 has validated our belief in the need for such a <a href="https://docs.google.com/presentation/d/1iNX_35sWUakee2q3zMFPyHE4IV2nC3lkCK_H6Y2qK84/edit#slide=id.p">"LLVM for database and AI systems"</a><a href="https://www.slideshare.net/AndrewLamb32/20220623-apache-arro [...]
-<p>While Velox and Acero focus on execution engines, DataFusion provides the entire suite of components needed to build most analytic systems, including a SQL frontend, a dataframe API, and extension points for just about everything. Some <a href="https://github.com/apache/arrow-datafusion#known-uses">DataFusion users</a> use a subset of the features such as the frontend (e.g. <a href="https://dask-sql.readthedocs.io/en/latest/">dask-sql</a>) or the execution engine, (e.g. <a href="htt [...]
-<p>One of DataFusion’s advantages is its implementation in <a href="https://www.rust-lang.org/">Rust</a> and thus its easy integration with the broader Rust ecosystem. Rust continues to be a major source of benefit, from the <a href="https://www.influxdata.com/blog/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/">ease of parallelization with the high quality and standardized <code>async</code> ecosystem</a> , as well as its modern dependency management system and wonderful perfo [...]
-<!--While we haven’t invested in the benchmarking ratings game datafusion continues to be quite speedy (todo quantity this, with some evidence) – maybe clickbench?-->
-<!--
-Maybe we can do this un a future post
-# DataFusion in Action
-
-While DataFusion really shines as an embeddable query engine, if you want to try it out and get a feel for its power, you can use the basic[`datafusion-cli`](https://docs.rs/datafusion-cli/13.0.0/datafusion_cli/) tool to get a sense for what is possible to add in your application
-
-(TODO example here of using datafusion-cli to query from local parquet files on disk)
-
-TODO: also mention you can use the same thing to query data from S3
--->
-<h1 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h1>
-<p>We have increased the frequency of DataFusion releases to monthly instead of quarterly. This
-makes it easier for the increasing number of projects that now depend on DataFusion.</p>
-<p>We have also completed the "graduation" of <a href="https://github.com/apache/arrow-ballista">Ballista to its own top-level arrow-ballista repository</a>
-which decouples the two projects and allows each project to move even faster.</p>
-<p>Along with numerous other bug fixes and smaller improvements, here are some of the major advances:</p>
-<h1 id="improved-support-for-cloud-object-stores">Improved Support for Cloud Object Stores<a class="headerlink" href="#improved-support-for-cloud-object-stores" title="Permanent link">¶</a></h1>
-<p>DataFusion now supports many major cloud object stores (Amazon S3, Azure Blob Storage, and Google Cloud Storage) "out of the box" via the <a href="https://crates.io/crates/object_store">object_store</a> crate. Using this integration, DataFusion optimizes reading parquet files by reading only the parts of the files that are needed.</p>
-<h2 id="advanced-sql">Advanced SQL<a class="headerlink" href="#advanced-sql" title="Permanent link">¶</a></h2>
-<p>DataFusion now supports correlated subqueries, by rewriting them as joins. See the <a href="https://arrow.apache.org/datafusion/user-guide/sql/subqueries.html">Subquery</a> page in the User Guide for more information.</p>
-<p>In addition to numerous other small improvements, the following SQL features are now supported:</p>
-<ul>
-<li><code>ROWS</code>, <code>RANGE</code>, <code>PRECEDING</code> and <code>FOLLOWING</code> in <code>OVER</code> clauses <a href="https://github.com/apache/arrow-datafusion/issues/3570">#3570</a></li>
-<li><code>ROLLUP</code> and <code>CUBE</code> grouping set expressions <a href="https://github.com/apache/arrow-datafusion/issues/2446">#2446</a></li>
-<li><code>SUM DISTINCT</code> aggregate support <a href="https://github.com/apache/arrow-datafusion/issues/2405">#2405</a></li>
-<li><code>IN</code> and <code>NOT IN</code> Subqueries by rewriting them to <code>SEMI</code> / <code>ANTI</code> <a href="https://github.com/apache/arrow-datafusion/issues/2885">#2421</a></li>
-<li>Non equality predicates in <code>ON</code> clause of <code>LEFT</code>, <code>RIGHT,</code>and <code>FULL</code> joins <a href="https://github.com/apache/arrow-datafusion/issues/2591">#2591</a></li>
-<li>Exact <code>MEDIAN</code> <a href="https://github.com/apache/arrow-datafusion/issues/3009">#3009</a></li>
-<li><code>GROUPING SETS</code>/<code>CUBE</code>/<code>ROLLUP</code> <a href="https://github.com/apache/arrow-datafusion/issues/2716">#2716</a></li>
-</ul>
-<h1 id="more-ddl-support">More DDL Support<a class="headerlink" href="#more-ddl-support" title="Permanent link">¶</a></h1>
-<p>Just as it is important to query, it is also important to give users the ability to define their data sources. We have added:</p>
-<ul>
-<li><code>CREATE VIEW</code> <a href="https://github.com/apache/arrow-datafusion/issues/2279">#2279</a></li>
-<li><code>DESCRIBE <table></code> <a href="https://github.com/apache/arrow-datafusion/issues/2642">#2642</a></li>
-<li>Custom / Dynamic table provider factories <a href="https://github.com/apache/arrow-datafusion/issues/3311">#3311</a></li>
-<li><code>SHOW CREATE TABLE</code> for support for views <a href="https://github.com/apache/arrow-datafusion/issues/2830">#2830</a></li>
-</ul>
-<h1 id="faster-execution">Faster Execution<a class="headerlink" href="#faster-execution" title="Permanent link">¶</a></h1>
-<p>Performance is always an important goal for DataFusion, and there are a number of significant new optimizations such as</p>
-<ul>
-<li>Optimizations of TopK (queries with a <code>LIMIT</code> or <code>OFFSET</code> clause): <a href="https://github.com/apache/arrow-datafusion/issues/3527">#3527</a>, <a href="https://github.com/apache/arrow-datafusion/issues/2521">#2521</a></li>
-<li>Reduce <code>left</code>/<code>right</code>/<code>full</code> joins to <code>inner</code> join <a href="https://github.com/apache/arrow-datafusion/issues/2750">#2750</a></li>
-<li>Convert cross joins to inner joins when possible <a href="https://github.com/apache/arrow-datafusion/issues/3482">#3482</a></li>
-<li>Sort preserving <code>SortMergeJoin</code> <a href="https://github.com/apache/arrow-datafusion/issues/2699">#2699</a></li>
-<li>Improvements in group by and sort performance <a href="https://github.com/apache/arrow-datafusion/issues/2375">#2375</a></li>
-<li>Adaptive <code>regex_replace</code> implementation <a href="https://github.com/apache/arrow-datafusion/issues/3518">#3518</a></li>
-</ul>
-<h1 id="optimizer-enhancements">Optimizer Enhancements<a class="headerlink" href="#optimizer-enhancements" title="Permanent link">¶</a></h1>
-<p>Internally the optimizer has been significantly enhanced as well.</p>
-<ul>
-<li>Casting / coercion now happens during logical planning <a href="https://github.com/apache/arrow-datafusion/issues/3396">#3185</a> <a href="https://github.com/apache/arrow-datafusion/issues/3636">#3636</a></li>
-<li>More sophisticated expression analysis and simplification is available</li>
-</ul>
-<h1 id="parquet">Parquet<a class="headerlink" href="#parquet" title="Permanent link">¶</a></h1>
-<ul>
-<li>The parquet reader can now read directly from parquet files on remote object storage <a href="https://github.com/apache/arrow-datafusion/issues/2677">#2489</a> <a href="https://github.com/apache/arrow-datafusion/issues/3051">#3051</a></li>
-<li>Experimental support for “predicate pushdown” with late materialization after filtering during the scan (another blog post on this topic is coming soon).</li>
-<li>Support reading directly from AWS S3 and other object stores via <code>datafusion-cli</code> <a href="https://github.com/apache/arrow-datafusion/issues/3631">#3631</a></li>
-</ul>
-<h1 id="datatype-support">DataType Support<a class="headerlink" href="#datatype-support" title="Permanent link">¶</a></h1>
-<ul>
-<li>Support for <code>TimestampTz</code> <a href="https://github.com/apache/arrow-datafusion/issues/3660">#3660</a></li>
-<li>Expanded support for the <code>Decimal</code> type, including <code>IN</code> list and better built in coercion.</li>
-<li>Expanded support for date/time manipulation such as <code>date_bin</code> built-in function , timestamp <code>+/-</code> interval, <code>TIME</code> literal values <a href="https://github.com/apache/arrow-datafusion/issues/3010">#3010</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3110">#3110</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3034">#3034</a></li>
-<li>Binary operations (<code>AND</code>, <code>XOR</code>, etc): <a href="https://github.com/apache/arrow-datafusion/issues/1619">#3037</a> <a href="https://github.com/apache/arrow-datafusion/issues/3430">#3420</a></li>
-<li><code>IS TRUE/FALSE</code> and <code>IS [NOT] UNKNOWN</code> <a href="https://github.com/apache/arrow-datafusion/issues/3235">#3235</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3246">#3246</a></li>
-</ul>
-<h2 id="upcoming-work">Upcoming Work<a class="headerlink" href="#upcoming-work" title="Permanent link">¶</a></h2>
-<p>With the community growing and code accelerating, there is so much great stuff on the horizon. Some features we expect to land in the next few months:</p>
-<ul>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/3462">Complete Parquet Pushdown</a></li>
-<li><a href="https://github.com/apache/arrow-datafusion/issues/3148">Additional date/time support</a></li>
-<li>Cost models, Nested Join Optimizations, analysis framework <a href="https://github.com/apache/arrow-datafusion/issues/128">#128</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3843">#3843</a>, <a href="https://github.com/apache/arrow-datafusion/issues/3845">#3845</a></li>
-</ul>
-<h1 id="community-growth">Community Growth<a class="headerlink" href="#community-growth" title="Permanent link">¶</a></h1>
-<p>The DataFusion 9.0.0 and 13.0.0 releases consists of 433 PRs from 64 distinct contributors. This does not count all the work that goes into our dependencies such as <a href="https://crates.io/crates/arrow">arrow</a>, <a href="https://crates.io/crates/parquet">parquet</a>, and <a href="https://crates.io/crates/object_store">object_store</a>, that much of the same community helps nurture.</p>
-<!--
-$ git log --pretty=oneline 9.0.0..13.0.0 . | wc -l
-433
-
-$ git shortlog -sn 9.0.0..13.0.0 . | wc -l
-65
--->
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>Kudos to everyone in the community who contributed ideas, discussions, bug reports, documentation and code. It is exciting to be building something so cool together!</p>
-<p>If you are interested in contributing to DataFusion, we would love to
-have you join us on our journey to create the most advanced open
-source query engine. You can try out DataFusion on some of your own
-data and projects and let us know how it goes or contribute a PR with
-documentation, tests or code. A list of open issues suitable for
-beginners is
-<a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
-<p>Check out our <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
-<h2 id="appendix-contributor-shoutout">Appendix: Contributor Shoutout<a class="headerlink" href="#appendix-contributor-shoutout" title="Permanent link">¶</a></h2>
-<p>To give a sense of the number of people who contribute to this project regularly, we present for your consideration the following list derived from <code>git shortlog -sn 9.0.0..13.0.0 .</code> Thank you all again!</p>
-<!-- Note: combined kmitchener and Kirk Mitchener -->
-<pre><code> 87 Andy Grove
- 71 Andrew Lamb
- 29 Kun Liu
- 29 Kirk Mitchener
- 17 Wei-Ting Kuo
- 14 Yang Jiang
- 12 Raphael Taylor-Davies
- 11 Batuhan Taskaya
- 10 Brent Gardner
- 10 Remzi Yang
- 10 comphead
- 10 xudong.w
- 8 AssHero
- 7 Ruihang Xia
- 6 Dan Harris
- 6 Daniël Heres
- 6 Ian Alexander Joiner
- 6 Mike Roberts
- 6 askoa
- 4 BaymaxHWY
- 4 gorkem
- 4 jakevin
- 3 George Andronchik
- 3 Sarah Yurick
- 3 Stuart Carnie
- 2 Dalton Modlin
- 2 Dmitry Patsura
- 2 JasonLi
- 2 Jon Mease
- 2 Marco Neumann
- 2 yahoNanJing
- 1 Adilet Sarsembayev
- 1 Ayush Dattagupta
- 1 Dezhi Wu
- 1 Dhamotharan Sritharan
- 1 Eduard Karacharov
- 1 Francis Du
- 1 Harbour Zheng
- 1 Ismaël Mejía
- 1 Jack Klamer
- 1 Jeremy Dyer
- 1 Jiayu Liu
- 1 Kamil Konior
- 1 Liang-Chi Hsieh
- 1 Martin Grigorov
- 1 Matthijs Brobbel
- 1 Mehmet Ozan Kabak
- 1 Metehan Yıldırım
- 1 Morgan Cassels
- 1 Nitish Tiwari
- 1 Renjie Liu
- 1 Rito Takeuchi
- 1 Robert Pack
- 1 Thomas Cameron
- 1 Vrishabh
- 1 Xin Hao
- 1 Yijie Shen
- 1 byteink
- 1 kamille
- 1 mateuszkj
- 1 nvartolomei
- 1 yourenawo
- 1 Özgür Akkurt
-</code></pre>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#background">Background</a></li>
-<li><a href="#summary">Summary</a></li>
-<li><a href="#improved-support-for-cloud-object-stores">Improved Support for Cloud Object Stores</a><ul>
-<li><a href="#advanced-sql">Advanced SQL</a></li>
-</ul>
-</li>
-<li><a href="#more-ddl-support">More DDL Support</a></li>
-<li><a href="#faster-execution">Faster Execution</a></li>
-<li><a href="#optimizer-enhancements">Optimizer Enhancements</a></li>
-<li><a href="#parquet">Parquet</a></li>
-<li><a href="#datatype-support">DataType Support</a><ul>
-<li><a href="#upcoming-work">Upcoming Work</a></li>
-</ul>
-</li>
-<li><a href="#community-growth">Community Growth</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a><ul>
-<li><a href="#appendix-contributor-shoutout">Appendix: Contributor Shoutout</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2022/10/28/ballista-0.9.0/index.html b/blog/2022/10/28/ballista-0.9.0/index.html
deleted file mode 100644
index 49f062d..0000000
--- a/blog/2022/10/28/ballista-0.9.0/index.html
+++ /dev/null
@@ -1,216 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow Ballista 0.9.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow Ballista 0.9.0 Release
- </h1>
- <p>Posted on: Fri 28 October 2022 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#support-for-cloud-object-stores-and-distributed-file-systems">Support for Cloud Object Stores and Distributed File Systems</a></li>
-<li><a href="#flight-sql-jdbc-support">Flight SQL & JDBC support</a></li>
-<li><a href="#python-bindings">Python Bindings</a></li>
-<li><a href="#scheduler-web-user-interface-and-rest-api">Scheduler Web User Interface and REST API</a></li>
-<li><a href="#simplified-kubernetes-deployment">Simplified Kubernetes Deployment</a></li>
-<li><a href="#user-guide">User Guide</a></li>
-</ul>
-</li>
-<li><a href="#roadmap">Roadmap</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
-<p><a href="https://github.com/apache/arrow-ballista">Ballista</a> is an Arrow-native distributed SQL query engine implemented in Rust.</p>
-<p>Ballista 0.9.0 is now available and is the most significant release since the project was <a href="http://arrow.apache.org/blog/2021/04/12/ballista-donation/">donated</a> to Apache
-Arrow in 2021.</p>
-<p>This release represents 4 weeks of work, with 66 commits from 14 contributors:</p>
-<pre><code> 22 Andy Grove
- 12 yahoNanJing
- 6 Daniël Heres
- 4 Brent Gardner
- 4 dependabot[bot]
- 4 r.4ntix
- 3 Stefan Stanciulescu
- 3 mingmwang
- 2 Ken Suenobu
- 2 Yang Jiang
- 1 Metehan Yıldırım
- 1 Trent Feda
- 1 askoa
- 1 yangzhong
-</code></pre>
-<h2 id="release-highlights">Release Highlights<a class="headerlink" href="#release-highlights" title="Permanent link">¶</a></h2>
-<p>The release notes below are not exhaustive and only expose selected highlights of the release. Many other bug fixes
-and improvements have been made: we refer you to the <a href="https://github.com/apache/arrow-ballista/blob/0.9.0-rc2/ballista/CHANGELOG.md">complete changelog</a>.</p>
-<h3 id="support-for-cloud-object-stores-and-distributed-file-systems">Support for Cloud Object Stores and Distributed File Systems<a class="headerlink" href="#support-for-cloud-object-stores-and-distributed-file-systems" title="Permanent link">¶</a></h3>
-<p>This is the first release of Ballista to have documented support for querying data from distributed file systems and
-object stores. Currently, S3 and HDFS are supported. Support for Google Cloud Storage and Azure Blob Storage is planned
-for the next release.</p>
-<h3 id="flight-sql-jdbc-support">Flight SQL & JDBC support<a class="headerlink" href="#flight-sql-jdbc-support" title="Permanent link">¶</a></h3>
-<p>The Ballista scheduler now implements the <a href="https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/">Flight SQL protocol</a>, enabling any compliant Flight SQL client
-to connect to and run queries against a Ballista cluster.</p>
-<p>The Apache Arrow Flight SQL JDBC driver can be used to connect Business Intelligence tools to a Ballista cluster.</p>
-<h3 id="python-bindings">Python Bindings<a class="headerlink" href="#python-bindings" title="Permanent link">¶</a></h3>
-<p>It is now possible to connect to a Ballista cluster from Python and execute queries using both the DataFrame and SQL
-interfaces.</p>
-<h3 id="scheduler-web-user-interface-and-rest-api">Scheduler Web User Interface and REST API<a class="headerlink" href="#scheduler-web-user-interface-and-rest-api" title="Permanent link">¶</a></h3>
-<p>The scheduler now has a web user interface for monitoring queries. It is also possible to view graphical query plans
-that show how the query was executed, along with metrics.</p>
-<p><img src="/blog/images/2022-10-28-ballista-web-ui.png" width="800"/></p>
-<p>The REST API that powers the user interface can also be accessed directly.</p>
-<h3 id="simplified-kubernetes-deployment">Simplified Kubernetes Deployment<a class="headerlink" href="#simplified-kubernetes-deployment" title="Permanent link">¶</a></h3>
-<p>Ballista now provides a <a href="https://github.com/apache/arrow-ballista/tree/master/helm">Helm chart</a> for simplified Kubernetes deployment.</p>
-<h3 id="user-guide">User Guide<a class="headerlink" href="#user-guide" title="Permanent link">¶</a></h3>
-<p>The user guide is published at <a href="https://arrow.apache.org/ballista/">https://arrow.apache.org/ballista/</a> and provides
-deployment instructions for Docker, Docker Compose, and Kubernetes, as well as references for configuring and
-tuning Ballista.</p>
-<h2 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">¶</a></h2>
-<p>The Ballista community is currently focused on the following tasks for the next release:</p>
-<ul>
-<li>Support for Azure Blob Storage and Google Cloud Storage</li>
-<li>Improve benchmark performance by implementing more query optimizations</li>
-<li>Improve scheduler web user interface</li>
-<li>Publish Docker images to GitHub Container Registry</li>
-</ul>
-<p>The detailed list of issues planned for the 0.10.0 release can be found in the <a href="https://github.com/apache/arrow-ballista/issues/361">tracking issue</a>.</p>
-<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
-<p>Ballista has a friendly community and we welcome contributions. A good place to start is to following the instructions
-in the <a href="https://arrow.apache.org/ballista/">user guide</a> and try using Ballista with your own SQL queries and ETL pipelines, and file issues
-for any bugs or feature suggestions.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#support-for-cloud-object-stores-and-distributed-file-systems">Support for Cloud Object Stores and Distributed File Systems</a></li>
-<li><a href="#flight-sql-jdbc-support">Flight SQL & JDBC support</a></li>
-<li><a href="#python-bindings">Python Bindings</a></li>
-<li><a href="#scheduler-web-user-interface-and-rest-api">Scheduler Web User Interface and REST API</a></li>
-<li><a href="#simplified-kubernetes-deployment">Simplified Kubernetes Deployment</a></li>
-<li><a href="#user-guide">User Guide</a></li>
-</ul>
-</li>
-<li><a href="#roadmap">Roadmap</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2023/01/19/datafusion-16.0.0/index.html b/blog/2023/01/19/datafusion-16.0.0/index.html
deleted file mode 100644
index 9acd3a7..0000000
--- a/blog/2023/01/19/datafusion-16.0.0/index.html
+++ /dev/null
@@ -1,361 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 16.0.0 Project Update - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow DataFusion 16.0.0 Project Update
- </h1>
- <p>Posted on: Thu 19 January 2023 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a><ul>
-<li><a href="#community-growth">Community Growth</a></li>
-<li><a href="#performance">Performance 🚀</a></li>
-<li><a href="#runtime-resource-limits">Runtime Resource Limits</a></li>
-<li><a href="#sql-window-functions">SQL Window Functions</a></li>
-</ul>
-</li>
-<li><a href="#improved-joins">Improved Joins</a></li>
-<li><a href="#streaming-execution">Streaming Execution</a></li>
-<li><a href="#better-support-for-distributed-catalogs">Better Support for Distributed Catalogs</a></li>
-<li><a href="#additional-sql-support">Additional SQL Support</a></li>
-<li><a href="#plan-serialization-and-substrait">Plan Serialization and Substrait</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a><ul>
-<li><a href="#appendix-contributor-shoutout">Appendix: Contributor Shoutout</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
-<p><a href="https://arrow.apache.org/datafusion/">DataFusion</a> is an extensible
-query execution framework, written in <a href="https://www.rust-lang.org/">Rust</a>,
-that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its
-in-memory format. It is targeted primarily at developers creating data
-intensive analytics, and offers mature
-<a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>,
-a DataFrame API, and many extension points.</p>
-<p>Systems based on DataFusion perform very well in benchmarks,
-especially considering they operate directly on parquet files rather
-than first loading into a specialized format. Some recent highlights
-include <a href="https://benchmark.clickhouse.com/">clickbench</a> and the
-<a href="https://www.cloudfuse.io/dashboards/standalone-engines">Cloudfuse.io standalone query
-engines</a> page.</p>
-<p>DataFusion is also part of a longer term trend, articulated clearly by
-<a href="http://www.cs.cmu.edu/~pavlo/">Andy Pavlo</a> in his <a href="https://ottertune.com/blog/2022-databases-retrospective/">2022 Databases
-Retrospective</a>.
-Database frameworks are proliferating and it is likely that all OLAP
-DBMSs and other data heavy applications, such as machine learning,
-will <strong>require</strong> a vectorized, highly performant query engine in the next
-5 years to remain relevant. The only practical way to make such
-technology so widely available without many millions of dollars of
-investment is though open source engine such as DataFusion or
-<a href="https://github.com/facebookincubator/velox">Velox</a>.</p>
-<p>The rest of this post describes the improvements made to DataFusion
-over the last three months and some hints of where we are heading.</p>
-<h2 id="community-growth">Community Growth<a class="headerlink" href="#community-growth" title="Permanent link">¶</a></h2>
-<p>We again saw significant growth in the DataFusion community since <a href="https://arrow.apache.org/blog/2022/10/25/datafusion-13.0.0/">our last update</a>. There are some interesting metrics on <a href="https://ossrank.com/p/1573-apache-arrow-datafusion">OSSRank</a>.</p>
-<p>The DataFusion 16.0.0 release consists of 543 PRs from 73 distinct contributors, not including all the work that goes into dependencies such as <a href="https://crates.io/crates/arrow">arrow</a>, <a href="https://crates.io/crates/parquet">parquet</a>, and <a href="https://crates.io/crates/object_store">object_store</a>, that much of the same community helps support. Thank you all for your help</p>
-<!--
-$ git log --pretty=oneline 13.0.0..16.0.0 . | wc -l
- 543
-
-$ git shortlog -sn 13.0.0..16.0.0 . | wc -l
- 73
--->
-<p>Several <a href="https://github.com/apache/arrow-datafusion#known-uses">new systems based on DataFusion</a> were recently added:</p>
-<ul>
-<li><a href="https://github.com/GreptimeTeam/greptimedb">Greptime DB</a></li>
-<li><a href="https://synnada.ai/">Synnada</a></li>
-<li><a href="https://github.com/PRQL/prql-query">PRQL</a></li>
-<li><a href="https://github.com/parseablehq/parseable">Parseable</a></li>
-<li><a href="https://github.com/splitgraph/seafowl">SeaFowl</a></li>
-</ul>
-<h2 id="performance">Performance 🚀<a class="headerlink" href="#performance" title="Permanent link">¶</a></h2>
-<p>Performance and efficiency are core values for
-DataFusion. While there is still a gap between DataFusion and the best of
-breed, tightly integrated systems such as <a href="https://duckdb.org">DuckDB</a>
-and <a href="https://www.pola.rs/">Polars</a>, DataFusion is
-closing the gap quickly. Performance highlights from the last three
-months:</p>
-<ul>
-<li>Up to 30% Faster Sorting and Merging using the new <a href="https://arrow.apache.org/blog/2022/11/07/multi-column-sorts-in-arrow-rust-part-1/">Row Format</a></li>
-<li><a href="https://arrow.apache.org/blog/2022/12/26/querying-parquet-with-millisecond-latency/">Advanced predicate pushdown</a>, directly on parquet, directly from object storage, enabling sub millisecond filtering. <!-- Andrew nots: we should really get this turned on by default --></li>
-<li><code>70%</code> faster <code>IN</code> expressions evaluation (<a href="https://github.com/apache/arrow-datafusion/issues/4057">#4057</a>)</li>
-<li>Sort and partition aware optimizations (<a href="https://github.com/apache/arrow-datafusion/issues/3969">#3969</a> and <a href="https://github.com/apache/arrow-datafusion/issues/4691">#4691</a>)</li>
-<li>Filter selectivity analysis (<a href="https://github.com/apache/arrow-datafusion/issues/3868">#3868</a>)</li>
-</ul>
-<h2 id="runtime-resource-limits">Runtime Resource Limits<a class="headerlink" href="#runtime-resource-limits" title="Permanent link">¶</a></h2>
-<p>Previously, DataFusion could potentially use unbounded amounts of memory for certain queries that included Sorts, Grouping or Joins.</p>
-<p>In version 16.0.0, it is possible to limit DataFusion's memory usage for Sorting and Grouping. We are looking for help adding similar limiting for Joins as well as expanding our algorithms to optionally spill to secondary storage. See <a href="https://github.com/apache/arrow-datafusion/issues/3941">#3941</a> for more detail.</p>
-<h2 id="sql-window-functions">SQL Window Functions<a class="headerlink" href="#sql-window-functions" title="Permanent link">¶</a></h2>
-<p><a href="https://en.wikipedia.org/wiki/Window_function_(SQL)">SQL Window Functions</a> are useful for a variety of analysis and DataFusion's implementation support expanded significantly:</p>
-<ul>
-<li>Custom window frames such as <code>... OVER (ORDER BY ... RANGE BETWEEN 0.2 PRECEDING AND 0.2 FOLLOWING)</code></li>
-<li>Unbounded window frames such as <code>... OVER (ORDER BY ... RANGE UNBOUNDED ROWS PRECEDING)</code></li>
-<li>Support for the <code>NTILE</code> window function (<a href="https://github.com/apache/arrow-datafusion/issues/4676">#4676</a>)</li>
-<li>Support for <code>GROUPS</code> mode (<a href="https://github.com/apache/arrow-datafusion/issues/4155">#4155</a>)</li>
-</ul>
-<h1 id="improved-joins">Improved Joins<a class="headerlink" href="#improved-joins" title="Permanent link">¶</a></h1>
-<p>Joins are often the most complicated operations to handle well in
-analytics systems and DataFusion 16.0.0 offers significant improvements
-such as</p>
-<ul>
-<li>Cost based optimizer (CBO) automatically reorders join evaluations, selects algorithms (Merge / Hash), and pick build side based on available statistics and join type (<code>INNER</code>, <code>LEFT</code>, etc) (<a href="https://github.com/apache/arrow-datafusion/issues/4219">#4219</a>)</li>
-<li>Fast non <code>column=column</code> equijoins such as <code>JOIN ON a.x + 5 = b.y</code></li>
-<li>Better performance on non-equijoins (<a href="https://github.com/apache/arrow-datafusion/issues/4562">#4562</a>) <!-- TODO is this a good thing to mention as any time this is usd the query is going to go slow or the data size is small --></li>
-</ul>
-<h1 id="streaming-execution">Streaming Execution<a class="headerlink" href="#streaming-execution" title="Permanent link">¶</a></h1>
-<p>One emerging use case for Datafusion is as a foundation for
-streaming-first data platforms. An important prerequisite
-is support for incremental execution for queries that can be computed
-incrementally.</p>
-<p>With this release, DataFusion now supports the following streaming features:</p>
-<ul>
-<li>Data ingestion from infinite files such as FIFOs (<a href="https://github.com/apache/arrow-datafusion/issues/4694">#4694</a>),</li>
-<li>Detection of pipeline-breaking queries in streaming use cases (<a href="https://github.com/apache/arrow-datafusion/issues/4694">#4694</a>),</li>
-<li>Automatic input swapping for joins so probe side is a data stream (<a href="https://github.com/apache/arrow-datafusion/issues/4694">#4694</a>),</li>
-<li>Intelligent elision of pipeline-breaking sort operations whenever possible (<a href="https://github.com/apache/arrow-datafusion/issues/4691">#4691</a>),</li>
-<li>Incremental execution for more types of queries; e.g. queries involving finite window frames (<a href="https://github.com/apache/arrow-datafusion/issues/4777">#4777</a>).</li>
-</ul>
-<p>These are a major steps forward, and we plan even more improvements over the next few releases.</p>
-<h1 id="better-support-for-distributed-catalogs">Better Support for Distributed Catalogs<a class="headerlink" href="#better-support-for-distributed-catalogs" title="Permanent link">¶</a></h1>
-<p>16.0.0 has been enhanced support for asynchronous catalogs (<a href="https://github.com/apache/arrow-datafusion/issues/4607">#4607</a>)
-to better support distributed metadata stores such as
-<a href="https://delta.io/">Delta.io</a> and <a href="https://iceberg.apache.org/">Apache
-Iceberg</a> which require asynchronous I/O
-during planning to access remote catalogs. Previously, DataFusion
-required synchronous access to all relevant catalog information.</p>
-<h1 id="additional-sql-support">Additional SQL Support<a class="headerlink" href="#additional-sql-support" title="Permanent link">¶</a></h1>
-<p>SQL support continues to improve, including some of these highlights:</p>
-<ul>
-<li>Add TPC-DS query planning regression tests <a href="https://github.com/apache/arrow-datafusion/issues/4719">#4719</a></li>
-<li>Support for <code>PREPARE</code> statement <a href="https://github.com/apache/arrow-datafusion/issues/4490">#4490</a></li>
-<li>Automatic coercions ast between Date and Timestamp <a href="https://github.com/apache/arrow-datafusion/issues/4726">#4726</a></li>
-<li>Support type coercion for timestamp and utf8 <a href="https://github.com/apache/arrow-datafusion/issues/4312">#4312</a></li>
-<li>Full support for time32 and time64 literal values (<code>ScalarValue</code>) <a href="https://github.com/apache/arrow-datafusion/issues/4156">#4156</a></li>
-<li>New functions, including <code>uuid()</code> <a href="https://github.com/apache/arrow-datafusion/issues/4041">#4041</a>, <code>current_time</code> <a href="https://github.com/apache/arrow-datafusion/issues/4054">#4054</a>, <code>current_date</code> <a href="https://github.com/apache/arrow-datafusion/issues/4022">#4022</a></li>
-<li>Compressed CSV/JSON support <a href="https://github.com/apache/arrow-datafusion/issues/3642">#3642</a></li>
-</ul>
-<p>The community has also invested in new <a href="https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/tests/sqllogictests/README.md">sqllogic based</a> tests to keep improving DataFusion's quality with less effort.</p>
-<h1 id="plan-serialization-and-substrait">Plan Serialization and Substrait<a class="headerlink" href="#plan-serialization-and-substrait" title="Permanent link">¶</a></h1>
-<p>DataFusion now supports serialization of physical plans, with a custom protocol buffers format. In addition, we are adding initial support for <a href="https://substrait.io/">Substrait</a>, a Cross-Language Serialization for Relational Algebra</p>
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>Kudos to everyone in the community who contributed ideas, discussions, bug reports, documentation and code. It is exciting to be building something so cool together!</p>
-<p>If you are interested in contributing to DataFusion, we would love to
-have you join us. You can try out DataFusion on some of your own
-data and projects and let us know how it goes or contribute a PR with
-documentation, tests or code. A list of open issues suitable for
-beginners is
-<a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
-<p>Check out our <a href="https://arrow.apache.org/datafusion/community/communication.html">Communication Doc</a> on more
-ways to engage with the community.</p>
-<h2 id="appendix-contributor-shoutout">Appendix: Contributor Shoutout<a class="headerlink" href="#appendix-contributor-shoutout" title="Permanent link">¶</a></h2>
-<p>Here is a list of people who have contributed PRs to this project over the last three releases, derived from <code>git shortlog -sn 13.0.0..16.0.0 .</code> Thank you all!</p>
-<pre><code> 113 Andrew Lamb
- 58 jakevin
- 46 Raphael Taylor-Davies
- 30 Andy Grove
- 19 Batuhan Taskaya
- 19 Remzi Yang
- 17 ygf11
- 16 Burak
- 16 Jeffrey
- 16 Marco Neumann
- 14 Kun Liu
- 12 Yang Jiang
- 10 mingmwang
- 9 Daniël Heres
- 9 Mustafa akur
- 9 comphead
- 9 mvanschellebeeck
- 9 xudong.w
- 7 dependabot[bot]
- 7 yahoNanJing
- 6 Brent Gardner
- 5 AssHero
- 4 Jiayu Liu
- 4 Wei-Ting Kuo
- 4 askoa
- 3 André Calado Coroado
- 3 Jie Han
- 3 Jon Mease
- 3 Metehan Yıldırım
- 3 Nga Tran
- 3 Ruihang Xia
- 3 baishen
- 2 Berkay Şahin
- 2 Dan Harris
- 2 Dongyan Zhou
- 2 Eduard Karacharov
- 2 Kikkon
- 2 Liang-Chi Hsieh
- 2 Marko Milenković
- 2 Martin Grigorov
- 2 Roman Nozdrin
- 2 Tim Van Wassenhove
- 2 r.4ntix
- 2 unconsolable
- 2 unvalley
- 1 Ajaya Agrawal
- 1 Alexander Spies
- 1 ArkashaJavelin
- 1 Artjoms Iskovs
- 1 BoredPerson
- 1 Christian Salvati
- 1 Creampanda
- 1 Data Psycho
- 1 Francis Du
- 1 Francis Le Roy
- 1 LFC
- 1 Marko Grujic
- 1 Matt Willian
- 1 Matthijs Brobbel
- 1 Max Burke
- 1 Mehmet Ozan Kabak
- 1 Rito Takeuchi
- 1 Roman Zeyde
- 1 Vrishabh
- 1 Zhang Li
- 1 ZuoTiJia
- 1 byteink
- 1 cfraz89
- 1 nbr
- 1 xxchan
- 1 yujie.zhang
- 1 zembunia
- 1 哇呜哇呜呀咦耶
-</code></pre>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a><ul>
-<li><a href="#community-growth">Community Growth</a></li>
-<li><a href="#performance">Performance 🚀</a></li>
-<li><a href="#runtime-resource-limits">Runtime Resource Limits</a></li>
-<li><a href="#sql-window-functions">SQL Window Functions</a></li>
-</ul>
-</li>
-<li><a href="#improved-joins">Improved Joins</a></li>
-<li><a href="#streaming-execution">Streaming Execution</a></li>
-<li><a href="#better-support-for-distributed-catalogs">Better Support for Distributed Catalogs</a></li>
-<li><a href="#additional-sql-support">Additional SQL Support</a></li>
-<li><a href="#plan-serialization-and-substrait">Plan Serialization and Substrait</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a><ul>
-<li><a href="#appendix-contributor-shoutout">Appendix: Contributor Shoutout</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2023/06/24/datafusion-25.0.0/index.html b/blog/2023/06/24/datafusion-25.0.0/index.html
deleted file mode 100644
index 793d2bd..0000000
--- a/blog/2023/06/24/datafusion-25.0.0/index.html
+++ /dev/null
@@ -1,333 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 26.0.0 - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow DataFusion 26.0.0
- </h1>
- <p>Posted on: Sat 24 June 2023 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#background">Background</a></li>
-<li><a href="#even-better-performance">(Even) Better Performance</a></li>
-<li><a href="#improved-documentation-and-website">Improved Documentation and Website</a></li>
-<li><a href="#new-features">New Features</a><ul>
-<li><a href="#more-streaming-less-memory">More Streaming, Less Memory</a></li>
-<li><a href="#dml-support-insert-delete-update-etc">DML Support (INSERT, DELETE, UPDATE, etc)</a></li>
-<li><a href="#timestamp-and-intervals">Timestamp and Intervals</a></li>
-<li><a href="#querying-structured-types-list-and-structs">Querying Structured Types (List and Structs)</a></li>
-<li><a href="#subqueries-all-the-way-down">Subqueries All the Way Down</a></li>
-</ul>
-</li>
-<li><a href="#community-and-project-growth">Community and Project Growth</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>It has been a whirlwind 6 months of DataFusion development since <a href="https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0">our
-last update</a>: the community has grown, many features have been added,
-performance improved and we are <a href="https://github.com/apache/arrow-datafusion/discussions/6475">discussing</a> branching out to our own
-top level Apache Project.</p>
-<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h2>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database
-toolkit, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory
-format.</p>
-<p>DataFusion, along with <a href="https://calcite.apache.org">Apache Calcite</a>, Facebook's <a href="https://github.com/facebookincubator/velox">Velox</a> and
-similar technology are part of the next generation "<a href="https://www.usenix.org/publications/login/winter2018/khurana">Deconstructed
-Database</a>" architectures, where new systems are built on a foundation
-of fast, modular components, rather as a single tightly integrated
-system.</p>
-<p>While single tightly integrated systems such as <a href="https://spark.apache.org/">Spark</a>, <a href="https://duckdb.org">DuckDB</a> and
-<a href="https://www.pola.rs/">Pola.rs</a> are great pieces of technology, our community believes that
-anyone developing new data heavy application, such as those common in
-machine learning in the next 5 years, will <strong>require</strong> a high
-performance, vectorized, query engine to remain relevant. The only
-practical way to gain access to such technology without investing many
-millions of dollars to build a new tightly integrated engine, is
-though open source projects like DataFusion and similar enabling
-technologies such as <a href="https://arrow.apache.org">Apache Arrow</a> and <a href="https://www.rust-lang.org/">Rust</a>.</p>
-<p>DataFusion is targeted primarily at developers creating other data
-intensive analytics, and offers:</p>
-<ul>
-<li>High performance, native, parallel streaming execution engine</li>
-<li>Mature <a href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">SQL support</a>, featuring subqueries, window functions, grouping sets, and more</li>
-<li>Built in support for Parquet, Avro, CSV, JSON and Arrow formats and easy extension for others</li>
-<li>Native DataFrame API and <a href="https://arrow.apache.org/datafusion-python/">python bindings</a></li>
-<li><a href="https://docs.rs/datafusion/latest/datafusion/index.html">Well documented</a> source code and architecture, designed to be customized to suit downstream project needs</li>
-<li>High quality, easy to use code <a href="https://crates.io/crates/datafusion/versions">released every 2 weeks to crates.io</a></li>
-<li>Welcoming, open community, governed by the highly regarded and well understood <a href="https://www.apache.org/">Apache Software Foundation</a></li>
-</ul>
-<p>The rest of this post highlights some of the improvements we have made
-to DataFusion over the last 6 months and a preview of where we are
-heading. You can see a list of all changes in the detailed
-<a href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/CHANGELOG.md">CHANGELOG</a>.</p>
-<h2 id="even-better-performance">(Even) Better Performance<a class="headerlink" href="#even-better-performance" title="Permanent link">¶</a></h2>
-<p><a href="https://voltrondata.com/resources/speeds-and-feeds-hardware-and-software-matter">Various</a> benchmarks show DataFusion to be quite close or <a href="https://github.com/tustvold/access-log-bench">even
-faster</a> to the state of the art in analytic performance (at the moment
-this seems to be DuckDB). We continually work on improving performance
-(see <a href="https://github.com/apache/arrow-datafusion/issues/5546">#5546</a> for a list) and would love additional help in this area.</p>
-<p>DataFusion now reads single large Parquet files significantly faster by
-<a href="https://github.com/apache/arrow-datafusion/pull/5057">parallelizing across multiple cores</a>. Native speeds for reading JSON
-and CSV files are also up to 2.5x faster thanks to improvements
-upstream in arrow-rs <a href="https://github.com/apache/arrow-rs/pull/3479#issuecomment-1384353159">JSON reader</a> and <a href="https://github.com/apache/arrow-rs/pull/3365">CSV reader</a>.</p>
-<p>Also, we have integrated the <a href="https://arrow.apache.org/blog/2022/11/07/multi-column-sorts-in-arrow-rust-part-1/">arrow-rs Row Format</a> into DataFusion resulting in up to <a href="https://github.com/apache/arrow-datafusion/pull/6163">2-3x faster sorting and merging</a>.</p>
-<h2 id="improved-documentation-and-website">Improved Documentation and Website<a class="headerlink" href="#improved-documentation-and-website" title="Permanent link">¶</a></h2>
-<p>Part of growing the DataFusion community is ensuring that DataFusion's
-features are understood and that it is easy to contribute and
-participate. To that end the <a href="https://arrow.apache.org/datafusion/">website</a> has been cleaned up, <a href="https://docs.rs/datafusion/latest/datafusion/index.html#architecture">the
-architecture guide</a> expanded, the <a href="https://arrow.apache.org/datafusion/contributor-guide/roadmap.html">roadmap</a> updated, and several
-overview talks created:</p>
-<ul>
-<li>Apr 2023 <em>Query Engine</em>: <a href="https://youtu.be/NVKujPxwSBA">recording</a> and <a href="https://docs.google.com/presentation/d/1D3GDVas-8y0sA4c8EOgdCvEjVND4s2E7I6zfs67Y4j8/edit#slide=id.p">slides</a></li>
-<li>April 2023 <em>Logical Plan and Expressions</em>: <a href="https://youtu.be/EzZTLiSJnhY">recording</a> and <a href="https://docs.google.com/presentation/d/1ypylM3-w60kVDW7Q6S99AHzvlBgciTdjsAfqNP85K30">slides</a></li>
-<li>April 2023 <em>Physical Plan and Execution</em>: <a href="https://youtu.be/2jkWU3_w6z0">recording</a> and <a href="https://docs.google.com/presentation/d/1cA2WQJ2qg6tx6y4Wf8FH2WVSm9JQ5UgmBWATHdik0hg">slides</a></li>
-</ul>
-<h2 id="new-features">New Features<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h2>
-<h3 id="more-streaming-less-memory">More Streaming, Less Memory<a class="headerlink" href="#more-streaming-less-memory" title="Permanent link">¶</a></h3>
-<p>We have made significant progress on the <a href="https://github.com/apache/arrow-datafusion/issues/4285">streaming execution roadmap</a>
-such as <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.unbounded_output">unbounded datasources</a>, <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/aggregates/enum.GroupByOrderMode.html">streaming group by</a>, sophisticated
-<a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/global_sort_selection/index.html">sort</a> and <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/repartition/index.html">repartitioning</a> improvements in the optimizer, and support
-for <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/joins/struct.SymmetricHashJoinExec.html">symmetric hash join</a> (read more about that in the great <a href="https://www.synnada.ai/blog/general-purpose-stream-joins-via-pruning-symmetric-hash-joins">Synnada
-Blog Post</a> on the topic). Together, these features both 1) make it
-easier to build streaming systems using DataFusion that can
-incrementally generate output before (or ever) seeing the end of the
-input and 2) allow general queries to use less memory and generate their
-results faster.</p>
-<p>We have also improved the runtime <a href="https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/index.html">memory management</a> system so that
-DataFusion now stays within its declared memory budget <a href="https://github.com/apache/arrow-datafusion/issues/3941">generate
-runtime errors</a>.</p>
-<h3 id="dml-support-insert-delete-update-etc">DML Support (<code>INSERT</code>, <code>DELETE</code>, <code>UPDATE</code>, etc)<a class="headerlink" href="#dml-support-insert-delete-update-etc" title="Permanent link">¶</a></h3>
-<p>Part of building high performance data systems includes writing data,
-and DataFusion supports several features for creating new files:</p>
-<ul>
-<li><code>INSERT INTO</code> and <code>SELECT ... INTO</code> support for memory backed and CSV tables</li>
-<li>New <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/insert/trait.DataSink.html">API for writing data into TableProviders</a></li>
-</ul>
-<p>We are working on easier to use <a href="https://github.com/apache/arrow-datafusion/issues/5654">COPY INTO</a> syntax, better support
-for writing parquet, JSON, and AVRO, and more -- see our <a href="https://github.com/apache/arrow-datafusion/issues/6569">tracking epic</a>
-for more details.</p>
-<h3 id="timestamp-and-intervals">Timestamp and Intervals<a class="headerlink" href="#timestamp-and-intervals" title="Permanent link">¶</a></h3>
-<p>One mark of the maturity of a SQL engine is how it handles the tricky
-world of timestamp, date, times and interval arithmetic. DataFusion is
-feature complete in this area and behaves as you would expect,
-supporting queries such as</p>
-<pre><code class="language-sql">SELECT now() + '1 month' FROM my_table;
-</code></pre>
-<p>We still have a long tail of <a href="https://github.com/apache/arrow-datafusion/issues/3148">date and time improvements</a>, which we are working on as well.</p>
-<h3 id="querying-structured-types-list-and-structs">Querying Structured Types (<code>List</code> and <code>Struct</code>s)<a class="headerlink" href="#querying-structured-types-list-and-structs" title="Permanent link">¶</a></h3>
-<p>Arrow and Parquet <a href="https://arrow.apache.org/blog/2022/10/08/arrow-parquet-encoding-part-2/">support nested data</a> well and DataFusion lets you
-easily query such <code>Struct</code> and <code>List</code>. For example, you can use
-DataFusion to read and query the <a href="https://data.mendeley.com/datasets/ct8f9skv97">JSON Datasets for Exploratory OLAP -
-Mendeley Data</a> like this:</p>
-<pre><code class="language-sql">----------
--- Explore structured data using SQL
-----------
-SELECT delete FROM 'twitter-sample-head-100000.parquet' WHERE delete IS NOT NULL limit 10;
-+---------------------------------------------------------------------------------------------------------------------------+
-| delete |
-+---------------------------------------------------------------------------------------------------------------------------+
-| {status: {id: {$numberLong: 135037425050320896}, id_str: 135037425050320896, user_id: 334902461, user_id_str: 334902461}} |
-| {status: {id: {$numberLong: 134703982051463168}, id_str: 134703982051463168, user_id: 405383453, user_id_str: 405383453}} |
-| {status: {id: {$numberLong: 134773741740765184}, id_str: 134773741740765184, user_id: 64823441, user_id_str: 64823441}} |
-| {status: {id: {$numberLong: 132543659655704576}, id_str: 132543659655704576, user_id: 45917834, user_id_str: 45917834}} |
-| {status: {id: {$numberLong: 133786431926697984}, id_str: 133786431926697984, user_id: 67229952, user_id_str: 67229952}} |
-| {status: {id: {$numberLong: 134619093570560002}, id_str: 134619093570560002, user_id: 182430773, user_id_str: 182430773}} |
-| {status: {id: {$numberLong: 134019857527214080}, id_str: 134019857527214080, user_id: 257396311, user_id_str: 257396311}} |
-| {status: {id: {$numberLong: 133931546469076993}, id_str: 133931546469076993, user_id: 124539548, user_id_str: 124539548}} |
-| {status: {id: {$numberLong: 134397743350296576}, id_str: 134397743350296576, user_id: 139836391, user_id_str: 139836391}} |
-| {status: {id: {$numberLong: 127833661767823360}, id_str: 127833661767823360, user_id: 244442687, user_id_str: 244442687}} |
-+---------------------------------------------------------------------------------------------------------------------------+
-
-----------
--- Select some deeply nested fields
-----------
-SELECT
- delete['status']['id']['$numberLong'] as delete_id,
- delete['status']['user_id'] as delete_user_id
-FROM 'twitter-sample-head-100000.parquet' WHERE delete IS NOT NULL LIMIT 10;
-
-+--------------------+----------------+
-| delete_id | delete_user_id |
-+--------------------+----------------+
-| 135037425050320896 | 334902461 |
-| 134703982051463168 | 405383453 |
-| 134773741740765184 | 64823441 |
-| 132543659655704576 | 45917834 |
-| 133786431926697984 | 67229952 |
-| 134619093570560002 | 182430773 |
-| 134019857527214080 | 257396311 |
-| 133931546469076993 | 124539548 |
-| 134397743350296576 | 139836391 |
-| 127833661767823360 | 244442687 |
-+--------------------+----------------+
-</code></pre>
-<h3 id="subqueries-all-the-way-down">Subqueries All the Way Down<a class="headerlink" href="#subqueries-all-the-way-down" title="Permanent link">¶</a></h3>
-<p>DataFusion can run many different subqueries by rewriting them to
-joins. It has been able to run the full suite of TPC-H queries for at
-least the last year, but recently we have implemented significant
-improvements to this logic, sufficient to run almost all queries in
-the TPC-DS benchmark as well.</p>
-<h2 id="community-and-project-growth">Community and Project Growth<a class="headerlink" href="#community-and-project-growth" title="Permanent link">¶</a></h2>
-<p>The six months since <a href="https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0">our last update</a> saw significant growth in
-the DataFusion community. Between versions <code>17.0.0</code> and <code>26.0.0</code>,
-DataFusion merged 711 PRs from 107 distinct contributors, not
-including all the work that goes into our core dependencies such as
-<a href="https://crates.io/crates/arrow">arrow</a>,
-<a href="https://crates.io/crates/parquet">parquet</a>, and
-<a href="https://crates.io/crates/object_store">object_store</a>, that much of
-the same community helps support.</p>
-<p>In addition, we have added 7 new committers and 1 new PMC member to
-the Apache Arrow project, largely focused on DataFusion, and we
-learned about some of the cool <a href="https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users">new systems</a> which are using
-DataFusion. Given the growth of the community and interest in the
-project, we also clarified the <a href="https://github.com/apache/arrow-datafusion/discussions/6441">mission statement</a> and are
-<a href="https://github.com/apache/arrow-datafusion/discussions/6475">discussing</a> "graduate"ing DataFusion to a new top level
-Apache Software Foundation project.</p>
-<!--
-$ git log --pretty=oneline 17.0.0..26.0.0 . | wc -l
- 711
-
-$ git shortlog -sn 17.0.0..26.0.0 . | wc -l
- 107
--->
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>Kudos to everyone in the community who has contributed ideas,
-discussions, bug reports, documentation and code. It is exciting to be
-innovating on the next generation of database architectures together!</p>
-<p>If you are interested in contributing to DataFusion, we would love to
-have you join us. You can try out DataFusion on some of your own
-data and projects and let us know how it goes or contribute a PR with
-documentation, tests or code. A list of open issues suitable for
-beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
-<p>Check out our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a> for more ways to engage with the
-community.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#background">Background</a></li>
-<li><a href="#even-better-performance">(Even) Better Performance</a></li>
-<li><a href="#improved-documentation-and-website">Improved Documentation and Website</a></li>
-<li><a href="#new-features">New Features</a><ul>
-<li><a href="#more-streaming-less-memory">More Streaming, Less Memory</a></li>
-<li><a href="#dml-support-insert-delete-update-etc">DML Support (INSERT, DELETE, UPDATE, etc)</a></li>
-<li><a href="#timestamp-and-intervals">Timestamp and Intervals</a></li>
-<li><a href="#querying-structured-types-list-and-structs">Querying Structured Types (List and Structs)</a></li>
-<li><a href="#subqueries-all-the-way-down">Subqueries All the Way Down</a></li>
-</ul>
-</li>
-<li><a href="#community-and-project-growth">Community and Project Growth</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2023/08/05/datafusion_fast_grouping/index.html b/blog/2023/08/05/datafusion_fast_grouping/index.html
deleted file mode 100644
index 1009a4d..0000000
--- a/blog/2023/08/05/datafusion_fast_grouping/index.html
+++ /dev/null
@@ -1,449 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0 - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Aggregating Millions of Groups Fast in Apache Arrow DataFusion 28.0.0
- </h1>
- <p>Posted on: Sat 05 August 2023 by alamb, Dandandan, tustvold</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#aggregating-millions-of-groups-fast-in-apache-arrow-datafusion">Aggregating Millions of Groups Fast in Apache Arrow DataFusion</a></li>
-<li><a href="#tldr">TLDR</a></li>
-<li><a href="#introduction-to-high-cardinality-grouping">Introduction to high cardinality grouping</a></li>
-<li><a href="#the-solution">The solution</a><ul>
-<li><a href="#two-phase-parallel-partitioned-grouping">Two phase parallel partitioned grouping</a></li>
-<li><a href="#hash-grouping">Hash grouping</a></li>
-<li><a href="#hash-grouping-in-2700">Hash grouping in 27.0.0</a></li>
-<li><a href="#hash-grouping-in-2800">Hash grouping in 28.0.0</a></li>
-<li><a href="#notes">Notes</a></li>
-</ul>
-</li>
-<li><a href="#clickbench-results">ClickBench results</a><ul>
-<li><a href="#notes_1">Notes</a></li>
-</ul>
-</li>
-<li><a href="#conclusion-performance-matters">Conclusion: performance matters</a></li>
-<li><a href="#acknowledgments">Acknowledgments</a></li>
-<li><a href="#about-datafusion">About DataFusion</a></li>
-<li><a href="#notes_2">Notes</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<!-- Converted from Google Docs using https://www.buymeacoffee.com/docstomarkdown -->
-<h2 id="aggregating-millions-of-groups-fast-in-apache-arrow-datafusion">Aggregating Millions of Groups Fast in Apache Arrow DataFusion<a class="headerlink" href="#aggregating-millions-of-groups-fast-in-apache-arrow-datafusion" title="Permanent link">¶</a></h2>
-<p>Andrew Lamb, Daniël Heres, Raphael Taylor-Davies,</p>
-<p><em>Note: this article was originally published on the <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">InfluxData Blog</a></em></p>
-<h2 id="tldr">TLDR<a class="headerlink" href="#tldr" title="Permanent link">¶</a></h2>
-<p>Grouped aggregations are a core part of any analytic tool, creating understandable summaries of huge data volumes. <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>’s parallel aggregation capability is 2-3x faster in the <a href="https://crates.io/crates/datafusion/28.0.0">newly released version <code>28.0.0</code></a> for queries with a large number (10,000 or more) of groups.</p>
-<p>Improving aggregation performance matters to all users of DataFusion. For example, both InfluxDB, a <a href="https://github.com/influxdata/influxdb">time series data platform</a> and Coralogix, a <a href="https://coralogix.com/?utm_source=InfluxDB&utm_medium=Blog&utm_campaign=organic">full-stack observability</a> platform, aggregate vast amounts of raw data to monitor and create insights for our customers. Improving DataFusion’s performance lets us provide better user experien [...]
-<p>With the new optimizations, DataFusion’s grouping speed is now close to DuckDB, a system that regularly reports <a href="https://duckdblabs.github.io/db-benchmark/">great</a> <a href="https://duckdb.org/2022/03/07/aggregate-hashtable.html#experiments">grouping</a> benchmark performance numbers. Figure 1 contains a representative sample of <a href="https://github.com/ClickHouse/ClickBench/tree/main">ClickBench</a> on a single Parquet file, and the full results are at the end of this ar [...]
-<p><img src="/blog/images/datafusion_fast_grouping/summary.png" width="700"/></p>
-<p><strong>Figure 1</strong>: Query performance for ClickBench queries on queries 16, 17, 18 and 19 on a single Parquet file for DataFusion <code>27.0.0</code>, DataFusion <code>28.0.0</code> and DuckDB <code>0.8.1</code>.</p>
-<h2 id="introduction-to-high-cardinality-grouping">Introduction to high cardinality grouping<a class="headerlink" href="#introduction-to-high-cardinality-grouping" title="Permanent link">¶</a></h2>
-<p>Aggregation is a fancy word for computing summary statistics across many rows that have the same value in one or more columns. We call the rows with the same values <em>groups</em> and “high cardinality” means there are a large number of distinct groups in the dataset. At the time of writing, a “large” number of groups in analytic engines is around 10,000.</p>
-<p>For example the <a href="https://github.com/ClickHouse/ClickBench">ClickBench</a> <em>hits</em> dataset contains 100 million anonymized user clicks across a set of websites. ClickBench Query 17 is:</p>
-<pre><code class="language-sql">SELECT "UserID", "SearchPhrase", COUNT(*)
-FROM hits
-GROUP BY "UserID", "SearchPhrase"
-ORDER BY COUNT(*)
-DESC LIMIT 10;
-</code></pre>
-<p>In English, this query finds “the top ten (user, search phrase) combinations, across all clicks” and produces the following results (there are no search phrases for the top ten users):</p>
-<pre><code class="language-text">+---------------------+--------------+-----------------+
-| UserID | SearchPhrase | COUNT(UInt8(1)) |
-+---------------------+--------------+-----------------+
-| 1313338681122956954 | | 29097 |
-| 1907779576417363396 | | 25333 |
-| 2305303682471783379 | | 10597 |
-| 7982623143712728547 | | 6669 |
-| 7280399273658728997 | | 6408 |
-| 1090981537032625727 | | 6196 |
-| 5730251990344211405 | | 6019 |
-| 6018350421959114808 | | 5990 |
-| 835157184735512989 | | 5209 |
-| 770542365400669095 | | 4906 |
-+---------------------+--------------+-----------------+
-</code></pre>
-<p>The ClickBench dataset contains</p>
-<ul>
-<li>99,997,497 total rows[^1]</li>
-<li>17,630,976 different users (distinct UserIDs)[^2]</li>
-<li>6,019,103 different search phrases[^3]</li>
-<li>24,070,560 distinct combinations[^4] of (UserID, SearchPhrase)
-Thus, to answer the query, DataFusion must map each of the 100M different input rows into one of the <strong>24 million different groups</strong>, and keep count of how many such rows there are in each group.</li>
-</ul>
-<h2 id="the-solution">The solution<a class="headerlink" href="#the-solution" title="Permanent link">¶</a></h2>
-<p>Like most concepts in databases and other analytic systems, the basic ideas of this algorithm are straightforward and taught in introductory computer science courses. You could compute the query with a program such as this[^5]:</p>
-<pre><code class="language-python">import pandas as pd
-from collections import defaultdict
-from operator import itemgetter
-
-# read file
-hits = pd.read_parquet('hits.parquet', engine='pyarrow')
-
-# build groups
-counts = defaultdict(int)
-for index, row in hits.iterrows():
- group = (row['UserID'], row['SearchPhrase']);
- # update the dict entry for the corresponding key
- counts[group] += 1
-
-# Print the top 10 values
-print (dict(sorted(counts.items(), key=itemgetter(1), reverse=True)[:10]))
-</code></pre>
-<p>This approach, while simple, is both slow and very memory inefficient. It requires over 40 seconds to compute the results for less than 1% of the dataset[^6]. Both DataFusion <code>28.0.0</code> and DuckDB <code>0.8.1</code> compute results in under 10 seconds for the <em>entire</em> dataset.</p>
-<p>To answer this query quickly and efficiently, you have to write your code such that it:</p>
-<ol>
-<li>Keeps all cores busy aggregating via parallelized computation</li>
-<li>Updates aggregate values quickly, using vectorizable loops that are easy for compilers to translate into the high performance <a href="https://en.wikipedia.org/wiki/Single_instruction,_multiple_data">SIMD</a> instructions available in modern CPUs.</li>
-</ol>
-<p>The rest of this article explains how grouping works in DataFusion and the improvements we made in <code>28.0.0</code>.</p>
-<h3 id="two-phase-parallel-partitioned-grouping">Two phase parallel partitioned grouping<a class="headerlink" href="#two-phase-parallel-partitioned-grouping" title="Permanent link">¶</a></h3>
-<p>Both DataFusion <code>27.0.</code> and <code>28.0.0</code> use state-of-the-art, two phase parallel hash partitioned grouping, similar to other high-performance vectorized engines like <a href="https://duckdb.org/2022/03/07/aggregate-hashtable.html">DuckDB’s Parallel Grouped Aggregates</a>. In pictures this looks like:</p>
-<pre><code class="language-text"> ▲ ▲
- │ │
- │ │
- │ │
-┌───────────────────────┐ ┌───────────────────┐
-│ GroupBy │ │ GroupBy │ Step 4
-│ (Final) │ │ (Final) │
-└───────────────────────┘ └───────────────────┘
- ▲ ▲
- │ │
- └────────────┬───────────┘
- │
- │
- ┌─────────────────────────┐
- │ Repartition │ Step 3
- │ HASH(x) │
- └─────────────────────────┘
- ▲
- │
- ┌────────────┴──────────┐
- │ │
- │ │
- ┌────────────────────┐ ┌─────────────────────┐
- │ GroupyBy │ │ GroupBy │ Step 2
- │ (Partial) │ │ (Partial) │
- └────────────────────┘ └─────────────────────┘
- ▲ ▲
- ┌──┘ └─┐
- │ │
- .─────────. .─────────.
- ,─' '─. ,─' '─.
-; Input : ; Input : Step 1
-: Stream 1 ; : Stream 2 ;
- ╲ ╱ ╲ ╱
- '─. ,─' '─. ,─'
- `───────' `───────'
-</code></pre>
-<p><strong>Figure 2</strong>: Two phase repartitioned grouping: data flows from bottom (source) to top (results) in two phases. First (Steps 1 and 2), each core reads the data into a core-specific hash table, computing intermediate aggregates without any cross-core coordination. Then (Steps 3 and 4) DataFusion divides the data (“repartitions”) into distinct subsets by group value, and each subset is sent to a specific core which computes the final aggregate.</p>
-<p>The two phases are critical for keeping cores busy in a multi-core system. Both phases use the same hash table approach (explained in the next section), but differ in how the groups are distributed and the partial results emitted from the accumulators. The first phase aggregates data as soon as possible after it is produced. However, as shown in Figure 2, the groups can be anywhere in any input, so the same group is often found on many different cores. The second phase uses a hash fun [...]
-<pre><code> ┌─────┐ ┌─────┐
- │ 1 │ │ 3 │
- │ 2 │ │ 4 │ 2. After Repartitioning: each
- └─────┘ └─────┘ group key appears in exactly
- ┌─────┐ ┌─────┐ one partition
- │ 1 │ │ 3 │
- │ 2 │ │ 4 │
- └─────┘ └─────┘
-
-─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
-
- ┌─────┐ ┌─────┐
- │ 2 │ │ 2 │
- │ 1 │ │ 2 │
- │ 3 │ │ 3 │
- │ 4 │ │ 1 │
- └─────┘ └─────┘ 1. Input Stream: groups
- ... ... values are spread
- ┌─────┐ ┌─────┐ arbitrarily over each input
- │ 1 │ │ 4 │
- │ 4 │ │ 3 │
- │ 1 │ │ 1 │
- │ 4 │ │ 3 │
- │ 3 │ │ 2 │
- │ 2 │ │ 2 │
- │ 2 │ └─────┘
- └─────┘
-
- Core A Core B
-
-</code></pre>
-<p><strong>Figure 3</strong>: Group value distribution across 2 cores during aggregation phases. In the first phase, every group value <code>1</code>, <code>2</code>, <code>3</code>, <code>4</code>, is present in the input stream processed by each core. In the second phase, after repartitioning, the group values <code>1</code> and <code>2</code> are processed by core A, and values <code>3</code> and <code>4</code> are processed only by core B.</p>
-<p>There are some additional subtleties in the <a href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/physical_plan/aggregates/row_hash.rs">DataFusion implementation</a> not mentioned above due to space constraints, such as:</p>
-<ol>
-<li>The policy of when to emit data from the first phase’s hash table (e.g. because the data is partially sorted)</li>
-<li>Handling specific filters per aggregate (due to the <code>FILTER</code> SQL clause)</li>
-<li>Data types of intermediate values (which may not be the same as the final output for some aggregates such as <code>AVG</code>).</li>
-<li>Action taken when memory use exceeds its budget.</li>
-</ol>
-<h3 id="hash-grouping">Hash grouping<a class="headerlink" href="#hash-grouping" title="Permanent link">¶</a></h3>
-<p>DataFusion queries can compute many different aggregate functions for each group, both <a href="https://arrow.apache.org/datafusion/user-guide/sql/aggregate_functions.html">built in</a> and/or user defined <a href="https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.AggregateUDF.html"><code>AggregateUDFs</code></a>. The state for each aggregate function, called an <em>accumulator</em>, is tracked with a hash table (DataFusion uses the excellent <a href="https://docs.rs/ha [...]
-<h3 id="hash-grouping-in-2700">Hash grouping in <code>27.0.0</code><a class="headerlink" href="#hash-grouping-in-2700" title="Permanent link">¶</a></h3>
-<p>As shown in Figure 3, DataFusion <code>27.0.0</code> stores the data in a <a href="https://github.com/apache/arrow-datafusion/blob/4d93b6a3802151865b68967bdc4c7d7ef425b49a/datafusion/core/src/physical_plan/aggregates/utils.rs#L38-L50"><code>GroupState</code></a> structure which, unsurprisingly, tracks the state for each group. The state for each group consists of:</p>
-<ol>
-<li>The actual value of the group columns, in <a href="https://docs.rs/arrow-row/latest/arrow_row/index.html">Arrow Row</a> format.</li>
-<li>In-progress accumulations (e.g. the running counts for the <code>COUNT</code> aggregate) for each group, in one of two possible formats (<a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/expr/src/accumulator.rs#L24-L49"><code>Accumulator</code></a> or <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/aggregate/row_accumulator.rs#L26-L46"><code>RowAc [...]
-<li>Scratch space for tracking which rows match each aggregate in each batch.</li>
-</ol>
-<pre><code> ┌──────────────────────────────────────┐
- │ │
- │ ... │
- │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
- │ ┃ ┃ │
- ┌─────────┐ │ ┃ ┌──────────────────────────────┐ ┃ │
- │ │ │ ┃ │group values: OwnedRow │ ┃ │
- │ ┌─────┐ │ │ ┃ └──────────────────────────────┘ ┃ │
- │ │ 5 │ │ │ ┃ ┌──────────────────────────────┐ ┃ │
- │ ├─────┤ │ │ ┃ │Row accumulator: │ ┃ │
- │ │ 9 │─┼────┐ │ ┃ │Vec<u8> │ ┃ │
- │ ├─────┤ │ │ │ ┃ └──────────────────────────────┘ ┃ │
- │ │ ... │ │ │ │ ┃ ┌──────────────────────┐ ┃ │
- │ ├─────┤ │ │ │ ┃ │┌──────────────┐ │ ┃ │
- │ │ 1 │ │ │ │ ┃ ││Accumulator 1 │ │ ┃ │
- │ ├─────┤ │ │ │ ┃ │└──────────────┘ │ ┃ │
- │ │ ... │ │ │ │ ┃ │┌──────────────┐ │ ┃ │
- │ └─────┘ │ │ │ ┃ ││Accumulator 2 │ │ ┃ │
- │ │ │ │ ┃ │└──────────────┘ │ ┃ │
- └─────────┘ │ │ ┃ │ Box<dyn Accumulator> │ ┃ │
- Hash Table │ │ ┃ └──────────────────────┘ ┃ │
- │ │ ┃ ┌─────────────────────────┐ ┃ │
- │ │ ┃ │scratch indices: Vec<u32>│ ┃ │
- │ │ ┃ └─────────────────────────┘ ┃ │
- │ │ ┃ GroupState ┃ │
- └─────▶ │ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ │
- │ │
- Hash table tracks an │ ... │
- index into group_states │ │
- └──────────────────────────────────────┘
- group_states: Vec<GroupState>
-
- There is one GroupState PER GROUP
-
-</code></pre>
-<p><strong>Figure 4</strong>: Hash group operator structure in DataFusion <code>27.0.0</code>. A hash table maps each group to a GroupState which contains all the per-group states.</p>
-<p>To compute the aggregate, DataFusion performs the following steps for each input batch:</p>
-<ol>
-<li>Calculate hash using <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/hash_utils.rs#L264-L307">efficient vectorized code</a>, specialized for each data type.</li>
-<li>Determine group indexes for each input row using the hash table (creating new entries for newly seen groups).</li>
-<li><a href="https://github.com/apache/arrow-datafusion/blob/4ab8be57dee3bfa72dd105fbd7b8901b873a4878/datafusion/core/src/physical_plan/aggregates/row_hash.rs#L562-L602">Update Accumulators for each group that had input rows,</a> assembling the rows into a contiguous range for vectorized accumulator if there are a sufficient number of them.</li>
-</ol>
-<p>DataFusion also stores the hash values in the table to avoid potentially costly hash recomputation when resizing the hash table.</p>
-<p>This scheme works very well for a relatively small number of distinct groups: all accumulators are efficiently updated with large contiguous batches of rows.</p>
-<p>However, this scheme is not ideal for high cardinality grouping due to:</p>
-<ol>
-<li><strong>Multiple allocations per group</strong> for the group value row format, as well as for the <code>RowAccumulator</code>s and each <code>Accumulator</code>. The <code>Accumulator</code> may have additional allocations within it as well.</li>
-<li><strong>Non-vectorized updates:</strong> Accumulator updates often fall back to a slower non-vectorized form because the number of distinct groups is large (and thus number of values per group is small) in each input batch.</li>
-</ol>
-<h3 id="hash-grouping-in-2800">Hash grouping in <code>28.0.0</code><a class="headerlink" href="#hash-grouping-in-2800" title="Permanent link">¶</a></h3>
-<p>For <code>28.0.0</code>, we rewrote the core group by implementation following traditional system optimization principles: fewer allocations, type specialization, and aggressive vectorization.</p>
-<p>DataFusion <code>28.0.0</code> uses the same RawTable and still stores group indexes. The major differences, as shown in Figure 4, are:</p>
-<ol>
-<li>Group values are stored either<ol>
-<li>Inline in the <code>RawTable</code> (for single columns of primitive types), where the conversion to Row format costs more than its benefit</li>
-<li>In a separate <a href="https://docs.rs/arrow-row/latest/arrow_row/struct.Row.html">Rows</a> structure with a single contiguous allocation for all groups values, rather than an allocation per group. Accumulators manage the state for all the groups internally, so the code to update intermediate values is a tight type specialized loop. The new <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/aggregate/groups_a [...]
-</ol>
-</li>
-</ol>
-<pre><code>┌───────────────────────────────────┐ ┌───────────────────────┐
-│ ┌ ─ ─ ─ ─ ─ ┐ ┌─────────────────┐│ │ ┏━━━━━━━━━━━━━━━━━━━┓ │
-│ │ ││ │ ┃ ┌──────────────┐ ┃ │
-│ │ │ │ ┌ ─ ─ ┐┌─────┐ ││ │ ┃ │┌───────────┐ │ ┃ │
-│ │ X │ 5 │ ││ │ ┃ ││ value1 │ │ ┃ │
-│ │ │ │ ├ ─ ─ ┤├─────┤ ││ │ ┃ │└───────────┘ │ ┃ │
-│ │ Q │ 9 │──┼┼──┐ │ ┃ │ ... │ ┃ │
-│ │ │ │ ├ ─ ─ ┤├─────┤ ││ └──┼─╋─▶│ │ ┃ │
-│ │ ... │ ... │ ││ │ ┃ │┌───────────┐ │ ┃ │
-│ │ │ │ ├ ─ ─ ┤├─────┤ ││ │ ┃ ││ valueN │ │ ┃ │
-│ │ H │ 1 │ ││ │ ┃ │└───────────┘ │ ┃ │
-│ │ │ │ ├ ─ ─ ┤├─────┤ ││ │ ┃ │values: Vec<T>│ ┃ │
-│ Rows │ ... │ ... │ ││ │ ┃ └──────────────┘ ┃ │
-│ │ │ │ └ ─ ─ ┘└─────┘ ││ │ ┃ ┃ │
-│ ─ ─ ─ ─ ─ ─ │ ││ │ ┃ GroupsAccumulator ┃ │
-│ └─────────────────┘│ │ ┗━━━━━━━━━━━━━━━━━━━┛ │
-│ Hash Table │ │ │
-│ │ │ ... │
-└───────────────────────────────────┘ └───────────────────────┘
- GroupState Accumulators
-
-
-Hash table value stores group_indexes One GroupsAccumulator
-and group values. per aggregate. Each
- stores the state for
-Group values are stored either inline *ALL* groups, typically
-in the hash table or in a single using a native Vec<T>
-allocation using the arrow Row format
-</code></pre>
-<p><strong>Figure 5</strong>: Hash group operator structure in DataFusion <code>28.0.0</code>. Group values are stored either directly in the hash table, or in a single allocation using the arrow Row format. The hash table contains group indexes. A single <code>GroupsAccumulator</code> stores the per-aggregate state for <em>all</em> groups.</p>
-<p>This new structure improves performance significantly for high cardinality groups due to:</p>
-<ol>
-<li><strong>Reduced allocations</strong>: There are no longer any individual allocations per group.</li>
-<li><strong>Contiguous native accumulator states</strong>: Type-specialized accumulators store the values for all groups in a single contiguous allocation using a <a href="https://doc.rust-lang.org/std/vec/struct.Vec.html">Rust Vec<T></a> of some native type.</li>
-<li><strong>Vectorized state update</strong>: The inner aggregate update loops, which are type-specialized and in terms of native <code>Vec</code>s, are well-vectorized by the Rust compiler (thanks <a href="https://llvm.org/">LLVM</a>!).</li>
-</ol>
-<h3 id="notes">Notes<a class="headerlink" href="#notes" title="Permanent link">¶</a></h3>
-<p>Some vectorized grouping implementations store the accumulator state row-wise directly in the hash table, which often uses modern CPU caches efficiently. Managing accumulator state in columnar fashion may sacrifice some cache locality, however it ensures the size of the hash table remains small, even when there are large numbers of groups and aggregates, making it easier for the compiler to vectorize the accumulator update.</p>
-<p>Depending on the cost of recomputing hash values, DataFusion <code>28.0.0</code> may or may not store the hash values in the table. This optimizes the tradeoff between the cost of computing the hash value (which is expensive for strings, for example) vs. the cost of storing it in the hash table.</p>
-<p>One subtlety that arises from pushing state updates into GroupsAccumulators is that each accumulator must handle similar variations with/without filtering and with/without nulls in the input. DataFusion <code>28.0.0</code> uses a templated <a href="https://github.com/apache/arrow-datafusion/blob/a6dcd943051a083693c352c6b4279156548490a0/datafusion/physical-expr/src/aggregate/groups_accumulator/accumulate.rs#L28-L54"><code>NullState</code></a> which encapsulates these common patterns ac [...]
-<p>The code structure is heavily influenced by the fact DataFusion is implemented using <a href="https://www.rust-lang.org/">Rust</a>, a new(ish) systems programming language focused on speed and safety. Rust heavily discourages many of the traditional pointer casting “tricks” used in C/C++ hash grouping implementations. The DataFusion aggregation code is almost entirely <a href="https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html#:~:text=Safe%20Rust%20is%20the%20true,Undefined%2 [...]
-<h2 id="clickbench-results">ClickBench results<a class="headerlink" href="#clickbench-results" title="Permanent link">¶</a></h2>
-<p>The full results of running the <a href="https://github.com/ClickHouse/ClickBench/tree/main">ClickBench</a> queries against the single Parquet file with DataFusion <code>27.0.0</code>, DataFusion <code>28.0.0</code>, and DuckDB <code>0.8.1</code> are below. These numbers were run on a GCP <code>e2-standard-8 machine</code> with 8 cores and 32 GB of RAM, using the scripts <a href="https://github.com/alamb/datafusion-duckdb-benchmark">here</a>.</p>
-<p>As the industry moves towards data systems assembled from components, it is increasingly important that they exchange data using open standards such as <a href="https://arrow.apache.org/">Apache Arrow</a> and <a href="https://parquet.apache.org/">Parquet</a> rather than custom storage and in-memory formats. Thus, this benchmark uses a single input Parquet file representative of many DataFusion users and aligned with the current trend in analytics of avoiding a costly load/transformati [...]
-<p>DataFusion now reaches near-DuckDB-speeds querying Parquet data. While we don’t plan to engage in a benchmarking shootout with a team that literally wrote <a href="https://dl.acm.org/doi/abs/10.1145/3209950.3209955">Fair Benchmarking Considered Difficult</a>, hopefully everyone can agree that DataFusion <code>28.0.0</code> is a significant improvement.</p>
-<p><img src="/blog/images/datafusion_fast_grouping/full.png" width="700"/></p>
-<p><strong>Figure 6</strong>: Performance of DataFusion <code>27.0.0</code>, DataFusion <code>28.0.0</code>, and DuckDB <code>0.8.1</code> on all 43 ClickBench queries against a single <code>hits.parquet</code> file. Lower is better.</p>
-<h3 id="notes_1">Notes<a class="headerlink" href="#notes_1" title="Permanent link">¶</a></h3>
-<p>DataFusion <code>27.0.0</code> was not able to run several queries due to either planner bugs (Q9, Q11, Q12, 14) or running out of memory (Q33). DataFusion <code>28.0.0</code> solves those issues.</p>
-<p>DataFusion is faster than DuckDB for query 21 and 22, likely due to optimized implementations of string pattern matching.</p>
-<h2 id="conclusion-performance-matters">Conclusion: performance matters<a class="headerlink" href="#conclusion-performance-matters" title="Permanent link">¶</a></h2>
-<p>Improving aggregation performance by more than a factor of two allows developers building products and projects with DataFusion to spend more time on value-added domain specific features. We believe building systems with DataFusion is much faster than trying to build something similar from scratch. DataFusion increases productivity because it eliminates the need to rebuild well-understood, but costly to implement, analytic database technology. While we’re pleased with the improvements [...]
-<h2 id="acknowledgments">Acknowledgments<a class="headerlink" href="#acknowledgments" title="Permanent link">¶</a></h2>
-<p>DataFusion is a <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">community effort</a> and this work was not possible without contributions from many in the community. A special shout out to <a href="https://github.com/sunchao">sunchao</a>, <a href="https://github.com/jyshen">yjshen</a>, <a href="https://github.com/yahoNanJing">yahoNanJing</a>, <a href="https://github.com/mingmwang">mingmwang</a>, <a href="https://github.com/ozankabak">ozankabak</a>, < [...]
-<h2 id="about-datafusion">About DataFusion<a class="headerlink" href="#about-datafusion" title="Permanent link">¶</a></h2>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine and database toolkit, written in <a href="https://www.rust-lang.org/">Rust</a>, that uses <a href="https://arrow.apache.org/">Apache Arrow</a> as its in-memory format. DataFusion, along with <a href="https://calcite.apache.org/">Apache Calcite</a>, Facebook’s <a href="https://github.com/facebookincubator/velox">Velox</a>, and similar technology are part of the next generation “<a h [...]
-<!-- Footnotes themselves at the bottom. -->
-<h2 id="notes_2">Notes<a class="headerlink" href="#notes_2" title="Permanent link">¶</a></h2>
-<p>[^1]: <code>SELECT COUNT(*) FROM 'hits.parquet';</code></p>
-<p>[^2]: <code>SELECT COUNT(DISTINCT "UserID") as num_users FROM 'hits.parquet';</code></p>
-<p>[^3]: <code>SELECT COUNT(DISTINCT "SearchPhrase") as num_phrases FROM 'hits.parquet';</code></p>
-<p>[^4]: <code>SELECT COUNT(*) FROM (SELECT DISTINCT "UserID", "SearchPhrase" FROM 'hits.parquet')</code></p>
-<p>[^5]: Full script at <a href="https://github.com/alamb/datafusion-duckdb-benchmark/blob/main/hash.py">hash.py</a></p>
-<p>[^6]: <a href="https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_%7B%7D.parquet">hits_0.parquet</a>, one of the files from the partitioned ClickBench dataset, which has <code>100,000</code> rows and is 117 MB in size. The entire dataset has <code>100,000,000</code> rows in a single 14 GB Parquet file. The script did not complete on the entire dataset after 40 minutes, and used 212 GB RAM at peak.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#aggregating-millions-of-groups-fast-in-apache-arrow-datafusion">Aggregating Millions of Groups Fast in Apache Arrow DataFusion</a></li>
-<li><a href="#tldr">TLDR</a></li>
-<li><a href="#introduction-to-high-cardinality-grouping">Introduction to high cardinality grouping</a></li>
-<li><a href="#the-solution">The solution</a><ul>
-<li><a href="#two-phase-parallel-partitioned-grouping">Two phase parallel partitioned grouping</a></li>
-<li><a href="#hash-grouping">Hash grouping</a></li>
-<li><a href="#hash-grouping-in-2700">Hash grouping in 27.0.0</a></li>
-<li><a href="#hash-grouping-in-2800">Hash grouping in 28.0.0</a></li>
-<li><a href="#notes">Notes</a></li>
-</ul>
-</li>
-<li><a href="#clickbench-results">ClickBench results</a><ul>
-<li><a href="#notes_1">Notes</a></li>
-</ul>
-</li>
-<li><a href="#conclusion-performance-matters">Conclusion: performance matters</a></li>
-<li><a href="#acknowledgments">Acknowledgments</a></li>
-<li><a href="#about-datafusion">About DataFusion</a></li>
-<li><a href="#notes_2">Notes</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/01/19/datafusion-34.0.0/index.html b/blog/2024/01/19/datafusion-34.0.0/index.html
deleted file mode 100644
index cf98623..0000000
--- a/blog/2024/01/19/datafusion-34.0.0/index.html
+++ /dev/null
@@ -1,392 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024 - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024
- </h1>
- <p>Posted on: Fri 19 January 2024 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#improved-performance">Improved Performance 🚀</a></li>
-<li><a href="#new-features">New Features ✨</a><ul>
-<li><a href="#dml-insert-creating-files">DML / Insert / Creating Files</a></li>
-<li><a href="#improved-struct-and-array-support">Improved STRUCT and ARRAY support</a></li>
-<li><a href="#other-features">Other Features</a></li>
-</ul>
-</li>
-<li><a href="#building-systems-is-easier-with-datafusion">Building Systems is Easier with DataFusion 🛠️</a><ul>
-<li><a href="#documentation">Documentation</a></li>
-<li><a href="#user-defined-window-and-table-functions">User Defined Window and Table Functions</a><ul>
-<li><a href="#growth-of-datafusion">Growth of DataFusion 📈</a></li>
-</ul>
-</li>
-</ul>
-</li>
-<li><a href="#datafusion-in-2024">DataFusion in 2024 🥳</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
-<p>We recently <a href="https://crates.io/crates/datafusion/34.0.0">released DataFusion 34.0.0</a>. This blog highlights some of the major
-improvements since we <a href="https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/.">released DataFusion 26.0.0</a> (spoiler alert there are many)
-and a preview of where the community plans to focus in the next 6 months.</p>
-<p><a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that
-uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
-create new, fast data centric systems such as databases, dataframe libraries,
-machine learning and streaming applications. While <a href="https://arrow.apache.org/datafusion/user-guide/introduction.html#project-goals">DataFusion’s primary design
-goal</a> is to accelerate creating other data centric systems, it has a
-reasonable experience directly out of the box as a <a href="https://arrow.apache.org/datafusion-python/">dataframe library</a> and
-<a href="https://arrow.apache.org/datafusion/user-guide/cli.html">command line SQL tool</a>.</p>
-<p>This may also be our last update on the Apache Arrow Site. Future
-updates will likely be on the DataFusion website as we are working to <a href="https://github.com/apache/arrow-datafusion/discussions/6475">graduate
-to a top level project</a> (Apache Arrow DataFusion → Apache DataFusion!) which
-will help focus governance and project growth. Also exciting, our <a href="https://github.com/apache/arrow-datafusion/discussions/8522">first
-DataFusion in person meetup</a> is planned for March 2024.</p>
-<p>DataFusion is very much a community endeavor. Our core thesis is that as a
-community we can build much more advanced technology than any of us as
-individuals or companies could alone. In the last 6 months between <code>26.0.0</code> and
-<code>34.0.0</code>, community growth has been strong. We accepted and reviewed over a
-thousand PRs from 124 different committers, created over 650 issues and closed 517
-of them.
-You can find a list of all changes in the detailed <a href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/CHANGELOG.md">CHANGELOG</a>.</p>
-<!--
-$ git log --pretty=oneline 26.0.0..34.0.0 . | wc -l
- 1009
-
-$ git shortlog -sn 26.0.0..34.0.0 . | wc -l
- 124
-
-https://crates.io/crates/datafusion/26.0.0
-DataFusion 26 released June 7, 2023
-
-https://crates.io/crates/datafusion/34.0.0
-DataFusion 34 released Dec 17, 2023
-
-Issues created in this time: 214 open, 437 closed
-https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+created%3A2023-06-23..2023-12-17
-
-Issues closes: 517
-https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2023-06-23..2023-12-17+
-
-PRs merged in this time 908
-https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2023-06-23..2023-12-17
--->
-<h1 id="improved-performance">Improved Performance 🚀<a class="headerlink" href="#improved-performance" title="Permanent link">¶</a></h1>
-<p>Performance is a key feature of DataFusion, DataFusion is
-more than 2x faster on <a href="https://benchmark.clickhouse.com/">ClickBench</a> compared to version <code>25.0.0</code>, as shown below:</p>
-<!--
- Scripts: https://github.com/alamb/datafusion-duckdb-benchmark/tree/datafusion-25-34
- Spreadsheet: https://docs.google.com/spreadsheets/d/1FtI3652WIJMC5LmJbLfT3G06w0JQIxEPG4yfMafexh8/edit#gid=1879366976
- Average runtime on 25.0.0: 7.2s (for the queries that actually ran)
- Average runtime on 34.0.0: 3.6s (for the same queries that ran in 25.0.0)
--->
-<figure class="text-center">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare-new.png"/>
-<figcaption>
-<b>Figure 1</b>: Performance improvement between <code>25.0.0</code> and <code>34.0.0</code> on ClickBench.
- Note that DataFusion <code>25.0.0</code>, could not run several queries due to
- unsupported SQL (Q9, Q11, Q12, Q14) or memory requirements (Q33).
- </figcaption>
-</figure>
-<figure class="text-center">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-34.0.0/compare.png"/>
-<figcaption>
-<b>Figure 2</b>: Total query runtime for DataFusion <code>34.0.0</code> and DataFusion <code>25.0.0</code>.
- </figcaption>
-</figure>
-<p>Here are some specific enhancements we have made to improve performance:
-* <a href="https://arrow.apache.org/blog/2023/08/05/datafusion_fast_grouping/">2-3x better aggregation performance with many distinct groups</a>
-* Partially ordered grouping / streaming grouping
-* [Specialized operator for "TopK" <code>ORDER BY LIMIT XXX</code>]
-* [Specialized operator for <code>min(col) GROUP BY .. ORDER by min(col) LIMIT XXX</code>]
-* <a href="https://github.com/apache/arrow-datafusion/pull/8126">Improved join performance</a>
-* Eliminate redundant sorting with sort order aware optimizers</p>
-<h1 id="new-features">New Features ✨<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h1>
-<h2 id="dml-insert-creating-files">DML / Insert / Creating Files<a class="headerlink" href="#dml-insert-creating-files" title="Permanent link">¶</a></h2>
-<p>DataFusion now supports writing data in parallel, to individual or multiple
-files, using <code>Parquet</code>, <code>CSV</code>, <code>JSON</code>, <code>ARROW</code> and user defined formats.
-<a href="https://github.com/apache/arrow-datafusion/pull/7655">Benchmark results</a> show improvements up to 5x in some cases.</p>
-<p>Similarly to reading, data can now be written to any [<code>ObjectStore</code>]
-implementation, including AWS S3, Azure Blob Storage, GCP Cloud Storage, local
-files, and user defined implementations. While reading from <a href="https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html#features">hive style
-partitioned tables</a> has long been supported, it is now possible to write to such
-tables as well.</p>
-<p>For example, to write to a local file:</p>
-<pre><code class="language-sql">❯ CREATE EXTERNAL TABLE awesome_table(x INT) STORED AS PARQUET LOCATION '/tmp/my_awesome_table';
-0 rows in set. Query took 0.003 seconds.
-
-❯ INSERT INTO awesome_table SELECT x * 10 FROM my_source_table;
-+-------+
-| count |
-+-------+
-| 3 |
-+-------+
-1 row in set. Query took 0.024 seconds.
-</code></pre>
-<p>You can also write to files with the [<code>COPY</code>], similarly to [DuckDB’s <code>COPY</code>]:</p>
-<pre><code class="language-sql">❯ COPY (SELECT x + 1 FROM my_source_table) TO '/tmp/output.json';
-+-------+
-| count |
-+-------+
-| 3 |
-+-------+
-1 row in set. Query took 0.014 seconds.
-</code></pre>
-<pre><code class="language-shell">$ cat /tmp/output.json
-{"x":1}
-{"x":2}
-{"x":3}
-</code></pre>
-<h2 id="improved-struct-and-array-support">Improved <code>STRUCT</code> and <code>ARRAY</code> support<a class="headerlink" href="#improved-struct-and-array-support" title="Permanent link">¶</a></h2>
-<p>DataFusion <code>34.0.0</code> has much improved <code>STRUCT</code> and <code>ARRAY</code>
-support, including a full range of <a href="https://arrow.apache.org/datafusion/user-guide/sql/scalar_functions.html#struct-functions">struct functions</a> and <a href="https://arrow.apache.org/datafusion/user-guide/sql/scalar_functions.html#array-functions">array functions</a>.</p>
-<!--
-❯ create table my_table as values ([1,2,3]), ([2]), ([4,5]);
--->
-<p>For example, you can now use <code>[]</code> syntax and <code>array_length</code> to access and inspect arrays:</p>
-<pre><code class="language-sql">❯ SELECT column1,
- column1[1] AS first_element,
- array_length(column1) AS len
- FROM my_table;
-+-----------+---------------+-----+
-| column1 | first_element | len |
-+-----------+---------------+-----+
-| [1, 2, 3] | 1 | 3 |
-| [2] | 2 | 1 |
-| [4, 5] | 4 | 2 |
-+-----------+---------------+-----+
-</code></pre>
-<pre><code class="language-sql">❯ SELECT column1, column1['c0'] FROM my_table;
-+------------------+----------------------+
-| column1 | my_table.column1[c0] |
-+------------------+----------------------+
-| {c0: foo, c1: 1} | foo |
-| {c0: bar, c1: 2} | bar |
-+------------------+----------------------+
-2 rows in set. Query took 0.002 seconds.
-</code></pre>
-<h2 id="other-features">Other Features<a class="headerlink" href="#other-features" title="Permanent link">¶</a></h2>
-<p>Other notable features include:
-* Support aggregating datasets that exceed memory size, with <a href="https://github.com/apache/arrow-datafusion/pull/7400">group by spill to disk</a>
-* All operators now track and limit their memory consumption, including Joins</p>
-<h1 id="building-systems-is-easier-with-datafusion">Building Systems is Easier with DataFusion 🛠️<a class="headerlink" href="#building-systems-is-easier-with-datafusion" title="Permanent link">¶</a></h1>
-<h2 id="documentation">Documentation<a class="headerlink" href="#documentation" title="Permanent link">¶</a></h2>
-<p>It is easier than ever to get started using DataFusion with the
-new <a href="https://arrow.apache.org/datafusion/library-user-guide/index.html">Library Users Guide</a> as well as significantly improved the <a href="https://docs.rs/datafusion/latest/datafusion/index.html">API documentation</a>. </p>
-<h2 id="user-defined-window-and-table-functions">User Defined Window and Table Functions<a class="headerlink" href="#user-defined-window-and-table-functions" title="Permanent link">¶</a></h2>
-<p>In addition to DataFusion's <a href="https://arrow.apache.org/datafusion/library-user-guide/adding-udfs.html#adding-a-scalar-udf">User Defined Scalar Functions</a>, and <a href="https://arrow.apache.org/datafusion/library-user-guide/adding-udfs.html#adding-an-aggregate-udf">User Defined Aggregate Functions</a>, DataFusion now supports <a href="https://arrow.apache.org/datafusion/library-user-guide/adding-udfs.html#adding-a-window-udf">User Defined Window Functions</a>
- and <a href="https://arrow.apache.org/datafusion/library-user-guide/adding-udfs.html#adding-a-user-defined-table-function">User Defined Table Functions</a>.</p>
-<p>For example, [the <code>datafusion-cli</code>] implements a DuckDB style [<code>parquet_metadata</code>]
-function as a user defined table function (<a href="https://github.com/apache/arrow-datafusion/blob/3f219bc929cfd418b0e3d3501f8eba1d5a2c87ae/datafusion-cli/src/functions.rs#L222-L248">source code here</a>): </p>
-<pre><code class="language-sql">❯ SELECT
- path_in_schema, row_group_id, row_group_num_rows, stats_min, stats_max, total_compressed_size
-FROM
- parquet_metadata('hits.parquet')
-WHERE path_in_schema = '"WatchID"'
-LIMIT 3;
-
-+----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
-| path_in_schema | row_group_id | row_group_num_rows | stats_min | stats_max | total_compressed_size |
-+----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
-| "WatchID" | 0 | 450560 | 4611687214012840539 | 9223369186199968220 | 3883759 |
-| "WatchID" | 1 | 612174 | 4611689135232456464 | 9223371478009085789 | 5176803 |
-| "WatchID" | 2 | 344064 | 4611692774829951781 | 9223363791697310021 | 3031680 |
-+----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
-3 rows in set. Query took 0.053 seconds.
-</code></pre>
-<h3 id="growth-of-datafusion">Growth of DataFusion 📈<a class="headerlink" href="#growth-of-datafusion" title="Permanent link">¶</a></h3>
-<p>DataFusion has been appearing more publicly in the wild. For example
-* New projects built using DataFusion such as <a href="https://lancedb.com/">lancedb</a>, <a href="https://glaredb.com/">GlareDB</a>, <a href="https://www.arroyo.dev/">Arroyo</a>, and <a href="https://github.com/cmu-db/optd">optd</a>.
-* Public talks such as <a href="https://www.youtube.com/watch?v=AJU9rdRNk9I">Apache Arrow Datafusion: Vectorized
- Execution Framework For Maximum Performance</a> in <a href="https://www.bagevent.com/event/8432178">CommunityOverCode Asia 2023</a>
-* Blogs posts such as <a href="https://www.synnada.ai/blog/apache-arrow-arrow-datafusion-ai-native-data-infra-an-interview-with-our-ceo-ozan">Apache Arrow, Arrow/DataFusion, AI-native Data Infra</a>,
- <a href="https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/">Flight, DataFusion, Arrow, and Parquet: Using the FDAP Architecture to build InfluxDB 3.0</a>, and
- <a href="https://www.linkedin.com/pulse/guide-user-defined-functions-apache-arrow-datafusion-dade-aderemi/">A Guide to User-Defined Functions in Apache Arrow DataFusion</a></p>
-<p>We have also <a href="https://github.com/apache/arrow-datafusion/issues/6782">submitted a paper</a> to <a href="https://2024.sigmod.org/">SIGMOD 2024</a>, one of the
-premiere database conferences, describing DataFusion in a technically formal
-style and making the case that it is possible to create a modular and extensive query engine
-without sacrificing performance. We hope this paper helps people
-evaluating DataFusion for their needs understand it better.</p>
-<h1 id="datafusion-in-2024">DataFusion in 2024 🥳<a class="headerlink" href="#datafusion-in-2024" title="Permanent link">¶</a></h1>
-<p>Some major initiatives from contributors we know of this year are:</p>
-<ol>
-<li>
-<p><em>Modularity</em>: Make DataFusion even more modular, such as <a href="https://github.com/apache/arrow-datafusion/issues/8045">unifying
- built in and user functions</a>, making it easier to customize
- DataFusion's behavior.</p>
-</li>
-<li>
-<p><em>Community Growth</em>: Graduate to our own top level Apache project, and
- subsequently add more committers and PMC members to keep pace with project
- growth.</p>
-</li>
-<li>
-<p><em>Use case white papers</em>: Write blog posts and videos explaining
- how to use DataFusion for real-world use cases.</p>
-</li>
-<li>
-<p><em>Testing</em>: Improve CI infrastructure and test coverage, more fuzz
- testing, and better functional and performance regression testing.</p>
-</li>
-<li>
-<p><em>Planning Time</em>: Reduce the time taken to plan queries, both <a href="https://github.com/apache/arrow-datafusion/issues/7698">wide
- tables of 1000s of columns</a>, and in <a href="https://github.com/apache/arrow-datafusion/issues/5637">general</a>.</p>
-</li>
-<li>
-<p><em>Aggregate Performance</em>: Improve the speed of <a href="https://github.com/apache/arrow-datafusion/issues/7000">aggregating "high cardinality"</a> data
- when there are many (e.g. millions) of distinct groups.</p>
-</li>
-<li>
-<p><em>Statistics</em>: <a href="https://github.com/apache/arrow-datafusion/issues/8227">Improved statistics handling</a> with an eye towards more
- sophisticated expression analysis and cost models.</p>
-</li>
-</ol>
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>If you are interested in contributing to DataFusion we would love to have you
-join us. You can try out DataFusion on some of your own data and projects and
-let us know how it goes, contribute suggestions, documentation, bug reports, or
-a PR with documentation, tests or code. A list of open issues
-suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a>.</p>
-<p>As the community grows, we are also looking to restart biweekly calls /
-meetings. Timezones are always a challenge for such meetings, but we hope to
-have two calls that can work for most attendees. If you are interested
-in helping, or just want to say hi, please drop us a note via one of
-the methods listed in our <a href="https://arrow.apache.org/datafusion/contributor-guide/communication.html">Communication Doc</a>.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#improved-performance">Improved Performance 🚀</a></li>
-<li><a href="#new-features">New Features ✨</a><ul>
-<li><a href="#dml-insert-creating-files">DML / Insert / Creating Files</a></li>
-<li><a href="#improved-struct-and-array-support">Improved STRUCT and ARRAY support</a></li>
-<li><a href="#other-features">Other Features</a></li>
-</ul>
-</li>
-<li><a href="#building-systems-is-easier-with-datafusion">Building Systems is Easier with DataFusion 🛠️</a><ul>
-<li><a href="#documentation">Documentation</a></li>
-<li><a href="#user-defined-window-and-table-functions">User Defined Window and Table Functions</a><ul>
-<li><a href="#growth-of-datafusion">Growth of DataFusion 📈</a></li>
-</ul>
-</li>
-</ul>
-</li>
-<li><a href="#datafusion-in-2024">DataFusion in 2024 🥳</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/03/06/comet-donation/index.html b/blog/2024/03/06/comet-donation/index.html
deleted file mode 100644
index f7353e7..0000000
--- a/blog/2024/03/06/comet-donation/index.html
+++ /dev/null
@@ -1,176 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Announcing Apache Arrow DataFusion Comet - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Announcing Apache Arrow DataFusion Comet
- </h1>
- <p>Posted on: Wed 06 March 2024 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#get-involved">Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
-<p>The Apache Arrow PMC is pleased to announce the donation of the <a href="https://github.com/apache/arrow-datafusion-comet">Comet project</a>,
-a native Spark SQL Accelerator built on <a href="https://arrow.apache.org/datafusion">Apache Arrow DataFusion</a>.</p>
-<p>Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to
-accelerate Spark workloads. It is designed as a drop-in
-replacement for Spark's JVM based SQL execution engine and offers significant
-performance improvements for some workloads as shown below.</p>
-<figure class="text-center">
-<img alt="Fig 1: Adaptive Arrow schema architecture overview." class="img-fluid" src="/blog/images/datafusion-comet/comet-architecture.png"/>
-<figcaption>
-<b>Figure 1</b>: With Comet, users interact with the same Spark ecosystem, tools
- and APIs such as Spark SQL. Queries still run through Spark's query optimizer and planner.
- However, the execution is delegated to Comet,
- which is significantly faster and more resource efficient than a JVM based
- implementation.
-</figcaption>
-</figure>
-<p>Comet is one of a growing class of projects that aim to accelerate Spark using
-native columnar engines such as the proprietary <a href="https://www.databricks.com/product/photon">Databricks Photon Engine</a> and
-open source projects <a href="https://incubator.apache.org/projects/gluten.html">Gluten</a>, <a href="https://github.com/NVIDIA/spark-rapids">Spark RAPIDS</a>, and <a href="https://github.com/kwai/blaze">Blaze</a> (also built using
-DataFusion).</p>
-<p>Comet was originally implemented at Apple and the engineers who worked on the
-project are also significant contributors to Arrow and DataFusion. Bringing
-Comet into the Apache Software Foundation will accelerate its development and
-grow its community of contributors and users.</p>
-<h1 id="get-involved">Get Involved<a class="headerlink" href="#get-involved" title="Permanent link">¶</a></h1>
-<p>Comet is still in the early stages of development and we would love to have you
-join us and help shape the project. We are working on an initial release, and
-expect to post another update with more details at that time.</p>
-<p>Before then, here are some ways to get involved:</p>
-<ul>
-<li>
-<p>Learn more by visiting the <a href="https://github.com/apache/arrow-datafusion-comet">Comet project</a> page and reading the <a href="https://lists.apache.org/thread/0q1rb11jtpopc7vt1ffdzro0omblsh0s">mailing list
- discussion</a> about the initial donation.</p>
-</li>
-<li>
-<p>Help us plan out the <a href="https://github.com/apache/arrow-datafusion-comet/issues/19">roadmap</a></p>
-</li>
-<li>
-<p>Try out the project and provide feedback, file issues, and contribute code.</p>
-</li>
-</ul>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#get-involved">Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/05/07/datafusion-tlp/index.html b/blog/2024/05/07/datafusion-tlp/index.html
deleted file mode 100644
index 5a4bd6d..0000000
--- a/blog/2024/05/07/datafusion-tlp/index.html
+++ /dev/null
@@ -1,176 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Announcing Apache Arrow DataFusion is now Apache DataFusion - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Announcing Apache Arrow DataFusion is now Apache DataFusion
- </h1>
- <p>Posted on: Tue 07 May 2024 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#background">Background</a></li>
-<li><a href="#why-now">Why now?</a></li>
-<li><a href="#looking-forward">Looking forward</a></li>
-<li><a href="#get-involved">Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
-<p>TLDR; <a href="https://arrow.apache.org/">Apache Arrow</a> DataFusion --> <a href="https://datafusion.apache.org/">Apache DataFusion</a></p>
-<p>The Arrow PMC and newly created DataFusion PMC are happy to announce that as of
-April 16, 2024 the Apache Arrow DataFusion subproject is now a top level
-<a href="https://www.apache.org/">Apache Software Foundation</a> project.</p>
-<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h2>
-<p>Apache DataFusion is a fast, extensible query engine for building high-quality
-data-centric systems in Rust, using the Apache Arrow in-memory format.</p>
-<p>When DataFusion was <a href="https://arrow.apache.org/blog/2019/02/04/datafusion-donation/">donated to the Apache Software Foundation</a> in 2019, the
-DataFusion community was not large enough to stand on its own and the Arrow
-project agreed to help support it. The community has grown significantly since
-2019, benefiting immensely from being part of Arrow and following <a href="https://www.apache.org/theapacheway/">The Apache
-Way</a>.</p>
-<h2 id="why-now">Why now?<a class="headerlink" href="#why-now" title="Permanent link">¶</a></h2>
-<p>The community <a href="https://github.com/apache/datafusion/discussions/6475">discussed graduating to a top level project publicly</a> for almost
-a year, as the project seemed ready to stand on its own and would benefit from
-more focused governance. For example, earlier in DataFusion's life many
-contributed to both <a href="https://github.com/apache/arrow-rs">arrow-rs</a> and DataFusion, but as DataFusion has matured many
-contributors, committers and PMC members focused more and more exclusively on
-DataFusion.</p>
-<h2 id="looking-forward">Looking forward<a class="headerlink" href="#looking-forward" title="Permanent link">¶</a></h2>
-<p>The future looks bright. There are now <a href="https://datafusion.apache.org/user-guide/introduction.html#known-users">10s of known projects built with
-DataFusion</a>, and that number continues to grow. We recently held our <a href="https://github.com/apache/datafusion/discussions/8522">first in
-person meetup</a> passed <a href="https://github.com/apache/datafusion/stargazers">5000 stars</a> on GitHub, <a href="https://github.com/apache/datafusion/issues/8373#issuecomment-2025133714">wrote a paper that was accepted
-at SIGMOD 2024</a>, and began work on <a href="https://github.com/apache/datafusion-comet">Comet</a>, an <a href="https://spark.apache.org/">Apache Spark</a> accelerator
-<a href="https://arrow.apache.org/blog/2024/03/06/comet-donation/">initially donated by Apple</a>.</p>
-<p>Thank you to everyone in the Arrow community who helped DataFusion grow and
-mature over the years, and we look forward to continuing our collaboration as
-projects. All future blogs and announcements will be posted on the <a href="https://datafusion.apache.org/">Apache
-DataFusion</a> website.</p>
-<h2 id="get-involved">Get Involved<a class="headerlink" href="#get-involved" title="Permanent link">¶</a></h2>
-<p>If you are interested in joining the community, we would love to have you join
-us. Get in touch using <a href="https://datafusion.apache.org/contributor-guide/communication.html">Communication Doc</a> and learn how to get involved in the
-<a href="https://datafusion.apache.org/contributor-guide/index.html">Contributor Guide</a>. We welcome everyone to try DataFusion on their
-own data and projects and let us know how it goes, contribute suggestions,
-documentation, bug reports, or a PR with documentation, tests or code.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#background">Background</a></li>
-<li><a href="#why-now">Why now?</a></li>
-<li><a href="#looking-forward">Looking forward</a></li>
-<li><a href="#get-involved">Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/07/20/datafusion-comet-0.1.0/index.html b/blog/2024/07/20/datafusion-comet-0.1.0/index.html
deleted file mode 100644
index aabfb3d..0000000
--- a/blog/2024/07/20/datafusion-comet-0.1.0/index.html
+++ /dev/null
@@ -1,214 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.1.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Comet 0.1.0 Release
- </h1>
- <p>Posted on: Sat 20 July 2024 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#project-status">Project Status</a></li>
-<li><a href="#current-performance">Current Performance</a></li>
-<li><a href="#roadmap">Roadmap</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>The Apache DataFusion PMC is pleased to announce the first official source release of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers five months of development work since the project was <a href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/">donated</a> to the Apache DataFusion
-project and is the result of merging 343 PRs from 41 contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.1.0.md">change log</a> for more information.</p>
-<p>This first release supports 15 <a href="https://datafusion.apache.org/comet/user-guide/datatypes.html#">data types</a>, 13 <a href="https://datafusion.apache.org/comet/user-guide/operators.html#">operators</a>, and 106 <a href="https://datafusion.apache.org/comet/user-guide/expressions.html#">expressions</a>. Comet is compatible with Apache
-Spark versions 3.3, 3.4, and 3.5. There is also experimental support for preview versions of Spark 4.0.</p>
-<h2 id="project-status">Project Status<a class="headerlink" href="#project-status" title="Permanent link">¶</a></h2>
-<p>The project's recent focus has been on fixing correctness and stability issues and implementing additional
-native operators and expressions so that a broader range of queries can be executed natively.</p>
-<p>Here are some of the highlights since the project was donated:</p>
-<ul>
-<li>Implemented native support for:</li>
-<li>SortMergeJoin</li>
-<li>HashJoin</li>
-<li>BroadcastHashJoin</li>
-<li>Columnar Shuffle</li>
-<li>More aggregate expressions</li>
-<li>Window aggregates</li>
-<li>Many Spark-compatible CAST expressions</li>
-<li>Implemented a simple Spark Fuzz Testing utility to find correctness issues</li>
-<li>Published a <a href="https://datafusion.apache.org/comet/user-guide/overview.html">User Guide</a> and <a href="https://datafusion.apache.org/comet/contributor-guide/contributing.html">Contributors Guide</a></li>
-<li>Created a <a href="https://github.com/apache/datafusion-benchmarks">DataFusion Benchmarks</a> repository with scripts and documentation for running benchmarks derived<br/>
- from TPC-H and TPC-DS with DataFusion and Comet</li>
-</ul>
-<h2 id="current-performance">Current Performance<a class="headerlink" href="#current-performance" title="Permanent link">¶</a></h2>
-<p>Comet already delivers a modest performance speedup for many queries, enabling faster data processing and
-shorter time-to-insights.</p>
-<p>We use benchmarks derived from the industry standard TPC-H and TPC-DS benchmarks for tracking progress with
-performance. The following chart shows the time it takes to run the 22 TPC-H queries against 100 GB of data in
-Parquet format using a single executor with eight cores. See the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>
-for details of the environment used for these benchmarks.</p>
-<p><img alt="Chart showing TPC-H benchmark results for Comet 0.1.0" class="img-fluid" src="/blog/images/comet-0.1.0/tpch_allqueries.png" width="100%"/></p>
-<p>Comet reduces the overall execution time from 626 seconds to 407 seconds, a 54% speedup (1.54x faster).</p>
-<p>Running the same queries with DataFusion standalone using the same number of cores results in a 3.9x speedup
-compared to Spark. Although this isn’t a fair comparison (DataFusion does not have shuffle or match Spark
-semantics in some cases, for example), it does give some idea about the potential future performance of
-Comet. Comet aims to provide a 2x-4x speedup for a wide range of queries once more operators and expressions
-can run natively.</p>
-<p>The following chart shows how much Comet currently accelerates each query from the benchmark.</p>
-<p><img alt="Chart showing TPC-H benchmark results for Comet 0.1.0" class="img-fluid" src="/blog/images/comet-0.1.0/tpch_queries_speedup.png" width="100%"/></p>
-<p>These benchmarks can be reproduced in any environment using the documentation in the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>. We
-encourage you to run these benchmarks in your environment or, even better, try Comet out with your existing Spark jobs.</p>
-<h2 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">¶</a></h2>
-<p>Comet is an open-source project, and contributors are welcome to work on any features they are interested in, but
-here are some current focus areas.</p>
-<ul>
-<li>Improve Performance & Reliability:</li>
-<li>Implement the remaining features needed so that all TPC-H queries can run entirely natively</li>
-<li>Implement spill support in SortMergeJoin</li>
-<li>Enable columnar shuffle by default</li>
-<li>Fully support Spark version 4.0.0</li>
-<li>Support more Spark operators and expressions</li>
-<li>We would like to support many more expressions natively in Comet, and this is a great place to start
- contributing. The contributors' guide has a section covering <a href="https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html">adding support for new expressions</a>.</li>
-<li>Move more Spark expressions into the <a href="https://crates.io/crates/datafusion-comet-spark-expr">datafusion-comet-spark-expr</a> crate. Although the main focus of the Comet
- project is to provide an accelerator for Apache Spark, we also publish a standalone crate containing
- Spark-compatible expressions that can be used by any project using DataFusion, without adding any dependencies
- on JVM or Apache Spark.</li>
-<li>Release Process & Documentation</li>
-<li>Implement a binary release process so that we can publish JAR files to Maven for all supported platforms</li>
-<li>Add documentation for running Spark and Comet in Kubernetes, and add example Dockerfiles.</li>
-</ul>
-<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project, and there is a Comet community video call held every four weeks on Wednesdays at 11:30 a.m. Eastern Time,
-which is 16:30 UTC during Eastern Standard Time and 15:30 UTC during Eastern Daylight Time. See the
-<a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">Comet Community Meeting</a> Google Document for the next scheduled meeting date, the video call link, and
-recordings of previous calls.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#project-status">Project Status</a></li>
-<li><a href="#current-performance">Current Performance</a></li>
-<li><a href="#roadmap">Roadmap</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/07/24/datafusion-40.0.0/index.html b/blog/2024/07/24/datafusion-40.0.0/index.html
deleted file mode 100644
index dd059e2..0000000
--- a/blog/2024/07/24/datafusion-40.0.0/index.html
+++ /dev/null
@@ -1,451 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion 40.0.0 Released - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion 40.0.0 Released
- </h1>
- <p>Posted on: Wed 24 July 2024 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#community-growth">Community Growth 📈</a></li>
-<li><a href="#improved-performance">Improved Performance 🚀</a></li>
-<li><a href="#improved-quality">Improved Quality 📋</a></li>
-<li><a href="#improved-documentation">Improved Documentation 📚</a></li>
-<li><a href="#new-features">New Features ✨</a></li>
-<li><a href="#sql">SQL</a></li>
-<li><a href="#sql-unparser-sql-formatter">SQL Unparser (SQL Formatter)</a></li>
-<li><a href="#low-level-apis-for-fast-parquet-access-indexing">Low Level APIs for Fast Parquet Access (indexing)</a><ul>
-<li><a href="#building-systems-is-easier-with-datafusion">Building Systems is Easier with DataFusion 🛠️</a></li>
-</ul>
-</li>
-<li><a href="#user-defined-sql-parsing-extensions">User Defined SQL Parsing Extensions</a></li>
-<li><a href="#pluggable-support-for-create-function">Pluggable Support for CREATE FUNCTION</a><ul>
-<li><a href="#looking-ahead-the-next-six-months">Looking Ahead: The Next Six Months 🔭</a></li>
-</ul>
-</li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<!-- see https://github.com/apache/datafusion/issues/9602 for details -->
-<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
-<p>We are proud to announce <a href="https://crates.io/crates/datafusion/40.0.0">DataFusion 40.0.0</a>. This blog highlights some of the
-many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/01/19/datafusion-34.0.0/">DataFusion 34.0.0</a> and a preview of
-what the community is thinking about in the next 6 months. We are hoping to make
-more regular blog posts -- if you are interested in helping write them, please
-reach out!</p>
-<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that
-uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
-create new, fast data centric systems such as databases, dataframe libraries,
-machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion’s primary design
-goal</a> is to accelerate the creation of other data centric systems, it has a
-reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/python/">dataframe library</a> and
-<a href="https://datafusion.apache.org/user-guide/cli/">command line SQL tool</a>.</p>
-<p>DataFusion's core thesis is that as a community, together we can build much more
-advanced technology than any of us as individuals or companies could do alone.
-Without DataFusion, highly performant vectorized query engines would remain
-the domain of a few large companies and world-class research institutions.
-With DataFusion, we can all build on top of a shared foundation, and focus on
-what makes our projects unique.</p>
-<h2 id="community-growth">Community Growth 📈<a class="headerlink" href="#community-growth" title="Permanent link">¶</a></h2>
-<p>In the last 6 months, between <code>34.0.0</code> and <code>40.0.0</code>, our community continues to
-grow in new and exciting ways.</p>
-<ol>
-<li>DataFusion became a top level Apache Software Foundation project (read the
- <a href="https://news.apache.org/foundation/entry/apache-software-foundation-announces-new-top-level-project-apache-datafusion">press release</a> and <a href="https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp/">blog post</a>).</li>
-<li>We added several PMC members and new
- committers: <a href="https://github.com/comphead">@comphead</a>, <a href="https://github.com/mustafasrepo">@mustafasrepo</a>, <a href="https://github.com/ozankabak">@ozankabak</a>, and <a href="https://github.com/waynexia">@waynexia</a> joined the PMC,
- <a href="https://github.com/jonahgao">@jonahgao</a> and <a href="https://github.com/lewiszlw">@lewiszlw</a> joined as committers. See the <a href="https://lists.apache.org/list.html?dev@datafusion.apache.org">mailing list</a> for
- more details.</li>
-<li><a href="https://datafusion.apache.org/comet/">DataFusion Comet</a> was <a href="https://arrow.apache.org/blog/2024/03/06/comet-donation/">donated</a> and is nearing its first release.</li>
-<li>In the <a href="https://github.com/apache/arrow-datafusion">core DataFusion repo</a> alone we reviewed and accepted almost 1500 PRs from 182 different
- committers, created over 1000 issues and closed 781 of them 🚀. This is up
- almost 50% from our last post (1000 PRs from 124 committers with 650 issues
- created in our last post) 🤯. All changes are listed in the detailed
- <a href="https://github.com/apache/datafusion/blob/main/datafusion/CHANGELOG.md">CHANGELOG</a>.</li>
-<li>DataFusion focused meetups happened or are happening in multiple cities
- around the world: <a href="https://github.com/apache/datafusion/discussions/8522">Austin</a>, <a href="https://github.com/apache/datafusion/discussions/10800">San Francisco</a>, <a href="https://www.huodongxing.com/event/5761971909400?td=1965290734055">Hangzhou</a>, <a href="https://github.com/apache/datafusion/discussions/11213">New York</a>, and
- <a href="https://github.com/apache/datafusion/discussions/11431">Belgrade</a>.</li>
-<li>Many new projects started in the <a href="https://github.com/datafusion-contrib">datafusion-contrib</a> organization, including
- <a href="https://github.com/datafusion-contrib/datafusion-table-providers">Table Providers</a>, <a href="https://github.com/datafusion-contrib/datafusion-sqlancer">SQLancer</a>, <a href="https://github.com/datafusion-contrib/datafusion-functions-variant">Open Variant</a>, <a href="https://github.com/datafusion-contrib/datafusion-functions-json">JSON</a>, and <a href="https://github.com/datafusion-contrib/datafusion-orc">ORC</a>. </li>
-</ol>
-<!--
-$ git log --pretty=oneline 34.0.0..40.0.0 . | wc -l
- 1453 (up from 1009)
-
-$ git shortlog -sn 34.0.0..40.0.0 . | wc -l
- 182 (up from 124)
-
-https://crates.io/crates/datafusion/34.0.0
-DataFusion 34 released Dec 17, 2023
-
-https://crates.io/crates/datafusion/40.0.0
-DataFusion 34 released July 12, 2024
-
-Issues created in this time: 321 open, 781 closed (up from 214 open, 437 closed)
-https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+created%3A2023-12-17..2024-07-12
-
-Issues closed: 911 (up from 517)
-https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2023-12-17..2024-07-12
-
-PRs merged in this time 1490 (up from 908)
-https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2023-12-17..2024-07-12
-
--->
-<p>In addition, DataFusion has been appearing publicly more and more, both online and offline. Here are some highlights:</p>
-<ol>
-<li><a href="https://dl.acm.org/doi/10.1145/3626246.3653368">Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine</a>, was presented in <a href="https://2024.sigmod.org/">SIGMOD '24</a>, one of the major database conferences</li>
-<li>As part of the trend to define "the POSIX of databases" in <a href="https://db.cs.cmu.edu/papers/2024/whatgoesaround-sigmodrec2024.pdf">"What Goes Around Comes Around... And Around..."</a> from Andy Pavlo and Mike Stonebraker</li>
-<li><a href="https://www.cpard.xyz/posts/datafusion/">"Why you should keep an eye on Apache DataFusion and its community"</a></li>
-<li><a href="https://www.tisonkun.org/2024/07/15/datafusion-meetup-san-francisco/">Apache DataFusion offline meetup in the Bay Area</a></li>
-</ol>
-<h2 id="improved-performance">Improved Performance 🚀<a class="headerlink" href="#improved-performance" title="Permanent link">¶</a></h2>
-<p>Performance is a key feature of DataFusion, and the community continues to work
-to keep DataFusion state of the art in this area. One major area DataFusion
-improved is the time it takes to convert a SQL query into a plan that can be
-executed. Planning is now almost 2x faster for TPC-DS and TPC-H queries, and
-over 10x faster for some queries with many columns.</p>
-<p>Here is a chart showing the improvement due to the concerted effort of many
-contributors including <a href="https://github.com/jackwener">@jackwener</a>, <a href="https://github.com/alamb">@alamb</a>, <a href="https://github.com/Lordworms">@Lordworms</a>, <a href="https://github.com/dmitrybugakov">@dmitrybugakov</a>,
-<a href="https://github.com/appletreeisyellow">@appletreeisyellow</a>, <a href="https://github.com/ClSlaid">@ClSlaid</a>, <a href="https://github.com/rohitrastogi">@rohitrastogi</a>, <a href="https://github.com/emgeee">@emgeee</a>, <a href="https://github.com/kevinmingtarja">@kevinmingtarja</a>,
-and <a href="https://github.com/peter-toth">@peter-toth</a> over several months (see <a href="https://github.com/apache/arrow-datafusion/issues/8045">ticket</a> for more details)</p>
-<p><img src="/blog/images/datafusion-40.0.0/improved-planning-time.png" width="700"/></p>
-<p>DataFusion is now up to 40% faster for queries that <code>GROUP BY</code> a single string
-or binary column due to a <a href="https://github.com/apache/datafusion/pull/8827">specialization for single
-Uft8/LargeUtf8/Binary/LargeBinary</a>. We are working on improving performance when
-there are [multiple variable length columns in the <code>GROUP BY</code> clause].</p>
-<p>We are also in the final phases of <a href="https://github.com/apache/datafusion/issues/10918">integrating</a> the new <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html">Arrow StringView</a>
-which significantly improves performance for workloads that scan, filter and
-group by variable length string and binary data. We expect the improvement to be
-especially pronounced for Parquet files due to <a href="https://github.com/apache/arrow-rs/issues/5530">upstream work in the parquet
-reader</a>. Kudos to <a href="https://github.com/XiangpengHong">@XiangpengHong</a>, <a href="https://github.com/AriesDevil">@AriesDevil</a>, <a href="https://github.com/PsiACE">@PsiACE</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>,
-<a href="https://github.com/a10y">@a10y</a>, and <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a> for driving this project.</p>
-<h2 id="improved-quality">Improved Quality 📋<a class="headerlink" href="#improved-quality" title="Permanent link">¶</a></h2>
-<p>DataFusion continues to improve overall in quality. In addition to ongoing bug
-fixes, one of the most exciting improvements is the addition of a new <a href="https://github.com/datafusion-contrib/datafusion-sqlancer">SQLancer</a>
-based <a href="https://github.com/apache/datafusion/issues/11030">DataFusion Fuzzing</a> suite thanks to <a href="https://github.com/2010YOUY01">@2010YOUY01</a> that has already found
-several bugs and thanks to <a href="https://github.com/jonahgao">@jonahgao</a>, <a href="https://github.com/tshauck">@tshauck</a>, <a href="https://github.com/xinlifoobar">@xinlifoobar</a>,
-<a href="https://github.com/LorrensP-2158466">@LorrensP-2158466</a> for fixing them so fast.</p>
-<h2 id="improved-documentation">Improved Documentation 📚<a class="headerlink" href="#improved-documentation" title="Permanent link">¶</a></h2>
-<p>We continue to improve the documentation to make it easier to get started using DataFusion with
-the <a href="https://datafusion.apache.org/library-user-guide/index.html">Library Users Guide</a>, <a href="https://docs.rs/datafusion/latest/datafusion/index.html">API documentation</a>, and <a href="https://github.com/apache/datafusion/tree/main/datafusion-examples">Examples</a>.</p>
-<p>Some notable new examples include:
-* <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_analysis.rs">sql_analysis.rs</a> to analyse SQL queries with DataFusion structures (thanks <a href="https://github.com/LorrensP-2158466">@LorrensP-2158466</a>)
-* <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/function_factory.rs">function_factory.rs</a> to create custom functions via SQL (thanks <a href="https://github.com/milenkovicm">@milenkovicm</a>)
-* <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/plan_to_sql.rs">plan_to_sql.rs</a> to generate SQL from DataFusion Expr and LogicalPlan (thanks <a href="https://github.com/edmondop">@edmondop</a>)
-* <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/parquet_index.rs">parquet_index.rs</a> and <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_parquet_index.rs">advanced_parquet_index.rs</a> for parquet indexing, described more below (thanks <a href="https://github.com/alamb">@alamb</a>)</p>
-<h2 id="new-features">New Features ✨<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h2>
-<p>There are too many new features in the last 6 months to list them all, but here
-are some highlights:</p>
-<h1 id="sql">SQL<a class="headerlink" href="#sql" title="Permanent link">¶</a></h1>
-<ul>
-<li>Support for <code>UNNEST</code> (thanks <a href="https://github.com/duongcongtoai">@duongcongtoai</a>, <a href="https://github.com/JasonLi-cn">@JasonLi-cn</a> and <a href="https://github.com/jayzhan211">@jayzhan211</a>) </li>
-<li>Support for <a href="https://github.com/apache/datafusion/issues/462">Recursive CTEs</a> (thanks <a href="https://github.com/jonahgao">@jonahgao</a> and <a href="https://github.com/matthewgapp">@matthewgapp</a>) </li>
-<li>Support for <code>CREATE FUNCTION</code> (see below) </li>
-<li>Many new SQL functions</li>
-</ul>
-<p>DataFusion now has much improved support for structured types such <code>STRUCT</code>,
-<code>LIST</code>/<code>ARRAY</code> and <code>MAP</code>. For example, you can now create <code>STRUCT</code> literals
-in SQL like this:</p>
-<pre><code class="language-rust">> select {'foo': {'bar': 2}};
-+--------------------------------------------------------------+
-| named_struct(Utf8("foo"),named_struct(Utf8("bar"),Int64(2))) |
-+--------------------------------------------------------------+
-| {foo: {bar: 2}} |
-+--------------------------------------------------------------+
-1 row(s) fetched.
-Elapsed 0.002 seconds.
-</code></pre>
-<h1 id="sql-unparser-sql-formatter">SQL Unparser (SQL Formatter)<a class="headerlink" href="#sql-unparser-sql-formatter" title="Permanent link">¶</a></h1>
-<p>DataFusion now supports converting <code>Expr</code>s and <code>LogicalPlan</code>s BACK to SQL text.
-This can be useful in query federation to push predicates down into other
-systems that only accept SQL, and for building systems that generate SQL.</p>
-<p>For example, you can now convert a logical expression back to SQL text:</p>
-<pre><code class="language-rust">// Form a logical expression that represents the SQL "a < 5 OR a = 8"
-let expr = col("a").lt(lit(5)).or(col("a").eq(lit(8)));
-// convert the expression back to SQL text
-let sql = expr_to_sql(&expr)?.to_string();
-assert_eq!(sql, "a < 5 OR a = 8");
-</code></pre>
-<p>You can also do complex things like parsing SQL, modifying the plan, and convert
-it back to SQL:</p>
-<pre><code class="language-rust">let df = ctx
- // Use SQL to read some data from the parquet file
- .sql("SELECT int_col, double_col, CAST(date_string_col as VARCHAR) FROM alltypes_plain")
- .await?;
-// Programmatically add new filters `id > 1 and tinyint_col < double_col`
-let df = df.filter(col("id").gt(lit(1)).and(col("tinyint_col").lt(col("double_col"))))?
-// Convert the new logical plan back to SQL
-let sql = plan_to_sql(df.logical_plan())?.to_string();
-assert_eq!(sql,
- "SELECT alltypes_plain.int_col, alltypes_plain.double_col, CAST(alltypes_plain.date_string_col AS VARCHAR) \
- FROM alltypes_plain WHERE ((alltypes_plain.id > 1) AND (alltypes_plain.tinyint_col < alltypes_plain.double_col))")
-);
-</code></pre>
-<p>See the <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/plan_to_sql.rs">Plan to SQL example</a> or the APIs <a href="https://docs.rs/datafusion/latest/datafusion/sql/unparser/fn.expr_to_sql.html">expr_to_sql</a> and <a href="https://docs.rs/datafusion/latest/datafusion/sql/unparser/fn.plan_to_sql.html">plan_to_sql</a> for more details.</p>
-<h1 id="low-level-apis-for-fast-parquet-access-indexing">Low Level APIs for Fast Parquet Access (indexing)<a class="headerlink" href="#low-level-apis-for-fast-parquet-access-indexing" title="Permanent link">¶</a></h1>
-<p>With their rising prevalence, supporting efficient access to Parquet files
-stored remotely on object storage is important. Part of doing this efficiently
-is minimizing the number of object store requests made by caching metadata and
-skipping over parts of the file that are not needed (e.g. via an index).</p>
-<p>DataFusion's Parquet reader has long internally supported <a href="https://arrow.apache.org/blog/2022/12/26/querying-parquet-with-millisecond-latency/">advanced predicate
-pushdown</a> by reading the parquet metadata from the file footer and pruning based
-on row group and data page statistics. DataFusion now also supports users
-supplying their own low level pruning information via the [<code>ParquetAccessPlan</code>]
-API.</p>
-<p>This API can be used along with index information to selectively skip decoding
-parts of the file. For example, Spice AI used this feature to add <a href="https://github.com/spiceai/spiceai/pull/1891">efficient
-support</a> for reading from DeltaLake tables and handling <a href="https://docs.delta.io/latest/delta-deletion-vectors.html">deletion vectors</a>.</p>
-<pre><code class="language-text"> ┌───────────────────────┐ If the RowSelection does not include any
- │ ... │ rows from a particular Data Page, that
- │ │ Data Page is not fetched or decoded.
- │ ┌───────────────────┐ │ Note this requires a PageIndex
- │ │ ┌──────────┐ │ │
-Row │ │ │DataPage 0│ │ │ ┌────────────────────┐
-Groups │ │ └──────────┘ │ │ │ │
- │ │ ┌──────────┐ │ │ │ ParquetExec │
- │ │ ... │DataPage 1│ ◀┼ ┼ ─ ─ ─ │ (Parquet Reader) │
- │ │ └──────────┘ │ │ └ ─ ─ ─ ─ ─│ │
- │ │ ┌──────────┐ │ │ │ ╔═══════════════╗ │
- │ │ │DataPage 2│ │ │ If only rows │ ║ParquetMetadata║ │
- │ │ └──────────┘ │ │ from DataPage 1 │ ╚═══════════════╝ │
- │ └───────────────────┘ │ are selected, └────────────────────┘
- │ │ only DataPage 1
- │ ... │ is fetched and
- │ │ decoded
- │ ╔═══════════════════╗ │
- │ ║ Thrift metadata ║ │
- │ ╚═══════════════════╝ │
- └───────────────────────┘
- Parquet File
-</code></pre>
-<p>See the <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/parquet_index.rs">parquet_index.rs</a> and <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_parquet_index.rs">advanced_parquet_index.rs</a> examples for more details. </p>
-<p>Thanks to <a href="https://github.com/alamb">@alamb</a> and <a href="https://github.com/Ted-Jiang">@Ted-Jiang</a> for this feature. </p>
-<h2 id="building-systems-is-easier-with-datafusion">Building Systems is Easier with DataFusion 🛠️<a class="headerlink" href="#building-systems-is-easier-with-datafusion" title="Permanent link">¶</a></h2>
-<p>In addition to many incremental API improvements, there are several new APIs that make
-it easier to build systems on top of DataFusion:</p>
-<ul>
-<li>Faster and easier to use <a href="https://docs.rs/datafusion/latest/datafusion/common/tree_node/trait.TreeNode.html#overview">TreeNode API</a> for traversing and manipulating plans and expressions.</li>
-<li>All functions now use the same <a href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html">Scalar User Defined Function API</a>, making it easier to customize
- DataFusion's behavior without sacrificing performance. See <a href="https://github.com/apache/arrow-datafusion/issues/8045">ticket</a> for more details.</li>
-<li>DataFusion can now be compiled to <a href="https://github.com/apache/datafusion/discussions/9834">WASM</a>. </li>
-</ul>
-<h1 id="user-defined-sql-parsing-extensions">User Defined SQL Parsing Extensions<a class="headerlink" href="#user-defined-sql-parsing-extensions" title="Permanent link">¶</a></h1>
-<p>As of DataFusion 40.0.0, you can use the [<code>ExprPlanner</code>] trait to extend
-DataFusion's SQL planner to support custom operators or syntax.</p>
-<p>For example the <a href="https://github.com/datafusion-contrib/datafusion-functions-json">datafusion-functions-json</a> project uses this API to support
-JSON operators in SQL queries. It provides a custom implementation for
-planning JSON operators such as <code>-></code> and <code>->></code> with code like:</p>
-<pre><code class="language-rust">struct MyCustomPlanner;
-
-impl ExprPlanner for MyCustomPlanner {
- // Provide custom implementation for planning a binary operators
- // such as `->` and `->>`
- fn plan_binary_op(
- &self,
- expr: RawBinaryExpr,
- _schema: &DFSchema,
- ) -> Result<PlannerResult<RawBinaryExpr>> {
- match &expr.op {
- BinaryOperator::Arrow => { /* plan -> operator */ }
- BinaryOperator::LongArrow => { /* plan ->> operator */ }
- ...
- }
- }
-}
-</code></pre>
-<p>Thanks to <a href="https://github.com/samuelcolvin">@samuelcolvin</a>, <a href="https://github.com/jayzhan211">@jayzhan211</a> and <a href="https://github.com/dharanad">@dharanad</a> for helping make this
-feature happen.</p>
-<h1 id="pluggable-support-for-create-function">Pluggable Support for <code>CREATE FUNCTION</code><a class="headerlink" href="#pluggable-support-for-create-function" title="Permanent link">¶</a></h1>
-<p>DataFusion's new [<code>FunctionFactory</code>] API let's users provide a handler for
-<code>CREATE FUNCTION</code> SQL statements. This feature lets you build systems that
-support defining functions in SQL such as</p>
-<pre><code class="language-sql">-- SQL based functions
-CREATE FUNCTION my_func(DOUBLE, DOUBLE) RETURNS DOUBLE
- RETURN $1 + $3
-;
-
--- ML Models
-CREATE FUNCTION iris(FLOAT[]) RETURNS FLOAT[]
-LANGUAGE TORCH AS 'models:/iris@champion';
-
--- WebAssembly
-CREATE FUNCTION func(FLOAT[]) RETURNS FLOAT[]
-LANGUAGE WASM AS 'func.wasm'
-</code></pre>
-<p>Huge thanks to <a href="https://github.com/milenkovicm">@milenkovicm</a> for this feature. There is an example of how to
-make macro like functions in <a href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/function_factory.rs">function_factory.rs</a>. It would be
-great if <a href="https://github.com/apache/datafusion/issues/9326">someone made a demo</a> showing how to create WASMs 🎣.</p>
-<h2 id="looking-ahead-the-next-six-months">Looking Ahead: The Next Six Months 🔭<a class="headerlink" href="#looking-ahead-the-next-six-months" title="Permanent link">¶</a></h2>
-<p>The community has been <a href="https://github.com/apache/datafusion/issues/11442">discussing what we will work on in the next six months</a>.
-Some major initiatives from that discussion are:</p>
-<ol>
-<li>
-<p><em>Performance</em>: Improve the speed of <a href="https://github.com/apache/arrow-datafusion/issues/7000">aggregating "high cardinality"</a>
- data when there are many (e.g. millions) of distinct groups as well as additional
- ideas to improve parquet performance. </p>
-</li>
-<li>
-<p><em>Modularity</em>: Make DataFusion even more modular, by completely unifying
- built in and user <a href="https://github.com/apache/datafusion/issues/8708">aggregate functions</a> and <a href="https://github.com/apache/datafusion/issues/8709">window functions</a>.</p>
-</li>
-<li>
-<p><em>LogicalTypes</em>: <a href="https://github.com/apache/datafusion/issues/11513">Introduce Logical Types</a> to make it easier to use
- different encodings like <code>StringView</code>, <code>RunEnd</code> and <code>Dictionary</code> arrays as well
- as user defined types. Thanks <a href="https://github.com/notfilippo">@notfilippo</a> for driving this. </p>
-</li>
-<li>
-<p><em>Improved Documentation</em>: Write blog posts and videos explaining
- how to use DataFusion for real-world use cases.</p>
-</li>
-<li>
-<p><em>Testing</em>: Improve CI infrastructure and test coverage, more fuzz
- testing, and better functional and performance regression testing.</p>
-</li>
-</ol>
-<h1 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h1>
-<p>DataFusion is not a project built or driven by a single person, company, or
-foundation. Rather, our community of users and contributors work together to
-build a shared technology that none of us could have built alone.</p>
-<p>If you are interested in joining us we would love to have you. You can try out
-DataFusion on some of your own data and projects and let us know how it goes,
-contribute suggestions, documentation, bug reports, or a PR with documentation,
-tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
-can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#community-growth">Community Growth 📈</a></li>
-<li><a href="#improved-performance">Improved Performance 🚀</a></li>
-<li><a href="#improved-quality">Improved Quality 📋</a></li>
-<li><a href="#improved-documentation">Improved Documentation 📚</a></li>
-<li><a href="#new-features">New Features ✨</a></li>
-<li><a href="#sql">SQL</a></li>
-<li><a href="#sql-unparser-sql-formatter">SQL Unparser (SQL Formatter)</a></li>
-<li><a href="#low-level-apis-for-fast-parquet-access-indexing">Low Level APIs for Fast Parquet Access (indexing)</a><ul>
-<li><a href="#building-systems-is-easier-with-datafusion">Building Systems is Easier with DataFusion 🛠️</a></li>
-</ul>
-</li>
-<li><a href="#user-defined-sql-parsing-extensions">User Defined SQL Parsing Extensions</a></li>
-<li><a href="#pluggable-support-for-create-function">Pluggable Support for CREATE FUNCTION</a><ul>
-<li><a href="#looking-ahead-the-next-six-months">Looking Ahead: The Next Six Months 🔭</a></li>
-</ul>
-</li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/08/20/python-datafusion-40.0.0/index.html b/blog/2024/08/20/python-datafusion-40.0.0/index.html
deleted file mode 100644
index 580213a..0000000
--- a/blog/2024/08/20/python-datafusion-40.0.0/index.html
+++ /dev/null
@@ -1,286 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Python 40.1.0 Released, Significant usability updates - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Python 40.1.0 Released, Significant usability updates
- </h1>
- <p>Posted on: Tue 20 August 2024 by timsaucer</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#background">Background</a></li>
-<li><a href="#whats-changed">What's Changed</a><ul>
-<li><a href="#improved-dataframe-filter-arguments">Improved DataFrame filter arguments</a></li>
-<li><a href="#comparison-against-literal-values">Comparison against literal values</a></li>
-<li><a href="#select-columns-by-name">Select columns by name</a></li>
-<li><a href="#creating-named-structs">Creating named structs</a></li>
-</ul>
-</li>
-<li><a href="#next-steps">Next Steps</a></li>
-<li><a href="#thank-you">Thank You</a></li>
-<li><a href="#get-involved">Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
-<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/40.1.0/">DataFusion in Python 40.1.0</a> has been released. In addition to
-bringing in all of the new features of the core <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> package, this release
-contains <em>significant</em> updates to the user interface and documentation. We listened to the python
-user community to create a more <em>pythonic</em> experience. If you have not used the python interface to
-DataFusion before, this is an excellent time to give it a try!</p>
-<h2 id="background">Background<a class="headerlink" href="#background" title="Permanent link">¶</a></h2>
-<p>Until now, the python bindings for DataFusion have primarily been a thin layer to expose the
-underlying Rust functionality. This has been worked well for early adopters to use DataFusion
-within their Python projects, but some users have found it difficult to work with. As compared to
-other DataFrame libraries, these issues were raised:</p>
-<ol>
-<li>Most of the functions had little or no documentation. Users often had to refer to the Rust
-documentation or code to learn how to use DataFusion. This alienated some python users.</li>
-<li>Users could not take advantage of modern IDE features such as type hinting. These are valuable
-tools for rapid testing and development.</li>
-<li>Some of the interfaces felt “clunky” to users since some Python concepts do not always map well
-to their Rust counterparts.</li>
-</ol>
-<p>This release aims to bring a better user experience to the DataFusion Python community.</p>
-<h2 id="whats-changed">What's Changed<a class="headerlink" href="#whats-changed" title="Permanent link">¶</a></h2>
-<p>The most significant difference is that we have added wrapper functions and classes for most of the
-user facing interface. These wrappers, written in Python, contain both documentation and type
-annotations.</p>
-<p>This documentation is now available on the <a href="https://datafusion.apache.org/python/autoapi/datafusion/index.html">DataFusion in Python API</a> website. There you can browse
-the available functions and classes to see the breadth of available functionality.</p>
-<p>Modern IDEs use language servers such as
-<a href="https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance">Pylance</a> or
-<a href="https://jedi.readthedocs.io/en/latest/">Jedi</a> to perform analysis of python code, provide useful
-hints, and identify usage errors. These are major tools in the python user community. With this
-release, users can fully use these tools in their workflow.</p>
-<figure class="text-center">
-<img alt="Fig 1: Enhanced tooltips in an IDE." class="img-fluid" src="/blog/images/python-datafusion-40.0.0/vscode_hover_tooltip.png"/>
-<figcaption>
-<b>Figure 1</b>: With the enhanced python wrappers, users can see helpful tool tips with
- type annotations directly in modern IDEs.
-</figcaption>
-</figure>
-<p>By having the type annotations, these IDEs can also identify quickly when a user has incorrectly
-used a function's arguments as shown in Figure 2.</p>
-<figure class="text-center">
-<img alt="Fig 2: Error checking in static analysis" class="img-fluid" src="/blog/images/python-datafusion-40.0.0/pylance_error_checking.png"/>
-<figcaption>
-<b>Figure 2</b>: Modern Python language servers can perform static analysis and quickly find
- errors in the arguments to functions.
-</figcaption>
-</figure>
-<p>In addition to these wrapper libraries, we have enhancements to some of the functions to feel more
-easy to use.</p>
-<h3 id="improved-dataframe-filter-arguments">Improved DataFrame filter arguments<a class="headerlink" href="#improved-dataframe-filter-arguments" title="Permanent link">¶</a></h3>
-<p>You can now apply multiple <code>filter</code> statements in a single step. When using <code>DataFrame.filter</code> you
-can pass in multiple arguments, separated by a comma. These will act as a logical <code>AND</code> of all of
-the filter arguments. The following two statements are equivalent:</p>
-<pre><code class="language-python">df.filter(col("size") < col("max_size")).filter(col("color") == lit("green"))
-df.filter(col("size") < col("max_size"), col("color") == lit("green"))
-</code></pre>
-<h3 id="comparison-against-literal-values">Comparison against literal values<a class="headerlink" href="#comparison-against-literal-values" title="Permanent link">¶</a></h3>
-<p>It is very common to write DataFrame operations that compare an expression to some fixed value.
-For example, filtering a DataFrame might have an operation such as <code>df.filter(col("size") < lit(16))</code>.
-To make these common operations more ergonomic, you can now simply use <code>df.filter(col("size") < 16)</code>.</p>
-<p>For the right hand side of the comparison operator, you can now use any Python value that can be
-coerced into a <code>Literal</code>. This gives an easy to ready expression. For example, consider these few
-lines from one of the
-<a href="https://github.com/apache/datafusion-python/tree/main/examples/tpch">TPC-H examples</a> provided in
-the DataFusion Python repository.</p>
-<pre><code class="language-python">df = (
- df_lineitem.filter(col("l_shipdate") >= lit(date))
- .filter(col("l_discount") >= lit(DISCOUNT) - lit(DELTA))
- .filter(col("l_discount") <= lit(DISCOUNT) + lit(DELTA))
- .filter(col("l_quantity") < lit(QUANTITY))
-)
-</code></pre>
-<p>The above code mirrors closely how these filters would need to be applied in rust. With this new
-release, the user can simplify these lines. Also shown in the example below is that <code>filter()</code>
-now accepts a variable number of arguments and filters on all such arguments (boolean AND).</p>
-<pre><code class="language-python">df = df_lineitem.filter(
- col("l_shipdate") >= date,
- col("l_discount") >= DISCOUNT - DELTA,
- col("l_discount") <= DISCOUNT + DELTA,
- col("l_quantity") < QUANTITY,
-)
-</code></pre>
-<h3 id="select-columns-by-name">Select columns by name<a class="headerlink" href="#select-columns-by-name" title="Permanent link">¶</a></h3>
-<p>It is very common for users to perform <code>DataFrame</code> selection where they simply want a column. For
-this we have had the function <code>select_columns("a", "b")</code> or the user could perform
-<code>select(col("a"), col("b"))</code>. In the new release, we accept either full expressions in <code>select()</code>
-or strings of the column names. You can mix these as well.</p>
-<p>Where before you may have to do an operation like</p>
-<pre><code class="language-python">df_subset = df.select(col("a"), col("b"), f.abs(col("c")))
-</code></pre>
-<p>You can now simplify this to</p>
-<pre><code class="language-python">df_subset = df.select("a", "b", f.abs(col("c")))
-</code></pre>
-<h3 id="creating-named-structs">Creating named structs<a class="headerlink" href="#creating-named-structs" title="Permanent link">¶</a></h3>
-<p>Creating a <code>struct</code> with named fields was previously difficult to use and allowed for potential
-user errors when specifying the name of each field. Now we have a cleaner interface where the
-user passes a list of tuples containing the name of the field and the expression to create.</p>
-<pre><code class="language-python">df.select(f.named_struct([
- ("a", col("a")),
- ("b", col("b"))
-]))
-</code></pre>
-<h2 id="next-steps">Next Steps<a class="headerlink" href="#next-steps" title="Permanent link">¶</a></h2>
-<p>While most of the user facing classes and functions have been exposed, there are a few that require
-exposure. Namely the classes in <code>datafusion.object_store</code> and the logical plans used by
-<code>datafusion.substrait</code>. The team is working on
-<a href="https://github.com/apache/datafusion-python/issues/767">these issues</a>.</p>
-<p>Additionally, in the next release of DataFusion there have been improvements made to the user-defined
-aggregate and window functions to make them easier to use. We plan on
-<a href="https://github.com/apache/datafusion-python/issues/780">bringing these enhancements</a> to this project.</p>
-<h2 id="thank-you">Thank You<a class="headerlink" href="#thank-you" title="Permanent link">¶</a></h2>
-<p>We would like to thank the following members for their very helpful discussions regarding these
-updates: <a href="https://github.com/andygrove">@andygrove</a>, <a href="https://github.com/max-muoto">@max-muoto</a>, <a href="https://github.com/slyons">@slyons</a>, <a href="https://github.com/Throne3d">@Throne3d</a>, <a href="https://github.com/Michael-J-Ward">@Michael-J-Ward</a>, <a href="https://github.com/datapythonista">@datapythonista</a>,
-<a href="https://github.com/austin362667">@austin362667</a>, <a href="https://github.com/kylebarron">@kylebarron</a>, <a href="https://github.com/simicd">@simicd</a>. The <a href="https://github.com/apache/datafusion-python/pull/750">primary PR (#750)</a> that includes these updates
-had an extensive conversation, leading to a significantly improved end product. Again, thank you
-to all who provided input!</p>
-<p>We would like to give an special thank you to <a href="https://github.com/3ok">@3ok</a> who created the initial version of the wrapper
-definitions. The work they did was time consuming and required exceptional attention to detail. It
-provided enormous value to starting this project. Thank you!</p>
-<h2 id="get-involved">Get Involved<a class="headerlink" href="#get-involved" title="Permanent link">¶</a></h2>
-<p>The DataFusion Python team is an active and engaging community and we would love
-to have you join us and help the project.</p>
-<p>Here are some ways to get involved:</p>
-<ul>
-<li>
-<p>Learn more by visiting the <a href="https://datafusion.apache.org/python/index.html">DataFusion Python project</a>
-page.</p>
-</li>
-<li>
-<p>Try out the project and provide feedback, file issues, and contribute code.</p>
-</li>
-</ul>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#background">Background</a></li>
-<li><a href="#whats-changed">What's Changed</a><ul>
-<li><a href="#improved-dataframe-filter-arguments">Improved DataFrame filter arguments</a></li>
-<li><a href="#comparison-against-literal-values">Comparison against literal values</a></li>
-<li><a href="#select-columns-by-name">Select columns by name</a></li>
-<li><a href="#creating-named-structs">Creating named structs</a></li>
-</ul>
-</li>
-<li><a href="#next-steps">Next Steps</a></li>
-<li><a href="#thank-you">Thank You</a></li>
-<li><a href="#get-involved">Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/08/28/datafusion-comet-0.2.0/index.html b/blog/2024/08/28/datafusion-comet-0.2.0/index.html
deleted file mode 100644
index 48c992f..0000000
--- a/blog/2024/08/28/datafusion-comet-0.2.0/index.html
+++ /dev/null
@@ -1,203 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.2.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Comet 0.2.0 Release
- </h1>
- <p>Posted on: Wed 28 August 2024 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#docker-images">Docker Images</a></li>
-<li><a href="#performance-improvements">Performance improvements</a></li>
-<li><a href="#new-features">New Features</a></li>
-</ul>
-</li>
-<li><a href="#current-performance">Current Performance</a><ul>
-<li><a href="#benchmark-derived-from-tpc-h">Benchmark derived from TPC-H</a></li>
-<li><a href="#benchmark-derived-from-tpc-ds">Benchmark derived from TPC-DS</a></li>
-</ul>
-</li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>The Apache DataFusion PMC is pleased to announce version 0.2.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately four weeks of development work and is the result of merging 87 PRs from 14
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.2.0.md">change log</a> for more information.</p>
-<h2 id="release-highlights">Release Highlights<a class="headerlink" href="#release-highlights" title="Permanent link">¶</a></h2>
-<h3 id="docker-images">Docker Images<a class="headerlink" href="#docker-images" title="Permanent link">¶</a></h3>
-<p>Docker images are now available from the <a href="https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet/265110454?tag=spark-3.4-scala-2.12-0.2.0">GitHub Container Registry</a>.</p>
-<h3 id="performance-improvements">Performance improvements<a class="headerlink" href="#performance-improvements" title="Permanent link">¶</a></h3>
-<ul>
-<li>Native shuffle is now enabled by default</li>
-<li>Improved handling of decimal types</li>
-<li>Reduced some redundant copying of batches in Filter/Scan operations</li>
-<li>Optimized performance of count aggregates</li>
-<li>Optimized performance of CASE expressions for specific uses:</li>
-<li>CASE WHEN expr THEN column ELSE null END</li>
-<li>CASE WHEN expr THEN literal ELSE literal END</li>
-<li>Optimized performance of IS NOT NULL</li>
-</ul>
-<h3 id="new-features">New Features<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h3>
-<ul>
-<li>Window operations now support count and sum aggregates</li>
-<li>CreateArray</li>
-<li>GetStructField</li>
-<li>Support nested types in hash join</li>
-<li>Basic implementation of RLIKE expression</li>
-</ul>
-<h2 id="current-performance">Current Performance<a class="headerlink" href="#current-performance" title="Permanent link">¶</a></h2>
-<p>We use benchmarks derived from the industry standard TPC-H and TPC-DS benchmarks for tracking progress with
-performance. The following charts shows the time it takes to run the queries against 100 GB of data in
-Parquet format using a single executor with eight cores. See the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>
-for details of the environment used for these benchmarks.</p>
-<h3 id="benchmark-derived-from-tpc-h">Benchmark derived from TPC-H<a class="headerlink" href="#benchmark-derived-from-tpc-h" title="Permanent link">¶</a></h3>
-<p>Comet 0.2.0 provides a 62% speedup compared to Spark. This is slightly better than the Comet 0.1.0 release.</p>
-<p><img alt="Chart showing TPC-H benchmark results for Comet 0.2.0" class="img-fluid" src="/blog/images/comet-0.2.0/tpch_allqueries.png" width="100%"/></p>
-<h3 id="benchmark-derived-from-tpc-ds">Benchmark derived from TPC-DS<a class="headerlink" href="#benchmark-derived-from-tpc-ds" title="Permanent link">¶</a></h3>
-<p>Comet 0.2.0 provides a 21% speedup compared to Spark, which is a significant improvement compared to
-Comet 0.1.0, which did not provide any speedup for this benchmark.</p>
-<p><img alt="Chart showing TPC-DS benchmark results for Comet 0.2.0" class="img-fluid" src="/blog/images/comet-0.2.0/tpcds_allqueries.png" width="100%"/></p>
-<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#docker-images">Docker Images</a></li>
-<li><a href="#performance-improvements">Performance improvements</a></li>
-<li><a href="#new-features">New Features</a></li>
-</ul>
-</li>
-<li><a href="#current-performance">Current Performance</a><ul>
-<li><a href="#benchmark-derived-from-tpc-h">Benchmark derived from TPC-H</a></li>
-<li><a href="#benchmark-derived-from-tpc-ds">Benchmark derived from TPC-DS</a></li>
-</ul>
-</li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/09/13/string-view-german-style-strings-part-1/index.html b/blog/2024/09/13/string-view-german-style-strings-part-1/index.html
deleted file mode 100644
index 57e4f52..0000000
--- a/blog/2024/09/13/string-view-german-style-strings-part-1/index.html
+++ /dev/null
@@ -1,243 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet
- </h1>
- <p>Posted on: Fri 13 September 2024 by Xiangpeng Hao, Andrew Lamb</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#what-is-stringview">What is StringView?</a></li>
-<li><a href="#faster-parquet-loading">Faster Parquet Loading</a></li>
-<li><a href="#from-binary-to-strings">From Binary to Strings</a></li>
-<li><a href="#be-careful-about-implicit-copies">Be Careful About Implicit Copies</a></li>
-<li><a href="#help-the-compiler-by-giving-it-more-information">Help the Compiler by Giving it More Information</a></li>
-<li><a href="#end-to-end-query-performance">End-to-End Query Performance</a><ul>
-<li><a href="#conclusion">Conclusion</a></li>
-</ul>
-</li>
-<li><a href="#footnotes">Footnotes</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p><em>Editor's Note: This is the first of a <a href="../string-view-german-style-strings-part-2/">two part</a> blog series that was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>This blog describes our experience implementing <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringView</a> in the <a href="https://github.com/apache/arrow-rs">Rust implementation</a> of <a href="https://arrow.apache.org/">Apache Arrow</a>, and integrating it into <a href="https://datafusion.apache.org/">Apache DataFusion</a>, significantly accelerating string-intensive queries in the <a href="https://benchmark.clickhouse.com/">ClickBen [...]
-<p>Getting significant end-to-end performance improvements was non-trivial. Implementing StringView itself was only a fraction of the effort required. Among other things, we had to optimize UTF-8 validation, implement unintuitive compiler optimizations, tune block sizes, and time GC to realize the <a href="https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/">FDAP ecosystem</a>’s benefit. With other members of the open source community, we were able [...]
-<p>StringView is based on a simple idea: avoid some string copies and accelerate comparisons with inlined prefixes. Like most great ideas, it is “obvious” only after <a href="https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf">someone describes it clearly</a>. Although simple, straightforward implementation actually <em>slows down performance for almost every query</em>. We must, therefore, apply astute observations and diligent engineering to realize the actual benefits from St [...]
-<p>Although this journey was successful, not all research ideas are as lucky. To accelerate the adoption of research into industry, it is valuable to integrate research prototypes with practical systems. Understanding the nuances of real-world systems makes it more likely that research designs[^2] will lead to practical system improvements.</p>
-<p>StringView support was released as part of <a href="https://crates.io/crates/arrow/52.2.0">arrow-rs v52.2.0</a> and <a href="https://crates.io/crates/datafusion/41.0.0">DataFusion v41.0.0</a>. You can try it by setting the <code>schema_force_view_types</code> <a href="https://datafusion.apache.org/user-guide/configs.html">DataFusion configuration option</a>, and we are<a href="https://github.com/apache/datafusion/issues/11682"> hard at work with the community to </a>make it the defaul [...]
-<p><img alt="End to end performance improvements for ClickBench queries" class="img-fluid" src="/blog/images/string-view-1/figure1-performance.png" width="100%"/></p>
-<p>Figure 1: StringView improves string-intensive ClickBench query performance by 20% - 200%</p>
-<h2 id="what-is-stringview">What is StringView?<a class="headerlink" href="#what-is-stringview" title="Permanent link">¶</a></h2>
-<p><img alt="Diagram of using StringArray and StringViewArray to represent the same string content" class="img-fluid" src="/blog/images/string-view-1/figure2-string-view.png" width="100%"/></p>
-<p>Figure 2: Use StringArray and StringViewArray to represent the same string content.</p>
-<p>The concept of inlined strings with prefixes (called “German Strings” <a href="https://x.com/andy_pavlo/status/1813258735965643203">by Andy Pavlo</a>, in homage to <a href="https://www.tum.de/">TUM</a>, where the <a href="https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf">Umbra paper that describes</a> them originated)
-has been used in many recent database systems (<a href="https://engineering.fb.com/2024/02/20/developer-tools/velox-apache-arrow-15-composable-data-management/">Velox</a>, <a href="https://pola.rs/posts/polars-string-type/">Polars</a>, <a href="https://duckdb.org/2021/12/03/duck-arrow.html">DuckDB</a>, <a href="https://cedardb.com/blog/german_strings/">CedarDB</a>, etc.)
-and was introduced to Arrow as a new <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout">StringViewArray</a>[^3] type. Arrow’s original <a href="https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout">StringArray</a> is very memory efficient but less effective for certain operations.
-StringViewArray accelerates string-intensive operations via prefix inlining and a more flexible and compact string representation.</p>
-<p>A StringViewArray consists of three components:</p>
-<ol>
-<li>The <code><em>view</em></code> array</li>
-<li>The buffers</li>
-<li>The buffer pointers (IDs) that map buffer offsets to their physical locations</li>
-</ol>
-<p>Each <code>view</code> is 16 bytes long, and its contents differ based on the string’s length:</p>
-<ul>
-<li>string length < 12 bytes: the first four bytes store the string length, and the remaining 12 bytes store the inlined string.</li>
-<li>string length > 12 bytes: the string is stored in a separate buffer. The length is again stored in the first 4 bytes, followed by the buffer id (4 bytes), the buffer offset (4 bytes), and the prefix (first 4 bytes) of the string.</li>
-</ul>
-<p>Figure 2 shows an example of the same logical content (left) using StringArray (middle) and StringViewArray (right):</p>
-<ul>
-<li>The first string – <code>"Apache DataFusion"</code> – is 17 bytes long, and both StringArray and StringViewArray store the string’s bytes at the beginning of the buffer. The StringViewArray also inlines the first 4 bytes – <code>"Apac"</code> – in the view.</li>
-<li>The second string, <code>"InfluxDB"</code> is only 8 bytes long, so StringViewArray completely inlines the string content in the <code>view</code> struct while StringArray stores the string in the buffer as well.</li>
-<li>The third string <code>"Arrow Rust Impl"</code> is 15 bytes long and cannot be fully inlined. StringViewArray stores this in the same form as the first string.</li>
-<li>The last string <code>"Apache DataFusion"</code> has the same content as the first string. It’s possible to use StringViewArray to avoid this duplication and reuse the bytes by pointing the view to the previous location.</li>
-</ul>
-<p>StringViewArray provides three opportunities for outperforming StringArray:</p>
-<ol>
-<li>Less copying via the offset + buffer format</li>
-<li>Faster comparisons using the inlined string prefix</li>
-<li>Reusing repeated string values with the flexible <code>view</code> layout</li>
-</ol>
-<p>The rest of this blog post discusses how to apply these opportunities in real query scenarios to improve performance, what challenges we encountered along the way, and how we solved them.</p>
-<h2 id="faster-parquet-loading">Faster Parquet Loading<a class="headerlink" href="#faster-parquet-loading" title="Permanent link">¶</a></h2>
-<p><a href="https://parquet.apache.org/">Apache Parquet</a> is the de facto format for storing large-scale analytical data commonly stored LakeHouse-style, such as <a href="https://iceberg.apache.org">Apache Iceberg</a> and <a href="https://delta.io">Delta Lake</a>. Efficiently loading data from Parquet is thus critical to query performance in many important real-world workloads.</p>
-<p>Parquet encodes strings (i.e., <a href="https://docs.rs/parquet/latest/parquet/data_type/struct.ByteArray.html">byte array</a>) in a slightly different format than required for the original Arrow StringArray. The string length is encoded inline with the actual string data (as shown in Figure 4 left). As mentioned previously, StringArray requires the data buffer to be continuous and compact—the strings have to follow one after another. This requirement means that reading Parquet string [...]
-<p>On the other hand, reading Parquet data as a StringViewArray can re-use the same data buffer as storing the Parquet pages because StringViewArray does not require strings to be contiguous. For example, in Figure 4, the StringViewArray directly references the buffer with the decoded Parquet page. The string <code>"Arrow Rust Impl"</code> is represented by a <code>view</code> with offset 37 and length 15 into that buffer.</p>
-<p><img alt="Diagram showing how StringViewArray can avoid copying by reusing decoded Parquet pages." class="img-fluid" src="/blog/images/string-view-1/figure4-copying.png" width="100%"/></p>
-<p>Figure 4: StringViewArray avoids copying by reusing decoded Parquet pages.</p>
-<p><strong>Mini benchmark</strong></p>
-<p>Reusing Parquet buffers is great in theory, but how much does saving a copy actually matter? We can run the following benchmark in arrow-rs to find out:</p>
-<p>Our benchmarking machine shows that loading <em>BinaryViewArray</em> is almost 2x faster than loading BinaryArray (see next section about why this isn’t <em>String</em> ViewArray).</p>
-<p>You can read more on this arrow-rs issue: <a href="https://github.com/apache/arrow-rs/issues/5904">https://github.com/apache/arrow-rs/issues/5904</a></p>
-<h1 id="from-binary-to-strings">From Binary to Strings<a class="headerlink" href="#from-binary-to-strings" title="Permanent link">¶</a></h1>
-<p>You may wonder why we reported performance for BinaryViewArray when this post is about StringViewArray. Surprisingly, initially, our implementation to read StringViewArray from Parquet was much <em>slower</em> than StringArray. Why? TLDR: Although reading StringViewArray copied less data, the initial implementation also spent much more time validating <a href="https://en.wikipedia.org/wiki/UTF-8#:~:text=UTF%2D8%20is%20a%20variable,Unicode%20Standard">UTF-8</a> (as shown in Figure 5).</p>
-<p>Strings are stored as byte sequences. When reading data from (potentially untrusted) Parquet files, a Parquet decoder must ensure those byte sequences are valid UTF-8 strings, and most programming languages, including Rust, include highly<a href="https://doc.rust-lang.org/std/str/fn.from_utf8.html"> optimized routines</a> for doing so.</p>
-<p><img alt="Figure showing time to load strings from Parquet and the effect of optimized UTF-8 validation." class="img-fluid" src="/blog/images/string-view-1/figure5-loading-strings.png" width="100%"/></p>
-<p>Figure 5: Time to load strings from Parquet. The UTF-8 validation advantage initially eliminates the advantage of reduced copying for StringViewArray.</p>
-<p>A StringArray can be validated in a single call to the UTF-8 validation function as it has a continuous string buffer. As long as the underlying buffer is UTF-8[^4], all strings in the array must be UTF-8. The Rust parquet reader makes a single function call to validate the entire buffer.</p>
-<p>However, validating an arbitrary StringViewArray requires validating each string with a separate call to the validation function, as the underlying buffer may also contain non-string data (for example, the lengths in Parquet pages).</p>
-<p>UTF-8 validation in Rust is highly optimized and favors longer strings (as shown in Figure 6), likely because it leverages SIMD instructions to perform parallel validation. The benefit of a single function call to validate UTF-8 over a function call for each string more than eliminates the advantage of avoiding the copy for StringViewArray.</p>
-<p><img alt="Figure showing UTF-8 validation throughput vs string length." class="img-fluid" src="/blog/images/string-view-1/figure6-utf8-validation.png" width="100%"/></p>
-<p>Figure 6: UTF-8 validation throughput vs string length—StringArray’s contiguous buffer can be validated much faster than StringViewArray’s buffer.</p>
-<p>Does this mean we should only use StringArray? No! Thankfully, there’s a clever way out. The key observation is that in many real-world datasets,<a href="https://www.vldb.org/pvldb/vol17/p148-zeng.pdf"> 99% of strings are shorter than 128 bytes</a>, meaning the encoded length values are smaller than 128, <strong>in which case the length itself is also valid UTF-8</strong> (in fact, it is <a href="https://en.wikipedia.org/wiki/ASCII">ASCII</a>).</p>
-<p>This observation means we can optimize validating UTF-8 strings in Parquet pages by treating the length bytes as part of a single large string as long as the length <em>value</em> is less than 128. Put another way, prior to this optimization, the length bytes act as string boundaries, which require a UTF-8 validation on each string. After this optimization, only those strings with lengths larger than 128 bytes (less than 1% of the strings in the ClickBench dataset) are string boundari [...]
-<p>The <a href="https://github.com/apache/arrow-rs/pull/6009/files">actual implementation</a> is only nine lines of Rust (with 30 lines of comments). You can find more details in the related arrow-rs issue:<a href="https://github.com/apache/arrow-rs/issues/5995"> https://github.com/apache/arrow-rs/issues/5995</a>. As expected, with this optimization, loading StringViewArray is almost 2x faster than loading StringArray.</p>
-<h1 id="be-careful-about-implicit-copies">Be Careful About Implicit Copies<a class="headerlink" href="#be-careful-about-implicit-copies" title="Permanent link">¶</a></h1>
-<p>After all the work to avoid copying strings when loading from Parquet, performance was still not as good as expected. We tracked the problem to a few implicit data copies that we weren't aware of, as described in<a href="https://github.com/apache/arrow-rs/issues/6033"> this issue</a>.</p>
-<p>The copies we eventually identified come from the following innocent-looking line of Rust code, where <code>self.buf</code> is a <a href="https://en.wikipedia.org/wiki/Reference_counting">reference counted</a> pointer that should transform without copying into a buffer for use in StringViewArray.</p>
-<p>However, Rust-type coercion rules favored a blanket implementation that <em>did</em> copy data. This implementation is shown in the following code block where the <code>impl<T: AsRef<[u8]>></code> will accept any type that implements <code>AsRef<[u8]></code> and copies the data to create a new buffer. To avoid copying, users need to explicitly call <code>from_vec</code>, which consumes the <code>Vec</code> and transforms it into a buffer.</p>
-<p>Diagnosing this implicit copy was time-consuming as it relied on subtle Rust language semantics. We needed to track every step of the data flow to ensure every copy was necessary. To help other users and prevent future mistakes, we also <a href="https://github.com/apache/arrow-rs/pull/6043">removed</a> the implicit API from arrow-rs in favor of an explicit API. Using this approach, we found and fixed several <a href="https://github.com/apache/arrow-rs/pull/6039">other unintentional co [...]
-<h1 id="help-the-compiler-by-giving-it-more-information">Help the Compiler by Giving it More Information<a class="headerlink" href="#help-the-compiler-by-giving-it-more-information" title="Permanent link">¶</a></h1>
-<p>The Rust compiler’s automatic optimizations mostly work very well for a wide variety of use cases, but sometimes, it needs additional hints to generate the most efficient code. When profiling the performance of <code>view</code> construction, we found, counterintuitively, that constructing <strong>long</strong> strings was 10x faster than constructing <strong>short</strong> strings, which made short strings slower on StringViewArray than on StringArray!</p>
-<p>As described in the first section, StringViewArray treats long and short strings differently. Short strings (<12 bytes) directly inline to the <code>view</code> struct, while long strings only inline the first 4 bytes. The code to construct a <code>view</code> looks something like this:</p>
-<p>It appears that both branches of the code should be fast: they both involve copying at most 16 bytes of data and some memory shift/store operations. How could the branch for short strings be 10x slower?</p>
-<p>Looking at the assembly code using <a href="https://godbolt.org/">Compiler Explorer</a>, we (with help from <a href="https://github.com/aoli-al">Ao Li</a>) found the compiler used CPU <strong>load instructions</strong> to copy the fixed-sized 4 bytes to the <code>view</code> for long strings, but it calls a function, <a href="https://doc.rust-lang.org/std/ptr/fn.copy_nonoverlapping.html"><code>ptr::copy_non_overlapping</code></a>, to copy the inlined bytes to the <code>view</code> for [...]
-<p>However, we know something the compiler doesn’t know: the short string size is not arbitrary—it must be between 0 and 12 bytes, and we can leverage this information to avoid the function call. Our solution generates 13 copies of the function using generics, one for each of the possible prefix lengths. The code looks as follows, and <a href="https://godbolt.org/z/685YPsd5G">checking the assembly code</a>, we confirmed there are no calls to <code>ptr::copy_non_overlapping</code>, and on [...]
-<h1 id="end-to-end-query-performance">End-to-End Query Performance<a class="headerlink" href="#end-to-end-query-performance" title="Permanent link">¶</a></h1>
-<p>In the previous sections, we went out of our way to make sure loading StringViewArray is faster than StringArray. Before going further, we wanted to verify if obsessing about reducing copies and function calls has actually improved end-to-end performance in real-life queries. To do this, we evaluated a ClickBench query (Q20) in DataFusion that counts how many URLs contain the word <code>"google"</code>:</p>
-<p>This is a relatively simple query; most of the time is spent on loading the “URL” column to find matching rows. The query plan looks like this:</p>
-<p>We ran the benchmark in the DataFusion repo like this:</p>
-<p>With StringViewArray we saw a 24% end-to-end performance improvement, as shown in Figure 7. With the <code>--string-view</code> argument, the end-to-end query time is <code>944.3 ms, 869.6 ms, 861.9 ms</code> (three iterations). Without <code>--string-view</code>, the end-to-end query time is <code>1186.1 ms, 1126.1 ms, 1138.3 ms</code>.</p>
-<p><img alt="Figure showing StringView improves end to end performance by 24 percent." class="img-fluid" src="/blog/images/string-view-1/figure7-end-to-end.png" width="100%"/></p>
-<p>Figure 7: StringView reduces end-to-end query time by 24% on ClickBench Q20.</p>
-<p>We also double-checked with detailed profiling and verified that the time reduction is indeed due to faster Parquet loading.</p>
-<h2 id="conclusion">Conclusion<a class="headerlink" href="#conclusion" title="Permanent link">¶</a></h2>
-<p>In this first blog post, we have described what it took to improve the
-performance of simply reading strings from Parquet files using StringView. While
-this resulted in real end-to-end query performance improvements, in our <a href="https://datafusion.apache.org/blog/2024/09/13//string-view-german-style-strings-part-2">next
-post</a>, we explore additional optimizations enabled by StringView in DataFusion,
-along with some of the pitfalls we encountered while implementing them.</p>
-<h1 id="footnotes">Footnotes<a class="headerlink" href="#footnotes" title="Permanent link">¶</a></h1>
-<p>[^1]: Benchmarked with AMD Ryzen 7600x (12 core, 24 threads, 32 MiB L3), WD Black SN770 NVMe SSD (5150MB/4950MB seq RW bandwidth)</p>
-<p>[^2]: Xiangpeng is a PhD student at the University of Wisconsin-Madison</p>
-<p>[^3]: There is also a corresponding <em>BinaryViewArray</em> which is similar except that the data is not constrained to be UTF-8 encoded strings.</p>
-<p>[^4]: We also make sure that offsets do not break a UTF-8 code point, which is <a href="https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/buffer/offset_buffer.rs#L62-L71">cheaply validated</a>.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#what-is-stringview">What is StringView?</a></li>
-<li><a href="#faster-parquet-loading">Faster Parquet Loading</a></li>
-<li><a href="#from-binary-to-strings">From Binary to Strings</a></li>
-<li><a href="#be-careful-about-implicit-copies">Be Careful About Implicit Copies</a></li>
-<li><a href="#help-the-compiler-by-giving-it-more-information">Help the Compiler by Giving it More Information</a></li>
-<li><a href="#end-to-end-query-performance">End-to-End Query Performance</a><ul>
-<li><a href="#conclusion">Conclusion</a></li>
-</ul>
-</li>
-<li><a href="#footnotes">Footnotes</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/09/13/string-view-german-style-strings-part-2/index.html b/blog/2024/09/13/string-view-german-style-strings-part-2/index.html
deleted file mode 100644
index e366917..0000000
--- a/blog/2024/09/13/string-view-german-style-strings-part-2/index.html
+++ /dev/null
@@ -1,230 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations
- </h1>
- <p>Posted on: Fri 13 September 2024 by Xiangpeng Hao, Andrew Lamb</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#faster-string-operations">Faster String Operations</a></li>
-<li><a href="#faster-comparison">Faster comparison</a></li>
-<li><a href="#faster-takeandfilter">Faster takeandfilter</a></li>
-<li><a href="#when-to-gc">When to GC?</a><ul>
-<li><a href="#the-art-of-function-inlining-not-too-much-not-too-little">The art of function inlining: not too much, not too little</a></li>
-<li><a href="#buffer-size-tuning">Buffer size tuning</a></li>
-<li><a href="#end-to-end-query-performance">End-to-end query performance</a></li>
-</ul>
-</li>
-<li><a href="#faster-string-aggregation">Faster String Aggregation</a></li>
-<li><a href="#stringview-pitfalls">StringView Pitfalls</a></li>
-<li><a href="#conclusion-and-takeaways">Conclusion and Takeaways</a><ul>
-<li><a href="#footnotes">Footnotes</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p><em>Editor's Note: This blog series was first published on the <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">InfluxData blog</a>. Thanks to InfluxData for sponsoring this work as <a href="https://haoxp.xyz/">Xiangpeng Hao</a>'s summer intern project</em></p>
-<p>In the <a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1">first post</a>, we discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies.
-In this second part of the post, we describe the rest of the journey: implementing additional efficient operations for real query processing.</p>
-<h2 id="faster-string-operations">Faster String Operations<a class="headerlink" href="#faster-string-operations" title="Permanent link">¶</a></h2>
-<h1 id="faster-comparison">Faster comparison<a class="headerlink" href="#faster-comparison" title="Permanent link">¶</a></h1>
-<p>String comparison is ubiquitous; it is the core of
-<a href="https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/index.html"><code>cmp</code></a>,
-<a href="https://docs.rs/arrow/latest/arrow/compute/fn.min.html"><code>min</code></a>/<a href="https://docs.rs/arrow/latest/arrow/compute/fn.max.html"><code>max</code></a>,
-and <a href="https://docs.rs/arrow/latest/arrow/compute/kernels/comparison/fn.like.html"><code>like</code></a>/<a href="https://docs.rs/arrow/latest/arrow/compute/kernels/comparison/fn.ilike.html"><code>ilike</code></a> kernels. StringViewArray is designed to accelerate such comparisons using the inlined prefix—the key observation is that, in many cases, only the first few bytes of the string determine the string comparison results.</p>
-<p>For example, to compare the strings <code>InfluxDB</code> with <code>Apache DataFusion</code>, we only need to look at the first byte to determine the string ordering or equality. In this case, since <code>A</code> is earlier in the alphabet than <code>I,</code> <code>Apache DataFusion</code> sorts first, and we know the strings are not equal. Despite only needing the first byte, comparing these strings when stored as a StringArray requires two memory accesses: 1) load the string offs [...]
-<p>For StringViewArray, typically, only one memory access is needed to load the view struct. Only if the result can not be determined from the prefix is the second memory access required. For the example above, there is no need for the second access. This technique is very effective in practice: the second access is never necessary for the more than <a href="https://www.vldb.org/pvldb/vol17/p148-zeng.pdf">60% of real-world strings which are shorter than 12 bytes</a>, as they are stored c [...]
-<p>However, functions that operate on strings must be specialized to take advantage of the inlined prefix. In addition to low-level comparison kernels, we implemented <a href="https://github.com/apache/arrow-rs/issues/5374">a wide range</a> of other StringViewArray operations that cover the functions and operations seen in ClickBench queries. Supporting StringViewArray in all string operations takes quite a bit of effort, and thankfully the Arrow and DataFusion communities are already ha [...]
-<h1 id="faster-takeandfilter">Faster <code>take</code>and<code>filter</code><a class="headerlink" href="#faster-takeandfilter" title="Permanent link">¶</a></h1>
-<p>After a filter operation such as <code>WHERE url <> ''</code> to avoid processing empty urls, DataFusion will often <em>coalesce</em> results to form a new array with only the passing elements.
-This coalescing ensures the batches are sufficiently sized to benefit from <a href="https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf">vectorized processing</a> in subsequent steps.</p>
-<p>The coalescing operation is implemented using the <a href="https://docs.rs/arrow/latest/arrow/compute/fn.take.html">take</a> and <a href="https://arrow.apache.org/rust/arrow/compute/kernels/filter/fn.filter.html">filter</a> kernels in arrow-rs. For StringArray, these kernels require copying the string contents to a new buffer without “holes” in between. This copy can be expensive especially when the new array is large.</p>
-<p>However, <code>take</code> and <code>filter</code> for StringViewArray can avoid the copy by reusing buffers from the old array. The kernels only need to create a new list of <code>view</code>s that point at the same strings within the old buffers.
-Figure 1 illustrates the difference between the output of both string representations. StringArray creates two new strings at offsets 0-17 and 17-32, while StringViewArray simply points to the original buffer at offsets 0 and 25.</p>
-<p><img alt="Diagram showing Zero-copy <code>take</code>/<code>filter</code> for StringViewArray" class="img-fluid" src="/blog/images/string-view-2/figure1-zero-copy-take.png" width="100%"/></p>
-<p>Figure 1: Zero-copy <code>take</code>/<code>filter</code> for StringViewArray</p>
-<h1 id="when-to-gc">When to GC?<a class="headerlink" href="#when-to-gc" title="Permanent link">¶</a></h1>
-<p>Zero-copy <code>take/filter</code> is great for generating large arrays quickly, but it is suboptimal for highly selective filters, where most of the strings are filtered out. When the cardinality drops, StringViewArray buffers become sparse—only a small subset of the bytes in the buffer’s memory are referred to by any <code>view</code>. This leads to excessive memory usage, especially in a <a href="https://github.com/apache/datafusion/issues/11628">filter-then-coalesce scenario</a>. [...]
-<p>To release unused memory, we implemented a <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html#method.gc">garbage collection (GC)</a> routine to consolidate the data into a new buffer to release the old sparse buffer(s). As the GC operation copies strings, similarly to StringArray, we must be careful about when to call it. If we call GC too early, we cause unnecessary copying, losing much of the benefit of StringViewArray. If we call GC too late, we hold [...]
-<p><code>arrow-rs</code> implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/coalesce_batches/struct.CoalesceBatchesExec.html"><code>CoalesceBatchesExec</code></a> operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, which aligns perfectly with the scenario of G [...]
-We, therefore,<a href="https://github.com/apache/datafusion/pull/11587"> implemented the GC procedure</a> inside <code>CoalesceBatchesExec</code>[^5] with a heuristic that estimates when the buffers are too sparse.</p>
-<h2 id="the-art-of-function-inlining-not-too-much-not-too-little">The art of function inlining: not too much, not too little<a class="headerlink" href="#the-art-of-function-inlining-not-too-much-not-too-little" title="Permanent link">¶</a></h2>
-<p>Like string inlining, <em>function</em> inlining is the process of embedding a short function into the caller to avoid the overhead of function calls (caller/callee save).
-Usually, the Rust compiler does a good job of deciding when to inline. However, it is possible to override its default using the <a href="https://doc.rust-lang.org/reference/attributes/codegen.html#the-inline-attribute"><code>#[inline(always)]</code> directive</a>.
-In performance-critical code, inlined code allows us to organize large functions into smaller ones without paying the runtime cost of function invocation.</p>
-<p>However, function inlining is <strong><em>not</em></strong> always better, as it leads to larger function bodies that are harder for LLVM to optimize (for example, suboptimal <a href="https://en.wikipedia.org/wiki/Register_allocation">register spilling</a>) and risk overflowing the CPU’s instruction cache. We observed several performance regressions where function inlining caused <em>slower</em> performance when implementing the StringViewArray comparison kernels. Careful inspection a [...]
-<h2 id="buffer-size-tuning">Buffer size tuning<a class="headerlink" href="#buffer-size-tuning" title="Permanent link">¶</a></h2>
-<p>StringViewArray permits multiple buffers, which enables a flexible buffer layout and potentially reduces the need to copy data. However, a large number of buffers slows down the performance of other operations.
-For example, <a href="https://docs.rs/arrow/latest/arrow/array/trait.Array.html#tymethod.get_array_memory_size"><code>get_array_memory_size</code></a> needs to sum the memory size of each buffer, which takes a long time with thousands of small buffers.
-In certain cases, we found that multiple calls to <a href="https://docs.rs/arrow/latest/arrow/compute/fn.concat_batches.html"><code>concat_batches</code></a> lead to arrays with millions of buffers, which was prohibitively expensive.</p>
-<p>For example, consider a StringViewArray with the previous default buffer size of 8 KB. With this configuration, holding 4GB of string data requires almost half a million buffers! Larger buffer sizes are needed for larger arrays, but we cannot arbitrarily increase the default buffer size, as small arrays would consume too much memory (most arrays require at least one buffer). Buffer sizing is especially problematic in query processing, as we often need to construct small batches of str [...]
-<p>To balance the buffer size trade-off, we again leverage the query processing (DataFusion) semantics to decide when to use larger buffers. While coalescing batches, we combine multiple small string arrays and set a smaller buffer size to keep the total memory consumption low. In string aggregation, we aggregate over an entire Datafusion partition, which can generate a large number of strings, so we set a larger buffer size (2MB).</p>
-<p>To assist situations where the semantics are unknown, we also <a href="https://github.com/apache/arrow-rs/pull/6136">implemented</a> a classic dynamic exponential buffer size growth strategy, which starts with a small buffer size (8KB) and doubles the size of each new buffer up to 2MB. We implemented this strategy in arrow-rs and enabled it by default so that other users of StringViewArray can also benefit from this optimization. See this issue for more details: <a href="https://githu [...]
-<h2 id="end-to-end-query-performance">End-to-end query performance<a class="headerlink" href="#end-to-end-query-performance" title="Permanent link">¶</a></h2>
-<p>We have made significant progress in optimizing StringViewArray filtering operations. Now, let’s test it in the real world to see how it works!</p>
-<p>Let’s consider ClickBench query 22, which selects multiple string fields (<code>URL</code>, <code>Title</code>, and <code>SearchPhase</code>) and applies several filters.</p>
-<p>We ran the benchmark using the following command in the DataFusion repo. Again, the <code>--string-view</code> option means we use StringViewArray instead of StringArray.</p>
-<p>To eliminate the impact of the faster Parquet reading using StringViewArray (see the first part of this blog), Figure 2 plots only the time spent in <code>FilterExec</code>. Without StringViewArray, the filter takes 7.17s; with StringViewArray, the filter only takes 4.86s, a 32% reduction in time. Moreover, we see a 17% improvement in end-to-end query performance.</p>
-<p><img alt="Figure showing StringViewArray reduces the filter time by 32% on ClickBench query 22." class="img-fluid" src="/blog/images/string-view-2/figure2-filter-time.png" width="100%"/></p>
-<p>Figure 2: StringViewArray reduces the filter time by 32% on ClickBench query 22.</p>
-<h1 id="faster-string-aggregation">Faster String Aggregation<a class="headerlink" href="#faster-string-aggregation" title="Permanent link">¶</a></h1>
-<p>So far, we have discussed how to exploit two StringViewArray features: reduced copy and faster filtering. This section focuses on reusing string bytes to repeat string values.</p>
-<p>As described in part one of this blog, if two strings have identical values, StringViewArray can use two different <code>view</code>s pointing at the same buffer range, thus avoiding repeating the string bytes in the buffer. This makes StringViewArray similar to an Arrow <a href="https://docs.rs/arrow/latest/arrow/array/struct.DictionaryArray.html">DictionaryArray</a> that stores Strings—both array types work well for strings with only a few distinct values.</p>
-<p>Deduplicating string values can significantly reduce memory consumption in StringViewArray. However, this process is expensive and involves hashing every string and maintaining a hash table, and so it cannot be done by default when creating a StringViewArray. We introduced an<a href="https://docs.rs/arrow/latest/arrow/array/builder/struct.GenericByteViewBuilder.html#method.with_deduplicate_strings"> opt-in string deduplication mode</a> in arrow-rs for advanced users who know their dat [...]
-<p>Once again, we leverage DataFusion query semantics to identify StringViewArray with duplicate values, such as aggregation queries with multiple group keys. For example, some <a href="https://github.com/apache/datafusion/blob/main/benchmarks/queries/clickbench/queries.sql">ClickBench queries</a> group by two columns:</p>
-<ul>
-<li><code>UserID</code> (an integer with close to 1 M distinct values)</li>
-<li><code>MobilePhoneModel</code> (a string with less than a hundred distinct values)</li>
-</ul>
-<p>In this case, the output row count is<code>count(distinct UserID) * count(distinct MobilePhoneModel)</code>, which is 100M. Each string value of <code>MobilePhoneModel</code> is repeated 1M times. With StringViewArray, we can save space by pointing the repeating values to the same underlying buffer.</p>
-<p>Faster string aggregation with StringView is part of a larger project to <a href="https://github.com/apache/datafusion/issues/7000">improve DataFusion aggregation performance</a>. We have a <a href="https://github.com/apache/datafusion/pull/11794">proof of concept implementation</a> with StringView that can improve the multi-column string aggregation by 20%. We would love your help to get it production ready!</p>
-<h1 id="stringview-pitfalls">StringView Pitfalls<a class="headerlink" href="#stringview-pitfalls" title="Permanent link">¶</a></h1>
-<p>Most existing blog posts (including this one) focus on the benefits of using StringViewArray over other string representations such as StringArray. As we have discussed, even though it requires a significant engineering investment to realize, StringViewArray is a major improvement over StringArray in many cases.</p>
-<p>However, there are several cases where StringViewArray is slower than StringArray. For completeness, we have listed those instances here:</p>
-<ol>
-<li><strong>Tiny strings (when strings are shorter than 8 bytes)</strong>: every element of the StringViewArray consumes at least 16 bytes of memory—the size of the <code>view</code> struct. For an array of tiny strings, StringViewArray consumes more memory than StringArray and thus can cause slower performance due to additional memory pressure on the CPU cache.</li>
-<li><strong>Many repeated short strings</strong>: Similar to the first point, StringViewArray can be slower and require more memory than a DictionaryArray because 1) it can only reuse the bytes in the buffer when the strings are longer than 12 bytes and 2) 32-bit offsets are always used, even when a smaller size (8 bit or 16 bit) could represent all the distinct values.</li>
-<li><strong>Filtering:</strong> As we mentioned above, StringViewArrays often consume more memory than the corresponding StringArray, and memory bloat quickly dominates the performance without GC. However, invoking GC also reduces the benefits of less copying so must be carefully tuned.</li>
-</ol>
-<h1 id="conclusion-and-takeaways">Conclusion and Takeaways<a class="headerlink" href="#conclusion-and-takeaways" title="Permanent link">¶</a></h1>
-<p>In these two blog posts, we discussed what it takes to implement StringViewArray in arrow-rs and then integrate it into DataFusion. Our evaluations on ClickBench queries show that StringView can improve the performance of string-intensive workloads by up to 2x.</p>
-<p>Given that DataFusion already <a href="https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQiI6ZmFsc2UsIkF0aGVuYSAocGFydGl0aW9uZWQpIjpmYWxzZSwiQXRoZW5hIChzaW5nbGUpIjpmYWxzZSwiQXVyb3JhIGZvciBNeVNRTCI6ZmFsc2UsIkF1cm9yYSBmb3IgUG9zdGdyZVNRTCI6ZmFsc2UsIkJ5Q29uaXR5IjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiY2hEQiAoUGFycXVldCwgcGFydGl0aW9uZWQpIjpmYWxzZSwiY2hEQiI6ZmFsc2UsIkNpdHVzIjpmYWxzZSwiQ2xpY2tIb3VzZSBDbG91ZCAoYXdzKSI6ZmFsc2UsIkNsaWNrSG91c2UgQ2xvdWQgKGF3cykgUGFyYWxsZWwgUmVwbGljYXMgT04iOmZh [...]
-<p>StringView is a big project that has received tremendous community support. Specifically, we would like to thank <a href="https://github.com/tustvold">@tustvold</a>, <a href="https://github.com/ariesdevil">@ariesdevil</a>, <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a>, <a href="https://github.com/ClSlaid">@ClSlaid</a>, <a href="https://github.com/2010YOUY01">@2010YOUY01</a>, <a href="https://github.com/chloro-pn">@chloro-pn</a>, <a href="https://github.com/a10y">@a10y</a [...]
-<p>As the introduction states, “German Style Strings” is a relatively straightforward research idea that avoid some string copies and accelerates comparisons. However, applying this (great) idea in practice requires a significant investment in careful software engineering. Again, we encourage the research community to continue to help apply research ideas to industrial systems, such as DataFusion, as doing so provides valuable perspectives when evaluating future research questions for th [...]
-<h3 id="footnotes">Footnotes<a class="headerlink" href="#footnotes" title="Permanent link">¶</a></h3>
-<p>[^5]: There are additional optimizations possible in this operation that the community is working on, such as <a href="https://github.com/apache/datafusion/issues/7957">https://github.com/apache/datafusion/issues/7957</a>.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#faster-string-operations">Faster String Operations</a></li>
-<li><a href="#faster-comparison">Faster comparison</a></li>
-<li><a href="#faster-takeandfilter">Faster takeandfilter</a></li>
-<li><a href="#when-to-gc">When to GC?</a><ul>
-<li><a href="#the-art-of-function-inlining-not-too-much-not-too-little">The art of function inlining: not too much, not too little</a></li>
-<li><a href="#buffer-size-tuning">Buffer size tuning</a></li>
-<li><a href="#end-to-end-query-performance">End-to-end query performance</a></li>
-</ul>
-</li>
-<li><a href="#faster-string-aggregation">Faster String Aggregation</a></li>
-<li><a href="#stringview-pitfalls">StringView Pitfalls</a></li>
-<li><a href="#conclusion-and-takeaways">Conclusion and Takeaways</a><ul>
-<li><a href="#footnotes">Footnotes</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/09/27/datafusion-comet-0.3.0/index.html b/blog/2024/09/27/datafusion-comet-0.3.0/index.html
deleted file mode 100644
index aa3b75d..0000000
--- a/blog/2024/09/27/datafusion-comet-0.3.0/index.html
+++ /dev/null
@@ -1,192 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.3.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Comet 0.3.0 Release
- </h1>
- <p>Posted on: Fri 27 September 2024 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#binary-releases">Binary Releases</a></li>
-<li><a href="#new-features">New Features</a></li>
-<li><a href="#performance-stability">Performance & Stability</a></li>
-<li><a href="#documentation-updates">Documentation Updates</a></li>
-</ul>
-</li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>The Apache DataFusion PMC is pleased to announce version 0.3.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately four weeks of development work and is the result of merging 57 PRs from 12
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.3.0.md">change log</a> for more information.</p>
-<h2 id="release-highlights">Release Highlights<a class="headerlink" href="#release-highlights" title="Permanent link">¶</a></h2>
-<h3 id="binary-releases">Binary Releases<a class="headerlink" href="#binary-releases" title="Permanent link">¶</a></h3>
-<p>Comet jar files are now published to Maven central for amd64 and arm64 architectures (Linux only).</p>
-<p>Files can be found at https://central.sonatype.com/search?q=org.apache.datafusion</p>
-<ul>
-<li>Spark versions 3.3, 3.4, and 3.5 are supported.</li>
-<li>Scala versions 2.12 and 2.13 are supported.</li>
-</ul>
-<h3 id="new-features">New Features<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h3>
-<p>The following expressions are now supported natively:</p>
-<ul>
-<li><code>DateAdd</code></li>
-<li><code>DateSub</code></li>
-<li><code>ElementAt</code></li>
-<li><code>GetArrayElement</code></li>
-<li><code>ToJson</code></li>
-</ul>
-<h3 id="performance-stability">Performance & Stability<a class="headerlink" href="#performance-stability" title="Permanent link">¶</a></h3>
-<ul>
-<li>Upgraded to DataFusion 42.0.0</li>
-<li>Reduced memory overhead due to some memory leaks being fixed</li>
-<li>Comet will now fall back to Spark for queries that use DPP, to avoid performance regressions because Comet does
- not have native support for DPP yet</li>
-<li>Improved performance when converting Spark columnar data to Arrow format</li>
-<li>Faster decimal sum and avg functions </li>
-</ul>
-<h3 id="documentation-updates">Documentation Updates<a class="headerlink" href="#documentation-updates" title="Permanent link">¶</a></h3>
-<ul>
-<li>Improved documentation for deploying Comet with Kubernetes and Helm in the <a href="https://datafusion.apache.org/comet/user-guide/kubernetes.html">Comet Kubernetes Guide</a></li>
-<li>More detailed architectural overview of Comet scan and execution in the <a href="https://datafusion.apache.org/comet/contributor-guide/plugin_overview.html">Comet Plugin Overview</a> in the contributor guide</li>
-</ul>
-<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#binary-releases">Binary Releases</a></li>
-<li><a href="#new-features">New Features</a></li>
-<li><a href="#performance-stability">Performance & Stability</a></li>
-<li><a href="#documentation-updates">Documentation Updates</a></li>
-</ul>
-</li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html b/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html
deleted file mode 100644
index e4e89d5..0000000
--- a/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/index.html
+++ /dev/null
@@ -1,341 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion is now the fastest single node engine for querying Apache Parquet files - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion is now the fastest single node engine for querying Apache Parquet files
- </h1>
- <p>Posted on: Mon 18 November 2024 by Andrew Lamb, Staff Engineer at InfluxData</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#a-strong-history-of-performance-improvements">A Strong History of Performance Improvements</a><ul>
-<li><a href="#stringview">StringView</a></li>
-<li><a href="#parquet">Parquet</a></li>
-<li><a href="#skipping-partial-aggregation-when-it-doesnt-help">Skipping Partial Aggregation When It Doesn't Help</a></li>
-<li><a href="#optimized-multi-column-grouping">Optimized Multi-Column Grouping</a></li>
-</ul>
-</li>
-<li><a href="#whats-next">What’s Next 🚀</a></li>
-<li><a href="#additional-thanks">Additional Thanks</a></li>
-<li><a href="#conclusion">Conclusion</a><ul>
-<li><a href="#notes">Notes</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>I am extremely excited to announce that <a href="https://crates.io/crates/datafusion">Apache DataFusion</a> is the
-fastest engine for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">ClickBench</a>. It is faster
-than <a href="https://duckdb.org/">DuckDB</a>, <a href="https://clickhouse.com/chdb">chDB</a> and <a href="https://clickhouse.com/">Clickhouse</a> using the same hardware. It also marks
-the first time a <a href="https://www.rust-lang.org/">Rust</a>-based engine holds the top spot, which has previously
-been held by traditional C/C++-based engines.</p>
-<p><img alt="Apache DataFusion Logo" class="img-fluid" src="/blog/images/2x_bgwhite_original.png" width="80%"/></p>
-<p><img alt="ClickBench performance for DataFusion 43.0.0" class="img-fluid" src="/blog/images/clickbench-datafusion-43/perf.png" width="100%"/></p>
-<p><strong>Figure 1</strong>: 2024-11-16 <a href="https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQiI6ZmFsc2UsIkFsbG95REIgKHR1bmVkKSI6ZmFsc2UsIkF0aGVuYSAocGFydGl0aW9uZWQpIjpmYWxzZSwiQXRoZW5hIChzaW5nbGUpIjpmYWxzZSwiQXVyb3JhIGZvciBNeVNRTCI6ZmFsc2UsIkF1cm9yYSBmb3IgUG9zdGdyZVNRTCI6ZmFsc2UsIkJ5Q29uaXR5IjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiY2hEQiAoRGF0YUZyYW1lKSI6ZmFsc2UsImNoREIgKFBhcnF1ZXQsIHBhcnRpdGlvbmVkKSI6dHJ1ZSwiY2hEQiI6ZmFsc2UsIkNpdHVzIjpmYWxzZSwiQ2xpY2tIb3VzZSBDbG91ZCAoYXdzKSI6 [...]
-partitioned 14 GB Parquet dataset (100 files, each ~140MB) on a <code>c6a.4xlarge</code> (16
-CPU / 32 GB RAM) VM. Measurements are relative (<code>1.x</code>) to results using
-different hardware.</p>
-<p>Best in class performance on Parquet is now available to anyone. DataFusion’s
-open design lets you start quickly with a full featured Query Engine, including
-SQL, data formats, catalogs, and more, and then customize any behavior you need.
-I predict the continued emergence of new classes of data systems now that
-creators can focus the bulk of their innovation on areas such as query
-languages, system integrations, and data formats rather than trying to play
-catchup with core engine performance.</p>
-<p>ClickBench also includes results for proprietary storage formats, which require
-costly load / export steps, making them useful in fewer use cases and thus much
-less important than open formats (though the idea of use case specific formats
-is interesting[^2]).</p>
-<p>This blog post highlights some of the techniques we used to achieve this
-performance, and celebrates the teamwork involved.</p>
-<h1 id="a-strong-history-of-performance-improvements">A Strong History of Performance Improvements<a class="headerlink" href="#a-strong-history-of-performance-improvements" title="Permanent link">¶</a></h1>
-<p>Performance has long been a core focus for DataFusion's community, and
-speed attracts users and contributors. Recently, we seem to have been
-even more focused on performance, including in July, 2024 when <a href="https://www.linkedin.com/in/mehmet-ozan-kabak/">Mehmet Ozan
-Kabak</a>, CEO of <a href="https://www.synnada.ai/">Synnada</a>, again <a href="https://github.com/apache/datafusion/issues/11442#issuecomment-2226834443">suggested focusing on performance</a>. This
-got many of us excited (who doesn’t love a challenge!), and we have subsequently
-rallied to steadily improve the performance release on release as shown in
-Figure 2.</p>
-<p><img alt="ClickBench performance results over time for DataFusion" class="img-fluid" src="/blog/images/clickbench-datafusion-43/perf-over-time.png" width="100%"/></p>
-<p><strong>Figure 2</strong>: ClickBench performance improved over 30% between DataFusion 34
-(released Dec. 2023) and DataFusion 43 (released Nov. 2024).</p>
-<p>Like all good optimization efforts, ours took sustained effort as DataFusion ran
-out of <a href="https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion">single 2x performance improvements</a> several years ago. Working together our
-community of engineers from around the world[^3] and all experience levels[^4]
-pulled it off (check out <a href="https://github.com/apache/datafusion/issues/12821">this discussion</a> to get a sense). It may be a "<a href="https://db.cs.cmu.edu/seminar2024/">hobo
-sandwich</a>" [^5], but it is a tasty one!</p>
-<p>Of course, most of these techniques have been implemented and described before,
-but until now they were only available in proprietary systems such as
-<a href="https://www.vertica.com/">Vertica</a>, <a href="https://www.databricks.com/product/photon">DataBricks
-Photon</a>, or
-<a href="https://www.snowflake.com/en/">Snowflake</a> or in tightly integrated open source
-systems such as <a href="https://duckdb.org/">DuckDB</a> or
-<a href="https://clickhouse.com/">ClickHouse</a> which were not designed to be extended.</p>
-<h2 id="stringview">StringView<a class="headerlink" href="#stringview" title="Permanent link">¶</a></h2>
-<p>Performance improved for all queries when DataFusion switched to using Arrow
-<code>StringView</code>. Using <code>StringView</code> “just” saves some copies and avoids one memory
-access for certain comparisons. However, these copies and comparisons happen to
-occur in many of the hottest loops during query processing, so optimizing them
-resulted in measurable performance improvements.</p>
-<p><img alt="Illustration of how take works with StringView" class="img-fluid" src="/blog/images/clickbench-datafusion-43/string-view-take.png" width="80%"/></p>
-<p><strong>Figure 3:</strong> Figure from <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">Using StringView / German Style Strings to Make
-Queries Faster: Part 1</a> showing how <code>StringView</code> saves copying data in many cases.</p>
-<p>Using StringView to make DataFusion faster for ClickBench required substantial
-careful, low level optimization work described in <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">Using StringView / German
-Style Strings to Make Queries Faster: Part 1</a> and <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">Part 2</a>. However, it <em>also</em>
-required extending the rest of DataFusion’s operations to support the new type.
-You can get a sense of the magnitude of the work required by looking at the 100+
-pull requests linked to the epic in arrow-rs
-(<a href="https://github.com/apache/arrow-rs/issues/5374">here</a>) and three major epics
-(<a href="https://github.com/apache/datafusion/issues/10918">here</a>,
-<a href="https://github.com/apache/datafusion/issues/11790">here</a> and
-<a href="https://github.com/apache/datafusion/issues/11752">here</a>) in DataFusion.</p>
-<p>Here is a partial list of people involved in the project (I am sorry to those whom I forgot)</p>
-<ul>
-<li><strong>Arrow</strong>: <a href="https://github.com/XiangpengHao">Xiangpeng Hao</a> (InfluxData’s amazing 2024 summer intern and UW Madison PhD), <a href="https://github.com/ariesdevil">Yijun Zhao</a> from DataBend Labs, and <a href="https://github.com/tustvold">Raphael Taylor-Davies</a> laid the foundation. <a href="https://github.com/RinChanNOWWW">RinChanNOW</a> from Tencent and <a href="https://github.com/a10y">Andrew Duffy</a> from SpiralDB helped push it along in the early day [...]
-<li><strong>DataFusion</strong>: <a href="https://github.com/XiangpengHao">Xiangpeng Hao</a>, again charted the initial path and <a href="https://github.com/Weijun-H">Weijun Huang</a>, <a href="https://github.com/dharanad">Dharan Aditya</a> <a href="https://github.com/Lordworms">Lordworms</a>, <a href="https://github.com/goldmedal">Jax Liu</a>, <a href="https://github.com/wiedld">wiedld</a>, <a href="https://github.com/tlm365">Tai Le Manh</a>, <a href="https://github.com/my-vegetable-h [...]
-<li><strong>DataFusion String Function Migration</strong>: <a href="https://github.com/tshauck">Trent Hauck</a> organized the effort and set the patterns, <a href="https://github.com/goldmedal">Jax Liu</a> made a clever testing framework, and <a href="https://github.com/austin362667">Austin Liu</a>, <a href="https://github.com/demetribu">Dmitrii Bu</a>, <a href="https://github.com/tlm365">Tai Le Manh</a>, <a href="https://github.com/PsiACE">Chojan Shang</a>, <a href="https://github.com/ [...]
-</ul>
-<h2 id="parquet">Parquet<a class="headerlink" href="#parquet" title="Permanent link">¶</a></h2>
-<p>Part of the reason for DataFusion's speed in ClickBench is reading Parquet files (really) quickly,
-which reflects invested effort in the Parquet reading system (see <a href="https://www.influxdata.com/blog/querying-parquet-millisecond-latency/">Querying
-Parquet with Millisecond Latency</a> )</p>
-<p>The <a href="https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/parquet/struct.ParquetExec.html">DataFusion ParquetExec</a> (built on the <a href="https://crates.io/crates/parquet">Rust Parquet Implementation</a>) is now the most
-sophisticated open source Parquet reader I know of. It has every optimization we
-can think of for reading Parquet, including projection pushdown, predicate
-pushdown (row group metadata, page index, and bloom filters), limit pushdown,
-parallel reading, interleaved I/O, and late materialized filtering (coming soon ™️
-by default). Some recent work from <a href="https://github.com/itsjunetime">June</a>
-<a href="https://github.com/apache/datafusion/pull/12135">recently unblocked a remaining hurdle</a> for enabling late materialized
-filtering, and conveniently <a href="https://github.com/XiangpengHao">Xiangpeng Hao</a> is
-working on the <a href="https://github.com/apache/arrow-datafusion/issues/3463">final piece</a> (no pressure😅)</p>
-<h2 id="skipping-partial-aggregation-when-it-doesnt-help">Skipping Partial Aggregation When It Doesn't Help<a class="headerlink" href="#skipping-partial-aggregation-when-it-doesnt-help" title="Permanent link">¶</a></h2>
-<p>Many ClickBench queries are aggregations that summarize millions of rows, a
-common task for reporting and dashboarding. DataFusion uses state of the art
-<a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.Accumulator.html#tymethod.state">two phase aggregation</a> plans. Normally, two phase aggregation works well as the
-first phase consolidates many rows immediately after reading, while the data is
-still in cache. However, for certain “high cardinality” aggregate queries (that
-have large numbers of groups), <a href="https://github.com/apache/datafusion/issues/6937">the two phase aggregation strategy used in
-DataFusion was inefficient</a>,
-manifesting in relatively slower performance compared to other engines for
-ClickBench queries such as</p>
-<pre><code class="language-sql">SELECT "WatchID", "ClientIP", COUNT(*) AS c, ...
-FROM hits
-GROUP BY "WatchID", "ClientIP" /* <----- 13M Distinct Groups!!! */
-ORDER BY c DESC
-LIMIT 10;
-</code></pre>
-<p>For such queries, the first aggregation phase does not significantly
-reduce the number of rows, which wastes significant effort. <a href="https://github.com/korowa">Eduard
-Karacharov</a> contributed a <a href="https://github.com/apache/datafusion/pull/11627">dynamic strategy</a> to
-bypass the first phase when it is not working efficiently, shown in Figure 4.</p>
-<p><img alt="Two phase aggregation diagram from DataFusion API docs annotated to show first phase not helping" class="img-fluid" src="/blog/images/clickbench-datafusion-43/skipping-partial-aggregation.png" width="100%"/></p>
-<p><strong>Figure 4</strong>: Diagram from <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.Accumulator.html#tymethod.state">DataFusion API docs</a> showing when the multi-phase
-grouping is not effective</p>
-<h2 id="optimized-multi-column-grouping">Optimized Multi-Column Grouping<a class="headerlink" href="#optimized-multi-column-grouping" title="Permanent link">¶</a></h2>
-<p>Another method for improving analytic database performance is specialized (aka
-highly optimized) versions of operations for different data types, which the
-system picks at runtime based on the query. Like other systems, DataFusion has
-specialized code for handling different types of group columns. For example,
-there is <a href="https://github.com/apache/datafusion/blob/73507c307487708deb321e1ba4e0d302084ca27e/datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs">special code</a> that handles <code>GROUP BY int_id</code> and <a href="https://github.com/apache/datafusion/blob/73507c307487708deb321e1ba4e0d302084ca27e/datafusion/physical-plan/src/aggregates/group_values/single_group_by/bytes.rs">different special
-code</a> that handles <code>GROUP BY string_id</code> .</p>
-<p>When a query groups by multiple columns, it is tricker to apply this technique.
-For example <code>GROUP BY string_id, int_id</code> and <code>GROUP BY int_id, string_id</code> have
-different optimal structures, but it is not possible to include specialized
-versions for all possible combinations of group column types.</p>
-<p>DataFusion includes <a href="https://github.com/apache/datafusion/blob/73507c307487708deb321e1ba4e0d302084ca27e/datafusion/physical-plan/src/aggregates/group_values/row.rs#L33-L39">a general Row based mechanism</a> that works for any
-combination of column types, but this general mechanism copies each value twice
-as shown in Figure 5. The cost of this copy <a href="https://github.com/apache/datafusion/issues/9403">is especially high for variable
-length strings and binary data</a>.</p>
-<p><img alt="Row based storage for multiple group columns" class="img-fluid" src="/blog/images/clickbench-datafusion-43/row-based-storage.png" width="100%"/></p>
-<p><strong>Figure 5</strong>: Prior to DataFusion 43.0.0, queries with multiple group columns
-used Row based group storage and copied each group value twice. This copy
-consumes a substantial amount of the query time for queries with many distinct
-groups, such as several of the queries in ClickBench.</p>
-<p>Many optimizations in Databases boil down to simply avoiding copies, and this
-was no exception. The trick was to figure out how to avoid copies without
-causing per-column comparison overhead to dominate or complexity to get out of
-hand. In a great example of diligent and disciplined engineering, <a href="https://github.com/jayzhan211">Jay
-Zhan</a> tried <a href="https://github.com/apache/datafusion/pull/10937">several</a>, <a href="https://github.com/apache/datafusion/pull/10976">different</a> approaches until arriving
-at the [one shipped in DataFusion <code>43.0.0</code>], shown in Figure 6.</p>
-<p><img alt="Column based storage for multiple group columns" class="img-fluid" src="/blog/images/clickbench-datafusion-43/column-based-storage.png" width="100%"/></p>
-<p><strong>Figure 6</strong>: DataFusion 43.0.0’s new columnar group storage copies each group
-value exactly once, which is significantly faster when grouping by multiple
-columns.</p>
-<p>Huge thanks as well to <a href="https://github.com/eejbyfeldt">Emil Ejbyfeldt</a> and
-<a href="https://github.com/Dandandan">Daniël Heres</a> for their help reviewing and to
-<a href="https://github.com/Rachelint">Rachelint (kamille</a>) for reviewing and
-contributing a faster <a href="https://github.com/apache/datafusion/pull/12996">vectorized append and compare for multiple groups</a> which
-will be released in DataFusion 44. The discussion on <a href="https://github.com/apache/datafusion/issues/9403">the ticket</a> is another
-great example of the power of the DataFusion community working together to build
-great software.</p>
-<h1 id="whats-next">What’s Next 🚀<a class="headerlink" href="#whats-next" title="Permanent link">¶</a></h1>
-<p>Just as I expect the performance of other engines to improve, DataFusion has
-several more performance improvements lined up itself:</p>
-<ol>
-<li><a href="https://github.com/apache/datafusion/pull/11943#top">Intermediate results blocked management</a> (thanks again <a href="https://github.com/Rachelint">Rachelint (kamille</a>)</li>
-<li><a href="https://github.com/apache/datafusion/issues/3463">Enable parquet filter pushdown by default</a></li>
-</ol>
-<p>We are also talking about what to focus on over the <a href="https://github.com/apache/datafusion/issues/13274">next three
-months</a> and are always
-looking for people to help! If you want to geek out (obsess??) about performance
-and other features with engineers from around the world, <a href="https://datafusion.apache.org/contributor-guide/communication.html">we would love you to
-join us</a>.</p>
-<h1 id="additional-thanks">Additional Thanks<a class="headerlink" href="#additional-thanks" title="Permanent link">¶</a></h1>
-<p>In addition to the people called out above, thanks:</p>
-<ol>
-<li><a href="https://github.com/pmcgleenon">Patrick McGleenon</a> for running ClickBench and gathering this data (<a href="https://github.com/apache/datafusion/issues/13099#issuecomment-2478314793">source</a>).</li>
-<li>Everyone I missed in the shoutouts – there are so many of you. We appreciate everyone.</li>
-</ol>
-<h1 id="conclusion">Conclusion<a class="headerlink" href="#conclusion" title="Permanent link">¶</a></h1>
-<p>I have dreamed about DataFusion being on top of the ClickBench leaderboard for
-several years. I often watched with envy improvements in systems backed by large
-VC investments, internet companies, or world class research institutions, and
-doubted that we could pull off something similar in an open source project with
-always limited time.</p>
-<p>The fact that we have now surpassed those other systems in query performance I
-think speaks to the power and possibility of focusing on community and aligning
-our collective enthusiasm and skills towards a common goal. Of course, being on
-the top in any particular benchmark is likely fleeting as other engines will
-improve, but so will DataFusion!</p>
-<p>I love working on DataFusion – the people, the quality of the code, my
-interactions and the results we have achieved together far surpass my
-expectations as well as most of my other software development experiences. I
-can’t wait to see what people will build next, and hope to <a href="https://github.com/apache/datafusion">see you
-online</a>.</p>
-<h2 id="notes">Notes<a class="headerlink" href="#notes" title="Permanent link">¶</a></h2>
-<p>[^1]: Note that DuckDB is slightly faster on the ‘cold’ run.</p>
-<p>[^2]: Want to try your hand at a custom format for ClickBench fame / glory?: <a href="https://github.com/apache/datafusion/issues/13448">Make DataFusion the fastest engine in ClickBench with custom file format</a></p>
-<p>[^3]: We have contributors from North America, South American, Europe, Asia, Africa and Australia</p>
-<p>[^4]: Undergraduates, PhD, Junior engineers, and getting-kind-of-crotchety experienced engineers</p>
-<p>[^5]: Thanks to Andy Pavlo, I love that nomenclature</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#a-strong-history-of-performance-improvements">A Strong History of Performance Improvements</a><ul>
-<li><a href="#stringview">StringView</a></li>
-<li><a href="#parquet">Parquet</a></li>
-<li><a href="#skipping-partial-aggregation-when-it-doesnt-help">Skipping Partial Aggregation When It Doesn't Help</a></li>
-<li><a href="#optimized-multi-column-grouping">Optimized Multi-Column Grouping</a></li>
-</ul>
-</li>
-<li><a href="#whats-next">What’s Next 🚀</a></li>
-<li><a href="#additional-thanks">Additional Thanks</a></li>
-<li><a href="#conclusion">Conclusion</a><ul>
-<li><a href="#notes">Notes</a></li>
-</ul>
-</li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/11/19/datafusion-python-udf-comparisons/index.html b/blog/2024/11/19/datafusion-python-udf-comparisons/index.html
deleted file mode 100644
index 08b11af..0000000
--- a/blog/2024/11/19/datafusion-python-udf-comparisons/index.html
+++ /dev/null
@@ -1,687 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Comparing approaches to User Defined Functions in Apache DataFusion using Python - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Comparing approaches to User Defined Functions in Apache DataFusion using Python
- </h1>
- <p>Posted on: Tue 19 November 2024 by timsaucer</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#personal-context">Personal Context</a></li>
-<li><a href="#user-defined-functions">User-Defined Functions</a><ul>
-<li><a href="#use-case-1-scalar-function">Use Case 1: Scalar Function</a></li>
-<li><a href="#use-case-2-aggregate-function">Use Case 2: Aggregate Function</a></li>
-</ul>
-</li>
-<li><a href="#pure-python-approach">Pure Python approach</a></li>
-<li><a href="#python-approach-using-pyarrow-compute">Python approach using PyArrow compute</a></li>
-<li><a href="#rust-udf-with-python-wrapper">Rust UDF with Python wrapper</a><ul>
-<li><a href="#rust-udf-with-initialization">Rust UDF with initialization</a></li>
-<li><a href="#rust-udf-with-direct-iteration">Rust UDF with direct iteration</a></li>
-</ul>
-</li>
-<li><a href="#user-defined-aggregate-function">User Defined Aggregate Function</a></li>
-<li><a href="#user-defined-window-functions">User Defined Window Functions</a></li>
-<li><a href="#performance-comparison">Performance Comparison</a></li>
-<li><a href="#final-thoughts-and-recommendations">Final Thoughts and Recommendations</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<h2 id="personal-context">Personal Context<a class="headerlink" href="#personal-context" title="Permanent link">¶</a></h2>
-<p>For a few months now I’ve been working with <a href="https://datafusion.apache.org/">Apache DataFusion</a>, a
-fast query engine written in Rust. From my experience the language that nearly all data scientists
-are working in is Python. In general, data scientists often use <a href="https://pandas.pydata.org/">Pandas</a>
-for in-memory tasks and <a href="https://spark.apache.org/">PySpark</a> for larger tasks that require
-distributed processing.</p>
-<p>In addition to DataFusion, there is another Rust based newcomer to the DataFrame world,
-<a href="https://pola.rs/">Polars</a>. The latter is growing extremely fast, and it serves many of the same
-use cases as DataFusion. For my use cases, I'm interested in DataFusion because I want to be able
-to build small scale tests rapidly and then scale them up to larger distributed systems with ease.
-I do recommend evaluating Polars for in-memory work.</p>
-<p>Personally, I would love a single query approach that is fast for both in-memory usage and can
-extend to large batch processing to exploit parallelization. I think DataFusion, coupled with
-<a href="https://datafusion.apache.org/ballista/">Ballista</a> or
-<a href="https://github.com/apache/datafusion-ray">DataFusion-Ray</a>, may provide this solution.</p>
-<p>As I’m testing, I’m primarily limiting my work to the
-<a href="https://datafusion.apache.org/python/">datafusion-python</a> project, a wrapper around the Rust
-DataFusion library. This wrapper gives you the speed advantages of keeping all of the data in the
-Rust implementation and the ergonomics of working in Python. Personally, I would prefer to work
-purely in Rust, but I also recognize that since the industry works in Python we should meet the
-people where they are.</p>
-<h2 id="user-defined-functions">User-Defined Functions<a class="headerlink" href="#user-defined-functions" title="Permanent link">¶</a></h2>
-<p>The focus of this post is User-Defined Functions (UDFs). The DataFusion library gives a lot of
-useful functions already for doing DataFrame manipulation. These are going to be similar to those
-you find in other DataFrame libraries. You’ll be able to do simple arithmetic, create substrings of
-columns, or find the average value across a group of rows. These cover most of the use cases
-you’ll need in a DataFrame.</p>
-<p>However, there will always arise times when you want a custom function. With UDFs you open a
-world of possibilities in your code. Sometimes there simply isn’t an easy way to use built-in
-functions to achieve your goals.</p>
-<p>In the following, I’m going to demonstrate two example use cases. These are based on real world
-problems I’ve encountered. Also I want to demonstrate the approach of “make it work, make it work
-well, make it work fast” that is a motto I’ve seen thrown around in data science.</p>
-<p>I will demonstrate three approaches to writing UDFs. In order of increasing performance they are</p>
-<ul>
-<li>Writing a pure Python function to do your computation</li>
-<li>Using the PyArrow libraries in Python to accelerate your processing</li>
-<li>Writing a UDF in Rust and exposing it to Python</li>
-</ul>
-<p>Additionally I will demonstrate two variants of this. The first will be nearly identical to the
-PyArrow library approach to simplify understanding how to connect the Rust code to Python. In the
-second version we will do the iteration through the input arrays ourselves to give even greater
-flexibility to the user.</p>
-<p>Here are the two example use cases, taken from my own work but generalized.</p>
-<h3 id="use-case-1-scalar-function">Use Case 1: Scalar Function<a class="headerlink" href="#use-case-1-scalar-function" title="Permanent link">¶</a></h3>
-<p>I have a DataFrame and a list of tuples that I’m interested in. I want to filter out the DataFrame
-to only have values that match those tuples from certain columns in the DataFrame.</p>
-<p>To give a concrete example, we will use data generated for the <a href="https://www.tpc.org/tpch/">TPC-H benchmarks</a>.
-Suppose I have a table of sales line items. There are many columns, but I am interested in three: a
-part key (<code>p_partkey</code>), supplier key (<code>p_suppkey</code>), and return status (<code>p_returnflag</code>). I want
-only to return a DataFrame with a specific combination of these three values. That is, I want
-to know if part number 1530 from supplier 4031 was sold (not returned), so I want a specific
-combination of <code>p_partkey = 1530</code>, <code>p_suppkey = 4031</code>, and <code>p_returnflag = 'N'</code>. I have a small
-handful of these combinations I want to return.</p>
-<p>Probably the most ergonomic way to do this without UDF is to turn that list of tuples into a
-DataFrame itself, perform a join, and select the columns from the original DataFrame. If we were
-working in PySpark we would probably broadcast join the DataFrame created from the tuple list since
-it is tiny. In practice, I have found that with some DataFrame libraries performing a filter rather
-than a join can be significantly faster. This is worth profiling for your specific use case.</p>
-<h3 id="use-case-2-aggregate-function">Use Case 2: Aggregate Function<a class="headerlink" href="#use-case-2-aggregate-function" title="Permanent link">¶</a></h3>
-<p>I have a DataFrame with many values that I want to aggregate. I have already analyzed it and
-determined there is a noise level below which I do not want to include in my analysis. I want to
-compute a sum of only values that are above my noise threshold.</p>
-<p>This can be done fairly easy without leaning on a User Defined Aggregate Function (UDAF). You can
-simply filter the DataFrame and then aggregate using the built-in <code>sum</code> function. Here, we
-demonstrate doing this as a UDF primarily as an example of how to write UDAFs. We will use the
-PyArrow compute approach.</p>
-<h2 id="pure-python-approach">Pure Python approach<a class="headerlink" href="#pure-python-approach" title="Permanent link">¶</a></h2>
-<p>The fastest way (developer time, not code time) for me to implement the scalar problem solution
-was to do something along the lines of “for each row, check the values of interest contains that
-tuple”. I’ve published this as
-<a href="https://github.com/apache/datafusion-python/blob/main/examples/python-udf-comparisons.py">an example</a>
-in the <a href="https://github.com/apache/datafusion-python">datafusion-python repository</a>. Here is an
-example of how this can be done:</p>
-<pre><code class="language-python">values_of_interest = [
- (1530, 4031, "N"),
- (6530, 1531, "N"),
- (5618, 619, "N"),
- (8118, 8119, "N"),
-]
-
-def is_of_interest_impl(
- partkey_arr: pa.Array,
- suppkey_arr: pa.Array,
- returnflag_arr: pa.Array,
-) -> pa.Array:
- result = []
- for idx, partkey in enumerate(partkey_arr):
- partkey = partkey.as_py()
- suppkey = suppkey_arr[idx].as_py()
- returnflag = returnflag_arr[idx].as_py()
- value = (partkey, suppkey, returnflag)
- result.append(value in values_of_interest)
-
- return pa.array(result)
-
-# Wrap our custom function with `datafusion.udf`, annotating expected
-# parameter and return types
-is_of_interest = udf(
- is_of_interest_impl,
- [pa.int64(), pa.int64(), pa.utf8()],
- pa.bool_(),
- "stable",
-)
-
-df_udf_filter = df_lineitem.filter(
- is_of_interest(col("l_partkey"), col("l_suppkey"), col("l_returnflag"))
-)
-</code></pre>
-<p>When working with a DataFusion UDF in Python, you define your function to take in some number of
-expressions. During the evaluation, these will get computed into their corresponding values and
-passed to your UDF as a PyArrow Array. We must return an Array also with the same number of
-elements (rows). So the UDF example just iterates through all of the arrays and checks to see if
-the tuple created from these columns matches any of those that we’re looking for.</p>
-<p>I’ll repeat because this is something that tripped me up the first time I wrote a UDF for
-datafusion: <strong>DataFusion UDFs, even scalar UDFs, process an array of values at a time not a single
-row.</strong> This is different from some other DataFrame libraries and you may need to recognize a slight
-change in mentality.</p>
-<p>Some important lines here are the lines like <code>partkey = partkey.as_py()</code>. When we do this, we pay a
-heavy cost. Now instead of keeping the analysis in the Rust code, we have to take the values in the
-array and convert them over to Python objects. In this case we end up getting two numbers and a
-string as real Python objects, complete with reference counting and all. Also we are iterating
-through the array in Python rather than Rust native. These will <strong>significantly</strong> slow down your
-code. Any time you have to cross the barrier where you change values inside the Rust arrays into
-Python objects or vice versa you will pay <strong>heavy</strong> cost in that transformation. You will want to
-design your UDFs to avoid this as much as possible.</p>
-<h2 id="python-approach-using-pyarrow-compute">Python approach using PyArrow compute<a class="headerlink" href="#python-approach-using-pyarrow-compute" title="Permanent link">¶</a></h2>
-<p>DataFusion uses <a href="https://arrow.apache.org/">Apache Arrow</a> as its in-memory data format. This can
-be seen in the way that Arrow Arrays are passed into the UDFs. We can take advantage of the fact
-that <a href="https://arrow.apache.org/docs/python/">PyArrow</a>, the canonical Python Arrow implementation,
-provides a variety of
-useful functions. In the example below, we are only using a few of the boolean functions and the
-equality function. Each of these functions takes two arrays and analyzes them row by row. In the
-below example, we shift the logic around a little since we are now operating on an entire array of
-values instead of checking a single row ourselves.</p>
-<pre><code class="language-python">import pyarrow.compute as pc
-
-def udf_using_pyarrow_compute_impl(
- partkey_arr: pa.Array,
- suppkey_arr: pa.Array,
- returnflag_arr: pa.Array,
-) -> pa.Array:
- results = None
- for partkey, suppkey, returnflag in values_of_interest:
- filtered_partkey_arr = pc.equal(partkey_arr, partkey)
- filtered_suppkey_arr = pc.equal(suppkey_arr, suppkey)
- filtered_returnflag_arr = pc.equal(returnflag_arr, returnflag)
-
- resultant_arr = pc.and_(filtered_partkey_arr, filtered_suppkey_arr)
- resultant_arr = pc.and_(resultant_arr, filtered_returnflag_arr)
-
- if results is None:
- results = resultant_arr
- else:
- results = pc.or_(results, resultant_arr)
-
- return results
-
-
-udf_using_pyarrow_compute = udf(
- udf_using_pyarrow_compute_impl,
- [pa.int64(), pa.int64(), pa.utf8()],
- pa.bool_(),
- "stable",
-)
-
-df_udf_pyarrow_compute = df_lineitem.filter(
- udf_using_pyarrow_compute(col("l_partkey"), col("l_suppkey"), col("l_returnflag"))
-)
-</code></pre>
-<p>The idea in the code above is that we will iterate through each of the values of interest, which we
-expect to be small. For each of the columns, we compare the value of interest to it’s corresponding
-array using <code>pyarrow.compute.equal</code>. This will give use three boolean arrays. We have a match to
-the tuple if we have a row in all three arrays that is true, so we use <code>pyarrow.compute.and_</code>. Now
-our return value from the UDF needs to include arrays for which any of the values of interest list
-of tuples exists, so we take the result from the current loop and perform a <code>pyarrow.compute.or_</code>
-on it.</p>
-<p>From my benchmarking, switching from approach of converting values into Python objects to this
-approach of using the PyArrow built-in functions leads to about a 10x speed improvement in this
-simple problem.</p>
-<p>It’s worth noting that almost all of the PyArrow compute functions expect to take one or two arrays
-as their arguments. If you need to write a UDF that is evaluating three or more columns, you’ll
-need to do something akin to what we’ve shown here.</p>
-<h2 id="rust-udf-with-python-wrapper">Rust UDF with Python wrapper<a class="headerlink" href="#rust-udf-with-python-wrapper" title="Permanent link">¶</a></h2>
-<p>This is the most complicated approach, but has the potential to be the most performant. What we
-will do here is write a Rust function to perform our computation and then expose that function to
-Python. I know of two use cases where I would recommend this approach. The first is the case when
-the PyArrow compute functions are insufficient for your needs. Perhaps your code is too complex or
-could be greatly simplified if you pulled in some outside dependency. The second use case is when
-you have written a UDF that you’re sharing across multiple projects and have hardened the approach.
-It is possible that you can implement your function in Rust to give a speed improvement and then
-every project that is using this shared UDF will benefit from those updates.</p>
-<p>When deciding to use this approach, it’s worth considering how much you think you’ll actually
-benefit from the Rust implementation to decide if it’s worth the additional effort to maintain and
-deploy the Python wheels you generate. It is certainly not necessary for every use case.</p>
-<p>Due to the excellent work by the Python arrow team, we can simplify our work to needing only two
-dependencies on the Rust side, <a href="https://github.com/apache/arrow-rs">arrow-rs</a> and
-<a href="https://pyo3.rs/">pyo3</a>. I have posted a <a href="https://github.com/timsaucer/tuple_filter_example">minimal example</a>.
-You’ll need <a href="https://github.com/PyO3/maturin">maturin</a> to build the project, and you must use
-release mode when building to get the expected performance.</p>
-<pre><code class="language-bash">maturin develop --release
-</code></pre>
-<p>When you write your UDF in Rust you generally will need to take these steps</p>
-<ol>
-<li>Write a function description that takes in some number of Python generic objects.</li>
-<li>Convert these objects to Arrow Arrays of the appropriate type(s).</li>
-<li>Perform your computation and create a resultant Array.</li>
-<li>Convert the array into a Python generic object.</li>
-</ol>
-<p>For the conversion to and from Python objects, we can take advantage of the
-<code>ArrayData::from_pyarrow_bound</code> and <code>ArrayData::to_pyarrow</code> functions. All that remains is to
-perform your computation.</p>
-<p>We are going to demonstrate doing this computation in two ways. The first is to mimic what we’ve
-done in the above approach using PyArrow. In the second we demonstrate iterating through the three
-arrays ourselves.</p>
-<p>In our first approach, we can expect the performance to be nearly identical to when we used the
-PyArrow compute functions. On the Rust side we will have slightly less overhead but the heavy
-lifting portions of the code are essentially the same between this Rust implementation and the
-PyArrow approach above.</p>
-<p>The reason for demonstrating this, even though it doesn’t provide a significant speedup over
-Python, is to primarily demonstrate how to make the Python to Rust with Python wrapper
-transition. In the second implementation you can see how we can iterate through all of the arrays
-ourselves.</p>
-<p>In this first example, we are hard coding the values of interest, but in the following section
-we demonstrate passing these in during initialization.</p>
-<pre><code class="language-rust">#[pyfunction]
-pub fn tuple_filter_fn(
- py: Python<'_>,
- partkey_expr: &Bound<'_, PyAny>,
- suppkey_expr: &Bound<'_, PyAny>,
- returnflag_expr: &Bound<'_, PyAny>,
-) -> PyResult<Py<PyAny>> {
- let partkey_arr: PrimitiveArray<Int64Type> =
- ArrayData::from_pyarrow_bound(partkey_expr)?.into();
- let suppkey_arr: PrimitiveArray<Int64Type> =
- ArrayData::from_pyarrow_bound(suppkey_expr)?.into();
- let returnflag_arr: StringArray = ArrayData::from_pyarrow_bound(returnflag_expr)?.into();
-
- let values_of_interest = vec![
- (1530, 4031, "N".to_string()),
- (6530, 1531, "N".to_string()),
- (5618, 619, "N".to_string()),
- (8118, 8119, "N".to_string()),
- ];
-
- let mut res: Option<BooleanArray> = None;
-
- for (partkey, suppkey, returnflag) in &values_of_interest {
- let filtered_partkey_arr = BooleanArray::from_unary(&partkey_arr, |p| p == *partkey);
- let filtered_suppkey_arr = BooleanArray::from_unary(&suppkey_arr, |s| s == *suppkey);
- let filtered_returnflag_arr =
- BooleanArray::from_unary(&returnflag_arr, |s| s == returnflag);
-
- let part_and_supp = compute::and(&filtered_partkey_arr, &filtered_suppkey_arr)
- .map_err(|e| PyValueError::new_err(e.to_string()))?;
- let resultant_arr = compute::and(&part_and_supp, &filtered_returnflag_arr)
- .map_err(|e| PyValueError::new_err(e.to_string()))?;
-
- res = match res {
- Some(r) => compute::or(&r, &resultant_arr).ok(),
- None => Some(resultant_arr),
- };
- }
-
- res.unwrap().into_data().to_pyarrow(py)
-}
-
-
-#[pymodule]
-fn tuple_filter_example(module: &Bound<'_, PyModule>) -> PyResult<()> {
- module.add_function(wrap_pyfunction!(tuple_filter_fn, module)?)?;
- Ok(())
-}
-</code></pre>
-<p>To use this we use the <code>udf</code> function in <code>datafusion-python</code> just as before.</p>
-<pre><code class="language-python">from datafusion import udf
-import pyarrow as pa
-from tuple_filter_example import tuple_filter_fn
-
-udf_using_custom_rust_fn = udf(
- tuple_filter_fn,
- [pa.int64(), pa.int64(), pa.utf8()],
- pa.bool_(),
- "stable",
-)
-</code></pre>
-<p>That's it! We've now got a third party Rust UDF with Python wrappers working with DataFusion's
-Python bindings!</p>
-<h3 id="rust-udf-with-initialization">Rust UDF with initialization<a class="headerlink" href="#rust-udf-with-initialization" title="Permanent link">¶</a></h3>
-<p>Looking at the code above, you can see that it is hard coding the values we're interested in. There
-are many types of UDFs that don't require any additional data provided to them before they start
-the computation. The code above is sloppy, so let's clean it up.</p>
-<p>We want to write the function to take some additional data. A limitation of the UDFs we create is
-that they expect to operate on entire arrays of data at a time. We can get around this problem by
-creating an initializer for our UDF. We do this by defining a Rust struct that contains the data we
-need and implement two methods on this struct, <code>new</code> and <code>__call__</code>. By doing this we will create a
-Python object that is callable, so it can be the function we provide to <code>udf</code>.</p>
-<pre><code class="language-rust">#[pyclass]
-pub struct TupleFilterClass {
- values_of_interest: Vec<(i64, i64, String)>,
-}
-
-#[pymethods]
-impl TupleFilterClass {
- #[new]
- fn new(values_of_interest: Vec<(i64, i64, String)>) -> Self {
- Self {
- values_of_interest,
- }
- }
-
- fn __call__(
- &self,
- py: Python<'_>,
- partkey_expr: &Bound<'_, PyAny>,
- suppkey_expr: &Bound<'_, PyAny>,
- returnflag_expr: &Bound<'_, PyAny>,
- ) -> PyResult<Py<PyAny>> {
- let partkey_arr: PrimitiveArray<Int64Type> =
- ArrayData::from_pyarrow_bound(partkey_expr)?.into();
- let suppkey_arr: PrimitiveArray<Int64Type> =
- ArrayData::from_pyarrow_bound(suppkey_expr)?.into();
- let returnflag_arr: StringArray = ArrayData::from_pyarrow_bound(returnflag_expr)?.into();
-
- let mut res: Option<BooleanArray> = None;
-
- for (partkey, suppkey, returnflag) in &self.values_of_interest {
- let filtered_partkey_arr = BooleanArray::from_unary(&partkey_arr, |p| p == *partkey);
- let filtered_suppkey_arr = BooleanArray::from_unary(&suppkey_arr, |s| s == *suppkey);
- let filtered_returnflag_arr =
- BooleanArray::from_unary(&returnflag_arr, |s| s == returnflag);
-
- let part_and_supp = compute::and(&filtered_partkey_arr, &filtered_suppkey_arr)
- .map_err(|e| PyValueError::new_err(e.to_string()))?;
- let resultant_arr = compute::and(&part_and_supp, &filtered_returnflag_arr)
- .map_err(|e| PyValueError::new_err(e.to_string()))?;
-
- res = match res {
- Some(r) => compute::or(&r, &resultant_arr).ok(),
- None => Some(resultant_arr),
- };
- }
-
- res.unwrap().into_data().to_pyarrow(py)
- }
-}
-
-#[pymodule]
-fn tuple_filter_example(module: &Bound<'_, PyModule>) -> PyResult<()> {
- module.add_class::<TupleFilterClass>()?;
- Ok(())
-}
-</code></pre>
-<p>When you write this, you don't have to call your constructor <code>new</code>. The more important part is that
-you have <code>#[new]</code> designated on the function. With this you can provide any kinds of data you need
-during processing. Using this initializer in Python is fairly straightforward.</p>
-<pre><code class="language-python">from datafusion import udf
-import pyarrow as pa
-from tuple_filter_example import TupleFilterClass
-
-tuple_filter_class = TupleFilterClass(values_of_interest)
-
-udf_using_custom_rust_fn_with_data = udf(
- tuple_filter_class,
- [pa.int64(), pa.int64(), pa.utf8()],
- pa.bool_(),
- "stable",
- name="tuple_filter_with_data"
-)
-</code></pre>
-<p>When you use this approach you will need to provide a <code>name</code> argument to <code>udf</code>. This is because our
-class/struct does not get the <code>__qualname__</code> attribute that the <code>udf</code> function is looking for. You
-can give this udf any name you choose.</p>
-<h3 id="rust-udf-with-direct-iteration">Rust UDF with direct iteration<a class="headerlink" href="#rust-udf-with-direct-iteration" title="Permanent link">¶</a></h3>
-<p>The final version of our scalar UDF is one where we implement it in Rust and iterate through all of
-the arrays ourselves. If you are iterating through more than 3 arrays at a time I recommend looking
-at <a href="https://docs.rs/itertools/latest/itertools/macro.izip.html">izip</a> in the
-<a href="https://crates.io/crates/itertools">itertools crate</a>. For ease of understanding and since we only
-have 3 arrays here I will just explicitly create my own tuple here.</p>
-<pre><code class="language-rust">#[pyclass]
-pub struct TupleFilterDirectIterationClass {
- values_of_interest: Vec<(i64, i64, String)>,
-}
-
-#[pymethods]
-impl TupleFilterDirectIterationClass {
- #[new]
- fn new(values_of_interest: Vec<(i64, i64, String)>) -> Self {
- Self { values_of_interest }
- }
-
- fn __call__(
- &self,
- py: Python<'_>,
- partkey_expr: &Bound<'_, PyAny>,
- suppkey_expr: &Bound<'_, PyAny>,
- returnflag_expr: &Bound<'_, PyAny>,
- ) -> PyResult<Py<PyAny>> {
- let partkey_arr: PrimitiveArray<Int64Type> =
- ArrayData::from_pyarrow_bound(partkey_expr)?.into();
- let suppkey_arr: PrimitiveArray<Int64Type> =
- ArrayData::from_pyarrow_bound(suppkey_expr)?.into();
- let returnflag_arr: StringArray = ArrayData::from_pyarrow_bound(returnflag_expr)?.into();
-
- let values_to_search: Vec<(&i64, &i64, &str)> = (&self.values_of_interest)
- .iter()
- .map(|(a, b, c)| (a, b, c.as_str()))
- .collect();
-
- let values = partkey_arr
- .values()
- .iter()
- .zip(suppkey_arr.values().iter())
- .zip(returnflag_arr.iter())
- .map(|((a, b), c)| (a, b, c.unwrap_or_default()))
- .map(|v| values_to_search.contains(&v));
-
- let res: BooleanArray = BooleanBuffer::from_iter(values).into();
-
- res.into_data().to_pyarrow(py)
- }
-}
-</code></pre>
-<p>We convert the <code>values_of_interest</code> into a vector of borrowed types so that we can do a fast search
-without creating additional memory. The other option is to turn the <code>returnflag</code> into a <code>String</code>
-but that memory allocation is unnecessary. After that we use two <code>zip</code> operations so that we can
-iterate over all three columns in a single pass. Since each <code>zip</code> will return a tuple of two
-elements, a quick <code>map</code> turns them into the tuple format we need. Also, <code>StringArray</code> is a little
-different in the buffer it uses, so it is treated slightly differently from the others.</p>
-<h2 id="user-defined-aggregate-function">User Defined Aggregate Function<a class="headerlink" href="#user-defined-aggregate-function" title="Permanent link">¶</a></h2>
-<p>Writing a user defined aggregate function or user defined window function is slightly more complex
-than scalar functions. This is because we must accumulate values and there is no guarantee that one
-batch will contain all the values we are aggregating over. For this we need to define an
-<code>Accumulator</code> which will do a few things.</p>
-<ul>
-<li>Process a batch and compute an internal state</li>
-<li>Share the state so that we can combine multiple batches</li>
-<li>Merge the results across multiple batches</li>
-<li>Return the final result</li>
-</ul>
-<p>In the example below, we're going to look at customer orders and we want to know per customer ID,
-how much they have ordered total. We want to ignore small orders, which we define as anything under
-5000.</p>
-<pre><code class="language-python">from datafusion import Accumulator, udaf
-import pyarrow as pa
-import pyarrow.compute as pc
-
-IGNORE_THRESHOLD = 5000.0
-class AboveThresholdAccum(Accumulator):
- def __init__(self) -> None:
- self._sum = 0.0
-
- def update(self, values: pa.Array) -> None:
- over_threshold = pc.greater(values, pa.scalar(IGNORE_THRESHOLD))
- sum_above = pc.sum(values.filter(over_threshold)).as_py()
- if sum_above is None:
- sum_above = 0.0
- self._sum = self._sum + sum_above
-
- def merge(self, states: List[pa.Array]) -> None:
- self._sum = self._sum + pc.sum(states[0]).as_py()
-
- def state(self) -> List[pa.Scalar]:
- return [pa.scalar(self._sum)]
-
- def evaluate(self) -> pa.Scalar:
- return pa.scalar(self._sum)
-
-sum_above_threshold = udaf(AboveThresholdAccum, [pa.float64()], pa.float64(), [pa.float64()], 'stable')
-
-df_orders.aggregate([col("o_custkey")],[sum_above_threshold(col("o_totalprice")).alias("sales")]).show()
-</code></pre>
-<p>Since we are doing a <code>sum</code> we can keep a single value as our internal state. When we call <code>update()</code>
-we will process a single array and update the internal state, which we share with the <code>state()</code>
-function. For larger batches we may <code>merge()</code> these states. It is important to note that the
-<code>states</code> in the <code>merge()</code> function are an array of the values returned from <code>state()</code>. It is
-entirely possible that the <code>merge</code> function is significantly different than the <code>update</code>, though in
-our example they are very similar.</p>
-<p>One example of implementing a user defined aggregate function where the <code>update()</code> and <code>merge()</code>
-operations are different is computing an average. In <code>update()</code> we would create a state that is both
-a sum and a count. <code>state()</code> would return a list of these two values, and <code>merge()</code> would compute
-the final result.</p>
-<h2 id="user-defined-window-functions">User Defined Window Functions<a class="headerlink" href="#user-defined-window-functions" title="Permanent link">¶</a></h2>
-<p>Writing a user defined window function is slightly more complex than an aggregate function due
-to the variety of ways that window functions are called. I recommend reviewing the
-<a href="https://datafusion.apache.org/python/user-guide/common-operations/udf-and-udfa.html">online documentation</a>
-for a description of which functions need to be implemented. The details of how to implement
-these generally follow the same patterns as described above for aggregate functions.</p>
-<h2 id="performance-comparison">Performance Comparison<a class="headerlink" href="#performance-comparison" title="Permanent link">¶</a></h2>
-<p>For the scalar functions above, we performed a timing evaluation, repeating the operation 100
-times. For this simple example these are our results.</p>
-<pre><code>+-----------------------------+--------------+---------+
-| approach | Average Time | Std Dev |
-+-----------------------------+--------------+---------+
-| python udf | 4.969 | 0.062 |
-| simple filter | 1.075 | 0.022 |
-| explicit filter | 0.685 | 0.063 |
-| pyarrow compute | 0.529 | 0.017 |
-| arrow rust compute | 0.511 | 0.034 |
-| arrow rust compute as class | 0.502 | 0.011 |
-| rust custom iterator | 0.478 | 0.009 |
-+-----------------------------+--------------+---------+
-</code></pre>
-<p>As expected, the conversion to Python objects is by far the worst performance. As soon as we drop
-into using any functions that keep the data entirely on the Native (Rust or C/C++) side we see a
-near 10x speed improvement. Then as we increase our complexity from using PyArrow compute functions
-to implementing the UDF in Rust we see incremental improvements. Our fastest approach - iterating
-through the arrays ourselves does operate nearly 10% faster than the PyArrow compute approach.</p>
-<h2 id="final-thoughts-and-recommendations">Final Thoughts and Recommendations<a class="headerlink" href="#final-thoughts-and-recommendations" title="Permanent link">¶</a></h2>
-<p>For anyone who is curious about <a href="https://datafusion.apache.org/">DataFusion</a> I highly recommend
-giving it a try. This post was designed to make it easier for new users to the Python implementation
-to work with User Defined Functions by giving a few examples of how one might implement these.</p>
-<p>When it comes to designing UDFs, I strongly recommend seeing if you can write your UDF using
-<a href="https://arrow.apache.org/docs/python/api/compute.html">PyArrow functions</a> rather than pure Python
-objects. As shown in the scalar example above, you can achieve a 10x speedup by using PyArrow
-functions. If you must do something that isn't well represented by the PyArrow compute functions,
-then I would consider using a Rust based UDF in the manner shown above.</p>
-<p>I would like to thank <a href="https://github.com/alamb">@alamb</a>, <a href="https://github.com/andygrove">@andygrove</a>, <a href="https://github.com/comphead">@comphead</a>, <a href="https://github.com/emgeee">@emgeee</a>, <a href="https://github.com/kylebarron">@kylebarron</a>, and <a href="https://github.com/Omega359">@Omega359</a>
-for their helpful reviews and feedback.</p>
-<p>Lastly, the Apache Arrow and DataFusion community is an active group of very helpful people working
-to make a great tool. If you want to get involved, please take a look at the
-<a href="https://datafusion.apache.org/python/">online documentation</a> and jump in to help with one of the
-<a href="https://github.com/apache/datafusion-python/issues">open issues</a>.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#personal-context">Personal Context</a></li>
-<li><a href="#user-defined-functions">User-Defined Functions</a><ul>
-<li><a href="#use-case-1-scalar-function">Use Case 1: Scalar Function</a></li>
-<li><a href="#use-case-2-aggregate-function">Use Case 2: Aggregate Function</a></li>
-</ul>
-</li>
-<li><a href="#pure-python-approach">Pure Python approach</a></li>
-<li><a href="#python-approach-using-pyarrow-compute">Python approach using PyArrow compute</a></li>
-<li><a href="#rust-udf-with-python-wrapper">Rust UDF with Python wrapper</a><ul>
-<li><a href="#rust-udf-with-initialization">Rust UDF with initialization</a></li>
-<li><a href="#rust-udf-with-direct-iteration">Rust UDF with direct iteration</a></li>
-</ul>
-</li>
-<li><a href="#user-defined-aggregate-function">User Defined Aggregate Function</a></li>
-<li><a href="#user-defined-window-functions">User Defined Window Functions</a></li>
-<li><a href="#performance-comparison">Performance Comparison</a></li>
-<li><a href="#final-thoughts-and-recommendations">Final Thoughts and Recommendations</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/11/20/datafusion-comet-0.4.0/index.html b/blog/2024/11/20/datafusion-comet-0.4.0/index.html
deleted file mode 100644
index 7383051..0000000
--- a/blog/2024/11/20/datafusion-comet-0.4.0/index.html
+++ /dev/null
@@ -1,222 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.4.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Comet 0.4.0 Release
- </h1>
- <p>Posted on: Wed 20 November 2024 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#performance-stability">Performance & Stability</a><ul>
-<li><a href="#unified-memory-management">Unified Memory Management</a></li>
-<li><a href="#faster-joins">Faster Joins</a></li>
-<li><a href="#bloom-filter-aggregates">Bloom Filter Aggregates</a></li>
-<li><a href="#complex-type-support">Complex Type support</a></li>
-</ul>
-</li>
-</ul>
-</li>
-<li><a href="#roadmap">Roadmap</a><ul>
-<li><a href="#use-datafusions-parquetexec">Use DataFusion’s ParquetExec</a></li>
-<li><a href="#use-arrows-parquet-batch-reader">Use Arrow’s Parquet Batch Reader</a></li>
-</ul>
-</li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>The Apache DataFusion PMC is pleased to announce version 0.4.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately six weeks of development work and is the result of merging 51 PRs from 10
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.4.0.md">change log</a> for more information.</p>
-<h2 id="release-highlights">Release Highlights<a class="headerlink" href="#release-highlights" title="Permanent link">¶</a></h2>
-<h3 id="performance-stability">Performance & Stability<a class="headerlink" href="#performance-stability" title="Permanent link">¶</a></h3>
-<p>There are a number of performance and stability improvements in this release. Here is a summary of some of the
-larger changes. Current benchmarking results can be found in the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>.</p>
-<h4 id="unified-memory-management">Unified Memory Management<a class="headerlink" href="#unified-memory-management" title="Permanent link">¶</a></h4>
-<p>Comet now uses a unified memory management approach that shares an off-heap memory pool with Apache Spark, resulting
-in a much simpler configuration. Comet now requires <code>spark.memory.offHeap.enabled=true</code>. This approach provides a
-holistic view of memory usage in Spark and Comet and makes it easier to optimize system performance.</p>
-<h4 id="faster-joins">Faster Joins<a class="headerlink" href="#faster-joins" title="Permanent link">¶</a></h4>
-<p>Apache Spark supports sort-merge and hash joins, which have similar performance characteristics. Spark defaults to
-using sort-merge joins because they are less likely to result in OutOfMemory exceptions. In vectorized query
-engines such as DataFusion, hash joins outperform sort-merge joins. Comet now has an experimental feature to
-replace Spark sort-merge joins with hash joins for improved performance. This feature is experimental because
-there is currently no spill-to-disk support in the hash join implementation. This feature can be enabled by
-setting <code>spark.comet.exec.replaceSortMergeJoin=true</code>.</p>
-<h4 id="bloom-filter-aggregates">Bloom Filter Aggregates<a class="headerlink" href="#bloom-filter-aggregates" title="Permanent link">¶</a></h4>
-<p>Spark’s optimizer can insert Bloom filter aggregations and filters to prune large result sets before a shuffle. However,
-Comet would fall back to Spark for the aggregation. Comet now has native support for Bloom filter aggregations
-after previously supporting Bloom filter testing. Users no longer need to set
-<code>spark.sql.optimizer.runtime.bloomFilter.enabled=false</code> when using Comet.</p>
-<h4 id="complex-type-support">Complex Type support<a class="headerlink" href="#complex-type-support" title="Permanent link">¶</a></h4>
-<p>This release has the following improvements to complex type support:</p>
-<ul>
-<li>Implemented <code>ArrayAppend</code> and <code>GetArrayStructFields</code>.</li>
-<li>Implemented native cast between structs</li>
-<li>Implemented native cast from structs to string</li>
-</ul>
-<h2 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">¶</a></h2>
-<p>One of the highest priority items on the roadmap is to add support for reading complex types (maps, structs, and arrays)
-from Parquet sources, both when reading Parquet directly and from Iceberg.</p>
-<p>Comet currently has proprietary native code for decoding Parquet pages, native column readers for all of Spark’s
-primitive types, and special handling for Spark-specific use cases such as timestamp rebasing and decimal type
-promotion. This implementation does not yet support complex types. File IO, decryption, and decompression are handled
-in JVM code, and Parquet pages are passed on to native code for decoding.</p>
-<p>Rather than add complex type support to this existing code, we are exploring two main options to allow us to
-leverage more of the upstream Arrow and DataFusion code.</p>
-<h3 id="use-datafusions-parquetexec">Use DataFusion’s ParquetExec<a class="headerlink" href="#use-datafusions-parquetexec" title="Permanent link">¶</a></h3>
-<p>For use cases where DataFusion can support reading a Parquet source, Comet could create a native plan that uses
-DataFusion’s ParquetExec. We are investigating using DataFusion’s SchemaAdapter to handle some Spark-specific
-handling of timestamps and decimals.</p>
-<h3 id="use-arrows-parquet-batch-reader">Use Arrow’s Parquet Batch Reader<a class="headerlink" href="#use-arrows-parquet-batch-reader" title="Permanent link">¶</a></h3>
-<p>For use cases not supported by DataFusion’s ParquetExec, such as integrating with Iceberg, we are exploring
-replacing our current native Parquet decoding logic with the Arrow readers provided by the Parquet crate.</p>
-<p>Iceberg already provides a vectorized Spark reader for Parquet. A <a href="https://github.com/apache/iceberg/pull/9841">PR</a> is open against Iceberg for adding a native
-version based on Comet, and we hope to update this to leverage the improvements outlined above.</p>
-<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#performance-stability">Performance & Stability</a><ul>
-<li><a href="#unified-memory-management">Unified Memory Management</a></li>
-<li><a href="#faster-joins">Faster Joins</a></li>
-<li><a href="#bloom-filter-aggregates">Bloom Filter Aggregates</a></li>
-<li><a href="#complex-type-support">Complex Type support</a></li>
-</ul>
-</li>
-</ul>
-</li>
-<li><a href="#roadmap">Roadmap</a><ul>
-<li><a href="#use-datafusions-parquetexec">Use DataFusion’s ParquetExec</a></li>
-<li><a href="#use-arrows-parquet-batch-reader">Use Arrow’s Parquet Batch Reader</a></li>
-</ul>
-</li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2024/12/14/datafusion-python-43.1.0/index.html b/blog/2024/12/14/datafusion-python-43.1.0/index.html
deleted file mode 100644
index 60d1c32..0000000
--- a/blog/2024/12/14/datafusion-python-43.1.0/index.html
+++ /dev/null
@@ -1,255 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Python 43.1.0 Released - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Python 43.1.0 Released
- </h1>
- <p>Posted on: Sat 14 December 2024 by timsaucer</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#arrow-pycapsule-import-and-export">Arrow PyCapsule import and export</a></li>
-<li><a href="#user-defined-window-functions">User-Defined Window Functions</a></li>
-<li><a href="#foreign-table-providers">Foreign Table Providers</a></li>
-<li><a href="#string-view-performance-enhancements">String View performance enhancements</a></li>
-<li><a href="#appreciation">Appreciation</a></li>
-<li><a href="#get-involved">Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>We are happy to announce that <a href="https://pypi.org/project/datafusion/43.1.0/">datafusion-python 43.1.0</a> has been released. This release
-brings in all of the new features of the core <a href="https://github.com/apache/datafusion/blob/main/dev/changelog/43.0.0.md">DataFusion 43.0.0</a> library. Since the last
-blog post for <a href="https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0/">datafusion-python 40.1.0</a>, a large number of improvements have been made
-that can be found in the <a href="https://github.com/apache/datafusion-python/tree/main/dev/changelog">changelogs</a>.</p>
-<p>We would like to point out four features that are particularly noteworthy.</p>
-<ul>
-<li>Arrow PyCapsule import and export</li>
-<li>User-Defined Window Functions</li>
-<li>Foreign Table Providers</li>
-<li>String View performance enhancements</li>
-</ul>
-<h2 id="arrow-pycapsule-import-and-export">Arrow PyCapsule import and export<a class="headerlink" href="#arrow-pycapsule-import-and-export" title="Permanent link">¶</a></h2>
-<p>Arrow has stable C interface for moving data between different libraries, but difficulties
-sometimes arise when different Python libraries expose this interface through different
-methods, requiring developers to write function calls for each library they are attempting
-to work with. A better approach is to use the <a href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html">Arrow PyCapsule Interface</a> which gives a
-consistent method for exposing these data structures across libraries.</p>
-<p>In <a href="https://github.com/apache/datafusion-python/pull/825">PR #825</a>, we introduced support for both importing and exporting Arrow data in
-<code>datafusion-python</code>. With this improvement, you can now use a single function call to import
-a table from <strong>any</strong> Python library that implements the <a href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html">Arrow PyCapsule Interface</a>.
-Many popular libraries, such as <a href="https://pandas.pydata.org/">Pandas</a> and <a href="https://pola.rs/">Polars</a>
-already support these interfaces.</p>
-<p>Suppose you have a Pandas and Polars DataFrames named <code>df_pandas</code> or <code>df_polars</code>, respectively:</p>
-<pre><code class="language-python">ctx = SessionContext()
-df_dfn1 = ctx.from_arrow(df_pandas)
-df_dfn1.show()
-
-df_dfn2 = ctx.from_arrow(df_polars)
-df_dfn2.show()
-</code></pre>
-<p>One great thing about using this interface is that as any new library is developed and
-uses these stable interfaces, they will work out of the box with DataFusion!</p>
-<p>Additionally, DataFusion DataFrames allow for exporting via the PyCapsule interface. For example,
-to convert a DataFrame to a PyArrow table, it is simply</p>
-<pre><code class="language-python">import pyarrow as pa
-table = pa.table(df)
-</code></pre>
-<h2 id="user-defined-window-functions">User-Defined Window Functions<a class="headerlink" href="#user-defined-window-functions" title="Permanent link">¶</a></h2>
-<p>In <code>datafusion-python 42.0.0</code> we released User-Defined Window Support in <a href="https://github.com/apache/datafusion-python/pull/880">PR #880</a>.
-For a detailed description of how these work please see the online documentation for
-all <a href="https://datafusion.apache.org/python/user-guide/common-operations/udf-and-udfa.html">user-defined functions</a>. Additionally the <a href="https://github.com/apache/datafusion-python/tree/main/examples">examples folder</a> contains a complete
-example demonstrating the four different modes of operation of window functions
-within DataFusion.</p>
-<h2 id="foreign-table-providers">Foreign Table Providers<a class="headerlink" href="#foreign-table-providers" title="Permanent link">¶</a></h2>
-<p>In the core <a href="https://github.com/apache/datafusion/blob/main/dev/changelog/43.0.0.md">DataFusion 43.0.0</a> release, support was added for a Foreign Function
-Interface to table providers. This creates a stable way for sharing functionality
-across different libraries, similar to the <a href="https://arrow.apache.org/docs/format/CDataInterface.html">Arrow C data interface</a> operates. This
-enables libraries, such as <a href="https://delta.io/docs/">delta lake</a> and <a href="https://github.com/datafusion-contrib/datafusion-table-providers">datafusion-contrib</a> to write their own
-table providers in Rust and expose them in Python without requiring a Rust dependency
-on <code>datafusion-python</code>. This is important because it allows these libraries to
-operate with <code>datafusion-python</code> regardless of which version of <code>datafusion</code> they
-were built against.</p>
-<p>To implement this feature in a table provider is quite simple. There is a complete
-example in the <a href="https://github.com/apache/datafusion-python/tree/main/examples">examples folder</a>, but the relevant code is here, exposed as a
-Python function via <a href="https://pyo3.rs/">pyo3</a>:</p>
-<pre><code class="language-rust"> fn __datafusion_table_provider__<'py>(
- &self,
- py: Python<'py>,
- ) -> PyResult<Bound<'py, PyCapsule>> {
- let name = CString::new("datafusion_table_provider").unwrap();
-
- let provider = self
- .create_table()
- .map_err(|e| PyRuntimeError::new_err(e.to_string()))?;
- let provider = FFI_TableProvider::new(Arc::new(provider), false);
-
- PyCapsule::new_bound(py, provider, Some(name.clone()))
- }
-</code></pre>
-<p>That's it! All of the work of converting the table provider to use the FFI interface
-is performed by the core library.</p>
-<h2 id="string-view-performance-enhancements">String View performance enhancements<a class="headerlink" href="#string-view-performance-enhancements" title="Permanent link">¶</a></h2>
-<p>In the core <a href="https://github.com/apache/datafusion/blob/main/dev/changelog/43.0.0.md">DataFusion 43.0.0</a> release, the option to enable StringView by default
-was turned on. This leads to some significant performance enhancements, but it <em>may</em>
-require some changes to users of <code>datafusion-python</code>.</p>
-<p>To learn more about the excellent work on this feature please read <a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1/">part 1</a> and <a href="https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2/">part 2</a>
-of the blog post describing how these enhancements can lead to 20-200% performance
-gains in some tests.</p>
-<p>During our testing we identified some cases where we needed to adjust workflows to
-account for the fact that StringView is now the default type for string based operations.
-First, when performing manipulations on string objects there is a performance loss when
-needing to cast from string to string view or vice versa. To reap the best performance,
-ideally all of your string type data will use StringView. For most users this should be
-transparent. However if you specify a schema for reading or creating data, then you
-likely need to change from <code>pa.string()</code> to <code>pa.string_view()</code>. For our testing, this
-primarily happens during data loading operations and in unit tests.</p>
-<p>If you wish to disable StringView as the default type to retain the old approach,
-you can do so following this example:</p>
-<pre><code class="language-python">from datafusion import SessionContext
-from datafusion import SessionConfig
-config = SessionConfig({"datafusion.execution.parquet.schema_force_view_types": "false"})
-ctx = SessionContext(config=config)
-</code></pre>
-<h2 id="appreciation">Appreciation<a class="headerlink" href="#appreciation" title="Permanent link">¶</a></h2>
-<p>We would like to thank everyone who has helped with these releases through their helpful
-conversations, code review, issue descriptions, and code authoring. We would especially
-like to thank the following authors of PRs who made these releases possible, listed in
-alphabetical order by username: <a href="https://github.com/andygrove">@andygrove</a>, <a href="https://github.com/drauschenbach">@drauschenbach</a>, <a href="https://github.com/emgeee">@emgeee</a>, <a href="https://github.com/ion-elgreco">@ion-elgreco</a>,
-<a href="https://github.com/jcrist">@jcrist</a>, <a href="https://github.com/kosiew">@kosiew</a>, <a href="https://github.com/mesejo">@mesejo</a>, <a href="https://github.com/Michael-J-Ward">@Michael-J-Ward</a>, and <a href="https://github.com/sir-sigurd">@sir-sigurd</a>.</p>
-<p>Thank you!</p>
-<h2 id="get-involved">Get Involved<a class="headerlink" href="#get-involved" title="Permanent link">¶</a></h2>
-<p>The DataFusion Python team is an active and engaging community and we would love
-to have you join us and help the project.</p>
-<p>Here are some ways to get involved:</p>
-<ul>
-<li>
-<p>Learn more by visiting the <a href="https://datafusion.apache.org/python/index.html">DataFusion Python project</a>
-page.</p>
-</li>
-<li>
-<p>Try out the project and provide feedback, file issues, and contribute code.</p>
-</li>
-</ul>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#arrow-pycapsule-import-and-export">Arrow PyCapsule import and export</a></li>
-<li><a href="#user-defined-window-functions">User-Defined Window Functions</a></li>
-<li><a href="#foreign-table-providers">Foreign Table Providers</a></li>
-<li><a href="#string-view-performance-enhancements">String View performance enhancements</a></li>
-<li><a href="#appreciation">Appreciation</a></li>
-<li><a href="#get-involved">Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2025/01/17/datafusion-comet-0.5.0/index.html b/blog/2025/01/17/datafusion-comet-0.5.0/index.html
deleted file mode 100644
index 0b66f05..0000000
--- a/blog/2025/01/17/datafusion-comet-0.5.0/index.html
+++ /dev/null
@@ -1,205 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.5.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Comet 0.5.0 Release
- </h1>
- <p>Posted on: Fri 17 January 2025 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#performance">Performance</a></li>
-<li><a href="#shuffle-improvements">Shuffle Improvements</a></li>
-<li><a href="#memory-management">Memory Management</a></li>
-<li><a href="#spark-sql-metrics">Spark SQL Metrics</a></li>
-<li><a href="#crate-reorganization">Crate Reorganization</a></li>
-</ul>
-</li>
-<li><a href="#update-on-complex-type-support">Update on Complex Type Support</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>The Apache DataFusion PMC is pleased to announce version 0.5.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately 8 weeks of development work and is the result of merging 69 PRs from 15
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.5.0.md">change log</a> for more information.</p>
-<h2 id="release-highlights">Release Highlights<a class="headerlink" href="#release-highlights" title="Permanent link">¶</a></h2>
-<h3 id="performance">Performance<a class="headerlink" href="#performance" title="Permanent link">¶</a></h3>
-<p>Comet 0.5.0 achieves a 1.9x speedup for single-node TPC-H @ 100 GB, an improvement from 1.7x in the previous release.</p>
-<p><img alt="Chart showing TPC-H benchmark results for Comet 0.5.0" class="img-fluid" src="/blog/images/comet-0.5.0/tpch_allqueries.png" width="100%"/></p>
-<p><img alt="Chart showing TPC-H benchmark results for Comet 0.5.0" class="img-fluid" src="/blog/images/comet-0.5.0/tpch_queries_compare.png" width="100%"/></p>
-<p>More benchmarking results can be found in the <a href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet Benchmarking Guide</a>.</p>
-<h3 id="shuffle-improvements">Shuffle Improvements<a class="headerlink" href="#shuffle-improvements" title="Permanent link">¶</a></h3>
-<p>Comet now supports multiple compression algorithms for compressing shuffle files. Previously, only ZSTD was supported
-but Comet now also supports LZ4 and Snappy. The default is now LZ4, which matches the default in Spark. ZSTD may be
-a better choice when the compression ratio is more important than CPU overhead.</p>
-<p>Previously, Comet used Arrow IPC to encode record batches into shuffle files. Although Arrow IPC is a good
-general-purpose framework for serializing Arrow record batches, we found that we could get better performance using
-a custom serialization approach optimized for Comet. One optimization is that the schema is encoded once per shuffle
-operation rather than once per batch. There are some planned performance improvements in the Rust implementation of
-Arrow IPC and Comet may switch back to Arrow IPC in the future.</p>
-<p>Comet provides two shuffle implementations. Comet native shuffle is the fastest and performs repartitioning in
-native code. Comet columnar shuffle delegates to Spark to perform repartitioning and is used in cases where native
-shuffle is not supported, such as with <code>RangePartitioning</code>. Comet generally tries to use native shuffle first, then
-columnar shuffle, and finally falls back to Spark if neither is supported. There was a bug in previous releases
-where Comet would sometimes fall back to Spark shuffle if native shuffle was not supported and missed opportunities
-to use columnar shuffle. This bug was fixed in this release but currently requires the configuration setting
-<code>spark.comet.exec.shuffle.fallbackToColumnar=true</code>. This will be enabled by default in the next release.</p>
-<h3 id="memory-management">Memory Management<a class="headerlink" href="#memory-management" title="Permanent link">¶</a></h3>
-<p>Comet 0.4.0 required Spark to be configured to use off-heap memory. In this release it is no longer required and
-there are multiple options for configuring Comet to use on-heap memory instead. More details are available in the
-<a href="https://datafusion.apache.org/comet/user-guide/tuning.html">Comet Tuning Guide</a>.</p>
-<h3 id="spark-sql-metrics">Spark SQL Metrics<a class="headerlink" href="#spark-sql-metrics" title="Permanent link">¶</a></h3>
-<p>Comet now provides detailed metrics for native shuffle, showing time for repartitioning, encoding and compressing,
-and writing to disk.</p>
-<h3 id="crate-reorganization">Crate Reorganization<a class="headerlink" href="#crate-reorganization" title="Permanent link">¶</a></h3>
-<p>One of the goals of the Comet project is to make Spark-compatible functionality available to other projects that
-are based on DataFusion. In this release, many implementations of Spark-compatible expressions were moved from the
-unpublished <code>datafusion-comet</code> crate, which provides the native part of the Spark plugin, into the
-<code>datafusion-comet-spark-expr</code> crate. There is also ongoing work to reorganize this crate to move expressions into
-subfolders named after the group name that Spark uses to organize expressions. For example, there are now subfolders
-named <code>agg_funcs</code>, <code>datetime_funcs</code>, <code>hash_funcs</code>, and so on.</p>
-<h2 id="update-on-complex-type-support">Update on Complex Type Support<a class="headerlink" href="#update-on-complex-type-support" title="Permanent link">¶</a></h2>
-<p>Good progress has been made with proof-of-concept work using DataFusion’s <code>ParquetExec</code>, which has the advantage of
-supporting complex types. This work is available on the <code>comet-parquet-exec</code> branch, and the current focus is on
-fixing test regressions, particularly regarding timestamp conversion issues.</p>
-<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#performance">Performance</a></li>
-<li><a href="#shuffle-improvements">Shuffle Improvements</a></li>
-<li><a href="#memory-management">Memory Management</a></li>
-<li><a href="#spark-sql-metrics">Spark SQL Metrics</a></li>
-<li><a href="#crate-reorganization">Crate Reorganization</a></li>
-</ul>
-</li>
-<li><a href="#update-on-complex-type-support">Update on Complex Type Support</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2025/02/02/datafusion-ballista-43.0.0/index.html b/blog/2025/02/02/datafusion-ballista-43.0.0/index.html
deleted file mode 100644
index bb2db7c..0000000
--- a/blog/2025/02/02/datafusion-ballista-43.0.0/index.html
+++ /dev/null
@@ -1,198 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Ballista 43.0.0 Released - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Ballista 43.0.0 Released
- </h1>
- <p>Posted on: Sun 02 February 2025 by milenkovicm</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#highlights-of-this-release">Highlights of this release</a><ul>
-<li><a href="#seamless-integration-with-datafusion">Seamless Integration with DataFusion</a></li>
-<li><a href="#removal-of-experimental-features">Removal of Experimental Features</a></li>
-<li><a href="#performance-scalability">Performance & Scalability</a></li>
-<li><a href="#new-logo">New Logo</a></li>
-</ul>
-</li>
-<li><a href="#roadmap">Roadmap</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>We are pleased to announce version <a href="https://github.com/apache/datafusion-ballista/blob/main/CHANGELOG.md#4300-2025-01-07">43.0.0</a> of the <a href="https://datafusion.apache.org/ballista/">DataFusion Ballista</a>. Ballista allows existing <a href="https://datafusion.apache.org">DataFusion</a> applications to be scaled out on a cluster for use cases that are not practical to run on a single node.</p>
-<h2 id="highlights-of-this-release">Highlights of this release<a class="headerlink" href="#highlights-of-this-release" title="Permanent link">¶</a></h2>
-<h3 id="seamless-integration-with-datafusion">Seamless Integration with DataFusion<a class="headerlink" href="#seamless-integration-with-datafusion" title="Permanent link">¶</a></h3>
-<p>The primary objective of this release has been to achieve a more seamless integration with the DataFusion ecosystem and try to achieve the same level of flexibility as DataFusion.</p>
-<p>In recent months, our development efforts have been directed toward providing a robust and extensible Ballista API. This new API empowers end-users to tailor Ballista's core functionality to their specific use cases. As a result, we have deprecated several experimental features from the Ballista core, allowing users to reintroduce them as custom extensions outside the core framework. This shift reduces the maintenance burden on Ballista's core maintainers and paves the way for optiona [...]
-<p>The most significant enhancement in this release is the deprecation of <code>BallistaContext</code>, which has been superseded by the DataFusion <code>SessionContext</code>. This change enables DataFusion applications written in Rust to execute on a Ballista cluster with minimal modifications. Beyond simplifying migration and reducing maintenance overhead, this update introduces distributed write functionality to Ballista for the first time, significantly enhancing its capabilities.</p>
-<pre><code class="language-rust">use ballista::prelude::*;
-use datafusion::prelude::*;
-
-#[tokio::main]
-async fn main() -> datafusion::error::Result<()> {
-
- // Instead of creating classic SessionContext
- // let ctx = SessionContext::new();
-
- // create DataFusion SessionContext with ballista standalone cluster started
- // let ctx = SessionContext::standalone().await;
-
- // create DataFusion SessionContext with ballista remote cluster started
- let ctx = SessionContext::remote("df://localhost:50050").await;
-
- // register the table
- ctx.register_csv("example", "tests/data/example.csv", CsvReadOptions::new()).await?;
-
- // create a plan to run a SQL query
- let df = ctx.sql("SELECT a, MIN(b) FROM example WHERE a <= b GROUP BY a LIMIT 100").await?;
-
- // execute and print results
- df.show().await?;
- Ok(())
-}
-</code></pre>
-<p>Additionally, Ballista’s versioning scheme has been aligned with that of DataFusion, ensuring that Ballista's version number reflects the compatible DataFusion version.</p>
-<p>At the moment there is a gap between DataFusion and Ballista, which we will try to bridge in the future.</p>
-<h3 id="removal-of-experimental-features">Removal of Experimental Features<a class="headerlink" href="#removal-of-experimental-features" title="Permanent link">¶</a></h3>
-<p>Ballista had grown in scope to include several experimental features in various states of completeness. Some features have been removed from this release in an effort to strip Ballista back to its core and make it easier to maintain and extend.</p>
-<p>Specifically, the caching subsystem, predefined object store registry, plugin subsystem, key-value stores for persistent scheduler state, and the UI have been removed.</p>
-<h3 id="performance-scalability">Performance & Scalability<a class="headerlink" href="#performance-scalability" title="Permanent link">¶</a></h3>
-<p>Ballista has significantly leveraged the advancements made in the DataFusion project over the past year. Benchmark results demonstrate notable improvements in performance, highlighting the impact of these enhancements:</p>
-<p>Per query comparison:</p>
-<p><img alt="Per query comparison" class="img-fluid" src="/blog/images/datafusion-ballista-43.0.0/tpch_queries_compare.png" width="100%"/></p>
-<p>Relative speedup:</p>
-<p><img alt="Relative speedup graph" class="img-fluid" src="/blog/images/datafusion-ballista-43.0.0/tpch_queries_speedup_rel.png" width="100%"/></p>
-<p>The overall speedup is 2.9x</p>
-<p><img alt="Overall speedup" class="img-fluid" src="/blog/images/datafusion-ballista-43.0.0/tpch_allqueries.png" width="50%"/></p>
-<h3 id="new-logo">New Logo<a class="headerlink" href="#new-logo" title="Permanent link">¶</a></h3>
-<p>Ballista now has a new logo, which is visually similar to other DataFusion projects. </p>
-<p><img alt="New logo" class="img-fluid" src="/blog/images/datafusion-ballista-43.0.0/ballista-logo.png" width="50%"/></p>
-<h2 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">¶</a></h2>
-<p>Moving forward, Ballista will adopt the same release cadence as DataFusion, providing synchronized updates across the ecosystem.
-Currently, there is no established long-term roadmap for Ballista. A plan will be formulated in the coming months based on community feedback and the availability of additional maintainers.</p>
-<p>In the short term, development efforts will concentrate on closing the feature gap between DataFusion and Ballista. Key priorities include implementing support for <code>INSERT INTO</code>, enabling table <code>URL</code> functionality, and achieving deeper integration with the Python ecosystem.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#highlights-of-this-release">Highlights of this release</a><ul>
-<li><a href="#seamless-integration-with-datafusion">Seamless Integration with DataFusion</a></li>
-<li><a href="#removal-of-experimental-features">Removal of Experimental Features</a></li>
-<li><a href="#performance-scalability">Performance & Scalability</a></li>
-<li><a href="#new-logo">New Logo</a></li>
-</ul>
-</li>
-<li><a href="#roadmap">Roadmap</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2025/02/17/datafusion-comet-0.6.0/index.html b/blog/2025/02/17/datafusion-comet-0.6.0/index.html
deleted file mode 100644
index 479cb66..0000000
--- a/blog/2025/02/17/datafusion-comet-0.6.0/index.html
+++ /dev/null
@@ -1,186 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.6.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Comet 0.6.0 Release
- </h1>
- <p>Posted on: Mon 17 February 2025 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#datafusion-upgrade">DataFusion Upgrade</a></li>
-<li><a href="#new-features">New Features</a></li>
-<li><a href="#performance-stability">Performance & Stability</a></li>
-</ul>
-</li>
-<li><a href="#bug-fixes">Bug Fixes</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>The Apache DataFusion PMC is pleased to announce version 0.6.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately four weeks of development work and is the result of merging 39 PRs from 12
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.6.0.md">change log</a> for more information.</p>
-<p>Starting with this release, we now plan on releasing new versions of Comet more frequently, typically within 1-2 weeks
-of each major DataFusion release. The main motivation for this change is to better support downstream Rust projects
-that depend on the <a href="https://docs.rs/datafusion-comet-spark-expr/latest/datafusion_comet_spark_expr/">datafusion_comet_spark_expr</a> crate.</p>
-<h2 id="release-highlights">Release Highlights<a class="headerlink" href="#release-highlights" title="Permanent link">¶</a></h2>
-<h3 id="datafusion-upgrade">DataFusion Upgrade<a class="headerlink" href="#datafusion-upgrade" title="Permanent link">¶</a></h3>
-<ul>
-<li>Comet 0.6.0 uses DataFusion 45.0.0</li>
-</ul>
-<h3 id="new-features">New Features<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h3>
-<ul>
-<li>Comet now supports <code>array_join</code>, <code>array_intersect</code>, and <code>arrays_overlap</code>. Note that these expressions are not
- yet guaranteed to be 100% compatible with Spark for all input data types, so these expressions are only enabled
- with the configuration setting <code>spark.comet.expression.allowIncompatible=true</code>. </li>
-</ul>
-<h3 id="performance-stability">Performance & Stability<a class="headerlink" href="#performance-stability" title="Permanent link">¶</a></h3>
-<ul>
-<li>Metrics from native execution are now updated in Spark every 3 seconds by default, rather than for each
- batch being processed. The mechanism for passing the metrics via JNI is also more efficient.</li>
-<li>New memory pool options "fair unified" and "unbounded" have been added. See the <a href="https://datafusion.apache.org/comet/user-guide/tuning.html">Comet Tuning Guide</a> for more information.</li>
-</ul>
-<h2 id="bug-fixes">Bug Fixes<a class="headerlink" href="#bug-fixes" title="Permanent link">¶</a></h2>
-<ul>
-<li>Hashing of decimal values with precision <= 18 is now compatible with Spark</li>
-<li>Comet falls back to Spark when hashing decimals with precision > 18</li>
-</ul>
-<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#datafusion-upgrade">DataFusion Upgrade</a></li>
-<li><a href="#new-features">New Features</a></li>
-<li><a href="#performance-stability">Performance & Stability</a></li>
-</ul>
-</li>
-<li><a href="#bug-fixes">Bug Fixes</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2025/02/20/datafusion-45.0.0/index.html b/blog/2025/02/20/datafusion-45.0.0/index.html
deleted file mode 100644
index a4bc0e2..0000000
--- a/blog/2025/02/20/datafusion-45.0.0/index.html
+++ /dev/null
@@ -1,306 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion 45.0.0 Released - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion 45.0.0 Released
- </h1>
- <p>Posted on: Thu 20 February 2025 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#community-growth">Community Growth 📈</a></li>
-<li><a href="#improved-performance">Improved Performance 🚀</a></li>
-<li><a href="#improved-quality">Improved Quality 📋</a></li>
-<li><a href="#improved-documentation">Improved Documentation 📚</a></li>
-<li><a href="#new-features">New Features ✨</a><ul>
-<li><a href="#functions">Functions</a></li>
-<li><a href="#ffi">FFI</a></li>
-<li><a href="#materialized-views">Materialized Views</a></li>
-<li><a href="#substrait">Substrait</a></li>
-</ul>
-</li>
-<li><a href="#looking-ahead-the-next-six-months">Looking Ahead: The Next Six Months 🔭</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
-<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
-<p>We are very proud to announce <a href="https://crates.io/crates/datafusion/45.0.0">DataFusion 45.0.0</a>. This blog highlights some of the
-many major improvements since we released <a href="https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/">DataFusion 40.0.0</a> and a preview of
-what the community is thinking about in the next 6 months. It has been an exciting
-period of development for DataFusion!</p>
-<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> is an extensible query engine, written in <a href="https://www.rust-lang.org/">Rust</a>, that
-uses <a href="https://arrow.apache.org">Apache Arrow</a> as its in-memory format. DataFusion is used by developers to
-create new, fast data centric systems such as databases, dataframe libraries,
-machine learning and streaming applications. While <a href="https://datafusion.apache.org/user-guide/introduction.html#project-goals">DataFusion’s primary design
-goal</a> is to accelerate the creation of other data centric systems, it has a
-reasonable experience directly out of the box as a <a href="https://datafusion.apache.org/user-guide/dataframe.html">dataframe library</a>,
-<a href="https://datafusion.apache.org/python/">python library</a> and <a href="https://datafusion.apache.org/user-guide/cli/">command line SQL tool</a>.</p>
-<p>DataFusion's core thesis is that as a community, together we can build much more
-advanced technology than any of us as individuals or companies could do alone.
-Without DataFusion, highly performant vectorized query engines would remain
-the domain of a few large companies and world-class research institutions.
-With DataFusion, we can all build on top of a shared foundation, and focus on
-what makes our projects unique.</p>
-<h2 id="community-growth">Community Growth 📈<a class="headerlink" href="#community-growth" title="Permanent link">¶</a></h2>
-<p>In the last 6 months, between <code>40.0.0</code> and <code>45.0.0</code>, our community continues to
-grow in new and exciting ways.</p>
-<ol>
-<li>We added several PMC members and new committers: <a href="https://github.com/jayzhan211">@jayzhan211</a> and <a href="https://github.com/jonahgao">@jonahgao</a> joined the PMC,
- <a href="https://github.com/2010YOUY01">@2010YOUY01</a>, <a href="https://github.com/rachelint">@rachelint</a>, <a href="https://github.com/findepi/">@findpi</a>, <a href="https://github.com/iffyio">@iffyio</a>, <a href="https://github.com/goldmedal">@goldmedal</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/Michael-J-Ward">@Michael-J-Ward</a> and <a href="https://github.com/korowa">@korowa</a>
- joined as committers. See the <a href="https://lists.apache.org/list.html?dev@datafusion.apache.org">mailing list</a> for more details.</li>
-<li>In the <a href="https://github.com/apache/arrow-datafusion">core DataFusion repo</a> alone we reviewed and accepted almost 1600 PRs from 206 different
- committers, created over 1100 issues and closed 751 of them 🚀. All changes are listed in the detailed
- <a href="https://github.com/apache/datafusion/tree/main/dev/changelog">changelogs</a>.</li>
-<li>DataFusion focused meetups happened in multiple cities around the world: <a href="https://github.com/apache/datafusion/discussions/10341#discussioncomment-10110273">Hangzhou</a>, <a href="https://github.com/apache/datafusion/discussions/11431">Belgrade</a>, <a href="https://github.com/apache/datafusion/discussions/11213">New York</a>,
- <a href="https://github.com/apache/datafusion/discussions/10348">Seattle</a>, <a href="https://github.com/apache/datafusion/discussions/12894">Chicago</a>, <a href="https://github.com/apache/datafusion/discussions/13165">Boston</a> and <a href="https://github.com/apache/datafusion/discussions/12988">Amsterdam</a> as well as a Rust NYC meetup in NYC focused on DataFusion.</li>
-</ol>
-<!--
-$ git log --pretty=oneline 40.0.0..45.0.0 . | wc -l
- 1532 (up from 1453)
-
-$ git shortlog -sn 40.0.0..45.0.0 . | wc -l
- 206 (up from 182)
-
-https://crates.io/crates/datafusion/45.0.0
-DataFusion 45 released Feb 7, 2025
-
-https://crates.io/crates/datafusion/40.0.0
-DataFusion 40 released July 12, 2024
-
-Issues created in this time: 375 open, 751 closed (from 321 open, 781 closed)
-https://github.com/apache/datafusion/issues?q=is%3Aissue+created%3A2024-07-12..2025-02-07
-
-Issues closed: 956 (up from 911)
-https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2024-07-12..2025-02-07
-
-PRs merged in this time 1597 (up from 1490)
-https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2024-07-12..2025-02-07
-
--->
-<p>DataFusion has put in an application to be part of <a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> with a
-<a href="https://github.com/apache/datafusion/issues/14478">number of ideas</a> for projects with mentors already selected. Additionally, <a href="https://github.com/apache/datafusion/issues/14373">some ideas</a> on
-how to make DataFusion an ideal selection for university database projects such as the
-<a href="https://15445.courses.cs.cmu.edu/spring2025/">CMU database classes</a> have been put forward.</p>
-<p>In addition, DataFusion has been appearing publicly more and more, both online and offline. Here are some highlights:</p>
-<ol>
-<li>A <a href="https://uwheel.rs/post/datafusion_uwheel/">demonstration of how uwheel</a> is integrated into DataFusion</li>
-<li>Integrating StringView into DataFusion - <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/">part 1</a> and <a href="https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/">part 2</a></li>
-<li><a href="https://techontherocks.show/3">Building streams</a> with DataFusion</li>
-<li><a href="https://blog.haoxp.xyz/posts/caching-datafusion">Caching in DataFusion</a>: Don't read twice</li>
-<li><a href="https://blog.haoxp.xyz/posts/parquet-to-arrow/">Parquet pruning in DataFusion</a>: Read no more than you need</li>
-<li>DataFusion is one of <a href="https://www.crn.com/news/software/2024/the-10-coolest-open-source-software-tools-of-2024?page=3">The 10 coolest open source software tools</a></li>
-<li><a href="https://www.denormalized.io/blog/building-databases">Building databases over a weekend</a></li>
-</ol>
-<h2 id="improved-performance">Improved Performance 🚀<a class="headerlink" href="#improved-performance" title="Permanent link">¶</a></h2>
-<p>DataFusion hit a milestone in its development by becoming <a href="https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/">the fastest single node engine</a>
-for querying Apache Parquet files in <a href="https://benchmark.clickhouse.com/">clickbench</a> benchmark for the 43.0.0 release. A <a href="https://github.com/apache/datafusion/issues/12821">lot
-of work</a> went into making this happen! While other engines have subsequently gotten faster,
-displacing DataFusion from the top spot, DataFusion still remains near the top and we <a href="https://github.com/apache/datafusion/issues/14586">are planning
-more improvements</a>.</p>
-<p><img alt="ClickBench performance results over time for DataFusion" class="img-fluid" src="/blog/images/datafusion-45.0.0/performance_over_time.png" width="100%"/></p>
-<p><strong>Figure 1</strong>: ClickBench performance improved over 33% between DataFusion 33
-(released Nov. 2023) and DataFusion 45 (released Feb. 2025). </p>
-<p>The task of <a href="https://github.com/apache/datafusion/issues/10918">integrating</a> the new <a href="https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html">Arrow StringView</a> which significantly improves performance
-for workloads that scan, filter and group by variable length string and binary data was completed
-and enabled by default in the past 6 months. The improvement is especially pronounced for Parquet
-files due to <a href="https://github.com/apache/arrow-rs/issues/5530">upstream work in the parquet reader</a>. Kudos to <a href="https://github.com/XiangpengHong">@XiangpengHong</a>, <a href="https://github.com/AriesDevil">@AriesDevil</a>,
-<a href="https://github.com/PsiACE">@PsiACE</a>, <a href="https://github.com/Weijun-H">@Weijun-H</a>, <a href="https://github.com/a10y">@a10y</a>, and <a href="https://github.com/RinChanNOWWW">@RinChanNOWWW</a> for driving this project.</p>
-<h2 id="improved-quality">Improved Quality 📋<a class="headerlink" href="#improved-quality" title="Permanent link">¶</a></h2>
-<p>DataFusion continues to improve overall in quality. In addition to ongoing bug
-fixes, one of the most exciting improvements in the last 6 months was the addition of the
-<a href="https://github.com/apache/datafusion/pull/13936">SQLite sqllogictest suite</a> thanks to <a href="https://github.com/Omega359">@Omega359</a>. These tests run over 5 million
-sql statements on every push to the main branch.</p>
-<p>Support for <a href="https://github.com/apache/datafusion/pull/13651">explicitly checking logical plan invariants</a> was added by <a href="https://github.com/wiedld">@wiedld</a> which
-can help catch implicit changes that might cause problems during upgrades.</p>
-<p>We have also started other quality initiatives to make it <a href="https://github.com/apache/datafusion/issues/13525">easier to use DataFusion</a>
-based on <a href="https://glaredb.com/">GlareDB</a>'s experience along with more <a href="https://github.com/apache/datafusion/issues/13661">extensive prerelease testing</a>. </p>
-<h2 id="improved-documentation">Improved Documentation 📚<a class="headerlink" href="#improved-documentation" title="Permanent link">¶</a></h2>
-<p>We continue to improve the documentation to make it easier to get started using DataFusion.
-During the last 6 months two projects were initiated to migrate the function documentation
-from strictly static markdown files. First, <a href="https://github.com/apache/datafusion/pull/12668">@Omega359</a> to allow function
-documentation to be generated from code and <a href="https://github.com/jonathanc-n">@jonathanc-n</a> and others helped with the migration,
-then <a href="https://github.com/comphead">@comphead</a> lead a project to <a href="https://github.com/apache/datafusion/pull/12822">create a doc macro</a> to allow for an even easier way to write
-function documentation. A special thanks to <a href="https://github.com/Chen-Yuan-Lai">@Chen-Yuan-Lai</a> for migrating many functions to
-the new syntax.</p>
-<p>Additionally, the <a href="https://github.com/apache/datafusion/pull/13877">examples</a> were <a href="https://github.com/apache/datafusion/pull/13905">refactored</a> and <a href="https://github.com/apache/datafusion/pull/13950">cleaned up</a> to improve their usefulness.</p>
-<h2 id="new-features">New Features ✨<a class="headerlink" href="#new-features" title="Permanent link">¶</a></h2>
-<p>There are too many new features in the last 6 months to list them all, but here
-are some highlights:</p>
-<h3 id="functions">Functions<a class="headerlink" href="#functions" title="Permanent link">¶</a></h3>
-<ul>
-<li>Uniform Window Functions: <code>BuiltInWindowFunctions</code> was removed and all now use UDFs (<a href="https://github.com/jcsherin">@jcsherin</a>)</li>
-<li>Uniform Aggregate Functions: <code>BuiltInAggregateFunctions</code> was removed and all now use UDFs</li>
-<li>As mentioned above function documentation was extracted from the markdown files</li>
-<li>Some new functions and sql support were added including '<a href="https://github.com/apache/datafusion/pull/13799">show functions</a>', '<a href="https://github.com/apache/datafusion/pull/11347">to_local_time</a>',
- '<a href="https://github.com/apache/datafusion/pull/12970">regexp_count</a>', '<a href="https://github.com/apache/datafusion/pull/11969">map_extract</a>', '<a href="https://github.com/apache/datafusion/pull/12211">array_distance</a>', '<a href="https://github.com/apache/datafusion/pull/12329">array_any_value</a>', '<a href="https://github.com/apache/datafusion/pull/12474">greatest</a>',
- '<a href="https://github.com/apache/datafusion/pull/13786">least</a>', '<a href="https://github.com/apache/datafusion/pull/14217">arrays_overlap</a>'</li>
-</ul>
-<h3 id="ffi">FFI<a class="headerlink" href="#ffi" title="Permanent link">¶</a></h3>
-<ul>
-<li>Foreign Function Interface work has started. This should allow for
- <a href="https://github.com/apache/datafusion/pull/12920">using table providers</a> across languages and versions of DataFusion. This
- is especially pertinent for integration with <a href="https://delta-io.github.io/delta-rs/">delta-rs</a> and other table formats.</li>
-</ul>
-<h3 id="materialized-views">Materialized Views<a class="headerlink" href="#materialized-views" title="Permanent link">¶</a></h3>
-<ul>
-<li><a href="https://github.com/suremarc">@suremarc</a> has added a <a href="https://github.com/datafusion-contrib/datafusion-materialized-views">materialized view implementation</a> in datafusion-contrib 🚀</li>
-</ul>
-<h3 id="substrait">Substrait<a class="headerlink" href="#substrait" title="Permanent link">¶</a></h3>
-<ul>
-<li>A lot of work was put into improving and enhancing substrait support (<a href="https://github.com/Blizzara">@Blizzara</a>, <a href="https://github.com/westonpace">@westonpace</a>, <a href="https://github.com/tokoko">@tokoko</a>, <a href="https://github.com/vbarua">@vbarua</a>, <a href="https://github.com/LatrecheYasser">@LatrecheYasser</a>, <a href="https://github.com/notfilippo">@notfilippo</a> and others)</li>
-</ul>
-<h2 id="looking-ahead-the-next-six-months">Looking Ahead: The Next Six Months 🔭<a class="headerlink" href="#looking-ahead-the-next-six-months" title="Permanent link">¶</a></h2>
-<p>One of the long term goals of <a href="https://github.com/alamb">@alamb</a>, DataFusion's PMC chair, has been to have
-<a href="https://www.influxdata.com/blog/datafusion-2025-influxdb/">1000 DataFusion based projects</a>. This may be the year that happens!</p>
-<p>The community has been <a href="https://github.com/apache/datafusion/issues/14580">discussing what we will work on in the next six months</a>.
-Some major initiatives are likely to be:</p>
-<ol>
-<li><em>Performance</em>: A <a href="https://github.com/apache/datafusion/issues/14482">number of items have been identified</a> as areas that could use additional work</li>
-<li><em>Memory usage</em>: Tracking and improving memory usage, statistics and spilling to disk </li>
-<li><em><a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> (GSOC)</em>: DataFusion is hopefully selected as a project and we start accepting and supporting student projects </li>
-<li><em>FFI</em>: Extending the FFI implementation to support to all types of UDF's and SessionContext</li>
-<li><em>Spark Functions</em>: A <a href="https://github.com/apache/datafusion/issues/5600">proposal has been made to add a crate</a> covering spark compatible builtin functions </li>
-</ol>
-<h2 id="how-to-get-involved">How to Get Involved<a class="headerlink" href="#how-to-get-involved" title="Permanent link">¶</a></h2>
-<p>DataFusion is not a project built or driven by a single person, company, or
-foundation. Rather, our community of users and contributors work together to
-build a shared technology that none of us could have built alone.</p>
-<p>If you are interested in joining us we would love to have you. You can try out
-DataFusion on some of your own data and projects and let us know how it goes,
-contribute suggestions, documentation, bug reports, or a PR with documentation,
-tests or code. A list of open issues suitable for beginners is <a href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">here</a> and you
-can find how to reach us on the <a href="https://datafusion.apache.org/contributor-guide/communication.html">communication doc</a>.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#community-growth">Community Growth 📈</a></li>
-<li><a href="#improved-performance">Improved Performance 🚀</a></li>
-<li><a href="#improved-quality">Improved Quality 📋</a></li>
-<li><a href="#improved-documentation">Improved Documentation 📚</a></li>
-<li><a href="#new-features">New Features ✨</a><ul>
-<li><a href="#functions">Functions</a></li>
-<li><a href="#ffi">FFI</a></li>
-<li><a href="#materialized-views">Materialized Views</a></li>
-<li><a href="#substrait">Substrait</a></li>
-</ul>
-</li>
-<li><a href="#looking-ahead-the-next-six-months">Looking Ahead: The Next Six Months 🔭</a></li>
-<li><a href="#how-to-get-involved">How to Get Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2025/03/11/ordering-analysis/index.html b/blog/2025/03/11/ordering-analysis/index.html
deleted file mode 100644
index 92d8acc..0000000
--- a/blog/2025/03/11/ordering-analysis/index.html
+++ /dev/null
@@ -1,526 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Using Ordering for Better Plans in Apache DataFusion - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Using Ordering for Better Plans in Apache DataFusion
- </h1>
- <p>Posted on: Tue 11 March 2025 by Mustafa Akur, Andrew Lamb</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a><ul>
-<li><a href="#removing-unnecessary-sorts">Removing Unnecessary Sorts</a></li>
-<li><a href="#optimized-operator-implementations">Optimized Operator Implementations</a></li>
-<li><a href="#streaming-friendly-execution">Streaming-Friendly Execution</a></li>
-</ul>
-</li>
-<li><a href="#analysis">Analysis</a><ul>
-<li><a href="#example-virtual-table">Example Virtual Table</a></li>
-</ul>
-</li>
-<li><a href="#key-concepts-for-analyzing-orderings">Key Concepts for Analyzing Orderings</a><ul>
-<li><a href="#1-constant-expressions">1. Constant Expressions</a></li>
-<li><a href="#2-equivalent-expression-groups">2. Equivalent Expression Groups</a></li>
-<li><a href="#3-succinct-encoding-of-valid-orderings">3. Succinct Encoding of Valid Orderings</a></li>
-<li><a href="#table-properties">Table Properties</a></li>
-<li><a href="#algorithm-for-analyzing-ordering-requirements">Algorithm for Analyzing Ordering Requirements</a></li>
-<li><a href="#example-walkthrough">Example Walkthrough</a></li>
-<li><a href="#algorithm-steps">Algorithm Steps</a></li>
-</ul>
-</li>
-<li><a href="#conclusion">Conclusion</a></li>
-<li><a href="#appendix">Appendix</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<!-- see https://github.com/apache/datafusion/issues/11631 for details -->
-<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
-<p>In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.</p>
-<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
-<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
-</blockquote>
-<p>There are various use cases where this type of analysis can be useful such as the following examples.</p>
-<h3 id="removing-unnecessary-sorts">Removing Unnecessary Sorts<a class="headerlink" href="#removing-unnecessary-sorts" title="Permanent link">¶</a></h3>
-<p>Imagine a user wants to execute the following query:</p>
-<pre><code class="language-SQL">SELECT hostname, log_line
-FROM telemetry ORDER BY time ASC limit 10
-</code></pre>
-<p>If we don't know anything about the <code>telemetry</code> table we need to sort it by <code>time ASC</code> and then retrieve the first 10 rows to get the correct result. However, if the table is already ordered by <code>time ASC</code>, we can simply retrieve the first 10 rows. This approach executes much faster and uses less memory compared to resorting the entire table, even when the <a href="https://docs.rs/datafusion/latest/datafusion/physical_plan/struct.TopK.html">TopK</a> ope [...]
-<p>In order to avoid the sort the query optimizer must determine the data is already sorted. For simple queries the analysis is straightforward however it gets complicated fast. For example, what if your data is sorted by <code>[hostname, time ASC]</code> and your query is</p>
-<pre><code class="language-sql">SELECT hostname, log_line
-FROM telemetry WHERE hostname = 'app.example.com' ORDER BY time ASC;
-</code></pre>
-<p>In this case a sort still isn't needed, but the analysis must reason about the sortedness of the stream when it knows <code>hostname</code> has a single value.</p>
-<h3 id="optimized-operator-implementations">Optimized Operator Implementations<a class="headerlink" href="#optimized-operator-implementations" title="Permanent link">¶</a></h3>
-<p>As another use case, some operators can utilize the ordering information to change its underlying algorithm to execute more efficiently. Consider the following query:</p>
-<pre><code class="language-SQL">SELECT COUNT(log_line)
-FROM telemetry GROUP BY hostname;
-</code></pre>
-<p>Most analytic systems, including DataFusion, by default implement such a query using a hash table keyed on values of <code>hostname</code> to store the counts. However, if the <code>telemetry</code> table is sorted by <code>hostname</code>, there are much more efficient algorithms for grouping on <code>hostname</code> values than hashing every value and storing it in memory. However, the more efficient algorithm can only be used when the input is sorted correctly. To see this in prac [...]
-<h3 id="streaming-friendly-execution">Streaming-Friendly Execution<a class="headerlink" href="#streaming-friendly-execution" title="Permanent link">¶</a></h3>
-<p>Stream processing aims to produce results immediately as they become available ensuring minimal latency for real-time workloads. However, some operators need to consume all input data before producing any output. Consider the <code>Sort</code> operation: before it can start generating output, the algorithm must first process all input data. As a result, data flow halts whenever such an operator is encountered until all input is consumed. When a physical query plan contains such an ope [...]
-<p>For a query to be executed in a streaming fashion we need to satisfy 2 conditions:</p>
-<p><strong>Logically Streamable</strong><br/>
-It should be possible to generate what user wants in streaming fashion. Consider following query:</p>
-<pre><code class="language-SQL">SELECT SUM(amount)
-FROM orders
-</code></pre>
-<p>Here, the user wants to compute the sum of all amounts in the orders table. By the nature of the query this requires scanning the entire table to generate a result making it impossible to execute in a streaming fashion.</p>
-<p><strong>Streaming Aware Planner</strong><br/>
-Being logically streamable does not guarantee that a query will execute in a streaming fashion. SQL is a declarative language, meaning it specifies 'WHAT' user wants. It is up to the planner 'HOW' to generate the result. In most cases there are many ways to compute the correct result for a given query. The query planner is responsible for choosing "a way" (ideally the best<sup id="optimal1"><a href="#optimal">*</a></sup> one) among the all alternatives to generate what user asks for. If [...]
-<h2 id="analysis">Analysis<a class="headerlink" href="#analysis" title="Permanent link">¶</a></h2>
-<p>Let's start by creating an example table that we will refer throughout the post. This table models the input data of an operator for the analysis:</p>
-<h3 id="example-virtual-table">Example Virtual Table<a class="headerlink" href="#example-virtual-table" title="Permanent link">¶</a></h3>
-<style>
- table {
- border-collapse: collapse;
- width: 80%;
- font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
- }
- th, td {
- padding: 12px 16px;
- text-align: left;
- border-bottom: 1px solid #e0e0e0;
- }
- th {
- background-color: #f9f9f9;
- font-weight: 600;
- }
- tr:hover {
- background-color: #f1f1f1;
- }
-</style>
-<table class="table">
-<tr>
-<th>amount</th> <th>price</th> <th>hostname</th><th>currency</th><th>time_bin</th> <th>time</th> <th>price_cloned</th> <th>time_cloned</th>
-</tr>
-<tr>
-<td>12</td> <td>25</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:01:30</td> <td>25</td> <td>08:01:30</td>
-</tr>
-<tr>
-<td>12</td> <td>26</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:11:30</td> <td>26</td> <td>08:11:30</td>
-</tr>
-<tr>
-<td>15</td> <td>30</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:41:30</td> <td>30</td> <td>08:41:30</td>
-</tr>
-<tr>
-<td>15</td> <td>32</td> <td>app.example.com</td> <td>USD</td> <td>08:00:00</td> <td>08:55:15</td> <td>32</td> <td>08:55:15</td>
-</tr>
-<tr>
-<td>15</td> <td>35</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:10:23</td> <td>35</td> <td>09:10:23</td>
-</tr>
-<tr>
-<td>20</td> <td>18</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:20:33</td> <td>18</td> <td>09:20:33</td>
-</tr>
-<tr>
-<td>20</td> <td>22</td> <td>app.example.com</td> <td>USD</td> <td>09:00:00</td> <td>09:40:15</td> <td>22</td> <td>09:40:15</td>
-</tr>
-</table>
-<p><br/></p>
-<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
-<strong>How can a table have multiple orderings?</strong> At first glance it may seem counterintuitive for a table to have more than one valid ordering. However, during query execution such scenarios can arise.
-
-For example consider the following query:
-
-
-<pre><code class="language-sql">SELECT time, date_bin('1 hour', time, '1970-01-01') as time_bin
-FROM table;
-</code></pre>
-
-If we know that the table is ordered by <code>time ASC</code> we can infer that <code>time_bin ASC</code> is also a valid ordering. This is because the <code>date_bin</code> function is monotonic, meaning it preserves the order of its input.
-
-DataFusion leverages these functional dependencies to infer new orderings as data flows through different query operators. For details on the implementation see the <a ,="" href="https://github.com/apache/datafusion/blob/main/datafusion/common/src/functional_dependencies.rs" target="_blank">source</a> code.
-</blockquote>
-<p>By inspection, you can see this table is sorted by the <code>amount</code> column, but It is also sorted by <code>time</code> and <code>time_bin</code> as well as the compound <code>(time_bin, amount)</code> and many other variations. While this example is an extreme case, real-world data often has multiple sort orders. </p>
-<p>A naive approach for analyzing whether the ordering requirement of an operator is satisfied by its input would be: </p>
-<ul>
-<li>Store all the valid ordering expressions that the tables satisfies </li>
-<li>Check whether the ordering requirement by the operator is among valid orderings. </li>
-</ul>
-<p>This naive algorithm works and correct. However, listing all valid orderings can be quite lengthy and is of exponential complexity as the number of orderings grows. For the example table here is a (small) subset of the valid orderings:</p>
-<p><code>[amount ASC]</code><br/>
-<code>[amount ASC, price ASC]</code><br/>
-<code>[amount ASC, price_cloned ASC]</code><br/>
-<code>[hostname ASC, amount ASC, price_cloned ASC]</code><br/>
-<code>[amount ASC, hostname ASC, price_cloned ASC]</code><br/>
-<code>[amount ASC, price_cloned ASC, hostname ASC]</code><br/>
-.<br/>
-.<br/>
-. </p>
-<p>As can be seen from the listing above storing all valid orderings is wasteful and contains significant redundancy. Here are some observations which suggest that we can do much better:</p>
-<ul>
-<li>
-<p>Storing a prefix of another valid ordering is redundant. If the table satisfies the lexicographic ordering<sup id="fn1"><a href="#footnote1">1</a></sup>: <code>[amount ASC, price ASC]</code>, it already satisfies ordering <code>[amount ASC]</code> trivially. Hence, once we store <code>[amount ASC, price ASC]</code> storing <code>[amount ASC]</code> is redundant.</p>
-</li>
-<li>
-<p>Using all columns that are equal to each other in the listings is redundant. If we know the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by <code>[amount ASC, price_cloned ASC]</code> since <code>price</code> and <code>price_cloned</code> are copy of each other. It is enough to use just one expression among the expressions that exact copy of each other.</p>
-</li>
-<li>
-<p>Constant expressions can be inserted anywhere in a valid ordering with an arbitrary direction (e.g. <code>ASC</code>, <code>DESC</code>). Hence, if the table is ordered by <code>[amount ASC, price ASC]</code>, it is also ordered by: <br/>
-<code>[hostname ASC, amount ASC, price ASC]</code>,<br/>
-<code>[hostname DESC, amount ASC, price ASC]</code>,<br/>
-<code>[amount ASC, hostname ASC, price ASC]</code>,<br/>
- .<br/>
- . </p>
-</li>
-</ul>
-<p>This is clearly redundant. For this reason, it is better to avoid explicitly encoding constant expressions in valid sort orders.</p>
-<p>In summary,</p>
-<ul>
-<li>We should store only the longest lexicographic ordering (shouldn't use any prefix of it)</li>
-<li>Using expressions that are exact copies of each other is redundant.</li>
-<li>Ordering expressions shouldn't contain any constant expression.</li>
-</ul>
-<h2 id="key-concepts-for-analyzing-orderings">Key Concepts for Analyzing Orderings<a class="headerlink" href="#key-concepts-for-analyzing-orderings" title="Permanent link">¶</a></h2>
-<p>To solve the shortcomings above DataFusion needs to track of following properties for the table:</p>
-<ul>
-<li>Constant Expressions </li>
-<li>Equivalent Expression Groups (will be explained shortly)</li>
-<li>Succinct Valid Orderings (will be explained shortly)</li>
-</ul>
-<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
-<strong>Note:</strong> These properties are implemented in the <code>EquivalenceProperties</code> structure in <code>DataFusion</code>, please see the <a href="https://github.com/apache/datafusion/blob/f47ea73b87eec4af044f9b9923baf042682615b2/datafusion/physical-expr/src/equivalence/properties/mod.rs#L134" target="_blank">source</a> for more details<br/>
-</blockquote>
-<p>These properties allow us to analyze whether the ordering requirement is satisfied by the data already.</p>
-<h3 id="1-constant-expressions">1. Constant Expressions<a class="headerlink" href="#1-constant-expressions" title="Permanent link">¶</a></h3>
-<p>Constant expressions are those where each row in the expression has the same value across all rows. Although constant expressions may seem odd in a table they can arise after operations like <code>Filter</code> or <code>Join</code> occur. </p>
-<p>For instance in the example table:</p>
-<ul>
-<li>Columns <code>hostname</code> and <code>currency</code> are constant because every row in the table has the same value (<code>'app.example.com'</code> for <code>hostname</code>, and <code>'USD'</code> for <code>currency</code>) for these columns.</li>
-</ul>
-<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
-<strong>Note:</strong> Constant expressions can arise during query execution. For example, in following query:<br/>
-<code>SELECT hostname FROM logs</code><br/><code>WHERE hostname='app.example.com'</code> <br/>
- after filtering is done, for subsequent operators the <code>hostname</code> column will be constant.
-</blockquote>
-<h3 id="2-equivalent-expression-groups">2. Equivalent Expression Groups<a class="headerlink" href="#2-equivalent-expression-groups" title="Permanent link">¶</a></h3>
-<p>Equivalent expression groups are expressions that always hold the same value across rows. These expressions can be thought of as clones of each other and may arise from operations like <code>Filter</code>, <code>Join</code>, or <code>Projection</code>.</p>
-<p>In the example table, the expressions <code>price</code> and <code>price_cloned</code> form one equivalence group, and <code>time</code> and <code>time_cloned</code> form another equivalence group.</p>
-<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
-<strong>Note:</strong> Equivalent expression groups can arise during the query execution. For example, in the following query:<br/>
-<code>SELECT time, time as time_cloned FROM logs</code> <br/>
- after the projection is done, for subsequent operators <code>time</code> and <code>time_cloned</code> will form an equivalence group. As another example, in the following query:<br/>
-<code>SELECT employees.id, employees.name, departments.department_name</code>
-<code>FROM employees</code>
-<code>JOIN departments ON employees.department_id = departments.id;</code> <br/>
-after joining, <code>employees.department_id</code> and <code>departments.id</code> will form an equivalence group.
-</blockquote>
-<h3 id="3-succinct-encoding-of-valid-orderings">3. Succinct Encoding of Valid Orderings<a class="headerlink" href="#3-succinct-encoding-of-valid-orderings" title="Permanent link">¶</a></h3>
-<p>Valid orderings are the orderings that the table already satisfies. However, naively listing them requires exponential space as the number of columns grows as discussed before. Instead, we list all valid orderings after following constraints are applied:</p>
-<ul>
-<li>Do not use any constant expressions in the valid ordering construction</li>
-<li>Use only one entry (by convention the first entry) in the equivalent expression group.</li>
-<li>Lexicographic ordering shouldn't contain any leading ordering<sup id="fn2"><a href="#footnote2">2</a></sup>except the first position <sup id="fn3"><a href="#footnote3">3</a></sup>.</li>
-<li>Do not use any prefix of a valid lexicographic ordering<sup id="fn4"><a href="#footnote4">4</a></sup>.</li>
-</ul>
-<p>After applying the first and second constraint, the example table simplifies to </p>
-<style>
- table {
- border-collapse: collapse;
- width: 80%;
- font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
- }
- th, td {
- padding: 12px 16px;
- text-align: left;
- border-bottom: 1px solid #e0e0e0;
- }
- th {
- background-color: #f9f9f9;
- font-weight: 600;
- }
- tr:hover {
- background-color: #f1f1f1;
- }
-</style>
-<table class="table">
-<tr>
-<th>amount</th> <th>price</th><th>time_bin</th> <th>time</th>
-</tr>
-<tr>
-<td>12</td> <td>25</td><td>08:00:00</td> <td>08:01:30</td>
-</tr>
-<tr>
-<td>12</td> <td>26</td><td>08:00:00</td> <td>08:11:30</td>
-</tr>
-<tr>
-<td>15</td> <td>30</td><td>08:00:00</td> <td>08:41:30</td>
-</tr>
-<tr>
-<td>15</td> <td>32</td><td>08:00:00</td> <td>08:55:15</td>
-</tr>
-<tr>
-<td>15</td> <td>35</td><td>09:00:00</td> <td>09:10:23</td>
-</tr>
-<tr>
-<td>20</td> <td>18</td><td>09:00:00</td> <td>09:20:33</td>
-</tr>
-<tr>
-<td>20</td> <td>22</td><td>09:00:00</td> <td>09:40:15</td>
-</tr>
-</table>
-<p><br/>
-Following third and fourth constraints for the simplified table, the succinct valid orderings are:<br/>
-<code>[amount ASC, price ASC]</code>, <br/>
-<code>[time_bin ASC]</code>,<br/>
-<code>[time ASC]</code> </p>
-<blockquote class="border-start border-primary border-4 ps-3 py-2 bg-light">
-<p><strong>How can DataFusion find orderings?</strong></p>
-DataFusion's <code>CREATE EXTERNAL TABLE</code> has a <code>WITH ORDER</code> clause (see <a href="https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table">docs</a>) to specify the known orderings of the table during table creation. For example the following query:<br/>
-<pre><code>
-CREATE EXTERNAL TABLE source (
- amount INT NOT NULL,
- price DOUBLE NOT NULL,
- time TIMESTAMP NOT NULL,
- ...
-)
-STORED AS CSV
-WITH ORDER (time ASC)
-WITH ORDER (amount ASC, price ASC)
-LOCATION '/path/to/FILE_NAME.csv'
-OPTIONS ('has_header' 'true');
-</code></pre>
-communicates that <code>source</code> table has the orderings: <code>[time ASC]</code> and <code>[amount ASC, price ASC]</code>.<br/>
-When orderings are communicated from the source, DataFusion tracks the orderings through each operator while optimizing the plan.<br/>
-<ul>
-<li>add new orderings (such as when "date_bin" function is applied to the "time" column)</li>
-<li>Remove orderings, if operation doesn't preserve the ordering of the data at its input</li>
-<li>Update equivalent groups</li>
-<li>Update constant expressions</li>
-</ul>
-
-Figure 1 shows an example how DataFusion generates an efficient plan for the query:
-<pre><code>
-SELECT
- row_number() OVER (ORDER BY time) as rn,
- time
-FROM events
-ORDER BY rn, time
-</code></pre>
-using the orderings of the query intermediates.<br/>
-<br/>
-<figure>
-<img alt="Window Query Datafusion Optimization" class="img-fluid" src="/blog/images/ordering_analysis/query_window_plan.png" width="80%"/>
-<figcaption><strong>Figure 1:</strong> DataFusion analyzes orderings of the sources and query intermediates to generate efficient plans</figcaption>
-</figure>
-</blockquote>
-<h3 id="table-properties">Table Properties<a class="headerlink" href="#table-properties" title="Permanent link">¶</a></h3>
-<p>In summary, for the example table, the following properties correctly describe the sort properties:</p>
-<ul>
-<li><strong>Constant Expressions</strong> = <code>hostname, currency</code> </li>
-<li><strong>Equivalent Expression Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code> </li>
-<li><strong>Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code> </li>
-</ul>
-<h3 id="algorithm-for-analyzing-ordering-requirements">Algorithm for Analyzing Ordering Requirements<a class="headerlink" href="#algorithm-for-analyzing-ordering-requirements" title="Permanent link">¶</a></h3>
-<p>After deriving these properties for the data, following algorithm can be used to check whether an ordering requirement is satisfied by the table:</p>
-<ol>
-<li><strong>Prune constant expressions</strong>: Remove any constant expressions from the ordering requirement.</li>
-<li><strong>Normalize the requirement</strong>: Replace each expression in the ordering requirement with the first entry from its equivalence group.</li>
-<li><strong>De-duplicate expressions</strong>: If an expression appears more than once, remove duplicates, keeping only the first occurrence.</li>
-<li><strong>Match leading orderings</strong>: Check whether the leading ordering requirement<sup id="fn5"><a href="#footnote5">5</a></sup> matches the leading valid orderings<sup id="fn6"><a href="#footnote6">6</a></sup> of table. If so:<ul>
-<li>Remove the leading ordering requirement from the ordering requirement </li>
-<li>Remove the matching leading valid ordering from the valid orderings of table. </li>
-</ul>
-</li>
-<li><strong>Iterate through the remaining expressions</strong>: Go back to step 4 until ordering requirement is empty or leading ordering requirement is not found among the leading valid orderings of table.</li>
-</ol>
-<p>If, at the end of the procedure above, the ordering requirement is an empty list, we can conclude that the requirement is satisfied by the table.</p>
-<h3 id="example-walkthrough">Example Walkthrough<a class="headerlink" href="#example-walkthrough" title="Permanent link">¶</a></h3>
-<p>Let's say the user provided a query such as the following:</p>
-<pre><code class="language-sql">SELECT * FROM table
-ORDER BY hostname DESC, amount ASC, time_bin ASC, price_cloned ASC, time ASC, currency ASC, price DESC;
-</code></pre>
-<p>And the input has the same properties explained above</p>
-<ul>
-<li><strong>Constant Expressions</strong> = <code>hostname, currency</code></li>
-<li><strong>Equivalent Expressions Groups</strong> = <code>[price, price_cloned], [time, time_cloned]</code></li>
-<li><strong>Succinct Valid Orderings</strong> = <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code></li>
-</ul>
-<p>In order to remove a sort the optimizer must check if the ordering requirement <code>[hostname DESC, amount ASC, time_bin ASC, price_cloned ASC, time ASC, currency ASC, price DESC]</code> is satisfied by the properties.</p>
-<h3 id="algorithm-steps">Algorithm Steps<a class="headerlink" href="#algorithm-steps" title="Permanent link">¶</a></h3>
-<ol>
-<li>
-<p><strong>Prune constant expressions</strong>:<br/>
- Remove <code>hostname</code> and <code>currency</code> from the requirement. The requirement becomes:<br/>
-<code>[amount ASC, time_bin ASC, price_cloned ASC, time ASC, price DESC]</code>.</p>
-</li>
-<li>
-<p><strong>Normalize using equivalent groups</strong>:<br/>
- Replace <code>price_cloned</code> with <code>price</code> and <code>time_cloned</code> with <code>time</code>. The requirement becomes:<br/>
-<code>[amount ASC, time_bin ASC, price ASC, time ASC, price DESC]</code>.</p>
-</li>
-<li>
-<p><strong>De-duplicate expressions</strong>:<br/>
- Since <code>price</code> appears twice, we simplify the requirement to:<br/>
-<code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> (keeping the first occurrence from the left side).</p>
-</li>
-<li>
-<p><strong>Match leading orderings</strong>:<br/>
- Check if leading ordering requirement <code>amount ASC</code> is among the leading valid orderings: <code>amount ASC, time_bin ASC, time ASC</code>. Since this is the case, we remove <code>amount ASC</code> from both the ordering requirement and the valid orderings of the table.</p>
-</li>
-<li><strong>Iterate through the remaining expressions</strong>:
-Now, the problem is converted from<br/>
-<em>"whether the requirement: <code>[amount ASC, time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[amount ASC, price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
-into<br/>
-<em>"whether the requirement: <code>[time_bin ASC, price ASC, time ASC]</code> is satisfied by valid orderings: <code>[price ASC], [time_bin ASC], [time ASC]</code>"</em><br/>
-We go back to step 4 until the ordering requirement list is exhausted or its length no longer decreases.</li>
-</ol>
-<p>At the end of stages above we end up with an empty ordering requirement list. Given this, we can conclude that the table satisfies the ordering requirement and thus no sort is required. </p>
-<h2 id="conclusion">Conclusion<a class="headerlink" href="#conclusion" title="Permanent link">¶</a></h2>
-<p>In this post, we described the conditions under which an ordering requirement is satisfied based on the properties of a table. We introduced key concepts such as constant expressions, equivalence groups, and valid orderings, and used them to determine whether a given ordering requirement are satisfied by an input table.</p>
-<p>This analysis plays a crucial role in:</p>
-<ul>
-<li>Choosing more efficient algorithm variants</li>
-<li>Generating streaming-friendly plans</li>
-</ul>
-<p>The <code>DataFusion</code> query engine employs this analysis (and many more) during its planning stage to ensure correct and efficient query execution. We <a href="https://datafusion.apache.org/contributor-guide/index.html">welcome you</a> to come and join the project.</p>
-<h2 id="appendix">Appendix<a class="headerlink" href="#appendix" title="Permanent link">¶</a></h2>
-<!--
-<p id="footnote1"><sup>[1]</sup>The ordering requirement refers to the condition that input data must be sorted in a certain way for a specific operator to function as intended.</p>
--->
-<p id="footnote1"><sup>[1]</sup>Lexicographic order is a way of ordering sequences (like strings, list of expressions) based on the order of their components, similar to how words are ordered in a dictionary. It compares each element of the sequences one by one, from left to right.</p>
-<p id="footnote2"><sup>[2]</sup>Leading ordering is the first ordering in a lexicographic ordering list. As an example, for the ordering: <code>[amount ASC, price ASC]</code>, leading ordering will be: <code>amount ASC</code>. </p>
-<p id="footnote3"><sup>[3]</sup>This means that, if we know that <code>[amount ASC]</code> and <code>[time ASC]</code> are both valid orderings for the table. We shouldn't enlist <code>[amount ASC, time ASC]</code> or <code>[time ASC, amount ASC]</code> as valid orderings. These orderings can be deduced if we know that table satisfies the ordering <code>[amount ASC]</code> and <code>[time ASC]</code>.</p>
-<p id="footnote4"><sup>[4]</sup>This means that, if ordering <code>[amount ASC, price ASC]</code> is a valid ordering for the table. We shouldn't enlist <code>[amount ASC]</code> as valid ordering. Validity of it can be deduced from the ordering: <code>[amount ASC, price ASC]</code>
-<p id="footnote5"><sup>[5]</sup>Leading ordering requirement is the first ordering requirement in the list of lexicographic ordering requirement expression. As an example for the requirement: <code>[amount ASC, time_bin ASC, prices ASC, time ASC]</code>, leading ordering requirement is: <code>amount ASC</code>.</p>
-<p id="footnote6"><sup>[6]</sup>Leading valid orderings are the first ordering for each valid ordering list in the table. As an example, for the valid orderings: <code>[amount ASC, prices ASC], [time_bin ASC], [time ASC]</code>, leading valid orderings will be: <code>amount ASC, time_bin ASC, time ASC</code>. </p>
-<p id="optimal"><sup>*</sup>Best depends on the use case, <code>DataFusion</code> has many various flags to communicate what user thinks the best plan is (e.g. streamable, fastest, lowest memory, etc.). See <a href="https://datafusion.apache.org/user-guide/configs.html" target="_blank">configurations</a> for detail.</p></p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#introduction">Introduction</a><ul>
-<li><a href="#removing-unnecessary-sorts">Removing Unnecessary Sorts</a></li>
-<li><a href="#optimized-operator-implementations">Optimized Operator Implementations</a></li>
-<li><a href="#streaming-friendly-execution">Streaming-Friendly Execution</a></li>
-</ul>
-</li>
-<li><a href="#analysis">Analysis</a><ul>
-<li><a href="#example-virtual-table">Example Virtual Table</a></li>
-</ul>
-</li>
-<li><a href="#key-concepts-for-analyzing-orderings">Key Concepts for Analyzing Orderings</a><ul>
-<li><a href="#1-constant-expressions">1. Constant Expressions</a></li>
-<li><a href="#2-equivalent-expression-groups">2. Equivalent Expression Groups</a></li>
-<li><a href="#3-succinct-encoding-of-valid-orderings">3. Succinct Encoding of Valid Orderings</a></li>
-<li><a href="#table-properties">Table Properties</a></li>
-<li><a href="#algorithm-for-analyzing-ordering-requirements">Algorithm for Analyzing Ordering Requirements</a></li>
-<li><a href="#example-walkthrough">Example Walkthrough</a></li>
-<li><a href="#algorithm-steps">Algorithm Steps</a></li>
-</ul>
-</li>
-<li><a href="#conclusion">Conclusion</a></li>
-<li><a href="#appendix">Appendix</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2025/03/20/datafusion-comet-0.7.0/index.html b/blog/2025/03/20/datafusion-comet-0.7.0/index.html
deleted file mode 100644
index 4936b93..0000000
--- a/blog/2025/03/20/datafusion-comet-0.7.0/index.html
+++ /dev/null
@@ -1,214 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 0.7.0 Release - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Comet 0.7.0 Release
- </h1>
- <p>Posted on: Thu 20 March 2025 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#performance">Performance</a></li>
-</ul>
-</li>
-<li><a href="#shuffle-improvements">Shuffle Improvements</a></li>
-<li><a href="#improved-hash-join-performance">Improved Hash Join Performance</a></li>
-<li><a href="#experimental-support-for-datafusions-parquet-scan">Experimental Support for DataFusion’s Parquet Scan</a></li>
-<li><a href="#complex-type-support">Complex Type Support</a></li>
-<li><a href="#updates-to-supported-spark-versions">Updates to supported Spark versions</a></li>
-<li><a href="#improved-tuning-guide">Improved Tuning Guide</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>The Apache DataFusion PMC is pleased to announce version 0.7.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>Comet runs on commodity hardware and aims to provide 100% compatibility with Apache Spark. Any operators or
-expressions that are not fully compatible will fall back to Spark unless explicitly enabled by the user. Refer
-to the <a href="https://datafusion.apache.org/comet/user-guide/compatibility.html">compatibility guide</a> for more information.</p>
-<p>This release covers approximately four weeks of development work and is the result of merging 46 PRs from 11
-contributors. See the <a href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.7.0.md">change log</a> for more information.</p>
-<h2 id="release-highlights">Release Highlights<a class="headerlink" href="#release-highlights" title="Permanent link">¶</a></h2>
-<h3 id="performance">Performance<a class="headerlink" href="#performance" title="Permanent link">¶</a></h3>
-<p>Comet 0.7.0 has improved performance compared to the previous release due to improvements in the native shuffle
-implementation and performance improvements in DataFusion 46.</p>
-<p>For single-node TPC-H at 100 GB, Comet now delivers a <strong>greater than 2x speedup</strong> compared to Spark using the same
-CPU and RAM. Even with <strong>half the resources</strong>, Comet still provides a measurable performance improvement.</p>
-<p><img alt="Chart showing TPC-H benchmark results for Comet 0.7.0" class="img-fluid" src="/blog/images/comet-0.7.0/performance.png" width="100%"/></p>
-<p><em>These benchmarks were performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and data
-stored locally in Parquet format on NVMe storage. Spark was running in Kubernetes with hard memory limits.</em></p>
-<h2 id="shuffle-improvements">Shuffle Improvements<a class="headerlink" href="#shuffle-improvements" title="Permanent link">¶</a></h2>
-<p>There are several improvements to shuffle in this release:</p>
-<ul>
-<li>When running in off-heap mode (which is the recommended approach), Comet was using the wrong memory allocator
- implementation for some types of shuffle operation, which could result in OOM rather than spilling to disk.</li>
-<li>The number of spill files is drastically reduced. In previous releases, each instance of ShuffleMapTask could
- potentially create a new spill file for each output partition each time that spill was invoked. Comet now creates
- a maximum of one spill file per output partition per instance of ShuffleMapTask, which is appended to in subsequent
- spills.</li>
-<li>There was a flaw in the memory accounting which resulted in Comet requesting approximately twice the amount of
- memory that was needed, resulting in premature spilling. This is now resolved.</li>
-<li>The metric for number of spilled bytes is now accurate. It was previously reporting invalid information.</li>
-</ul>
-<h2 id="improved-hash-join-performance">Improved Hash Join Performance<a class="headerlink" href="#improved-hash-join-performance" title="Permanent link">¶</a></h2>
-<p>When using the <code>spark.comet.exec.replaceSortMergeJoin</code> setting to replace sort-merge joins with hash joins, Comet
-will now do a better job of picking the optimal build side. Thanks to <a href="https://github.com/hayman42">@hayman42</a> for suggesting this, and thanks to the
-<a href="https://github.com/apache/incubator-gluten/">Apache Gluten(incubating)</a> project for the inspiration in implementing this feature.</p>
-<h2 id="experimental-support-for-datafusions-parquet-scan">Experimental Support for DataFusion’s Parquet Scan<a class="headerlink" href="#experimental-support-for-datafusions-parquet-scan" title="Permanent link">¶</a></h2>
-<p>It is now possible to configure Comet to use DataFusion’s Parquet reader instead of Comet’s current Parquet reader. This
-has the advantage of supporting complex types, and also has performance optimizations that are not present in Comet's
-existing reader.</p>
-<p>Support should still be considered experimental, but most of Comet’s unit tests are now passing with the new reader.
-Known issues include handling of <code>INT96</code> timestamps and unsigned bytes and shorts.</p>
-<p>To enable DataFusion’s Parquet reader, either set <code>spark.comet.scan.impl=native_datafusion</code> or set the environment
-variable <code>COMET_PARQUET_SCAN_IMPL=native_datafusion</code>.</p>
-<h2 id="complex-type-support">Complex Type Support<a class="headerlink" href="#complex-type-support" title="Permanent link">¶</a></h2>
-<p>With DataFusion’s Parquet reader enabled, there is now some early support for reading structs from Parquet. This is
-not thoroughly tested yet. We would welcome additional testing from the community to help determine what is and isn’t
-working, as well as contributions to improve support for structs and other complex types. The tracking issue is
-<a href="https://github.com/apache/datafusion-comet/issues/1043">https://github.com/apache/datafusion-comet/issues/1043</a>.</p>
-<h2 id="updates-to-supported-spark-versions">Updates to supported Spark versions<a class="headerlink" href="#updates-to-supported-spark-versions" title="Permanent link">¶</a></h2>
-<ul>
-<li>Comet 0.7.0 is now tested against Spark 3.5.4 rather than 3.5.1</li>
-<li>This will be the last Comet release to support Spark 3.3.x</li>
-</ul>
-<h2 id="improved-tuning-guide">Improved Tuning Guide<a class="headerlink" href="#improved-tuning-guide" title="Permanent link">¶</a></h2>
-<p>The <a href="https://datafusion.apache.org/comet/user-guide/tuning.html">Comet Tuning Guide</a> has been improved and now provides guidance on determining how much memory to allocate to
-Comet.</p>
-<h2 id="getting-involved">Getting Involved<a class="headerlink" href="#getting-involved" title="Permanent link">¶</a></h2>
-<p>The Comet project welcomes new contributors. We use the same <a href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack and Discord</a> channels as the main DataFusion
-project and have a weekly <a href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion video call</a>.</p>
-<p>The easiest way to get involved is to test Comet with your current Spark jobs and file issues for any bugs or
-performance regressions that you find. See the <a href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting Started</a> guide for instructions on downloading and installing
-Comet.</p>
-<p>There are also many <a href="https://github.com/apache/datafusion-comet/contribute">good first issues</a> waiting for contributions.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#release-highlights">Release Highlights</a><ul>
-<li><a href="#performance">Performance</a></li>
-</ul>
-</li>
-<li><a href="#shuffle-improvements">Shuffle Improvements</a></li>
-<li><a href="#improved-hash-join-performance">Improved Hash Join Performance</a></li>
-<li><a href="#experimental-support-for-datafusions-parquet-scan">Experimental Support for DataFusion’s Parquet Scan</a></li>
-<li><a href="#complex-type-support">Complex Type Support</a></li>
-<li><a href="#updates-to-supported-spark-versions">Updates to supported Spark versions</a></li>
-<li><a href="#improved-tuning-guide">Improved Tuning Guide</a></li>
-<li><a href="#getting-involved">Getting Involved</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2025/03/20/parquet-pruning/index.html b/blog/2025/03/20/parquet-pruning/index.html
deleted file mode 100644
index 03a2dde..0000000
--- a/blog/2025/03/20/parquet-pruning/index.html
+++ /dev/null
@@ -1,200 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Parquet Pruning in DataFusion: Read Only What Matters - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Parquet Pruning in DataFusion: Read Only What Matters
- </h1>
- <p>Posted on: Thu 20 March 2025 by Xiangpeng Hao</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#the-pipeline">The pipeline</a><ul>
-<li><a href="#background-parquet-file-structure">Background: Parquet file structure</a></li>
-<li><a href="#1-read-metadata">1. Read metadata</a></li>
-<li><a href="#2-prune-by-projection">2. Prune by projection</a></li>
-<li><a href="#3-prune-by-row-group-stats-and-bloom-filters">3. Prune by row group stats and Bloom filters</a></li>
-<li><a href="#4-prune-by-page-stats">4. Prune by page stats</a></li>
-<li><a href="#5-read-from-storage">5. Read from storage</a></li>
-</ul>
-</li>
-<li><a href="#preview-of-coming-attractions-filter-pushdown">Preview of coming attractions: filter pushdown</a></li>
-<li><a href="#conclusion">Conclusion</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-to-arrow/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
-<hr/>
-<p><a href="https://parquet.apache.org/">Apache Parquet</a> has become the industry standard for storing columnar data, and reading Parquet efficiently -- especially from remote storage -- is crucial for query performance.</p>
-<p><a href="https://datafusion.apache.org/">Apache DataFusion</a> implements advanced Parquet pruning techniques to effectively read only the data that matters for a given query.</p>
-<p>Achieving high performance adds complexity.
-This post provides an overview of the techniques used in DataFusion to selectively read Parquet files.</p>
-<h3 id="the-pipeline">The pipeline<a class="headerlink" href="#the-pipeline" title="Permanent link">¶</a></h3>
-<p>The diagram below illustrates the <a href="https://docs.rs/datafusion/46.0.0/datafusion/datasource/physical_plan/parquet/source/struct.ParquetSource.html">Parquet reading pipeline</a> in DataFusion, highlighting how data flows through various pruning stages before being converted to Arrow format:</p>
-<p><img alt="Parquet pruning pipeline in DataFusion" class="img-fluid" src="/blog/images/parquet-pruning/read-parquet.jpg" width="100%"/></p>
-<h4 id="background-parquet-file-structure">Background: Parquet file structure<a class="headerlink" href="#background-parquet-file-structure" title="Permanent link">¶</a></h4>
-<p>As shown in the figure above, each Parquet file has multiple row groups. Each row group contains a set of columns, and each column contains a set of pages.</p>
-<p>Pages are the smallest units of data in Parquet files and typically contain compressed and encoded values for a specific column. This hierarchical structure enables efficient columnar access and forms the foundation for the pruning techniques we'll discuss.</p>
-<p>Check out <a href="https://www.influxdata.com/blog/querying-parquet-millisecond-latency/">Querying Parquet with Millisecond Latency</a> for more details on the Parquet file structure.</p>
-<h4 id="1-read-metadata">1. Read metadata<a class="headerlink" href="#1-read-metadata" title="Permanent link">¶</a></h4>
-<p>DataFusion first reads the <a href="https://parquet.apache.org/docs/file-format/metadata/">Parquet metadata</a> to understand the data in the file.
-Metadata often includes data schema, the exact location of each row group and column chunk, and their corresponding statistics (e.g., min/max values).
-It also optionally includes <a href="https://parquet.apache.org/docs/file-format/pageindex/">page-level stats</a> and <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>.
-This information is used to prune the file before reading the actual data.</p>
-<p><a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L118">Fetching metadata</a> requires up to two network requests: one to read the footer size from the end of the file, and another to read the footer itself. </p>
-<p><a href="https://www.influxdata.com/blog/how-good-parquet-wide-tables/">Decoding metadata</a> is generally fast since it only requires parsing a small amount of data. However, for tables with hundreds or thousands of columns, the metadata can become quite large and decoding it can become a bottleneck. This is particularly noticeable when scanning many small files.</p>
-<p>Reading metadata is latency-critical, so DataFusion allows users to cache metadata through the <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L39">ParquetFileReaderFactory</a> trait.</p>
-<h4 id="2-prune-by-projection">2. Prune by projection<a class="headerlink" href="#2-prune-by-projection" title="Permanent link">¶</a></h4>
-<p>The simplest yet perhaps most effective pruning is to read only the columns that are needed.
-This is because queries usually don't select all columns, e.g., <code>SELECT a FROM table</code> only reads column <code>a</code>.
-As a <strong>columnar</strong> format, Parquet allows DataFusion to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/mod.rs#L778">only read</a> the <strong>columns</strong> that are needed.</p>
-<p>This projection pruning happens at the column level and can dramatically reduce I/O when working with wide tables where queries typically access only a small subset of columns.</p>
-<h4 id="3-prune-by-row-group-stats-and-bloom-filters">3. Prune by row group stats and Bloom filters<a class="headerlink" href="#3-prune-by-row-group-stats-and-bloom-filters" title="Permanent link">¶</a></h4>
-<p>Each row group has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/physical_optimizer/pruning.rs#L81">basic stats</a> like min/max values for each column.
-DataFusion applies the query predicates to these stats to prune row groups, e.g., <code>SELECT * FROM table WHERE a > 10</code> will only read row groups where <code>a</code> has a max value greater than 10.</p>
-<p>Sometimes min/max stats are too simple to prune effectively, so Parquet also supports <a href="https://www.influxdata.com/blog/using-parquets-bloom-filters/">Bloom filters</a>. DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L202">uses Bloom filters when available</a>.</p>
-<p>Bloom filters are particularly effective for equality predicates (<code>WHERE a = 10</code>) and can significantly reduce the number of row groups that need to be read for point queries or queries with highly selective predicates.</p>
-<h4 id="4-prune-by-page-stats">4. Prune by page stats<a class="headerlink" href="#4-prune-by-page-stats" title="Permanent link">¶</a></h4>
-<p>Parquet optionally supports <a href="https://github.com/apache/parquet-format/blob/master/PageIndex.md">page-level stats</a> -- similar to row group stats but more fine-grained.
-DataFusion implements <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L219">page pruning</a> when the stats are present.</p>
-<p>Page-level pruning provides an additional layer of filtering after row group pruning. It allows DataFusion to skip individual pages within a row group, further reducing the amount of data that needs to be read and decoded.</p>
-<h4 id="5-read-from-storage">5. Read from storage<a class="headerlink" href="#5-read-from-storage" title="Permanent link">¶</a></h4>
-<p>Now we (hopefully) have pruned the Parquet file into small ranges of bytes, i.e., the <a href="https://github.com/apache/datafusion/blob/76a7789ace33ced54c973fa0d5fc9d1866e1bf19/datafusion/datasource-parquet/src/access_plan.rs#L86">Access Plan</a>.
-The last step is to <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/reader.rs#L103">make requests</a> to fetch those bytes and decode them into Arrow RecordBatch. </p>
-<h3 id="preview-of-coming-attractions-filter-pushdown">Preview of coming attractions: filter pushdown<a class="headerlink" href="#preview-of-coming-attractions-filter-pushdown" title="Permanent link">¶</a></h3>
-<p>So far we have discussed techniques that prune the Parquet file using only the metadata, i.e., before reading the actual data.</p>
-<p>Filter pushdown, also known as predicate pushdown or late materialization, is a technique that prunes data during scanning, with filters being generated and applied in the Parquet reader.</p>
-<p><img alt="Filter pushdown in DataFusion" class="img-fluid" src="/blog/images/parquet-pruning/filter-pushdown.jpg" width="100%"/></p>
-<p>Unlike metadata-based pruning which works at the row group or page level, filter pushdown operates at the row level, allowing DataFusion to filter out individual rows that don't match the query predicates during the decoding process.</p>
-<p>DataFusion <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/core/src/datasource/physical_plan/parquet/row_filter.rs#L154">implements filter pushdown</a> but has <a href="https://github.com/apache/datafusion/blob/31701b8dc9c6486856c06a29a32107d9f4549cec/datafusion/common/src/config.rs#L382">not enabled it by default</a> due to <a href="https://github.com/apache/datafusion/issues/3463">some performance regressions</a>.</p>
-<p>We are working to remove the <a href="https://github.com/apache/arrow-rs/issues/5523#issuecomment-2429470872">remaining performance issues</a> and enable it by default, which we will discuss in the next blog post.</p>
-<h3 id="conclusion">Conclusion<a class="headerlink" href="#conclusion" title="Permanent link">¶</a></h3>
-<p>DataFusion employs a multi-step approach to Parquet pruning, from column projection to row group stats, page stats, and potentially row-level filtering.
-Each step may reduce the amount of data to be read and processed, significantly improving query performance.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#the-pipeline">The pipeline</a><ul>
-<li><a href="#background-parquet-file-structure">Background: Parquet file structure</a></li>
-<li><a href="#1-read-metadata">1. Read metadata</a></li>
-<li><a href="#2-prune-by-projection">2. Prune by projection</a></li>
-<li><a href="#3-prune-by-row-group-stats-and-bloom-filters">3. Prune by row group stats and Bloom filters</a></li>
-<li><a href="#4-prune-by-page-stats">4. Prune by page stats</a></li>
-<li><a href="#5-read-from-storage">5. Read from storage</a></li>
-</ul>
-</li>
-<li><a href="#preview-of-coming-attractions-filter-pushdown">Preview of coming attractions: filter pushdown</a></li>
-<li><a href="#conclusion">Conclusion</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
- Apache® and the Apache feather logo are trademarks of The Apache Software Foundation.
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2025/03/21/parquet-pushdown/index.html b/blog/2025/03/21/parquet-pushdown/index.html
deleted file mode 100644
index eeec154..0000000
--- a/blog/2025/03/21/parquet-pushdown/index.html
+++ /dev/null
@@ -1,383 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Efficient Filter Pushdown in Parquet - Apache DataFusion Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Efficient Filter Pushdown in Parquet
- </h1>
- <p>Posted on: Fri 21 March 2025 by Xiangpeng Hao</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#why-filter-pushdown-in-parquet">Why filter pushdown in Parquet?</a></li>
-<li><a href="#how-can-filter-pushdown-be-slower">How can filter pushdown be slower?</a></li>
-<li><a href="#attempt-cache-filter-columns">Attempt: cache filter columns</a></li>
-<li><a href="#real-solution">Real solution</a><ul>
-<li><a href="#what-pages-are-cached">What pages are cached?</a></li>
-<li><a href="#then-why-cache-2-pages-per-column-instead-of-1">Then why cache 2 pages per column instead of 1?</a></li>
-</ul>
-</li>
-<li><a href="#how-does-it-perform">How does it perform?</a></li>
-<li><a href="#conclusion">Conclusion</a></li>
-</ul>
-</div>
- </aside>
-
- <style>
-figure {
- margin: 20px 0;
-}
-
-figure img {
- display: block;
- max-width: 80%;
-}
-
-figcaption {
- font-style: italic;
- margin-top: 10px;
- color: #555;
- font-size: 0.9em;
- max-width: 80%;
-}
-</style>
-<!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p><em>Editor's Note: This blog was first published on <a href="https://blog.xiangpeng.systems/posts/parquet-pushdown/">Xiangpeng Hao's blog</a>. Thanks to <a href="https://www.influxdata.com/">InfluxData</a> for sponsoring this work as part of his PhD funding.</em></p>
-<hr/>
-<p>In the <a href="https://datafusion.apache.org/blog/2025/03/20/parquet-pruning">previous post</a>, we discussed how <a href="https://datafusion.apache.org/">Apache DataFusion</a> prunes <a href="https://parquet.apache.org/">Apache Parquet</a> files to skip irrelevant <strong>files/row_groups</strong> (sometimes also <a href="https://parquet.apache.org/docs/file-format/pageindex/">pages</a>).</p>
-<p>This post discusses how Parquet readers skip irrelevant <strong>rows</strong> while scanning data,
... 88582 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org