You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by gi...@apache.org on 2023/04/04 14:36:20 UTC
[arrow-datafusion] branch asf-site updated: Publish built docs triggered by b87871fdd1f4ce64201eb1f7c79a0547627f37e9
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b0a46af93e Publish built docs triggered by b87871fdd1f4ce64201eb1f7c79a0547627f37e9
b0a46af93e is described below
commit b0a46af93e10520bcdc62e83cfa0c6c995349136
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Apr 4 14:36:14 2023 +0000
Publish built docs triggered by b87871fdd1f4ce64201eb1f7c79a0547627f37e9
---
_images/DataFusion-Logo-Background-White.png | Bin 0 -> 12401 bytes
_sources/contributor-guide/architecture.md.txt | 26 +++
_sources/contributor-guide/index.md.txt | 44 ++---
_sources/index.rst.txt | 18 +-
_sources/user-guide/comparison.md.txt | 52 ++++++
_sources/user-guide/integration.md.txt | 35 ++++
_sources/user-guide/introduction.md.txt | 2 +-
_sources/user-guide/users.md.txt | 67 +++++++
.../{communication.html => architecture.html} | 125 +++++--------
contributor-guide/communication.html | 33 +++-
contributor-guide/index.html | 197 ++++++++++++---------
contributor-guide/quarterly_roadmap.html | 27 ++-
contributor-guide/roadmap.html | 33 +++-
contributor-guide/specification/index.html | 27 ++-
contributor-guide/specification/invariants.html | 27 ++-
.../specification/output-field-name-semantic.html | 27 ++-
genindex.html | 27 ++-
index.html | 80 +++++----
objects.inv | Bin 780 -> 875 bytes
search.html | 27 ++-
searchindex.js | 2 +-
user-guide/cli.html | 27 ++-
user-guide/{faq.html => comparison.html} | 101 +++++++----
user-guide/configs.html | 27 ++-
user-guide/dataframe.html | 27 ++-
user-guide/example-usage.html | 37 +++-
user-guide/expressions.html | 27 ++-
user-guide/faq.html | 27 ++-
user-guide/{faq.html => integration.html} | 85 ++++++---
user-guide/introduction.html | 33 +++-
user-guide/library.html | 33 +++-
user-guide/sql/aggregate_functions.html | 27 ++-
user-guide/sql/data_types.html | 27 ++-
user-guide/sql/ddl.html | 27 ++-
user-guide/sql/explain.html | 27 ++-
user-guide/sql/index.html | 27 ++-
user-guide/sql/information_schema.html | 27 ++-
user-guide/sql/scalar_functions.html | 27 ++-
user-guide/sql/select.html | 27 ++-
user-guide/sql/sql_status.html | 27 ++-
user-guide/sql/subqueries.html | 27 ++-
user-guide/{faq.html => users.html} | 96 ++++++----
42 files changed, 1310 insertions(+), 356 deletions(-)
diff --git a/_images/DataFusion-Logo-Background-White.png b/_images/DataFusion-Logo-Background-White.png
new file mode 100644
index 0000000000..023c2373fc
Binary files /dev/null and b/_images/DataFusion-Logo-Background-White.png differ
diff --git a/_sources/contributor-guide/architecture.md.txt b/_sources/contributor-guide/architecture.md.txt
new file mode 100644
index 0000000000..3150060ff3
--- /dev/null
+++ b/_sources/contributor-guide/architecture.md.txt
@@ -0,0 +1,26 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Architecture
+
+There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.
+
+- (July 2022): DataFusion and Arrow: Supercharge Your Data Analytical Tool with a Rusty Query Engine: [recording](https://www.youtube.com/watch?v=Rii1VTn3seQ) and [slides](https://docs.google.com/presentation/d/1q1bPibvu64k2b7LPi7Yyb0k3gA1BiUYiUbEklqW1Ckc/view#slide=id.g11054eeab4c_0_1165)
+- (March 2021): The DataFusion architecture is described in _Query Engine Design and the Rust-Based DataFusion in Apache Arrow_: [recording](https://www.youtube.com/watch?v=K6eCAVEk4kU) (DataFusion content starts [~ 15 minutes in](https://www.youtube.com/watch?v=K6eCAVEk4kU&t=875s)) and [slides](https://www.slideshare.net/influxdata/influxdb-iox-tech-talks-query-engine-design-and-the-rustbased-datafusion-in-apache-arrow-244161934)
+- (February 2021): How DataFusion is used within the Ballista Project is described in \*Ballista: Distributed Compute with Rust and Apache Arrow: [recording](https://www.youtube.com/watch?v=ZZHQaOap9pQ)
diff --git a/_sources/contributor-guide/index.md.txt b/_sources/contributor-guide/index.md.txt
index 5d3d02930d..f54d146521 100644
--- a/_sources/contributor-guide/index.md.txt
+++ b/_sources/contributor-guide/index.md.txt
@@ -31,7 +31,9 @@ You can find a curated
[good-first-issue](https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
list to help you get started.
-# Pull Requests
+# Developer's guide
+
+## Pull Requests
We welcome pull requests (PRs) from anyone from the community.
@@ -39,8 +41,6 @@ DataFusion is a very active fast-moving project and we try to review and merge P
Review bandwidth is currently our most limited resource, and we highly encourage reviews by the broader community. If you are waiting for your PR to be reviewed, consider helping review other PRs that are waiting. Such review both helps the reviewer to learn the codebase and become more expert, as well as helps identify issues in the PR (such as lack of test coverage), that can be addressed and make future reviews faster and more efficient.
-## Merging PRs
-
Since we are a worldwide community, we have contributors in many timezones who review and comment. To ensure anyone who wishes has an opportunity to review a PR, our committers try to ensure that at least 24 hours passes between when a "major" PR is approved and when it is merged.
A "major" PR means there is a substantial change in design or a change in the API. Committers apply their best judgment to determine what constitutes a substantial change. A "minor" PR might be merged without a 24 hour delay, again subject to the judgment of the committer. Examples of potential "minor" PRs are:
@@ -50,11 +50,11 @@ A "major" PR means there is a substantial change in design or a change in the AP
3. Non-controversial build-related changes (clippy, version upgrades etc.)
4. Smaller non-controversial feature additions
-# Developer's guide
+## Getting Started
This section describes how you can get started at developing DataFusion.
-## Windows setup
+### Windows setup
```shell
wget https://az792536.vo.msecnd.net/vms/VMBuild_20190311/VirtualBox/MSEdge/MSEdge.Win10.VirtualBox.zip
@@ -63,7 +63,7 @@ git-bash.exe
cargo build
```
-## Protoc Installation
+### Protoc Installation
Compiling DataFusion from sources requires an installed version of the protobuf compiler, `protoc`.
@@ -85,7 +85,7 @@ libprotoc 3.12.4
Alternatively a binary release can be downloaded from the [Release Page](https://github.com/protocolbuffers/protobuf/releases) or [built from source](https://github.com/protocolbuffers/protobuf/blob/main/src/README.md).
-## Bootstrap environment
+### Bootstrap environment
DataFusion is written in Rust and it uses a standard rust toolkit:
@@ -110,7 +110,7 @@ or run them all at once:
- [dev/rust_lint.sh](https://github.com/apache/arrow-datafusion/blob/main/dev/rust_lint.sh)
-## Test Organization
+### Test Organization
DataFusion has several levels of tests in its [Test
Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html)
@@ -118,13 +118,13 @@ and tries to follow [Testing Organization](https://doc.rust-lang.org/book/ch11-0
This section highlights the most important test modules that exist
-### Unit tests
+#### Unit tests
Tests for the code in an individual module are defined in the same source file with a `test` module, following Rust convention
-### Rust Integration Tests
+#### Rust Integration Tests
-There are several tests of the public interface of the DataFusion library in the [tests](https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests) directory.
+There are several tests of the public interface of the DataFusion library in the [tests](https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/tests) directory.
You can run these tests individually using a command such as
@@ -134,16 +134,16 @@ cargo test -p datafusion --tests sql_integration
One very important test is the [sql_integration](https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/sql_integration.rs) test which validates DataFusion's ability to run a large assortment of SQL queries against an assortment of data setups.
-### sqllogictests Tests
+#### sqllogictests Tests
-The [sqllogictests](https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/sqllogictests) also validate DataFusion SQL against an assortment of data setups.
+The [sqllogictests](https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/tests/sqllogictests) also validate DataFusion SQL against an assortment of data setups.
Data Driven tests have many benefits including being easier to write and maintain. We are in the process of [migrating sql_integration tests](https://github.com/apache/arrow-datafusion/issues/4460) and encourage
you to add new tests using sqllogictests if possible.
-## Benchmarks
+### Benchmarks
-### Criterion Benchmarks
+#### Criterion Benchmarks
[Criterion](https://docs.rs/criterion/latest/criterion/index.html) is a statistics-driven micro-benchmarking framework used by DataFusion for evaluating the performance of specific code-paths. In particular, the criterion benchmarks help to both guide optimisation efforts, and prevent performance regressions within DataFusion.
@@ -153,7 +153,7 @@ Criterion integrates with Cargo's built-in [benchmark support](https://doc.rust-
cargo bench --bench BENCHMARK_NAME
```
-A full list of benchmarks can be found [here](https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/benches).
+A full list of benchmarks can be found [here](https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/benches).
_[cargo-criterion](https://github.com/bheisler/cargo-criterion) may also be used for more advanced reporting._
@@ -171,13 +171,15 @@ If the environment variable `PARQUET_FILE` is set, the benchmark will run querie
The benchmark will automatically remove any generated parquet file on exit, however, if interrupted (e.g. by CTRL+C) it will not. This can be useful for analysing the particular file after the fact, or preserving it to use with `PARQUET_FILE` in subsequent runs.
-### Upstream Benchmark Suites
+#### Upstream Benchmark Suites
-Instructions and tooling for running upstream benchmark suites against DataFusion can be found in [benchmarks](https://github.com/apache/arrow-datafusion/blob/main/benchmarks).
+Instructions and tooling for running upstream benchmark suites against DataFusion can be found in [benchmarks](https://github.com/apache/arrow-datafusion/tree/main/benchmarks).
These are valuable for comparative evaluation against alternative Arrow implementations and query engines.
-## How to add a new scalar function
+## HOWTOs
+
+### How to add a new scalar function
Below is a checklist of what you need to do to add a new scalar function to DataFusion:
@@ -197,7 +199,7 @@ Below is a checklist of what you need to do to add a new scalar function to Data
- In [expr/src/expr_fn.rs](https://github.com/apache/arrow-datafusion/blob/main/datafusion/expr/src/expr_fn.rs), add:
- a new entry of the `unary_scalar_expr!` macro for the new function.
-## How to add a new aggregate function
+### How to add a new aggregate function
Below is a checklist of what you need to do to add a new aggregate function to DataFusion:
@@ -215,7 +217,7 @@ Below is a checklist of what you need to do to add a new aggregate function to D
- tests to the function.
- In [tests/sql](https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/sql), add a new test where the function is called through SQL against well known data and returns the expected result.
-## How to display plans graphically
+### How to display plans graphically
The query plans represented by `LogicalPlan` nodes can be graphically
rendered using [Graphviz](https://www.graphviz.org/).
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index 57290d5a26..09071a7511 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -15,12 +15,21 @@
.. specific language governing permissions and limitations
.. under the License.
+.. image:: _static/images/DataFusion-Logo-Background-White.png
+ :alt: DataFusion Logo
+
=======================
Apache Arrow DataFusion
=======================
-Table of Contents
-=================
+DataFusion is a very fast, extensible query engine for building high-quality data-centric systems in
+`Rust <http://rustlang.org>`_, using the `Apache Arrow <https://arrow.apache.org>`_
+in-memory format.
+
+DataFusion offers SQL and Dataframe APIs, excellent
+`performance <https://benchmark.clickhouse.com>`_, built-in support for
+CSV, Parquet, JSON, and Avro, extensive customization, and a great
+community.
.. _toc.guide:
@@ -30,6 +39,9 @@ Table of Contents
user-guide/introduction
user-guide/example-usage
+ user-guide/users
+ user-guide/comparison
+ user-guide/integration
user-guide/library
user-guide/cli
user-guide/dataframe
@@ -47,8 +59,10 @@ Table of Contents
contributor-guide/index
contributor-guide/communication
+ contributor-guide/architecture
contributor-guide/roadmap
contributor-guide/quarterly_roadmap
contributor-guide/specification/index
+ Github <https://github.com/apache/arrow-datafusion>
Issue tracker <https://github.com/apache/arrow-datafusion/issues>
Code of conduct <https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md>
diff --git a/_sources/user-guide/comparison.md.txt b/_sources/user-guide/comparison.md.txt
new file mode 100644
index 0000000000..2cb13f326a
--- /dev/null
+++ b/_sources/user-guide/comparison.md.txt
@@ -0,0 +1,52 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Comparisons to Other Projects
+
+When compared to similar systems, DataFusion typically is:
+
+1. Targeted at developers, rather than end users / data scientists.
+2. Designed to be embedded, rather than a complete file based SQL system.
+3. Governed by the [Apache Software Foundation](https://www.apache.org/) process, rather than a single company or individual.
+4. Implemented in `Rust`, rather than `C/C++`
+
+Here is a comparison with similar projects that may help understand
+when DataFusion might be be suitable and unsuitable for your needs:
+
+- [DuckDB](http://www.duckdb.org) is an open source, in process analytic database.
+ Like DataFusion, it supports very fast execution, both from its custom file format
+ and directly from parquet files. Unlike DataFusion, it is written in C/C++ and it
+ is primarily used directly by users as a serverless database and query system rather
+ than as a library for building such database systems.
+
+- [Polars](http://pola.rs): Polars is one of the fastest DataFrame
+ libraries at the time of writing. Like DataFusion, it is also
+ written in Rust and uses the Apache Arrow memory model, but unlike
+ DataFusion it does not provide SQL nor as many extension points.
+
+- [Facebook Velox](https://engineering.fb.com/2022/08/31/open-source/velox/)
+ is an execution engine. Like DataFusion, Velox aims to
+ provide a reusable foundation for building database-like systems. Unlike DataFusion,
+ it is written in C/C++ and does not include a SQL frontend or planning /optimization
+ framework.
+
+- [Databend](https://github.com/datafuselabs/databend) is a complete
+ database system. Like DataFusion it is also written in Rust and
+ utilizes the Apache Arrow memory model, but unlike DataFusion it
+ targets end-users rather than developers of other database systems.
diff --git a/_sources/user-guide/integration.md.txt b/_sources/user-guide/integration.md.txt
new file mode 100644
index 0000000000..bffa6b1893
--- /dev/null
+++ b/_sources/user-guide/integration.md.txt
@@ -0,0 +1,35 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Integrations and Extensions
+
+There are a number of community projects that extend DataFusion or
+provide integrations with other systems.
+
+## Language Bindings
+
+- [datafusion-c](https://github.com/datafusion-contrib/datafusion-c)
+- [datafusion-python](https://github.com/apache/arrow-datafusion-python)
+- [datafusion-ruby](https://github.com/datafusion-contrib/datafusion-ruby)
+- [datafusion-java](https://github.com/datafusion-contrib/datafusion-java)
+
+## Integrations
+
+- [datafusion-bigtable](https://github.com/datafusion-contrib/datafusion-bigtable)
+- [datafusion-catalogprovider-glue](https://github.com/datafusion-contrib/datafusion-catalogprovider-glue)
diff --git a/_sources/user-guide/introduction.md.txt b/_sources/user-guide/introduction.md.txt
index 55fc59b320..f906eac78c 100644
--- a/_sources/user-guide/introduction.md.txt
+++ b/_sources/user-guide/introduction.md.txt
@@ -17,7 +17,7 @@
under the License.
-->
-# Introduction
+# Features, and Usecases
DataFusion is a very fast, extensible query engine for building
high-quality data-centric systems in [Rust](http://rustlang.org),
diff --git a/_sources/user-guide/users.md.txt b/_sources/user-guide/users.md.txt
new file mode 100644
index 0000000000..0d259c8de3
--- /dev/null
+++ b/_sources/user-guide/users.md.txt
@@ -0,0 +1,67 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Known Users
+
+Here are some of the projects known to use DataFusion:
+
+- [Ballista](https://github.com/apache/arrow-ballista) Distributed SQL Query Engine
+- [Blaze](https://github.com/blaze-init/blaze) Spark accelerator with DataFusion at its core
+- [CeresDB](https://github.com/CeresDB/ceresdb) Distributed Time-Series Database
+- [Cloudfuse Buzz](https://github.com/cloudfuse-io/buzz-rust)
+- [CnosDB](https://github.com/cnosdb/cnosdb) Open Source Distributed Time Series Database
+- [Cube Store](https://github.com/cube-js/cube.js/tree/master/rust)
+- [Dask SQL](https://github.com/dask-contrib/dask-sql) Distributed SQL query engine in Python
+- [datafusion-tui](https://github.com/datafusion-contrib/datafusion-tui) Text UI for DataFusion
+- [delta-rs](https://github.com/delta-io/delta-rs) Native Rust implementation of Delta Lake
+- [Flock](https://github.com/flock-lab/flock)
+- [GreptimeDB](https://github.com/GreptimeTeam/greptimedb) Open Source & Cloud Native Distributed Time Series Database
+- [InfluxDB IOx](https://github.com/influxdata/influxdb_iox) Time Series Database
+- [Kamu](https://github.com/kamu-data/kamu-cli/) Planet-scale streaming data pipeline
+- [Parseable](https://github.com/parseablehq/parseable) Log storage and observability platform
+- [qv](https://github.com/timvw/qv) Quickly view your data
+- [ROAPI](https://github.com/roapi/roapi)
+- [Seafowl](https://github.com/splitgraph/seafowl) CDN-friendly analytical database
+- [Synnada](https://synnada.ai/) Streaming-first framework for data products
+- [Tensorbase](https://github.com/tensorbase/tensorbase)
+- [VegaFusion](https://vegafusion.io/) Server-side acceleration for the [Vega](https://vega.github.io/) visualization grammar
+- [ZincObserve](https://github.com/zinclabs/zincobserve) Distributed cloud native observability platform
+
+[ballista]: https://github.com/apache/arrow-ballista
+[blaze]: https://github.com/blaze-init/blaze
+[ceresdb]: https://github.com/CeresDB/ceresdb
+[cloudfuse buzz]: https://github.com/cloudfuse-io/buzz-rust
+[cnosdb]: https://github.com/cnosdb/cnosdb
+[cube store]: https://github.com/cube-js/cube.js/tree/master/rust
+[dask sql]: https://github.com/dask-contrib/dask-sql
+[datafusion-tui]: https://github.com/datafusion-contrib/datafusion-tui
+[delta-rs]: https://github.com/delta-io/delta-rs
+[flock]: https://github.com/flock-lab/flock
+[kamu]: https://github.com/kamu-data/kamu-cli
+[greptime db]: https://github.com/GreptimeTeam/greptimedb
+[influxdb iox]: https://github.com/influxdata/influxdb_iox
+[parseable]: https://github.com/parseablehq/parseable
+[prql-query]: https://github.com/prql/prql-query
+[qv]: https://github.com/timvw/qv
+[roapi]: https://github.com/roapi/roapi
+[seafowl]: https://github.com/splitgraph/seafowl
+[synnada]: https://synnada.ai/
+[tensorbase]: https://github.com/tensorbase/tensorbase
+[vegafusion]: https://vegafusion.io/
+[zincobserve]: https://github.com/zinclabs/zincobserve "if you know of another project, please submit a PR to add a link!"
diff --git a/contributor-guide/communication.html b/contributor-guide/architecture.html
similarity index 66%
copy from contributor-guide/communication.html
copy to contributor-guide/architecture.html
index 161be1fd62..cff8bd234e 100644
--- a/contributor-guide/communication.html
+++ b/contributor-guide/architecture.html
@@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
- <title>Communication — Arrow DataFusion documentation</title>
+ <title>Architecture — Arrow DataFusion documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
@@ -35,7 +35,7 @@
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Roadmap" href="roadmap.html" />
- <link rel="prev" title="Introduction" href="index.html" />
+ <link rel="prev" title="Communication" href="communication.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -74,7 +74,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/library.html">
Using DataFusion as a library
@@ -186,9 +201,14 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="communication.html">
+ Communication
+ </a>
+ </li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Communication
+ Architecture
</a>
</li>
<li class="toctree-l1">
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -256,36 +281,9 @@
<div class="toc-item">
-<div class="tocsection onthispage pt-5 pb-3">
- <i class="fas fa-list"></i> On this page
-</div>
<nav id="bd-toc-nav">
- <ul class="visible nav section-nav flex-column">
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#questions">
- Questions?
- </a>
- <ul class="nav section-nav flex-column">
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#mailing-list">
- Mailing list
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#slack-and-discord">
- Slack and Discord
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#sync-up-video-calls">
- Sync up video calls
- </a>
- </li>
- </ul>
- </li>
-</ul>
-
+
</nav>
</div>
@@ -293,7 +291,7 @@
<div class="tocsection editthispage">
- <a href="https://github.com/apache/arrow-datafusion/edit/main/docs/source/contributor-guide/communication.md">
+ <a href="https://github.com/apache/arrow-datafusion/edit/main/docs/source/contributor-guide/architecture.md">
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -330,55 +328,14 @@
specific language governing permissions and limitations
under the License.
-->
-<section id="communication">
-<h1>Communication<a class="headerlink" href="#communication" title="Permalink to this heading">¶</a></h1>
-<p>We welcome participation from everyone and encourage you to join us, ask
-questions, and get involved.</p>
-<p>All participation in the Apache Arrow DataFusion project is governed by the
-Apache Software Foundation’s <a class="reference external" href="https://www.apache.org/foundation/policies/conduct.html">code of
-conduct</a>.</p>
-<p>The vast majority of communication occurs in the open on our
-<a class="reference external" href="https://github.com/apache/arrow-datafusion">github repository</a>.</p>
-<section id="questions">
-<h2>Questions?<a class="headerlink" href="#questions" title="Permalink to this heading">¶</a></h2>
-<section id="mailing-list">
-<h3>Mailing list<a class="headerlink" href="#mailing-list" title="Permalink to this heading">¶</a></h3>
-<p>We use arrow.apache.org’s <code class="docutils literal notranslate"><span class="pre">dev@</span></code> mailing list for project management, release
-coordination and design discussions
-(<a class="reference external" href="mailto:dev-subscribe%40arrow.apache.org">subscribe</a>,
-<a class="reference external" href="mailto:dev-unsubscribe%40arrow.apache.org">unsubscribe</a>,
-<a class="reference external" href="https://lists.apache.org/list.html?dev@arrow.apache.org">archives</a>).</p>
-<p>When emailing the dev list, please make sure to prefix the subject line with a
-<code class="docutils literal notranslate"><span class="pre">[DataFusion]</span></code> tag, e.g. <code class="docutils literal notranslate"><span class="pre">"[DataFusion]</span> <span class="pre">New</span> <span class="pre">API</span> <span class="pre">for</span> <span class="pre">remote</span> <span class="pre">data</span> <span class="pre">sources"</span></code>, so
-that the appropriate people in the Apache Arrow community notice the message.</p>
-</section>
-<section id="slack-and-discord">
-<h3>Slack and Discord<a class="headerlink" href="#slack-and-discord" title="Permalink to this heading">¶</a></h3>
-<p>We use the official <a class="reference external" href="https://s.apache.org/slack-invite">ASF</a> Slack workspace
-for informal discussions and coordination. This is a great place to meet other
-contributors and get guidance on where to contribute. Join us in the
-<code class="docutils literal notranslate"><span class="pre">#arrow-rust</span></code> channel.</p>
-<p>We also have a backup Arrow Rust Discord
-server (<a class="reference external" href="https://discord.gg/Qw5gKqHxUM">invite link</a>) in case you are not able
-to join the Slack workspace. If you need an invite to the Slack workspace, you
-can also ask for one in our Discord server.</p>
-</section>
-<section id="sync-up-video-calls">
-<h3>Sync up video calls<a class="headerlink" href="#sync-up-video-calls" title="Permalink to this heading">¶</a></h3>
-<p>We have biweekly sync calls every other Thursdays at both 04:00 UTC
-and 16:00 UTC (starting September 30, 2021) depending on if there are
-items on the agenda to discuss and someone being willing to host.</p>
-<p>Please see the <a class="reference external" href="https://docs.google.com/document/d/1atCVnoff5SR4eM4Lwf2M1BBJTY6g3_HUNR6qswYJW_U/edit">agenda</a>
-for the video call link, add topics and to see what others plan to discuss.</p>
-<p>The goals of these calls are:</p>
-<ol class="arabic simple">
-<li><p>Help “put a face to the name” of some of other contributors we are working with</p></li>
-<li><p>Discuss / synchronize on the goals and major initiatives from different stakeholders to identify areas where more alignment is needed</p></li>
-</ol>
-<p>No decisions are made on the call and anything of substance will be discussed on the mailing list or in github issues / google docs.</p>
-<p>We will send a summary of all sync ups to the dev@arrow.apache.org mailing list.</p>
-</section>
-</section>
+<section id="architecture">
+<h1>Architecture<a class="headerlink" href="#architecture" title="Permalink to this heading">¶</a></h1>
+<p>There is no formal document describing DataFusion’s architecture yet, but the following presentations offer a good overview of its different components and how they interact together.</p>
+<ul class="simple">
+<li><p>(July 2022): DataFusion and Arrow: Supercharge Your Data Analytical Tool with a Rusty Query Engine: <a class="reference external" href="https://www.youtube.com/watch?v=Rii1VTn3seQ">recording</a> and <a class="reference external" href="https://docs.google.com/presentation/d/1q1bPibvu64k2b7LPi7Yyb0k3gA1BiUYiUbEklqW1Ckc/view#slide=id.g11054eeab4c_0_1165">slides</a></p></li>
+<li><p>(March 2021): The DataFusion architecture is described in <em>Query Engine Design and the Rust-Based DataFusion in Apache Arrow</em>: <a class="reference external" href="https://www.youtube.com/watch?v=K6eCAVEk4kU">recording</a> (DataFusion content starts <a class="reference external" href="https://www.youtube.com/watch?v=K6eCAVEk4kU&amp;t=875s">~ 15 minutes in</a>) and <a class="reference external" href="https://www.slideshare.net/influxdata/influxdb-iox-tech-talks-query-engi [...]
+<li><p>(February 2021): How DataFusion is used within the Ballista Project is described in *Ballista: Distributed Compute with Rust and Apache Arrow: <a class="reference external" href="https://www.youtube.com/watch?v=ZZHQaOap9pQ">recording</a></p></li>
+</ul>
</section>
@@ -387,11 +344,11 @@ for the video call link, add topics and to see what others plan to discuss.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="index.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="communication.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Introduction</p>
+ <p class="prev-next-title">Communication</p>
</div>
</a>
<a class='right-next' id="next-link" href="roadmap.html" title="next page">
diff --git a/contributor-guide/communication.html b/contributor-guide/communication.html
index 161be1fd62..b63fd04c95 100644
--- a/contributor-guide/communication.html
+++ b/contributor-guide/communication.html
@@ -34,7 +34,7 @@
<script src="../_static/sphinx_highlight.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Roadmap" href="roadmap.html" />
+ <link rel="next" title="Architecture" href="architecture.html" />
<link rel="prev" title="Introduction" href="index.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -74,7 +74,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -394,10 +419,10 @@ for the video call link, add topics and to see what others plan to discuss.</p>
<p class="prev-next-title">Introduction</p>
</div>
</a>
- <a class='right-next' id="next-link" href="roadmap.html" title="next page">
+ <a class='right-next' id="next-link" href="architecture.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Roadmap</p>
+ <p class="prev-next-title">Architecture</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/contributor-guide/index.html b/contributor-guide/index.html
index 0268c1d52a..acfe935284 100644
--- a/contributor-guide/index.html
+++ b/contributor-guide/index.html
@@ -74,7 +74,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -267,98 +292,103 @@
Introduction
</a>
</li>
- <li class="toc-h1 nav-item toc-entry">
- <a class="reference internal nav-link" href="#pull-requests">
- Pull Requests
- </a>
- <ul class="visible nav section-nav flex-column">
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#merging-prs">
- Merging PRs
- </a>
- </li>
- </ul>
- </li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#developer-s-guide">
Developer’s guide
</a>
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#windows-setup">
- Windows setup
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#protoc-installation">
- Protoc Installation
+ <a class="reference internal nav-link" href="#pull-requests">
+ Pull Requests
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#bootstrap-environment">
- Bootstrap environment
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#test-organization">
- Test Organization
+ <a class="reference internal nav-link" href="#getting-started">
+ Getting Started
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#unit-tests">
- Unit tests
+ <a class="reference internal nav-link" href="#windows-setup">
+ Windows setup
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#rust-integration-tests">
- Rust Integration Tests
+ <a class="reference internal nav-link" href="#protoc-installation">
+ Protoc Installation
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#sqllogictests-tests">
- sqllogictests Tests
+ <a class="reference internal nav-link" href="#bootstrap-environment">
+ Bootstrap environment
</a>
</li>
- </ul>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#benchmarks">
- Benchmarks
- </a>
- <ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#criterion-benchmarks">
- Criterion Benchmarks
+ <a class="reference internal nav-link" href="#test-organization">
+ Test Organization
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#parquet-sql-benchmarks">
- Parquet SQL Benchmarks
+ <a class="reference internal nav-link" href="#unit-tests">
+ Unit tests
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#rust-integration-tests">
+ Rust Integration Tests
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#sqllogictests-tests">
+ sqllogictests Tests
</a>
</li>
</ul>
</li>
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#upstream-benchmark-suites">
- Upstream Benchmark Suites
+ <a class="reference internal nav-link" href="#benchmarks">
+ Benchmarks
</a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#criterion-benchmarks">
+ Criterion Benchmarks
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#parquet-sql-benchmarks">
+ Parquet SQL Benchmarks
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#upstream-benchmark-suites">
+ Upstream Benchmark Suites
+ </a>
+ </li>
+ </ul>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#how-to-add-a-new-scalar-function">
- How to add a new scalar function
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#how-to-add-a-new-aggregate-function">
- How to add a new aggregate function
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#how-to-display-plans-graphically">
- How to display plans graphically
+ <a class="reference internal nav-link" href="#howtos">
+ HOWTOs
</a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#how-to-add-a-new-scalar-function">
+ How to add a new scalar function
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#how-to-add-a-new-aggregate-function">
+ How to add a new aggregate function
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#how-to-display-plans-graphically">
+ How to display plans graphically
+ </a>
+ </li>
+ </ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#specifications">
@@ -437,13 +467,13 @@
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">good-first-issue</a>
list to help you get started.</p>
</section>
+<section id="developer-s-guide">
+<h1>Developer’s guide<a class="headerlink" href="#developer-s-guide" title="Permalink to this heading">¶</a></h1>
<section id="pull-requests">
-<h1>Pull Requests<a class="headerlink" href="#pull-requests" title="Permalink to this heading">¶</a></h1>
+<h2>Pull Requests<a class="headerlink" href="#pull-requests" title="Permalink to this heading">¶</a></h2>
<p>We welcome pull requests (PRs) from anyone from the community.</p>
<p>DataFusion is a very active fast-moving project and we try to review and merge PRs quickly to keep the review backlog down and the pace up. After review and approval, one of the <a class="reference external" href="https://arrow.apache.org/committers/">many people with commit access</a> will merge your PR.</p>
<p>Review bandwidth is currently our most limited resource, and we highly encourage reviews by the broader community. If you are waiting for your PR to be reviewed, consider helping review other PRs that are waiting. Such review both helps the reviewer to learn the codebase and become more expert, as well as helps identify issues in the PR (such as lack of test coverage), that can be addressed and make future reviews faster and more efficient.</p>
-<section id="merging-prs">
-<h2>Merging PRs<a class="headerlink" href="#merging-prs" title="Permalink to this heading">¶</a></h2>
<p>Since we are a worldwide community, we have contributors in many timezones who review and comment. To ensure anyone who wishes has an opportunity to review a PR, our committers try to ensure that at least 24 hours passes between when a “major” PR is approved and when it is merged.</p>
<p>A “major” PR means there is a substantial change in design or a change in the API. Committers apply their best judgment to determine what constitutes a substantial change. A “minor” PR might be merged without a 24 hour delay, again subject to the judgment of the committer. Examples of potential “minor” PRs are:</p>
<ol class="arabic simple">
@@ -453,12 +483,11 @@ list to help you get started.</p>
<li><p>Smaller non-controversial feature additions</p></li>
</ol>
</section>
-</section>
-<section id="developer-s-guide">
-<h1>Developer’s guide<a class="headerlink" href="#developer-s-guide" title="Permalink to this heading">¶</a></h1>
+<section id="getting-started">
+<h2>Getting Started<a class="headerlink" href="#getting-started" title="Permalink to this heading">¶</a></h2>
<p>This section describes how you can get started at developing DataFusion.</p>
<section id="windows-setup">
-<h2>Windows setup<a class="headerlink" href="#windows-setup" title="Permalink to this heading">¶</a></h2>
+<h3>Windows setup<a class="headerlink" href="#windows-setup" title="Permalink to this heading">¶</a></h3>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>wget<span class="w"> </span>https://az792536.vo.msecnd.net/vms/VMBuild_20190311/VirtualBox/MSEdge/MSEdge.Win10.VirtualBox.zip
choco<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>git<span class="w"> </span>rustup.install<span class="w"> </span>visualcpp-build-tools
git-bash.exe
@@ -467,7 +496,7 @@ cargo<span class="w"> </span>build
</div>
</section>
<section id="protoc-installation">
-<h2>Protoc Installation<a class="headerlink" href="#protoc-installation" title="Permalink to this heading">¶</a></h2>
+<h3>Protoc Installation<a class="headerlink" href="#protoc-installation" title="Permalink to this heading">¶</a></h3>
<p>Compiling DataFusion from sources requires an installed version of the protobuf compiler, <code class="docutils literal notranslate"><span class="pre">protoc</span></code>.</p>
<p>On most platforms this can be installed from your system’s package manager</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ apt install -y protobuf-compiler
@@ -484,7 +513,7 @@ libprotoc<span class="w"> </span><span class="m">3</span>.12.4
<p>Alternatively a binary release can be downloaded from the <a class="reference external" href="https://github.com/protocolbuffers/protobuf/releases">Release Page</a> or <a class="reference external" href="https://github.com/protocolbuffers/protobuf/blob/main/src/README.md">built from source</a>.</p>
</section>
<section id="bootstrap-environment">
-<h2>Bootstrap environment<a class="headerlink" href="#bootstrap-environment" title="Permalink to this heading">¶</a></h2>
+<h3>Bootstrap environment<a class="headerlink" href="#bootstrap-environment" title="Permalink to this heading">¶</a></h3>
<p>DataFusion is written in Rust and it uses a standard rust toolkit:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">cargo</span> <span class="pre">build</span></code></p></li>
@@ -510,18 +539,18 @@ libprotoc<span class="w"> </span><span class="m">3</span>.12.4
</ul>
</section>
<section id="test-organization">
-<h2>Test Organization<a class="headerlink" href="#test-organization" title="Permalink to this heading">¶</a></h2>
+<h3>Test Organization<a class="headerlink" href="#test-organization" title="Permalink to this heading">¶</a></h3>
<p>DataFusion has several levels of tests in its <a class="reference external" href="https://martinfowler.com/articles/practical-test-pyramid.html">Test
Pyramid</a>
and tries to follow <a class="reference external" href="https://doc.rust-lang.org/book/ch11-03-test-organization.html">Testing Organization</a> in the The Book.</p>
<p>This section highlights the most important test modules that exist</p>
<section id="unit-tests">
-<h3>Unit tests<a class="headerlink" href="#unit-tests" title="Permalink to this heading">¶</a></h3>
+<h4>Unit tests<a class="headerlink" href="#unit-tests" title="Permalink to this heading">¶</a></h4>
<p>Tests for the code in an individual module are defined in the same source file with a <code class="docutils literal notranslate"><span class="pre">test</span></code> module, following Rust convention</p>
</section>
<section id="rust-integration-tests">
-<h3>Rust Integration Tests<a class="headerlink" href="#rust-integration-tests" title="Permalink to this heading">¶</a></h3>
-<p>There are several tests of the public interface of the DataFusion library in the <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests">tests</a> directory.</p>
+<h4>Rust Integration Tests<a class="headerlink" href="#rust-integration-tests" title="Permalink to this heading">¶</a></h4>
+<p>There are several tests of the public interface of the DataFusion library in the <a class="reference external" href="https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/tests">tests</a> directory.</p>
<p>You can run these tests individually using a command such as</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>cargo<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-p<span class="w"> </span>datafusion<span class="w"> </span>--tests<span class="w"> </span>sql_integration
</pre></div>
@@ -529,23 +558,24 @@ and tries to follow <a class="reference external" href="https://doc.rust-lang.or
<p>One very important test is the <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/sql_integration.rs">sql_integration</a> test which validates DataFusion’s ability to run a large assortment of SQL queries against an assortment of data setups.</p>
</section>
<section id="sqllogictests-tests">
-<h3>sqllogictests Tests<a class="headerlink" href="#sqllogictests-tests" title="Permalink to this heading">¶</a></h3>
-<p>The <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/sqllogictests">sqllogictests</a> also validate DataFusion SQL against an assortment of data setups.</p>
+<h4>sqllogictests Tests<a class="headerlink" href="#sqllogictests-tests" title="Permalink to this heading">¶</a></h4>
+<p>The <a class="reference external" href="https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/tests/sqllogictests">sqllogictests</a> also validate DataFusion SQL against an assortment of data setups.</p>
<p>Data Driven tests have many benefits including being easier to write and maintain. We are in the process of <a class="reference external" href="https://github.com/apache/arrow-datafusion/issues/4460">migrating sql_integration tests</a> and encourage
you to add new tests using sqllogictests if possible.</p>
</section>
</section>
<section id="benchmarks">
-<h2>Benchmarks<a class="headerlink" href="#benchmarks" title="Permalink to this heading">¶</a></h2>
+<h3>Benchmarks<a class="headerlink" href="#benchmarks" title="Permalink to this heading">¶</a></h3>
<section id="criterion-benchmarks">
-<h3>Criterion Benchmarks<a class="headerlink" href="#criterion-benchmarks" title="Permalink to this heading">¶</a></h3>
+<h4>Criterion Benchmarks<a class="headerlink" href="#criterion-benchmarks" title="Permalink to this heading">¶</a></h4>
<p><a class="reference external" href="https://docs.rs/criterion/latest/criterion/index.html">Criterion</a> is a statistics-driven micro-benchmarking framework used by DataFusion for evaluating the performance of specific code-paths. In particular, the criterion benchmarks help to both guide optimisation efforts, and prevent performance regressions within DataFusion.</p>
<p>Criterion integrates with Cargo’s built-in <a class="reference external" href="https://doc.rust-lang.org/cargo/commands/cargo-bench.html">benchmark support</a> and a given benchmark can be run with</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cargo</span> <span class="n">bench</span> <span class="o">--</span><span class="n">bench</span> <span class="n">BENCHMARK_NAME</span>
</pre></div>
</div>
-<p>A full list of benchmarks can be found <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/benches">here</a>.</p>
+<p>A full list of benchmarks can be found <a class="reference external" href="https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/benches">here</a>.</p>
<p><em><a class="reference external" href="https://github.com/bheisler/cargo-criterion">cargo-criterion</a> may also be used for more advanced reporting.</em></p>
+</section>
<section id="parquet-sql-benchmarks">
<h4>Parquet SQL Benchmarks<a class="headerlink" href="#parquet-sql-benchmarks" title="Permalink to this heading">¶</a></h4>
<p>The parquet SQL benchmarks can be run with</p>
@@ -556,15 +586,17 @@ you to add new tests using sqllogictests if possible.</p>
<p>If the environment variable <code class="docutils literal notranslate"><span class="pre">PARQUET_FILE</span></code> is set, the benchmark will run queries against this file instead of a randomly generated one. This can be useful for performing multiple runs, potentially with different code, against the same source data, or for testing against a custom dataset.</p>
<p>The benchmark will automatically remove any generated parquet file on exit, however, if interrupted (e.g. by CTRL+C) it will not. This can be useful for analysing the particular file after the fact, or preserving it to use with <code class="docutils literal notranslate"><span class="pre">PARQUET_FILE</span></code> in subsequent runs.</p>
</section>
-</section>
<section id="upstream-benchmark-suites">
-<h3>Upstream Benchmark Suites<a class="headerlink" href="#upstream-benchmark-suites" title="Permalink to this heading">¶</a></h3>
-<p>Instructions and tooling for running upstream benchmark suites against DataFusion can be found in <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/benchmarks">benchmarks</a>.</p>
+<h4>Upstream Benchmark Suites<a class="headerlink" href="#upstream-benchmark-suites" title="Permalink to this heading">¶</a></h4>
+<p>Instructions and tooling for running upstream benchmark suites against DataFusion can be found in <a class="reference external" href="https://github.com/apache/arrow-datafusion/tree/main/benchmarks">benchmarks</a>.</p>
<p>These are valuable for comparative evaluation against alternative Arrow implementations and query engines.</p>
</section>
</section>
+</section>
+<section id="howtos">
+<h2>HOWTOs<a class="headerlink" href="#howtos" title="Permalink to this heading">¶</a></h2>
<section id="how-to-add-a-new-scalar-function">
-<h2>How to add a new scalar function<a class="headerlink" href="#how-to-add-a-new-scalar-function" title="Permalink to this heading">¶</a></h2>
+<h3>How to add a new scalar function<a class="headerlink" href="#how-to-add-a-new-scalar-function" title="Permalink to this heading">¶</a></h3>
<p>Below is a checklist of what you need to do to add a new scalar function to DataFusion:</p>
<ul class="simple">
<li><p>Add the actual implementation of the function:</p>
@@ -594,7 +626,7 @@ you to add new tests using sqllogictests if possible.</p>
</ul>
</section>
<section id="how-to-add-a-new-aggregate-function">
-<h2>How to add a new aggregate function<a class="headerlink" href="#how-to-add-a-new-aggregate-function" title="Permalink to this heading">¶</a></h2>
+<h3>How to add a new aggregate function<a class="headerlink" href="#how-to-add-a-new-aggregate-function" title="Permalink to this heading">¶</a></h3>
<p>Below is a checklist of what you need to do to add a new aggregate function to DataFusion:</p>
<ul class="simple">
<li><p>Add the actual implementation of an <code class="docutils literal notranslate"><span class="pre">Accumulator</span></code> and <code class="docutils literal notranslate"><span class="pre">AggregateExpr</span></code>:</p>
@@ -619,7 +651,7 @@ you to add new tests using sqllogictests if possible.</p>
</ul>
</section>
<section id="how-to-display-plans-graphically">
-<h2>How to display plans graphically<a class="headerlink" href="#how-to-display-plans-graphically" title="Permalink to this heading">¶</a></h2>
+<h3>How to display plans graphically<a class="headerlink" href="#how-to-display-plans-graphically" title="Permalink to this heading">¶</a></h3>
<p>The query plans represented by <code class="docutils literal notranslate"><span class="pre">LogicalPlan</span></code> nodes can be graphically
rendered using <a class="reference external" href="https://www.graphviz.org/">Graphviz</a>.</p>
<p>To do so, save the output of the <code class="docutils literal notranslate"><span class="pre">display_graphviz</span></code> function to a file.:</p>
@@ -635,6 +667,7 @@ can be displayed. For example, the following command creates a
</pre></div>
</div>
</section>
+</section>
<section id="specifications">
<h2>Specifications<a class="headerlink" href="#specifications" title="Permalink to this heading">¶</a></h2>
<p>We formalize DataFusion semantics and behaviors through specification
diff --git a/contributor-guide/quarterly_roadmap.html b/contributor-guide/quarterly_roadmap.html
index fdf84b6e50..93fff81114 100644
--- a/contributor-guide/quarterly_roadmap.html
+++ b/contributor-guide/quarterly_roadmap.html
@@ -74,7 +74,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/contributor-guide/roadmap.html b/contributor-guide/roadmap.html
index fb3050c83a..fa137bdb9c 100644
--- a/contributor-guide/roadmap.html
+++ b/contributor-guide/roadmap.html
@@ -35,7 +35,7 @@
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Quarterly Roadmap" href="quarterly_roadmap.html" />
- <link rel="prev" title="Communication" href="communication.html" />
+ <link rel="prev" title="Architecture" href="architecture.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -74,7 +74,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -496,11 +521,11 @@ be smaller until execution time.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="communication.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="architecture.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Communication</p>
+ <p class="prev-next-title">Architecture</p>
</div>
</a>
<a class='right-next' id="next-link" href="quarterly_roadmap.html" title="next page">
diff --git a/contributor-guide/specification/index.html b/contributor-guide/specification/index.html
index fd297a1374..f80b7ce737 100644
--- a/contributor-guide/specification/index.html
+++ b/contributor-guide/specification/index.html
@@ -74,7 +74,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../../user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../user-guide/library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/contributor-guide/specification/invariants.html b/contributor-guide/specification/invariants.html
index 8bb5cc4531..5287dcc7cf 100644
--- a/contributor-guide/specification/invariants.html
+++ b/contributor-guide/specification/invariants.html
@@ -74,7 +74,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../../user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../user-guide/library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/contributor-guide/specification/output-field-name-semantic.html b/contributor-guide/specification/output-field-name-semantic.html
index ebc56735ae..3488b5ae25 100644
--- a/contributor-guide/specification/output-field-name-semantic.html
+++ b/contributor-guide/specification/output-field-name-semantic.html
@@ -73,7 +73,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../../user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -81,6 +81,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../user-guide/library.html">
Using DataFusion as a library
@@ -190,6 +205,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../roadmap.html">
Roadmap
@@ -222,6 +242,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/genindex.html b/genindex.html
index d720beaa92..b5cbec51d8 100644
--- a/genindex.html
+++ b/genindex.html
@@ -71,7 +71,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -79,6 +79,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/library.html">
Using DataFusion as a library
@@ -188,6 +203,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="contributor-guide/roadmap.html">
Roadmap
@@ -220,6 +240,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/index.html b/index.html
index a731d3cf5b..0d6caf42f0 100644
--- a/index.html
+++ b/index.html
@@ -34,7 +34,7 @@
<script src="_static/sphinx_highlight.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
- <link rel="next" title="Introduction" href="user-guide/introduction.html" />
+ <link rel="next" title="Features, and Usecases" href="user-guide/introduction.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -73,7 +73,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -81,6 +81,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/library.html">
Using DataFusion as a library
@@ -190,6 +205,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="contributor-guide/roadmap.html">
Roadmap
@@ -222,6 +242,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -255,21 +280,9 @@
<div class="toc-item">
-<div class="tocsection onthispage pt-5 pb-3">
- <i class="fas fa-list"></i> On this page
-</div>
<nav id="bd-toc-nav">
- <ul class="visible nav section-nav flex-column">
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#table-of-contents">
- Table of Contents
- </a>
- <ul class="nav section-nav flex-column">
- </ul>
- </li>
-</ul>
-
+
</nav>
</div>
@@ -296,16 +309,25 @@
<div>
- <section id="apache-arrow-datafusion">
+ <img alt="DataFusion Logo" src="_images/DataFusion-Logo-Background-White.png" />
+<section id="apache-arrow-datafusion">
<h1>Apache Arrow DataFusion<a class="headerlink" href="#apache-arrow-datafusion" title="Permalink to this heading">¶</a></h1>
-<section id="table-of-contents">
-<h2>Table of Contents<a class="headerlink" href="#table-of-contents" title="Permalink to this heading">¶</a></h2>
+<p>DataFusion is a very fast, extensible query engine for building high-quality data-centric systems in
+<a class="reference external" href="http://rustlang.org">Rust</a>, using the <a class="reference external" href="https://arrow.apache.org">Apache Arrow</a>
+in-memory format.</p>
+<p>DataFusion offers SQL and Dataframe APIs, excellent
+<a class="reference external" href="https://benchmark.clickhouse.com">performance</a>, built-in support for
+CSV, Parquet, JSON, and Avro, extensive customization, and a great
+community.</p>
<div class="toctree-wrapper compound" id="toc-guide">
<p aria-level="2" class="caption" role="heading"><span class="caption-text">User Guide</span></p>
<ul>
-<li class="toctree-l1"><a class="reference internal" href="user-guide/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/introduction.html">Features, and Usecases</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-guide/example-usage.html">Example Usage</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-guide/example-usage.html#identifiers-and-capitalization">Identifiers and Capitalization</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/users.html">Known Users</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/comparison.html">Comparisons to Other Projects</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/integration.html">Integrations and Extensions</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-guide/library.html">Using DataFusion as a library</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-guide/cli.html">DataFusion Command-line SQL Utility</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-guide/dataframe.html">DataFrame API</a></li>
@@ -320,19 +342,10 @@
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Contributor Guide</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="contributor-guide/index.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="contributor-guide/index.html#pull-requests">Pull Requests</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#merging-prs">Merging PRs</a></li>
-</ul>
-</li>
<li class="toctree-l1"><a class="reference internal" href="contributor-guide/index.html#developer-s-guide">Developer’s guide</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#windows-setup">Windows setup</a></li>
-<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#protoc-installation">Protoc Installation</a></li>
-<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#bootstrap-environment">Bootstrap environment</a></li>
-<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#test-organization">Test Organization</a></li>
-<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#benchmarks">Benchmarks</a></li>
-<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#how-to-add-a-new-scalar-function">How to add a new scalar function</a></li>
-<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#how-to-add-a-new-aggregate-function">How to add a new aggregate function</a></li>
-<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#how-to-display-plans-graphically">How to display plans graphically</a></li>
+<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#pull-requests">Pull Requests</a></li>
+<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#getting-started">Getting Started</a></li>
+<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#howtos">HOWTOs</a></li>
<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#specifications">Specifications</a></li>
<li class="toctree-l2"><a class="reference internal" href="contributor-guide/index.html#how-to-format-md-document">How to format <code class="docutils literal notranslate"><span class="pre">.md</span></code> document</a></li>
</ul>
@@ -341,6 +354,7 @@
<li class="toctree-l2"><a class="reference internal" href="contributor-guide/communication.html#questions">Questions?</a></li>
</ul>
</li>
+<li class="toctree-l1"><a class="reference internal" href="contributor-guide/architecture.html">Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributor-guide/roadmap.html">Roadmap</a><ul>
<li class="toctree-l2"><a class="reference internal" href="contributor-guide/roadmap.html#datafusion">DataFusion</a></li>
<li class="toctree-l2"><a class="reference internal" href="contributor-guide/roadmap.html#ballista">Ballista</a></li>
@@ -355,11 +369,11 @@
<li class="toctree-l2"><a class="reference internal" href="contributor-guide/specification/output-field-name-semantic.html">Output field name semantics</a></li>
</ul>
</li>
+<li class="toctree-l1"><a class="reference external" href="https://github.com/apache/arrow-datafusion">Github</a></li>
<li class="toctree-l1"><a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">Issue tracker</a></li>
<li class="toctree-l1"><a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">Code of conduct</a></li>
</ul>
</div>
-</section>
</section>
@@ -371,7 +385,7 @@
<a class='right-next' id="next-link" href="user-guide/introduction.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Introduction</p>
+ <p class="prev-next-title">Features, and Usecases</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/objects.inv b/objects.inv
index 6e01b94920..447eb59a00 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/search.html b/search.html
index d7b7caa0ab..0ac0e5e0be 100644
--- a/search.html
+++ b/search.html
@@ -77,7 +77,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -85,6 +85,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/library.html">
Using DataFusion as a library
@@ -194,6 +209,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="contributor-guide/roadmap.html">
Roadmap
@@ -226,6 +246,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/searchindex.js b/searchindex.js
index 46ef778dd2..fd2b54686a 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"docnames": ["contributor-guide/communication", "contributor-guide/index", "contributor-guide/quarterly_roadmap", "contributor-guide/roadmap", "contributor-guide/specification/index", "contributor-guide/specification/invariants", "contributor-guide/specification/output-field-name-semantic", "index", "user-guide/cli", "user-guide/configs", "user-guide/dataframe", "user-guide/example-usage", "user-guide/expressions", "user-guide/faq", "user-guide/introduction", "user-guide [...]
\ No newline at end of file
+Search.setIndex({"docnames": ["contributor-guide/architecture", "contributor-guide/communication", "contributor-guide/index", "contributor-guide/quarterly_roadmap", "contributor-guide/roadmap", "contributor-guide/specification/index", "contributor-guide/specification/invariants", "contributor-guide/specification/output-field-name-semantic", "index", "user-guide/cli", "user-guide/comparison", "user-guide/configs", "user-guide/dataframe", "user-guide/example-usage", "user-guide/expressions [...]
\ No newline at end of file
diff --git a/user-guide/cli.html b/user-guide/cli.html
index f47163c98f..b117f04043 100644
--- a/user-guide/cli.html
+++ b/user-guide/cli.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/faq.html b/user-guide/comparison.html
similarity index 72%
copy from user-guide/faq.html
copy to user-guide/comparison.html
index 30c5202403..e9d89a0d5f 100644
--- a/user-guide/faq.html
+++ b/user-guide/comparison.html
@@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
- <title>Frequently Asked Questions — Arrow DataFusion documentation</title>
+ <title>Comparisons to Other Projects — Arrow DataFusion documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
@@ -34,8 +34,8 @@
<script src="../_static/sphinx_highlight.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Introduction" href="../contributor-guide/index.html" />
- <link rel="prev" title="Configuration Settings" href="configs.html" />
+ <link rel="next" title="Integrations and Extensions" href="integration.html" />
+ <link rel="prev" title="Known Users" href="users.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -164,8 +179,8 @@
Configuration Settings
</a>
</li>
- <li class="toctree-l1 current active">
- <a class="current reference internal" href="#">
+ <li class="toctree-l1">
+ <a class="reference internal" href="faq.html">
Frequently Asked Questions
</a>
</li>
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -256,19 +281,9 @@
<div class="toc-item">
-<div class="tocsection onthispage pt-5 pb-3">
- <i class="fas fa-list"></i> On this page
-</div>
<nav id="bd-toc-nav">
- <ul class="visible nav section-nav flex-column">
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#what-is-the-relationship-between-apache-arrow-datafusion-and-ballista">
- What is the relationship between Apache Arrow, DataFusion, and Ballista?
- </a>
- </li>
-</ul>
-
+
</nav>
</div>
@@ -276,7 +291,7 @@
<div class="tocsection editthispage">
- <a href="https://github.com/apache/arrow-datafusion/edit/main/docs/source/user-guide/faq.md">
+ <a href="https://github.com/apache/arrow-datafusion/edit/main/docs/source/user-guide/comparison.md">
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -313,17 +328,37 @@
specific language governing permissions and limitations
under the License.
-->
-<section id="frequently-asked-questions">
-<h1>Frequently Asked Questions<a class="headerlink" href="#frequently-asked-questions" title="Permalink to this heading">¶</a></h1>
-<section id="what-is-the-relationship-between-apache-arrow-datafusion-and-ballista">
-<h2>What is the relationship between Apache Arrow, DataFusion, and Ballista?<a class="headerlink" href="#what-is-the-relationship-between-apache-arrow-datafusion-and-ballista" title="Permalink to this heading">¶</a></h2>
-<p>Apache Arrow is a library which provides a standardized memory representation for columnar data. It also provides
-“kernels” for performing common operations on this data.</p>
-<p>DataFusion is a library for executing queries in-process using the Apache Arrow memory
-model and computational kernels. It is designed to run within a single process, using threads
-for parallel query execution.</p>
-<p><a class="reference external" href="https://github.com/apache/arrow-ballista">Ballista</a> is a distributed compute platform built on DataFusion.</p>
-</section>
+<section id="comparisons-to-other-projects">
+<h1>Comparisons to Other Projects<a class="headerlink" href="#comparisons-to-other-projects" title="Permalink to this heading">¶</a></h1>
+<p>When compared to similar systems, DataFusion typically is:</p>
+<ol class="arabic simple">
+<li><p>Targeted at developers, rather than end users / data scientists.</p></li>
+<li><p>Designed to be embedded, rather than a complete file based SQL system.</p></li>
+<li><p>Governed by the <a class="reference external" href="https://www.apache.org/">Apache Software Foundation</a> process, rather than a single company or individual.</p></li>
+<li><p>Implemented in <code class="docutils literal notranslate"><span class="pre">Rust</span></code>, rather than <code class="docutils literal notranslate"><span class="pre">C/C++</span></code></p></li>
+</ol>
+<p>Here is a comparison with similar projects that may help understand
+when DataFusion might be be suitable and unsuitable for your needs:</p>
+<ul class="simple">
+<li><p><a class="reference external" href="http://www.duckdb.org">DuckDB</a> is an open source, in process analytic database.
+Like DataFusion, it supports very fast execution, both from its custom file format
+and directly from parquet files. Unlike DataFusion, it is written in C/C++ and it
+is primarily used directly by users as a serverless database and query system rather
+than as a library for building such database systems.</p></li>
+<li><p><a class="reference external" href="http://pola.rs">Polars</a>: Polars is one of the fastest DataFrame
+libraries at the time of writing. Like DataFusion, it is also
+written in Rust and uses the Apache Arrow memory model, but unlike
+DataFusion it does not provide SQL nor as many extension points.</p></li>
+<li><p><a class="reference external" href="https://engineering.fb.com/2022/08/31/open-source/velox/">Facebook Velox</a>
+is an execution engine. Like DataFusion, Velox aims to
+provide a reusable foundation for building database-like systems. Unlike DataFusion,
+it is written in C/C++ and does not include a SQL frontend or planning /optimization
+framework.</p></li>
+<li><p><a class="reference external" href="https://github.com/datafuselabs/databend">Databend</a> is a complete
+database system. Like DataFusion it is also written in Rust and
+utilizes the Apache Arrow memory model, but unlike DataFusion it
+targets end-users rather than developers of other database systems.</p></li>
+</ul>
</section>
@@ -332,17 +367,17 @@ for parallel query execution.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="configs.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="users.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Configuration Settings</p>
+ <p class="prev-next-title">Known Users</p>
</div>
</a>
- <a class='right-next' id="next-link" href="../contributor-guide/index.html" title="next page">
+ <a class='right-next' id="next-link" href="integration.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Introduction</p>
+ <p class="prev-next-title">Integrations and Extensions</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 4ef6d14e50..7ca4e49a45 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/dataframe.html b/user-guide/dataframe.html
index fcfb9f5c42..cfa8699ef5 100644
--- a/user-guide/dataframe.html
+++ b/user-guide/dataframe.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/example-usage.html b/user-guide/example-usage.html
index 1663ca8fbd..ebabd60f01 100644
--- a/user-guide/example-usage.html
+++ b/user-guide/example-usage.html
@@ -34,8 +34,8 @@
<script src="../_static/sphinx_highlight.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Using DataFusion as a library" href="library.html" />
- <link rel="prev" title="Introduction" href="introduction.html" />
+ <link rel="next" title="Known Users" href="users.html" />
+ <link rel="prev" title="Features, and Usecases" href="introduction.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1 current active">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -496,13 +521,13 @@
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Introduction</p>
+ <p class="prev-next-title">Features, and Usecases</p>
</div>
</a>
- <a class='right-next' id="next-link" href="library.html" title="next page">
+ <a class='right-next' id="next-link" href="users.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Using DataFusion as a library</p>
+ <p class="prev-next-title">Known Users</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/user-guide/expressions.html b/user-guide/expressions.html
index 9300481ce1..2d07180150 100644
--- a/user-guide/expressions.html
+++ b/user-guide/expressions.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/faq.html b/user-guide/faq.html
index 30c5202403..88477cc6de 100644
--- a/user-guide/faq.html
+++ b/user-guide/faq.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/faq.html b/user-guide/integration.html
similarity index 77%
copy from user-guide/faq.html
copy to user-guide/integration.html
index 30c5202403..ddfb417e79 100644
--- a/user-guide/faq.html
+++ b/user-guide/integration.html
@@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
- <title>Frequently Asked Questions — Arrow DataFusion documentation</title>
+ <title>Integrations and Extensions — Arrow DataFusion documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
@@ -34,8 +34,8 @@
<script src="../_static/sphinx_highlight.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Introduction" href="../contributor-guide/index.html" />
- <link rel="prev" title="Configuration Settings" href="configs.html" />
+ <link rel="next" title="Using DataFusion as a library" href="library.html" />
+ <link rel="prev" title="Comparisons to Other Projects" href="comparison.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -164,8 +179,8 @@
Configuration Settings
</a>
</li>
- <li class="toctree-l1 current active">
- <a class="current reference internal" href="#">
+ <li class="toctree-l1">
+ <a class="reference internal" href="faq.html">
Frequently Asked Questions
</a>
</li>
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -263,8 +288,13 @@
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#what-is-the-relationship-between-apache-arrow-datafusion-and-ballista">
- What is the relationship between Apache Arrow, DataFusion, and Ballista?
+ <a class="reference internal nav-link" href="#language-bindings">
+ Language Bindings
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#integrations">
+ Integrations
</a>
</li>
</ul>
@@ -276,7 +306,7 @@
<div class="tocsection editthispage">
- <a href="https://github.com/apache/arrow-datafusion/edit/main/docs/source/user-guide/faq.md">
+ <a href="https://github.com/apache/arrow-datafusion/edit/main/docs/source/user-guide/integration.md">
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -313,16 +343,25 @@
specific language governing permissions and limitations
under the License.
-->
-<section id="frequently-asked-questions">
-<h1>Frequently Asked Questions<a class="headerlink" href="#frequently-asked-questions" title="Permalink to this heading">¶</a></h1>
-<section id="what-is-the-relationship-between-apache-arrow-datafusion-and-ballista">
-<h2>What is the relationship between Apache Arrow, DataFusion, and Ballista?<a class="headerlink" href="#what-is-the-relationship-between-apache-arrow-datafusion-and-ballista" title="Permalink to this heading">¶</a></h2>
-<p>Apache Arrow is a library which provides a standardized memory representation for columnar data. It also provides
-“kernels” for performing common operations on this data.</p>
-<p>DataFusion is a library for executing queries in-process using the Apache Arrow memory
-model and computational kernels. It is designed to run within a single process, using threads
-for parallel query execution.</p>
-<p><a class="reference external" href="https://github.com/apache/arrow-ballista">Ballista</a> is a distributed compute platform built on DataFusion.</p>
+<section id="integrations-and-extensions">
+<h1>Integrations and Extensions<a class="headerlink" href="#integrations-and-extensions" title="Permalink to this heading">¶</a></h1>
+<p>There are a number of community projects that extend DataFusion or
+provide integrations with other systems.</p>
+<section id="language-bindings">
+<h2>Language Bindings<a class="headerlink" href="#language-bindings" title="Permalink to this heading">¶</a></h2>
+<ul class="simple">
+<li><p><a class="reference external" href="https://github.com/datafusion-contrib/datafusion-c">datafusion-c</a></p></li>
+<li><p><a class="reference external" href="https://github.com/apache/arrow-datafusion-python">datafusion-python</a></p></li>
+<li><p><a class="reference external" href="https://github.com/datafusion-contrib/datafusion-ruby">datafusion-ruby</a></p></li>
+<li><p><a class="reference external" href="https://github.com/datafusion-contrib/datafusion-java">datafusion-java</a></p></li>
+</ul>
+</section>
+<section id="integrations">
+<h2>Integrations<a class="headerlink" href="#integrations" title="Permalink to this heading">¶</a></h2>
+<ul class="simple">
+<li><p><a class="reference external" href="https://github.com/datafusion-contrib/datafusion-bigtable">datafusion-bigtable</a></p></li>
+<li><p><a class="reference external" href="https://github.com/datafusion-contrib/datafusion-catalogprovider-glue">datafusion-catalogprovider-glue</a></p></li>
+</ul>
</section>
</section>
@@ -332,17 +371,17 @@ for parallel query execution.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="configs.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="comparison.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Configuration Settings</p>
+ <p class="prev-next-title">Comparisons to Other Projects</p>
</div>
</a>
- <a class='right-next' id="next-link" href="../contributor-guide/index.html" title="next page">
+ <a class='right-next' id="next-link" href="library.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Introduction</p>
+ <p class="prev-next-title">Using DataFusion as a library</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/user-guide/introduction.html b/user-guide/introduction.html
index 4b4cc9537b..5f27d98009 100644
--- a/user-guide/introduction.html
+++ b/user-guide/introduction.html
@@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
- <title>Introduction — Arrow DataFusion documentation</title>
+ <title>Features, and Usecases — Arrow DataFusion documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -323,8 +348,8 @@
specific language governing permissions and limitations
under the License.
-->
-<section id="introduction">
-<h1>Introduction<a class="headerlink" href="#introduction" title="Permalink to this heading">¶</a></h1>
+<section id="features-and-usecases">
+<h1>Features, and Usecases<a class="headerlink" href="#features-and-usecases" title="Permalink to this heading">¶</a></h1>
<p>DataFusion is a very fast, extensible query engine for building
high-quality data-centric systems in <a class="reference external" href="http://rustlang.org">Rust</a>,
using the <a class="reference external" href="https://arrow.apache.org">Apache Arrow</a> in-memory format.</p>
diff --git a/user-guide/library.html b/user-guide/library.html
index fe9fb132e1..424e8e8bc4 100644
--- a/user-guide/library.html
+++ b/user-guide/library.html
@@ -35,7 +35,7 @@
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="DataFusion Command-line SQL Utility" href="cli.html" />
- <link rel="prev" title="Example Usage" href="example-usage.html" />
+ <link rel="prev" title="Integrations and Extensions" href="integration.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -445,11 +470,11 @@ with <code class="docutils literal notranslate"><span class="pre">native</span><
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="example-usage.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="integration.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Example Usage</p>
+ <p class="prev-next-title">Integrations and Extensions</p>
</div>
</a>
<a class='right-next' id="next-link" href="cli.html" title="next page">
diff --git a/user-guide/sql/aggregate_functions.html b/user-guide/sql/aggregate_functions.html
index 8b3a5a6c48..08694a6a90 100644
--- a/user-guide/sql/aggregate_functions.html
+++ b/user-guide/sql/aggregate_functions.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/sql/data_types.html b/user-guide/sql/data_types.html
index 54a97aa284..717b5116d4 100644
--- a/user-guide/sql/data_types.html
+++ b/user-guide/sql/data_types.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/sql/ddl.html b/user-guide/sql/ddl.html
index 15b5b03bca..534aede5f7 100644
--- a/user-guide/sql/ddl.html
+++ b/user-guide/sql/ddl.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/sql/explain.html b/user-guide/sql/explain.html
index f0a74f5f99..fa1b9d4d43 100644
--- a/user-guide/sql/explain.html
+++ b/user-guide/sql/explain.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/sql/index.html b/user-guide/sql/index.html
index 40881dada0..37e475aad5 100644
--- a/user-guide/sql/index.html
+++ b/user-guide/sql/index.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/sql/information_schema.html b/user-guide/sql/information_schema.html
index 9b0dd909c5..b20bae2388 100644
--- a/user-guide/sql/information_schema.html
+++ b/user-guide/sql/information_schema.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/sql/scalar_functions.html b/user-guide/sql/scalar_functions.html
index 08f7bfc956..d131609377 100644
--- a/user-guide/sql/scalar_functions.html
+++ b/user-guide/sql/scalar_functions.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/sql/select.html b/user-guide/sql/select.html
index 3a1ec0e4a8..84266e0340 100644
--- a/user-guide/sql/select.html
+++ b/user-guide/sql/select.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/sql/sql_status.html b/user-guide/sql/sql_status.html
index 8d19060987..19818f995a 100644
--- a/user-guide/sql/sql_status.html
+++ b/user-guide/sql/sql_status.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/sql/subqueries.html b/user-guide/sql/subqueries.html
index aede5205e2..34687b4b1a 100644
--- a/user-guide/sql/subqueries.html
+++ b/user-guide/sql/subqueries.html
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../users.html">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../library.html">
Using DataFusion as a library
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
diff --git a/user-guide/faq.html b/user-guide/users.html
similarity index 69%
copy from user-guide/faq.html
copy to user-guide/users.html
index 30c5202403..838d3ad003 100644
--- a/user-guide/faq.html
+++ b/user-guide/users.html
@@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
- <title>Frequently Asked Questions — Arrow DataFusion documentation</title>
+ <title>Known Users — Arrow DataFusion documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
@@ -34,8 +34,8 @@
<script src="../_static/sphinx_highlight.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Introduction" href="../contributor-guide/index.html" />
- <link rel="prev" title="Configuration Settings" href="configs.html" />
+ <link rel="next" title="Comparisons to Other Projects" href="comparison.html" />
+ <link rel="prev" title="Example Usage" href="example-usage.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -74,7 +74,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Introduction
+ Features, and Usecases
</a>
</li>
<li class="toctree-l1">
@@ -82,6 +82,21 @@
Example Usage
</a>
</li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Known Users
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="comparison.html">
+ Comparisons to Other Projects
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="integration.html">
+ Integrations and Extensions
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="library.html">
Using DataFusion as a library
@@ -164,8 +179,8 @@
Configuration Settings
</a>
</li>
- <li class="toctree-l1 current active">
- <a class="current reference internal" href="#">
+ <li class="toctree-l1">
+ <a class="reference internal" href="faq.html">
Frequently Asked Questions
</a>
</li>
@@ -191,6 +206,11 @@
Communication
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/architecture.html">
+ Architecture
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/roadmap.html">
Roadmap
@@ -223,6 +243,11 @@
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion">
+ Github
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external" href="https://github.com/apache/arrow-datafusion/issues">
Issue tracker
@@ -256,19 +281,9 @@
<div class="toc-item">
-<div class="tocsection onthispage pt-5 pb-3">
- <i class="fas fa-list"></i> On this page
-</div>
<nav id="bd-toc-nav">
- <ul class="visible nav section-nav flex-column">
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#what-is-the-relationship-between-apache-arrow-datafusion-and-ballista">
- What is the relationship between Apache Arrow, DataFusion, and Ballista?
- </a>
- </li>
-</ul>
-
+
</nav>
</div>
@@ -276,7 +291,7 @@
<div class="tocsection editthispage">
- <a href="https://github.com/apache/arrow-datafusion/edit/main/docs/source/user-guide/faq.md">
+ <a href="https://github.com/apache/arrow-datafusion/edit/main/docs/source/user-guide/users.md">
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -313,17 +328,32 @@
specific language governing permissions and limitations
under the License.
-->
-<section id="frequently-asked-questions">
-<h1>Frequently Asked Questions<a class="headerlink" href="#frequently-asked-questions" title="Permalink to this heading">¶</a></h1>
-<section id="what-is-the-relationship-between-apache-arrow-datafusion-and-ballista">
-<h2>What is the relationship between Apache Arrow, DataFusion, and Ballista?<a class="headerlink" href="#what-is-the-relationship-between-apache-arrow-datafusion-and-ballista" title="Permalink to this heading">¶</a></h2>
-<p>Apache Arrow is a library which provides a standardized memory representation for columnar data. It also provides
-“kernels” for performing common operations on this data.</p>
-<p>DataFusion is a library for executing queries in-process using the Apache Arrow memory
-model and computational kernels. It is designed to run within a single process, using threads
-for parallel query execution.</p>
-<p><a class="reference external" href="https://github.com/apache/arrow-ballista">Ballista</a> is a distributed compute platform built on DataFusion.</p>
-</section>
+<section id="known-users">
+<h1>Known Users<a class="headerlink" href="#known-users" title="Permalink to this heading">¶</a></h1>
+<p>Here are some of the projects known to use DataFusion:</p>
+<ul class="simple">
+<li><p><a class="reference external" href="https://github.com/apache/arrow-ballista">Ballista</a> Distributed SQL Query Engine</p></li>
+<li><p><a class="reference external" href="https://github.com/blaze-init/blaze">Blaze</a> Spark accelerator with DataFusion at its core</p></li>
+<li><p><a class="reference external" href="https://github.com/CeresDB/ceresdb">CeresDB</a> Distributed Time-Series Database</p></li>
+<li><p><a class="reference external" href="https://github.com/cloudfuse-io/buzz-rust">Cloudfuse Buzz</a></p></li>
+<li><p><a class="reference external" href="https://github.com/cnosdb/cnosdb">CnosDB</a> Open Source Distributed Time Series Database</p></li>
+<li><p><a class="reference external" href="https://github.com/cube-js/cube.js/tree/master/rust">Cube Store</a></p></li>
+<li><p><a class="reference external" href="https://github.com/dask-contrib/dask-sql">Dask SQL</a> Distributed SQL query engine in Python</p></li>
+<li><p><a class="reference external" href="https://github.com/datafusion-contrib/datafusion-tui">datafusion-tui</a> Text UI for DataFusion</p></li>
+<li><p><a class="reference external" href="https://github.com/delta-io/delta-rs">delta-rs</a> Native Rust implementation of Delta Lake</p></li>
+<li><p><a class="reference external" href="https://github.com/flock-lab/flock">Flock</a></p></li>
+<li><p><a class="reference external" href="https://github.com/GreptimeTeam/greptimedb">GreptimeDB</a> Open Source & Cloud Native Distributed Time Series Database</p></li>
+<li><p><a class="reference external" href="https://github.com/influxdata/influxdb_iox">InfluxDB IOx</a> Time Series Database</p></li>
+<li><p><a class="reference external" href="https://github.com/kamu-data/kamu-cli/">Kamu</a> Planet-scale streaming data pipeline</p></li>
+<li><p><a class="reference external" href="https://github.com/parseablehq/parseable">Parseable</a> Log storage and observability platform</p></li>
+<li><p><a class="reference external" href="https://github.com/timvw/qv">qv</a> Quickly view your data</p></li>
+<li><p><a class="reference external" href="https://github.com/roapi/roapi">ROAPI</a></p></li>
+<li><p><a class="reference external" href="https://github.com/splitgraph/seafowl">Seafowl</a> CDN-friendly analytical database</p></li>
+<li><p><a class="reference external" href="https://synnada.ai/">Synnada</a> Streaming-first framework for data products</p></li>
+<li><p><a class="reference external" href="https://github.com/tensorbase/tensorbase">Tensorbase</a></p></li>
+<li><p><a class="reference external" href="https://vegafusion.io/">VegaFusion</a> Server-side acceleration for the <a class="reference external" href="https://vega.github.io/">Vega</a> visualization grammar</p></li>
+<li><p><a class="reference external" href="https://github.com/zinclabs/zincobserve">ZincObserve</a> Distributed cloud native observability platform</p></li>
+</ul>
</section>
@@ -332,17 +362,17 @@ for parallel query execution.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="configs.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="example-usage.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Configuration Settings</p>
+ <p class="prev-next-title">Example Usage</p>
</div>
</a>
- <a class='right-next' id="next-link" href="../contributor-guide/index.html" title="next page">
+ <a class='right-next' id="next-link" href="comparison.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Introduction</p>
+ <p class="prev-next-title">Comparisons to Other Projects</p>
</div>
<i class="fas fa-angle-right"></i>
</a>