You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by al...@apache.org on 2021/10/19 16:25:33 UTC
[arrow-datafusion] branch master updated: Add Roadmap to
Documentation (#1104)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/master by this push:
new ff243a4 Add Roadmap to Documentation (#1104)
ff243a4 is described below
commit ff243a40e84a0bf86b69c976ff0ed317fae6df64
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Tue Oct 19 12:25:26 2021 -0400
Add Roadmap to Documentation (#1104)
* Add Roadmap
* Fix English, add comments from xudong963
* Add datafusion-cli thoughts
* add more links
* Apply suggestions from code review
Co-authored-by: Loïc Sharma <73...@users.noreply.github.com>
Co-authored-by: QP Hou <da...@gmail.com>
* Incorporate comments from QP Hou
* prettier
* Update docs/source/specification/roadmap.md
Co-authored-by: Daniël Heres <da...@gmail.com>
* Apply suggestions from code review
Co-authored-by: Carlos <wx...@gmail.com>
Co-authored-by: rdettai <rd...@gmail.com>
Co-authored-by: Loïc Sharma <73...@users.noreply.github.com>
Co-authored-by: QP Hou <da...@gmail.com>
Co-authored-by: Daniël Heres <da...@gmail.com>
Co-authored-by: Carlos <wx...@gmail.com>
Co-authored-by: rdettai <rd...@gmail.com>
---
README.md | 4 ++
docs/source/index.rst | 1 +
docs/source/specification/roadmap.md | 99 ++++++++++++++++++++++++++++++++++++
3 files changed, 104 insertions(+)
diff --git a/README.md b/README.md
index 458f197..e1f96f0 100644
--- a/README.md
+++ b/README.md
@@ -356,6 +356,10 @@ are mapped to Arrow types according to the following table
| `CUSTOM` | _Not yet supported_ |
| `ARRAY` | _Not yet supported_ |
+# Roadmap
+
+Please see [Roadmap](docs/source/specification/roadmap.md) for information of where the project is headed.
+
# Architecture Overview
There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 6956d0b..bf6b250 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -52,6 +52,7 @@ Table of content
:maxdepth: 1
:caption: Specification
+ specification/roadmap
specification/invariants
specification/output-field-name-semantic
diff --git a/docs/source/specification/roadmap.md b/docs/source/specification/roadmap.md
new file mode 100644
index 0000000..520815b
--- /dev/null
+++ b/docs/source/specification/roadmap.md
@@ -0,0 +1,99 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Roadmap
+
+This document describes high level goals of the DataFusion and
+Ballista development community. It is not meant to restrict
+possibilities, but rather help newcomers understand the broader
+context of where the community is headed, and inspire
+additional contributions.
+
+DataFusion and Ballista are part of the [Apache
+Arrow](https://arrow.apache.org/) project and governed by the Apache
+Software Foundation governance model. These projects are entirely
+driven by volunteers, and we welcome contributions for items not on
+this roadmap. However, before submitting a large PR, we strongly
+suggest you start a coversation using a github issue or the
+dev@arrow.apache.org mailing list to make review efficient and avoid
+surprises.
+
+# DataFusion
+
+DataFusion's goal is to become the embedded query engine of choice
+for new analytic applications, by leveraging the unique features of
+[Rust](https://www.rust-lang.org/) and [Apache Arrow](https://arrow.apache.org/)
+to provide:
+
+1. Best-in-class single node query performance
+2. A Declarative SQL query interface compatible with PostgreSQL
+3. A Dataframe API, similar to those offered by Pandas and Spark
+4. A Procedural API for programatically creating and running execution plans
+5. High performance, data race free, erogonomic extensibility points at at every layer
+
+## Additional SQL Language Features
+
+- Complete support list on [status](https://github.com/apache/arrow-datafusion/blob/master/README.md#status)
+- Timestamp Arithmetic [#194](https://github.com/apache/arrow-datafusion/issues/194)
+- SQL Parser extension point [#533](https://github.com/apache/arrow-datafusion/issues/533)
+- Support for nested structures (fields, lists, structs) [#119](https://github.com/apache/arrow-datafusion/issues/119)
+- Remaining Set Operators (`INTERSECT` / `EXCEPT`) [#1082](https://github.com/apache/arrow-datafusion/issues/1082)
+- Run all queries from the TPCH benchmark (see [milestone](https://github.com/apache/arrow-datafusion/milestone/2) for more details)
+
+## Query Optimizer
+
+- Additional constant folding / partial evaluation [#1070](https://github.com/apache/arrow-datafusion/issues/1070)
+- More sophisticated cost based optimizer for join ordering
+- Implement advanced query optimization framework (Tokomak) #440
+
+## Datasources
+
+- Better support for reading data from remote filesystems (e.g. S3) without caching it locally [#907](https://github.com/apache/arrow-datafusion/issues/907) [#1060](https://github.com/apache/arrow-datafusion/issues/1060)
+- Support for partitioned datasources [#1139](https://github.com/apache/arrow-datafusion/issues/1139) and make the integration of other table formats (Delta, Iceberg...) simpler
+- Improve performances of file format datasources (parallelize file listings, async Arrow readers, file chunk prefetching capability...)
+
+## Runtime / Infrastructure
+
+- Migrate to some sort of arrow2 based implementation (see [milestone](https://github.com/apache/arrow-datafusion/milestone/3) for more details)
+- Add DataFusion to h2oai/db-benchmark [147](https://github.com/apache/arrow-datafusion/issues/147)
+- Improve build time [348](https://github.com/apache/arrow-datafusion/issues/348)
+
+## Resource Management
+
+- Finer grain control and limit of runtime memory [#587](https://github.com/apache/arrow-datafusion/issues/587) and CPU usage [#54](https://github.com/apache/arrow-datafusion/issues/64)
+
+## Python Interface
+
+TBD
+
+## DataFusion CLI (`datafusion-cli`)
+
+Note: There are some additional thoughts on a datafusion-cli vision on [#1096](https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-939418770).
+
+- Better abstraction between REPL parsing and queries so that commands are separated and handled correctly
+- Connect to the `Statistics` subsystem and have the cli print out more stats for query debugging, etc.
+- Improved error handling for interactive use and shell scripting usage
+- publishing to apt, brew, and possible NuGet registry so that people can use it more easily
+- adopt a shorter name, like dfcli?
+
+## Ballista
+
+# Vision
+
+TBD