You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ag...@apache.org on 2022/10/14 14:39:35 UTC
[arrow-ballista] branch master updated: Update README in preparation for 0.9.0 release (#318)
This is an automated email from the ASF dual-hosted git repository.
agrove pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-ballista.git
The following commit(s) were added to refs/heads/master by this push:
new f08db4dc Update README in preparation for 0.9.0 release (#318)
f08db4dc is described below
commit f08db4dc211a65433f7e1f58c36010667636e4a5
Author: Andy Grove <an...@gmail.com>
AuthorDate: Fri Oct 14 08:39:29 2022 -0600
Update README in preparation for 0.9.0 release (#318)
* Update README
* copy roadmap from df
* Update README.md
Co-authored-by: Daniël Heres <da...@gmail.com>
* minor edits
Co-authored-by: Daniël Heres <da...@gmail.com>
---
README.md | 100 +++++++++++++++++++++++++++++++++++++++++---------------------
1 file changed, 66 insertions(+), 34 deletions(-)
diff --git a/README.md b/README.md
index f2a73c13..b8814515 100644
--- a/README.md
+++ b/README.md
@@ -17,63 +17,95 @@
under the License.
-->
-# Ballista: Distributed Compute with Rust, Apache Arrow, and DataFusion
+# Ballista: Distributed SQL Query Engine, built on Apache Arrow
-Ballista is a distributed SQL query engine primarily implemented in Rust, and powered by Apache Arrow and
-DataFusion. It is built on an architecture that allows other programming languages (such as Python, C++, and
-Java) to be supported as first-class citizens without paying a penalty for serialization costs.
+Ballista is a distributed SQL query engine powered by the Rust implementation of [Apache Arrow](arrow) and
+[DataFusion](datafusion).
-The foundational technologies in Ballista are:
+If you are looking for documentation for a released version of Ballista, please refer to the
+[Ballista User Guide](user-guide).
-- [Apache Arrow](https://arrow.apache.org/) memory model and compute kernels for efficient processing of data.
-- [DataFusion Query Engine](https://github.com/apache/arrow-datafusion) for query execution
-- [Apache Arrow Flight Protocol](https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/) for efficient
- data transfer between processes.
-- [Google Protocol Buffers](https://developers.google.com/protocol-buffers) for serializing query plans, with [plans to
- eventually use substrait.io here](https://github.com/apache/arrow-ballista/issues/32).
+## Overview
Ballista implements a similar design to Apache Spark (particularly Spark SQL), but there are some key differences:
-- The choice of Rust as the main execution language avoids the overhead of GC pauses.
+- The choice of Rust as the main execution language avoids the overhead of GC pauses and results in deterministic
+ processing times.
- Ballista is designed from the ground up to use columnar data, enabling a number of efficiencies such as vectorized
- processing (SIMD and GPU) and efficient compression. Although Spark does have some columnar support, it is still
+ processing (SIMD) and efficient compression. Although Spark does have some columnar support, it is still
largely row-based today.
- The combination of Rust and Arrow provides excellent memory efficiency and memory usage can be 5x - 10x lower than
Apache Spark in some cases, which means that more processing can fit on a single node, reducing the overhead of
distributed compute.
-- The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors
- in any programming language with minimal serialization overhead.
+- The use of Apache Arrow as the memory model and network protocol means that data can be exchanged efficiently between
+ executors using the [Flight Protocol](flight), and between clients and schedulers/executors using the
+ [Flight SQL Protocol](flight-sql)
-Ballista can be deployed as a standalone cluster and also supports [Kubernetes](https://kubernetes.io/). In either
-case, the scheduler can be configured to use [etcd](https://etcd.io/) as a backing store to (eventually) provide
-redundancy in the case of a scheduler failing.
+## Features
-# Project Status and Roadmap
+- Supports HDFS as well as cloud object stores. S3 is supported today and GCS and Azure support is planned.
+- DataFrame and SQL APIs available from Python and Rust.
+- Clients can connect to a Ballista cluster using [Flight SQL](flight-sql).
+- JDBC support via Arrow Flight SQL JDBC Driver
+- Scheduler web interface and REST UI for monitoring query progress and viewing query plans and metrics.
+- Support for Docker, Docker Compose, and Kubernetes deployment, as well as manual deployment on bare metal.
-Ballista is currently a proof-of-concept and provides batch execution of SQL queries. Although it is already capable of
-executing complex queries, it is not yet scalable or robust.
+# Getting Started
-There is an excellent discussion in https://github.com/apache/arrow-ballista/issues/30 about the future of the project
-and we encourage you to participate and add your feedback there if you are interested in using or contributing to
-Ballista.
+The easiest way to get started is to run one of the standalone or distributed [examples](./examples/README.md). After
+that, refer to the [Getting Started Guide](ballista/rust/client/README.md).
-The current initiatives being considered are:
+## Project Status
-- Continue to improve the current batch-based execution
-- Add support for low-latency query execution based on a streaming model
-- Adopt [substrait.io](https://substrait.io/) to allow other query engines to be integrated
+Ballista supports a wide range of SQL, including CTEs, Joins, and Subqueries and can execute complex queries at scale.
-# Getting Started
+Refer to the [DataFusion SQL Reference](https://arrow.apache.org/datafusion/user-guide/sql/index.html) for more
+information on supported SQL.
-The easiest way to get started is to run one of the standalone or distributed [examples](./examples/README.md). After
-that, refer to the [Getting Started Guide](ballista/rust/client/README.md).
+Ballista is maturing quickly and is now working towards being production ready. See the following roadmap for more details.
+
+## Roadmap
+
+There is an excellent discussion in https://github.com/apache/arrow-ballista/issues/30 about the future of the project,
+and we encourage you to participate and add your feedback there if you are interested in using or contributing to
+Ballista.
+
+The current focus is on the following items:
+
+- Make production ready
+ - Shuffle file cleanup
+ - Periodically
+ - Add gRPC & REST interfaces for clients/UI to actively call the cleanup for a job or the whole system
+ - Fill functional gaps between DataFusion and Ballista
+ - Improve task scheduling and data exchange efficiency
+ - Better error handling
+ - Schedule restart
+ - Improve monitoring, logging, and metrics
+ - Auto scaling support
+ - Better configuration management
+- All-at-once job task scheduling
+- Shuffle improvement
+ - Shuffle memory control
+ - Improve shuffle IO to avoid producing too many files
+ - Support sort-based shuffle
+ - Support range partition
+ - Support broadcast shuffle
+- Support for multi-scheduler deployments. Initially for resiliency and fault tolerance but ultimately to support
+ sharding for scalability and more efficient caching.
+- Executor deployment grouping based on resource allocation
## Architecture Overview
-- Refer to the [developer documentation](docs/developer) for the [Architecture Overview](/docs/developer/architecture.md)
-- Watch the [Ballista: Distributed Compute with Rust and Apache Arrow](https://www.youtube.com/watch?v=ZZHQaOap9pQ)
- talk from the New York Open Statistical Programming Meetup (Feb 2021)
+There are currently no up-to-date architecture documents available. You can get a general overview of the architecture
+by watching the [Ballista: Distributed Compute with Rust and Apache Arrow](ballista-talk) talk from the New York Open
+Statistical Programming Meetup (Feb 2021).
## Contribution Guide
Please see [Contribution Guide](CONTRIBUTING.md) for information about contributing to DataFusion.
+
+[arrow]: https://arrow.apache.org/
+[datafusion]: https://github.com/apache/arrow-datafusion
+[flight]: https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/
+[flight-sql]: https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/
+[ballista-talk]: https://www.youtube.com/watch?v=ZZHQaOap9pQ