You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ag...@apache.org on 2022/05/22 16:13:15 UTC
[arrow-datafusion] branch master updated: MINOR: Remove ballista files that are no longer needed (#2589)
This is an automated email from the ASF dual-hosted git repository.
agrove pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/master by this push:
new 2d7e1fb1e MINOR: Remove ballista files that are no longer needed (#2589)
2d7e1fb1e is described below
commit 2d7e1fb1eb746d6a3ddca0ee1bbd913d21ae08ba
Author: Andy Grove <ag...@apache.org>
AuthorDate: Sun May 22 10:13:11 2022 -0600
MINOR: Remove ballista files that are no longer needed (#2589)
---
.asf.yaml | 2 +-
CONTRIBUTING.md | 7 +-
README.md | 2 +-
benchmarks/README.md | 69 +----
dev/build-ballista-docker-arm64.sh | 34 ---
dev/build-ballista-docker.sh | 24 --
dev/build-ui.sh | 23 --
dev/docker/ballista-arm64.Dockerfile | 27 --
dev/docker/ballista-base.dockerfile | 99 -------
dev/docker/ballista-scheduler-ui.dockerfile | 38 ---
dev/docker/ballista.dockerfile | 96 -------
dev/integration-tests.sh | 28 --
dev/release/crate-deps.dot | 19 --
dev/release/crate-deps.svg | 216 ++++-----------
dev/release/rat_exclude_files.txt | 3 -
dev/release/update_change_log-ballista.sh | 34 ---
dev/update_ballista_versions.py | 105 -------
dev/update_datafusion_versions.py | 10 -
docs/source/index.rst | 2 -
docs/source/user-guide/distributed/clients/cli.rst | 111 --------
.../user-guide/distributed/clients/index.rst | 26 --
.../user-guide/distributed/clients/python.md | 22 --
docs/source/user-guide/distributed/clients/rust.md | 99 -------
.../distributed/deployment/cargo-install.md | 50 ----
.../distributed/deployment/configuration.md | 34 ---
.../distributed/deployment/docker-compose.md | 97 -------
.../user-guide/distributed/deployment/docker.md | 106 -------
.../user-guide/distributed/deployment/index.rst | 29 --
.../distributed/deployment/kubernetes.md | 308 ---------------------
.../distributed/deployment/raspberrypi.md | 129 ---------
docs/source/user-guide/distributed/index.rst | 26 --
docs/source/user-guide/distributed/introduction.md | 50 ----
32 files changed, 59 insertions(+), 1866 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index 7b21e927e..65592acbb 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -21,7 +21,7 @@ notifications:
pullrequests: github@arrow.apache.org
jira_options: link label worklog
github:
- description: "Apache Arrow DataFusion and Ballista query engines"
+ description: "Apache Arrow DataFusion SQL Query Engine"
homepage: https://arrow.apache.org/datafusion
labels:
- arrow
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 94ad6605c..465a68ef8 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -35,9 +35,6 @@ list to help you get started.
This section describes how you can get started at developing DataFusion.
-For information on developing with Ballista, see the
-[Ballista developer documentation](ballista/docs/README.md).
-
### Bootstrap environment
DataFusion is written in Rust and it uses a standard rust toolkit:
@@ -168,7 +165,7 @@ The benchmark will automatically remove any generated parquet file on exit, howe
### Upstream Benchmark Suites
-Instructions and tooling for running upstream benchmark suites against DataFusion and/or Ballista can be found in [benchmarks](./benchmarks).
+Instructions and tooling for running upstream benchmark suites against DataFusion can be found in [benchmarks](./benchmarks).
These are valuable for comparative evaluation against alternative Arrow implementations and query engines.
@@ -263,5 +260,5 @@ $ prettier --version
After you've confirmed your prettier version, you can format all the `.md` files:
```bash
-prettier -w {ballista,datafusion,datafusion-cli,datafusion-examples,dev,docs}/**/*.md
+prettier -w {datafusion,datafusion-cli,datafusion-examples,dev,docs}/**/*.md
```
diff --git a/README.md b/README.md
index 58a511da4..455c04704 100644
--- a/README.md
+++ b/README.md
@@ -60,7 +60,7 @@ Projects that adapt to or serve as plugins to DataFusion:
Here are some of the projects known to use DataFusion:
-- [Ballista](ballista) Distributed Compute Platform
+- [Ballista](https://github.com/apache/arrow-ballista) Distributed Compute Platform
- [Cloudfuse Buzz](https://github.com/cloudfuse-io/buzz-rust)
- [Cube Store](https://github.com/cube-js/cube.js/tree/master/rust)
- [delta-rs](https://github.com/delta-io/delta-rs)
diff --git a/benchmarks/README.md b/benchmarks/README.md
index fb541e4e7..c7e7f1264 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -17,7 +17,7 @@
under the License.
-->
-# DataFusion and Ballista Benchmarks
+# DataFusion Benchmarks
This crate contains benchmarks based on popular public data sets and open source benchmark suites, making it easy to
run real-world benchmarks to help with performance and scalability testing and for comparing performance with other Arrow
@@ -91,58 +91,6 @@ docker run -v /mnt:/mnt -it ballistacompute/spark-benchmarks:0.4.0-SNAPSHOT \
--partitions 64
```
-## Running the Ballista Benchmarks
-
-To run the benchmarks it is necessary to have at least one Ballista scheduler and one Ballista executor running.
-
-To run the scheduler from source:
-
-```bash
-cd $ARROW_HOME/ballista/rust/scheduler
-RUST_LOG=info cargo run --release
-```
-
-By default the scheduler will bind to `0.0.0.0` and listen on port 50050.
-
-To run the executor from source:
-
-```bash
-cd $ARROW_HOME/ballista/rust/executor
-RUST_LOG=info cargo run --release
-```
-
-By default the executor will bind to `0.0.0.0` and listen on port 50051.
-
-You can add SIMD(`cargo nightly` is required)/snmalloc/LTO flags to improve speed (with longer build times):
-
-```
-RUST_LOG=info RUSTFLAGS='-C target-cpu=native -C lto -C codegen-units=1 -C embed-bitcode' cargo run --release --bin executor --features "simd snmalloc" --target x86_64-unknown-linux-gnu
-```
-
-To run the benchmarks:
-
-```bash
-cd $ARROW_HOME/benchmarks
-cargo run --release --bin tpch benchmark ballista --host localhost --port 50050 --query 1 --path $(pwd)/data --format tbl
-```
-
-## Running the Ballista Benchmarks on docker-compose
-
-To start a Rust scheduler and executor using Docker Compose:
-
-```bash
-cd $ARROW_HOME
-./dev/build-rust.sh
-cd $ARROW_HOME/benchmarks
-docker-compose up
-```
-
-Then you can run the benchmark with:
-
-```bash
-docker-compose run ballista-client bash -c '/tpch benchmark ballista --host ballista-scheduler --port 50050 --query 1 --path /data --format tbl'
-```
-
## Expected output
The result of query 1 should produce the following output when executed against the SF=1 dataset.
@@ -178,20 +126,5 @@ Query 'fare_amt_by_passenger' iteration 1 took 7599 ms
Query 'fare_amt_by_passenger' iteration 2 took 7969 ms
```
-## Running the Ballista Loadtest
-
-```bash
- cargo run --bin tpch -- loadtest ballista-load
- --query-list 1,3,5,6,7,10,12,13
- --requests 200
- --concurrency 10
- --data-path /****
- --format parquet
- --host localhost
- --port 50050
- --sql-path /***
- --debug
-```
-
[1]: http://www.tpc.org/tpch/
[2]: https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page
diff --git a/dev/build-ballista-docker-arm64.sh b/dev/build-ballista-docker-arm64.sh
deleted file mode 100755
index 5d951773a..000000000
--- a/dev/build-ballista-docker-arm64.sh
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-if [ -z "${DOCKER_REPO}" ]; then
- echo "DOCKER_REPO env var must be set"
- exit -1
-fi
-cargo install cross
-cross build --release --target aarch64-unknown-linux-gnu
-rm -rf temp-ballista-docker
-mkdir temp-ballista-docker
-cp target/aarch64-unknown-linux-gnu/release/ballista-executor temp-ballista-docker
-cp target/aarch64-unknown-linux-gnu/release/ballista-scheduler temp-ballista-docker
-cp target/aarch64-unknown-linux-gnu/release/tpch temp-ballista-docker
-mkdir temp-ballista-docker/queries/
-cp benchmarks/queries/*.sql temp-ballista-docker/queries/
-docker buildx build --push -t $DOCKER_REPO/ballista-arm64 --platform=linux/arm64 -f dev/docker/ballista-arm64.Dockerfile temp-ballista-docker
-rm -rf temp-ballista-docker
\ No newline at end of file
diff --git a/dev/build-ballista-docker.sh b/dev/build-ballista-docker.sh
deleted file mode 100755
index 7add135d9..000000000
--- a/dev/build-ballista-docker.sh
+++ /dev/null
@@ -1,24 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-set -e
-
-. ./dev/build-set-env.sh
-docker build -t ballista-base:$BALLISTA_VERSION -f dev/docker/ballista-base.dockerfile .
-docker build -t ballista:$BALLISTA_VERSION -f dev/docker/ballista.dockerfile .
\ No newline at end of file
diff --git a/dev/build-ui.sh b/dev/build-ui.sh
deleted file mode 100755
index 070839702..000000000
--- a/dev/build-ui.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-set -e
-
-. ./dev/build-set-env.sh
-docker build -t ballista-scheduler-ui:$BALLISTA_VERSION -f dev/docker/ballista-scheduler-ui.dockerfile ballista/ui/scheduler
diff --git a/dev/docker/ballista-arm64.Dockerfile b/dev/docker/ballista-arm64.Dockerfile
deleted file mode 100644
index eb82c03e3..000000000
--- a/dev/docker/ballista-arm64.Dockerfile
+++ /dev/null
@@ -1,27 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-FROM arm64v8/ubuntu
-
-ADD ballista-scheduler /
-ADD ballista-executor /
-
-# Add benchmarks
-ADD tpch /
-RUN mkdir /queries
-ADD queries/*.sql /queries/
-
-ENV RUST_LOG=info
\ No newline at end of file
diff --git a/dev/docker/ballista-base.dockerfile b/dev/docker/ballista-base.dockerfile
deleted file mode 100644
index 504b33683..000000000
--- a/dev/docker/ballista-base.dockerfile
+++ /dev/null
@@ -1,99 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# Turn .dockerignore to .dockerallow by excluding everything and explicitly
-# allowing specific files and directories. This enables us to quickly add
-# dependency files to the docker content without scanning the whole directory.
-# This setup requires to all of our docker containers have arrow's source
-# as a mounted directory.
-
-
-# Base image extends debian:buster-slim
-FROM rust:1.59.0-buster AS builder
-
-RUN apt update && apt -y install musl musl-dev musl-tools libssl-dev openssl
-
-#NOTE: the following was copied from https://github.com/emk/rust-musl-builder/blob/master/Dockerfile under Apache 2.0 license
-
-# The OpenSSL version to use. We parameterize this because many Rust
-# projects will fail to build with 1.1.
-#ARG OPENSSL_VERSION=1.0.2r
-ARG OPENSSL_VERSION=1.1.1b
-
-# Build a static library version of OpenSSL using musl-libc. This is needed by
-# the popular Rust `hyper` crate.
-#
-# We point /usr/local/musl/include/linux at some Linux kernel headers (not
-# necessarily the right ones) in an effort to compile OpenSSL 1.1's "engine"
-# component. It's possible that this will cause bizarre and terrible things to
-# happen. There may be "sanitized" header
-RUN echo "Building OpenSSL" && \
- ls /usr/include/linux && \
- mkdir -p /usr/local/musl/include && \
- ln -s /usr/include/linux /usr/local/musl/include/linux && \
- ln -s /usr/include/x86_64-linux-gnu/asm /usr/local/musl/include/asm && \
- ln -s /usr/include/asm-generic /usr/local/musl/include/asm-generic && \
- cd /tmp && \
- curl -LO "https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz" && \
- tar xvzf "openssl-$OPENSSL_VERSION.tar.gz" && cd "openssl-$OPENSSL_VERSION" && \
- env CC=musl-gcc ./Configure no-shared no-zlib -fPIC --prefix=/usr/local/musl -DOPENSSL_NO_SECURE_MEMORY linux-x86_64 && \
- env C_INCLUDE_PATH=/usr/local/musl/include/ make depend && \
- env C_INCLUDE_PATH=/usr/local/musl/include/ make && \
- make install && \
- rm /usr/local/musl/include/linux /usr/local/musl/include/asm /usr/local/musl/include/asm-generic && \
- rm -r /tmp/*
-
-RUN echo "Building zlib" && \
- cd /tmp && \
- ZLIB_VERSION=1.2.12 && \
- curl -LO "http://zlib.net/zlib-$ZLIB_VERSION.tar.gz" && \
- tar xzf "zlib-$ZLIB_VERSION.tar.gz" && cd "zlib-$ZLIB_VERSION" && \
- CC=musl-gcc ./configure --static --prefix=/usr/local/musl && \
- make && make install && \
- rm -r /tmp/*
-
-RUN echo "Building libpq" && \
- cd /tmp && \
- POSTGRESQL_VERSION=11.2 && \
- curl -LO "https://ftp.postgresql.org/pub/source/v$POSTGRESQL_VERSION/postgresql-$POSTGRESQL_VERSION.tar.gz" && \
- tar xzf "postgresql-$POSTGRESQL_VERSION.tar.gz" && cd "postgresql-$POSTGRESQL_VERSION" && \
- CC=musl-gcc CPPFLAGS=-I/usr/local/musl/include LDFLAGS=-L/usr/local/musl/lib ./configure --with-openssl --without-readline --prefix=/usr/local/musl && \
- cd src/interfaces/libpq && make all-static-lib && make install-lib-static && \
- cd ../../bin/pg_config && make && make install && \
- rm -r /tmp/*
-
-ENV OPENSSL_DIR=/usr/local/musl/ \
- OPENSSL_INCLUDE_DIR=/usr/local/musl/include/ \
- DEP_OPENSSL_INCLUDE=/usr/local/musl/include/ \
- OPENSSL_LIB_DIR=/usr/local/musl/lib/ \
- OPENSSL_STATIC=1 \
- PQ_LIB_STATIC_X86_64_UNKNOWN_LINUX_MUSL=1 \
- PG_CONFIG_X86_64_UNKNOWN_LINUX_GNU=/usr/bin/pg_config \
- PKG_CONFIG_ALLOW_CROSS=true \
- PKG_CONFIG_ALL_STATIC=true \
- LIBZ_SYS_STATIC=1 \
- TARGET=musl
-
-# The content copied mentioned in the NOTE above ends here.
-
-## Download the target for static linking.
-RUN rustup target add x86_64-unknown-linux-musl
-RUN cargo install cargo-build-deps
-
-# prepare toolchain
-RUN rustup update && \
- rustup component add rustfmt
diff --git a/dev/docker/ballista-scheduler-ui.dockerfile b/dev/docker/ballista-scheduler-ui.dockerfile
deleted file mode 100644
index 99892e95e..000000000
--- a/dev/docker/ballista-scheduler-ui.dockerfile
+++ /dev/null
@@ -1,38 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# Turn .dockerignore to .dockerallow by excluding everything and explicitly
-# allowing specific files and directories. This enables us to quickly add
-# dependency files to the docker content without scanning the whole directory.
-# This setup requires to all of our docker containers have arrow's source
-# as a mounted directory.
-
-FROM node:14.16.0-alpine as build
-WORKDIR /app
-ENV PATH /app/node_modules/.bin:$PATH
-
-COPY package.json ./
-COPY yarn.lock ./
-RUN yarn
-
-COPY . ./
-RUN yarn build
-
-FROM nginx:stable-alpine
-COPY --from=build /app/build /usr/share/nginx/html
-EXPOSE 80
-CMD ["nginx", "-g", "daemon off;"]
diff --git a/dev/docker/ballista.dockerfile b/dev/docker/ballista.dockerfile
deleted file mode 100644
index dac756633..000000000
--- a/dev/docker/ballista.dockerfile
+++ /dev/null
@@ -1,96 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# Turn .dockerignore to .dockerallow by excluding everything and explicitly
-# allowing specific files and directories. This enables us to quickly add
-# dependency files to the docker content without scanning the whole directory.
-# This setup requires to all of our docker containers have arrow's source
-# as a mounted directory.
-
-ARG RELEASE_FLAG=--release
-FROM ballista-base:0.6.0 AS base
-WORKDIR /tmp/ballista
-RUN apt-get -y install cmake
-RUN cargo install cargo-chef --version 0.1.34
-
-FROM base as planner
-ADD Cargo.toml .
-COPY ballista ./ballista/
-COPY ballista-cli ./ballista-cli/
-COPY ballista-examples ./ballista-examples/
-COPY benchmarks ./benchmarks/
-COPY datafusion ./datafusion/
-COPY datafusion-cli ./datafusion-cli/
-COPY datafusion-examples ./datafusion-examples/
-RUN cargo chef prepare --recipe-path recipe.json
-
-FROM base as cacher
-COPY --from=planner /tmp/ballista/recipe.json recipe.json
-RUN cargo chef cook $RELEASE_FLAG --recipe-path recipe.json
-
-FROM base as builder
-RUN mkdir /tmp/ballista/ballista
-RUN mkdir /tmp/ballista/ballista-cli
-RUN mkdir /tmp/ballista/ballista-examples
-RUN mkdir /tmp/ballista/benchmarks
-RUN mkdir /tmp/ballista/datafusion
-RUN mkdir /tmp/ballista/datafusion-cli
-RUN mkdir /tmp/ballista/datafusion-examples
-ADD Cargo.toml .
-COPY ballista ./ballista/
-COPY ballista-cli ./ballista-cli/
-COPY ballista-examples ./ballista-examples/
-COPY benchmarks ./benchmarks/
-COPY datafusion ./datafusion/
-COPY datafusion-cli ./datafusion-cli/
-COPY datafusion-examples ./datafusion-examples/
-COPY --from=cacher /tmp/ballista/target target
-ARG RELEASE_FLAG=--release
-
-# force build.rs to run to generate configure_me code.
-ENV FORCE_REBUILD='true'
-RUN cargo build $RELEASE_FLAG
-
-# put the executor on /executor (need to be copied from different places depending on FLAG)
-ENV RELEASE_FLAG=${RELEASE_FLAG}
-RUN if [ -z "$RELEASE_FLAG" ]; then mv /tmp/ballista/target/debug/ballista-executor /executor; else mv /tmp/ballista/target/release/ballista-executor /executor; fi
-
-# put the scheduler on /scheduler (need to be copied from different places depending on FLAG)
-ENV RELEASE_FLAG=${RELEASE_FLAG}
-RUN if [ -z "$RELEASE_FLAG" ]; then mv /tmp/ballista/target/debug/ballista-scheduler /scheduler; else mv /tmp/ballista/target/release/ballista-scheduler /scheduler; fi
-
-# put the tpch on /tpch (need to be copied from different places depending on FLAG)
-ENV RELEASE_FLAG=${RELEASE_FLAG}
-RUN if [ -z "$RELEASE_FLAG" ]; then mv /tmp/ballista/target/debug/tpch /tpch; else mv /tmp/ballista/target/release/tpch /tpch; fi
-
-# Copy the binary into a new container for a smaller docker image
-FROM ballista-base:0.6.0
-
-COPY --from=builder /executor /
-
-COPY --from=builder /scheduler /
-
-COPY --from=builder /tpch /
-
-ADD benchmarks/run.sh /
-RUN mkdir /queries
-COPY benchmarks/queries/ /queries/
-
-ENV RUST_LOG=info
-ENV RUST_BACKTRACE=full
-
-CMD ["/executor"]
\ No newline at end of file
diff --git a/dev/integration-tests.sh b/dev/integration-tests.sh
deleted file mode 100755
index 3c1306319..000000000
--- a/dev/integration-tests.sh
+++ /dev/null
@@ -1,28 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-set -e
-./dev/build-ballista-docker.sh
-pushd benchmarks
-./tpch-gen.sh
-
-docker-compose up -d
-docker-compose run ballista-client /run.sh
-docker-compose down
-
-popd
diff --git a/dev/release/crate-deps.dot b/dev/release/crate-deps.dot
index 5176814e3..66dd71fb3 100644
--- a/dev/release/crate-deps.dot
+++ b/dev/release/crate-deps.dot
@@ -41,24 +41,5 @@ digraph G {
datafusion_proto -> datafusion
- ballista_core -> datafusion
- ballista_core -> datafusion_proto
-
- ballista_scheduler -> ballista_core
- ballista_scheduler -> datafusion
-
- ballista_executor -> ballista_core
- ballista_executor -> datafusion
-
- ballista -> datafusion
- ballista -> ballista_core
- ballista -> ballista_scheduler
- ballista -> ballista_executor
-
datafusion_cli -> datafusion
-
- ballista_cli -> datafusion
- ballista_cli -> datafusion_cli
- ballista_cli -> ballista
-
}
diff --git a/dev/release/crate-deps.svg b/dev/release/crate-deps.svg
index eb98a00ce..a4998cf80 100644
--- a/dev/release/crate-deps.svg
+++ b/dev/release/crate-deps.svg
@@ -4,262 +4,154 @@
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: G Pages: 1 -->
-<svg width="812pt" height="692pt"
- viewBox="0.00 0.00 812.33 692.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
-<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 688)">
+<svg width="568pt" height="404pt"
+ viewBox="0.00 0.00 567.79 404.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 400)">
<title>G</title>
-<polygon fill="white" stroke="transparent" points="-4,4 -4,-688 808.33,-688 808.33,4 -4,4"/>
+<polygon fill="white" stroke="transparent" points="-4,4 -4,-400 563.79,-400 563.79,4 -4,4"/>
<!-- datafusion_common -->
<g id="node1" class="node">
<title>datafusion_common</title>
-<ellipse fill="none" stroke="black" cx="395.39" cy="-18" rx="102.88" ry="18"/>
-<text text-anchor="middle" x="395.39" y="-14.3" font-family="Times,serif" font-size="14.00">datafusion_common</text>
+<ellipse fill="none" stroke="black" cx="319.55" cy="-18" rx="102.88" ry="18"/>
+<text text-anchor="middle" x="319.55" y="-14.3" font-family="Times,serif" font-size="14.00">datafusion_common</text>
</g>
<!-- datafusion_data_access -->
<g id="node2" class="node">
<title>datafusion_data_access</title>
-<ellipse fill="none" stroke="black" cx="685.39" cy="-234" rx="118.88" ry="18"/>
-<text text-anchor="middle" x="685.39" y="-230.3" font-family="Times,serif" font-size="14.00">datafusion_data_access</text>
+<ellipse fill="none" stroke="black" cx="146.55" cy="-234" rx="118.88" ry="18"/>
+<text text-anchor="middle" x="146.55" y="-230.3" font-family="Times,serif" font-size="14.00">datafusion_data_access</text>
</g>
<!-- datafusion_expr -->
<g id="node3" class="node">
<title>datafusion_expr</title>
-<ellipse fill="none" stroke="black" cx="282.39" cy="-90" rx="85.29" ry="18"/>
-<text text-anchor="middle" x="282.39" y="-86.3" font-family="Times,serif" font-size="14.00">datafusion_expr</text>
+<ellipse fill="none" stroke="black" cx="319.55" cy="-90" rx="85.29" ry="18"/>
+<text text-anchor="middle" x="319.55" y="-86.3" font-family="Times,serif" font-size="14.00">datafusion_expr</text>
</g>
<!-- datafusion_expr->datafusion_common -->
<g id="edge1" class="edge">
<title>datafusion_expr->datafusion_common</title>
-<path fill="none" stroke="black" d="M308.6,-72.76C323.78,-63.36 343.11,-51.39 359.67,-41.13"/>
-<polygon fill="black" stroke="black" points="361.88,-43.87 368.54,-35.63 358.19,-37.92 361.88,-43.87"/>
+<path fill="none" stroke="black" d="M319.55,-71.7C319.55,-63.98 319.55,-54.71 319.55,-46.11"/>
+<polygon fill="black" stroke="black" points="323.05,-46.1 319.55,-36.1 316.05,-46.1 323.05,-46.1"/>
</g>
<!-- datafusion_physical_expr -->
<g id="node4" class="node">
<title>datafusion_physical_expr</title>
-<ellipse fill="none" stroke="black" cx="127.39" cy="-162" rx="127.28" ry="18"/>
-<text text-anchor="middle" x="127.39" y="-158.3" font-family="Times,serif" font-size="14.00">datafusion_physical_expr</text>
+<ellipse fill="none" stroke="black" cx="176.55" cy="-162" rx="127.28" ry="18"/>
+<text text-anchor="middle" x="176.55" y="-158.3" font-family="Times,serif" font-size="14.00">datafusion_physical_expr</text>
</g>
<!-- datafusion_physical_expr->datafusion_common -->
<g id="edge2" class="edge">
<title>datafusion_physical_expr->datafusion_common</title>
-<path fill="none" stroke="black" d="M134.58,-143.8C143.96,-123.51 162.29,-90.25 188.39,-72 220.21,-49.75 260.61,-36.83 297.25,-29.34"/>
-<polygon fill="black" stroke="black" points="298.32,-32.7 307.48,-27.37 297,-25.82 298.32,-32.7"/>
+<path fill="none" stroke="black" d="M182.68,-143.8C190.29,-124.43 204.83,-92.93 225.55,-72 238.55,-58.87 255.49,-47.99 271.4,-39.57"/>
+<polygon fill="black" stroke="black" points="273.27,-42.55 280.61,-34.9 270.11,-36.3 273.27,-42.55"/>
</g>
<!-- datafusion_physical_expr->datafusion_expr -->
<g id="edge3" class="edge">
<title>datafusion_physical_expr->datafusion_expr</title>
-<path fill="none" stroke="black" d="M163.73,-144.59C186.16,-134.46 215.08,-121.4 238.75,-110.71"/>
-<polygon fill="black" stroke="black" points="240.2,-113.89 247.87,-106.59 237.32,-107.51 240.2,-113.89"/>
+<path fill="none" stroke="black" d="M210.08,-144.59C230.35,-134.66 256.37,-121.92 277.96,-111.36"/>
+<polygon fill="black" stroke="black" points="279.77,-114.37 287.22,-106.83 276.69,-108.08 279.77,-114.37"/>
</g>
<!-- datafusion_jit -->
<g id="node5" class="node">
<title>datafusion_jit</title>
-<ellipse fill="none" stroke="black" cx="383.39" cy="-162" rx="73.39" ry="18"/>
-<text text-anchor="middle" x="383.39" y="-158.3" font-family="Times,serif" font-size="14.00">datafusion_jit</text>
+<ellipse fill="none" stroke="black" cx="432.55" cy="-162" rx="73.39" ry="18"/>
+<text text-anchor="middle" x="432.55" y="-158.3" font-family="Times,serif" font-size="14.00">datafusion_jit</text>
</g>
<!-- datafusion_jit->datafusion_common -->
<g id="edge4" class="edge">
<title>datafusion_jit->datafusion_common</title>
-<path fill="none" stroke="black" d="M384.84,-143.87C386.88,-119.67 390.64,-75.21 393.07,-46.39"/>
-<polygon fill="black" stroke="black" points="396.58,-46.45 393.94,-36.19 389.61,-45.86 396.58,-46.45"/>
+<path fill="none" stroke="black" d="M432.92,-143.92C432.5,-124.66 429.18,-93.26 413.55,-72 403.16,-57.87 387.8,-46.98 372.49,-38.86"/>
+<polygon fill="black" stroke="black" points="374.03,-35.72 363.52,-34.39 370.91,-41.99 374.03,-35.72"/>
</g>
<!-- datafusion_jit->datafusion_expr -->
<g id="edge5" class="edge">
<title>datafusion_jit->datafusion_expr</title>
-<path fill="none" stroke="black" d="M359.96,-144.76C346.42,-135.38 329.19,-123.44 314.41,-113.19"/>
-<polygon fill="black" stroke="black" points="316.25,-110.21 306.04,-107.39 312.26,-115.96 316.25,-110.21"/>
+<path fill="none" stroke="black" d="M406.91,-145.12C391.38,-135.5 371.37,-123.1 354.42,-112.6"/>
+<polygon fill="black" stroke="black" points="356.22,-109.6 345.87,-107.31 352.53,-115.55 356.22,-109.6"/>
</g>
<!-- datafusion_row -->
<g id="node6" class="node">
<title>datafusion_row</title>
-<ellipse fill="none" stroke="black" cx="429.39" cy="-234" rx="81.49" ry="18"/>
-<text text-anchor="middle" x="429.39" y="-230.3" font-family="Times,serif" font-size="14.00">datafusion_row</text>
+<ellipse fill="none" stroke="black" cx="478.55" cy="-234" rx="81.49" ry="18"/>
+<text text-anchor="middle" x="478.55" y="-230.3" font-family="Times,serif" font-size="14.00">datafusion_row</text>
</g>
<!-- datafusion_row->datafusion_common -->
<g id="edge6" class="edge">
<title>datafusion_row->datafusion_common</title>
-<path fill="none" stroke="black" d="M444.62,-216.24C452.48,-206.44 461.27,-193.4 465.39,-180 470.09,-164.71 469.43,-159.48 465.39,-144 455.6,-106.49 431.78,-68.32 414.65,-44.2"/>
-<polygon fill="black" stroke="black" points="417.39,-42.01 408.68,-35.97 411.72,-46.12 417.39,-42.01"/>
+<path fill="none" stroke="black" d="M493.78,-216.24C501.64,-206.44 510.43,-193.4 514.55,-180 519.25,-164.71 521.01,-158.64 514.55,-144 490.23,-88.94 427.68,-55.53 379.92,-37.28"/>
+<polygon fill="black" stroke="black" points="380.88,-33.9 370.28,-33.72 378.45,-40.47 380.88,-33.9"/>
</g>
<!-- datafusion_row->datafusion_jit -->
<g id="edge7" class="edge">
<title>datafusion_row->datafusion_jit</title>
-<path fill="none" stroke="black" d="M418.25,-216.05C412.85,-207.84 406.25,-197.79 400.25,-188.66"/>
-<polygon fill="black" stroke="black" points="403.16,-186.71 394.74,-180.28 397.31,-190.55 403.16,-186.71"/>
+<path fill="none" stroke="black" d="M467.41,-216.05C462.01,-207.84 455.41,-197.79 449.41,-188.66"/>
+<polygon fill="black" stroke="black" points="452.32,-186.71 443.9,-180.28 446.47,-190.55 452.32,-186.71"/>
</g>
<!-- datafusion -->
<g id="node7" class="node">
<title>datafusion</title>
-<ellipse fill="none" stroke="black" cx="374.39" cy="-306" rx="59.59" ry="18"/>
-<text text-anchor="middle" x="374.39" y="-302.3" font-family="Times,serif" font-size="14.00">datafusion</text>
+<ellipse fill="none" stroke="black" cx="312.55" cy="-306" rx="59.59" ry="18"/>
+<text text-anchor="middle" x="312.55" y="-302.3" font-family="Times,serif" font-size="14.00">datafusion</text>
</g>
<!-- datafusion->datafusion_common -->
<g id="edge8" class="edge">
<title>datafusion->datafusion_common</title>
-<path fill="none" stroke="black" d="M430.68,-299.87C462.34,-293.78 499.42,-280.56 519.39,-252 554.53,-201.74 521.44,-158.92 457.39,-72 449.24,-60.94 438.35,-50.64 427.98,-42.13"/>
-<polygon fill="black" stroke="black" points="429.86,-39.15 419.84,-35.7 425.52,-44.64 429.86,-39.15"/>
+<path fill="none" stroke="black" d="M253.26,-302.98C176.17,-298.97 47.86,-287.07 18.55,-252 -63.22,-154.18 149.21,-71.47 258.85,-36.65"/>
+<polygon fill="black" stroke="black" points="259.99,-39.96 268.48,-33.63 257.89,-33.28 259.99,-39.96"/>
</g>
<!-- datafusion->datafusion_data_access -->
<g id="edge9" class="edge">
<title>datafusion->datafusion_data_access</title>
-<path fill="none" stroke="black" d="M420.95,-294.52C471.49,-283.15 552.88,-264.83 612.03,-251.51"/>
-<polygon fill="black" stroke="black" points="613.07,-254.86 622.06,-249.25 611.53,-248.04 613.07,-254.86"/>
+<path fill="none" stroke="black" d="M279.35,-291C254.89,-280.69 221.28,-266.51 193.99,-255"/>
+<polygon fill="black" stroke="black" points="195.2,-251.72 184.62,-251.05 192.48,-258.17 195.2,-251.72"/>
</g>
<!-- datafusion->datafusion_expr -->
<g id="edge10" class="edge">
<title>datafusion->datafusion_expr</title>
-<path fill="none" stroke="black" d="M359.17,-288.18C350.59,-278.15 340.09,-264.9 332.39,-252 314.53,-222.09 311.97,-213.2 301.39,-180 294.92,-159.71 290,-136.11 286.79,-118.32"/>
-<polygon fill="black" stroke="black" points="290.21,-117.58 285.05,-108.32 283.31,-118.77 290.21,-117.58"/>
+<path fill="none" stroke="black" d="M313.11,-287.85C314.32,-250.83 317.19,-163.18 318.65,-118.39"/>
+<polygon fill="black" stroke="black" points="322.16,-118.34 318.98,-108.23 315.16,-118.11 322.16,-118.34"/>
</g>
<!-- datafusion->datafusion_physical_expr -->
<g id="edge12" class="edge">
<title>datafusion->datafusion_physical_expr</title>
-<path fill="none" stroke="black" d="M343.35,-290.49C321.99,-280.26 293.12,-265.94 268.39,-252 230.73,-230.77 189.03,-204.02 160.81,-185.42"/>
-<polygon fill="black" stroke="black" points="162.42,-182.28 152.15,-179.68 158.55,-188.12 162.42,-182.28"/>
+<path fill="none" stroke="black" d="M308.82,-287.73C303.86,-268.29 293.45,-236.73 274.55,-216 262.24,-202.5 245.67,-191.74 229.66,-183.55"/>
+<polygon fill="black" stroke="black" points="230.86,-180.24 220.34,-179.02 227.8,-186.54 230.86,-180.24"/>
</g>
<!-- datafusion->datafusion_jit -->
<g id="edge11" class="edge">
<title>datafusion->datafusion_jit</title>
-<path fill="none" stroke="black" d="M359.57,-288.17C351.93,-278.35 343.39,-265.31 339.39,-252 334.79,-236.68 333.96,-231.05 339.39,-216 343.2,-205.43 350.15,-195.48 357.41,-187.12"/>
-<polygon fill="black" stroke="black" points="360.13,-189.33 364.37,-179.62 355,-184.56 360.13,-189.33"/>
+<path fill="none" stroke="black" d="M326.55,-288.43C347.48,-263.66 387.08,-216.8 411.51,-187.9"/>
+<polygon fill="black" stroke="black" points="414.48,-189.8 418.27,-179.9 409.14,-185.28 414.48,-189.8"/>
</g>
<!-- datafusion->datafusion_row -->
<g id="edge13" class="edge">
<title>datafusion->datafusion_row</title>
-<path fill="none" stroke="black" d="M387.42,-288.41C394.11,-279.91 402.38,-269.37 409.81,-259.91"/>
-<polygon fill="black" stroke="black" points="412.64,-261.98 416.07,-251.96 407.14,-257.66 412.64,-261.98"/>
+<path fill="none" stroke="black" d="M345.74,-291C370.79,-280.44 405.45,-265.82 433.08,-254.17"/>
+<polygon fill="black" stroke="black" points="434.68,-257.3 442.54,-250.19 431.96,-250.85 434.68,-257.3"/>
</g>
<!-- datafusion_proto -->
<g id="node8" class="node">
<title>datafusion_proto</title>
-<ellipse fill="none" stroke="black" cx="374.39" cy="-378" rx="89.08" ry="18"/>
-<text text-anchor="middle" x="374.39" y="-374.3" font-family="Times,serif" font-size="14.00">datafusion_proto</text>
+<ellipse fill="none" stroke="black" cx="221.55" cy="-378" rx="89.08" ry="18"/>
+<text text-anchor="middle" x="221.55" y="-374.3" font-family="Times,serif" font-size="14.00">datafusion_proto</text>
</g>
<!-- datafusion_proto->datafusion -->
<g id="edge14" class="edge">
<title>datafusion_proto->datafusion</title>
-<path fill="none" stroke="black" d="M374.39,-359.7C374.39,-351.98 374.39,-342.71 374.39,-334.11"/>
-<polygon fill="black" stroke="black" points="377.89,-334.1 374.39,-324.1 370.89,-334.1 377.89,-334.1"/>
-</g>
-<!-- ballista_core -->
-<g id="node9" class="node">
-<title>ballista_core</title>
-<ellipse fill="none" stroke="black" cx="368.39" cy="-450" rx="69.59" ry="18"/>
-<text text-anchor="middle" x="368.39" y="-446.3" font-family="Times,serif" font-size="14.00">ballista_core</text>
-</g>
-<!-- ballista_core->datafusion -->
-<g id="edge15" class="edge">
-<title>ballista_core->datafusion</title>
-<path fill="none" stroke="black" d="M326.02,-435.61C307.33,-427.35 287.18,-414.67 276.39,-396 268.38,-382.15 268.2,-373.74 276.39,-360 286.68,-342.74 304.76,-330.76 322.61,-322.62"/>
-<polygon fill="black" stroke="black" points="324.03,-325.82 331.88,-318.7 321.3,-319.37 324.03,-325.82"/>
-</g>
-<!-- ballista_core->datafusion_proto -->
-<g id="edge16" class="edge">
-<title>ballista_core->datafusion_proto</title>
-<path fill="none" stroke="black" d="M369.87,-431.7C370.53,-423.98 371.33,-414.71 372.06,-406.11"/>
-<polygon fill="black" stroke="black" points="375.56,-406.37 372.92,-396.1 368.58,-405.77 375.56,-406.37"/>
-</g>
-<!-- ballista_scheduler -->
-<g id="node10" class="node">
-<title>ballista_scheduler</title>
-<ellipse fill="none" stroke="black" cx="491.39" cy="-522" rx="94.78" ry="18"/>
-<text text-anchor="middle" x="491.39" y="-518.3" font-family="Times,serif" font-size="14.00">ballista_scheduler</text>
-</g>
-<!-- ballista_scheduler->datafusion -->
-<g id="edge18" class="edge">
-<title>ballista_scheduler->datafusion</title>
-<path fill="none" stroke="black" d="M494.17,-503.74C498.29,-472.21 502.58,-404.48 472.39,-360 461.01,-343.24 442.54,-331.32 424.68,-323.09"/>
-<polygon fill="black" stroke="black" points="426.01,-319.85 415.44,-319.11 423.24,-326.28 426.01,-319.85"/>
-</g>
-<!-- ballista_scheduler->ballista_core -->
-<g id="edge17" class="edge">
-<title>ballista_scheduler->ballista_core</title>
-<path fill="none" stroke="black" d="M462.86,-504.76C445.54,-494.91 423.27,-482.23 404.7,-471.66"/>
-<polygon fill="black" stroke="black" points="406.34,-468.57 395.92,-466.67 402.88,-474.65 406.34,-468.57"/>
-</g>
-<!-- ballista_executor -->
-<g id="node11" class="node">
-<title>ballista_executor</title>
-<ellipse fill="none" stroke="black" cx="250.39" cy="-522" rx="89.88" ry="18"/>
-<text text-anchor="middle" x="250.39" y="-518.3" font-family="Times,serif" font-size="14.00">ballista_executor</text>
-</g>
-<!-- ballista_executor->datafusion -->
-<g id="edge20" class="edge">
-<title>ballista_executor->datafusion</title>
-<path fill="none" stroke="black" d="M247.08,-503.9C241.98,-472.36 235.71,-404.19 266.39,-360 279.09,-341.71 299.94,-329.46 319.95,-321.39"/>
-<polygon fill="black" stroke="black" points="321.23,-324.65 329.36,-317.87 318.77,-318.1 321.23,-324.65"/>
-</g>
-<!-- ballista_executor->ballista_core -->
-<g id="edge19" class="edge">
-<title>ballista_executor->ballista_core</title>
-<path fill="none" stroke="black" d="M277.76,-504.76C294.22,-495 315.35,-482.46 333.07,-471.95"/>
-<polygon fill="black" stroke="black" points="335.17,-474.78 341.98,-466.67 331.59,-468.76 335.17,-474.78"/>
-</g>
-<!-- ballista -->
-<g id="node12" class="node">
-<title>ballista</title>
-<ellipse fill="none" stroke="black" cx="368.39" cy="-594" rx="44.69" ry="18"/>
-<text text-anchor="middle" x="368.39" y="-590.3" font-family="Times,serif" font-size="14.00">ballista</text>
-</g>
-<!-- ballista->datafusion -->
-<g id="edge21" class="edge">
-<title>ballista->datafusion</title>
-<path fill="none" stroke="black" d="M325.2,-588.69C268.21,-582 172.58,-566.99 151.39,-540 101.01,-475.84 142.07,-446.76 247.39,-360 268.62,-342.51 296.44,-330.03 320.63,-321.59"/>
-<polygon fill="black" stroke="black" points="322.02,-324.82 330.4,-318.34 319.81,-318.18 322.02,-324.82"/>
-</g>
-<!-- ballista->ballista_core -->
-<g id="edge22" class="edge">
-<title>ballista->ballista_core</title>
-<path fill="none" stroke="black" d="M368.39,-575.87C368.39,-551.67 368.39,-507.21 368.39,-478.39"/>
-<polygon fill="black" stroke="black" points="371.89,-478.19 368.39,-468.19 364.89,-478.19 371.89,-478.19"/>
-</g>
-<!-- ballista->ballista_scheduler -->
-<g id="edge23" class="edge">
-<title>ballista->ballista_scheduler</title>
-<path fill="none" stroke="black" d="M393.28,-578.83C410.69,-568.93 434.25,-555.52 453.92,-544.32"/>
-<polygon fill="black" stroke="black" points="455.69,-547.34 462.65,-539.35 452.23,-541.26 455.69,-547.34"/>
-</g>
-<!-- ballista->ballista_executor -->
-<g id="edge24" class="edge">
-<title>ballista->ballista_executor</title>
-<path fill="none" stroke="black" d="M344.22,-578.67C327.65,-568.83 305.37,-555.61 286.67,-544.52"/>
-<polygon fill="black" stroke="black" points="288.21,-541.37 277.83,-539.28 284.64,-547.39 288.21,-541.37"/>
+<path fill="none" stroke="black" d="M243.11,-360.41C255.19,-351.12 270.4,-339.42 283.51,-329.34"/>
+<polygon fill="black" stroke="black" points="285.97,-331.86 291.76,-322.99 281.7,-326.31 285.97,-331.86"/>
</g>
<!-- datafusion_cli -->
-<g id="node13" class="node">
+<g id="node9" class="node">
<title>datafusion_cli</title>
-<ellipse fill="none" stroke="black" cx="594.39" cy="-450" rx="74.99" ry="18"/>
-<text text-anchor="middle" x="594.39" y="-446.3" font-family="Times,serif" font-size="14.00">datafusion_cli</text>
+<ellipse fill="none" stroke="black" cx="403.55" cy="-378" rx="74.99" ry="18"/>
+<text text-anchor="middle" x="403.55" y="-374.3" font-family="Times,serif" font-size="14.00">datafusion_cli</text>
</g>
<!-- datafusion_cli->datafusion -->
-<g id="edge25" class="edge">
+<g id="edge15" class="edge">
<title>datafusion_cli->datafusion</title>
-<path fill="none" stroke="black" d="M581.21,-432.21C565.1,-412.65 536.18,-380.39 505.39,-360 481.01,-343.86 450.94,-331.26 425.76,-322.4"/>
-<polygon fill="black" stroke="black" points="426.61,-318.99 416.01,-319.07 424.35,-325.62 426.61,-318.99"/>
-</g>
-<!-- ballista_cli -->
-<g id="node14" class="node">
-<title>ballista_cli</title>
-<ellipse fill="none" stroke="black" cx="614.39" cy="-666" rx="59.59" ry="18"/>
-<text text-anchor="middle" x="614.39" y="-662.3" font-family="Times,serif" font-size="14.00">ballista_cli</text>
-</g>
-<!-- ballista_cli->datafusion -->
-<g id="edge26" class="edge">
-<title>ballista_cli->datafusion</title>
-<path fill="none" stroke="black" d="M631.99,-648.48C656.35,-623.72 697.39,-574.4 697.39,-523 697.39,-523 697.39,-523 697.39,-449 697.39,-339.22 539.08,-313.55 444.13,-308.03"/>
-<polygon fill="black" stroke="black" points="444.18,-304.53 434.01,-307.51 443.81,-311.52 444.18,-304.53"/>
-</g>
-<!-- ballista_cli->ballista -->
-<g id="edge28" class="edge">
-<title>ballista_cli->ballista</title>
-<path fill="none" stroke="black" d="M572.38,-653.05C528.24,-640.49 459.04,-620.8 413.7,-607.89"/>
-<polygon fill="black" stroke="black" points="414.42,-604.46 403.84,-605.09 412.5,-611.19 414.42,-604.46"/>
-</g>
-<!-- ballista_cli->datafusion_cli -->
-<g id="edge27" class="edge">
-<title>ballista_cli->datafusion_cli</title>
-<path fill="none" stroke="black" d="M612.79,-647.85C609.33,-610.83 601.13,-523.18 596.95,-478.39"/>
-<polygon fill="black" stroke="black" points="600.41,-477.86 596,-468.23 593.45,-478.51 600.41,-477.86"/>
+<path fill="none" stroke="black" d="M381.98,-360.41C369.91,-351.12 354.7,-339.42 341.59,-329.34"/>
+<polygon fill="black" stroke="black" points="343.39,-326.31 333.33,-322.99 339.12,-331.86 343.39,-326.31"/>
</g>
</g>
</svg>
diff --git a/dev/release/rat_exclude_files.txt b/dev/release/rat_exclude_files.txt
index b236d3a5d..98e0c6a38 100644
--- a/dev/release/rat_exclude_files.txt
+++ b/dev/release/rat_exclude_files.txt
@@ -15,7 +15,6 @@ ci/etc/*.patch
ci/vcpkg/*.patch
CHANGELOG.md
datafusion/CHANGELOG.md
-ballista/CHANGELOG.md
python/CHANGELOG.md
conbench/benchmarks.json
conbench/requirements.txt
@@ -110,8 +109,6 @@ requirements.txt
.gitattributes
rust-toolchain
benchmarks/queries/q*.sql
-ballista/rust/scheduler/testdata/*
-ballista/ui/scheduler/yarn.lock
python/rust-toolchain
python/requirements*.txt
**/testdata/*
diff --git a/dev/release/update_change_log-ballista.sh b/dev/release/update_change_log-ballista.sh
deleted file mode 100755
index 56a3a7af7..000000000
--- a/dev/release/update_change_log-ballista.sh
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-# Usage:
-# CHANGELOG_GITHUB_TOKEN=<TOKEN> ./update_change_log-ballista.sh master ballista-0.7.0 ballista-0.6.0
-
-RELEASE_BRANCH=$1
-RELEASE_TAG=$2
-BASE_TAG=$3
-
-SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-${SOURCE_DIR}/update_change_log.sh \
- ballista \
- "${BASE_TAG}" \
- --exclude-tags-regex "python-.+" \
- --future-release "${RELEASE_TAG}" \
- --release-branch "${RELEASE_BRANCH}"
diff --git a/dev/update_ballista_versions.py b/dev/update_ballista_versions.py
deleted file mode 100755
index 0fb3f4bf8..000000000
--- a/dev/update_ballista_versions.py
+++ /dev/null
@@ -1,105 +0,0 @@
-#!/usr/bin/env python
-
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Script that updates verions for ballista crates, locally
-#
-# dependencies:
-# pip install tomlkit
-
-import os
-import re
-import argparse
-from pathlib import Path
-import tomlkit
-
-
-def update_cargo_toml(cargo_toml: str, new_version: str):
- print(f'updating {cargo_toml}')
- with open(cargo_toml) as f:
- data = f.read()
-
- doc = tomlkit.parse(data)
- if "ballista/" in cargo_toml or "ballista-cli/" in cargo_toml:
- doc.get('package')['version'] = new_version
-
- # ballista crates also depend on each other
- ballista_deps = (
- 'ballista',
- 'ballista-core',
- 'ballista-executor',
- 'ballista-scheduler',
- 'ballista-cli',
- )
- for ballista_dep in ballista_deps:
- dep = doc.get('dependencies', {}).get(ballista_dep)
- if dep is not None:
- dep['version'] = new_version
- dep = doc.get('dev-dependencies', {}).get(ballista_dep)
- if dep is not None:
- dep['version'] = new_version
-
- with open(cargo_toml, 'w') as f:
- f.write(tomlkit.dumps(doc))
-
-
-def update_docker_compose(docker_compose_path: str, new_version: str):
- print(f'Updating ballista versions in {docker_compose_path}')
- with open(docker_compose_path, "r+") as fd:
- data = fd.read()
- pattern = re.compile(r'(^\s+image:\sballista:)\d+\.\d+\.\d+(-SNAPSHOT)?', re.MULTILINE)
- data = pattern.sub(r"\g<1>"+new_version, data)
- fd.truncate(0)
- fd.seek(0)
- fd.write(data)
-
-
-def main():
- parser = argparse.ArgumentParser(description='Update ballista crate versions.')
- parser.add_argument('new_version', type=str, help='new ballista version')
- args = parser.parse_args()
-
- repo_root = Path(__file__).parent.parent.absolute()
- ballista_crates = set([
- os.path.join(repo_root, rel_path, "Cargo.toml")
- for rel_path in [
- 'ballista-cli',
- 'ballista-examples',
- 'ballista/rust/core',
- 'ballista/rust/scheduler',
- 'ballista/rust/executor',
- 'ballista/rust/client',
- ]
- ])
- new_version = args.new_version
-
- print(f'Updating ballista versions in {repo_root} to {new_version}')
-
- for cargo_toml in ballista_crates:
- update_cargo_toml(cargo_toml, new_version)
-
- for path in (
- "benchmarks/docker-compose.yaml",
- "docs/source/user-guide/distributed/deployment/docker-compose.md",
- ):
- path = os.path.join(repo_root, path)
- update_docker_compose(path, new_version)
-
-
-if __name__ == "__main__":
- main()
diff --git a/dev/update_datafusion_versions.py b/dev/update_datafusion_versions.py
index 177c0b61d..2560f79dc 100755
--- a/dev/update_datafusion_versions.py
+++ b/dev/update_datafusion_versions.py
@@ -40,14 +40,6 @@ crates = {
'datafusion-row': 'datafusion/row/Cargo.toml'
}
-ballista_crates = {
- 'ballista-cli': 'ballista-cli/Cargo.toml',
- 'core': 'ballista/rust/core/Cargo.toml',
- 'client': 'ballista/rust/client/Cargo.toml',
- 'executor': 'ballista/rust/executor/Cargo.toml',
- 'scheduler': 'ballista/rust/scheduler/Cargo.toml',
-}
-
def update_datafusion_version(cargo_toml: str, new_version: str):
print(f'updating {cargo_toml}')
with open(cargo_toml) as f:
@@ -110,8 +102,6 @@ def main():
print(f'Updating datafusion dependency versions in {repo_root} to {new_version}')
for cargo_toml in crates.values():
update_downstream_versions(cargo_toml, new_version)
- for cargo_toml in ballista_crates.values():
- update_downstream_versions(cargo_toml, new_version)
update_docs("README.md", new_version)
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 07c930401..81cd572d6 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -43,7 +43,6 @@ Table of content
user-guide/library
user-guide/cli
user-guide/sql/index
- user-guide/distributed/index
user-guide/faq
.. _toc.specs:
@@ -64,7 +63,6 @@ Table of content
:caption: README
DataFusion <https://github.com/apache/arrow-datafusion/blob/master/README.md>
- Ballista <https://github.com/apache/arrow-datafusion/tree/master/ballista/README.md>
.. _toc.community:
diff --git a/docs/source/user-guide/distributed/clients/cli.rst b/docs/source/user-guide/distributed/clients/cli.rst
deleted file mode 100644
index d5cf30b8a..000000000
--- a/docs/source/user-guide/distributed/clients/cli.rst
+++ /dev/null
@@ -1,111 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-=======================
-Ballista Command-line
-=======================
-
-The Arrow Ballista CLI is a command-line interactive SQL utility that allows
-queries to be executed against CSV and Parquet files. It is a convenient way to
-try Ballista out with your own data sources.
-
-Install and run using Cargo
-===========================
-
-The easiest way to install Ballista CLI a spin is via `cargo install ballista-cli`.
-
-Run using Docker
-================
-
-There is no officially published Docker image for the Ballista CLI, so it is necessary to build from source
-instead.
-
-Use the following commands to clone this repository and build a Docker image containing the CLI tool. Note that there is :code:`.dockerignore` file in the root of the repository that may need to be deleted in order for this to work.
-
-.. code-block:: bash
-
- git clone https://github.com/apache/arrow-datafusion
- git checkout 8.0.0
- cd arrow-datafusion
- docker build -f ballista-cli/Dockerfile . --tag ballista-cli
- docker run -it -v $(your_data_location):/data ballista-cli
-
-
-Usage
-=====
-
-.. code-block:: bash
-
- Apache Arrow <de...@arrow.apache.org>
- Command Line Client for Ballista distributed query engine.
-
- USAGE:
- ballista-cli [OPTIONS]
-
- OPTIONS:
- -c, --batch-size <BATCH_SIZE> The batch size of each query, or use DataFusion default
- -f, --file <FILE>... Execute commands from file(s), then exit
- --format <FORMAT> [default: table] [possible values: csv, tsv, table, json,
- nd-json]
- -h, --help Print help information
- --host <HOST> Ballista scheduler host
- -p, --data-path <DATA_PATH> Path to your data, default to current directory
- --port <PORT> Ballista scheduler port
- -q, --quiet Reduce printing other than the results and work quietly
- -r, --rc <RC>... Run the provided files on startup instead of ~/.datafusionrc
- -V, --version Print version information
-
-Type `exit` or `quit` to exit the CLI.
-
-
-Registering Parquet Data Sources
-================================
-
-Parquet data sources can be registered by executing a :code:`CREATE EXTERNAL TABLE` SQL statement. It is not necessary to provide schema information for Parquet files.
-
-.. code-block:: sql
-
- CREATE EXTERNAL TABLE taxi
- STORED AS PARQUET
- LOCATION '/mnt/nyctaxi/tripdata.parquet';
-
-
-Registering CSV Data Sources
-============================
-
-CSV data sources can be registered by executing a :code:`CREATE EXTERNAL TABLE` SQL statement. It is necessary to provide schema information for CSV files since DataFusion does not automatically infer the schema when using SQL to query CSV files.
-
-.. code-block:: sql
-
- CREATE EXTERNAL TABLE test (
- c1 VARCHAR NOT NULL,
- c2 INT NOT NULL,
- c3 SMALLINT NOT NULL,
- c4 SMALLINT NOT NULL,
- c5 INT NOT NULL,
- c6 BIGINT NOT NULL,
- c7 SMALLINT NOT NULL,
- c8 INT NOT NULL,
- c9 BIGINT NOT NULL,
- c10 VARCHAR NOT NULL,
- c11 FLOAT NOT NULL,
- c12 DOUBLE NOT NULL,
- c13 VARCHAR NOT NULL
- )
- STORED AS CSV
- WITH HEADER ROW
- LOCATION '/path/to/aggregate_test_100.csv';
diff --git a/docs/source/user-guide/distributed/clients/index.rst b/docs/source/user-guide/distributed/clients/index.rst
deleted file mode 100644
index 6199bca5e..000000000
--- a/docs/source/user-guide/distributed/clients/index.rst
+++ /dev/null
@@ -1,26 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Clients
-=======
-
-.. toctree::
- :maxdepth: 2
-
- cli
- rust
- python
diff --git a/docs/source/user-guide/distributed/clients/python.md b/docs/source/user-guide/distributed/clients/python.md
deleted file mode 100644
index dac06408b..000000000
--- a/docs/source/user-guide/distributed/clients/python.md
+++ /dev/null
@@ -1,22 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Python
-
-Coming soon.
diff --git a/docs/source/user-guide/distributed/clients/rust.md b/docs/source/user-guide/distributed/clients/rust.md
deleted file mode 100644
index 8788d0583..000000000
--- a/docs/source/user-guide/distributed/clients/rust.md
+++ /dev/null
@@ -1,99 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Ballista Rust Client
-
-Ballista usage is very similar to DataFusion. Tha main difference is that the starting point is a `BallistaContext`
-instead of the DataFusion `SessionContext`. Ballista uses the same DataFrame API as DataFusion.
-
-The following code sample demonstrates how to create a `BallistaContext` to connect to a Ballista scheduler process.
-
-```rust
-let config = BallistaConfig::builder()
- .set("ballista.shuffle.partitions", "4")
- .build()?;
-
-// connect to Ballista scheduler
-let ctx = BallistaContext::remote("localhost", 50050, &config);
-```
-
-Here is a full example using the DataFrame API.
-
-```rust
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = BallistaConfig::builder()
- .set("ballista.shuffle.partitions", "4")
- .build()?;
-
- // connect to Ballista scheduler
- let ctx = BallistaContext::remote("localhost", 50050, &config);
-
- let testdata = datafusion::test_util::parquet_test_data();
-
- let filename = &format!("{}/alltypes_plain.parquet", testdata);
-
- // define the query using the DataFrame trait
- let df = ctx
- .read_parquet(filename)?
- .select_columns(&["id", "bool_col", "timestamp_col"])?
- .filter(col("id").gt(lit(1)))?;
-
- // print the results
- df.show().await?;
-
- Ok(())
-}
-```
-
-Here is a full example demonstrating SQL usage.
-
-```rust
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = BallistaConfig::builder()
- .set("ballista.shuffle.partitions", "4")
- .build()?;
-
- // connect to Ballista scheduler
- let ctx = BallistaContext::remote("localhost", 50050, &config);
-
- let testdata = datafusion::test_util::arrow_test_data();
-
- // register csv file with the execution context
- ctx.register_csv(
- "aggregate_test_100",
- &format!("{}/csv/aggregate_test_100.csv", testdata),
- CsvReadOptions::new(),
- )?;
-
- // execute the query
- let df = ctx.sql(
- "SELECT c1, MIN(c12), MAX(c12) \
- FROM aggregate_test_100 \
- WHERE c11 > 0.1 AND c11 < 0.9 \
- GROUP BY c1",
- )?;
-
- // print the results
- df.show().await?;
-
- Ok(())
-}
-```
diff --git a/docs/source/user-guide/distributed/deployment/cargo-install.md b/docs/source/user-guide/distributed/deployment/cargo-install.md
deleted file mode 100644
index 8e68278fd..000000000
--- a/docs/source/user-guide/distributed/deployment/cargo-install.md
+++ /dev/null
@@ -1,50 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Deploying a standalone Ballista cluster using cargo install
-
-A simple way to start a local cluster for testing purposes is to use cargo to install
-the scheduler and executor crates.
-
-```bash
-cargo install --locked ballista-scheduler
-cargo install --locked ballista-executor
-```
-
-With these crates installed, it is now possible to start a scheduler process.
-
-```bash
-RUST_LOG=info ballista-scheduler
-```
-
-The scheduler will bind to port 50050 by default.
-
-Next, start an executor processes in a new terminal session with the specified concurrency
-level.
-
-```bash
-RUST_LOG=info ballista-executor -c 4
-```
-
-The executor will bind to port 50051 by default. Additional executors can be started by
-manually specifying a bind port. For example:
-
-```bash
-RUST_LOG=info ballista-executor --bind-port 50052 -c 4
-```
diff --git a/docs/source/user-guide/distributed/deployment/configuration.md b/docs/source/user-guide/distributed/deployment/configuration.md
deleted file mode 100644
index 56ca92895..000000000
--- a/docs/source/user-guide/distributed/deployment/configuration.md
+++ /dev/null
@@ -1,34 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Configuration
-
-The rust executor and scheduler can be configured using toml files, environment variables and command line arguments. The specification for config options can be found in `rust/ballista/src/bin/[executor|scheduler]_config_spec.toml`.
-
-Those files fully define Ballista's configuration. If there is a discrepancy between this documentation and the files, assume those files are correct.
-
-To get a list of command line arguments, run the binary with `--help`
-
-There is an example config file at `ballista/rust/ballista/examples/example_executor_config.toml`
-
-The order of precedence for arguments is: default config file < environment variables < specified config file < command line arguments.
-
-The executor and scheduler will look for the default config file at `/etc/ballista/[executor|scheduler].toml` To specify a config file use the `--config-file` argument.
-
-Environment variables are prefixed by `BALLISTA_EXECUTOR` or `BALLISTA_SCHEDULER` for the executor and scheduler respectively. Hyphens in command line arguments become underscores. For example, the `--scheduler-host` argument for the executor becomes `BALLISTA_EXECUTOR_SCHEDULER_HOST`
diff --git a/docs/source/user-guide/distributed/deployment/docker-compose.md b/docs/source/user-guide/distributed/deployment/docker-compose.md
deleted file mode 100644
index c43e775ae..000000000
--- a/docs/source/user-guide/distributed/deployment/docker-compose.md
+++ /dev/null
@@ -1,97 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Starting a Ballista cluster using Docker Compose
-
-Docker Compose is a convenient way to launch a cluster when testing locally.
-
-## Build Docker image
-
-There is no officially published Docker image so it is currently necessary to build the image from source instead.
-
-Run the following commands to clone the source repository and build the Docker image.
-
-```bash
-git clone git@github.com:apache/arrow-datafusion.git -b 8.0.0
-cd arrow-datafusion
-./dev/build-ballista-docker.sh
-```
-
-This will create an image with the tag `ballista:0.7.0`.
-
-## Start a cluster
-
-The following Docker Compose example demonstrates how to start a cluster using one scheduler process and one
-executor process, with the scheduler using etcd as a backing store. A data volume is mounted into each container
-so that Ballista can access the host file system.
-
-```yaml
-version: "2.2"
-services:
- etcd:
- image: quay.io/coreos/etcd:v3.4.9
- command: "etcd -advertise-client-urls http://etcd:2379 -listen-client-urls http://0.0.0.0:2379"
- ballista-scheduler:
- image: ballista:0.7.0
- command: "/scheduler --config-backend etcd --etcd-urls etcd:2379 --bind-host 0.0.0.0 --bind-port 50050"
- ports:
- - "50050:50050"
- environment:
- - RUST_LOG=info
- volumes:
- - ./data:/data
- depends_on:
- - etcd
- ballista-executor:
- image: ballista:0.7.0
- command: "/executor --bind-host 0.0.0.0 --bind-port 50051 --scheduler-host ballista-scheduler"
- ports:
- - "50051:50051"
- environment:
- - RUST_LOG=info
- volumes:
- - ./data:/data
- depends_on:
- - ballista-scheduler
-```
-
-With the above content saved to a `docker-compose.yaml` file, the following command can be used to start the single
-node cluster.
-
-```bash
-docker-compose up
-```
-
-This should show output similar to the following:
-
-```bash
-$ docker-compose up
-Creating network "ballista-benchmarks_default" with the default driver
-Creating ballista-benchmarks_etcd_1 ... done
-Creating ballista-benchmarks_ballista-scheduler_1 ... done
-Creating ballista-benchmarks_ballista-executor_1 ... done
-Attaching to ballista-benchmarks_etcd_1, ballista-benchmarks_ballista-scheduler_1, ballista-benchmarks_ballista-executor_1
-ballista-executor_1 | [2021-08-28T15:55:22Z INFO ballista_executor] Running with config:
-ballista-executor_1 | [2021-08-28T15:55:22Z INFO ballista_executor] work_dir: /tmp/.tmpLVx39c
-ballista-executor_1 | [2021-08-28T15:55:22Z INFO ballista_executor] concurrent_tasks: 4
-ballista-scheduler_1 | [2021-08-28T15:55:22Z INFO ballista_scheduler] Ballista v0.7.0 Scheduler listening on 0.0.0.0:50050
-ballista-executor_1 | [2021-08-28T15:55:22Z INFO ballista_executor] Ballista v0.7.0 Rust Executor listening on 0.0.0.0:50051
-```
-
-The scheduler listens on port 50050 and this is the port that clients will need to connect to.
diff --git a/docs/source/user-guide/distributed/deployment/docker.md b/docs/source/user-guide/distributed/deployment/docker.md
deleted file mode 100644
index e00acac98..000000000
--- a/docs/source/user-guide/distributed/deployment/docker.md
+++ /dev/null
@@ -1,106 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Starting a Ballista cluster using Docker
-
-## Build Docker image
-
-There is no officially published Docker image so it is currently necessary to build the image from source instead.
-
-Run the following commands to clone the source repository and build the Docker image.
-
-```bash
-git clone git@github.com:apache/arrow-datafusion.git -b 8.0.0
-cd arrow-datafusion
-./dev/build-ballista-docker.sh
-```
-
-This will create an image with the tag `ballista:0.7.0`.
-
-### Start a Scheduler
-
-Start a scheduler using the following syntax:
-
-```bash
-docker run --network=host \
- -d ballista:0.7.0 \
- /scheduler --bind-port 50050
-```
-
-Run `docker ps` to check that the process is running:
-
-```
-$ docker ps
-CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
-1f3f8b5ed93a ballista:0.7.0 "/scheduler --bind-p…" 2 seconds ago Up 1 second tender_archimedes
-```
-
-Run `docker logs CONTAINER_ID` to check the output from the process:
-
-```
-$ docker logs 1f3f8b5ed93a
-[2021-08-28T15:45:11Z INFO ballista_scheduler] Ballista v0.7.0 Scheduler listening on 0.0.0.0:50050
-```
-
-### Start executors
-
-Start one or more executor processes. Each executor process will need to listen on a different port.
-
-```bash
-docker run --network=host \
- -d ballista:0.7.0 \
- /executor --external-host localhost --bind-port 50051
-```
-
-Use `docker ps` to check that both the scheduer and executor(s) are now running:
-
-```
-$ docker ps
-CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
-7c6941bb8dc0 ballista:0.7.0 "/executor --externa…" 3 seconds ago Up 2 seconds tender_goldberg
-1f3f8b5ed93a ballista:0.7.0 "/scheduler --bind-p…" 50 seconds ago Up 49 seconds tender_archimedes
-```
-
-Use `docker logs CONTAINER_ID` to check the output from the executor(s):
-
-```
-$ docker logs 7c6941bb8dc0
-[2021-08-28T15:45:58Z INFO ballista_executor] Running with config:
-[2021-08-28T15:45:58Z INFO ballista_executor] work_dir: /tmp/.tmpeyEM76
-[2021-08-28T15:45:58Z INFO ballista_executor] concurrent_tasks: 4
-[2021-08-28T15:45:58Z INFO ballista_executor] Ballista v0.7.0 Rust Executor listening on 0.0.0.0:50051
-```
-
-### Using etcd as backing store
-
-_NOTE: This functionality is currently experimental_
-
-Ballista can optionally use [etcd](https://etcd.io/) as a backing store for the scheduler. Use the following commands
-to launch the scheduler with this option enabled.
-
-```bash
-docker run --network=host \
- -d ballista:0.7.0 \
- /scheduler --bind-port 50050 \
- --config-backend etcd \
- --etcd-urls etcd:2379
-```
-
-Please refer to the [etcd](https://etcd.io/) web site for installation instructions. Etcd version 3.4.9 or later is
-recommended.
diff --git a/docs/source/user-guide/distributed/deployment/index.rst b/docs/source/user-guide/distributed/deployment/index.rst
deleted file mode 100644
index f5e41d010..000000000
--- a/docs/source/user-guide/distributed/deployment/index.rst
+++ /dev/null
@@ -1,29 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Start a Ballista Cluster
-========================
-
-.. toctree::
- :maxdepth: 2
-
- cargo-install
- docker
- docker-compose
- kubernetes
- raspberrypi
- configuration
diff --git a/docs/source/user-guide/distributed/deployment/kubernetes.md b/docs/source/user-guide/distributed/deployment/kubernetes.md
deleted file mode 100644
index 3047f3253..000000000
--- a/docs/source/user-guide/distributed/deployment/kubernetes.md
+++ /dev/null
@@ -1,308 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Deploying Ballista with Kubernetes
-
-Ballista can be deployed to any Kubernetes cluster using the following instructions. These instructions assume that
-you are already comfortable managing Kubernetes deployments.
-
-The Ballista deployment consists of:
-
-- k8s deployment for one or more scheduler processes
-- k8s deployment for one or more executor processes
-- k8s service to route traffic to the schedulers
-- k8s persistent volume and persistent volume claims to make local data accessible to Ballista
-- _(optional)_ a [keda](http://keda.sh) instance for autoscaling the number of executors
-
-## Testing locally
-
-[Microk8s](https://microk8s.io/) is recommended for installing a local k8s cluster. Once Microk8s is installed, DNS
-must be enabled using the following command.
-
-```bash
-microk8s enable dns
-```
-
-## Limitations
-
-Ballista is at an early stage of development and therefore has some significant limitations:
-
-- There is no support for shared object stores such as S3. All data must exist locally on each node in the
- cluster, including where any client process runs.
-- Only a single scheduler instance is currently supported unless the scheduler is configured to use `etcd` as a
- backing store.
-
-## Build Docker image
-
-There is no officially published Docker image so it is currently necessary to build the image from source instead.
-
-Run the following commands to clone the source repository and build the Docker image.
-
-```bash
-git clone git@github.com:apache/arrow-datafusion.git -b 8.0.0
-cd arrow-datafusion
-./dev/build-ballista-docker.sh
-```
-
-This will create an image with the tag `ballista:0.7.0`.
-
-## Publishing your images
-
-Once the images have been built, you can retag them and can push them to your favourite docker registry.
-
-```bash
-docker tag ballista:0.7.0 <your-repo>/ballista:0.7.0
-docker push <your-repo>/ballista:0.7.0
-```
-
-## Create Persistent Volume and Persistent Volume Claim
-
-Copy the following yaml to a `pv.yaml` file and apply to the cluster to create a persistent volume and a persistent
-volume claim so that the specified host directory is available to the containers. This is where any data should be
-located so that Ballista can execute queries against it.
-
-```yaml
-apiVersion: v1
-kind: PersistentVolume
-metadata:
- name: data-pv
- labels:
- type: local
-spec:
- storageClassName: manual
- capacity:
- storage: 10Gi
- accessModes:
- - ReadWriteOnce
- hostPath:
- path: "/mnt"
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
- name: data-pv-claim
-spec:
- storageClassName: manual
- accessModes:
- - ReadWriteOnce
- resources:
- requests:
- storage: 3Gi
-```
-
-To apply this yaml:
-
-```bash
-kubectl apply -f pv.yaml
-```
-
-You should see the following output:
-
-```bash
-persistentvolume/data-pv created
-persistentvolumeclaim/data-pv-claim created
-```
-
-## Deploying Ballista Scheduler and Executors
-
-Copy the following yaml to a `cluster.yaml` file and change `<your-image>` with the name of your Ballista Docker image.
-
-```yaml
-apiVersion: v1
-kind: Service
-metadata:
- name: ballista-scheduler
- labels:
- app: ballista-scheduler
-spec:
- ports:
- - port: 50050
- name: scheduler
- selector:
- app: ballista-scheduler
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: ballista-scheduler
-spec:
- replicas: 1
- selector:
- matchLabels:
- app: ballista-scheduler
- template:
- metadata:
- labels:
- app: ballista-scheduler
- ballista-cluster: ballista
- spec:
- containers:
- - name: ballista-scheduler
- image: <your-repo>/ballista:0.7.0
- command: ["/scheduler"]
- args: ["--bind-port=50050"]
- ports:
- - containerPort: 50050
- name: flight
- volumeMounts:
- - mountPath: /mnt
- name: data
- volumes:
- - name: data
- persistentVolumeClaim:
- claimName: data-pv-claim
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: ballista-executor
-spec:
- replicas: 2
- selector:
- matchLabels:
- app: ballista-executor
- template:
- metadata:
- labels:
- app: ballista-executor
- ballista-cluster: ballista
- spec:
- containers:
- - name: ballista-executor
- image: <your-repo>/ballista:0.7.0
- command: ["/executor"]
- args:
- - "--bind-port=50051"
- - "--scheduler-host=ballista-scheduler"
- - "--scheduler-port=50050"
- ports:
- - containerPort: 50051
- name: flight
- volumeMounts:
- - mountPath: /mnt
- name: data
- volumes:
- - name: data
- persistentVolumeClaim:
- claimName: data-pv-claim
-```
-
-```bash
-$ kubectl apply -f cluster.yaml
-```
-
-This should show the following output:
-
-```
-service/ballista-scheduler created
-deployment.apps/ballista-scheduler created
-deployment.apps/ballista-executor created
-```
-
-You can also check status by running `kubectl get pods`:
-
-```bash
-$ kubectl get pods
-NAME READY STATUS RESTARTS AGE
-ballista-executor-78cc5b6486-4rkn4 0/1 Pending 0 42s
-ballista-executor-78cc5b6486-7crdm 0/1 Pending 0 42s
-ballista-scheduler-879f874c5-rnbd6 0/1 Pending 0 42s
-```
-
-You can view the scheduler logs with `kubectl logs ballista-scheduler-0`:
-
-```
-$ kubectl logs ballista-scheduler-0
-[2021-02-19T00:24:01Z INFO scheduler] Ballista v0.7.0 Scheduler listening on 0.0.0.0:50050
-[2021-02-19T00:24:16Z INFO ballista::scheduler] Received register_executor request for ExecutorMetadata { id: "b5e81711-1c5c-46ec-8522-d8b359793188", host: "10.1.23.149", port: 50051 }
-[2021-02-19T00:24:17Z INFO ballista::scheduler] Received register_executor request for ExecutorMetadata { id: "816e4502-a876-4ed8-b33f-86d243dcf63f", host: "10.1.23.150", port: 50051 }
-```
-
-## Port Forwarding
-
-If you want to run applications outside of the cluster and have them connect to the scheduler then it is necessary to
-set up port forwarding.
-
-First, check that the `ballista-scheduler` service is running.
-
-```bash
-$ kubectl get services
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
-kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 26h
-ballista-scheduler ClusterIP 10.152.183.21 <none> 50050/TCP 24m
-```
-
-Use the following command to set up port-forwarding.
-
-```bash
-kubectl port-forward service/ballista-scheduler 50050:50050
-```
-
-## Deleting the Ballista cluster
-
-Run the following kubectl command to delete the cluster.
-
-```bash
-kubectl delete -f cluster.yaml
-```
-
-## Adding autoscaling for executors
-
-Ballista supports autoscaling for executors through [Keda](http://keda.sh). Keda allows scaling a deployment
-through custom metrics which are exposed through the Ballista scheduler, and it can even scale the number of
-executors down to 0 if there is no activity in the cluster.
-
-Keda can be installed in your kubernetes cluster through a single command line:
-
-```bash
-kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.3.0/keda-2.3.0.yaml
-```
-
-Once you have deployed Keda on your cluster, you can now deploy a new kubernetes object called `ScaledObject`
-which will let Keda know how to scale your executors. In order to do that, copy the following YAML into a
-`scale.yaml` file:
-
-```yaml
-apiVersion: keda.sh/v1alpha1
-kind: ScaledObject
-metadata:
- name: ballista-executor
-spec:
- scaleTargetRef:
- name: ballista-executor
- minReplicaCount: 0
- maxReplicaCount: 5
- triggers:
- - type: external
- metadata:
- # Change this DNS if the scheduler isn't deployed in the "default" namespace
- scalerAddress: ballista-scheduler.default.svc.cluster.local:50050
-```
-
-And then deploy it into the cluster:
-
-```bash
-kubectl apply -f scale.yaml
-```
-
-If the cluster is inactive, Keda will now scale the number of executors down to 0, and will scale them up when
-you launch a query. Please note that Keda will perform a scan once every 30 seconds, so it might take a bit to
-scale the executors.
-
-Please visit Keda's [documentation page](https://keda.sh/docs/2.3/concepts/scaling-deployments/) for more information.
diff --git a/docs/source/user-guide/distributed/deployment/raspberrypi.md b/docs/source/user-guide/distributed/deployment/raspberrypi.md
deleted file mode 100644
index 3bf36c722..000000000
--- a/docs/source/user-guide/distributed/deployment/raspberrypi.md
+++ /dev/null
@@ -1,129 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Running Ballista on Raspberry Pi
-
-The Raspberry Pi single-board computer provides a fun and relatively inexpensive way to get started with distributed
-computing.
-
-These instructions have been tested using an Ubuntu Linux desktop as the host, and a
-[Raspberry Pi 4 Model B](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/) with 4 GB RAM as the target.
-
-## Preparing the Raspberry Pi
-
-We recommend installing the 64-bit version of [Ubuntu for Raspberry Pi](https://ubuntu.com/raspberry-pi).
-
-The Rust implementation of Arrow does not work correctly on 32-bit ARM architectures
-([issue](https://github.com/apache/arrow-rs/issues/109)).
-
-## Cross Compiling DataFusion for the Raspberry Pi
-
-We do not yet publish official Docker images as part of the release process, although we do plan to do this in the
-future ([issue #228](https://github.com/apache/arrow-datafusion/issues/228)).
-
-Although it is technically possible to build DataFusion directly on a Raspberry Pi, it really isn't very practical.
-It is much faster to use [cross](https://github.com/rust-embedded/cross) to cross-compile from a more powerful
-desktop computer.
-
-Docker must be installed and the Docker daemon must be running before cross-compiling with cross. See the
-[cross](https://github.com/rust-embedded/cross) project for more detailed instructions.
-
-Run the following command to install cross.
-
-```bash
-cargo install cross
-```
-
-From the root of the DataFusion project, run the following command to cross-compile for ARM 64 architecture.
-
-```bash
-cross build --release --target aarch64-unknown-linux-gnu
-```
-
-It is even possible to cross-test from your desktop computer:
-
-```bash
-cross test --target aarch64-unknown-linux-gnu
-```
-
-## Deploying the binaries to Raspberry Pi
-
-You should now be able to copy the executable to the Raspberry Pi using scp on Linux. You will need to change the IP
-address in these commands to be the IP address for your Raspberry Pi. The easiest way to find this is to connect a
-keyboard and monitor to the Pi and run `ifconfig`.
-
-```bash
-scp ./target/aarch64-unknown-linux-gnu/release/ballista-scheduler ubuntu@10.0.0.186:
-scp ./target/aarch64-unknown-linux-gnu/release/ballista-executor ubuntu@10.0.0.186:
-```
-
-Finally, ssh into the Pi and make the binaries executable:
-
-```bash
-ssh ubuntu@10.0.0.186
-chmod +x ballista-scheduler ballista-executor
-```
-
-It is now possible to run the Ballista scheduler and executor natively on the Pi.
-
-## Docker
-
-Using Docker's `buildx` cross-platform functionality, we can also build a docker image targeting ARM64
-from any desktop environment. This will require write access to a Docker repository
-on [Docker Hub](https://hub.docker.com/) because the resulting Docker image will be pushed directly
-to the repo.
-
-```bash
-DOCKER_REPO=myrepo ./dev/build-ballista-docker-arm64.sh
-```
-
-On the Raspberry Pi:
-
-```bash
-docker pull myrepo/ballista-arm64
-```
-
-Run a scheduler:
-
-```bash
-docker run -it myrepo/ballista-arm64 /ballista-scheduler
-```
-
-Run an executor:
-
-```bash
-docker run -it myrepo/ballista-arm64 /ballista-executor
-```
-
-Run the benchmarks:
-
-```bash
-docker run -it myrepo/ballista-arm64 \
- /tpch benchmark datafusion --query=1 --path=/path/to/data --format=parquet \
- --concurrency=24 --iterations=1 --debug --host=ballista-scheduler --bind-port=50050
-```
-
-Note that it will be necessary to mount appropriate volumes into the containers and also configure networking
-so that the Docker containers can communicate with each other. This can be achieved using Docker compose or Kubernetes.
-
-## Kubernetes
-
-With Docker images built using the instructions above, it is now possible to deploy Ballista to a Kubernetes cluster
-running on one of more Raspberry Pi computers. Refer to the instructions in the [Kubernetes](kubernetes.md) chapter
-for more information, and remember to change the Docker image name to `myrepo/ballista-arm64`.
diff --git a/docs/source/user-guide/distributed/index.rst b/docs/source/user-guide/distributed/index.rst
deleted file mode 100644
index abb3c7b15..000000000
--- a/docs/source/user-guide/distributed/index.rst
+++ /dev/null
@@ -1,26 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Ballista Distributed Compute
-============================
-
-.. toctree::
- :maxdepth: 2
-
- introduction
- deployment/index
- clients/index
diff --git a/docs/source/user-guide/distributed/introduction.md b/docs/source/user-guide/distributed/introduction.md
deleted file mode 100644
index 77db6261c..000000000
--- a/docs/source/user-guide/distributed/introduction.md
+++ /dev/null
@@ -1,50 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Overview
-
-Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is
-built on an architecture that allows other programming languages to be supported as first-class citizens without paying
-a penalty for serialization costs.
-
-The foundational technologies in Ballista are:
-
-- [Apache Arrow](https://arrow.apache.org/) memory model and compute kernels for efficient processing of data.
-- [Apache Arrow Flight Protocol](https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/) for efficient data transfer between processes.
-- [Google Protocol Buffers](https://developers.google.com/protocol-buffers) for serializing query plans.
-- [DataFusion](https://github.com/apache/arrow-datafusion/) for query execution.
-
-## How does this compare to Apache Spark?
-
-Although Ballista is largely inspired by Apache Spark, there are some key differences.
-
-- The choice of Rust as the main execution language means that memory usage is deterministic and avoids the overhead
- of GC pauses.
-- Ballista is designed from the ground up to use columnar data, enabling a number of efficiencies such as vectorized
- processing (SIMD and GPU) and efficient compression. Although Spark does have some columnar support, it is still
- largely row-based today.
-- The combination of Rust and Arrow provides excellent memory efficiency and memory usage can be 5x - 10x lower than
- Apache Spark in some cases, which means that more processing can fit on a single node, reducing the overhead of
- distributed compute.
-- The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors
- in any programming language with minimal serialization overhead.
-
-## Status
-
-Ballista is still in the early stages of development but is capable of executing complex analytical queries at scale.